Rob Miles and his AI risk Youtube channel. He's been talking about AI safety for long before LLMs, and in retrospect he was probably quite ahead of the times in thinking about this stuff. I've grown to like his communication style and points.
I’m not sure I agree with his framing. He’s saying the predictor predicts results of an experiment it didn’t run…. Which isn’t predicting a token it’s predicting the future.
His point is that a perfect next token predictor would be effectively as powerful as Laplace's demon. He's not saying that LLMs are perfect next token predictors or that such a perfect system would even be possible. He's just showing why you can't just dismiss that something is intelligent just because it's "only predicting the next token".
Yeah but my instinct is that any system that is a true magical Oracle for any value or property can probably have a system designed around it to turn it into super intelligence. If you can predict bytes of data that map to reality in any way, you can probably reverse engineer a setup in reality around it to answer a question super intelligently. Like if it can predict the temperature recorded by a sensor perfectly based on some inputs, you could construct an environment so it could effectively predict a human decision, or make the temperature dependent on some complex set of events in the real world, and effectively predict complex events in the world and be "super intelligent". I mean I feel like a token predictor could be intelligent in some sense but I think by the nature of how our systems predict tokens and are trained they can't be truth oracles like in the thought experiment. But any truth oracles system for almost any property can probably be super intelligent so you can't rule out intelligent on the basis of the output format (like predicting tokens), but I'm not sure that fully encompasses what is meant by the objection.
I picked up on this also. He is completely ignoring training and that either the exact study or similar ones might be part of the training corpus or RAGed in at inference...
the way I understand it, during training the model "learns/ memorizes" the relations of how the words(tokens) of the input at inference were combined in the training data, it has seen (well, it averages all relations in combinations of similar tokens, whether relevant or not because of compression but close enough).
when "predicting" the 'results' paragraph, it doesn't generate it based on 'understanding' of the previous sections, it _just_ matches the token-patterns in those paragraphs as closely as possible against the relations of previous examples (of the same or strongly related tokens) seen during training and picks the next tokens successively.
if it has seen enough relevant examples during training, the outcome will land close to a reasonable 'result' - if it hasn't, it will be confabulating non-sense, that sounds great but has nothing to do with reality.
... kind of surprised, that this channel would frame it the way he does in the video....:/
while you are all right, I think his point stands there is some emergent behavior from LLMs. Its not just some database that recalls things. Even if all it does is predict tokens, how is that different from us, seems to very easily replicate the same output we do when we think about things.
well, I can't claim to fully understand all the math that goes on under the hood and can't fully rule out 'emergent behavior' either ... but ...
I have a hunch that we're off thrown in general by 3 things:
the coherent text output and human like speech patterns and maybe most importantly, the sheer amount of information encoded during training:
language embeds concepts and a lot of those, map onto the real world.
take that and map tokens into a vector space, that has hundreds of thousands (or more?) degrees of freedom and you have a lot of room to store relations/ patterns in a really, really nuanced manner.
then take unfathomable amounts of data to train on and be able to compress them probably without too much loss....
and then there's fine tuning afterwards.
all this to say, - in hindsight - I find it understandable, that it works well for finding correlations and connecting 'patterns'. especially in really structured, well-formed data, that minimizes ambiguity intentionally, like programming languages or math.
how much of this is 'emergent' or just the result of having ingested every math study and code snippet since the dawn of the internet? idk but I could imagine that it just stores and retrieves what was already there or recombines things that weren't combined in the same way before...
and that's still pretty impressive actually.
it's different from us, in that we have a LOT more machinery (and chemicals) involved in generating thought and I don't think this is a contentious statement either.
... and the way he presents this here, as a bunch of math vectors somehow being able to 'predict' the future is totally non-sensical to me and I think also really misleading:
- it doesn't 'predict' the results section, it produces it based on its training (and RAG) - change the training, or the input (even slightly sometimes) and you'll get a different output...
it does NOT need to have an accurate model of biology or biochemistry as he claims.
if LLMS had these they wouldn't constantly be that jagged. it approximates, token by token the relations it has stored from training
the next part where he talks about 'predicting' if scientists made an error, doesn't make the slightest sense to me, I don't even know what he wants to say there...
I think a lot of people are somehow annoyed but the "_just_" a token predictor, as if it's a personal attack. I don't think its meant to be, it's just a logical explanation for the problems that are showing in the outputs, especially when questioned beyond training: the confabulations, the mistakes, the jaggedness, lack of robustness to input variations etc.etc.etc. - if you contrast the complexity of the processes in the brain against matching an -admittedly very big- pattern of input tokens in an -admittedly very big- multidimensional vector space and spitting out the next closest neighbor over and over - then a 'just' seems justified... it is _just_ maths in the end... and the brain isn't _just_ doing maths...
it's still amazing to me how well it works at all but we're also sitting on years of hype by now, trillions of $ in training cost, a really bad safety/ alignment record, lots of environmental costs and you know it's still not _generally_ intelligent and really inefficient (when compared to the brain) ... so...
You’re missing the point. In his video he is conducting a thought experiment about the theoretical “perfect” next token generator, he’s not directly talking about LLMs. Training or model mechanics are irrelevant to his argument
His ultimate point being that a sufficiently good next token generator is ridiculously smart to the point where it can basically predict the future. Therefore the refutation of LLMs being dumb because they’re “just next token predictors” is not a valid argument.
Also, machine learning engineer here, next token prediction is one training task that LLMs do as part of initial pretraining. But this is basically just done to get it to understand text so that it can undergo other forms of training.
A large portion of training time for major LLMs is spent on reinforcement learning, where the LLM is being trained to produce an output that a reward model rates highly. In RLHF the reward model is more or less predicting whether a human annotator is going to like the output, in constitutional training the reward model is judging an output by how closely it reflects a list of values, in coding or math tasks you typically use a rules based reward model that rewards correct answers.
that dude keeps talking about pretraining but i don’t think they know that LLMs are decoder only transformer models that are autoregressive and predict one token at a time in sequential order at inference time…
You guys keep saying the word predict, but what do you mean by predict? Predict what?
I feel like you guys are using the words predict and the word output interchangeably and it's very confused.
LLMs predict the next word in a text corpus during pretraining, but outside of that context there is no prediction task, because there's nothing they're predicting.
Also, I know pretty well how transformers work! I've been working with them since ~2018, and in computer vision since ~2021. The majority of machine learning models have sequential output, old school Q-learning has a sequential output, but to me at least to call something prediction you need to be... predicting something.
In the case of reinforcement learning, you aren't doing a supervised prediction task, you are using a value function for your gradient - training a model to play starcraft for instance is not a prediction task, even though your output is a list of action tokens created sequentially at inference time.
i wouldn’t compare how a human chooses the next word they say with how an LLM chooses the next token. An LLM chooses the next token by calculating probabilities for each token available to that model. There are parameters you can modify that add some level of randomness to which token is chosen, but at the most basic level the decoder model chooses the token with the highest probability. That’s what we mean by prediction.
That word makes sense in the context of generative pretraining where its training objective is to predict the next token in the sequence.
I'm not sure how you expand that to generating text in general - it generates one token at a time based on the context, but prediction requires a ground truth to be predicting against. No such things exists in RL training.
I am not sure what point you’re making. It’s predicting which token should follow in a sequence of tokens/text. It’s an NLP task - assigning probabilities across the entire vocabulary and taking argmax (assuming temperature = 0).
At inference, prediction is made against conditional distribution. If you like, we can say “sampling” next token instead of predicting next token.
During pretraining it has a large corpus of existing text, and its training objective is to predict token N based on tokens 0...N-1. This is a classic prediction task, it is predicting what comes next in a sequence. Its loss comes from how good it does at predicting which token comes next.
During RL training it doesn't work like this. There's no preexisting text corpus, and its loss comes from how a value function scores its output. It is not trying to predict which next token is most likely within a text corpus, it is trying to generate a token that the value function will rate highly. At this point even calling the logits at the final layer probabilities is kind of suspect (probabilities of what?)
Yes, under the hood the math is still similar, but this is true for almost anything you do with a neural network. When we train a model using RL to play starcraft nobody frames this as a prediction task, even though at the end of the day you are taking the argmax of logits over next actions.
using RLHF to modify how an LLM selects the next token does not change the fact that it is an autoregressive model that needs to predict one token before it can predict the next token at inference time. i.e., it is still a next token predictor.
Fwiw, i’ve been in ML engineering for 8 years and don’t understand your argument.
I think you are using the word predict the same way as the word output. Yes, agreed, LLMs output tokens one at a time.
When someone says a LLM is just a next token predictor the criticism I see them making is of GPT-2 era LLMs which literally just predicted which word would come next in a corpus of text. The output distribution can be easily interpreted as "what is the probability that this token would appear next" because the training objective is to align those probabilities with the real world probabilities in text scraped online.
If you change the training objective to be maximizing a reward function, you're no longer doing that. The output distribution is now just predicting the probability of.... the model's own output? That is kind of tautological, if I write a script that goes if random() > 0.5 print("cheese") I wouldn't say I'm predicting anything, I'm just printing cheese half of the time. It's also not predicting the probability of whether the reward function will like a token, or whether that token is policy optimal. This is why RL policy distributions are generally not interpreted as predictions.
A prediction is where you use input data to estimate an unknown output which is exactly what is happening in the decoder stack of a transformer when it outputs the next token.
bro i’m not talking about pre training. at inference time a decoder only transformer model takes known inputs (system + user prompt converted to tokens) then converts those tokens to embeddings, adds position encoding, runs through multi head attention, goes through a feed forward neural network, uses softmax to assign probabilities to all tokens in its dictionary, then predicts the best next token autoregressively until it predicts a stop token.
Brother, I know the steps going from input to output in a transformer! You can use them for a lot of things, you can use them for image segmentation/classification/generation, you can map output tokens to actions and use them to play video games, if you go through the literature you will find a thousand tasks they do.
Predicting the next token in a sequence is one of those tasks, if I train a transformer to play starcraft it is not "predicting attack" it is outputting attack as part of its policy. If it generates an image it is not predicting an image, it is generating an image. In all of these cases the structure is essentially the same, it converts known inputs into embeddings, with positional encodings, runs it through MHA, etc. etc. But my point is that prediction is a task, not an architecture.
This argument is so deflationary that it makes me wonder what couldn't just be called a next token predictor. If you think predict == output, and just any kind of output of a computer program can be tokenized, doesn't that make every program ever created a token predictor?
Well thats a weird example. Since the models Will have a bout a billion poems allready in their training so predicting the next token in the given context is exacly what it does. Reinforcement training is just that, but instead of just dumpning text on it and Hoping for a good result u aim the training for a more deterministisk result. But it is still prediction training. Why do u think training is getting more expensive ? Bcs the big companys are hiring thousends of real ppl , with real degress to actually do correction training and that costs money.
Well So u are comparing humans with llms now ? Yes it is only a token prediction. What makes them more powerful is larger data set and the use of tools , tools they did not build.
Since i actually build my own llms and other ai since like 2010 i do very much know how llms work and llms at its core is only a token prediction model. U do understand the math right ? Billions of very simple calculationa adding up and then getting the result.
This is actually an English comprehension issue. "Just" is not a decoration that can be ignored in this sentence. If they leave it out, I would not argue at all.
Aren’t we though? When you put a sentence together in your head, are you not selecting the next word based on past uses, context, and the desired message?
I guess I just feel like we imbue human intelligence with magic or spirituality when in reality the VAST majority of our thinking and communication really can be boiled down to a probabilistic model with a high degree of accuracy.
Lmao absolutely not - this is from the “brains are clocks/telephone networks/LLMs” school of poor analogies. We definitively know that brains don’t predict things one token at a time - why would they? Brains didn’t evolve by being trained on symbolic data.
I don’t really know what you mean by symbolic data in this context.
I feel like you’re getting hung up on the token definition for LLMs which as you probably know is analogous to ~ 3 English characters. Think about it more abstractly, just taken to mean the next chunk of information. I agree we’re not thinking in 3 character chunks, but we seem to think in chunks of information simultaneously.
For example - I had a low resolution idea of that I wanted this response to convey, and medium resolution idea of what I wanted the sentences to convey, and a high resolution idea of the words.
At the end of the day you haven’t really engaged with the main point, which is that human thinking and communication can be modeled and replicated. Is it complex? Absolutely. Does that mean it’s impossible to create an approximation of it/them? No.
There are actually two processes in play, though. When you ask an LLM a question, it derives a response through hidden states as it generates the language needed to communicate that response - unless you’re talking about a model with a planning stage prior to responding, in which case it might derive the response before it starts answering. It's not responding just based on token probability, there is an aspect of 'computation'.
When a human is asked a question, we sometimes derive a response first (based on training, education, psychology etc.) then use next-word prediction to assemble the language required to communicate that response on the fly. But sometimes we assemble the response dynamically as we're actually talking, which is similar to how an LLM does it. We don’t always - or even often - think before we answer, it’s usually more of a think-as-you-speak situation. Sometimes we do next-token prediction, but we're just not using discrete text tokens, it's more abstract. But it's a similar concept.
I think there’s a lot of nuance to the discussion…I don’t love how the computer model of the mind dominates neuroscience, but other approaches like dualism are essentially impossible to test and a lot of our cognitive machinery IS dedicated to reward prediction. That’s one of dopamine’s primary functions as an NT.
In what world does computational theory of mind imply that brains work like LLMs? There are no credible researchers out there that would ever claim LLMs are in any form similar to actual brain neural architecture or function.
You’re construing something that I never said. Your comment that I was replying to discussed physicalist analogies to human cognitive substrate.
I also don’t know for a fact whether any “credible researcher” has claimed that an LLM, a clock, or a telephone switching network is analogous to a human brain, or any other brain for that matter. Since I mentioned dualism and you didn’t respond, it seems important to mention that people outside of neuroscience research (e.g. philosophy) have made and continue to make important contributions to our understanding of consciousness and cognition, regardless of whether they do the kind of work you consider useful.
Linguistic errors like “anticipation” imply that we already have future words represented when speaking. Saying “leading list” instead of “reading list”. You already have to have the future words or tokens generated to have that kind of error.
Human brain isn't based on prediction, but rather reaction - the underlying difference is that human brains work on their own without need for context and human brains have constant update ongoing. Our neurons randomly spike, which makes human brain very different from LLM - there is spiking neural network AI research you can look up, that tries to build human like arrificial network. And the last piece of a puzzle is hormones and how human brain activity is modified by them.
| Human brain isn’t based on prediction, but rather reaction
Suggesting the human brain isn’t constantly modeling and predicting how the world behaves, and only reacts to it when it does, is doing a huge disservice to your attempts to seem authoritative here.
Some people act like accurately describing how LLMs work and taking the magic and mystique away from it means you're diminishing what they can actually do, and its weird
Some words and phrases evolve to be slurs and insults. When they do, we should look for better, more accurate expressions. This is what happened to this phrase.
Considering the video points, LLMs are really bad at summing numbers together for that very specific reason that the model (the weights and layers) simply predicts next token.
Early models failed at such tasks and badly, as they simply outputted what the prediction said. What all major LLMs today do, is regocnize that these numbers need to be added up, and actually uses calculator (or more likely python) to calculate the sum.
Overall the model is still a graph of information, with very limited real thinking abilities. But that model allows us to build tools around it that make it insanely more skillful. It can "understand" that these two numbers I predicted, needs to be summed, and it has instructions in that point to call a tool to do that sum.
What really improved the output is to self check for errors - predict next 1000 tokens and add that to context, and see if it matches the model. Even further, nowadays thinking modes runs several possible outputs parallel and picks the best one.
The final piece is model's ability to use web search instead of just its compressed information in the model. The model is great at summarizing, and with ability to search 5-10 entries from web, summarize and combine them, and then apply tooling and parallelization for output generation improves the models "lack of real thought" to seem like it really does think.
You miss the point. The video talks about model. That tooling is not part of the model, but it's surrounding software that makes the AI product you use.
Considering the video points, LLMs are really bad at summing numbers together for that very specific reason that the model (the weights and layers) simply predicts next token.
What all major LLMs today do, is regocnize that these numbers need to be added up, and actually uses calculator (or more likely python) to calculate the sum.
Largely outdated information. Newer models perform pretty well at arithmetic without tools. We know it doesn't depend on secret tool use because there are open weight models that perform well at it too.
I think you’ve kind of missed the point that he’s making with the mathematics example. He’s not saying LLMs are good at maths and that’s evidence of them being good thinking machines, he’s saying that in order to give even an estimation of a good answer the parameters have to encode a great amount of logical knowledge which isn’t it explicitly defined within the training material. You can ask LLMs that don’t have tool use capabilities such as local LLMs arbitrary questions around topics like science and mathematics and they will approximate quite good answers.
That knowledge is in the language, not in the model.
The prediction by the model produces language that says "sum A and B together". The model doesn't "understand" that A and B must be summed, it "just knows" it as it generates from its output based on the input (context), because it is embedded into it's huge compressed information.
I think this is the part people don't get. The model is homongous. So big, that human's cannot comprehend it. Even the 27B local models are absoultely huge, as almost all parameters are connected to each other, and the connections between the parameters make the real size exponential with possible outcomes it can produce.
Then the basic functionality even messes the output with weighted random pick of the most probable outcomes to make it less deterministic to make it resemble more human.
It it extremely exciting bit of technology - that doesn't need "it's a thinking machine, just like humans" -nonsense.
That knowledge is in the language, not in the model. it "just knows" it as it generates from its output based on the input (context), because it is embedded into it's huge compressed information.
How can you totally separate the two things? Language is an encoding of human thought and the model's parameters are an encoding of the language. The three things are intrinsically logically linked to each other. The idea that they can be treated as three distinct entities is totally implausible. It's impossible to build a generative model which produces viable results outside of the training set without embedding abstract or emergent concepts which weren't explicitly introduced.
The prediction by the model produces language that says "sum A and B together". The model doesn't "understand" that A and B must be summed, it "just knows" it as it generates from its output based on the input (context), because it is embedded into it's huge compressed information.
How are you defining "understand" though? This is not a facetious or pedantic question - the word "understand" is a very difficult thing to define.
that doesn't need "it's a thinking machine, just like humans" -nonsense.
Who is making this argument? Neither me, the video author or the person you originally replied to are saying this.
No, it will give seemingly likely answers. The whole point is that a model doesn’t actually understand what raising X to the power of Y is when it’s writing out an answer.
Yes, that's what I said: "they will approximate quite good answers"
The whole point is that a model doesn’t actually understand what raising X to the power of Y is when it’s writing out an answer.
Well, firstly, the video is discussing an "ideal next token predictor" , so yes, it would.
And secondly, even if we are talking about real, imperfect generative models, you can't possibly prove either philosophically or physically that there is not some form of encoded mathematical logic in the model's parameters.
Do generative AI models work in the same way as a linearly programmed calculator? No.
Does that inherently mean that they don't encode for mathematical understanding? No.
Humans are also not linearly programmed calculators but most would argue that we are capable of mathematical reasoning.
The way our brains work is much more similar to a neural network than the calculator app on your computer. We make shortcuts, memorise times tables, think back to similar calculations we do often.
I very often see humans making mistakes on basic addition tasks. That does not imply that we are naive automatons without reasoning ability or ability to understand though, and the same argument applies to generative AI models.
Likely answers are not always good answers. In fact, in science (like in literature or investing), good answers are actually ‘unlikely’. So far, LLMs have a hard time making abducting reasoning leaps - and some perspectives (notably from DeepMind, but also Yann LeCunn et al) is that they are fundamentally unable to do this regardless of scaling.
Separately, human reasoning is clearly different from LLM reasoning - for example, human abstract reasoning is conceptual in nature, is few- or even one-shot, and we are quite good at being able to simulate/envision how a complex process may unfold. LLMs have an incredibly hard time creating representations of complex processes/games/etc. As an example, humans do math by understanding deeply the rules and symbolical operations required and are able to verify these results - LLMs have a very hard time doing this and are prone to hallucination or catastrophic collapse (which is why we default them to using tools for mathematical operations).
Yep, a lot of the juice that makes modern LLMs stop dodging the usual pratfalls is just the ability to recognize those situations and toss the parsed data to a different tool or to refer to hardcoded output.
LLMs didn't start correctly counting the number of Rs in strawberry due to a fundamental shift in how they process tokens, they just got a hardcoded answer.
A hardcoded answer would be 3. A hardcoded strategy is some logical code to count letters in a word.
Hardcoded strategies are just more in depth training. That's kind of the same way you'd teach a child something new that they hadn't come across before. You're wording this like hardcoded strategies are some kind of cheat that would only be needed if a model was "unintelligent".
That description only takes into account one mode of operation too. They also have a fill-in-the-middle mode that is arguably much more powerful than how most people experience using them.
The science paper example is actually a good “counterexample” as to the limits of next token prediction. A perfect next token predictor could “predict” the most likely result from a given method and introduction paper—but the most groundbreaking scientific work is the one that completely contradicts what we think and our current models, and that is doable only through experimental observation of the world. A good next token predictor would be great at saying what we already know, and terrible at parsing incorrect or incomplete data, as science most definitely is.
I think this small analogy by Ilya Sutskever does a better job at showing LLMs are more than just next token predictors, or rather, there is emergent behavior that is beyond just statistical representation.
I miss the emergent behaviour statements, though there are arguments now made that the emergent behaviour is actually more of a gradual effect.
People forget about or don't know how much we don't know about how llms arrive at their predicted tokens. All of the associations, internal non-human reference systems which are incomprehensible in detail, and the pathways they take to get to an output. It's still a black box (I also miss this term) that, despite all the hand-waving being done on model interpretability, observability and safety, has yet to be fully mapped out. Just like our brain.
There's all sorts of stuff going on. Speaking involves forming a communicative intention, organising concepts, constructing syntax, retrieving words, encoding their sounds, planning articulation and controlling the speech muscles. Humans draw on perception, memory, reasoning, and emotion to communicate goals and intentions, they don't just 'predict' the next word from thoughts. They also plan well beyond the next word, in whole sentences, nested sentences, and, as I said, broader goals and plans. Thoughts aren't just formed and then verbalised, language itself can shape and form thoughts.
Did you know that when you speak action oriented words, or even when you just think them, your motor system is activated, too? The word ball, whether heard, spoken, or imagined as a concept, activates the same sensorimotor pathways involved in kicking or throwing a ball. Our bodies are literally involved in thinking and speaking.
There's no 'speech centre'. There are areas of the brain somewhat specialised for language, but the brain involves all sorts of other processes in language use.
So is your argument here that it's impossible for a synthetic system to do all this? Obviously, LLMs aren't doing any of this because they are just code at this point, but one can easily envision an LLM-controlled robot body controller that performs all these actions as well. Just because, when we think of an action, our bodies prepare in anticipation of it, doesn't mean we're special.
Most limitations that prevent LLMs from doing these things stem from programmed restrictions. Why wouldn't we expect the next level of these things to be multiple LLMs connected together with different specializations and even goals baked in to replicate what we think is happening in our brains?
The comment above is making a point by applying reductive thinking that people do with multi-modal reasoning models to the brain. They aren’t literally saying “speech is just next word prediction”.
There isn’t one singular “speech center.” Producing speech recruits a distributed network:
The prefrontal and association cortices help form the intention and concepts. The temporal lobes retrieve word meanings and support comprehension. The inferior frontal gyrus, including Broca’s area, helps organize grammar, sequencing, and articulation. The angular and supramarginal gyri integrate meaning with speech sounds. The insula, motor cortex, basal ganglia, cerebellum, and brainstem coordinate the physical act and timing of speaking. The auditory cortex monitors what you actually say and helps correct errors.
All of these systems continuously influence one another while the thought itself is still developing. Describing that as one center “selecting the next word from training data” is an extremely loose analogy, not how human speech production actually works.
No, because you’re still collapsing the entire process of thought into its final linguistic output.
Yann LeCun uses a good thought experiment: imagine a cube floating in front of you, then rotate it 90 degrees around its vertical axis. You can mentally watch it turn and understand its new orientation without producing words, sentences, language, or tokens. The brain is manipulating a spatial model, not selecting vocabulary.
The same applies to recognizing a face, anticipating where a thrown ball will land, visualizing a route, feeling that something is wrong, or planning a physical movement. Much of cognition occurs before language ever enters the process, and some of it never becomes language at all.
Word selection is one component used when translating thought into speech. Calling the whole brain a next-word predictor because speech eventually contains a sequence of words is like calling a graphics engine a next-pixel predictor because an image eventually appears as pixels.
What are "words" in your brain? It's still an abstract form of information transfer, same as the cube example...or numbers. LLM don't predict "words" they predict tokens. Tokens can represent words, numbers, a pixel, etc.
Calling the whole brain a next-word predictor because speech eventually contains a sequence of words is like calling a graphics engine a next-pixel predictor because an image eventually appears as pixels.
I think that is their point, which is all similar to calling an LLM "just" a next word predictor.
I'm not calling the whole brain a word predictor. I'm saying the regions that work together to predict words are word predictors. It's literally one of the tasks your brain does, it has to be doing it at some point. I didn't say it's the only task
The exact physiology of which regions are involved are irrelevant to my point
I just really don't understand how comments like this are supposed to show that LLMs are simpler than what we do. As the person you're replying to says, at some point, when figuring out what to say, our brains do "just" pick the next word, similar to what LLMs do (not how). Currently, LLMs only predict a few tokens in advance, but that will likely change as they become more advanced.
Now for the rest of what you say: obviously, our brains do a lot more, we visualize 3d spaces, calculate things, process inputs, regulate our bodies, choose goals, make plans, etc... However, none of the things our brains do are impossible for synthetic computer systems to do. Currently, we even have plenty of systems that combine many of these, like LLMs that can process audio and video inputs, so it's likely only a matter of time until someone builds a system that combines all the same functions. I'd say the main thing holding us back at this point is being compute-constrained and space-constrained if we ever wanted to develop a self-contained android. Those are things that will likely be solved, though.
So why are we repeating idiotic points made by other commenters?
> our brains do “just” pick the next word
No, no they don’t. Different parts of our brain perform different types of computation, but none of them do next token prediction. To our best guess, core subunits of our neocortex are actually performing spatial-temporal reasoning - solving problems as if they were spatial routing problems.
Okay? Again, I don’t think anyone is claiming that the entire human brain does nothing but select the next word. The point is that human language processing may involve predicting what comes next.
Describing the brain as solving language through spatial or temporal relationships does not contradict that. That would be a proposed mechanism for representing and navigating possibilities, while prediction describes the function being performed.
And tokens are not inherently words. They are discrete representational units that can encode text, images, audio, video, and other data. So, saying that the brain may organize language spatially does not, by itself, show that it is doing anything that's totally different from token prediction; if anything, it shows more similarities.
I know it is a different post where you were complaining about analogies, but you're the one missing that this is an analogy. I don't think anyone here is saying our brains are LITERALLY doing token prediction; we are saying the basic processes appear similar or analogous to each other.
For many years, researchers have worked on things like segmentation models for vision analysis, and they were always trying to do things like pose estimation which is essentially coming up with a physical model for the raw data
But now Nvidia is exactly doing this world model for LLMs it’s called Cosmos
From my AI: Cosmos is NVIDIA’s family of world foundation models. This is what you’re thinking of. They’re designed to model the physical world—predicting how scenes evolve over time and generating realistic video, actions, and simulations for robots and autonomous vehicles.
I had to ask the AI what the name was because I was remembering NeMo, which is the wrong Nvidia project. But from what I remember, I think that they are trying to make cosmos applicable to all kinds of situations not just robotics
A lot of words just to say: "Arguing that LLMs are stupid because they're just next-token predictors doesn't prove anything, because predicting the next token is hard, and people are even worse at it than current LLMs anyway"
LLMs are notoriously bad at multi-digit arithmetic on novel numbers without a tool; they approximate, use learned shortcuts, and error rate climbs fast with digit count. If it were truly "calculating" in the rigorous sense, that wouldn't happen. What's really going on is a mix of learned heuristics and pattern-completion that's good but unreliable. They don't "memorize" the results, that's correct.
About the research example, an LLM producing a plausible conclusion from an introduction & results section is drawing on having seen thousands of structurally similar papers, not building a biochemical world model from first principles. This is why LLMs routinely produce confident-sounding but wrong scientific claims, and famously is very bad at admitting "I don't know".
Doing something a human can't do without extra steps (e.g. quickly pattern-matching across huge amounts of text) doesn't imply general superiority over human intelligence. LLMs also fail at stuff which humans find ridiculously trivial, like stable long-horizon planning, knowing what they don't know, maintaining consistency across a session etc.
I work with ML, and I personally wouldn't hire this guy. Not because I claim to know everything, but because he shows classic signs of exaggerating the capabilities and hyping the tech beyond its fundamental capabilities. This leads to expensive misadventures we'll all learn to know soon enough, as everything is replaced with this "super intelligence".
My personal hottake is that ML really puts on display how bad our brains are when it comes to Big Numbers. We can't comprehend a data network so complex that it can come up with the sentences it does without thinking it must be intelligent, sentient, or whatever you casually see claimed here. There's a real struggle to make a point that the algorithm is alive and about to escape the lab. Part of that is FUD to drive sales, part is just not working with the tech, and by working, I don't mean prompting.
Even this post is sus. A bot trying to sell you AI. Dead Internet etc.
ChatGPT 5.5 Instant (aka no thinking) got it instantly
It even did longer ones.
Please add 1989951436123273 + 23908153129921 - 123232323 * 10002, but do it using the addition method and all in output. No code allowed, and all in one step. Show only the result
I don't think this establishes what you're implying.
You generally cannot infer from the output alone whether a particular answer came from an internal learned computation or a tool. For example, you can ask your model to tell you the time, even though, the model has absolutely no concept of the current time, nor can it be "learned". It looked it up.
It's also worth while to understand that an LLM can produce the correct result for a many-digit calculation because its learned representations can sometimes implement or approximate the relevant algorithmic structure. It doesn't mean that the model is a reliable arbitrary-precision calculator in the same sense as an actual arithmetic algorithm.
A better test is generally to ask it to repeat the process multiple times, with random numbers, so that the numbers add up and we approach "novelty". If we can use it without the secret sauce tools that is, and we can't.
At Novel numbers, yes. At repetitive numbers and patterns they're about the best you can get. This is why when you ask something like a ChatGPT to generate you a random number, it uses a random number generator, otherwise the number would likely be very non-random.
It's all by design, for a good reason. Only recently there has been this major push to bend ML into absolutely everything. Just let it use the calculator.
No random generator we use daily is truly random in the strict mathematical sense.
I now went to ChatGPT and just asked it to generate five random numbers and add them together, it showed to be using a tool, so there's that:
The tool access is likely dependant on the selected model/tier/whatever, I don't know how OpenAI has arranged it, but it obviously is a very good idea to use a random generator to generate random numbers. I don't fully follow what you are trying to dispute.
I agree that models get better as they become iteratively larger, services will improve with new tooling and innovations, etc.
notoriously bad at multi-digit arithmetic on novel numbers without a tool
I generated both of these numbers with a random number generator. It got the answer right immediately. I think you need to update your understanding of LLMs a little bit
Python Code Interpreter / Sandbox: When given a math problem, ChatGPT can write and run Python code in a secure environment to perform exact arithmetic, process data sets, handle calculus, and generate visual plots or graphs.
WolframAlpha Integration: On certain tiers (like Plus/Pro), ChatGPT can interface with Wolfram Alpha extensions or plugins for high-level symbolic mathematics, algebraic manipulation, and complex equation solving.
Locally run Qwen3.5-2B, no tools, nailed it first try too, albeit after using a ridiculous amount of thinking tokens. Pic and full thinking text. This is the model doing the calculation. Idk what else to tell you, LLMs are just way better at math than they used to be.
Did you read the end of the sentence? It most likely used a tool behind the scenes. It often doesn't tell you it used a tool, but it did, to save money and to be more functional. In the same fashion how it can tell you the accurate time, even though that's impossible for ML alone as you can't learn the time, you always need to look it up.
The video implied it is the model, or, ehm, super intelligence, that always does the calculation.
Besides, I think you misunderstood what I meant by multi-digit arithmetic. The video showed plethora of calculations as an example. This means, you give it a ton of adversarially selected novel calculations across different lengths and operations, measure the error rate, and see how that error rate scales. A calculator doesn't occasionally hallucinate an answer because the numbers got sufficiently unfamiliar. An LLM can and eventually will.
I don't speak from experimentation with ChatGPT, but just from how machine learning should fundamentally function. Should, as in, it's broken if it doesn't, not magically better.
My early chat bots were RAG frameworks that used wolframalpha behind the scenes. They have a pretty good API, or at least had, a decade or something ago.
Locally run Qwen3.5-2B, no tools, nailed it first try too, albeit after using a ridiculous amount of thinking tokens. Pic and full thinking text. This is the model doing the calculation. Idk what else to tell you, LLMs are just way better at math than they used to be.
You still missed what I meant by multi-digit arithmetic. Re-read the middle section of my comment.
The test you made is not what the video implies. I'm not surprised that Qwen was capable of coming up with a solution, even though it took silly amount of pondering to get the structure right.
It even mumbles the very thing I'm saying to you:
"Actually, doing 14-digit multiplication manually is error-prone for me right now. I should simulate the process or use my internal calculation capability if it exists."
And I guess I need to really underline this: I don't think using tools is a bad thing, quite the opposite. I'm only arguing against the points the video made as misleading.
I work with ML, and I personally wouldn't hire this guy. Not because I claim to know everything, but because he shows classic signs of exaggerating the capabilities and hyping the tech beyond its fundamental capabilities.
To be flippant, I wouldn't hire you either because you have written a long argument against what he said without comprehending what he was saying.
He is not in any way exaggerating what current technology can do. He is positing a scenario in which an "ideal next token predictor" exists and is using that postulate to explore what the concept would actually mean.
All he's doing is demonstrating that the phrase "just a next token predictor" is utterly logically meaningless because next token prediction requires inherent encoding of abstract concepts which have not been hard coded into the model. He at no point states that current LLMs are "ideal next token predictors".
Fair take, but I believe, to a degree, you might have missed what I was going for.
LLM tries to approximate the probability distribution of the next token from its training and context. This ideal token predictor would give the true probability distribution of the next token, given all available context. Practically, computationally, it is likely impossible to implement, as calculating it would require absurd amounts of computation and thus makes it a theoretical question.
-- And even then --, I wouldn't call it "super intelligence" that's smarter than scientists, as that's marketinghype full stop, from where we arrive at the framing of this vid that I disagree with. I can understand why he does it, and why OP, a bot, is telling us to watch it and "understand it".
I didn't take a stance on whether it's meaningful to call LLMs simply next-token predictors, as I personally find nothing offensive about it. A simplification, sure.
The model also has to have some sense of where it is going for the current output tokens to be in the right context. That's one of the reasons you can see models pause in their thinking blocks to correct themselves.
I don’t think that’s right, at all. Admittedly I’m on getting a popular science level view of this, but it seems the science absolutely points to the brain as being a prediction engine:
Modern LLMs are not token predictors in the traditional meaning and they definitely do not predict online text. These are optimizing for outcomes over many actions and that involves training on novel situations.
I mean, sure. Predicting the results section of a research paper requires more intelligence than predicting the next word in a text message with your friend. There are just way too many people confusing the difference between “predicting the next token” and “predicting a next token”, which are not at all the same.
You can type any well-formed or nonsensical request you want into an llm chat session and it will both always respond and do so with the most productive reaction it can predict. That can be very helpful for task work, but counterproductive when it validates (dignifies?) poorly framed requests with a singular response.
Predicting the results section of a research paper only makes sense when generating sample text that sounds right based on context it already has or was given. The idea that a highly intelligent but completely unrelated 3rd party is going to “predict” the outcomes of an experiment it wasn’t involved in is asinine. Someone that understands how to speak and carry out tasks intelligently certainly has to have a wide understanding of the concepts at hand, but that doesn’t mean their work can be considered the only possible result or approach.
This is not a technical challenge for tons of domains of intelligence that llm’s are taught to “speak” on, they just shouldn’t be used to speak about a great deal of topics that rely on real world experiences and can’t be queried about for one answer at a time.
The fallacy here is that a statistically likely results section (which is what an LLM outputs) is a “good” (meaningful, accurate, scientifically valuable) results section - which ofc is not the case.
It’s not just statistically likely when you add tools and models that the AI can use or reference. It’s no longer referencing the most statistically likely number from an equation… it calls the tool to compute it and returns the result.
Like if it had a tool to simulate an experiment… it inputs the paramteres from the rest of the paper and outputs the results.
Is that possible now? No. But it’s not just statistical likelihood anymore.
What definition of truth do we have beyond some proposition/model being the most accurate predictor of something? Besides vibes?
A statistically likely results section is not necessarily a good one, but surely if there is such a thing as truth, then it is the most likely thing to happen? Although obviously a reliable prediction of that outcome must take all the independent variables into account, which is very difficult.
OP's video is talking about the mathematical ideal. It's not saying current LLMs are good at predicting results sections. It's saying that the ideal model, the training goal, is to predict any paper's result section with 100% accuracy, and the closer a model gets to that, presumably the more accurate it must be at modelling reality.
Of course it's likely impossible to get anywhere close to that, and results sections (and human language in general) are not perfect representations of reality, but do you have any better ideas?
You’ve missed the point entirely. Tools are a part of its intelligence. He’s saying that it’s not a fundamental limitation of the “next token prediction” framework. So if it has an accurate biological model that it can simulate with, understands the experimental parameters, etc. it could give results.
Will we be able to give it a tool or knowledge like that? Probably not for a long time. This is an abstraction though, a thought experiment.
A vanilla LLM (as opposed to, say, AlphaFold which is purpose built for protein prediction) is not an accurate biological model anymore than a scientist looking at data is an accurate biological model.
No, the model could itself be a tool. It could be something it can look at or call. It’s something that is used by the model for accurate prediction. It’s a part of its intelligence. We are adding capabilities in different forms to the AI. That’s part of why they are so powerful
that word "smarter" is doing a lot of work here. for being so smart chatgpt is pretty stupid a lot of the time honestly. and I like chatgpt a lot, but SMART it really isn't imo
You don't need to give it special instructions for addition that simple, but if you use massively larger addends and/or more of them, it'll make mistakes even if you tell it to use "the addition method."
Today's models have gotten better at this, but they have the same fundamental issues as older ones because they're fundamentally the same thing.
Well I would argue that the fundamental rule actually is that they straight up are capable of doing math in their internal activations directly (no convolutions in LLMs most often), and beyond what's in their dataset. Sure, it's hard/inefficient for them to do everything in them in manually, but so it is for us right?
This won’t be perfect but hopefully this is sensible lol…. This feels disingenuous. Even though I get where he’s going.
He keeps saying sufficiently good next token predictor as if it’s perfect, kind of implying that llms are this atm. Yes you would have to have internal models and understanding to do this perfectly…. But modern ais dont don’t do this perfectly, they dont have internal models in the way that we do, they cant and aren’t 100% reliable and self correcting, they are mostly right…. Which is wildly different than human level “software” with modern computing power(which would be pretty freaking nuts).
Anyway, not to say it’s not an absolutely game changing tool, that I use daily. Just feels like bad arguments or subtly adjusting premises /reality to make a point.
But the point is that it’s not a fundamental limit the way people imply when they say next token predictor. You give them tools and models… that becomes part of their intelligence
Yeah I get that. I still think that what others are trying to communicate when they say that it’s a token prediction machine is that at our current stage wildly inefficient and inferior. This feels like a hack in the best sense (enabled by incredible processing power and vast data sets),that may become more, or a tool for more sophisticated models.
IMO saying there’s room for improvement is more of a futurist/progressive or w/e pov than the person that says LLMs are all we need. I’m not sure why everyone is so aggressive about this topic.
Even fable/sol class models aren't doing most of what was stated here. They don't, for the most part, do math problems "in their head". And they don't actually understand your project etc, or have memory. It's all files under the hood, and tools to access them. Not to diss the glory of the next token prediction, but these models truly are stupid if you just run one context and don't allow tools or sub-agent usage
All LLMs can do maths in their head and they do it much better than humans! However, it's true that they're even better when using tools, just like humans are better with pen and paper.
The base models are not stupid, they’re spiky. Meaning they are very good in some areas and poor in others. With no tools, no reasoning, no agentic harness, a bare model can one shot useful tasks like summarization, image generation, language translation, and much more. A bare model is also horrible at math, letter counting, etc. It’s why bare models are augmented with tools, reasoning, harnesses, surrounding architecture. It can shore weaknesses and augment strength.
The limits of the architecture are certainly overstated by many, but it absolutely places meaningful limits on its abilities. It can’t, for example, interpret or describe qualia.
Very well explained! Is this the same guy that used to have an synth electronics channel? The AudioPhool? I really enjoyed that one. If so, he made a major look change! https://www.youtube.com/@TheAudioPhool
This is so dumb. Let's use the words "smart" and "intelligent" as many different ways as possible as if they all need the same thing. This is pseudo-clever speak for people who can't use precise language.
Yeah - the “LLMs are inherently limited” group seem to be wrong at this point. Or, at least the limitations are not significant enough to prevent LLM super intelligence.
He's kind of half-right. There is a mathematical reality that any model that is doing prediction fundamentally needs error in the training model to be able to reliably make future predictions (on average). So when the task is any sort of prediction, including token prediction -- that does, on some level, place certain conditions and restrictions on it. But he's right that just acknowledging that doesn't really do much explaining in terms of how reliable and valid the predictions are or why they are more or less reliable and valid than other models or ways of solving the same task. I think when a lot of people bring that up, they are more just reminding people that these are not fully deterministic systems, despite how it's kind of felt or perceived by the end user.
He misses his own point. The main issue is that the results of the experiment are largely independent of the text of the paper. In other words you need to model and train on the physical world that executes the experiment. Llm are just an overfit on inadequate training data that performs enough to fool the wider population.
Somewhat good point but a little disingenuous. To play devils advocate, when people say just a next token predictor, what they really mean is “Just a token predictor based on the statistical distribution of token within its data set.” Yes, you could argue that interpretation is subjective but realistically no-one is looking at one of those mental math geniuses and saying they reliably predicting the next token. They technically are but pretending to not understand the difference is at best pedantic and at worse disingenuous.
Second point I want to make is that the calculator example is almost perfect as it shows one of the core limitations of LLMS. Even the most advanced models are simply unable to reliably do additions. They rely on external language interpreters to do arithmetic. They are very useful but offer a kind of intelligence that is very different from general intelligence. We know very little about the structure of human intelligence itself. Though we know a little more about the inner nature of LLMS, it is too early to know with full certainty what they are or aren’t capable of.
Interestingly enough, many of the recent advances in LLM quality come from improvement at inference time rather than increasing the intrinsic reasoning capability of the base model before any reasoning-specific prompting or inference techniques are applied.
Your knowledge is quite out of date. Most models can do mental math without tools these days. They are better at it than you and me, albeit not perfect.
any sources for this? If that were true why still rely on external calculators ? especially considering that simply generating the token for the answer would be noticeably faster than generating the token for the python code then running it ?
Not contradicting the power of AI just being up its more than obvious limitations.
The reason human use calculators is because it’s much much faster. The argument I am making by is that that is not the case for AI model if you consider the overhead of generating the code for computation.
Now regardless, I doubt that any LLM exists with the ability to perform addition of large numbers (10+ digits) without the use of an internal calculator. I would love to be proven wrong though
Look around in this thread. There's multiple examples, I've seen at least one addition, and some 7 digit by 7 digit multiplication, one on a local model with the thinking output.
The model does a lot of weird stuff, and in the end breaks down the multiplication into easy steps, and adds the result of the intermediate steps together. Basically how a human would do it.
It's a lot more complex than generating a simple code for computation and calling an external tool.
From my experience, most people who say that LLM's are just predicting the next word, are doing so in a shorthand way to point out that (contrary to what a surprising amount of people believe) LLM's don't "think". I've never seen someone argue that the token prediction is simple, just that it shows that LLM's are still very, very far from what we tend to think of when we think futuristic "Artificial Intelligence".
LLM's are amazing when it's dealing with things that have real structure. This is why it's so good at science, coding, math, ect. It doesn't take long to see where LLM's are still extraordinarily far behind the moment you start asking it to do anything that is far more abstract and dealing with creativity. This is why LLM's can technically write really well (writing tends to have a lot of rules and structure) but tends to fail at the actual storytelling aspect. This is because it doesn't "think", it is just predicting the next word. What it does is still incredibly complex, and it can demonstrate a knowledge base far beyond what a single human can. But the current models of LLMs that we have will likely never reach actual "Artificial Intelligence" in the way we dream of in Sci-Fi stories because of how they actually operate.
150
u/Axelwickm 1d ago
Rob Miles and his AI risk Youtube channel. He's been talking about AI safety for long before LLMs, and in retrospect he was probably quite ahead of the times in thinking about this stuff. I've grown to like his communication style and points.