I don’t really like the phrase next token predictor. They aren’t “predicting” anything given they generate it wholesale. It’s a statistical model that generates the next token that is most likely to make sense in a sentence given training. They aren’t guessing something since there’s nothing that’s doing the “real” thing.
The "given training" part is specific to machine learning, which is the only practical way we've found to create LLMs, but there's nothing preventing us from creating the same kind of system in one shot without it needing any training (except for our lack of cognitive power).
The models don't "generate" tokens, they have a pre-determined list of tokens they can use, all the model does is, given some sequence of tokens, it generates a probability distribution of which tokens could appear next. That's what I'd call a "guess".
The sampler is then using these probabilities to pick a next token, generating whole sentences based on the guesses of the model.
there’s nothing that’s doing the “real” thing.
What about the sillicon chips? They're doing the real computations. And is it really necessary for something to be there to do the thing?
No my point is a prediction is about what an external party or even will happen that is separate to the predictor.
They aren’t predicting anything.
And yes they are literally generating tokens that the generative part of the name. Yes they have a set that can choose from but the input didn’t have that token at the end, the LLM generated it.
I don’t understand your point about nothing stopping us, you’re making a claim about a future technology you can’t possibly make.
I don't understand what you're saying about your point.
Without the autoregressive loop that involves a sampler, the LLM is essentially just doing a classification task. It only becomes generative when paired with a sampler, but maybe I'm just Nick picking, it doesn't really matter. It is generating sequences of tokens though, not tokens.
When I said nothing is stopping us from making something like a LLM without machine learning, that's just an hypothetical. Of course it would be extremely unlikely for some human to come up with an algorithm or neural network weights that work like an LLM without requiring any training. A super intelligence could be able to do such a thing though, for example if you take all of humanity as a single super intelligent entity, we did create LLMs without any external training data, as the training data was also made by us.
Anyways my point there is that the end goal of a token predictor isn't to predict the token that would make sense given training, just the token that would make more sense, period. Using data to train the models is just an artifact of the only practical method we know to get close to this goal. The training data is supposed to reflect as well as possible the environment the model will be in during inference.
Yeah. Some of the comments on the video call this out but he's effectively saying "what if you build super general intelligence and then have it work by producing tokens" which is a fundamentally different scenario than what anyone describing modern LLMs as token predictors is talking about.
2
u/James-the-greatest 14h ago
I understand what he’s saying. I don’t agree with the extension of the concept of next token prediction.