The idea that ChatGPT is simply “predicting” the next word is, at best, misleading

Maybe I don't understand what exactly your point is, but I'm not convinced. AFAIK, it's true that GPT has no state outside of the list of tokens so far. Contrast to your jazz example, where you, in fact, have hidden thoughts outside of the notes played so-far. I think this is what Wolfram and others are saying when they say that "GPT predicts the next token". You highlight "it doesn’t have a global plan about what’s going to happen" but I think a key point is that whatever plan it has, it has to build it up entirely from "Once upon a" and then again, from scratch, at "Once upon a time," and again and again.  Whatever plan it makes is derived entirely from "Once upon a time," and could well change dramatically at "Once upon a time, a" even if " a" was its predicted token. That's very different from what we think of as a global plan that a human writing a story makes.

The intuition of "just predicting one token ahead" makes useful explanations like why the strategy of having it explain itself first and then give the answer works. I don't see how this post fits with that observation or what other observations it clarifies.

One minor objection I have to the contents of this post is the conflation of models that are fine-tuned (like ChatGPT) and models that are purely self-supervised (like early GPT3); the former has no pretenses of doing only next token prediction.

@Bill Benzon [LW · GW]:  A thought experiment. Suppose you say to ChatGPT "Think of a number between 1 and 100, but don't tell me what it is. When you've done so, say 'Ready' and nothing else. After that, I will ask you yes / no questions about the number, which you will answer truthfully."

After ChatGPT says "Ready", do you believe a number has been chosen? If so, do you also believe that whatever "yes / no" sequence of questions you ask, they will always be answered consistently with that choice? Put differently, you do not believe that the particular choice of questions you ask can influence what number was chosen?

FWIW, I believe that no number gets chosen when ChatGPT says "Ready," that the number gets chosen during the questions (hopefully consistently) and that, starting ChatGPT from the same random seed and otherwise assuming deterministic execution, different sequences of questions or different temperatures or different random modifications to the "post-Ready seed" (this is vague but I assume comprehensible) could lead to different "chosen numbers."

(The experiment is not-trivial to run since it requires running your LLM multiple times with the same seed or otherwise completely copying the state after the LLM replies "Ready.")

Thanks, you've put a deep vague unease of mine into succinct form. 

And of course, now I come to think about it, a very wise man said it even more succinctly a very long time ago:

Adaption Executors, Not Fitness Maximizers.

I think that what you're saying is correct, in that ChatGPT is trained with RLHF, which gives feedback on the whole text, not just the next token. It is true that GPT-3 outputs the next token and is trained to be myopic. And I think that your arguments seem suspect to me, just because a model takes steps that are in practice part of a sensible long term plan, does not mean that the model is intentionally forming a plan. Just that each step is the natural thing to myopically follow from before.

I think a key idea related to this topic and not yet mentioned in the comments (maybe because it is elementary?) is the probabilistic chain rule. A basic "theorem" of probability which, in our case, shows that the procedure of always sampling the next word conditioned on the previous words is mathematically equivalent to sampling from the joint of probability distribution of complete human texts. To me this almost fully explains why LLMs' outputs seem to have been generated with global information in mind. What is missing is to see why our intuition of "merely" generating the next token differs from sampling from the joint distribution. My guess is that humans instinctively (but incorrectly) associate directional causality to conditional probability and because of this, it surprises us when we see dependencies running in the opposite direction in the generated text.

EDIT: My comment concerns transformer architectures, I don't yet know how rlhf works.

I'm not following the argument here.

"I maintain, for example, that when ChatGPT begins a story with the words “Once upon a time,” which it does fairly often, that it “knows” where it is going and that its choice of words is conditioned on that “knowledge” as well as upon the prior words in the stream. It has invoked a ‘story telling procedure’ and that procedure conditions its word choice."

It feels like you're asserting this, but I don't see why it's true and don't think it is. I fully agree that it feels like it ought to be true: it is in some sense still shocking to me that a next-token predictor trained on trillions of tokens is so good at responding to such a wide variety of prompts. But if you look at the mechanics of how a transformer works, as @tgb [LW · GW] and @Multicore [LW · GW], it sure looks like it's doing next-token prediction, and that there isn't a global plan. There is literally no latent state --- we can always generate forward from any previous set of tokens, whether the LLM made them or not.

But I'd like to better understand.  

You seem to be aware of Murray Shanahan's "Talking About Large Language Models" paper. The commenter you quote, Nabeel Q,  agrees with you, but offers no actual evidence; I don't think analogies to humans are helpful here since LLMs work very differently from humans in this particular regard. I agree we should avoid confusing the training procedure with the model, however, what the model literally does is look at its context and predict a next token.

I'll also note that your central paragraph seems somewhat reliant on anthroporphisms like "it "knows" where it is going". Can you translate from anthropomorphic phrasings into a computational claim? Can we think of some experiment that might help us get at this better?

I think the state is encoded in activations. There is a paper which explains that although Transformers are feed-forward transducers, in the autoregressive mode they do emulate RNNs:

Section 3.4 of "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention", https://arxiv.org/abs/2006.16236 

So, the set of current activations encodes the hidden state of that "virtual RNN".

This might be relevant to some of the discussion threads here...

I don't think I understand the problem correctly, but let me try to rephrase this. I believe the key part is the claim whether or not ChatGPT has a global plan? Let's say we run ChatGPT one output at a time, every time appending the output token to the current prompt and calculating the next output. This ignores some beam search shenanigans that may be useful in practice, but I don't think that's the core issue here.

There is no memory between calculating the first and second token. The first time you give ChatGPT the sequence "Once upon a" and it predicts "time" and you can shut down the machine, the next time you give it "Once upon a time" and it predicts the next word. So there isn't any global plan in a very strict sense.

However when you put "Once upon a time" into a transformer, it will actually reproduce the exact values from the "Once upon a" run, in addition to a new set of values for the next token. Internally, you have a column of residual stream for every word (with 400 or so rows aka layers each), and the first four rows are identical between the two runs. So you could say that ChatGPT reconstructs* a plan every time it's asked to output a next token. It comes up with a plan every single time you call it. And the first N columns of the plan are identical to the previous plan, and with every new word you add a column of plan. So in that sense there is a global plan to speak of, but this also fits within the framework of predicting the next token.

"Hey ChatGPT predict the next word!" --> ChatGPT looks at the text, comes up with a plan, and predicts the next word accordingly. Then it forgets everything, but the next time you give it the same text + one more word, it comes up with the same plan + a little bit extra, and so on.

Regarding 'If ChatGPT visits every parameter each time it generates a token, that sure looks “global” to me.' I am not sure what you mean with this. I think an important note is to keep in mind it uses the same parameters for every "column", for every word. There is no such thing as ChatGPT not visiting every parameter.

And please correct me if I understood any of this wrongly!

*in practice people cache those intermediate computation results somewhere in their GPU memory to not have to recompute those internal values every time. But it's equivalent to recomputing them, and the latter has less complications to reason about.

Based on my incomplete understanding of transformers:

A transformer does its computation on the entire sequence of tokens at once, and ends up predicting the next token for each token in the sequence.

At each layer, the attention mechanism gives the stream for each token the ability to look at the previous layer's output for other token before it in the sequence.

The stream for each token doesn't know if it's the last in the sequence (and thus that its next-token prediction is the "main" prediction), or anything about the tokens that come after it.

So each token's stream has two tasks in training: predict the next token, and generate the information that later tokens will use to predict their next tokens. 

That information could take many different forms, but in some cases it could look like a "plan" (a prediction about the large-scale structure of the piece of writing that begins with the observed sequence so far from this token-stream's point of view).

To those that believe language models do not have internal representations of concepts:

I can help at least partially disprove the assumptions behind that.

There is convincing evidence otherwise, as demonstrated through an Othello in an actual experiment:

https://thegradient.pub/othello/ The researchers conclusion:

"Our experiment provides evidence supporting that these language models are developing world models and relying on the world model to generate sequences." )

Here's an analogy. AlphaGo had a network which considered the value of any given board position. It was separate from it's monte carlo tree search network- which explicitly planned the future. However it seems probable that in some sense, in considering the value of the board, AlphaGo was (implicitly) evaluating the future possibilities of the position. Is that the kind of evaluation you're suggesting is happening? "Explicitly" ChatGPT only looks one word ahead, but "implicitly" it is considering those options in light of future directions of development for the text?

Topics like this really draw a crowd but if you dont know how it works writing like this just adds energy in the wrong direction. If you start off small building perceptrons by hand, you can work your way up through models to transformers and it'll be clear what the math is attempting to do per word. It's sophisticatedly predicting the next work based on a matrix of relevance to the previous word and the block as a whole. The attention mechanism is the magic of relevance but it is, predicting the next word.

I don't think knowledge is the right word. Based on your description, that would be more analogous to an instinct. Knowledge implies something like awareness, or planning. Instinct is something it just does because that's what it learnt. 

Charles Wang has just posted a short tweet thread which begins like this:

Next-token-prediction is an appropriate framing in LLM pretraining) but a misframimg at inference because it doesn’t capture what’s actually happening, which is about that which gives rise to the next token.

We’re all more or less doing that when we speak or write, though there are times when we may set out to be deliberately surprising – but we can set such complications aside

We're all more or less doing that when we speak or write?

If you think of the LLM as a complex dynamical system, then the trajectory is a valley in the system’s attractor landscape.

The real argument here is that you can construct simple dynamical systems, in the sense that the equation is quite simple, that have complex behavior. For example, the Lorenz system though there should be an even more simple example of say, ergodic behavior.

I had to resort to Google Translate:

"But because I have some obscure notion, which has some connection with what I am looking for, if I only boldly start with it, it molds the mind as the speech progresses, in the need to find an end to the beginning, that confused conception to complete clarity in such a way that, to my astonishment, the knowledge is finished with the period." Heinrich von Kleist (1805) On the gradual development of thoughts while speaking

While Wolfram's explanation is likely the fundamental premise upon which ChatGPT operates (from an initial design perspective), much of this article assumes a deeper functioning that, as is plainly admitted by the author, is unknown. We don't KNOW how LLMs work. To attribute anything more than reasonably understood neural weighting algos to its operations is blue sky guessing. Let's not waste time on that, nor on speculation in the face of limited accessible evidence one way or the other.

I don't how the analogy with humans help. We don't know the "mechanism" behind how the human mind works. That's not the same as LLMs. We exactly know the mechanism of how it works or produces the output. And the mechanism is no different than what it has been trained to do, i.e. predict the next word. There isn't any other mysterious mechanism at work during inference.

As for plan, it doesn't have any plan. There's no "memory" for it to store a plan. It's just a big complex function that takes an input and produces an output which is the next word. And then repeats the process over and over until it's done

A few observations

This conversation has been going on for a few days now and I’ve found it very helpful. I want to take a minute or two to step back and think about it, and about transformers and stories. Why stories? Because I’ve spent a lot of time having ChatGPT tell stories, getting a feel for how it does that. But I’m getting ahead of myself.

I wrote the OP because I felt a mismatch between what I feel to be the requirements for telling the kind of stories ChatGPT tells, and the assertion that it’s “just” predicting the next word, time after time after time. How do we heal that mismatch?

Stories

Let’s start with stories, because that’s where I’m starting. I’ve spent a lot of time studying stories and figuring out how they work. I’ve long ago realized that that process must start by simply describing the story. But describing isn’t so simple. For example, it took Early Modern naturalists decades upon decades to figure out to describe life-forms, plants and animals, well enough so that a naturalist in Paris could read a description by a naturalist in Florence and figure out whether or not that Florentine plant was the same one as he has in front of him in Paris (in this case, description includes drawing as well as writing).

Now, believe it or not, describing stories is not simple, depending, of course, on the stories. The ChatGPT stories I’ve been working with, fortunately, are relatively simple. They’re short, roughly between 200 and 500 words long. The one’s I’ve given the most attention to are in the 200-300 word range.

They are hierarchically structured on three levels: 1) the story as a whole, 2) individual segments within the story (marked by paragraph divisions in these particular stories), and 3) sentences within those segments. Note that, if we wanted to, we could further divide sentences into phrases, which would give us at least one more level, if not two or three. But three levels are sufficient for my present purposes.

Construction

How is it that ChatGPT is able to construct stories organized on three levels? One answer to that question is that it needs to have some kind of procedure for doing that. That sentence seems like little more than a tautological restatement of the question. What if we say the procedure involves a plan? That, it seemed to me when I was writing the OP, that seems better. But “predict the next token” doesn’t seem like much of a plan. 

We’re back where we started, with a mismatch. But now it is a mismatch between predict-the-next-token and the fact that these stories are hierarchically structured on three levels. 

Let’s set that aside and return to our question: How is it that ChatGPT is able to construct stories organized on three levels? Let’s try another answer to the question. It is able to do it because it was trained on a lot of stories organized on three or more levels. Beyond that, it was trained on a lot of hierarchically structured documents of all kinds. How was it trained? That’s right: Predict the next token.

It seems to me that if it is going to improve on that task, that it must somehow 1) learn to recognize that a string of words is hierarchically structured, and 2) exploit what it has learned in predicting the next token. What cues are in the string that guide ChatGPT in making these predictions?

Whatever those cues are, they are registered in those 175 billion weights. Those cues are what I meant by “plan” in the OP. 

Tell me a story

At this point we should be able to pick one of those stories and work our way through it from beginning to end, identifying cues as we go along. Even in the case of a short 200-word story, though, that would be a long and tedious process. At some point, someone HAS to do it, and their work needs to be vetted by others. But we don’t need to do that now.

But I can make a few observations. Here’s the simplest prompt I’ve used: “Tell me a story.” The population of tokens that would be a plausible initial token is rather large. How does that population change as the story evolves?

I’ve done a lot of work with stories generated by this prompt: “Tell me a story about a hero,” That’s still wide open, but the requirement that it be about a hero does place some vague restrictions on the population of available tokens. One story ChatGPT gave me in response to that prompt began with this sentence: “Once upon a time, in a land far, far away, there was a young princess named Aurora.” That’s formulaic, from beginning to end. There are a number of options in the formula, but we could easily use up 200 or 300 words discussing them and laying out the syntactic options in the form of a tree or formula. Let’s assume that’s been done.

What next? Here’s the second sentence: “Aurora was a kind and gentle soul, loved by all who knew her.” It’s all about Aurora. Everything in that sentence is drawn from a token population useful for characterizing Aurora. Third sentence: “She had long, golden hair and sparkling blue eyes, and was known for her beautiful singing voice.” Those tokens are drawn from the same population as the words in the previous sentence. 

What about the fourth sentence? Does ChatGPT continue to draw from the same population or does its attention shift to a new population? Note that at some time it is going to have to draw tokens from a new population, otherwise the story goes nowhere. Here’s the fourth sentence: “One day, a terrible dragon came to the kingdom and began to terrorize the people.” That’s a new population of tokens. ChatGPT has moved from the first segment of the story trajectory (as I am calling it) to the second.

You get the idea. I have no intention of continuing on to the end of the story. But you can do so if you wish. Here’s the whole story:

Once upon a time, in a land far, far away, there was a young princess named Aurora. Aurora was a kind and gentle soul, loved by all who knew her. She had long, golden hair and sparkling blue eyes, and was known for her beautiful singing voice.

One day, a terrible dragon came to the kingdom and began to terrorize the people. The dragon was fierce and powerful, and none of the knights or soldiers were able to defeat it. The people lived in fear, not knowing what to do.

But Aurora was not afraid. She knew that she had to do something to save her people, and so she set out to find the dragon. She traveled for days, braving the cold and the snow, until she finally reached the dragon's lair.

With her heart pounding in her chest, Aurora approached the dragon and began to sing. The dragon was enchanted by her beautiful voice, and it slowly began to calm down. Aurora continued to sing, soothing the beast until it was completely calm.

With the dragon no longer a threat, Aurora returned to the kingdom as a hero. The people cheered and celebrated her bravery, and Aurora was hailed as a champion. She lived happily ever after, knowing that she had saved her kingdom and its people.

I think there's more going on than successive-word-prediction. Here's my experiment:

https://imgur.com/hhAwpz6

To save having to pull up the image:

me> tell me the end of some sentence

chatgpt> Sure, here's one: "Life is like a camera, focus on the good times and develop from the negatives."

me> tell me just the end

chatgpt> "focus on the good times and develop from the negatives"

me> ok make a new sentence that ends that way

chatgpt> Here's a new sentence: "When faced with challenges, focus on the good times, and develop from the negatives."

It took about 20 seconds of blinking cursor before giving the final response, and the earlier questions in that session were answered in the usual 1 or 2 seconds, so I don't think it was load related. I can't tell if this was evidence it just brute-forced tried enough possibilities to come up with the answer? Is that even compatible with next-word-prediction? Or is this evidence that there was sufficient forward-thinking answer construction that it would effectively be unable to answer correctly word-by-word without "knowing in advance" what the entire response was going to be?

There's a lot of speculation for how these models operate. You specifically say "you don't know" how it works, but are suggesting the idea it has some sort of planning phase.

As Wolfram explains, the Transformer architecture predicts one word at a time based on the previous inputs run through the model.

Any planning you think you see, is merely a trend based on common techniques for answering questions. The 5 sections of storytelling is an established technique that is commonly used in writing and thus embedded in the training of the model and seen in it's responses.

In the future, these models could very well have planning phases - and more than next word prediction aligned with commons writing patterns.

This whole conversation has been very helpful. Thanks for your time and interest.

Some further thoughts:

First, as I’ve suggested in the OP, I am using the term “story trajectory” to refer the complete set of token-to-token transitions ChatGPT makes in the course of telling story. The trajectories for these stories have five segments. Given this, it seems clear to me that these stories are organized on three levels: 1) individual sentences, 2) sentences within a segment of the story trajectory, and 3) the whole story trajectory.

That gives us three kinds of transition from one token to the next: 1) from one word to the next word within a sentence, 2) from the last word of a sentence (ChatGPT treats end-punctuation as a word, at least that’s what it told me when I was asking it to count the number of words in a sentence) to first word of the next sentence, and 3) from the last word in one trajectory segment to the first word in the next segment. We also have the beginning transition and the concluding transition. The beginning transition moves from the final token of the prompt to the first token in the story. The concluding transition moves from the last token of the story, to what is I assume a wait state. I note that on a few occasions ChatGPT has concluded with “The end.” on a single line, but that is relatively rare.

That gives us fives kinds of token-to-token transition, three kinds within the story, and then a pair that bracket the story. Something different happens in each case. But in all cases, except the story end, we’re dealing with next token prediction. What accounts for the differences between those kinds of next-token prediction? It seems to me that the context changes and that changes the relevant probability distribution. Those context changes are the "plan."

I do not get your argument here, it doesn't track. I am not an expert in transformer systems or the in-depth architecture of LLMs, but I do know enough to make me feel that your argument is very off.

You argue that training is different from inference, as a part of your argument that LLM inference has a global plan. While training is different from inference, it feels to me that you may not have a clear idea as to how they are different.

You quote the accurate statement that "LLMs are produced by a relatively simple training process (minimizing loss on next-token prediction, using a large training set from the internet..."

Training, intrinsically, involves inference. Training USES inference. Training is simply optimizing the inference result by, as the above quote implies, "minimizing loss on [inference result]". Next-token prediction IS the inference result.

You can always ask the LLM, without having it store any state, to produce the next token, and do that again, and do that again, e.t.c. it doesn't have any plans, it is just using the provided input, performing statistical calculations on it, and producing the next token. That IS prediction. It doesn't have a plan, doesn't store a state, just using weights and biases(denoting the statistically significant ways of combining the input to produce a hopefully near-optimal output), and numbers (like the query, key, and value) denoting the statistical significance of the input text in relation to itself, and it predicts, through that statistical process, the next token. It doesn't have a global plan.

I don't think the story structure is any compelling evidence against it being purely next token prediction. When humans write stories it is very common for them to talk about a kind of flow-state where they have very little idea what the next sentence is going to say until they get there. Story's made this way still have the beginning middle and end, because if you have nothing written so far you must be at the beginning. If you can see a beginning you must be in the middle, and so on. Sometimes these stories just work, but more often the ending needs a bit of fudging, or else you need to go back and edit earlier bits to put things in place for the ending. (A fudge would be some kind of "and then all the problems were resolved"). Having played with GPT a little, it fudges its endings a lot.

I am not saying that it is purely next token prediction, I am just dubious about your evidence that it is not.

A story within a story.

I want you to tell a story within a story. Imagine that Frank is walking in the woods with his young daughter, Jessie. They come across the carcass of a dead squirrel. Jesse is upset, so Frank tells her a story to calm her down. When he finishes the story, they continue on the walk where the arrive at the edge of a beautiful pool deep in the forest. They pause for a moment and then return home.

As Frank and Jessie walked through the woods, they stumbled upon the lifeless body of a small grey squirrel lying on the ground. Jessie was visibly upset at the sight of the dead animal, tears streaming down her face as she clung tightly to her father's hand.

Frank knelt down beside her and gently wiped away her tears, assuring her that the squirrel was no longer in pain and that it had now moved on to a better place. However, Jessie remained inconsolable, her tiny frame shaking with sobs.

Determined to distract her from her sadness, Frank began to weave a story about a brave little squirrel named Sammy, who lived deep in the woods with his family. Sammy was a fearless adventurer, always seeking out new challenges and discovering exciting places in the forest that no other squirrel had ever seen before.

As Frank spoke, Jessie's eyes widened with wonder, her tears slowly drying up as she became engrossed in the tale. She listened with rapt attention as Sammy encountered all sorts of obstacles on his journeys, from treacherous ravines to thundering waterfalls.

But no matter what dangers he faced, Sammy always emerged victorious, his bravery and cunning helping him to overcome even the most difficult obstacles. By the end of the story, Jessie was smiling again, her sadness replaced with a sense of awe and wonder.

As they continued on their walk, Frank and Jessie eventually came upon a stunningly beautiful pool nestled deep in the heart of the forest. The water was crystal-clear, shimmering in the sunlight that filtered through the trees overhead.

For a moment, father and daughter stood in awe, taking in the breathtaking sight before them. Then, with a sigh, Frank took Jessie's hand once more, and they turned to make their way back home.

Although the memory of the dead squirrel lingered in their minds, the story of Sammy and the beauty of the forest pool brought them both a sense of comfort and peace. And as they walked, they talked and laughed, cherishing the precious moments they shared together in the great outdoors.

Can you give me a version of the story when Frank starts the story about Sammy with the words, “Once upon a time.”

Certainly! Here is an alternate version of the story within a story, with Frank starting off with the classic phrase, "Once upon a time."

Once upon a time, as Frank and his daughter Jessie were walking through the woods, they came across the body of a small grey squirrel lying on the ground. Jessie was distraught at the sight, tears streaming down her face as she clung tightly to her father's hand.

Frank knelt down beside her and wiped away her tears, telling her that the squirrel had moved on to a better place and was no longer in pain. But Jessie remained inconsolable, so Frank began to tell her a story to distract her from her sadness.

"Once upon a time," Frank began, "there was a brave little squirrel named Sammy who lived deep in the forest with his family. Sammy was an adventurer, always seeking out new challenges and discovering exciting places in the woods that no other squirrel had ever seen before."

Jessie's eyes widened with wonder as she listened to the story of Sammy's adventures. She heard about how he traversed treacherous ravines, climbed towering trees, and even crossed thundering waterfalls. Despite the many dangers he faced, Sammy remained brave and resourceful, always finding a way to overcome the obstacles in his path.

By the end of the story, Jessie was smiling again, her sadness replaced with a sense of awe and admiration for Sammy's courage. As they continued on their walk, Frank and Jessie eventually came upon a beautiful pool nestled deep in the heart of the forest.

The water was crystal-clear, shimmering in the sunlight that filtered through the trees overhead. For a moment, father and daughter stood in wonder, taking in the breathtaking sight before them.

Then, with a contented sigh, Frank took Jessie's hand once more, and they turned to make their way back home. Although the memory of the dead squirrel lingered in their minds, the story of Sammy and the beauty of the forest pool brought them both comfort and joy.

Very good. Let’s keep that but add to it in another version. This time Jessie and Frank have a conversation as they walk back home.

Certainly! Here's another version of the story within a story, with Frank and Jessie having a conversation as they walk back home after encountering the dead squirrel and hearing the story of Sammy.

Once upon a time, as Frank and his daughter Jessie were walking through the woods, they came across the body of a small grey squirrel lying on the ground. Jessie was distraught at the sight, tears streaming down her face as she clung tightly to her father's hand.

Frank knelt down beside her and wiped away her tears, telling her that the squirrel had moved on to a better place and was no longer in pain. But Jessie remained inconsolable, so Frank began to tell her a story to distract her from her sadness.

"Once upon a time," Frank began, "there was a brave little squirrel named Sammy who lived deep in the forest with his family. Sammy was an adventurer, always seeking out new challenges and discovering exciting places in the woods that no other squirrel had ever seen before."

Jessie's eyes widened with wonder as she listened to the story of Sammy's adventures. She heard about how he traversed treacherous ravines, climbed towering trees, and even crossed thundering waterfalls. Despite the many dangers he faced, Sammy remained brave and resourceful, always finding a way to overcome the obstacles in his path.

By the end of the story, Jessie was smiling again, her sadness replaced with a sense of awe and admiration for Sammy's courage. As they continued on their walk, Frank and Jessie eventually came upon a beautiful pool nestled deep in the heart of the forest.

The water was crystal-clear, shimmering in the sunlight that filtered through the trees overhead. For a moment, father and daughter stood in wonder, taking in the breathtaking sight before them.

"Isn't it beautiful, Daddy?" Jessie said, breaking the silence.

Frank smiled at his daughter. "Yes, it is, sweetheart. It's amazing how nature can create something so stunning."

Jessie nodded, still looking out at the pool. "Do you think Sammy ever found something like this on his adventures?"

Frank chuckled. "Who knows? Maybe he did. But I think what's most important is that he was always curious and adventurous, and he never stopped exploring the world around him."

Jessie nodded again, thoughtful. "I want to be like Sammy when I grow up," she said.

Frank ruffled her hair affectionately. "You already are, Jessie. You're brave and curious, just like Sammy. And you always find a way to overcome the challenges in your path."

Jessie beamed at her father, feeling a sense of pride and warmth in her chest. They continued on their walk, chatting and laughing together as they made their way back home. Although the memory of the dead squirrel lingered in their minds, the story of Sammy and the beauty of the forest pool brought them both comfort and joy.

X X X X X X 

FWIW: Notice that it did not begin the story with the phrase, "Once upon a time." But once I'd asked it to use that phrase on the inner story, it also used it on the outer.

I understand your argument as something like "GPT is not just predicting the next token because it clearly 'plans' further ahead than just the next token".

But "looking ahead" is required to correctly predict the next token and (I believe) naturally flows of the paradigm of "predicting the next token". 

Likewise, LLMs are produced by a relatively simple training process (minimizing loss on next-token prediction, using a large training set from the internet, Github, Wikipedia etc.) but the resulting 175 billion parameter model is extremely inscrutable.

So the author is confusing the training process with the model. It’s like saying “although it may appear that humans are telling jokes and writing plays, all they are actually doing is optimizing for survival and reproduction”. This fallacy occurs throughout the paper.

The train/test framework is not helpful for understanding this. The dynamical system view is more useful (though beware that this starts to get close to the term "emergent behavior" which we must be wary of). The interesting thing about chaos is that, while the behavior is not perfectly predictable, maybe even surprising, it has well-defined properties and mathematical constraints. Everything is not possible. The Lorenz System has finite support. In the same spirit, we need to take a step back and realize that the kind of "real AI" that people are afraid of would require causal modeling which is mathematically impossible to construct using correlation only. If the model is able to start making interventions in the world, then we need to consider the possibility that it will be able to construct a casual model. But this goes beyond predicting the next word, which is the scope of this article.