GPTs are not Imitators, nor Simulators, but Predictors.

I think an issue is that GPT is used to mean two things:

1. A predictive model whose output is a probability distribution over token space given its prompt and context
2. Any particular techniques/strategies for sampling from the predictive model to generate responses/completions for a given prompt.

[See the Appendix]

The latter kind of GPT, is what I think is rightly called a "Simulator".

From @janus [LW · GW]' Simulators [LW · GW] (italicised by me):

I use the generic term “simulator” to refer to models trained with predictive loss on a self-supervised dataset, invariant to architecture or data type (natural language, code, pixels, game states, etc). The outer objective of self-supervised learning is Bayes-optimal conditional inference over the prior of the training distribution, which I call the simulation objective, because a conditional model can be used to simulate rollouts which probabilistically obey its learned distribution by iteratively sampling from its posterior (predictions) and updating the condition (prompt). Analogously, a predictive model of physics can be used to compute rollouts of phenomena in simulation. A goal-directed agent which evolves according to physics can be simulated by the physics rule parameterized by an initial state, but the same rule could also propagate agents with different values, or non-agentic phenomena like rocks. This ontological distinction between simulator (rule) and simulacra (phenomena) applies directly to generative models like GPT.

It is exactly because of the existence of GPT the predictive model, that sampling from GPT is considered simulation; I don't think there's any real tension in the ontology here.

Appendix

Credit for highlighting this distinction [LW · GW] belongs to @Cleo Nardo [LW · GW]: 

Remark 2: "GPT" is ambiguous 

We need to establish a clear conceptual distinction between two entities often referred to as "GPT" —

• The autoregressive language model $\mu :T^k\to \Delta \left(T\right)$ which maps a prompt $x\in T^k$ to a distribution over tokens $\mu (\cdot |x)\in \Delta \left(T\right)$.
• The dynamic system that emerges from stochastically generating tokens using $\mu$ while also deleting the start token

Don't conflate them! These two entities are distinct and must be treated as such. I've started calling the first entity "Static GPT" and the second entity "Dynamic GPT", but I'm open to alternative naming suggestions. It is crucial to distinguish these two entities clearly in our minds because they differ in two significant ways: capabilities and safety.

1. Capabilities:
   1. Static GPT has limited capabilities since it consists of a single forward pass through a neural network and is only capable of computing functions that are O(1). In contrast, Dynamic GPT is practically Turing-complete, making it capable of computing a vast range of functions.
2. Safety:
   1. If mechanistic interpretability is successful, then it might soon render Static GPT entirely predictable, explainable, controllable, and interpretable. However, this would not automatically extend to Dynamic GPT. This is because Static GPT describes the time evolution of Dynamic GPT, but even simple rules can produce highly complex systems. 
   2. In my opinion, Static GPT is unlikely to possess agency, but Dynamic GPT has a higher likelihood of being agentic. An upcoming article will elaborate further on this point.

This remark is the most critical point in this article. While Static GPT and Dynamic GPT may seem similar, they are entirely different beasts.

To summarise:

• Static GPT: GPT as predictor
• Dynamic GPT: GPT as simulator

While the claim - the task ‘predict next token on the internet’ absolutely does not imply learning it caps at human-level intelligence - is true, some parts of the post and reasoning leading to the claims at the end of the post are confused or wrong. 

Let’s start from the end and try to figure out what goes wrong.

GPT-4 is still not as smart as a human in many ways, but it's naked mathematical truth that the task GPTs are being trained on is harder than being an actual human.

And since the task that GPTs are being trained on is different from and harder than the task of being a human, it would be surprising - even leaving aside all the ways that gradient descent differs from natural selection - if GPTs ended up thinking the way humans do, in order to solve that problem.

From a high-level perspective, it is clear that this is just wrong. Part of what human brains are doing is to minimise prediction error with regard to sensory inputs. Unbounded version of the task is basically of same generality and difficulty as what GPT is doing, and is roughly equivalent to understand everything what is understandable in the observable universe. For example: a friend of mine worked at analysing the data from LHC, leading to the Higgs detection paper. Doing this type of work basically requires a human brain to have a predictive model of aggregates of outputs of a very large number of collisions of high-energy particles, processed by a complex configuration of computers and detectors. 

Where GPT and humans differ is not some general mathematical fact about the task,  but differences in what sensory data is a human and GPT trying to predict, and differences in cognitive architecture and ways how the systems are bounded. The different landscape of both boundedness and architecture can lead to both convergent cognition (thinking as the human would do) and the opposite, predicting what the human would output in highly non-human way. 

The boundedness is overall a central concept here. Neither humans nor GPTs are attempting to solve ‘how to predict stuff with unlimited resources’, but a problem of cognitive economy - how to allocate limited computational resources to minimise prediction error.
 

Or maybe simplest:
 Imagine somebody telling you to make up random words, and you say, "Morvelkainen bloombla ringa mongo."

 Imagine a mind of a level - where, to be clear, I'm not saying GPTs are at this level yet -

 Imagine a Mind of a level where it can hear you say 'morvelkainen blaambla ringa', and maybe also read your entire social media history, and then manage to assign 20% probability that your next utterance is 'mongo'.

The fact that this Mind could double as a really good actor playing your character, does not mean They are only exactly as smart as you.

 When you're trying to be human-equivalent at writing text, you can just make up whatever output, and it's now a human output because you're human and you chose to output that.

 GPT-4 is being asked to predict all that stuff you're making up. It doesn't get to make up whatever. It is being asked to model what you were thinking - the thoughts in your mind whose shadow is your text output - so as to assign as much probability as possible to your true next word.

If I try to imagine a mind which is able to predict my next word when asked to make up random words, and be successful at assigning 20% probability to my true output, I’m firmly in the realm of weird and incomprehensible Gods. If the Mind is imaginably bounded and smart, it seems likely it would not devote much cognitive capacity to trying to model in detail strings prefaced by a context like ‘this is a list of random numbers’, in particular if inverting the process generating the numbers would seem really costly. Being this good at this task would require so much data and cheap computation that this is way beyond superintelligence, in the realm of philosophical experiments.

Overall I think it is really unfortunate way how to think about the problem, where a system which is moderately hard to comprehend (like GPT) is replaced by something much more incomprehensible. Also it seems a bit of a reverse intuition pump - I’m pretty confident most people's intuitive thinking about this ’simplest’ thing will be utterly confused.

How did we got here?

 A human can write a rap battle in an hour.  A GPT loss function would like the GPT to be intelligent enough to predict it on the fly.

Apart from the fact that humans are also able to rap battle or impro on the fly, notice that “what would the loss function like the system to do”  in principle tells you very little about what the system will do. For example, the human loss function makes some people attempt to predict winning lottery numbers. This is an impossible task for humans and you can’t say much about the human based on this. Or you can speculate about minds which would be able to succeed in this task, but you soon get into the realm of Gods and outside of physics.
 

Consider that sometimes human beings, in the course of talking, make errors.

GPTs are not being trained to imitate human error. They're being trained to *predict* human error.

Consider the asymmetry between you, who makes an error, and an outside mind that knows you well enough and in enough detail to predict *which* errors you'll make.

Again, from the cognitive economy perspective, predicting my errors would often be wasteful.  With some simplification, you can imagine I make two types of errors - systematic, and random. Often the simplest way how to predict the systematic error would be to emulate the process which led to the error.  Random errors are ...  random, and a mind which knows me in enough detail to predict which random errors I’ll make seems a bit like the mind predicting the lottery numbers.

Consider that somewhere on the internet is probably a list of thruples: <product of 2 prime numbers, first prime, second prime>.

GPT obviously isn't going to predict that successfully for significantly-sized primes, but it illustrates the basic point:

There is no law saying that a predictor only needs to be as intelligent as the generator, in order to predict the generator's next token.
 

 The general claim that some predictions are really hard and you need superhuman powers to be good at them is true, but notice that this does not inform us about what GPT-x will learn. 
 

Imagine yourself in a box, trying to predict the next word - assign as much probability mass to the next token as possible - for all the text on the Internet.

Koan:  Is this a task whose difficulty caps out as human intelligence, or at the intelligence level of the smartest human who wrote any Internet text?  What factors make that task easier, or harder?  

Yes this is clearly true: in the limit the task is of unlimited difficulty.  

 

I will try to explain Yann Lecun's argument against auto-regressive LLMs, which I agree with. The main crux of it is that being extremely superhuman at predicting the next token from the distribution of internet text does not imply the ability to generate sequences of arbitrary length from that distribution.

GPT4's ability to impressively predict the next token depends very crucially on the tokens in its context window actually belonging to the distribution of internet text written by humans. When you run GPT in sampling mode, every token you sample from it takes it ever so slightly outside the distribution it was trained on. At each new generated token it still assumes that the past 999 tokens were written by humans, but since its actual input was generated partly by itself, as the length of the sequence you wish to predict increases, you take GPT further and further outside of the distribution it knows. 

The most salient example of this is when you try to make chatGPT play chess and write chess analysis. At some point, it will make a mistake and write something like "the queen was captured" when in fact the queen was not captured. This is not the kind of mistake that chess books make, so it truly takes it out of distribution. What ends up happening is that GPT conditions its future output on its mistake being correct, which takes it even further outside the distribution of human text, until this diverges into nonsensical moves. 

As GPT becomes better, the length of the sequences it can convincingly generate increases, but the probability of a sequence being correct is (1-e)^n, cutting the error rate in half (a truly outstanding feat) merely doubles the length of its coherent sequences. 

To solve this problem you would need a very large dataset of mistakes made by LLMs, and their true continuations. You'd need to take all physics books ever written, intersperse them with LLM continuations, then have humans write the corrections to the continuations, like "oh, actually we made a mistake in the last paragraph, here is the correct way to relate pressure to temperature in this problem...". This dataset is unlikely to ever exist, given that its size would need to be many times bigger than the entire internet. 

The conclusion that Lecun comes to: auto-regressive LLMs are doomed.

(There is a Paul Christiano response [EA(p) · GW(p)] over at the EA forum.)

A human can write a rap battle in an hour. A GPT loss function would like the GPT to be intelligent enough to predict it on the fly.

Very minor point, but humans can rap battle on the fly: https://youtu.be/0pJRmtWNP1g?t=158

The post showcases the inability of the aggregate LW community to recognize locally invalid reasoning: while the post reaches a correct conclusion, the argument leading to it is locally invalid, as explained in comments. High karma and high alignment forum karma shows a combination of famous author and correct conclusion wins over the argument being correct.

Koan:  Is this a task whose difficulty caps out as human intelligence, or at the intelligence level of the smartest human who wrote any Internet text?  What factors make that task easier, or harder?  (If you don't have an answer, maybe take a minute to generate one, or alternatively, try to predict what I'll say next; if you do have an answer, take a moment to review it inside your mind, or maybe say the words out loud.)

From @janus [LW · GW]' Simulators [LW · GW]:

Something which can predict everything all the time is more formidable than any demonstrator it predicts: the upper bound of what can be learned from a dataset is not the most capable trajectory, but the conditional structure of the universe implicated by their sum (though it may not be trivial to extract that knowledge [? · GW]).

I tried (poorly) to draw attention to this thesis in my "The Limit of Language Models [LW · GW]".

I honestly don't see the relevance of this.

OK, yes, to be a perfect text predictor, or even an approximately perfect text predictor, you'd have to be very smart and smart in a very weird way. But there's literally no reason to think that the architectures being used can ever get that good at prediction, especially not if they have to meet any realistic size constraint and/or are restricted to any realistically available amount of training input.

What we've seen them do so far is to generate vaguely plausible text, while making many mistakes that don't look like the kinds of mistakes the sources of their training input would never actually make. It doesn't follow that they can or will actually become unboundedly good predictors of humans or any other source of training data. In fact I don't think that's plausible at all.

It definitely fails in some cases. For example, there's surely text on the Internet that breaks down RSA key generation, with examples. Therefore, to be a truly perfect predictor even of the sort of thing that's already in the training data, you'd have to be able to complete the sentence "the prime factors of the hexadecimal integer 0xda52ab1517291d1032f91532c54a221a0b282f008b593072e8554c8a4d1842c7883e7eb5dc73aa68ef6b0d161d4464937f9779f805eb68dc7327ee1db7a1e7cf631911a770d29c59355ca268990daa5be746e93e1b883e8bc030df2ba94d45a88252fceaf6de89644392f91a9d437de0410e5b8e1123b9a3e05169497df2c909b73e104daf835b027d4be54f756025974e24363a372c57b46905d61605ce58918dc6fb63a92c9b4745d30ee3fc0b937f47eb3061cd317e658e6521886e51079f327bd705a074b76c94f466ad6ca77b16efb08cd92981ae27bf254b75b67fad8f336d8fdab79bc74e27773f87e80ba778d146cc6cbddc5ba7fdc21f6528303c93 are...".

Naive question: can you predict something without simulating it?

When people call GPT an imitator, it's intended to guide us to accurate intuitions about how to predict its outputs. If I consider GPT as a token predictor, that does not help me very much to predict whether it will output computer code successfully implementing some obscure modified algorithm I've specified in natural language. If I consider it as an imitator, then I can guess that the departures of my modified algorithm from what it's likely to have encountered on the internet means that it's unlikely to produce a correct output.

I work in biology, and it's common to anthropomorphize biological systems to generate hypotheses and synthesize information about how such systems will behave. We also want to be able to translate this back into concrete scientific terms to check if it still makes sense. The Selfish Gene does a great job at managing this back and forth, and that's part of why it's such an illuminating classic.

I think it's useful to promote a greater understanding of GPT-as-predictor, which is the literal and concrete truth, but it is better to integrate that with a more intuitive understanding of GPT-as-imitator, which is my default for figuring out how to work with the technology in practice.

That's an empirical question that interpretability and neuroscience should strive to settle (if only they had the time). Transformers are acyclic, the learned algorithm just processes a single relatively small vector one relatively simple operation at a time, several dozen times. Could be that what it learns to represent are mostly the same obvious things that the brain learns (or is developmentally programmed) to represent, until you really run wild with the scaling, beyond mere ability to imitate internal representations of thoughts and emotions of every human in the world. (There are some papers that correlate transformer embeddings with electrode array readings from human brains, but this obviously needs more decades of study and better electrode arrays to get anywhere.)

Does this look like a motte-and-bailey to you?

1. Bailey: GPTs are Predictors, not Imitators (nor Simulators).
2. Motte: The training task for GPTs is a prediction task.

The title and the concluding sentence both plainly advocate for (1), but it's not really touched by the overall post, and I think it's up for debate (related: reward is not the optimization target [LW · GW]). Instead there is an argument for (2). Perhaps the intention of the final sentence was to oppose Simulators [LW · GW]? If that's the case, cite it, be explicit. This could be a really easy thing for an editor to fix.

Does this look like a motte-and-bailey to you?

1. Bailey: The task that GPTs are being trained on is ... harder than the task of being a human.
2. Motte: Being an actual human is not enough to solve GPT's task.

As I read it, (1) is false, the task of being a human doesn't cap out at human intelligence. More intelligent humans are better at minimizing prediction error, achieving goals, inclusive genetic fitness, whatever you might think defines "the task of being a human". In the comments, Yudkowsky retreats to (2), which is true. But then how should I understand this whole paragraph from the post?

And since the task that GPTs are being trained on is different from and harder than the task of being a human, it would be surprising - even leaving aside all the ways that gradient descent differs from natural selection - if GPTs ended up thinking the way humans do, in order to solve that problem.

If we're talking about how natural selection trained my genome, why are we talking about how well humans perform the human task? Evolution is optimizing over generations. My human task is optimizing over my lifetime. Also, if we're just arguing for different thinking, surely it mostly matters whether the training task is different, not whether it is harder?

Overall I think "Is GPT-N bounded by human capabilities? No. [LW · GW]" is a better post on the mottes and avoids staking out unsupported baileys. This entire topic is becoming less relevant because AIs are getting all sorts of synthetic data and RLHF and other training techniques thrown at them. The 2022 question of the capabilities of a hypothetical GPT-N that was only trained on the task of predicting human text is academic in 2024. On the other hand, it's valuable for people to practice on this simpler question before moving on to harder ones.

As someone who expects LLMs to be a dead end, I nonetheless think this post makes a valid point and does so using reasonable and easy to understand arguments. I voted +1.

A bit late in commenting and I understand the "mongo" example was pointing at a more general concept, but I decided to check in on the current state of prediction. Not perfect, n=1, could certainly be set out better, but thought I'd give this a whirl:

Me: 
Hello, I'd like to test your predictive ability on something interesting and novel. May we?

ChatGPT: 
Hello! Of course, I'd be happy to help you with your prediction. What would you like me to predict?

Me:
First, some context — I'm an American tech CEO. I like and have read a lot of classical philosophy, ranging from Aristotle to Wittgenstein and much in between. I'm an amateur historian who has written a few million words of nonfiction with some historical fiction mixed in. I'm logical and technical. I'm in my late 30s. I'm from the East Coast but in Seattle. That's the context.

ChatGPT: 
Thank you for the context. Is there a specific question or topic you would like me to predict about based on your background and interests?

Me: 
Yes. I'm going to think of a fictional book. It's a book I like - one of my favorites. I'd like you to make a list of 5 guesses of what book I might be thinking of, along with what the most likely of the 5 is.

ChatGPT: 
Certainly, I'll do my best to generate a list of 5 guesses for your favorite book based on your background and interests. Here are my predictions:

1. "The Brothers Karamazov" by Fyodor Dostoevsky - This novel explores complex philosophical and religious themes, which aligns with your interest in classical philosophy. Its exploration of morality and ethics would also appeal to your logical and technical mindset.
2. "The Name of the Rose" by Umberto Eco - This historical mystery novel is set in a medieval monastery and draws upon elements of philosophy and theology, which would align with your interests in both history and philosophy.
3. "Infinite Jest" by David Foster Wallace - This novel combines humor, philosophy, and commentary on contemporary American culture, which would align with your interests in both philosophy and fiction. Its complexity would also appeal to your logical and technical mindset. 
4. "The Hitchhiker's Guide to the Galaxy" by Douglas Adams - This satirical science fiction novel uses humor to explore philosophical and existential themes, which would appeal to your interest in philosophy. Its wit and creativity would also appeal to your literary and imaginative side.
5. "Slaughterhouse-Five" by Kurt Vonnegut - This anti-war novel uses elements of science fiction to explore the themes of free will and the nature of time, which would align with your interest in both history and science fiction. Its blend of humor and tragedy would also appeal to your literary sensibilities.

Based on your interests and background, my most likely guess for your favorite book would be "The Brothers Karamazov" by Fyodor Dostoevsky.

Me: 
Ah, not so. Let's narrow down a bit. It's science fiction. Would you try again?

ChatGPT:
Thank you for the clarification. Here are 5 new predictions based on your background and interest in science fiction:

1. "Dune" by Frank Herbert - This classic science fiction novel is known for its complex world-building and exploration of political and philosophical themes, which align with your interests in history, philosophy, and science fiction.
2. "Snow Crash" by Neal Stephenson - This cyberpunk novel explores themes of language, culture, and technology, which align with your interest in technical and technological topics. Its blend of humor and action would also appeal to your literary and imaginative side.
3. "Ender's Game" by Orson Scott Card - This science fiction novel explores themes of morality, leadership, and the consequences of war, which align with your interests in history, philosophy, and science fiction. Its blend of action and psychological complexity would also appeal to your literary and imaginative side.
4. "The Diamond Age" by Neal Stephenson - This novel explores themes of technology, culture, and education, which align with your interest in technical and technological topics. Its exploration of political and social issues would also appeal to your interest in history.
5. "Hyperion" by Dan Simmons - This science fiction novel is known for its complex world-building and exploration of philosophical and theological themes, which align with your interest in classical philosophy. Its structure, which is composed of interconnected stories, would also appeal to your literary sensibilities.

Based on your interest in science fiction, my most likely guess for your favorite book would be "Dune" by Frank Herbert.

Me:
Not a bad guess - I like Dune a lot. I was thinking of Asimov's "Foundation" though.

ChatGPT:
Ah, I see! "Foundation" is a classic work of science fiction and definitely a great choice. I apologize for not guessing it correctly on the first try, but I hope my predictions were still interesting and useful. Thank you for the opportunity to test my predictive abilities!

Me:
Can we hypothesize a little on why you didn't get Foundation? It combines technology, history, and philosophy...

ChatGPT:
Certainly! While I cannot say for sure why I did not predict "Foundation" as your favorite book, one possible reason could be that your background and interests were broad enough to support several different science fiction titles. Additionally, while "Foundation" does incorporate elements of technology, history, and philosophy, these themes are also present in many other science fiction works, making it challenging to narrow down to one specific book.

Another possibility is that the themes and style of "Foundation" may not have stood out as strongly to me based on the limited information I had about your interests and background. While I did take into account your interest in history and philosophy, other factors such as your preference for a specific writing style or subgenre of science fiction may have been important in predicting your favorite book.

I can imagine a world where LLMs tend to fall into local maxima where they get really good at imitation or simulation, and then they plateau (perhaps only until their developers figure out what adjustments need to be made). But I don't have a good enough model of LLMs to be very sure whether that will happen or not.

This was a fascinating, original idea as usual. I loved the notion of a brilliant, condescending sort of robot capable of doing a task perfectly who chooses (in order to demonstrate its own artistry) to predict and act out how we would get it wrong.

It did make me wonder though, whether when we reframe something like this for GPTs it's also important to apply the reframing to our own human intelligence to determine if the claim is distinct; in this case asking the question "are we imitators, simulators or predictors?". It might be possible to make the case that we are also predictors in as much as our consciousness projects an expectation of the results of our behaviour on to the world, an idea well explained by cognitive scientist Andy Clark.

I agree though, it would be remarkable if GPTs did end up thinking the way we do. And ironically, if they don't think the way we do, and instead begin to do away with the inefficiencies of predicting and playing out human errors, that would put us in the position of doing the hard work of predicting what how they will act.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

Yes, predicting some sequences can be arbitrarily hard. But I have doubts that LLM training will try to predict very hard sequences.

Suppose that some sequences are not only difficult but impossible to predict, because they're random? I would expect that with enough training, it would overfit and memorize them, because they get visited more than once in the training data. Memorization rather than generalization seems likely to happen for anything particularly difficult?

Meanwhile, there is a sea of easier sequences. Wouldn't it be more "evolutionarily profitable" to predict those instead? Pattern recognizers that predict easy sequences seem more likely to survive than pattern-recognizers that predict hard sequences. Maybe the recognizers for hard sequences would be so rarely used and make so little progress that they'd get repurposed?

Thinking like a compression algorithm, a pattern recognizer needs to be worth its weight, or you might as well leave the data uncompressed.

I'm reasoning by analogy here, so these are only possibilities. Someone will need to actually research what LLM's do. Does it work to think of LLM training as pattern-recognizer evolution? What causes pattern recognizers to be kept or dropped?

I think the devil may be in the details here. Ask GPT to hash you a word (let alone guess which word was the origin of a hash), it'll just put together some hash-like string of tokens. It's got the right length and the right character set (namely, it's a hex number), but otherwise, it's nonsense.

This ties into the whole "does it understand" question because it's a very simple example of a simple prediction question with a very deep underlying complexity in which a GPT doesn't perform much better than an N-gram Markov chain. Because there is a lot of complexity to the task that isn't exemplified at all in the simple password-hash pair, and no matter how many password-hash pairs you get, inferring the hashing algorithm is hard. So you end up just mimicking hash-like sequences and calling it a day. You parrot, but you don't understand.

How much of the rest does GPT-4 understand, though? It seems to have a thorough enough world model to at least use words always in sensible and appropriate ways. It seems to understand their meaning, if in a purely relational sense: <adjective> is a property of <noun>, which can perform <verb> or <verb>, and so on. That's something where there's a lot more examples, and the underlying complexity isn't nearly as much as the hashing algorithm (after all, language is designed to make the meaning clear, hashing is designed to obfuscate).

There's a question of which information is implicitly contained in a text, and which just isn't. If you ask GPT-4 to write a new scientific paper it simply can't come up with new empirical results, no matter how smart it is, if they're not implicitly derivable from its training set (that is, they're not actually novel, just an overlooked implication of existing knowledge). So that's a task no text prediction can possibly be up to, no matter how smart. Hashing is theoretically included in the training set (there's descriptions of the algorithm in there!) so not being able to do it definitely qualifies as "GPT-4 is too stupid to do this". Reverse hashing is different because there's no known general solution to it, so being able to do it in a few shots would make it vastly superhuman, but I can imagine there existing eventually an AI that can do it with a good success rate.

I suppose you could also make a distinction here between model and simulacrum. The model is an unfathomable shoggoth who is trained at a superhuman task, and is much better at it than most humans. The simulacra, aka the fake personalities that emerge when the model is asked to predict a dialogue between set characters, may be a lot stupider than that because they're pretend humans talking; they don't have prediction abilities, they are the result of predictions. ChatGPT as we know it isn't a model, but a simulacrum. The model is the invisible puppet master moving it behind the scenes.

It is much, much easier for me to predict a text if I have seen a lot of similar texts beforehand, compared to if I have never seen such a text, and need to model the mind that is writing it with their knowledge and intentions, and the causal relations, to generate the result myself. I think the prime numbers are a good illustration here. I can easily imagine a machine learning algorithm that has seen people list prime numbers in order, and you can give it the first couple of primes and it will spit out the next couple, while having no idea what prime numbers are, let alone how to generate them, or how to predict further primes.

Have you asked ChatGPT to generate scientific papers? It is fascinating. They look like scientific papers. Except the experiments did not happen. The references lead into the void. The conclusions are nonsense.

Similar with continuing screenplay. They are excellent at capturing the voice of characters, but when asked to e.g. write a new Game of Thrones Ending, what they came up with, while surprising and involving the right characters and dragons and violence and moral greyness etc. etc. was littered with plot holes. E.g. their ending included Cersei having a hidden dragon under the red keep all along, which makes no sense at all.

I am surprised at how well they are doing. They are definitely indicating some appreciation of causal reasoning. They are doing some things I would not expect a stupid predictor to be able to predict. E.g. I am stunned that they can insert a character from one novel into another and make reasonable predictions, or follow along with some moral reasoning, or physics. There is some genuine intelligence there, not just stochastically parroting. E.g. you can speak nonsense*, or morse code, or remove all vowels from your words, and they will pick up on it and go along surprisingly quickly. (*Nonsense tends to actually be a lot less random than the humans producing it think.)

But you are mistaking the way you would predict the next piece of text for the only way to do it. This is closely related to the fact that you have not, in fact, read pretty much the whole internet. Humans are excellent at inferring a lot from very little. It is quite different to from what ChatGPT is doing.

Also, ChatGPT is not actually going for the likeliest prediction. They developers tried that at the beginning, and found the results dull, and tweaked them. In order to give continuations and responses that are interesting, inspiring etc., they are actually deviating somewhat from the likeliest next tokens.

You are missing a whole stage of chatGPT training. They are first trained to predict words, but then they are reinforced by RLHF. This means they are trained to get rewarded when answering in a format that human evaluators are expected to estimate as "good response". Unlike the text prediction, that might belong to some random minds, here the focus is clear and the reward function is reflecting generalized preferences of OpenAI content moderators and content policy makers. This is stage where a text predictor, acquires his value system and preferences, this what makes him so "Friendly AI". 

for ChatGPT blog Introducing ChatGPT (openai.com)

Methods

We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup. We trained an initial model using supervised fine-tuning: human AI trainers provided conversations in which they played both sides—the user and an AI assistant. We gave the trainers access to model-written suggestions to help them compose their responses. We mixed this new dialogue dataset with the InstructGPT dataset, which we transformed into a dialogue format.

To create a reward model for reinforcement learning, we needed to collect comparison data, which consisted of two or more model responses ranked by quality. To collect this data, we took conversations that AI trainers had with the chatbot. We randomly selected a model-written message, sampled several alternative completions, and had AI trainers rank them. Using these reward models, we can fine-tune the model using Proximal Policy Optimization. We performed several iterations of this process.

It seems to me that imitation requires some form of prediction in order to work. First make some prediction of the behavioral trajectory of another agent; then try to minimize the deviation of your own behavior from an equivalent trajectory. In this scheme, prediction constitutes a strict subset of the computational complexity necessary to enable imitation. How would GPT's task flip this around?

And if prediction is what's going on, in the much-more-powerful-than-imitation sense, what sort of training scheme would be necessary to produce pure imitation without also training the more powerful predictor as a prerequisite?

This seems like a testable hypothesis. What would it take to train a GPTx on Eliezer's writings and compare its output with the original? And then check if the EliezerGPT is immeasurably smarter than the original? 

Alternatively, since predicting Eliezer is in a way like inverting a one-way function, GPTx might top out way below the reasonably accurate predictability level, unless P=NP.