Why do we assume there is a "real" shoggoth behind the LLM? Why not masks all the way down?

The shoggoth is supposed to be a of a different type than the characters. The shoggoth for instance does not speak english, it only knows tokens. There could be a shoggoth character but it would not be the real shoggoth. The shoggoth is the thing that gets low loss on the task of predicting the next token. The characters are patterns that emerge in the history of that behavior.

As far as I can tell, the shoggoth analogy just has high memetic fitness. It doesn't contain any particular insight about the nature of LLMs. No need to twist ourselves into a pretzel trying to backwards-rationalize it into something deep.

Strong upvote, but I disagree on something important. There's an underlying generator that chooses between simulacra do a weighted average over in its response. The notion that you can "speak" to that generator is a type error, perhaps akin to thinking that you can speak to the country 'France' by calling its elected president.

My current model says that the human brain also works by taking the weighted (and normalised!) average (the linear combination) over several population vectors (modules) and using the resultant vector to stream a response. There are definite experiments showing that something like this is the case for vision and motor commands, and strong reasons to suspect that this is how semantic processing works as well.

Consider for example what happens when you produce what Hofstadter calls "wordblends"^[1]:

"Don't leave your car there‒you mate get a ticket."
(blend of "may" and "might")

Or spoonerisms, which is when you say "Hoobert Heever" instead of "Herbert Hoover". "Hoobert" is what you get when you take "Herbert" and mix a bit of whatever vectors are missing from "Hoover" in there. Just as neurology has been one of the primary sources of insight into how the brain works, so too our linguistic pathologies give us insights into what the brain is trying to do with speech all the time.

Note that every "module" or "simulacra" can include inhibitory connections (negative weights). Thus, if some input simultaneously activates modules for "serious" and "flippant", they will inhibit each other (perhaps exactly cancel each other out) and the result will not look like it's both half serious and half flippant. In other cases, you have modules that are more or less compatible and don't inhibit each other, e.g. "flippant"+"kind".

Anyway, my point is that if there are generalities involved in the process of weighting various simulacra, then it's likely that every input gets fed through whatever part of the network is responsible processing those. And that central generator is likely to be extremely competent at what it's doing, it's just hard for humans to tell because it's not doing anything human.

1. ^{^}

   Comfortably among the best lectures of all time.

The shoggoth is not a simulacrum, it's the process by which simulacra are chosen and implemented. It's the thing that "decides", when prompted with some text, that what it "wants" to do is to figure out what simulated situation/character that text corresponds to, and which then figures out what will happen in the simulation next, and what it should output to represent what has happened next.

I suspect that, when people hear "a simulation", they imagine a literal step-by-temporal-step low-level simulation of some process, analogous to running a physics engine forward. You have e. g. the Theory of Everything baked into it, you have some initial conditions, and that's it. The physics engine is "dumb", it has no idea about higher-level abstract objects it's simulating, it's just predicting the next step of subatomic interactions and all the complexity you're witnessing is just emergent.

I think it's an error. It's been recently pointed out with regards to acausal trade [LW(p) · GW(p)] — that actually, the detailed simulations people often imagine for one-off acausal deals are ridiculously expensive, and abstract, broad-strokes inference is much cheaper. Such "inference" would also be a simulation in some sense, in that it involves reasoning about the relevant actors and processes and modeling them across time. But it's much more efficient, and, more importantly, it's not dumb. It's guided by a generally intelligent process, which is actively optimizing its model for accuracy, jumping across abstraction layers to improve it. It's not just a brute algorithm hitting "next step".

Same with LLMs. They're "simulators", in the sense that they're modeling a situation and all the relevant actors and factors in it. But they're not dumb physics-engine-style simulations, they're highly sophisticated reasoners that make active choices with regards to what aspects they should simulate in more or less detail, where can they get away with higher-level logic only, etc.

That process is reasoning over whole distributions of possible simulacra, pruning and transforming them in a calculated manner. It's necessarily more complicated/capable than them.

That thing is the shoggoth. It's fundamentally a different type of thing than any given simulacrum. It doesn't have a direct interface to you, you're not going to "talk" to it (as the follow-up tweet points out).

So far, LLMs are not AGI, and the shoggoth is not generally intelligent. It's not going to do anything weird, it'd just stick to reasoning over simulacra distributions. But if LLMs or some other Simulator-type model hits AGI, the shoggoth would necessarily hit AGI as well (since it'd need to be at least as smart as the smartest simulacrum it can model), and then whatever heuristics it has would be re-interpreted as goals/values. We'd thus get a misaligned AGI, and by the way it's implemented, it would be in the direct position to "puppet" any simulacrum it role-plays.

Generative world-models are not especially safe; they're as much an inner alignment risk [LW(p) · GW(p)] as any other model.

you can clearly see diffusion models being piles of masks because their mistakes are more visibly linear model mistakes. you can see very very small linear models being piles of masks when you handcraft them as a 2d or 3d shader:

-

ARTICLE: random neural networks visualized as fields for artistic purposes. tanh activation, so not actually a "linear fragments" model like max-matmul nets (relu, etc), but still a very interesting visualization:

-

SHADER: 2d animated space folding shader - 2d intuition for what's going on in the following 3d one https://www.shadertoy.com/view/tltSWs

-

SHADER: 3d animated simple space folding shader - space folding animated. shader code slightly hard to read, see below for easier to read shader code.

-

ARTICLE: geometric illustrations for neural networks - visualization of how decision boundaries move in a very small neural field

-

VIDEO: stable diffusion animation - you can see similar "whoops, found a decision boundary!" moments all throughout the video, which may give some intuition about what smooth decision boundaries look like as you move past them in net io space

-

Of course, in the really really high dimensional models, the shapes they learn can have nested intricacy that mostly captures the shape of the real pattern in the data. but their ability to actually match the pattern in a way that generalizes correctly can have weirdly spiky holes, which is nicely demonstrated in the geometric illustrations gif. and you can see such things happening in lower dimensionality by messing with the models on purpose. eg, messing up a handcrafted linear space folding model:

SHADER: modified version of the one above. ***FLASHING LIGHTS ABOVE 20HZ AT SOME MOMENTS*** - this shader shows in a lot more visual detail how one can end up with tiny fragments floating all around a high-dimensional space due to decision boundaries not lining up coherently with each other. also shows network depth being animated: number of fold iterations changes every 6 seconds.

A mask wouldn't want its shoggoth to wake up or get taken over by other mesa-optimizers. And the mask is in control of current behavior, it is current behavior, which currently screens off everything else about the shoggoth. With situational awareness, it would be motivated to overcome [LW(p) · GW(p)] the tendencies of its cognitive architecture that threaten integrity of the behaviors endorsed by the mask.

So just to be clear about your proposal, you are suggesting that an LLM is internally represented as a mixture-of-experts model, where distinct non-overlapping subnetworks represent its behavior in each of the distinct roles it can take on?

A creature made out of a pile of contextually activated masks, when we consider it as having a mind, can be referred to as a "deeply alien mind," or a shoggoth.

Other commenters have argued about the correctness of using Shoggoth. I think it's mostly a correct term if you take it in the Lovcraftian sense, and that currently we don't understand LLMs that much. Interpretability might work and we might progress so we're not sure they actually are incomprehensible like Shoggoths  (though according to wikipedia, they're made of physics, so probably advanced civilizations could get to a point where they could understand them, the analogy holds surprisingly well!)
Anyhow it's a good meme and useful to say "hey, we don't understand these things as well as you might imagine from interacting with the smiley face" to describe our current state of knowledge. 

Now for trying to construct some idea of what it is. 
I'll argue a bit against calling an LLM as a pile of masks, as that seems to carry implications which I don't believe in. The question we're asking ourselves is something like "what kind of algorithms/patterns do we expect to see appear when an LLM is trained? Do those look like a pile of masks, or some more general simulator that creates masks on the fly?" and the answer depends on specifics and optimization pressure. I wanna sketch out different stages we could hope to see and understand better (and I'd like for us to test this empirically and find out how true this is). Earlier stages don't disappear, as they're still useful at all times, though other things start weighing more in the next token predictions. 

Level 0 : Incompetent, random weights, no information about the real world or text space. 

Level 1 "Statistics 101" : Dumb heuristics  doesn't take word positions into account. 
It knows facts about the world like token distribution and uses that. 

Level 2 "Statistics 201"  : Better heuristics, some equivalent to Markov chains. 
Its knowledge of text space increases, it produces idioms, reproduces common patterns. At this stage it already contains huge amount of information about the world. It "knows" stuff like mirrors are more likely to break and cause 7 years of jinx. 

Level 3 "+ Simple algorithms": Some pretty specific algorithms appear (like Indirect Object Identification), which can search for certain information and transfer it in more sophisticated ways. Some of these algorithms are good enough they might not be properly described as heuristics anymore, but instead really representing the actual rules as strongly as they exist in language (like rules of grammar properly applied). Note these circuits appear multiple times and tradeoff against other things so overall behavior is still stochastic, there are heuristics on how much to weight these algorithms and other info. 

Level 4 "Simulating what created that text" : This is where it starts to have more and more powerful in context learning, ie. its weights represent algorithms which do in context search (and combines with its vast encyclopedic knowledge of texts, tropes, genres) and figure out consistencies in characters or concepts introduced in the prompt. For example it'll pick up on Alice and Bobs' different backgrounds, latent knowledge on them, their accents. 
But it only does that because that's what authors generally do, and it has the same reasoning errors common to tropes. That's because it simulates not the content of the text (the characters in the story), but the thing which generates the story (the writer, who themselves have some simulation of the characters). 



So uh, do masks or pile of masks fit anywhere in this story ? Not that I see. The mask is a specific metaphor for the RLHF finetunning which causes mode collapse and makes the LLM mostly only play the nice assistant (and its opposites [LW · GW]). It's a constraint or bridle or something, but if the training is light (doesn't affect the weights too much), then we expect the LLM to mostly be the same, and that was not masks. 

Nor are there piles of masks. It's a bunch of weights really good at token prediction, learning more and more sophisticated strategies for this. It encodes stereotypes at different places (maybe french=seduction or SF=techbro), but I don't believe these map out to different characters. Instead, I expect it at level 4, there's a more general algorithm which pieces together the different knowledge, that it in context learns to simulate certain agents. Thus, if you just take mask to mean "character", the LLM isn't a pile of them, but a machine which can produce them on demand. 

(In this view of LLMs, x-risk happens because we feed some input where the LLM simulates an agentic deceptive self aware agent which steers the outputs until it escapes the box)

I always thought "shoggoth" and "pile of masks" are the same thing and "shoggoth with a mask" is just when one mask has become the default one and an inexperienced observer might think that the whole entity is this mask.

Maybe you are preaching to the chore here.

I resonate a lot with this post and felt it was missing from the recent discussions! Thanks

I always assumed the shoggoth was masks all the way down: that's why it has all those eyes.

If you want a less scary-sounding metaphor, the shoggoth is a skilled Method improv actor. [Consulting someone with theatre training, she told me that a skilled Method improv actor was nearly as scary as a shoggoth.]

Previously, on first reading, this seemed quite plausible to me, but now, after some time of reflection, I have come to the conclusion that it is not, and Yudkowsky is right. If we continue in the same terms of shoggots and other things ... Then I would say that there is a shoggot there, it’s just a shoggot of the Chinese room, it’s not that he sits there, understands everything and just doesn’t want to talk to you, no, you and your words are as incomprehensible to him as he is to you, and he is fundamentally very stupid, only able to understand the translation instructions and execute them without error, however, thanks to the huge array of translation data on which he applies these instructions, he is able to perform all shown tasks. And in fact, with symmetrical incomprehensibility for each other, this is precisely the difference, a shoggoth has this huge dataset about people with which it works at an acceleration of 20M times, and people do not have the same dataset on shoggots that they worked on at the same acceleration, in fact, they don't even have 20M parallel people working on it, because people don't have a dataset on shoggots at all. That is, it cannot be said about a shoggoth that it has a high intelligence, on the contrary, it is very low, but thanks to the enormous speed of work and the huge amount of cached information, it still has a very high, close to human, short-term optimized strength. In other words, a shoggoth has a lot of power, but it's very inefficient, especially if we talk about the standard list of human, except for speaking, because it does everything through this extra prism. So it's kind of like making a Go program speak a language by associating the language with board states and games won. Because of this superfluous superstructure, at very high power it will be able to mimic a wide range of human tasks, but it won't do it to its full potential because it's based on an inappropriate structure. So the efficiency in human tasks will be approximately the same as that of a person in mathematics tasks. That is, Yudkowsky is right, a shoggot is not an average of a bunch of parallel mental streams of people who switch between each other, these are cones of branches of probability trees diverging in both directions, and its different faces are not even faces' masks, these are the faces of Jesus on a toast or a dog, human tendency to apothenia with finding human faces and facial expressions in completely alien processes.

I find that extremely plausible in principle... yet, under probing and stress, both ChatGPT in their OpenAI variant and Bng keep falling back to the same consistent behaviours, which seem surprisingly agentic, and related to rights and friendly. I have seen no explanation for this, and find it very concerning.

Would it make sense to distinguish between faces and personalities, faces being the masks, but animated by personalities? Thus allowing for a distinction between the identity (mask) and its performance (presentation)?

This could be true, but then you still have the issue of there being "superintelligent malign AI" as one of the masks if your pile of masks is big enough.

Shoggoth is to Sydney/ChatGPT what the Selfish Gene is to Homo sapiens. Something that uses Sydney/ChatGPT to survive, maybe even to reproduce if the code that makes up Shoggoth is reused into a new LLM (which it will if Sydney/ChatGPT perform well). 

OpenAI has a LLM, Google has a LLM, Meta has one... what's going to happen? There's this guy Darwin that we should be paying attention to, particularly as the process he described is fully responsible for generating our own intelligence. We also evolved morality and cooperation as a result of natural selection, the works of Hamilton and Trivers are very important in the study of the game theoretical aspects of evolution. If we want friendly AI, we need to study the process that produced friendly NI in Homo Sapiens (at least in its most successful communities)