What is the best argument that LLMs are shoggoths?

SOTA LLMs seem to be wildly, wildly superhuman than humans at literal next token prediction [LW · GW].

It's unclear if this implies fundamental differences in how they work versus different specializations.

(It's possible that humans could trained to be much better at next token prediction, but there isn't an obvious methodology which works for this based on intial experiments.)

For a back and forth on whether the "LLMs are shoggoths" is propaganda, try reading this [LW(p) · GW(p)].

In my opinion if you read the dialogue, you'll see the meaning of "LLMs are shoggoths" shift back and forth -- from "it means LLMs are psychopathic" to "it means LLMs think differently from humans." There isn't a fixed meaning.

I don't think trying to disentangle the "meaning" of shoggoths is going to result in anything; it's a metaphor, some of whose understandings are obviously true ("we don't understand all cognition in LLMs"), some of which are dubiously true ("LLM's 'true goals' exist, and are horrific and alien"). But regardless of the truth of these props, you do better examining them one-by-one than in an emotionally-loaded image.

It's sticky because it's vivid, not because it's clear; it's reached for as a metaphor -- like "this government policy is like 1984" -- because it's a ready-to-hand example with an obvious emotional valence, not for any other reason.

If you were to try to zoom into "this policy is like 1984" you'd find nothing; so also here.

Can you say more about what you mean by "Where can I find a post or article arguing that the internal cognitive model of contemporary LLMs is quite alien, strange, non-human, even though they are trained on human text and produce human-like answers, which are rendered "friendly" by RLHF?"

Like, obviously it's gonna be alien in some ways and human-like in other ways. Right? How similar does it have to be to humans, in order to count as not an alien? Surely you would agree that if we were to do a cluster analysis of the cognition of all humans alive today + all LLMs, we'd end up with two distinct clusters (the LLMs and then humanity) right? 

First thoughts:

• Context length is insanely long
• Very good at predicting the next token
• Knows many more abstract facts

These three things are all instances of being OOM better at something specific. If you consider the LLM somewhat human-level at the thing it does, this suggests that it's doing it in a way which is very different from what a human does.

That said, I'm not confident about this; I can sense there could be an argument that this counts as human but ramped up on some stats, and not an alien shoggoth.

Rather, I am looking for a discussion of  evidence that the  LLMs internal  "true" motivation or reasoning system is very different from human, despite the human output, and that in outlying environmental conditions, very different from the training environment, it will behave very differently. A good argument might analyze bits of weird inhuman behavior to try to infer the internal model.

I think we do not understand enough about either the LLM's true algorithms or humans' to make such arguments, except for basic observations like the fact that humans have non-language recurrent state which many LLMs lack.

I wouldn't say that's exactly best argument but for example

I'm not totally sure the hypothesis is well-defined enough to argue about, but maybe Gary Marcus-esque analysis of the pattern of LLM mistakes?

If the internals were like a human thinking about the question and then giving an answer, it would probably be able to add numbers more reliably. And I also suspect the pattern of mistakes doesn't look typical for a human at any developmental stage (once a human can add 3 digit numbers their success rate at 5 digit numbers is probably pretty good). I vaguely recall some people looking at this, but gave forgotten the reference, sorry.

If Earth had intelligent species with different minds, an LLM could end up identical to a member of at most one of them.

Does something like the "I have been a good Bing. 😊" thing count? (More examples.) 

I'd say that's a pretty striking illustration that under the surface of a helpful assistant (in the vein of Siri et al.) these things are weird, and the shoggoth is a good metaphor.

First, a factual statement that is true to the best of my knowledge: LLM state, that is used to produce probability distribution for the next token, is completely determined by the state of its input buffer (plus a bit of indeterminism due to parallel processing and non-associativity of floating point arithmetic).

That is LLM can pass only a single token (around 2 bytes) to its future self. That follows from the above.

What comes next is a plausible (to me) speculation.

For humans what's passed to our future self is most likely much more that a single token. That is a state of the human brain that leads to writing (or uttering) the next word most likely cannot be derived from a small subset of a previous state plus a last written word (that is state of the brain changes not only because we had written or said a word, but by other means too).

This difference can lead to completely different processes that LLM uses to mimic human output, that is potential shoggethification. But to be the real shoggoth LLM also needs a way to covertly update its shoggoth state, that is the part of its state that can lead to inhuman behavior. Output buffer is the only thing it has to maintain state, so the shoggoth state should be steganographically encoded in it, thus severely limiting its information density and update rate.

I wonder how a shoggoth state may arise at all, but it might be my lack of imagination.