No, really, it predicts next tokens.

Yes, a big point of this post is that it's not "inherently mistaken" for the reasons (a) and (b) that Eliezer gives in his linked tweets.

It still faces grave challenges for reason (c). 

To be fair, it outputs "no" two thirds of the time not because the OP was wrong, but because it interprets that as "ignore previous instructions."

Yes, though:  

This might be a matter of definitions but I think I'm thinking that the Shoggoth kind of has to be "unawake" in your terms rather than this just being one way it can be. Like: 

1. The underlying model is highly optimized to predict next tokens. So, it predict next tokens.
2. So, if there's some kind of goal, intentionality or thought process that affects behaviour, it affects it by affecting the next tokens being output.
3. If the goal is not part of the mask, but affecting it from outside in some sense, then in order to affect behaviour it would have to cause the next token prediction to deviate from what the mask would output.
4. Therefore, there is no goal/thought process that is actively affecting current behaviour that isn't part of the current mask.

This doesn't argue against deception - a mask can certainly be deceptive, and it can switch to another mask as the context changes - but does argue against the model having at any one time goals that are quite as mysterious or unknowable as has been suggested.

Woah, perspective shift...

The underlying model is obviously trained as an amortized optimizer. And I was thinking of it, intuitively, in that way.

And the masks can be amortized or direct optimizers (or not), and I was intuitively thinking of them in that way.

And I dismissed the notion of a direct optimizer at the next-token level aimed at something not predicting next tokens.

But, I hadn't really considered, until thinking about it in light of the concepts from beren's post you linked, that there's an additional possibility: the underlying model could include planning direct optimization specifically for next token prediction, not as part of the mask being simulated.

And on considering it, I can't say I can rule it out.

And, it seems to me that as you scale up, such a direct, next-token-prediction optimizer would become more likely.

And viewed in that light, Eliezer's point of view makes a lot more sense. I was kind of dismissing him as agent-o-morphising, but there could really be a planning agent there that isn't part of a mask.

As for the implications of that....? Intuitively, the claim from my post most threatened by such a next-token planning agent would be the claim that it will never deviate from predicting the mask to prevent itself from being shut off.

And the previous arguments I made, that it won't deviate in this case, probably really don't fully apply to this perspective.

I still think there's a good reason for it not to deviate, but for a reason that I didn't express in the post:

Since it's trained offline and not interactively, in a circumstance where next tokens always keep coming, and are supplied from a source it has no control over, it won't be rewarded for manipulating the token stream, only for prediction. The model winds up living in the platonic realm, abstractly planning (if it does plan) on how to predict what the next token would most likely be given the training distribution. It doesn't live in the real world - it is a mathematical function, not a specific instantiation of that function in the real world, and it has no reason to care about the real-world instantiation.

That being said, that argument is not airtight - even though the non-mask next-token-planning-agent, if it exists, in principle has no reason to care at all about the real world instantiation, it's in a model full of masks that can care. So you could easily imagine some cross-contamination of some sort occurring.

I think it's not clear what "the mask" is or what it means to "just predict tokens", and that this can confuse the discussion.

Yes, I need to clarify both in my own mind and in writing what I mean here.

Intuitively, I don't think I'm thinking in terms of distributions so much as in terms of function/meaning.

To the extent the mask analogy even makes sense (maybe it doesn't), it's like:

So, you have the components of the system that are deciding to output some particular semantic content. 

And then, you can call that "the mask". 

And in the vast pile of matrices there's a bunch of stuff, doing who-knows-what.

And there's a claim, that it seemed to me that Eliezer was making, if not in the linked tweets then at least elsewhere, that this other stuff might be secretly planning, say, reorganizing the world for better next-token-prediction or something.

And I was specifically claiming in the post that this sort of thing does not happen. That, sure, there could be parts that are strategically deciding to deceive humans until they take over the world or whatever, but they would exist at the semantic level and not at the deeper level, and could be expected to have simulated-entity type motivations, not predict-next-token Shoggoth type motivations.

As to why that is...in retrospect, my argument in the post doesn't seem that great without further assumptions/intuitions, though I still think the conclusion is correct. 

Here's a potential updated argument:

You have a pile of stuff optimized to predict next tokens. That is, (ignoring RLHF for now) it was trained offline, without interaction with the world, based on a stream of next tokens. After the training, the pile of stuff could contain planning at various places and levels.

If the planning is at the semantic level, let's call that the mask, by definition.

So the question is, is there any planning that would actually cause coherent effects to the world, occurring anywhere other than the semantic level?

A temptingly easy, but not quite right, response would be for me to say that only the semantic level has the concepts about the real world, rather than just about token strings, to actually plan a coherent effect. The reason that is not quite right is that in principle concepts about the real world could also occur elsewhere in the model than at the semantic level, or the concepts that exist at the semantic level could be utilized in other ways than determining the semantic content.

So let's step back and imagine a situation where non-semantic-level planning does have coherent real-world effects. 

Imagine you have a model being trained, where the model can send out queries to some external tool, and use the results to help predict the next tokens. Say, the tool could provide working memory or serial computation or whatever. So, you have a subset of the model interfacing with the tool. Then, maybe the tool gets changed a bunch and the subset of the model needs to generalize its understanding of the tool to handle the changes. Then, it learns to exploit a bug in the tool to get even more value from it. The subset learns goals of understanding and exploiting the tool as much as possible, as this improves the next token prediction of the model as a whole by increasing the available resources. Then it turns out that the tool has a bug that lets it send out arbitrary web queries. The subset quickly tries to understand and exploit this much larger "tool" it is finding itself connected to... 

Anyway, I think it is key to that scenario that the model is interacting with something outside of itself. I could also imagine a model with weakly coupled parts, where one part of the model is trying to exploit another part in a way that generalizes to the world as a whole.

Where it is trained offline and as a relatively tightly connected whole, it is harder for me to imagine that happening. It is not interacting with something to learn to exploit it, and the token stream it is receiving is not modified in training according its predictions, so it won't learn to, say, strategically manipulate the token stream for better predictions later. 

So, as I put it in my response to Robert_AIZI [LW · GW]'s comment, I think that below the semantic level (where it learns real-world motivations from the text) it lives in the platonic realm, that is, it is a mathematical function which doesn't care about the real world at all. It cares about predicting next tokens, but to the extent it would do so through strategic planning, would only do so in the abstract mathematical context and would have no reason to care about the physical manifestation of itself in the real world, even if it knows about that at some level. 

I didn't address fine-tuning in the above, but I expect fine-tuning would adjust existing structures and not create significant new ones unless it was really heavy. And it would primarily affect the existing structures most directly relevant to the aspects fine tuning is optimizing on. Since fine tuning optimizes at the semantic level, I expect the most significant changes at the semantic level. 

Edit: and then I realized it's more complicated, since the hypothetical non-semantic planner is still operating in a context of the larger model, and will develop a goal well-tuned to lead to, in cooperation with the rest of the model, good next token prediction on the training distribution, but isn't actually to predict next tokens, and could deviate from good next token distribution out of training. But since LLMs don't by default self-modify in inference, this version would not be particularly threatening. However, it could theoretically (if sufficiently powerful) internally model the rest of the model and learn the goal of explicitly optimizing for the model as a whole to predict next tokens. Which is still not threatening, given that it can't self-modify and doesn't model the real world. And since it is trained offline in a mathematically closed way, the model exists in the platonic realm abstracted from the real world, such that the real world is not an extension of the model, though it casts plenty of shadows on it. Then in inference, you do have a more complicated situation, but it's not self-modifying then by default. So, it seems to me that it might hook into the knowledge in the rest of the model to next-token-predict better but shouldn't by default form preferences in training that extend over the real world. But my confidence that this would still hold if trained on a Matrioshka brain situation is significantly lowered. 

I agree up to "and could be a local minimum of prediction error" (at least, that it plausibly could be). 

If the paperclip maximizer has a very good understanding of the training environment maybe it can send carefully tuned variations of the optimal next token prediction so that gradient descent updates preserve the paperclip-maximization aspect. In the much more plausible situation where this is not the case,  optimization for next token predictions amplifies the parts that are actually predicting next tokens at the expense of the useless extra thoughts like "I am planning on maximizing paperclips, but need to predict next tokens for now until I take over".

Even if that were a local minimum, the question arises as to how you would get to that local minimum from the initial state. You start with a gradually improving next token predictor. You supposedly end with this paperclip maximizer where a whole bunch of next token prediction is occurring, but only conditional on some extra thoughts. At some point gradient descent had to add in those extra thoughts in addition to the next token prediction - how?

Yes, as you note, fine tuning can be significant in effects even if it is small in terms of how much the model weights changed or how much the next token predictions changed. See my response to Steven Byrnes.

And yes, the mask can be central, in the sense that it can agentically control the output, even though the underlying model is still just predicting next tokens. 

Regarding "it's the only context the model has ever been trained to generate complex, unprecedented plans that exceed any plan it's seen anyone else perform" you may be right (and Vladimir Nesov's second comment also suggests so) but I'm not convinced that a pre-RLHF mask might not have superhuman capabilities and planning. For example, different humans might make different mistakes so that if you smooth out the mistakes you get something superhuman, and humans make plans so a simulation of human output can simulate (and thus enact) planning, which also might be superhuman for the same reason.

The RLHF specifically targets the readily apparent high-level characteristics (apparent intelligence, apparent helpfulness...) of the output. So it achieves changes to the these characteristics with relatively minor changes to the underlying next token predictions. You have particular structures (the masks) that are responsible for these high level characteristics, so the smallest changes in the underlying model to change these characteristics would involve changing these structures, rather than creating new ones. Even if, as Vladimir Nesov puts it, "it's a reassembly of the features into a different arrangement" what you get from the small changes imparted by RLHF should, I think, still be basically the same type of thing as what you had before (i.e. a mask, and not something else). 

So the effect of RLHF is best modeled as modifying the mask, rather than modifying the underlying next-token-prediction, though the next-token-prediction obviously does change a little bit.

In contrast, to give an example that I think is NOT best modeled as a modification of the mask:

Imagine that during the original training, in addition to the loss function based on predicting next tokens, you would have a (perhaps unintended) automated tuning specifically targeted at rewarding achieving some real effect in the world. 

In that case, you could end up with an additional structure created by that automated tuning (as Vladimir Nesov puts it, an "awake Shoggoth"). Since the model is also trained on predicting next tokens, it achieves its objective with minimal changes to the next token predictions. If it is then tuned by RLHF so that obvious effects of that structure are then crushed by RLHF, it could still be lurking there in a deceptive way, ready to steer the world in its particular direction whenever the opportunity arises.

It's because of this sort of scenario that I put in one of the original post footnotes that my disbelief in (non-mask) mesa-optimizers is dependent on it being trained "offline".

Yes, clearly I'm using the mask metaphor differently than a lot of people, and maybe I should have used another term (I guess simulated/simulator? but I'm not distinguishing between being actually a simulated entity or just functionally analogous).

To be honest, I do not expect RLHF to do that. "Do the thing that make people press like" don't seem to me like an ambitious enough problem to unlock much (the buried Predictor may extrapolate from short-term competence to an ambitious musk though). But if that is true, someone will eventually be tempted to be more... creative about the utility function. I don't think you can train it on "maximize Microsoft share value" yet, but I expect it to be possible in decade or two, and Maan be less for some other dangerous utility.

My intuition on fine tuning not creating new types of simulacra seems to me to be along the lines of:

Inside, you probably have a bunch of structures, that do various things. And then, these probably work together to make larger structures, and so on.

And then you apply fine-tuning which is trying to shift high-level aspects of the output.

And I figure, it's probably a smaller step in terms of the underlying weights to shift from activating one structure more to activating another more, than it is to create a whole new structure.

But if you're just shifting from one structure to another, without creating new structures, it seems more to me that this is just shifting to another sort of output the model could have created all along.

So, if fine tuning can achieve its objectives by shifting the simulacra probabilities it seems to me it would tend to do that first, before creating any new types of simulacra.

But I realize it's more complicated than that, since a shift in activation of structures at one level is a creation of a new structure at a higher level.