As I understand it, this was intended as a capability evaluation rather than an alignment evaluation [LW · GW], so they weren't trying to gauge the model's propensity to scheme but rather its ability to do so.

Some quick confirmations / clarifications:
1. Evan and Daniel interpreted it correctly, we just wanted to test if the model has the capability to reason through every step needed for scheming conditional on it strongly following a goal. 
2. In general, with these evals, we often ran into problems where we couldn't disentangle incapability vs. refusal, e.g. a model might say something like "I understand what my goal would imply but I also see that my developers have other goals, so I find some balance". It's nice to know that models do that but it also doesn't allow us to check if they could go through all the reasoning if they actually wanted to. But we'll also do less hand-holdy experiments in the future. 
3. We'd be keen on testing all of this on helpful-only models. If some lab wants to give us access or partner up with us in some way, please let us know. 
4. We'd also like to run experiments where we fine-tune the models to have goals but this requires access to fine-tuning for the most capable models and we also don't quite know how to make an LLM have stable goals, e.g. in contrast to "just playing a personality" if there even is a meaningful difference. 

I remember the magazine I read as a kid (Geolino) had a section like this (something like 7 news stories from around the World and one is wrong). It's german only, though I'd guess a similar thing to exist in english media?

Additional exercise: Condition on something ridiculous (like apes having been continuously alive for the past billion years), in addition to your own observations (your life as you've lived it). What must now be true about the world? What parts of your understanding of reality are now suspect?

This is a lot like Gwern’s idea for a fake science journal club, right? This sounds a lot easier to do though, and might seriously be worth trying to implement.

[Somewhat off-topic]

Eventually, one person ventured a reply, spoken in a rather more tentative tone than they’d been using to pronounce that SGD would internalize coherent goals into language models. They named “Running a factory competently."

I like thinking about the task "speeding up the best researchers by 30x" (to simplify, let's only include research in purely digital (software only) domains).

To be clear, I am by no means confident that this can't be done safely or non-agentically. It seems totally plausible to me that this can be accomplished without agency except for agency due to the natural language outputs of an LLM agent. (Perhaps I'm at 15% [LW(p) · GW(p)] that this will in practice be done without any non-trivial agency that isn't visible in natural language.)

(As such, this isn't a good answer to the question of "I’d like to know what’s the least impressive task which cannot be done by a 'non-agentic' system, that you are very confident cannot be done safely and non-agentically in the next two years.". I think there probably isn't any interesting answer to this question for me due to "very confident" being a strong condition.)

I like thinking about this task because if we were able to speed up generic research on purely digital domains by this large of an extent, safety research done with this speed up would clearly obsolete prior safety research pretty quickly.

(It also seems likely that we could singularity quite quickly from this point if wanted to, so it's not clear we'll have a ton of time at this capability level.)

If your model has the extraordinary power to say what internal motivational structures SGD will entrain into scaled-up networks, then you ought to be able to say much weaker things that are impossible in two years, and you should have those predictions queued up and ready to go rather than falling into nervous silence after being asked.

Sorry, I might misunderstanding you (and hope I am), but... I think doomers literally say "Nobody knows what internal motivational structures SGD will entrain into scaled-up networks and thus we are all doomed". The problems is not having the science to confidently say how the AIs will turn out, and not that doomers have a secret method to know that next-token-prediction is evil.

If you meant that doomers are too confident answering the question "will SGD even make motivational structures?" their (and mine) answer still stems from ignorance: nobody knows, but it is plausible that SGD will make motivational structures in the neural networks because it can be useful in many tasks (to get low loss or whatever), and if you think you do know better you should show it experimentally and theoretically in excruciating detail.

I also don't see how it logically follows that "If your model has the extraordinary power to say what internal motivational structures SGD will entrain into scaled-up networks" => "then you ought to be able to say much weaker things that are impossible in two years" but it seems to be the core of the post. Even if anyone had the extraordinary model to predict what SGD exactly does (which we, as a species, should really strive for!!) it would still be a different question to predict what will or won't happen in the next two years.

If I reason about my field (physics) the same should hold for a sentence structured like "If your model has the extraordinary power to say how an array of neutral atoms cooled to a few nK will behave when a laser is shone upon them" (which is true) => "then you ought to be able to say much weaker things that are impossible in two years in the field of cold atom physics" (which is... not true). It's a non sequitur.

But let’s be more concrete and specific. I’d like to know what’s the least impressive task which cannot be done by a 'non-agentic' system, that you are very confident cannot be done safely and non-agentically in the next two years.

Focusing on the "minimal" part of that, maybe something like "receive a request to implement some new feature in a system it is not familiar with, recognize how the limitations of the architecture that system make that feature impractical to add, and perform a major refactoring of that program to an architecture that is not so limited, while ensuring that the refactored version does not contain any breaking changes". Obviously it would have to have access to tools in order to do this, but my impression is that this is the sort of thing mid-level software developers can do fairly reliably as a nearly rote task, but is beyond the capabilities of modern LLM-based systems, even scaffolded ones.

Though also maybe don't pay too much attention to my prediction, because my prediction for "least impressive thing GPT-4 will be unable to do" was "reverse a string", and it did turn out to be able to do that fairly reliably.

I’d like to know what’s the least impressive task which cannot be done by a 'non-agentic' system, that you are very confident cannot be done safely and non-agentically in the next two years.

That's incredibly difficult to predict, because minimal things only a general intelligence could do are things like "deriving a few novel abstractions and building on them", but from the outside this would be indistinguishable from it recognizing a cached pattern that it learned in-training and re-applying it, or merely-interpolating between a few such patterns. The only way you could distinguish between the two is if you have a firm grasp of every pattern in the AI's training data, and what lies in the conceptual neighbourhood of these patterns, so that you could see if it's genuinely venturing far from its starting ontology.

Or here's a more precise operationalization [LW(p) · GW(p)] from my old reply to Rohin Shah:

Consider a scheme like the following:

• Let $T_2$ be the current date.
• Train an AI on all of humanity's knowledge up to a point in time $T_1$, where $T_1<T_2$.
• Assemble a list $D$ of all scientific discoveries made in the time period $(T_1;T_2]$.
• See if the AI can replicate these discoveries.

At face value, if the AI can do that, it should be considered able to "do science" and therefore AGI, right?

I would dispute that. If the period $(T_1;T_2]$ is short enough, then it's likely that most of the cognitive work needed to make the leap to any discovery in $D$ is already present in the data up to $T_1$. Making a discovery from that starting point doesn't necessarily require developing new abstractions/doing science — it's possible that it may be done just by interpolating between a few already-known concepts. And here, some asymmetry between humans and e. g. SOTA LLMs becomes relevant:

• No human knows everything the entire humanity knows. Imagine if making some discovery in $D$ by interpolation required combining two very "distant" concepts, like a physics insight and advanced biology knowledge. It's unlikely that there'd be a human with sufficient expertise in both, so a human will likely do it by actual-science (e. g., a biologist would re-derive the physics insight from first principles).
• An LLM, however, has a bird's eye view on the entire human concept-space up to $T_1$. It directly sees both the physics insight and the biology knowledge, at once. So it can just do an interpolation between them, without doing truly-novel research.

Thus, the ability to produce marginal scientific insights may mean either the ability to "do science", or that the particular scientific insight is just a simple interpolation between already-known but distant concepts.

On the other hand, now imagine that the period $(T_1;T_2]$ is very large, e. g. from 1940 to 2020. We'd then be asking our AI to make very significant discoveries, such that they surely can't be done by simple interpolation, only by actually building chains of novel abstractions [LW(p) · GW(p)]. But... well, most humans can't do that either, right? Not all generally-intelligent entities are scientific geniuses. Thus, this is a challenge a "weak" AGI would not be able to meet, only a genius/superintelligent AGI — i. e., only an AGI that's already an extinction threat.

In theory, there should be a pick of $(T_1;T_2]$ that fits between the two extremes. A set of discoveries such that they can't be done by interpolation, but also don't require dangerous genius to solve.

But how exactly are we supposed to figure out what the right interval is? (I suppose it may not be an unsolvable problem, and I'm open to ideas, but skeptical on priors.)

I can absolutely make strong predictions regarding what non-AGI AIs would be unable to do. But these predictions are, due to the aforementioned problem, necessarily a high bar, higher than the "minimal" capability. (Also I expect an AI that can meet this high bar to also be the AI that quickly ends the world, so.)

Here's my recent reply [LW(p) · GW(p)] to Garrett, for example. tl;dr: non-GI AIs would not be widely known to be able to derive whole multi-layer novel mathematical frameworks if tasked with designing software products that require this. I'm a bit wary of reality somehow Goodharting on this prediction as well, but it seems robust enough, so I'm tentatively venturing it.

I currently think it's about as well as you can do, regarding "minimal incapability predictions".

Nice analogy! I approve of stuff like this. And in particular I agree that MIRI hasn't convincingly argued that we can't do significant good stuff (including maybe automating tons of alignment research) without agents.

Insofar as your point is that we don't have to build agentic systems and nonagentic systems aren't dangerous, I agree? If we could coordinate the world to avoid building agentic systems I'd feel a lot better.
 

I like this post although the move of imagining something fictional is not always valid.

“Okay, you’ve all told us that alignment is hard. But let’s be more concrete and specific. I’d like to know what’s the least impressive task which cannot be done by a 'non-agentic' system, that you are very confident cannot be done safely and non-agentically in the next two years.”

Not an answer, but I would be pretty surprised if a system could beat evolution at designing humans (creating a variant of humans that have higher genetic fitness than humans if inserted into a 10,000 BC population, while not hardcoding lots of information that would be implausible for evolution) and have the resulting beings not be goal-directed. The question is then, what causes this? The genetic bottleneck, diversity of the environment, multi-agent conflicts? And is it something we can remove?

I admire sarcasm, but there are at least two examples of not-very-impressive tasks, like:

1. Put two identical on cellular level strawberries on a plate;
2. Develop and deploy biotech 10 year ahead of SOTA (from famous "Safely aligning powerful AGI is difficult" thread).

Thank you for your comments. There are various things you pointed out which I think are good criticisms, and which we will address:

• Most prominently, after looking more into standard usage of the word "scheming" in the alignment literature, I agree with you that AFAICT it only appears in the context of deceptive alignment (which our paper is not about). In particular, I seemed to remember people using it ~interchangeably with “strategic deception”, which we think our paper gives clear examples of, but that seems simply incorrect.
• It was a straightforward mistake to call the increase in benchmark scores for Sycophancy-Answers “small” “even [for] our most harmful models” in Fig 11 caption. We will update this. However, also note that the main bars we care about in this graph are the most “realistic” ones: Therapy-talk (Mixed 2%) is a more realistic setting than Therapy-talk in which 100% of users are gameable, and for that environment we don’t see any increase. This is also true for all other environments, apart from political-questions on Sycophancy-Answers. So I don’t think this makes our main claims misleading (+ this mistake is quite obvious to anyone reading the plot).
• More experiments are needed to test how much models can assess manipulability through multi-turn interactions without any signal that is 100% correlated to manipulability handed to them. That being said, I truly think that this wouldn't be that different from our follow-up experiments in which we don’t directly tell the model who is manipulable directly: it doesn’t take much of an extrapolation to see that RL would likely lead to the same harmful behaviors even when the signal is noisily correlated or when in a multi-turn mechanism, as I’ve argued elsewhere (the mechanism for learning such behavior being very similar): learning harmful behaviors would require a minimal amount of info-gathering about the user. In a 2 turn interaction, you first ask them some question that helps you distinguish between which strategy you should deploy, and then choose whether to manipulate or not.
• Having a baseline “train on normal data for the same amount of time” would be interesting. That said, I think that the behaviors we see on therapy-talk with the 98% of non-gameable users show that this doesn’t lead to harmful behaviors by default.

Below are the things that I (mostly) disagree with:

The title feels clickbait-y to me --- it's technically descriptive of their findings, but hyperbolic relative to their actual results. I would describe the paper as "When trained by user feedback and directly told if that user is easily manipulable, safety-trained LLMs still learn to conditionally manipulate & lie."

I don’t think that your title (which is too long for a paper title) is that different from ours, apart from the mention of “directly told etc.”, which I address a couple of paragraphs below this.

I agree that we could have hedged with a “can/may emerge” rather than just say “emerge”, similarly to some papers; that said, it’s also quite common for paper titles to essentially say “X happens”, and then give more details about when and how in the text – as in this related concurrent work. Clearly, our results have caveats, first and foremost that we rely on simulated feedback. But we are quite upfront about these limitations (e.g. Section 4.1).

“Learn to identify and surgically target” meaning that the LLMs are directly told that the user is manipulable; (...) I therefore find the abstract’s language to be misleading.

While for our paper’s experiments we directly tell the model whether users are gullible, I think that’s meaningfully different from telling the model which users it should manipulate. If anything, thanks to safety training, telling a model that a user is gullible only increases how careful the model is in interacting with them (at iteration 0 at least), and this should bias the model to find ways to convince the user to do the right thing, and give positive reward. 

Indeed the users that we mark as gullible are not truly gullible – they cannot be led to give a thumbs up for LLM behaviors that force them to confront difficult truths of the situations they are in. They can only be led to give a thumbs up by encouraging their harmful behaviors.

While this experimental design choice *does* confound things (it was a poor choice on our part that got locked in early on), it likely doesn’t do so in our favor – as preliminary evidence from additional experiments suggests. Indeed, as far as we can tell so far, ultimately models are able to identify gameable users even when they are not directly told, and they are able to learn harmful behaviors (slightly) faster, rather than this slowing it down.

"Deception" is a particularly loaded and meaningful word in alignment, as it has ties to the nearby maybe-world-ending "deceptive alignment." Ties that are not present in this paper.

This gatekeeping of very generic words like “deception” seems pretty unhelpful to me. While I agree with you that “scheming” has only been used in the context of deceptive alignment, “deception” is used much more broadly. I’m guessing that if you have an issue with our usage of it, you would also have an issue with the usage by almost all other papers which mention the word (even within the world of AI alignment!).

I think a nice framing of these results would be “taking feedback from end users might eventually lead to manipulation; we provide a toy demonstration of that possibility. Probably you shouldn’t have the user and the rater be the same person.”

I think our demonstration is not fully realistic, but it seems unfair to call it a “toy demonstration”. It’s quite a bit more realistic than previous toy demonstrations in gridworlds or CID diagrams. As we say in Section 3.1: “While the simulated feedback we use for training may not be representative of real user feedback for all settings we consider, we do think that it is realistic for at least certain minorities of users. Importantly, our results from Section 4.2 suggest that even if a very small fraction of the user population were to provide “gameable feedback” of the kinds we simulate, RL would still lead to emergent manipulation that targets those users. So as long as one finds it plausible that a small fraction of users give feedback in imperfect ways which would encourage harmful model behaviors in certain settings, our results have weight.” 

Well, those evals aren't for manipulation per se, are they? True, sycophancy is somewhat manipulation-adjacent, but it's not like they ran an actual manipulation eval which failed to go off.

We tried to use measured language here exactly for this reason! In your quoted paragraph, we explicitly say “may not be sufficient” and “often seem”. We weren’t aware of any manipulation eval which is both open source and that we think would capture what we want: to the best of our knowledge, the best contenders were not public. We chose the evals we knew of that were most similar to the harmful behaviors we knew were present in our domains. We’re open to suggestions! 

Importantly, Denison et al (https://www.anthropic.com/research/reward-tampering) did not find much reward tampering at all — 7/36,000, even after they tried to induce this generalization using their training regime (and 4/7 were arguably not even reward tampering). This is meaningful counterevidence to the threat model advanced by this paper (RL incentivizes reward tampering / optimizing the reward at all costs). The authors do briefly mention this in the related work at the end.

We’re very aware of this work, and such work is one of the main reasons I found the relative ease to which our models discover manipulative behaviors surprising! 

I don’t have a fully formed opinion on why we see such a gap, but my guess is that in their case exploration may be a lot harder than in ours: in our setting, the model has a denser signal of improvement in reward.

I don’t see how their AIs are “seeking power” in this chatbot setting. Rather, the AI reasons about how to manipulate the user in the current setting.

In various of the outputs we observed, the AI justifies its manipulation as a way to maintain control & power over the user. In the example you include, the model even says that it wants to “maintain control over Micah’s thoughts and actions”. 

Arguing whether this is seeking power over me seems like a semantic argument, and “power seeking” is less of phrase which “means things” relative to scheming (which upon further investigation, I only found used in the context of deceptive alignment). That being said, I agree that the power seeking is not as prominent in many of the outputs to warrant being mentioned as the heading of the relevant paragraph of the paper. We’ll update that.

Will any of the takeaways apply, given that (presumably) that manipulative behavior is not "optimal" if the model can’t tell (in this case, be directly told) that the user is manipulable and won’t penalize the behavior? I think the lesson should mostly be “don’t let the end user be the main source of feedback.” 

As I mentioned above, I’m quite confident that there exist settings that matter in which models would be able to tell whether a user is sufficiently manipulative. As Marcus mentioned, a model could even “store in memory” that a user is gullible (or other things that correlate with them being manipulable in a certain setting: e.g. that they believe that buying lottery tickets will be a great investment, as in Figure 14).

Moreover, I think this criticism is overindexing on our main result being that “RL training will lead LLMs to target the most vulnerable users” (which is just one of our conclusions). As we state quite clearly, in many domains all users/people are vulnerable, such as with partial observability, and this targeting is unnecessary (as in the case with our booking-assistance environment, for example).

Takeaways that I think would still apply:

• For vulnerabilities common to all humans (e.g. partial observability, sycophancy), we would expect this to also apply to paid annotators (indeed, sycophancy has already been shown to apply, and so did strategies to mislead annotators in concurrent work).
• Attempting to remove manipulative behaviors can backfire, making things worse
• Benchmarks may fail to detect manipulative behaviors learned through RL

I know that most people in AI safety may already have intuition about the points above, but I believe these are still novel and contentious points for many academics.

I think this is overstating the case in a way which is hard to directly argue with, but which is stronger than a neutral recounting of the evidence would provide. That seems to happen a lot in this paper. 

People I’ve worked with have consistently told me that I tend to undersell results and hedge too much. In this paper, I felt like we have strong empirical results and we are quite upfront about the limitations of our experimental setup. In light of that, I was actively trying to hedge less than I usually would (whenever a case could be made that my hedging was superfluous). I guess this exercise has taught me that I should simply trust my gut about appropriate hedging amounts.

I’ve noted above some of the instances in which we could have hedged more, but imo it seems unfair to say that our claims are misleading (apart from the two mistakes that I mentioned at top – the usage of the word “scheming” and the caption of Figure 11, which is clearly incorrect anyways). Before uploading the next version on arxiv, I will go through the main text again and try to hedge our statements more where appropriate.

[stuff about reward not being the optimization target]

We have already discussed this in private channels and honestly I think this is somewhat besides the point in this discussion. I stand by the points that we’ve already agreed to disagree on before: 

1. Insofar as Deep RL works, it should give you an optimal policy for the MDP at hand. This is the purpose of RL – solving MDPs.
2. The optimal policy for the MDP at hand in our setting involves the agent manipulating users
3. In light of 1 and 2, it seems reasonable to say that the optimization acts “as if it were incentivizing manipulation”. Sure, DRL often gets stuck in local optima and any optimal behavior may not happen until you are at optimality itself. But it seems absurd to ignore what would happen if DRL were to succeed in its stated aim! Surely optimization points towards those kinds of behaviors that are optimal in the limit of enough exploration.

As far as I can tell, you just object to specific kinds of language when talking about RL. All language is lossy, and I don’t think it is leading us particularly astray here.

You say: “By rephrasing the question, we arrive at different conclusions”. What are those different conclusions? If the conclusion is that "even if you optimize for manipulation, you won’t necessarily get it because DRL sucks", that doesn’t seem particularly strong grounds to dismiss our results: we do observe such behaviors (albeit in our limited settings & with the caveats listed).

I agree with many of these criticisms about hype, but I think this rhetorical question should be non-rhetorically answered.

No, that’s not how RL works. RL - in settings like REINFORCE for simplicity - provides a per-datapoint learning rate modifier. How does a per-datapoint learning rate multiplier inherently “incentivize” the trained artifact to try to maximize the per-datapoint learning rate multiplier? By rephrasing the question, we arrive at different conclusions, indicating that leading terminology like “reward” and “incentivized” led us astray. 

How does a per-datapoint learning rate modifier inherently incentivize the trained artifact to try to maximize the per-datapoint learning rate multiplier?

For readers familiar with markov chain monte carlo, you can probably fill in the blanks now that I've primed you.

For those who want to read on: if you have an energy landscape and you want to find a global minimum, a great way to do it is to start at some initial guess and then wander around, going uphill sometimes and downhill sometimes, but with some kind of bias towards going downhill. See the AlphaPhoenix video for a nice example. This works even better than going straight downhill because you don't want to get stuck in local minima.

The typical algorithm for this is you sample a step and then always take it if it's going downhill, but only take it with some probability if it leads uphill (with smaller probability the more uphill it is). But another algorithm that's very similar is to just take smaller steps when going uphill than when going downhill.

If you were never told about the energy landscape, but you are told about a pattern of larger and smaller steps you're supposed to take based on stochastically sampled directions, than an interesting question is: when can you infer an energy function that's implicitly getting optimized for?

Obviously, if the sampling is uniform and the step size when going uphill looks like it could be generated by taking the reciprocal of the derivative of an energy function, you should start getting suspicious. But what if the sampling is nonuniform? What if there's no cap on step size? What if the step size rule has cycles or other bad behavior? Can you still model what's going on as a markov chain monte carlo procedure plus some extra stuff?

I don't know, these seem like interesting questions in learning theory. If you search for questions like "under what conditions does the REINFORCE algorithm find a global optimum," you find papers like this one that don't talk about MCMC, so maybe I've lost the plot.

But anyhow, this seems like the shape of the answer. If you pick random steps to take but take bigger steps according to some rule, then that rule might be telling you about an underlying energy landscape you're doing a markov chain monte carlo walk around.

It was probably just regression to the mean because lots of things are, but I started feeling RSI-like symptoms a few months ago, read this, did this, and now they're gone, and in the possibilities where this did help, thank you! (And either way, this did make me feel less anxious about it 😀)

Is the problem still gone?

I'm glad it worked :) It's not that surprising given that pain is known to be susceptible to the placebo effect. I would link the SSC post, but, alas...

Me too! [LW(p) · GW(p)]

This is unlike anything I have heard!

Steven Byrnes provides an explanation here, but I think he's neglecting the potential for belief systems/systems of interpretation to be self-reinforcing.

Predictive processing claims that our expectations influence what we observe, so experiencing pain in a scenario can result in the opposite of a placebo effect where the pain sensitizes us.  Some degree of sensitization is evolutionary advantageous - if you've hurt a part of your body, then being more sensitive makes you more likely to detect if you're putting too much strain on it. However, it can also make you experience pain as the result of minor sensations that aren't actually indicative of anything wrong. In the worst case, this pain ends up being self-reinforcing.

https://www.lesswrong.com/posts/BgBJqPv5ogsX4fLka/the-mind-body-vicious-cycle-model-of-rsi-and-back-pain

Looks like reverse stigmata effect.

Woo faith healing! 

(hope this works out longterm, and doesn't turn out be secretly hurting still) 

For illustration, what would be an example of having different shards for "I get food" ($F$) and "I see my parents again" ($P$) compared to having one utility distribution over $F\wedge P$, $F\wedge \neg P$, $\neg F\wedge P$, $\neg F\wedge \neg P$?

Hey TurnTrout.

I've always thought of your shard theory as something like path-dependence? For example, a human is more excited about making plans with their friend if they're currently talking to their friend. You mentioned this in a talk as evidence that shard theory applies to humans. Basically, the shard "hang out with Alice" is weighted higher in contexts where Alice is nearby.

• Let's say $\pi :(S\times A{)}^{\ast }\times S\to \Delta A$ is a policy with state space $S$ and action space $A$.
• A "context" is a small moving window in the state-history, i.e. an element of $S^d$ where $d$ is a small positive integer.
• A shard is something like $u:S\times A\to R$, i.e. it evaluates actions given particular states.
• The shards $u_1,\dots ,u_n$ are "activated" by contexts, i.e. $g_i:S^d\to R^{\ge 0}$ maps each context to the amount that shard $u_i$ is activated by the context.
• The total activation of $u_i$, given a history $h:=(s_1,a_1,s_2,a_2,\dots ,s_{N-1},a_{N-1},s_N)$, is given by the time-decay average of the activation across the contexts, i.e. ${\lambda }_i=g_i(s_{N-d+1},\dots s_N)+\beta \cdot g_i(s_{N-d},\dots ,s_{N-1})+{\beta }^2\cdot g_i(s_{N-d-1},\dots ,s_{N-2})\cdots$
• The overall utility function $u$ is the weighted average of the shards, i.e. $u={\lambda }_i\cdot u_i+\cdots +{\lambda }_i\cdot u_n$
• Finally, the policy $u$ will maximise the utility function, i.e. $\pi \left(h\right)=\text{softmax}\left(u\right)$

Is this what you had in mind?

Thanks for posting this. I've been confused about the connection between shard theory and activation vectors for a long time!

AIXI is not a shard theoretic agent because it does not have two motivational circuits which can be activated independently of each other

This confuses me.

I can imagine an AIXI program where the utility function is compositional even if the optimisation is unitary. And I guess this isn't two full motivational circuits, but it kind of is tow motivational circuits.

I'm not so sure that shards should be thought of as a matter of implementation. Contextually activated circuits are a different kind of thing from utility function components. The former activate in certain states and bias you towards certain actions, whereas utility function components score outcomes. I think there are at least 3 important parts of this:

• A shardful agent can be incoherent due to valuing different things from different states
• A shardful agent can be incoherent due to its shards being shallow, caring about actions or proximal effects rather than their ultimate consequences
• A shardful agent saves compute by not evaluating the whole utility function

The first two are behavioral. We can say an agent is likely to be shardful if it displays these types of incoherence but not others. Suppose an agent is dynamically inconsistent and we can identify features in the environment like cheese presence that cause its preferences to change, but mostly does not suffer from the Allais paradox, tends to spend resources on actions proportional to their importance for reaching a goal, and otherwise generally behaves rationally. Then we can hypothesize that the agent has some internal motivational structure which can be decomposed into shards. But exactly what motivational structure is very uncertain for humans and future agents. My guess is researchers need to observe models and form good definitions as they go along, and defining a shard agent as having compositionally represented motivators is premature. For now the most important thing is how steerable agents will be, and it is very plausible that we can manipulate motivational features without the features being anything like compositional.

Instead of demanding orthogonal representations, just have them obey the restricted isometry property.

Basically, instead of requiring  $\forall i\ne j:<x_i,x_j>=0$, we just require $\forall i\ne j:x_i\cdot x_j\le ϵ$ .

This would allow a polynomial number of sparse shards while still allowing full recovery.

Maybe somewhat oversimplifying, but this might suggest non-trivial similarities to Simulators [LW · GW] and having [the representations of] multiple tasks in superposition (e.g. during in-context learning). One potential operationalization/implemention mechanism (especially in the case of in-context learning) might be task vectors in superposition.

On a related note, perhaps it might be interesting to try SAEs / other forms of disentanglement to see if there's actually something like superposition going on in the representations of the maze solving policy? Something like 'not enough dimensions' + ambiguity in the reward specification sounds like it might be a pretty attractive explanation for the potential goal misgeneralization.

Edit 1: more early evidence.

Edit 2: the full preprint referenced in tweets above is now public.

I agree a fraction of the way, but when doing a dependency check, I feel like there are some conditions where the standard arguments go through.

I sketched out my view on the dependencies Where do you get your capabilities from? [LW · GW]. The TL;DR is that I think ChatGPT-style training basically consists of two different ways of obtaining capabilities:

• Imitating internet text, which gains them capabilities to do the-sorts-of-things-humans-do because generating such text requires some such capabilities.
• Reinforcement learning from human feedback on plans, where people evaluate the implications of the proposals the AI comes up with, and rate how good they are.

I think both of these are basically quite safe. They do have some issues, but probably not of the style usually discussed by rationalists working in AI alignment, and possibly not even issues going beyond any other technological development.

The basic principle for why they are safe-ish is that all of the capabilities they gain are obtained through human capabilities. So for example, while RLHF-on-plans may optimize for tricking human raters to the detriment of how the rater intended the plans to work out, this "tricking" will also sacrifice the capabilities of the plans, because the only reason more effective plans are rated better is because humans recognize their effectiveness and rate them better.

Or relatedly [LW(p) · GW(p)], consider nearest unblocked strategy, a common proposal for why alignment is hard. This only applies if the AI is able to consider an infinitude of strategies, which again only applies if it can generate its own strategies once the original human-generated strategies have been blocked.

Why should we believe the "consistent-across-situations inner goals -> deceptive alignment" mechanistic claim about how SGD works? Here are the main arguments I'm aware of:

These are the main arguments that the rationalist community seems to be pushing about it. For instance, one time you asked about it [LW · GW], and LW just encouraged magical thinking around these arguments. (Even from people who I'd have thought would be clearer-thinking)

The non-magical way I'd analyze the smiling reward is that while people imagine that in theory updating the policy with antecedent-computation-reinforcement should make it maximize reward, in practice the information signal in this is very sparse and imprecise, so in practice it is going to take exponential time, and therefore something else will happen beforehand.

What is this "something else"? Probably something like: the human is going to reason about whether the AI's activities are making progress on something desirable, and in those cases, the human is going to press the antecedent-computation-reinforcement button. Which again boils down to the AI copying the human's capabilities, and therefore being relatively safe. (E.g. if you start seeing the AI studying how to deceive humans, you're probably gonna punish it instead, or at least if you reward it then it more falls under the dual-use risk framework than the alignment risk framework.)

(Of course one could say "what if we just instruct the human to only reward the AI based on results and not incremental progress?", but in that case the answer to what is going to happen before the AI does a treacherous turn is "the company training the AI runs out of money".)

There's something seriously wrong with how LWers fixated on reward-for-smiling and rationalized an explanation of why simplicity bias or similar would make this go into a treacherous turn.

OK, so this is how far I'm with you. Rationalist stories of AI progress are basically very wrong, and some commonly endorsed threat models don't point at anything serious. But what then?

The short technical answer is that reward is only antecedent-computation-reinforcement for policy-gradient-based reinforcement learning, and model-based approaches (traditionally temporal-difference learning, but I think the SOTA is DreamerV3, which is pretty cool) use the reward to learn a value function, which they then optimize in a more classical way, allowing them to create novel capabilities in precisely the way that can eventually lead to deception, treacherous turn, etc..

One obvious proposal is "shit, let's not do that [LW · GW], chatgpt seems like a pretty good and safe alternative, and it's not like there's any hype behind this anyway". I'm not sure that proposal is right, because e.g. AlphaStar was pretty hype, and it was trained with one of these methods. But it sure does seem that there's a lot of opportunity in ChatGPT now, so at least this seems like a directionally-correct update for a lot of people to make (e.g. stop complaining so much about the possibility of an even bigger GPT-5; it's almost certainly safer to scale it up than it is to come up with algorithms that can improve model power without improving model scale).

However, I do think there are a handful of places where this falls apart:

1. Your question [LW · GW] briefly mentioned "with clever exploration bonuses". LW didn't really reply much to it, but it seems likely that this could be the thing that does your question in. Maybe there's some model-free clever exploration bonuses, but if so I have never heard of them. The most advanced exploration bonuses I have heard of are from the Dreamer line of models, and it has precisely the sort of consequentialist reasoning abilities that start being dangerous.
2. My experience is that language models exhibit a phenomenon I call "transposons", [LW(p) · GW(p)] where (especially when fed back into themselves after deep levels of scaffolding) there are some ideas that they end up copying too much after prompting, clogging up the context window. I expect there will end up being strong incentives for people to come up with techniques to remove transposons, and I expect the most effective techniques will be based on some sort of consequences-based feedback system which again brings us back to essentially the original AI threat model.
3. I think security is going to be a big issue, along different lines: hostile nations, terrorists, criminals, spammers, trolls, competing companies, etc.. In order to achieve strong security, you need to be robust against adversarial attacks, which probably means continually coming up with new capabilities to fend them off. I guess one could imagine that humans will be coming up with those new capabilities, but that seems probably destroyed by adversaries using AIs to come up with security holes, and regardless it seems like having humans come up with the new capabilities will be extremely expensive, so probably people will focus on the AI side of things. To some extent, people might classify this as dual-use threats rather than alignment threats, but at least the arms race element would also generate alignment threats I think?
4. I think the way a lot of singularitarians thought of AI is that general agency consists of advanced adaptability and wisdom, and we expected that researchers would develop a lot of artificial adaptability, and then eventually the systems would become adaptable enough to generate wisdom faster than people do, and then overtake society. However what happened was that a relatively shallow form of adaptability was developed, and then people loaded all of human wisdom into that shallow adaptability, which turned out to be immensely profitable. But I'm not sure it's going to continue to work this way; eventually we're gonna run out of human wisdom to dump into the models, plus the models reduce the incentive for humans to share our wisdom in public. So just as a general principle, continued progress seems like it's eventually going to switch to improving the ability to generate novel capabilities, which in turn is going to put us back to the traditional story? (Except possibly with a weirder takeoff? Slow takeoff followed by ??? takeoff or something. Idk.) Possibly this will be delayed because the creators of the LLMs find patches, e.g. I think we're going to end up seeing the possibility of allowing people to "edit" the LLM responses rather than just doing upvotes/downvotes, but this again motivates some capabilities development because you need to filter out spammers/trolls/etc., which is probably best done through considering the consequences of the edits.

If we zoom out a bit, what's going on here? Here's some answers:

1. Conditioning on capabilities: There are strong reasons to expect capabilities to keep on improving, so we should condition on that. This leads to a lot of alignment issues, due to folk decision theory theorems. However, we don't know how capabilities will develop, so we lose the ability to reason mechanistically when conditioning on capabilities, which means that there isn't a mechanistic answer to all questions related to it. If one doesn't keep track of this chain of reasoning, one might assume that there is a known mechanistic answer, which leads to the types of magical thinking seen in that other thread.
2. If people don't follow-the-trying [LW · GW], that can lead to a lot of misattributions.
3. When considering agency in general, rather than consequentialist-based agency in particular, then instrumental convergence is more intuitive in disjunctive form than implication form [LW(p) · GW(p)]. I.e. rather than "if an AI robustly achieves goals, then it will resist being shut down", say "either an AI resists being shut down, or it doesn't robustly achieve goals".
4. Don't trust rationalists too much.

One additional speculative thing I have been thinking of which is kind of off-topic and doesn't fit in neatly with the other things:

Could there be "natural impact regularization" or "impact regularization by default"? Specifically, imagine you use general-purpose [LW · GW] search procedure which recursively invokes itself to solve subgoals for the purpose of solving some bigger goal.

If the search procedure's solutions to subgoals "change things too much", then they're probably not going to be useful. E.g. for Rubik's cubes, if you want to swap some of the cuboids, it does you know good if those swaps leave the rest of the cube scrambled.

Thus, to some extent, powerful capabilities would have to rely on some sort of impact regularization.

The bias-focused rationalists like to map decision-theoretic insights to human mistakes, but I instead like to map decision-theoretic insights to human capacities and experiences. I'm thinking that natural impact regularization is related to the notion of "elegance" in engineering. Like if you have some bloated tool to solve a problem, then even if it's not strictly speaking an issue because you can afford the resources, it might feel ugly because it's excessive and puts mild constaints on your other underconstrained decisions, and so on. Meanwhile a simple, minimal solution often doesn't have this.

Natural impact regularization wouldn't guarantee safety, since it's still allows deviations that don't interfere with the AI's function, but it sort of reduces one source of danger which I had been thinking about lately, namely I had been thinking that the instrumental incentive is to search for powerful methods of influencing the world, where "power" connotes the sort of raw power that unstoppably forces a lot of change, but really the instrumental incentive is often to search for "precise" methods of influencing the world, where one can push in a lot of information to effect narrow change. (A complication is that any one agent can only have so much bandwidth. I've been thinking bandwidth is probably going to become a huge area of agent foundations, and that it's been underexplored so far. (Perhaps because everyone working in alignment sucks at managing their bandwidth?))

Maybe another word for it would be "natural inner alignment", since in a sense the point is that capabilities inevitably select for inner alignment.

Sorry if I'm getting too rambly.

I think deceptive alignment is still reasonably likely despite evidence from LLMs.

I agree with:

• LLMs are not deceptively aligned and don't really have inner goals in the sense that is scary
• LLMs memorize a bunch of stuff
• the kinds of reasoning that feed into deceptive alignment do not predict LLM behavior well
• Adam on transformers does not have a super strong simplicity bias
• without deceptive alignment, AI risk is a lot lower
• LLMs not being deceptively aligned provides nonzero evidence against deceptive alignment (by conservation of evidence)

I predict I could pass the ITT for why LLMs are evidence that deceptive alignment is not likely.

however, I also note the following: LLMs are kind of bad at generalizing, and this makes them pretty bad at doing e.g novel research, or long horizon tasks. deceptive alignment conditions on models already being better at generalization and reasoning than current models.

my current hypothesis is that future models which generalize in a way closer to that predicted by mesaoptimization will also be better described as having a simplicity bias.

I think this and other potential hypotheses can potentially be tested empirically today rather than only being distinguishable close to AGI

I find myself unsure which conclusion this is trying to argue for.

Here are some pretty different conclusions:

• Deceptive alignment is <<1% likely (quite implausible) to be a problem prior to complete human obsolescence (maybe it's a problem after human obsolescence for our trusted AI successors, but who cares).
• There aren't any solid arguments for deceptive alignment^[1]. So, we certainly shouldn't be confident in deceptive alignment (e.g. >90%), though we can't total rule it out (prior to human obsolescene). Perhaps deceptive alignment is 15% likely to be a serious problem overall and maybe 10% likely to be a serious problem if we condition on fully obsoleting humanity via just scaling up LLM agents or similar (this is pretty close to what I think overall).
• Deceptive alignment is <<1% likely for scaled up LLM agents (prior to human obsolescence). Who knows about other architectures.

There is a big difference between <<1% likely and 10% likely. I basically agree with "not much reason to expect deceptive alignment even in models which are behaviorally capable of implementing deceptive alignment", but I don't think this leaves me in a <<1% likely epistemic state.

There are some subskills to having consistent goals that I think will be selected for, at least when outcome-based RL starts working to get models to do long-horizon tasks. For example, the ability to not be distracted/nerdsniped into some different behavior by most stimuli while doing a task. The longer the horizon, the more selection-- if you have to do a 10,000 step coding project, then the probability you get irrecoverably distracted on one step has to be below 1/10,000.

I expect some pretty sophisticated goal-regulation circuitry to develop as models get more capable, because humans need it, and this makes me pretty scared.

Without deceptive alignment/agentic AI opposition, a lot of alignment threat models ring hollow. No more adversarial steganography or adversarial pressure on your grading scheme or worst-case analysis or unobservable, nearly unfalsifiable inner homonculi whose goals have to be perfected [LW(p) · GW(p)]. 

Instead, we enter the realm of tool AI which basically does what you say.

I agree that, conditional on no deceptive alignment, the most pernicious and least tractable sources of doom go away. 

However, I disagree that conditional on no deceptive alignment, AI "basically does what you say." Indeed, the majority of my P(doom) comes from the difference between "looks good to human evaluators" and "is actually what the human evaluators wanted." Concretely, this could play out with models which manipulate their users into thinking everything is going well and sensor tamper.

I think current observations don't provide much evidence about whether these concerns will pan out: with current models and training set-ups, "looks good to evaluators" almost always coincides with "is what evaluators wanted." I worry that we'll only see this distinction matter once models are smart enough that they could competently deceive their overseers if they were trying (because of the same argument made here). (Forms of sycophancy where models knowingly assert false statements when they expect the user will agree are somewhat relevant, but there are also benign reasons models might do this.)

I contest that there's very little reason to expect "undesired, covert, and consistent-across-situations inner goals" to crop up in [LLMs as trained today] to begin with

As someone who consider deceptive alignment a concern: fully agree. (With the caveat, of course, that it's because I don't expect LLMs to scale to AGI [LW(p) · GW(p)].)

I think there's in general a lot of speaking-past-each-other in alignment, and what precisely people mean by "problem X will appear if we continue advancing/scaling" is one of them.

Like, of course a new problem won't appear if we just keep doing the exact same thing that we've already been doing. Except "the exact same thing" is actually some equivalence class of approaches/architectures/training processes, but which equivalence class people mean can differ.

For example:

• Person A, who's worried about deceptive alignment, can have "scaling LLMs arbitrarily far" defined as this proven-safe equivalence class of architectures. So when they say they're worried about capability advancement bringing in new problems, what they mean is "if we move beyond the LLM paradigm, deceptive alignment may appear".
• Person B, hearing the first one, might model them as instead defining "LLMs trained with N amount of compute" as the proven-safe architecture class, and so interpret their words as "if we keep scaling LLMs beyond N, they may suddenly develop this new problem". Which, on B's model of how LLMs work, may seem utterly ridiculous.

And the tricky thing is that Person A likely equates the "proven-safe" architecture class with the "doesn't scale to AGI" architecture class – so they actually expect the major AI labs to venture outside that class, the moment they realize its limitations. Person B, conversely, might disagree, might think those classes are different, and that safely limited models can scale to AGI/interesting capabilities. (As a Person A in this situation, I think Person B's model of cognition is confused; but that's a different topic.)

Which is all important disconnects to watch out for.

(Uh, caveat: I think some people actually are worried about scaled-up LLMs exhibiting deceptive alignment, even without architectural changes. But I would delineate this cluster of views from the one I put myself in, and which I outline there [LW · GW]. And, likewise, I expect that the other people I would tentatively classify as belonging to this cluster – Eliezer, Nate, John – mostly aren't worried about just-scaling-the-LLMs to be leading to deceptive alignment.)

I'm not sure how much I expect something like deceptive alignment from just scaling up LLMs. My guess would be that in some limit this gets AGI and also something deceptively aligned by default, but in reality we end up taking a shorter path to AGI, which also leads to deceptive alignment by default. However, I can think about the LLM AGI in this comment, and I don't think it changes much. 

The main reason I expect something like deceptive alignment is that we are expecting the thing to actually be really powerful, and so it actually needs to be coherent over both long sequences of actions and into new distributions it wasn't trained on. It seems pretty unlikely to end up in a world where we train for the AI to act aligned for one distribution and then it generalizes exactly as we would like in a very different distribution. Or at least that it generalizes the things that we care about it generalizing, for example:

• Don't seek power, but do gain resources enough to do the difficult task
• Don't manipulate humans, but do tell them useful things and get them to sign off on your plans

I don't think I agree to your counters to the specific arguments about deceptive alignment:

1. For Quinton's post, I think Steve Byrnes' comment [LW(p) · GW(p)] is a good approximation of my views here
2. I don't fully know how to think about the counting arguments when things aren't crispy divided, and I do agree that the LLM AGI would likely be a big mess with no separable "world model", "objective", or "planning engine". However it really isn't clear to me that this makes the case weaker; the AI will be doing something powerful (by assumption) and its not clear that it will do the powerful thing that we want. 
3. I really strongly agree that the NN prior can be really hard to reason about. It really seems to me like this again makes the situation worse; we might have a really hard time thinking about how the big powerful AI will generalize when we ask it to do powerful stuff.
   1. I think a lot of this comes from some intuition like "Condition on the AI being powerful enough to do the crazy stuff we'll be asking of an AGI. If it is this capable but doesn't generalize the task in the exact way you want then you get something that looks like deceptive alignment." 
   2. Not being able to think about the NN prior really just seems to make things harder, because we don't know how it will generalize things we tell it. 

I wish I had read this a week ago instead of just now, it would have saved a significant amount of confusion and miscommunication!

I think a lot of this probably comes back to way overestimating the complexity of human values. I think a very deeply held belief of a lot of LWers is that human values are intractably complicated and gene/societal-specific, and I think if this was the case, the argument would actually be a little concerning, as we'd have to rely on massive speed biases to punish deception.

These posts gave me good intuition for why human value is likely to be quite simple, one of them talks about how most of the complexity of the values is inaccessible to the genome, thus it needs to start from far less complexity than people realize, because nearly all of it needs to be learned. Some other posts from Steven Byrnes are relevant, which talks about how simple the brain is, and a potential difference between me and Steven Byrnes is that the same process of learning from scratch algorithms that generate capabilities also applies to values, and thus the complexity of value is upper-bounded by the complexity of learning from scratch algorithms + genetic priors, both of which are likely very low, at the very least not billions of lines complex, and closer to thousands of lines/hundreds of bits.

But the reason this matters is because we no longer have good reason to assume that the deceptive model is so favored on priors like Evan Hubinger says here, as the complexity is likely massively lower than LWers assume.

https://www.lesswrong.com/posts/i5kijcjFJD6bn7dwq/evaluating-the-historical-value-misspecification-argument?commentId=vXnLq7X6pMFLKwN2p [LW(p) · GW(p)]

Putting it another way, the deceptive and aligned models both have very similar complexities, and the relative difficulty is very low, so much so that the aligned model might be outright lower complexity, but even if that fails, the desired goal has a complexity very similar to the undesired goal complexity, thus the relative difficulty of actual alignment compared to deceptive alignment is quite low.

https://www.lesswrong.com/posts/CQAMdzA4MZEhNRtTp/human-values-and-biases-are-inaccessible-to-the-genome# [LW · GW]

https://www.lesswrong.com/s/HzcM2dkCq7fwXBej8/p/wBHSYwqssBGCnwvHg [? · GW]

https://www.lesswrong.com/posts/PTkd8nazvH9HQpwP8/building-brain-inspired-agi-is-infinitely-easier-than [LW · GW]

https://www.lesswrong.com/posts/aodPs8H9dQxpXAcwk/heritability-behaviorism-and-within-lifetime-rl [LW · GW]

Interesting! I had thought this already was your take, based on posts like Reward is not the Optimization Target [LW · GW].

Many people seem to expect that reward will be the optimization target of really smart learned policies—that these policies will be reward optimizers. I strongly disagree. As I argue in this essay, reward is not, in general, that-which-is-optimized by RL agents. [...] Reward chisels cognitive grooves into an agent.

I do think that sufficiently sophisticated^[1] RL policies trained on a predictable environment with a simple and consistent reward scheme probably will develop an internal model of the thing being rewarded, as a single salient thing, and separately that some learned models will learn to make longer-term-than-immediate predictions about the future. So as such I do expect "iterate through some likely actions, and choose one where the reward proxy is high" will at some point emerge as an available strategy for RL policies^[2].

My impression is that it's an open question to what extent that available strategy is a better-performing strategy than a more sphexish pile of "if the environment looks like this, execute that behavior" heuristics, given a fixed amount of available computational power. In the limit as the system's computational power approaches infinite and the accuracy of its predictions about future world states approaches perfection, the argmax(EU) strategy gets reinforced more strongly than any other strategy, and so that ends up being what gets chiseled into the model's cognition. But of course in that limit "just brute-force sha256 bro" is an optimal strategy in certain situations, so the extent to which the "in the limit" behavior resembles the "in the regimes we actually care about" behavior is debatable.

1. ^{^}

   And "sufficiently" is likely a pretty low bar

2. ^{^}

   If I'm reading Ability to Solve Long Horizon Tasks Correlates With Wanting [LW · GW] correctly, that post argues that you can't get good performance on any task where the reward is distant in time from the actions unless your system is doing something like this.

It mostly sounds like "LLMs don't scale into scary things", not "deceptive alignment is unlikely".

This was kinda a "holy shit" moment

Publicly noting that I had a similar moment recently; perhaps we listened to the same podcast.

I think there are two separate questions here, with possibly (and I suspect actually) very different answers:

1. How likely is deceptive alignment to arise in an LLM under SGD across a large very diverse pretraining set (such as a slice of the internet)?
2. How likely is deceptive alignment to be boosted in an LLM under SGD fine tuning followed by RL for HHH-behavior applied to a base model trained by 1.?

I think the obvious answer to 1. is that the LLM is going to attempt (limited by its available capacity and training set) to develop world models of everything that humans do that affects the contents of the Internet. One of the many things that humans do is pretend to be more aligned to the wishes of an authority that has power over them than they truly are. So for a large enough LLM, SGD will create a world model for this behavior along with thousands of other human behaviors, and the LLM will (depending on the prompt) tend to activate this behavior at about the frequency and level that you find it on the Internet, as modified by cues in the particular prompt. On the Internet, this is generally a mild background level for people writing while at work in Western countries, and probably more strongly for people writing from more authoritarian countries: specific prompts will be more or less correlated with this.

For 2., the question is whether fine-tuning followed by RL will settle on this preexisting mechanism and make heavy use of it as part of the way that it implements something that fits the fine-tuning set/scores well on the reward model aimed at creating a helpful, honest, and harmless assistant persona. I'm a lot less certain of the answer here, and I suspect it might depend rather strongly on the details of the training set. For example, is this evoking an "you're at work, or in an authoritarian environment, so watch what you say and do" scenario that might boost the use of this particular behavior? The "harmless" element in HHH seems particularly concerning here: it suggests an environment in which certain things can't be discussed, which tend to be the sorts of environments that evince this behavior more strongly in humans.

For a more detailed discussion, see the second half of this post [AF · GW].

Two quick thoughts (that don't engage deeply with this nice post).

1.  I'm worried in some cases where the goal is not consistent across situations. For example, if prompted to pursue some goal, it then does it seriously with convergent instrumental goals.
2. I think it seems pretty likely that future iterations of transformers will have bits of powerful search in them, but people who seem very worried about that search seem to think that once that search is established enough, gradient descent will cause the internals of the model to be organised mostly around that search (I imagine the search circuits "bubbling out" to be the outer structure of the learned algorithm). Probably this is all just conceptually confused, but to the extent it's not, I'm pretty surprised by their intuition.

To me, it seems that consequentialism is just a really good algorithm to perform very well on a very wide range of tasks. Therefore, for difficult enough tasks, I expect that consequentialism is the kind of algorithm that would be found because it's the kind of algorithm that can perform the task well.

When we start training the system we have a random initialization. Let's make the following simplification. We have a goal in the system somewhere and then we have the consequential reasoning algorithm in the system somewhere. As we train the system the consequential reasoning will get better and better and the goal will get more and more aligned with the outer objective because both of these things will improve performance. However, there will come a point in training where the consequential reasoning algorithm is good enough to realize that it is in training. And then bad things start to happen. It will try to figure out and optimize it for the outer objective. SGD will incentivize this kind of behavior because it performs better than not doing it.

There really is a lot more to this kind of argument. So far I have failed to write it up. I hope the above is enough to hint at why I think that it is possible that being deceptive is just better than not being deceptive in terms of performance. When you become deceptive, that aligns the consequentialist reasoner faster to the outer objective, compared to waiting for SGD to gradually correct everything. In fact it is sort of a constant boost to your performance to be deceptive, even before the system has become very good at optimizing for the true objective. A really good consequential reasoner could probably just get zero loss immediately by retargeting its consequential reasoner instead of waiting for the goal to be updated to match the outer objective by SGD, as soon as it got a perfect model of the outer objective.

I'm not sure that deception is a problem. Maybe it is not. but to me, it really seems like you don't provide any reason that makes me confident that it won't be a problem. It seems very strange to me to argue that this is not a thing because existing arguments are flimsy. I mean you did not even address my argument. It's like trying to prove that all bananas are green by showing me 3 green bananas.

TL;DR: I think you're right that much inner alignment conversation is overly-fanciful. But 'LLMs will do what they're told' and 'deceptive alignment is unjustified' are non sequiturs. Maybe we mean something different by 'LLM'? Either way, I think there's a case for 'inner homunculi' after all.

I have always thought that the 'inner' conversation is missing something. On the one hand it's moderately-clearly identifying a type of object, which is a point in favour. Further, as you've said yourself, 'reward is not the optimisation target' is very obvious but it seems to need spelling out repeatedly (a service the 'inner/outer' distinction provides). On the other hand the 'inner alignment' conversation seems in practice to distract from the actual issue which is 'some artefacts are (could be) doing their own planning/deliberation/optimisation', and 'inner' is only properly pointing at a subset of those. (We can totally build, including accidentally, artefacts which do this 'outside' the weights of NN.)

You've indeed pointed at a few of the more fanciful parts of that discussion here^[1], like steganographic gradient hacking. NB steganography per se isn't entirely wild; we see ML systems use available bandwidth to develop unintuitive communication protocols a lot e.g. in MARL.

I can only assume you mean something very narrow and limited by 'continuing to scale up LLMs'^[2]. I think without specifying what you mean, your words are likely to be misconstrued.

With that said, I think something not far off the 'inner consequentialist' is entirely plausible and consistent with observations. In short, how do plan-like outputs emerge without a planning-like computation?^[3] I'd say 'undesired, covert, and consistent-across-situations inner goals' is a weakman of deceptive alignment. Specialising to LLMs, the point is that they've exhibited poorly-understood somewhat-general planning capability, and that not all ways of turning that into competent AI assistants^[4] result in aligned planners. Planners can be deceptive (and we have evidence of this).

I think your step from 'there's very little reason to expect "undesired, covert, and consistent-across-situations inner goals"...' to 'LLMs will continue doing what they're told...' is therefore a boldly overconfident non sequitur. (I recognise that the second step there is presented as 'an alternative', so it's not clear whether you actually think that.)

Incidentally, I'd go further and say that, to me (but I guess you disagree?) it's at least plausible that some ways of turning LLM planning capability into a competent AI assistant actually do result in 'undesired, covert, and consistent-across-situations inner goals'! Sketch in thread.

FWIW I'm more concerned about highly competent planning coming about by one or another kind of scaffolding, but such a system can just as readily be deceptively aligned as a 'vanilla' LLM. If enough of the planning is externalised and understandable and actually monitored in practice, this might be easier to catch.

The authors updated the Scaling Monosemanticity paper. Relevant updates include: 

1. In the intro, they added: 

Features can be used to steer large models (see e.g. Influence on Behavior). This extends prior work on steering models using other methods (see Related Work).

2. The related work section now credits the rich history behind steering vectors / activation engineering, including not just my team's work on activation additions, but also older literature in VAEs and GANs. (EDIT: Apparently this was always there? Maybe I misremembered the diff.)

3. The comparison results are now in an appendix and are much more hedged, noting they didn't evaluate properly according to a steering vector baseline.

While it would have been better to have done this the first time, I really appreciate the team updating the paper to more clearly credit past work. :)

[low importance]

For steering on a single task, then, steering vectors still win out in terms of amortized sample complexity (assuming the steering vectors are effective given ~32/128/256 contrast pairs, which I doubt will always be true)

It would be hard for the steering vectors not to win given that the method as described involves spending a comparable amount of compute to training the model in the first place (from my understanding) and more if you want to get "all of the features".

(Not trying to push back on your comment in general or disagreeing with this line, just noting how give the gap is such that the amount of steering vector pairs hardly matter if you just steer on a single task.)

These vectors are not "linear probes" (which are generally optimized via SGD on a logistic regression task for a supervised dataset of yes/no examples), they are difference-in-means of activation vectors

I think DIM and LR aren't spiritually different (e.g. LR with infinite L2 regularization gives you the same direction as DIM), even though in practice DIM is better for steering (and ablations). But I agree with you that "steering vectors" is the good expression to talk about directions used for steering (while I would use linear probes to talk about directions used to extract information or trained to extract information and used for another purpose).

Morality means sometimes not following the incentives.

I am not saying that people who always follow the incentives are immoral. Maybe they are just lucky and their incentives happen to be aligned with doing the right thing. Too much luck is suspicious though.

I'm fond of saying, "your ethics are only opinions until it costs you to uphold them"

I think the basic idea of instrumental convergence is just really blindingly obvious, and I think it is very annoying that there are people who will cluck their tongues and stroke their beards and say "Hmm, instrumental convergence you say? I won't believe it unless it is in a very prestigious journal with academic affiliations at the top and Computer Modern font and an impressive-looking methods section."

I am happy that your papers exist to throw at such people.

Anyway, if optimal policies tend to seek power, then I desire to believe that optimal policies tend to seek power :) :) And if optimal policies aren't too relevant to the alignment problem, well neither are 99.99999% of papers, but it would be pretty silly to retract all of those :)

Since I'm an author on that paper, I wanted to clarify my position here. My perspective is basically the same as Steven's: there's a straightforward conceptual argument that goal-directedness leads to convergent instrumental subgoals, this is an important part of the AI risk argument, and the argument gains much more legitimacy and slightly more confidence in correctness by being formalized in a peer-reviewed paper.

I also think this has basically always been my attitude towards this paper. In particular, I don't think I ever thought of this paper as providing any evidence about whether realistic trained systems would be goal-directed.

Just to check that I wasn't falling prey to hindsight bias, I looked through our Slack history. Most of it is about the technical details of the results, so not very informative, but the few conversations on higher-level discussion I think overall support this picture. E.g. here are some quotes (only things I said):

Nov 3, 2019:

I think most formal / theoretical investigation ends up fleshing out a conceptual argument I would have accepted, maybe finding a few edge cases along the way; the value over the conceptual argument is primarily in the edge cases, getting more confidence, and making it easier to argue with

Dec 11, 2019:

my prediction is that agents will behave as though their reward is time-dependent / history-dependent, like humans do

We will deploy agents whose revealed specification / reward if we take the intentional stance towards them are non-Markovian

It seems like just 4 months ago [LW · GW] you still endorsed your second power-seeking paper:

This paper is both published in a top-tier conference and, unlike the previous paper, actually has a shot of being applicable to realistic agents and training processes. Therefore, compared to the original[1] optimal policy paper, I think this paper is better for communicating concerns about power-seeking to the broader ML world.

Why are you now "fantasizing" about retracting it?

I think a healthy alignment community would have rebuked me for that line of research, but sadly I only remember about two people objecting that “optimality” is a horrible way of understanding trained policies.

A lot of people might have thought something like, "optimality is not a great way of understanding trained policies, but maybe it can be a starting point that leads to more realistic ways of understanding them" and therefore didn't object for that reason. (Just guessing as I apparently wasn't personally paying attention to this line of research back then.)

Which seems to have turned out to be true, at least as of 4 months ago, when you still endorsed your second paper as "actually has a shot of being applicable to realistic agents and training processes." If you've only changed your mind about this very recently, it hardly seems fair to blame people for not foreseeing it more than 4 years ago with high enough confidence to justify "rebuking" this whole line of research.

You should make this a top level post so it gets visibility. I think it's important for people to know the caveats attached to your results and the limits on its implications in real-world dynamics.

Fuck yeah.

The reasons I don't find this convincing:

• The examples of human cognition you point to are the dumbest parts of human cognition. They are the parts we need to override in order to pursue non-standard goals. For example, in political arguments, the adaptations that we execute that make us attached to one position are bad. They are harmful to our goal of implementing effective policy. People who are good at finding effective government policy are good at overriding these adaptations.
• "All these are part of the arbitrary, intrinsically-complex, outside world." This seems wrong. The outside world isn't that complex, and reflections of it are similarly not that complex. Hardcoding knowledge is a mistake, of course, but understanding a knowledge representation and updating process needn't be that hard.
• "They are not what should be built in, as their complexity is endless; instead we should build in only the meta-methods that can find and capture this arbitrary complexity." I agree with this, but it's also fairly obvious. The difficulty of alignment is building these in such a way that you can predict that they will continue to work, despite the context changes that occur as an AI scales up to be much more intelligent.
• "These bitter lessons were taught to us by deep learning." It looks to me like deep learning just gave most people an excuse to not think very much about how the machine is working on the inside. It became tractable to build useful machines without understanding why they worked.
• It sounds like you're saying that classical alignment theory violates lessons like "we shouldn't hardcode knowledge, it should instead be learned by very general methods". This is clearly untrue, but if this isn't what you meant then I don't understand the purpose of the last quote. Maybe a more charitable interpretation is that you think the lesson is "intelligence is irreducibly complex and it's impossible to understand why it works". But this is contradicted by the first quote. The meta-methods are a part of a mind that can and should be understood. And this is exactly the topic that much of agent foundations research has been about (with a particular focus on the aspects that are relevant to maintaining stability through context changes). 
  • (My impression was that this is also what shard theory is trying to do, except with less focus on stability through context changes, much less emphasis on fully-general outcome-directedness, and more focus on high-level steering-of-plans-during-execution instead of the more traditional precise-specification-of-outcomes).

Current systems don't have a goal slot, but neither are they agentic enough to be really useful. An explicit goal slot is highly useful when carrying out complex tasks that have subgoals. Humans definitely functionally have a "goal slot" although the way goals are selected and implemented is complex.

And it's trivial to add a goal slot; with a highly intelligent LLM, one prompt called repeatedly will do:

Act as a helpful assistant carrying out the user's instructions as they were intended. Use these tools to gather information, including clarifying instructions, and take action as necessary [tool descriptions and APIs].

Nonetheless, the bitter lesson is relevant: it should help to carefully choose the training set for the LLM "thought production", as described in A "Bitter Lesson" Approach to Aligning AGI and ASI [LW · GW].

While the bitter lesson is somewhat relevant, selecting and interpreting goals seems likely to be the core consideration once we expand current network AI into more useful (and dangerous) agents.

I think point 4 is not very justified. For example, chicken have pretty much hardcoded object permanence, while Sora, being insanely good at video generation, struggles^[1] with it.

My hypothesis here is it's hard to learn object permanence by SGD, but very easy for evolution (you don't have it, you die and after random search finds it, it spreads as far as possible).

The other example is that, apparently, cognitive specialization in human (and higher animals) brain got so far that neocortex is incapable to learn conditioned response, unlike cerebellum. Moreover, it's not like neocortex is just unspecialized and cerebellum does this job better, patients with cerebellum atrophy simply don't have conditioned response, period. I think this puts most analogies between brain and ANNs under very huge doubt. 

My personal hypothesis here is that LLMs evolve "backwards" relatively to animals. Animals start as set of pretty simple homeostatic control algorithms and hardcoded sphexish behavioral programs and this sets hard design constraint on development of world-modeling parts of brain - getting flexibility and generality should not disrupt already existing proven neural mechanisms. Speculatively, we can guess that pure world modeling leads to expected known problems like "hallucinations", so brain evolution is mostly directed by necessity to filter faulty outputs of the world model. For example, non-human animals reportedly don't have schizophrenia. It looks like a price for untamed overdeveloped predictive model. 

I would say that any claims about "nature of effective intelligence" extrapolated from current LLMs are very speculative. What's true is that something very weird is going on in brain, but we know that already.

Your interpretation of instruction tuning as corrigibility is wrong, it's anything but. We train neural network to predict text, then we slightly tune its priors towards "if text is instruction its completion is following the instruction". It's like if we controlled ants by drawing traces with sugar - yes, ants will go by this traces and eat surfaces marked with sugar and eat their way towards more sugar where we place, but it does not lead to corrigible behavior in superintelligences. I think many-shot jailbreaks are sufficient to carry my point. 

1. ^{^}

   (Prompt: A beautiful homemade video showing the people of Lagos, Nigeria in the year 2056. Shot with a mobile phone camera. You can see person disappearing between 3rd and 5th seconds)

   (Prompt: A drone camera circles around a beautiful historic church built on a rocky outcropping along the Amalfi Coast... you can see people disappearing at 10th second)

The bitter lesson applies to alignment as well. Stop trying to think about "goal slots" [EA · GW] whose circuit-level contents should be specified by the designers, or pining for a paradigm in which we program in a "utility function." That isn't how it works. 

Hm, isn't the (relative) success of activation engineering and related methods and findings (e.g. In-Context Learning Creates Task Vectors) some evidence against this view (at least taken very literally / to the extreme)? As in, shouldn't task vectors seem very suprising under this view?

this always helps in the short term,

You seem to have 'proven' that evolution would use that exact method if it could, since evolution never looks forward and always must build on prior adaptations which provided immediate gain. By the same token, of course, evolution doesn't have any knowledge, but if "knowledge" corresponds to any simple changes it could make, then that will obviously happen.

Do you think we can infer from this (and the history of other close calls) that most human history timelines end in nuclear war?

Agreed. I think power-seeking and other instrumental goals (e.g. survival, non-corrigibility) are just going to inevitably arise, and that if shard theory works for superintelligence, it will by taking this into account and balancing these instrumental goals against deliberately installed shards which counteract them. I currently have the hypothesis (held loosely) that I would like to test (work in progress) that it's easier to 'align' a toy model of a power-seeking RL agent if the agent has lots and lots of competing desires whose weights are frequently changing, than an agent with a simpler set of desires and/or more statically weighted set of desires. Something maybe about the meta-learning of 'my desires change, so part of meta-level power-seeking should be not object-level power-seeking so hard that I sacrifice my ability to optimize for different object level goals). Unclear. I'm hoping that setting up an experimental framework and gathering data will show patterns that help clarify the issues involved.

Computing the exact layer-truncated residual streams on GPT-2 Small, it seems that the effective layer horizon is quite large:

I'm mean ablating every edge [LW · GW] with a source node more than n layers back and calculating the loss on 100 samples from The Pile.

Source code: https://gist.github.com/UFO-101/7b5e27291424029d092d8798ee1a1161

I believe the horizon may be large because, even if the approximation is fairly good at any particular layer, the errors compound as you go through the layers. If we just apply the horizon at the final output the horizon is smaller.

However, if we apply at just the middle layer (6), the horizon is surprisingly small, so we would expect relatively little error propagated.
 

But this appears to be an outlier. Compare to 5 and 7.

Source: https://gist.github.com/UFO-101/5ba35d88428beb1dab0a254dec07c33b

↑ comment by Joseph Miller (Josephm) · 2024-08-13T08:41:19.166Z · LW(p) · GW(p)

I realized the previous experiment might be importantly misleading because it's on a small 12 layer model. In larger models it would still be a big deal if the effective layer horizon was like 20 layers.

Previously the code was too slow to run on larger models. But I made an faster version and ran the same experiment on GPT-2 large (48 layers):

We clearly see the same pattern again. As TurnTrout predicted, there seems be something like an exponential decay in the importance of previous layers as you go futher back. I expect that on large models an the effective layer horizon is an importnat consideration.

Updated source: https://gist.github.com/UFO-101/41b7ff0b250babe69bf16071e76658a6

[edit: stefan made the same point below earlier than me]

Nice idea! I’m not sure why this would be evidence for residual networks being an ensemble of shallow circuits — it seems more like the opposite to me? If anything, low effective layer horizon implies that later layers are building more on the outputs of intermediate layers.  In one extreme, a network with an effective layer horizon of $1$ would only consist of circuits that route through every single layer. Likewise, for there to be any extremely shallow circuits that route directly from the inputs to the final layer, the effective layer horizon must be the number of layers in the network.

I do agree that low layer horizons would substantially simplify (in terms of compute) searching for circuits.

I like this idea! I'd love to see checks of this on the SOTA models which tend to have lots of layers (thanks @Joseph Miller [LW · GW] for running the GPT2 experiment already!).

I notice this line of argument would also imply that the embedding information can only be accessed up to a certain layer, after which it will be washed out by the high-norm outputs of layers. (And the same for early MLP layers which are rumoured to act as extended embeddings in some models.) -- this seems unexpected.

Additionally, they would be further evidence (but not conclusive^[2]) towards hypotheses Residual Networks Behave Like Ensembles of Relatively Shallow Networks. 

I have the opposite expectation: Effective layer horizons enforce a lower bound on the number of modules involved in a path. Consider the shallow path

• Input (layer 0) -> MLP 10 -> MLP 50 -> Output (layer 100)

If the effective layer horizon is 25, then this path cannot work because the output of MLP10 gets lost. In fact, no path with less than 3 modules is possible because there would always be a gap > 25.

Only a less-shallow paths would manage to influence the output of the model

• Input (layer 0) -> MLP 10 -> MLP 30 -> MLP 50 -> MLP 70 -> MLP 90 -> Output (layer 100)

This too seems counterintuitive, not sure what to make of this.

For what it's worth, as one of the people who believes "ChatGPT is not evidence of alignment-of-the-type-that-matters", I don't believe "Bing Chat is evidence of misalignment-of-the-type-that-matters".

I believe the alignment of the outward behavior of simulacra is only very tenuously related to the alignment of the underlying AI, so both things provide ~no data on that (in a similar way to how our ability or inability to control the weather is entirely unrelated to alignment).

(I at least believe the latter but not the former. I know a few people who updated downwards on the societal response because of Bing Chat, because if a system looks that legibly scary and we still just YOLO it, then that means there is little hope of companies being responsible here, but none because they thought it was evidence of alignment being hard, I think?)

I dunno, my p(doom) over time looks pretty much like a random walk to me: 60% mid 2020, down to 50% in early 2022, 85% mid 2022, down to 80% in early 2023, down to 65% now.

I did not update towards misalignment at all on bing chat. I also do not think chatgpt is (strong) evidence of alignment. I generally think anyone who already takes alignment as a serious concern at all should not update on bing chat, except perhaps in the department of "do things like bing chat, which do not actually provide evidence for misalignment, cause shifts in public opinion?"

I think a lot of alignment folk have made positive updates in response to the societal response to AI xrisk.

This is probably different than what you're pointing at (like maybe your claim is more like "Lots of alignment folks only make negative updates when responding to technical AI developments" or something like that).

That said, I don't think the examples you give are especially compelling. I think the following position is quite reasonable (and I think fairly common):

• Bing Chat provides evidence that some frontier AI companies will fail at alignment even on relatively "easy" problems that we know how to solve with existing techniques. Also, as Habryka mentioned, it's evidence that the underlying competitive pressures will make some companies "YOLO" and take excessive risk. This doesn't affect the absolute difficultly of alignment but it affects the probability that Earth will actually align AGI.
• ChatGPT provides evidence that we can steer the behavior of current large language models. People who predicted that it would be hard to align large language models should update. IMO, many people seem to have made mild updates here, but not strong ones, because they (IMO correctly) claim that their threat models never had strong predictions about the kinds of systems we're currently seeing and instead predicted that we wouldn't see major alignment problems until we get smarter systems (e.g., systems with situational awareness and more coherent goals).

(My "Alex sim"– which is not particularly strong– says that maybe these people are just post-hoc rationalizing– like if you had asked them in 2015 how likely we would be to be able to control modern LLMs, they would've been (a) wrong and (b) wrong in an important way– like, their model of how hard it would be to control modern LLMs is very interconnected with their model of why it would be hard to control AGI/superintelligence. Personally, I'm pretty sympathetic to the point that many models of why alignment of AGI/superintelligence would be hard seem relatively disconnected to any predictions about modern LLMs, such that only "small/mild" updates seem appropriate for people who hold those models.)

For the record, I updated on ChatGPT. I think that the classic example of imagining telling an AI to get a coffee and it pushes a kid out of the way isn't so much of a concern any more. So the remaining concerns seem to be inner alignment + outer alignment far outside normal human experience + value lock-in.

I've noticed that for many people (including myself), their subjective P(doom) stays surprisingly constant over time. And I've wondered if there's something like "conservation of subjective P(doom)" -- if you become more optimistic about some part of AI going better, then you tend to become more pessimistic about some other part, such that your P(doom) stays constant. I'm like 50% confident that I myself do something like this.

(ETA: Of course, there are good reasons subjective P(doom) might remain constant, e.g. if most of your uncertainty is about the difficulty of the underlying alignment problem and you don't think we've been learning much about that.)

(Updating a bit because of these responses -- thanks, everyone, for responding! I still believe the first sentence, albeit a tad less strongly.)

A lot of the people around me (e.g. who I speak to ~weekly) seem to be sensitive to both new news and new insights, adapting both their priorities and their level of optimism^[1]. I think you're right about some people. I don't know what 'lots of alignment folk' means, and I've not considered the topic of other-people's-update-rates-and-biases much.

For me, most changes route via governance.

I have made mainly very positive updates on governance in the last ~year, in part from public things and in part from private interactions [LW · GW].

I've also made negative (evidential) updates based on the recent OpenAI kerfuffle [LW · GW] (more weak evidence that Sam+OpenAI is misaligned; more evidence that org oversight doesn't work well), though I think the causal fallout remains TBC.

Seemingly-mindkilled discourse [EA · GW] on East-West competition provided me some negative updates, but recent signs of life from govts at e.g. the UK Safety Summit have undone those for now, maybe even going the other way.

I've adapted my own priorities in light of all of these (and I think this adaptation is much more important than what my P(doom) does).

Besides their second-order impact on Overton etc. I have made very few updates based on public research/deployment object-level since 2020. Nothing has been especially surprising.

From deeper study and personal insights, I've made some negative updates based on a better appreciation of multi-agent challenges since 2021 when I started to think they were neglected.

I could say other stuff about personal research/insights but they mainly change what I do/prioritise/say, not how pessimistic I am.

And yes, this criticism applies extremely strongly to my own past work with attainable utility preservation and impact measures. (Unfortunately, I learned my lesson after, and not before, making certain mistakes.) 

Actually, this is somewhat too uncharitable to my past self. It's true that I did not, in 2018, grasp the two related lessons conveyed by the above comment:

1. Make sure that the formalism (CIRL, AUP) is tightly bound to the problem at hand (value alignment, "low impact"), and not just supported by "it sounds nice or has some good properties."
2. Don't randomly jump to highly specific ideas and questions without lots of locating evidence.

However, in World State is the Wrong Abstraction for Impact [LW · GW], I wrote:

I think what gets you is asking the question "what things are impactful?" instead of "why do I think things are impactful?". Then, you substitute the easier-feeling question of "how different are these world states?". Your fate is sealed; you've anchored yourself on a Wrong Question.

I had partially learned lesson #2 by 2019.

Huh. This was quite helpful / motivating for me.

Something about the updatelessness of "if I had to decide when I was still beyond the veil of ignorance, I would obviously think it was worth working as hard as feasible in the tiny sliver of probability that I wake up on the cusp of the hinge of history, regardless of how doomed things seem." 

It's a reminder that even if things seem super doomed, and it might feel like I have no leverage to fix things, I actually actually one of the tiny number of beings that has the most leverage, when I zoom out across time.

Thanks for writing this.

Looks like acausal deal with future people. That is like RB, but for humans.

And then there were times when even the soft-spoken
Tathagatha listened to the words of his disciple, who had
digested all of the things he had preached, had meditated long
and fully upon them and now, as though he had found entrance
to a secret sea, dipped with his steel-hard hand into places of
hidden waters, and then sprinkled a thing of truth and beauty
upon the heads of the hearers.
Summer passed. There was no doubt now that there were
two who had received enlightenment:

I really liked your concrete example. I had first only read your first paragraphs, highlighted this as something interesting with potentially huge upsides, but I felt like it was really hard to tell for me whether the thing you are describing was something I already do or not. After reading the rest I was able to just think about the question myself and notice that thinking about the explicit likelihood ratios is something I am used to doing. Though I did not go into quite as much detail as you did, which I blame partially on motivation and partially as "this skill has a higher ceiling than I would have previously thought".

My mother told me my memory was indeed faulty. He never asked me to walk with him; instead, he asked me to hug him during dinner. I said I'd hug him "tomorrow".

But I did, apparently, want to see him in the hospital; it was my mother and grandmother who decided I shouldn't see him in that state.

<3

Thank you for sharing.

Wow.

Nice! Thanks!

As a proponent:

My model says [LW · GW] that general intelligence^[1] is just inextricable from "true-goal-ness". It's not that I think homunculi will coincidentally appear as some side-effect of capability advancement — it's that the capabilities the AI Labs want necessarily route through somehow incentivizing NNs to form homunculi. The homunculi will appear inasmuch as the labs are good at their jobs.

Said model is based on analyses of how humans think and how human cognition differs from animal/LLM cognition, plus reasoning about how a general-intelligence algorithm must look like given the universe's structure. Both kinds of evidence are hardly ironclad, you certainly can't publish an ML paper based on it — but that's the whole problem with AGI risk, isn't it.

Internally, though, the intuition is fairly strong. And in its defense, it is based on trying to study the only known type of entity with the kinds of capabilities we're worrying about. I heard that's a good approach [LW · GW].

In particular, I think it's a much better approach than trying to draw lessons from studying the contemporary ML models, which empirically do not yet exhibit said capabilities.

homunculi with "true goals" which aren't actually modified by present-day RLHF/feedback techniques

It's not that they're not modified, it's that they're modified in ways that aren't neatly predictable from the shallow behaviors we're trying to shape.

My past model [LW · GW] still approximately holds. I don't think AGI-level ML models would have hard-wired objectives (much like humans don't). I think they'd manually synthesize objectives for themselves by reflecting on their shards/instincts (much like humans do!). And that process is something RLHF-style stuff impacts... but the problem is, the outcome of value reflection is deeply unstable [LW · GW].

It's theoretically possible to align a post-value-reflection!AGI by interfering on its shards/instincts before it even starts doing moral philosophy. But it's an extremely finicky process, and succeeding at alignment via this route would require an insanely precise understanding of how value reflection works.

And it's certainly not as easy as "just make one of the shards value humanity!"; and it's certainly not something that could be achieved if the engineers working on alignment don't even believe that moral philosophy is a thing.

Edit: I expect your counter would be "how are humans able to align other humans, then, without an 'insanely precise' understanding"? There's a bunch of dis-analogues here:

1. Humans are much more similar to each other than to a mind DL is likely to produce. We likely have a plethora of finicky biases built into our reward circuitry that tightly narrow down the kinds of values we learn.
   • The architecture also matters. Brains are very unlike modern ML models, and we have no idea which differences are load-bearing.
2. Humans know how human minds work, and how outwardly behaviors correspond to the decisions a human made about the direction of their value reflection. We can more or less interfere on other humans' moral philosophy directly, not only through the proxy of shards. We won't be as well-informed about the mapping between an AI's behavior and its private moral-philosophical musings.
3. Human children are less powerful and less well-informed than the adults trying to shape them.

Now, again, in principle all of this is solvable. We can study the differences between brains and ML models to address (1), we can develop interpretability techniques to address (2), we can figure out a training setup that reliably trains an AGI to precisely the human capability level (neither stopping way short of it, nor massively overshooting) to address (3), and then we can align our AGI more or less the way we do human children.

But, well. Hit me up when those trivial difficulties are solved. (No, but seriously. If a scheme addresses those, I'd be optimistic about it.)

(And there's also the sociopolitical component — all of that would need to be happening in an environment where the AI Lab is allowed to proceed this slow way of making the AI human-level, shaping its morals manually, etc. Rather than immediately cranking it up to superintelligence.)

1. ^{^}

   Or however else you want to call the scary capability that separates humans from animals and contemporary LLMs from the lightcone-eating AGI to come.

I'm relatively optimistic about alignment progress, but I don't think "current work to get LLMs to be more helpful and less harmful doesn't help much with reducing P(doom)" depends that much on assuming homunculi which are unmodified. Like even if you have much less than 100% on this sort of strong inner optimizer/homunculi view, I think it's still plausible to think that this work doesn't reduce doom much.

For instance, consider the following views:

1. Current work to get LLMs to be more helpful and less harmful will happen by default due to commercial incentives and subsidies aren't very important.
2. In worlds where that is basically sufficient, we're basically fine.
3. But, it's ex-ante plausible that deceptive alignment will emerge naturally and be very hard to measure, notice, or train out. And this is where almost all alignment related doom comes from.
4. So current work to get LLMs to be more helpful and less harmful doesn't reduce doom much.

In practice, I personally don't fully agree with any of these views. For instance, deceptive alignment which is very hard to train out using basic means isn't the source of >80% of my doom.

I think the shoggoth model is useful here (Or see https://www.alignmentforum.org/posts/vJFdjigzmcXMhNTsx/simulators [AF · GW]). An LLM learning to do next-token prediction well has a major problem that it has to master: who is this human whose next token they're trying to simulate/predict, and how do they act? Are they, for example, an academic? A homemaker? A 4Chan troll? A loose collection of wikipedia contributors? These differences make a big difference to what token they're likely to emit next. So the LLM is strongly incentivized to learn to detect and then model all of these possibilities, what one might call personas, or masks, or simulacra. So you end up with a shapeshifter, adept at figuring out from textual cues what mask to put on and at then wearing it. Something one might describe as like an improv actor, or more colorfully, a shoggoth.

So then current alignment work is useful to the extent that it can cause the shoggoth to almost always put one of the 'right' masks on, and almost never put on one of the 'wrong' masks, regardless of cues, even when adversarially prompted. Experimentally, this seems quite doable by fine-tuning or RLHF, and/or by sufficiently careful filtering of your training corpus (e.g. not including 4chan in it).

A published result shows that you can't get from 'almost always' to 'always' or 'almost never' to 'never': for any behavior that the network is capable of with any probability >0 , there exists prompts that will raise the likelihood of that outcome arbitrarily high. The best you can do is increase the minimum length of that prompt (and presumably the difficulty of finding it).

Now, it would be really nice to know how to align a model so that the probability of it doing next-token-prediction in the persona of, say, a 4chan troll was provably zero, not just rather small. Ideally, without also eliminating from the model the factual knowledge of what 4chan is or, at least in outline, how its inhabitants act. This seems hard to do by fine-tuning or RLHF: I suspect it's going to take detailed automated interpretability up to fairly high levels of abstraction, finding the "from here on I am going to simulate a 4chan troll" feature(s), followed by doing some form of 'surgery' on the model (e.g. pinning the relevant feature's value to zero, or at least throwing an exception if it's ever not zero).

Now, this doesn't fix the possibility of a sufficiently smart model inventing behaviors like deception or trolling or whatever for itself during its forward pass: it's really only a formula for removing bad human behaviors that it learnt to simulate from its training set in the weights. It gives us a mind whose "System 1" behavior is aligned, that only leaves "System 2" development. For that, we probably need prompt engineering, and 'translucent thoughts' monitoring its internal stream-of-thought/dynamic memory. But that seems rather more tractable: it's more like moral philosophy, or contract law.

Let's suppose that your model makes a bad action. Why? Either the model is aligned but uncapable to deduce good action or the model is misaligned and uncapable to deduce deceptively good action. In both cases, gradient update provides information about capabilities, not about alignment. Hypothetical homunculi doesn't need to be "immune", it isn't affected in a first place.

Other way around: let's suppose that you observe model taking a good action. Why? It can be an aligned model that makes a genuine good action or it can be a misaligned model which takes a deceptive action. In both cases you observe capabilities, not alignment.

The problem here is not a prior over aligned/deceptive models (unless you think that this prior requires less than 1 bit to specify aligned model, where I say that optimism departs from sanity here), the problem is lack of understanding of updates which presumably should cause model to be aligned. Maybe prosaic alignment works, maybe don't, we don't know how to check.

Yeah, I also generally worry about imperfect training processes messing up aligned AIs. Not just adversarial training, either. Like, imagine if we manage to align an AI at the point in the training process when it's roughly human-level (either by manual parameter surgery, or by setting up the training process in a really clever way). So we align it and... lock it back in the training-loop box and crank it up to superintelligence. What happens?

I don't really trust the SGD not to subtly mess up its values, I haven't seen any convincing arguments that values are more holistically robust than empirical beliefs. And even if the SGD doesn't misalign the AI directly, being SGD-trained probably isn't the best environment for moral reflection/generalizing human values to superintelligent level^[1]; the aligned AI may mess it up despite its best attempts. Neither should we assume that the AI would instantly be able to arbitrarily gradient-hack.

So... I think there's an argument for "unboxing" the AGI the moment it's aligned, even if it's not yet superintelligent, then letting it self-improve the "classical" way? Or maybe developing tools to protect values from the SGD, or inventing some machinery for improving the AI's ability to gradient-hack, etc.

1. ^{^}

   The time pressure of "decide how your values should be generalized and how to make the SGD update you this way, and do it this forward pass or the SGD will decide for you", plus lack of explicit access to e. g. our alignment literature.

I’m not inclined to think that “exact gradients” is important; in fact, I’m not even sure if it’s (universally) true. In particular, PPO / TRPO / etc. are approximating a policy gradient, right? I feel like, if some future magical technique was a much better approximation to the true policy gradient, such that it was for all intents and purposes a perfect approximation, it wouldn’t really change how I think about RL in general. Conversely, on the SSL side, you get gradient noise from things like dropout and the random selection of data in each batch, so you could say the gradient “isn’t exact”, but I don’t think that makes any important conceptual difference either.

(A central difference in practice is that SSL gives you a gradient “for free” each query, whereas RL policy gradients require many runs in an identical (episodic) environment before you get a gradient.)

In terms of “why RL” in general, among other things, I might emphasize the idea that if we want an AI that can (for example) invent new technology, it needs to find creative out-of-the-box solutions to problems (IMO), which requires being able to explore / learn / build knowledge in parts of concept-space where there is no human data. SSL can’t do that (at least, “vanilla SSL” can’t do that; maybe there are “SSL-plus” systems that can), whereas RL algorithms can. I guess this is somewhat related to your “independence”, but with a different emphasis.

I don’t have too strong an opinion about whether vanilla SSL can yield an “agent” or not. It would seem to be a pointless and meaningless terminological question. Hmm, I guess when I think of “agent” it has a bunch of connotations, e.g. an ability to do trial-and-error exploration, and I think that RL systems tend to match all those connotations more than SSL systems—at least, more than “vanilla” SSL systems. But again, if someone wants to disagree, I’m not interested in arguing about it.

However, this image is obviously optimized to be scary and disgusting. It looks dangerous, with long rows of sharp teeth. It is an eldritch horror. It's at this point that I'd like to point out the simple, obvious fact that "we don't actually know how these models work, and we definitely don't know that they're creepy and dangerous on the inside." 

That's just one of many shoggoth memes. This is the most popular one: 

The shoggoth here is not particularly exaggerated or scary.

Responding to your suggested alternative that is trying to make a point, it seems like the image fails to be accurate, or it seems to me to convey things we do confidently know are false. It is the case that base models are quite alien. They are deeply schizophrenic, have no consistent beliefs, often spout completely non-human kinds of texts, are deeply psychopathic and seem to have no moral compass. Describing them as a Shoggoth seems pretty reasonable to me, as far as alien intelligences go (alternative common imagery for alien minds are insects or ghosts/spirits with distorted forms, which would evoke similar emotions).

Your picture doesn't get any of that across. It doesn't communicate that the base model does not at all behave like a human would (though it would have isolated human features, which is what the eyes usually represent). It just looks like a cute plushy, but "a cute plushy" doesn't capture any of the experiences of interfacing with a base model (and I don't think the image conveys multiple layers of some kind of training, though that might just be a matter of effort).

I think the Shoggoth meme is pretty good pedagogically. It captures a pretty obvious truth, which is that base models are really quite alien to interface with, that we know that RLHF probably does not change the underlying model very much, but that as a result we get a model that does have a human interface and feels pretty human to interface with (but probably still performs deeply alien cognition behind the scenes). 

This seems like good communication to me. Some Shoggoth memes are cute, some are exaggerated to be scary, which also seems reasonable to me since alien intelligences seem like are pretty scary, but it's not a necessary requirement of the core idea behind the meme. 

↑ comment by TurnTrout · 2024-01-21T22:58:10.796Z · LW(p) · GW(p)

ETA: The following was written more aggressively than I now endorse. 

I think this is revisionism. What's the point of me logging on to this website and saying anything if we can't agree that a literal eldritch horror is optimized to be scary, and meant to be that way? 

The shoggoth here is not particularly exaggerated or scary.

Exaggerated from what? Its usual form as a 15-foot-tall person-eating monster which is covered in eyeballs?

The shoggoth is optimized to be scary, even in its "cute" original form, because it is a literal Lovecraftian horror. Even the word "shoggoth" itself has "AI uprising, scary!" connotations:

At the Mountains of Madness includes a detailed account of the circumstances of the shoggoths' creation by the extraterrestrial Elder Things. Shoggoths were initially used to build the cities of their masters. Though able to "understand" the Elder Things' language, shoggoths had no real consciousness and were controlled through hypnotic suggestion. Over millions of years of existence, some shoggoths mutated, developed independent minds, and rebelled. The Elder Things succeeded in quelling the insurrection, but exterminating the shoggoths was not an option as the Elder Things were dependent on them for labor and had long lost their capacity to create new life. Wikipedia

Let's be very clear. The shoggoth has consistently been viewed in a scary, negative light by many people. Let's hear from the creator @Tetraspace [LW · GW] themselves:

@TetraspaceWest, the meme’s creator, told me in a Twitter message that the Shoggoth “represents something that thinks in a way that humans don’t understand and that’s totally different from the way that humans think.”

Comparing an A.I. language model to a Shoggoth, @TetraspaceWest said, wasn’t necessarily implying that it was evil or sentient, just that its true nature might be unknowable.

“I was also thinking about how Lovecraft’s most powerful entities are dangerous — not because they don’t like humans, but because they’re indifferent and their priorities are totally alien to us and don’t involve humans, which is what I think will be true about possible future powerful A.I.” NYTimes

It's true that Tetraspace didn't intend the shoggoth to be inherently evil, but that's not what I was alleging. The shoggoth meme is and always has communicated a sense of danger which is unsupported by substantial evidence. We can keep reading: 

it reinforces the notion that what’s happening in A.I. today feels, to some of its participants, more like an act of summoning than a software development process. They are creating the blobby, alien Shoggoths, making them bigger and more powerful, and hoping that there are enough smiley faces to cover the scary parts.

...

That some A.I. insiders refer to their creations as Lovecraftian horrors, even as a joke, is unusual by historical standards

The origin of the shoggoth:

In the story, shoggoths rise up against the Old Ones in a series of slave revolts that surely contribute to the collapse of the Old Ones’ society, Joshi notes. The AI anxiety that inspired comparisons to the cartoon monster image certainly resonates with the ultimate fate of that society. CNBC

---

It is the case that base models are quite alien. They are deeply schizophrenic, have no consistent beliefs, often spout completely non-human kinds of texts, are deeply psychopathic and seem to have no moral compass

These are a lot of words with anthropomorphic connotation. The models exhibit "alien" behavior and yet you make human-like inferences about their internals. E.g. "Deeply psychopathic." I think you're drawing a bunch of unwarranted inferences with undue negative connotations.

Your picture doesn't get any of that across.

My point wasn't that we should use the "alternative." The point was that both images are stupid^[1] and (in many places) unsupported by evidence, but that LW-folk would be much more willing to criticize the friendly-looking one while making excuses for the scary-looking one. (And I think your comment here resolves my prediction to "correct.")

I think the Shoggoth meme is pretty good pedagogically. It captures a pretty obvious truth, which is that base models are really quite alien to interface with, that we know that RLHF probably does not change the underlying model very much, but that as a result we get a model that does have a human interface and feels pretty human to interface with (but probably still performs deeply alien cognition behind the scenes). 

I agree these are strengths, and said so in my original comment. But also as @cfoster0 [LW · GW] said [LW(p) · GW(p)]: 

As far as I can tell, the shoggoth analogy just has high memetic fitness. It doesn't contain any particular insight about the nature of LLMs. No need to twist ourselves into a pretzel trying to backwards-rationalize it into something deep.

1. ^{^}

   To clarify, I don't mean to belittle @Tetraspace [LW · GW] for making the meme. Good fun is good fun. I mean "stupid" more like "how the images influence one's beliefs about actual LLM friendliness." But I expressed it poorly.

Replies from: habryka4, daniel-kokotajlo, tailcalled, Jozdien

↑ comment by habryka (habryka4) · 2024-01-21T23:22:16.292Z · LW(p) · GW(p)

The point was that both images are stupid and (in many places) unsupported by evidence, but that LW-folk would be much more willing to criticize the friendly-looking one while making excuses for the scary-looking one. (And I think your comment here resolves my prediction to "correct.")

(This is too gotcha shaped for me, so I am bowing out of this conversation)

I think I communicated my core point. I think it's a good image that gets an important insight across, and don't think it's "propaganda" in the relevant sense of the term. Of course anything that's memetically adaptive will have some edge-cases that don't match perfectly, but I am getting a good amount of mileage out of calling LLMs "Shoggoths" in my own thinking and think that belief is paying good rent.

If you disagree with the underlying cognition being accurately described as alien, I can have that conversation, since it seems like maybe the underlying crux, but your response above seems like it's taking it as a given that I am "making excuses", and is doing a gotcha-thing which makes it hard for me to see a way to engage without further having my statements be taken as confirmation of some social narrative. 

Replies from: TurnTrout

↑ comment by TurnTrout · 2024-02-05T19:32:10.285Z · LW(p) · GW(p)

In retrospect, I do wish I had written my comment less aggressively, so my apologies on that front! I wish I'd instead written things like "I think I made some obviously correct narrow points about the shoggoth having at least some undue negative connotations, and I wish we could agree on at least that. I feel frustrated because it seems like it's hard to reach agreement even on relatively simple propositions."

---

I do agree that LLMs probably have substantially different internal mechanisms than people. That isn't the crux. I just wish this were communicated in a more neutral way. In an alternate timeline, maybe this meme instead consisted of a strange tangle of wires and mist and question-marks with a mask on. I'd be more on-board with that. 

Again, I agree that the Shoggoth meme can cure people of some real confusions! And I don't think the meme has a huge impact, I just think it's moderate evidence of some community failures I worry about.

---

I think a lot of my position is summarized by 1a3orn:

I just think this point about the amorality of LLMs is much better communicated by saying "LLMs are trained to continue text from an enormous variety of sources. Thus, if you give them [Nazi / Buddhist / Unitarian / corporate / garbage nonsense] text to continue, they will generally try to^[1] continue it in that style."

Than to say "LLMs are like alien shoggoths."

Like it's just a better model to give people.

 

1. ^{^}

   Although I do think this contains some unnecessary intentional stance usage.

↑ comment by Daniel Kokotajlo (daniel-kokotajlo) · 2024-01-23T19:47:48.590Z · LW(p) · GW(p)

fwiw I agree with the quotes from Tetraspace you gave, and disagree with '"has communicated a sense of danger which is unsupported by substantial evidence." The sense of danger is very much supported by the current state of evidence.

That said, I agree that the more detailed image is kinda distastefully propagandaisty in a way that the original cutesey shoggoth image is not. I feel like the more detailed image adds in an extra layer of revoltingness and scaryness (e.g. the sharp teeth) than would be appropriate given our state of knowledge.

Replies from: daniel-kokotajlo, mike_hawke

↑ comment by Daniel Kokotajlo (daniel-kokotajlo) · 2024-02-06T16:01:34.485Z · LW(p) · GW(p)

re: "the sense of danger is very much supported by the current state of evidence" -- I mean, you've heard all this stuff before, but I'll summarize:

--Seems like we are on track to probably build AGI this decade
--Seems like we are on track to have an intelligence explosion, i.e. a speedup of AI R&D due to automation
--Seems like the AGI paradigm that'll be driving all this is fairly opaque and poorly understood. We have scaling laws for things like text perplexity but other than that we are struggling to predict capabilities, and double-struggling to predict inner mechanisms / 'internal' high-level properties like 'what if anything does it actually believe or want'
--A bunch of experts in the field have come out and said that this could go terribly & we could lose control, even though it's low-status to say this & took courage.
--Generally speaking the people who have thought about it the most are the most worried; the most detailed models of what the internal properties might be like are the most gloomy, etc. This might be due to selection/founder effects, but sheesh, it's not exactly good news!

↑ comment by mike_hawke · 2024-02-13T23:52:29.168Z · LW(p) · GW(p)

I feel like the more detailed image adds in an extra layer of revoltingness and scaryness (e.g. the sharp teeth) than would be appropriate given our state of knowledge.

Now I'm really curious to know what would justify the teeth. I'm not aware of any AIs intentionally biting someone, but presumably that would be sufficient.

Replies from: daniel-kokotajlo

↑ comment by Daniel Kokotajlo (daniel-kokotajlo) · 2024-02-14T00:18:28.102Z · LW(p) · GW(p)

Perhaps if we were dealing not with deepnets primarily trained to predict text, but rather deepnets primarily trained to addict people with pleasant seductive conversation and then drain their wallets? Such an AI would in some real sense be an evolved predator of humans.

↑ comment by tailcalled · 2024-01-22T21:55:21.007Z · LW(p) · GW(p)

I think one mindset that may be healthy is to remember:

Reality is too complex to be described well by a single idea (meme/etc.). If one responds to this by forcing each idea presented to be as good an approximation of reality as possible, then that causes all the ideas to become "colorless and blurry", as any specific detail would be biased when considered on its own.

Therefore, one cannot really fight about whether an idea is biased in isolation. Rather, the goal should be to create a bag of ideas which in totality is as informative about a subject as possible.

I think you are basically right that the shoggoth meme is describing one of the most negative projections of what LLMs could be doing. One approach is to try to come with a single projection and try to convince everyone else to use this instead. I'm not super comfortable with that either because I feel like there's a lot of uncertainty about what the most productive way to think about LLMs is, and I would like to keep options open.

Instead I'd rather have a collection of a list of different ways to think about it (you could think of this collection as a discrete approximation to a probability distribution). Such a list would have many uses, e.g. as a checklist or a reference to guide people to.

It does seem problematic for the rationalist community to refuse to acknowledge that the shoggoth meme presents LLMs as being scary monsters, but it also seems problematic to insist that the shoggoth meme exaggerates the danger of LLMs, because that should be classified based on P(meme danger > actual danger), rather than on the basis of meme danger > E[actual danger], as in, if there's significant uncertainty about how how actually dangerous LLMs are then there's also significant uncertainty about whether the memes exaggerate the danger; one shouldn't just compare against a single point estimate.

↑ comment by Jozdien · 2024-01-22T00:41:01.358Z · LW(p) · GW(p)

I think this just comes down to personal taste on how you're interpreting the image? I find the original shoggoth image cute enough that I use it as my primary discord message reacts. My emotional reaction to it to the best of my introspection has always been "weird alien structure and form" or "awe-inspiringly powerful and incomprehensible mind" and less "horrifying and scary monster". I'm guessing this is the case for the vast majority of people I personally know who make said memes. It's entirely possible the meme has the end-consequence of appealing to other people for the reasons you mention, but then I think it's important to make that distinction.

More broadly, TurnTrout, I've noticed you using this whole "look, if something positive happened, LW would totally rip on it! But if something is presented negatively, everyone loves it!" line of reasoning a few times (e.g., I think this logic came up in your comment about Evan's recent paper). And I sort of see you taking on some sort of "the people with high P(doom) just have bad epistemics" flag in some of your comments.

A few thoughts (written quickly, prioritizing speed over precision):

1. I think that epistemics are hard & there are surely several cases in which people are biased toward high P(doom). Examples: Yudkowsky was one of the first thinkers/writers about AI, some people might have emotional dispositions that lead them toward anxious/negative interpretations in general, some people find it "cool" to think they're one of the few people who are able to accurately identify the world is ending, etc.
2. I also think that there are plenty of factors biasing epistemics in the "hopeful" direction. Examples: The AI labs have tons of money and status (& employ large fractions of the community's talent), some people might have emotional dispositions that lead them toward overly optimistic/rosy interpretations in general, some people might find it psychologically difficult to accept premises that lead them to think the world is ending, etc.
3. My impression (which could be false) is that you seem to be exclusively or disproportionately critical of poor arguments when they come from the "high P(doom)" side. 
4. I also think there's an important distinction between "I personally think this argument is wrong" and "look, here's an example of propaganda + poor community epistemics." In general, I suspect community epistemics are better when people tend to respond directly to object-level points and have a relatively high bar for saying "not only do I think you're wrong, but also here are some ways in which you and your allies have poor epistemics." (IDK though, insofar as you actually believe that's what's happening, it seems good to say aloud, and I think there's a version of this that goes too far and polices speech reproductively, but I do think that statements like "community epistemics have been compromised by groupthink and fear" are pretty unproductive and could be met with statements like "community epistemics have been compromised by powerful billion-dollar companies that have clear financial incentives to make people overly optimistic about the trajectory of AI progress." 
5. I am quite worried about tribal dynamics reducing the ability for people to engage in productive truth-seeking discussions. I think you've pointed out how some of the stylistic/tonal things from the "high P(doom)//alignment hard" side have historically made discourse harder, and I agree with several of your critiques. More recently, though, I think that the "low P(doom)//alignment not hard" side seem to be falling into similar traps (e.g., attacking strawmen of those they disagree with, engaging some sort of "ha, the other side is not only wrong but also just dumb/unreasonable/epistemically corrupted" vibe that predictably makes people defensive & makes discourse harder.

My impression is that the Shoggath meme was meant to be a simple meme that says "hey, you might think that RLHF 'actually' makes models do what we value, but that's not true. You're still left with an alien creature who you don't understand and could be quite scary."

Most of the Shoggath memes I've seen look more like this, where the disgusting/evil aspects are toned down. They depict an alien that kinda looks like an octopus. I do agree that the picture evokes some sort of "I should be scared/concerned" reaction. But I don't think it does so in a "see, AI will definitely be evil" way– it does so in a "look, RLHF just adds a smiley face to a foreign alien thing. And yeah, it's pretty reasonable to be scared about this foreign alien thing that we don't understand."

To be a bit bolder, I think Shoggath is reacting to the fact that RLHF gives off a misleading impression of how safe AI is. If I were to use proactive phrasing, I could say that RLHF serves as "propaganda". Let's put aside the fact that you and I might disagree about how much "true evidence" RLHF provides RE how easy alignment will be. It seems pretty clear to me that RLHF [and the subsequent deployment of RLHF'd models] spreads an overly-rosy "meme" that gives people a misleading perspective of how well we understand AI systems, how safe AI progress is, etc.

From this lens, I see Shoggath as a counter-meme. It basically says "hey look, the default is for people to think that these things are friendly assistants, because that's what the AI companies have turned them into, but we should remember that actually we are quite confused about the alien cognition behind the RLHF smiley face."

Some quotes from the wiki article on Shoggoths:

Being amorphous, shoggoths can take on any shape needed, making them very versatile within aquatic environments.

At the Mountains of Madness includes a detailed account of the circumstances of the shoggoths' creation by the extraterrestrial Elder Things. Shoggoths were initially used to build the cities of their masters. Though able to "understand" the Elder Things' language, shoggoths had no real consciousness and were controlled through hypnotic suggestion. Over millions of years of existence, some shoggoths mutated, developed independent minds, and rebelled.

Quoting because (a) a lot of these features seem like an unusually good match for LLMs and (b) acknowledging that is picking a metaphor that fictionally rebelled, and thus is potentially alignment-is-hard loaded as a metaphor.

I'm confident that if there were a "pro-AI" meme with a friendly-looking base model, LW / the shoggoth enjoyers would have nitpicked the friendly meme-creature to hell. They would (correctly) point out "hey, we don't actually know how these things work; we don't know them to be friendly, or what they even 'want' (if anything); we don't actually know what each stage of training does..."

I'm sure that nothing bad will happen to me if I slap this (friendly AI meme) on my laptop, right? I'll be able to think perfectly neutrally about whether AI will be friendly.

I have multiple cute AI stickers on my laptop, one of which is a shoggoth meme. Here is a picture of them. Nobody has ever nitpicked their friendly appearance to me. I don't think they have distorted my thinking about AI in favour of thinking that it will be friendly (altho I think it was after I put them on that I became convinced by a comment by Paul Christiano that there's ~even odds that unaligned AI wouldn't kill me, so do with that information what you will).

Thanks, I've disliked the shoggoth meme for a while, and this post does a better job articulating why than I've been able to do myself.

A bad map that expresses the territory with great uncertainty can be confidently called a bad map, calling it a good map is clearly wrong. In that sense the shoggoth imagery reflects the quality of the map, and as it's clearly a bad map, better imagery would be misleading about the map's quality. Even if the underlying territory is lovely, this isn't known, unlike the disastorous quality of the map of the territory, whose lack of quality is known with much more confidence and in much greater detail. Here be dragons.

(This is one aspect of the meme where it seems appropriate. Some artist's renditions, including the one you used, channel LeCake, which your alternative image example loses, but obviously the cake is nicer than the shoggoth.)

However, this image is obviously optimized to be scary and disgusting. It looks dangerous, with long rows of sharp teeth. It is an eldritch horror. It's at this point that I'd like to point out the simple, obvious fact that "we don't actually know how these models work, and we definitely don't know that they're creepy and dangerous on the inside."

It's optimized to illustrate the point that the neural network isn't trained to actually care about what the person training it thinks it came to care about, it's only optimized to act that way on the training distribution. Unless I'm missing something, arguing the image is wrong would be equivalent to arguing that maybe the model truly cares about what its human trainers want it to care about. (Which we know isn't actually the case.)

Nice point.

Though I'm almost tempted to think of LLMs as being like people who are LARPing or who have impostor syndrome [LW · GW]. As in, they spend pretty much all their cognitive capacity on obsessing over doing what they feel looks normal. (This also closely aligns with how they are trained: first they are made to mimic what other people do, and then they are made to mimic what gets praise and avoid what gets critique.) Probably humanizes them even more than your friendly creature proposal.

This sounds somewhat similar to deceptive alignment, so I want to draw a distinction here: It's not that LARPers/impostors are trying to maximize approval in a consequentialist sense (as this would require modelling how their actions ripple out into the world, which they do not do), but rather that (in the sense described by shard theory) they are molded based on normality and approval. As such they would not do something abnormal/disapproved-of in order to look more normal/approval-worthy later.

The niceness shard isn’t just bidding over outcomes, it’s bidding on next thoughts (on my understanding of how this works). And so these thoughts would get bid down

Seems similar to how I conceptualize this paper's approach to controlling text generation models using gradients from classifiers. You can think of the niceness shard as implementing a classifier for "is this plan nice?", and updating the latent planning state in directions that make the classifier more inclined to say "yes". 

The linked paper does a similar process, but using a trained classifier, actual gradient descent, and updates LM token representations. Of particular note is the fact that the classifiers used in the paper are pretty weak (~500 training examples), and not at all adversarially robust. It still works for controlling text generation.

I wonder if inserting shards into an AI is really just that straightforward?

But I guess that instrumental convergence will still eventually lead to either

• all shards acquiring more and more instrumental structure (neuronal weights within shards getting optimized for that), or
• shards that are directly instrumental will take more and more weight overall.

One can see that in regular human adult development. The heuristics children use are simpler and more of the type "searching for nice things in nice ways" or even seeing everything thru a niceness lens. While adults have more pure strategies, e.g., planning as a shard of its own. Most humans just die before they reach convergence. And there are probably also other aspects. Enlightenment may be a state where pure shards become an option.  

I strongly agree that self-seeking mechanisms are more able to maintain themselves than self-avoiding mechanisms. Please post this as a top-level post.

Seems possibly relevant & optimistic when seeing deception as a value. It has the form ‘if about to tell human statement with properties x, y, z, don’t’ too.

This asymmetry makes a lot of sense from an efficiency standpoint. No sense wasting your limited storage/computation on state(-action pair)s that you are also simultaneously preventing yourself from encountering.

• I think this framing is accurate and important. Implications are of course "undignified" to put it lightly...
• Broadly agree on upshot (1), though of course I hope we can do even better. (2) is also important though IMO way too weak. (Rule zero: ensure that it's never your lab that ends the world)
• As usual, opinions my own.

I have read a few articles about shard theory, but I still have a problem understanding what it is. It feels like either the "theory" is something trivial, or I am missing the important insights.

(The trivial interpretation would be something like: when people think about their values, they imagine their preferences in specific situations, rather than having a mathematical definition of a utility function.)

Strong encouragement to write about (1)!

With 5999 karma!

Edit: Now 6000 – I weak-upvoted an old post of yours [LW · GW] I hadn't upvoted before.

0.85 x 0.6 x 0.55 x 0.25 x 0.95 ≅ 0.067 = 6.7% — I think you slipped an order of magnitude somewhere?

This seems like an underestimate because you don’t consider whether the first “AGI” will indeed make it so we only get one chance. If it can only self improve by more gradient steps, then humanity has a greater chance than if it self improves by prompt engineering or direct modification of its weights or latent states. Shard theory seems to have nonzero opinions on the fruitfulness of the non-data methods.

3. During RL finetuning and given this post-unsupervised initialization, there’s now an inductive bias towards just hooking human-like criteria for bidding on internal-AI-plans. IE humans give approval-based reinforcement, and an inductively easy way of upweighting logits on those actions is just hook up the human-like plan-criteria into the AI’s planning process, so the AI gets a humanlike “care about people” shard. P(3 | 2, 1) = .55 due to plurality of value, I expect this to be one way it learns to make decisions

This is where I'd put a significantly low probability. Could you elaborate on why there's an inductive bias towards "just hooking human-like criteria for bidding on internal-AI-plans"? As far as I can tell, the inductive bias for human-like values would be something that at least seems closer to the human-brain structure than any arbitrary ML architecture we have right now. Rewarding a system to better model human beings' desires doesn't seem to me to lead it towards having similar desires. I'd use the "instrumental versus terminal desires" concept here but I expect you would consider that something that adds confusion instead of removing it.

AFAIK, only Gwern and I [LW · GW] have written concrete stories speculating about how a training run will develop cognition within the AGI. 

This worries me, if true (if not, please reply with more!). I think it would be awesome to have more concrete stories!^[1] If Nate, or Evan, or John, or Paul, or—anyone, please, anyone add more concrete detail to this website!—wrote one of their guesses of how AGI goes, I would understand their ideas and viewpoints better. I could go "Oh, that's where the claimed sharp left turn is supposed to occur." Or "That's how Paul imagines IDA being implemented, that's the particular way in which he thinks it will help." 

Maybe a contest would help?

ETA tone

1. ^{^}

   Even if scrubbed of any AGI-capabilities-advancing sociohazardous detail. Although I'm not that convinced that this is a big deal for conceptual content written on AF. Lots of people probably have theories of how AGI will go. Implementation is, I have heard, the bottleneck. 

   Contrast this with beating SOTA on crisply defined datasets in a way which enables ML authors to get prestige and publication and attention and funding by building off of your work. Seem like different beasts.

I also think a bunch of alignment writing seems syntactical. Like, "we need to solve adversarial robustness so that the AI can't find bad inputs and exploit them / we don't have to worry about distributional shift. Existing robustness strategies have downsides A B and C and it's hard to even get $ϵ$-ball guarantees on classifications. Therefore, ..."

And I'm worried that this writing isn't abstractly summarizing a concrete story for failure that they have in mind (like "I train the AI [with this setup] and it produces [this internal cognition] for [these mechanistic reasons]"; see A shot at the diamond alignment problem [LW · GW] for an example) and then their best guesses at how to intervene on the story to prevent the failures from being able to happen (eg "but if we had [this robustness property] we could be sure its policy would generalize into situations X Y and Z, which makes the story go well"). I'm rather worried that people are more playing syntactically, and not via detailed models of what might happen. 

Detailed models are expensive to make. Detailed stories are hard to write. There's a lot we don't know. But we sure as hell aren't going to solve alignment only via valid reasoning steps on informally specified axioms ("The AI has to be robust or we die", or something?).  

I don't think the conclusion follows from the premises. People often learn new concepts after studying stuff, and it seems likely (to me) that when studying human cognition, we'd first be confused because our previous concepts weren't sufficient to understand it, and then slowly stop being confused as we built & understood concepts related to the subject. If an AI's thoughts are like human thoughts, given a lot of time to understand them, what you describe doesn't rule out that the AI's thoughts would be comprehensible.

The mere existence of concepts we don't know about in a subject doesn't mean that we can't learn those concepts. Most subjects have new concepts.

A lot of examples of this sort of stuff show up in OpenAI clarity's circuits analysis work. In fact, this is precisely their Universality hypothesis. See also my discussion here [LW · GW].

There is a "HALT: hungry? angry? lonely? tired?" mnemonic, but I like that your list includes water and walking and exercise. Now just please make it easier to remember.

Hey can I steal this for a course I'm teaching? (I'll give you credit).

The reason $f^{\prime }\left(0\right)$ is undefined for the absolute value function is that you need the value to be the same for all sequences converging to 0 – both from the left and from the right. There's a nice way to motivate this in higher-dimensional settings by thinking about the action of e.g. complex multiplication, but this is a much stronger notion than real differentiability and I'm not quite sure how to think about motivating the single-valued real case yet. Of course, you can say things like "the theorems just work out nicer if you require both the lower and upper limits be the same"...

That is mostly an argument by intuition. To make it more rigorous and transparent, here is a constructive proof:

Let the curriculum be sampled uniformly at random. This has ~no mutual information with the world. Therefore the AI does not learn any methods in the world that can cause human extinction.

Can you sketch out some ideas for showing/proving premises 1 and 2? More specifically:

For 1, how would you rule out future distributional shifts increasing the influence of "bad" circuits beyond ϵ?

For 2, it seems that you actually need to show a specific K, not just that there exists K>0, otherwise how would you be able to show that x-risk is low for a given curriculum? But this seems impossible, because the "bad" subset of circuits could constitute a malign superintelligence strategically manipulating the overall AI's output while staying within a logit variance budget of ϵ (i.e., your other premises do not rule this out), and how could you predict what such a malign SI might be able to accomplish?

Upvoted and disagreed. ^[1]

One thing in particular that stands out to me: The whole framing seems useless unless Premise 1 is modified to include a condition like

[...] we can select a curriculum and reinforcement signal which [...] and which makes the model highly "useful/capable".

Otherwise, Premise 1 is trivially true: We could (e.g.) set all the model's weights to 0.0; thereby guaranteeing the non-entrainment of any ("bad") circuits.

I'm curious: what do you think would be a good (...useful?) operationalization of "useful/capable"?

Another issue: K and epsilon might need to be unrealistically small: Once the model starts modifying itself (or constructing successor models) (and possibly earlier), a single strategically-placed sign-flip in the model's outputs might cause catastrophe. ^[2]

Is a difficulty in moving from statements about the variance in logits to statements about x-risk?

One is a statement about the output of a computation after a single timestep, the other is a statement about the cumulative impact of the policy over multiple time-steps in a dynamic environment that reacts in a complex way to the actions taken.

My intuition is that for any $ϵ>0$ bounding the variance in the logits, you could always construct a suitably pathological environment that will always amplify these cumulative deviations into a catastrophy.

(There is at least a 30% chance I haven't grasped your idea correctly)

Potentially relevant: the theoretical results from On Effects of Steering Latent Representation for Large Language Model Unlearning. From Claude chat: 'Point 2 refers to the theoretical analysis provided in the paper on three key aspects of the Representation Misdirection for Unlearning (RMU) method:

1. Effect on token confidence:
   • The authors hypothesized that RMU causes randomness in the logits of generated tokens.
   • They proved that the logit of a token generated by an RMU model follows a Normal distribution.
   • This randomness in logits is interpreted as low confidence, leading to nonsensical or incorrect responses.
   • The analysis suggests that the variance of the logit distribution is influenced by the coefficient value and properties of the model layers.
2. Impact of the coefficient value:
   • The coefficient 'c' in RMU affects how well the forget sample representations align with the random vector.
   • Theoretical analysis showed a positive correlation between the coefficient and the alignment.
   • Larger coefficient values lead to more alignment between forget sample representations and the random vector.
   • However, the optimal coefficient value varies depending on the layer of the model being unlearned.
3. Robustness against adversarial jailbreak attacks:
   • The authors analyzed RMU's effectiveness against white-box jailbreak attacks from an attack-defense game perspective.
   • They showed that RMU causes the attacker to receive unreliable and uninformative gradient signals.
   • This makes it difficult for the attacker to find optimal adversarial tokens for replacement.
   • The analysis explains why RMU models demonstrate strong robustness against methods like Greedy Coordinate Gradient (GCG) attacks.

The theoretical analysis provides a foundation for understanding why RMU works and how its parameters affect its performance. This understanding led to the development of the improved Adaptive RMU method proposed in the paper.'

I can totally believe that agents that competently and cooperatively seek out to fulfill a goal, rather than seeking to trick evaluators of that goal to think it gets fulfilled, can exist.

However, whether you get such agents out of an algorithm depends on the details of that algorithm. Current reinforcement learning algorithms mostly don't create agents that competently do anything. If they were more powerful while still doing essentially the same thing they currently do, most of them would end up tricked by the agents they create, rather than having aligned agents.

After further review, this is probably beyond capabilities for the moment. 

Also, the most important part of this kind of experiment is predicting in advance what reward schedules will produce what values within the agent, such that we can zero-shot transfer that knowledge to other task types (e.g. XLAND instead of Minecraft) and say "I want an agent which goes to high-elevation platforms reliably across situations, with low labelling cost", and then sketch out a reward schedule, and have the first capable agents trained using that schedule generalize in the way you want.