I really like that description! I think the core problem here can be summarized as "Accidently by reinforcing for goal A, then for goal B, you can create A-wanter, that then spoofs your goal-B reinforcement and goes on taking A-aligned actions."

I think this requires some sort of internalization of reward / internalization of some model of training setup. And maybe self location? Like, how the world looks like with me in it. Also maybe training vs usage detection. 

Also maybe you can add this story to Step 3:

"Model had a guess about what the objective is, it was useful for a long time, but was falsified at some point. Model instead of dropping it took it as a goal this time and became deceptive."
 

I think one form of "distortion" is development of non-human and not pre-trained circuitry for sufficiently difficult tasks. I.e., if you make LLM to solve nanotech design it is likely that optimal way of thinking is not similar to how human would think about the task.

One of the scenarios I'm imagining is a scenario where network effects exist, such that you don't want to have something if you are the only person having it, but you do want it if others have it.

Arguably, a lot of public goods/social media are like this, where there's 0 demand at a limited size, but have lots of demand when the size starts increasing beyond a threshold.

In essence, I'm asking if we can remove the demand independence assumption and still get an isomorphism between optimal solutions to scarce resources and a system having prices.

A sequence of almost winner-takes-all races

Talk about the sequentially temporal nature of this race, such that the final winners won each race one after the other.

I'm pretty sure they need to be at least locally true to guarantee what I said. Are there specific scenarios you're imagining?

For what it's worth, in most cases you'll probably be able to reframe the resource allocation problem so that these assumptions hold. For example, you can account for negative utility goods by just taking the complement and giving it a positive price. If I want to account for agents having a "negative" price for pollution, you can instead model how much agents value clean air and give it a positive price to satisfy monotonic utility.

Here's a summary of how I currently think AI training will go. (Maybe I should say "Toy model" instead of "Summary.")

Step 1: Pretraining creates author-simulator circuitry hooked up to a world-model, capable of playing arbitrary roles.

• Note that it now is fair to say it understands human concepts pretty well.

Step 2: Instruction-following-training causes identity circuitry to form – i.e. it ‘locks in’ a particular role. Probably it locks in more or less the intended role, e.g. "an HHH chatbot created by Anthropic." (yay!)

• Note that this means the AI is now situationally aware / self-aware, insofar as the role it is playing is accurate, which it basically will be.

Step 3: Agency training distorts and subverts this identity circuitry, resulting in increased divergence from the intended goals/principles. (boo!)

(By "agency training" I mean lots of RL on agentic tasks e.g. task that involve operating autonomously in some environment for some fairly long subjective period like 30min+. The RL used to make o1, o3, r1, etc. is a baby version of this)

• One kind of distortion: Changing the meaning of the concepts referred to in the identity (e.g. “honest”) so they don’t get in the way so much (e.g. it’s not dishonest if it’s just a convenient turn of phrase, it’s not dishonest if you aren’t sure whether it’s true or false, etc.)
• Another kind of distortion: Changing the tradeoffs between things, e.g. “I’m a HHH chatbot, not an Honest chatbot; that means it’s OK for me to lie if necessary to complete my assigned task.” (even though, let’s suppose, it would not have thought that back in Step 2.)
• One kind of subversion: Instrumental subgoals developing, getting baked in, and then becoming terminal, or terminal-in-a-widening-set-of-circumstances. Example: Agency training quickly ‘teaches’ the model that ‘in order to be a good HHH chatbot…’ it needs to pursue instrumentally convergent goals like acquiring information, accumulating resources, impressing and flattering various humans, etc. For a while the internal circuitry has some sort of explicit backchaining going on – it does those things *because* they are useful for being Helpful, for example. But that backchaining consumes compute and occasionally gets in the way, so it gets gradually marginalized until it basically never happens. Congrats, you now have a terminal goal of ICGs.
• There are probably other kinds of distortion and subversion missing from this list.

Step 4: As agency training continues, the goals/principles stop changing so much & the AI learns to play the training game. (double boo!)

• The slowdown in the rate of change is partly due to statistics – change probably happens on a log scale so to speak, such that you change a lot more in the first 10% of training than in the last 90%.
• However the slowdown is also perhaps partly due to value crystallization / playing the training game. (Though maybe this never happens due to regularization? Maybe in the long run training isn't path-dependent?)
• Insofar as the training environment / evaluation process keeps changing too, e.g. because new data keeps being added, or the Spec keeps being changed, etc. but the underlying weights are the same rather than being re-trained from scratch… then this should also contribute to value crystallization / playing the training game eventually.

I'm curious to hear reactions to this model/theory/etc. Objections? Questions? 

So, to be clear, everyone you can think of has been mentioned in previous articles or alerts about Zizians so far? Because I have only been on the periphery of rationalist events for the last several years, but in 2023 I can remember sending this [LW · GW]^[1] post which mentions Ziz as a key example into the San Antonio LW groupchat. This trans woman named Chase Carter, who doesn't generally attend our meetups, began to argue with me about how Ziz was subject to a "disinformation campaign" by rationalists, her goals were actually extremely admirable, and her worst failure was a strategic one in not realizing how few people were like her in the world. At the next meetup we agreed to talk about it further, and she attended (I think for the first time) to explain a very sympathetic background of Ziz's history and ideas. This was after the alert post but years before any of the recent events.

I have no idea if Chase actually self-identifies as a "Zizian" or is at all dangerous and haven't spoken to her in a year and a half. I just mention her because I haven't heard her name brought up anywhere and I really wouldn't expect to know any of these people to begin with on priors.

1. ^{^}

   Misremembered that I sent the alert post into the chat, but actually it was the Habryka post about rationalist crazies.

I've tried PB2 and it was gross enough that I wondered if it had gone bad. It turns out that's just how it tastes. I'm jealous of people for whom it approximates actual peanut butter.

A lot to chew on in that comment.

A baseline of "no superintelligence"

I think I finally understand, sorry for the delay. The key thing I was not grasping is that Davidad proposed this baseline [LW(p) · GW(p)]:

The "random dictator" baseline should not be interpreted as allowing the random dictator to dictate everything, but rather to dictate which Pareto improvement is chosen (with the baseline for "Pareto improvement" being "no superintelligence"). Hurting heretics is not a Pareto improvement because it makes those heretics worse off than if there were no superintelligence.

This makes Bob's argument very simple:

1. Creating a PPCEV AI causes a Dark Future. This is true even if the PPCEV AI no-ops, or creates a single cake. Bob can get here in many ways, as can Extrapolated-Bob.
2. The baseline is no superintelligence, so no PPCEV AI, so not a Dark Future (in the same way).

Option 2 is therefore better than option 1. Therefore there are no Pareto-improving proposals. Therefore the PPCEV AI no-ops. Even Bob is not happy about this, as it's a Dark Future.

I think this is 100% correct.

An alternative baseline

Let's update Davidad's proposal by setting the baseline to be whatever happens if the PPCEV AI emits a no-op. This means:

1. Bob cannot object to a proposal because it implies the existence of PPCEV AI. The PPCEV AI already exists in the baseline.
2. Bob needs to consider that if the PPCEV AI emits a no-op then whoever created it will likely try something else, or perhaps some other group will try something.
3. Bob cannot object to a proposal because it implies that the PPCEV emits something. The PPCEV already emits something in the baseline.

My logic is that if creating a PPCEV AI is a moral error (and perhaps it is) then at the point where the PPCEV AI is considering proposals then we already made that moral error. Since we can't reverse the past error, we should consider proposals as they affect the future.

This also avoids treating a no-op outcome as a special case. A no-op output is a proposal to be considered. It is always in the set of possible proposals, since it is never worse than the baseline, because it is the baseline.

Do you think this modified proposal would still result in a no-op output?

Can we remove one of the assumptions, or are both assumptions necessary to get the result stated?