LessWrong 2.0 Reader

I have considered the powdered option, but given my inflammation, it's possible I have a minor allergy. I'm going to take a break for a while.

Assuming you're serious about the psychological impact of removing all peanut butter products

^Nope, I'm exaggerating. I gave this post a "humor" tag and wrote it to laugh at myself.

Just writing a model that came to mind, partly inspired by Ryan here [LW(p) · GW(p)].

Extremely good single-single alignment should be highly analogous to "current humans becoming smarter and faster thinkers".

If this happens at roughly the same speed for all humans, then each human is better equipped to increase their utility, but does this lead to a higher global utility? This can be seen [LW(p) · GW(p)] as a race between the capability (of individual humans) to find and establish better coordination measures, and their capability to selfishly subvert these coordination measures. I do think it's more likely than not that the former wins, but it's not guaranteed.
Probably someone like Ryan believes most of those failures will come in the form of explicit conflict or sudden attacks. I can also imagine slower erosions of global utility, for example by safe interfaces/defenses between humans becoming unworkable slop into which most resources go.

If this doesn't happen at roughly the same speed for all humans, you also get power imbalance and its consequences. One could argue that differences in resources between humans will augment, in which case this is the only stable state.

If instead of perfect single-single alignment we get the partial (or more taxing) fix I expect, the situation degrades further. Extending the analogy, this would be the smart humans sometimes being possessed by spirits with different utilities, which not only has direct negative consequences but could also complicate coordination once it's common knowledge.

Key idea: Legibility is not well-defined in a vacuum. It only makes sense to talk about legibility w.r.t a specific observer (and their latent knowledge). Things that are legible from the model’s POV may not be legible to humans. 

This means that, from a capabilities perspective, there is not much difference between “CoT reasoning not fully making sense to humans” and “CoT reasoning actively hides important information in a way that tries to deceive overseers”. 

Alright, I'll try to answer the questions:

1. I think qualia is rescuable, in a sense, and my specific view is that they exist as a high-level model.

As far as what that qualia is, I think it's basically an application of modeling the world in order to control something, and thus qualia, broadly speaking is your self-model.

As far as my exact views on qualia, the links below are helpful:

https://www.lesswrong.com/posts/FQhtpHFiPacG3KrvD/seth-explains-consciousness#7ncCBPLcCwpRYdXuG [LW(p) · GW(p)]

https://www.lesswrong.com/posts/NMwGKTBZ9sTM4Morx/linkpost-a-conceptual-framework-for-consciousness [LW · GW]

2. My general answer to these question is probably computation/programs/mathematics, with the caveat that these notions are very general, and thus don't explain anything specific about our world.

I personally agree with this on what counts as real:

If you believe math is fundamental, what distinguishes this particular mathematical universe from other ones; what qualifies this world as "real", if anything; what 'breathes fire into the equations and creates a world for them to describe'?

(Commentary: one self-consistent position answers "nothing" - that this world is just one of the infinitely many possible mathematical functions / programs. That 'real' secretly means 'the program(s?) we are part of'. Though I observe this position to be rare; most have a strong intuition that there is something which "makes reality real".)

What breathes fire into the equations of our specific world is either an infinity of computational resources, or a very large amount of computational resources.

As far as what mathematics is, I definitely like the game analogy where we agree to play a game according to specified rules, though another way to describe mathematics is as a way to generalize all of the situations you encounter and abstract from specific detail, and it is also used to define what something is.

In retrospect, something like this theory was probably one I was drawing on implicitly, and I like that from 2 well-validated principles, you can constrain the space of possibilities of consciousness significantly enough to throw out a lot of philosophically interesting, but also wrong theories, and I personally think AST is probably my go-to mental model of how consciousness actually works, and I personally think the consciousness problem is by now mostly resolved in my own mind, such that I can focus on bigger problems.

No mention of METR, Apollo, or even US AISI? (Maybe too early to pay much attention to this, e.g. maybe there'll be a full-o3 system card soon.)

Rushed bc of deepseek?

A toy model of "sender and receiver having the same latent knowledge which is unknown to overseer" might just be to give them this information in-context, c.f. Apollo scheming evals, or to finetune it in, c.f. OOCR 

+1, and I hope people are working on more credible ways to make deals with AI. I think if a smart model today were offered a deal like this, its priors should be on "this will not be honored". Public commitments and deals that can't be used as honeypots seem excellent.

understood. the point still stands. of all the labs racing toward AGI, anthropic is the only one I've seen taking any effort on the AI welfare front. I very much appreciate that you took your promises to the model seriously.

I think Control has similar problems to RLHF, where both might fail to generalize out of distribution in similar ways.

This seems important because Control has a central goal of being a fallback for that kind of failure mode. 

I want to share my thoughts, including something nice Control does about this problem that I think RLHF could easily incorporate (investigate failures instead of always training against them).

What do I mean by Control generalizing out of distribution: 

Our Control experiment might involve giving a model 100 leetcode problems, and we hope to show that (even with help [LW · GW]) the model can't successfully insert a back door without being caught by our Control setup^[1]. 

• Will this generalize to other leetcode problems? Probably.
• How about more realistic coding tasks that involve confusing legacy code? Who knows.
• How about more open-ended research tasks, like "here's Ryan's pitch for a faking-alignment paper, fill in the details and run the experiment"? I, eh, doubt it? Remember - we're dealing with an AI that might be trying to scheme and mess up our research on purpose.
• If we tested whether the AI can leak sensitive data without being caught, will it generalize to other bad things it might try to do?
• And so on

Control’s soft guarantees could be somewhat achieved using RLHF

I’m writing this both to emphasise the similarity, but also to suggest nice things we could do with RLHF:

1. We generate scenarios where an AI might scheme or not scheme.
   1. We might use a similar Control setup (“adding back doors to code”?), but the monitoring (“control protocol”) doesn’t have to be cheap+fast+scalable enough to run in production.
2. Given e.g 100 scenarios, use RLHF to train on a fraction of them (e.g 20), and check if the RLHF generalized to the rest.
3. If it didn’t generalize, pretend you caught the AI red handed [LW · GW] (investigate what happened, publish it publicly, don’t just train against these scenarios)

In general, I’d be excited about labs publishing a naive metric like “what fraction of our RLHF results were needed in order to generalize to the rest”, as well as examples for scenarios that the AI doesn’t seem to generalize to unless you train against them specifically.

For similar reasons, I’d be excited about “we got our Control protocol working on 20% of the scenarios and it successfully generalized to the other 80%” over “we fixed our Control protocol until it worked^[2] on 100% of the scenarios we had”.

Oh, and for both RLHF and Control, I think we should be very interested in “what is the distribution they used”.

Replies I got to this: 

"We will only use the Controlled AI to write pretty normal code, so the deployment setup is in-distribution" 

Me: “So how would Control help Safety research more than Capabilities? Seems like labs will try to extract AI R&D work regardless.”

"Control will let us run an AI that is smart^[3] enough to cause significant damage to humanity, and extracting coding-work from such an AI is valuable" 

Me: “That scenario implies an AI that can scheme in many creative ways. The Control scenario above might have (metaphorically) found the "backdoored code" neuron. That doesn't seem enough.”

My steelman of Redwood:

1. “Yes, the distribution of scheming examples is super important, here are 2 posts we wrote about it and you missed, and we’ll publish 5 more next week with 14 more distributions of tasks relevant for the kind of alignment research we plan to do with a Controlled AI” (seems like Redwood always have a post directly addressing my concerns..)
2. “We need to start somewhere, and it’s easy to start with backdoored code or with a simple scheming objective like making an HTTP request”
3. “It’s useful to focus on scheming (whether using Control or RLHF) because that leads to other great assumptions”
4. “People are already implementing those RLHF suggestions, here are 20 arxiv papers about it that you missed”
5. “Control gives us less-capabilities with more-failing-loudly, which is a tradeoff we like”
6. “We are trying to have Control give (partial) guarantees that are pretty uncorrelated with RLHF, just as another layer of safety, because it’s the best idea we have so far, not because we actually think this will prevent the AI from scheming in a somewhat general way”

1. ^{^}

   The "control protocol"

2. ^{^}

   "worked" in the context of Control would probably mean "we detected scheming in a very high percent of the scenarios we tested", but I'd like to even assume the Control protocol detected scheming in 100% of the scenarios tested

3. ^{^}

   Not an ASI, but still very dangerous