LessWrong 2.0 Reader

Still, IMO, exploiting the frozen planner through adversarial inputs in a single step seems pretty unlikely to be a fruitful strategy for the optimized planner. If the optimized planner is simply trying to accurately convey information to the frozen planner, probably >99% of that information is best to convey through human-understandable text.

Well, I'm not sure. As you mention, it depends on the step size. It also depends on how vulnerable to adversarial inputs LLMs are and how easy they are to find. I haven't looked into the research on this, but it sounds empirically checkable. If there are lots of adversarial inputs which have a wide array of impacts on LLM behavior, then it would seem very plausible that the optimized planner could find useful ones without being specifically steered in that direction.

This is especially likely to hold under the following conditions:

We can also combine this with other proposals, such as paraphrasing.

This cognitive phenomenon is usually lumped in with “confirmation bias.” However, it seems to me that the phenomenon of trying to test positive rather than negative examples, ought to be distinguished from the phenomenon of trying to preserve the belief you started with. “Positive bias” is sometimes used as a synonym for “confirmation bias,” and fits this particular flaw much better.

Subtle distinction I almost missed here. Worth expanding.

Yeah, you're right, I no longer think it's an interesting proposal.

For next time: https://www.lesswrong.com/posts/LfrNFfJFcqnG9WuFf/activated-charcoal-for-hangover-prevention-way-more-than-you [LW · GW] ;)

I was really sleep deprived and slightly intoxicated yesterday and wrote this. It was amusing to me, at least, in the state I was in.

Also this:

From Altman: [...] Admitted that he lost a lot of trust with Greg and Ilya through this process. Felt their messaging was inconsistent and felt childish at times. [...] Sam was bothered by how much Greg and Ilya keep the whole team in the loop with happenings as the process unfolded. Felt like it distracted the team.

Apparently airing such concerns is "childish" and should only be done behind closed doors, otherwise it "distracts the team", hm.

I can totally believe this. But, I also think that responsibly wearing the scientist hat entails prep work before engaging in a four hour public discussion with a domain expert in a field. At minimum that includes skimming the titles and ideally the abstracts/outlines of their key writings. Maybe ask Claude to summarize the highlights for you. If he'd done that he'd have figured out many of the answers to many of these questions on his own, or much faster during discussion. He's too smart not to.

Otherwise, you're not actually ready to have a meaningful scientific discussion with that person on that topic.

A few quotes that stood out to me:

Greg:

I hope for us to enter the field as a neutral group, looking to collaborate widely and shift the dialog towards being about humanity winning rather than any particular group or company. 

Greg and Ilya (to Elon):

The goal of OpenAI is to make the future good and to avoid an AGI dictatorship. You are concerned that Demis could create an AGI dictatorship. So do we. So it is a bad idea to create a structure where you could become a dictator if you chose to, especially given that we can create some other structure that avoids this possibility.

Greg and Ilya (to Altman):

But we haven't been able to fully trust your judgements throughout this process, because we don't understand your cost function.

We don't understand why the CEO title is so important to you. Your stated reasons have changed, and it's hard to really understand what's driving it.

Is AGI truly your primary motivation? How does it connect to your political goals? How has your thought process changed over time?

I'm talking about training only the Face, not training the policy (shoggoth) at all with the proposal I'm imagining.

And, these should clearly be separate models such that the training of one doesn't generalize to the other.

So, making the Face more deceptive doesn't kill the canary?

Ah, yes, I had neglected this important distinction.

So what do you do in the end, throw away the face?

It seems worth pointing out that although the CoT isn't trained to be deceptive, it is trained to think thoughts which help out a deceptive face. So it seems plausible that a multi-step training process like:

GPT -> basic chat training -> shoggoth+face training -> discard face & align shoggoth

Would result in deceptive capabilities: the shoggoth CoT learns to explicitly reason about what helps deceive the alignment training in the shoggoth+face step, and then in the final step, some of this is destroyed but some is driven underground. IE, learning to explicitly reason about deception creates latent structure which is useful for that task, and this latent structure will somewhat transfer to the case where the deception has to be hidden from the CoT.

The question is whether we end up with a highly misaligned policy (and this is clear from the chain of thought such that we can discover this). I think Daniel was expecting in the baseline proposal that you end up with a Face that does at least some censoring/mild deception due to training incentives (I think we already see a bit of this with o1).

So it sounds like we have some disagreement about what degree of deception is considered problematic. Again, I'm worried about basin-of-attraction type dynamics. We can think of a whole frontier lab as collectively being a sort of self-improving AGI; we want that recursive self-improvement to be going down a good path rather than a bad one. The LLMs are already helping to improve themselves in little ways (EG as coding assistants) and big ways (EG constitutional AI). So it's an environment where seeds of deceptiveness can grow and take some control of the overall trajectory.

Maybe your concern is that it is plausible that the policy won't be misaligned, but the Face will be and this will screw us over. This seems plausible and is worth worrying about (with both proposals). However, I thought that a core part of the hope is that the Face isn't smart enough to cleverly scheme (unless this schemeing happens in the CoT itself). And indeed, if the Face can cleverly scheme without CoT, what is stopping the policy from doing this too?

I'm not sure what you're getting at here. It seems like the idea is that Face + CoT is, together, a clever schemer -- in the o1 report, we already see cases of explicit scheming in the CoT. So then, using the Face+CoT together (EG trying to train it further, using it in supervision steps for later iterations of the system, etc) is dangerous and difficult for the same reason that a single deceptive LLM would be. If we use Face+CoT together, we have to reason about them together in that way. Or, if we are using them as separate pieces, then we have to account for the part of the overall deceptive-schemer strategy they've learned.

I guess you could use Face+CoT as a supervisor (EG in constitutional AI) by looking at the CoT (both by hand and using some automation) and throwing out any deceptive examples, so as to not use them for supervision. 

But the overall goal should be to produce a highly non-deceptive system in the end, I think.

Another idea would be to do the whole Face+CoT thing, and then filter for honesty on a bunch of examples, and then re-train a whole new system (well, you can start from the same GPT) only on those honest examples. 

give better names to actual formal math things, jesus christ.