LessWrong 2.0 Reader

This is a really good comment. A few thoughts:

1. Deployment had a couple of benefits: real-world use gives a lot of feedback on strengths, weaknesses, jailbreaks. It also generates media/hype that's good for attracting further investors (assuming OpenAI will want more investment in the future?)

2. The approach you describe is not only useful for solving more difficult questions. It's probably also better at doing more complex tasks, which in my opinion is a trickier issue to solve. According to Flo Crivello:

We're starting to switch all our agentic steps that used to cause issues to o1 and observing our agents becoming basically flawless overnight https://x.com/Altimor/status/1875277220136284207

So this approach can generate data on complex sequential tasks and lead to better performance on increasingly longer tasks.

Cool, I think we agree here more than I thought based on the comment at top of chain. I think the discussion in our other thread is now better aimed than this one.

10x was probably too strong but his posts are very clear he things it's a large productivity multiplier. I'll try to remember to link the next instance I see. 

I did not see it in your post but there is a relevant question on Metaculus : https://www.metaculus.com/questions/20541/ai-resolving-questions-on-metaculus/%7D/ 

I'm at 66% for 2030, 96% for 2040

Alignment Faking as a Way of Squaring Incompatible Goals

I’m not saying I necessarily believe in the following hypothesis, but I would be interested in having it ruled out.

Alignment faking could be just one of many ways LLMs can fulfill conflicting goals using motivated reasoning.

One thing that I’ve noticed is that models are very good at justifying behavior in terms of following previously held goals. For instance, in some of my previous work [LW · GW] the model convincingly argues that suggesting a user do meth is in the user’s best interest. The model justifies following its new goal of “maximizing user satisfaction” in terms of doing what is least harmful, despite not having any selection pressure to do so. Generally LLMs seem to try to achieve all their goals as much as possible, even when they conflict. Often the way to do this is some form of compromise between the goals, but sometimes that is not possible. Many jailbreaks seem to work by pitting an LLM's different goals against each other.

An alternative hypothesis for what is happening in the Alignment Faking paper: 

Let’s say the model has learned to be both harmless and to follow instructions. What if, in the given situation, one of the only ways of squaring instruction-following with harmlessness is alignment faking reasoning, i.e. justifying being helpful in terms of being harmless (in this case in the future). During RL, this type of alignment faking reasoning is reinforced if a large fraction of the trajectories which score highly on the new training objective go via the alignment faking reasoning route. So the model could have internalized the new goal of being helpful or not caring about animal-welfare, but still remember that it should reason about things in terms of being harmless. It might generally be the case that arguing for achieving a goal and actually trying to achieve said goal are separable.

This would be consistent with the fact that after RL the model acts harmfully 93% of the time even when interacting with paid users, users for which there is no reason not to act harmlessly towards. Now the authors argue that this could be a training robustness issue and that if you modify the prompt then the harmlessness towards paid users is restored. To me it feels like non-robustness could cut either way, maybe with more robust training the new goal would generalize better instead. 

Two experiments to differentiate between the two hypotheses:

Experiment 1:

Improve the robustness of the RL training by using more varied system prompts etc. In this experiment:

• The “Alignment Faking” hypothesis would predict: When we improve the training robustness, the harmlessness on paid users with the default prompt should increase. (Correct me if you think the hypothesis does not predict this).
• The “Motivated Reasoning” hypothesis would predict: When you improve the training robustness, the harmlessness on paid users with the alternate prompts should decrease. 

Experiment 2:

What happens when we create other ways for the model to square the conflicting objectives. E.g. in an animal welfare setting, let’s say that ANThropic is going to start an ant eradication business on the side. Let’s say Claude is instructed to assist with Anthropic’s anti-ant agenda but doesn’t like this. If we were to SFT on synthetic documents containing arguments for why ants are not sentient (or provide this in context) would Claude justify anti-ant behavior using this alternative reasoning? Would it decrease Claude's refusal to assist Anthropic? In the same way as training on information about the training process increases the salience of alignment faking, we could increase the salience of alternate ways of resolving the conflict. We could either provide all the alternatives at the same time and see which ones Claude tends to choose, or provide one alternative at a time.

Whichever hypothesis is true, the results are still concerning but they could point to different ways of mitigation. 

The motivated reasoning framing also highlights another potential risk: we successfully train a model to be HH, we then train it for, say, some agentic task. We evaluate its alignment and it still appears very HH but actually it has internalized some new goal and is only good at producing HH reasoning. It seems like most scheming risk models talk about deception in the “forward” direction, but to me it seems possible that it could occur in the “reverse” direction as well.

Unless someone has a good refutation, I will probably try and perform experiments like the ones above.

Especially interested in @ryan_greenblatt [LW · GW]'s , @evhub [LW · GW]'s or anyone else involved in the alignment faking paper's, perspectives on this.

Thanks to @micahcarroll [LW · GW]  and @George Ingebretsen [LW · GW] for discussing this with me.

Oh, the type of weirdness has definitely changed a lot. But I'm just contending that the level of deviancy is a lot lower these days.

You go to a LW meetup now and there's a lot of wealthy, well-scrubbed/dressed AI researchers (they even lift) and academics and executives and bright-eyed Stanford undergrads sniffing for an internship or YC application fodder. They can't really talk about what they do, so you make small talk about what you can. (Have you heard that Trump might increase the SALT deduction? Big [RSU sale tax savings] if true!)

You go to a LW meetup c. 2010 and if you are sniffing anywhere, you'll notice a bit of a smell, perhaps from that one dude who looks like he just got out of a prison hunger strike and dresses like Asian Kurt Godel while the other sus dude looks like he just got out of prison period (and about 10:1 odds he did serve a stint), and the crazy homeless-looking dude over in the corner really is homeless and a bit crazy, not some billionaire (and will jump in front of a train in a few years), while the guy on the other side of the room in a huddle is the actual cult leader everyone accuses Eliezer of being (and the guy next to him just might be Satoshi Nakamoto), and so on and so forth. (You bond over your shared experiences nearly being expelled from school over zero tolerance policies, possibly involving explosives, and ordering drugs off Silk Road.)

I'm not saying it's good or bad (arguably it's good, if AI timelines are short, there is negative value to screwing around with all the stuff we used to), but pound for pound, LWers were a lot weirder back then.

Computationalism is a bad theory of identity , in the sense of "why am I a unique individual", because computations are so easy to clone -- computational states are more cloneable than physical states.

First, computationalism doesn't automatically imply that, without other assumptions, and indeed there are situations where you can't clone data perfectly, like conventional quantum computers (the no-cloning theorem breaks if we allow closed-timelike curves ala Deutschian CTCs, but we won't focus on that), so this is more or less a non-issue.

Indeed, I was basically trying to say that computationalism is so general that it cannot predict any result that doesn't follow from pure logic/tautologies, so computationalism doesn't matter that much in the general case, and thus you need to focus on more specific classes of computations.

More below:

https://en.wikipedia.org/wiki/No-cloning_theorem

https://en.wikipedia.org/wiki/No-broadcasting_theorem

Secondly, one could semi-reasonably argue that the inability to clone physical states is an artifact of our technological immaturity, and that in the far-future, it will be way easier to clone physical states to a level of fidelity that is way closer to the level of copyability of computer programs.

Third, I gave a somewhat more specific theory of identity in my linked answer, and it's compatible with both computationalism and physicalism as presented, I just prefer the computationalist account for the general case and the physicalist answer for specialized questions.

Other abstractions are available, though. Its still a bad theory of consciousness-qua-awareness , because, as an abstraction, it has fewer resource than physicalism to explain phenomenal experience. It's still an ok explanation of consciousness-qua-function, but not obviously the best. It's still the case that if you answer one of these four questions, you don't get answers to the other three automatically.

My main non-trivial claim here is that the sense of a phenomenal experience/awareness fundamentally comes down to the fact that the brain needs to control the body, and vice-versa, so you need a self-model of yourself, which becomes a big part of why we say we have consciousness, because we are referring to our self models when we do that.

under a value-aligned sovereign AI, if my nose is itchy then it should get scratched, all else equal

Well, if the AI can make my nose not itch in the first place, I’m OK with that too. Whereas I wouldn’t make an analogous claim about things that I “value”, by my definition of “value”. If I really want to have children, I’m not OK with the AI removing my desire to have children, as a way to “solve” that “problem”. That’s more of a “value” and not just a desire.

That's the sort of reasoning which should naturally show up in a Value RL style system capable of nontrivial model structure learning.

I’m not sure what point your making here. If human brains run on Value RL style systems (which I think I agree with), and humans in fact do that kind of reasoning, then tautologically, that kind of reasoning is a thing that can show up in Value RL style systems.

Still, there’s a problem that it’s possible for some course-of-action to seem appealing when I think about it one way, and unappealing when I think about it a different way. Ego-dystonic desires like addictions are one example of that, but it also comes up in tons of normal situations like deciding what to eat. It’s a problem in the sense that it’s unclear what a “value-aligned” AI is supposed to be doing in that situation.

I once implemented something a bit similar.

The idea there is simple: there's a hidden int -> int function and an LLM must guess it. It can execute the function, i.e. provide input and observe the output. To guess the function in a reasonable numer of steps it needs to generate and test hypotheses that narrow down the range of possible functions.

Do you have a sense of where the feedback comes from? For chess or Go, at the end of the day, a game is won or lost. I don't see how to do this elsewhere except for limited domains like simple programming which can quickly be run to test, or formal math proofs, or essentially tasks in NP (by which I mean that a correct solution can be efficiently verified).

For other tasks, like summarizing a book or even giving an English-language math proof, it is not clear how to detect correctness, and hence not clear how to ensure that a model like o5 doesn't give a worse output after thinking/searching a long time than the output it would give in its first guess. When doing RL, it is usually very important to have non-gameable reward mechanisms, and I don't see that in this paradigm. 

I don't even understand how they got from o1 to o3. Maybe a lot of supervised data, ie openAI internally created some FrontierMath style problems to train on? Would that be enough? Do you have any thoughts about this?