LessWrong 2.0 Reader

Thanks!

This could be taken as an argument for using some type of goals selected from learned knowledge [LW · GW] for alignment if possible.

Yeah that’s what I was referring to in the paragraph:

“Well OK,” says the optimist. “…so much the worse for Ev! She didn’t have interpretability, and she didn’t have intelligent supervision after the training has already been running, etc. But we do! Let’s just engineer the AI’s explicit motivation!”

Separately, you also wrote:

we're really training LLMs mostly to have a good world model and to follow instructions

I think I mostly agree with that, but it’s less true of o1 / r1-type stuff than what came before, right? (See: o1 is a bad idea [LW · GW].) Then your reply is: DeepSeek r1 was post-trained for “correctness at any cost”, but it was post-post-trained for “usability”. Even if we’re not concerned about alignment faking [LW · GW] during post-post-training (should we be?), I also have the idea at the back of my mind that future SOTA AI with full-fledged (1-3) loops probably (IMO) won’t be trained in the exact same way than present SOTA AI, just as present SOTA AI is not trained in the exact same way as SOTA AI as recently as like six months ago. Just something to keep in mind.

Anyway, I kinda have three layers of concerns, and this is just discussing one of them. See “Optimist type 2B” in this comment [LW(p) · GW(p)].

Re-reading this a couple days later, I think my §5.1 discussion didn’t quite capture the LLM-scales-to-AGI optimist position. I think there are actually 2½ major versions, and I only directly talked about one in the post. Let me try again:

• Optimist type 1: They make the §5.1.1 argument, imagining that humans will remain in the loop, in a way that’s not substantially different from the present.
  • My pessimistic response: see §5.1.1 in the post.
• Optimist type 2: They make the §5.1.3 argument that our methods are better than Ev’s, because we’re engineering the AI’s explicit desires in a more direct way. And the explicit desire is corrigibility / obedience. And then they also make the §5.1.2 argument that “AIs solving specific technical problems that the human wants them to solve” will not undermine those explicit motivations, despite the (1-3) loop running freely with minimal supervision, because the (1-3) loop will work in vaguely intuitive and predictable ways on the object-level technical question.
  • My pessimistic response has three parts: First, the idea that a full-fledged (1-3) loop will not undermine corrigibility / obedience is as yet untested and at least open to question (as I wrote in §5.1.2). Second, my expectation is that some training and/or algorithm change will happen between now and “AIs that really have the full (1-3) triad”, and that change may well make it less true that we are directly engineering the AI’s explicit desires in the first place—for example, see o1 is a bad idea [LW · GW]. Third, what are the “specific technical problems that the human wants the AI to solve”??
• …Optimist type 2A answers that last question by saying the “specific technical problems that the human wants the AI to solve” is just, whatever random things that people want to happen in the world economy.
  • My pessimistic response: See discussion in §5.1.2 plus What does it take to defend the world against out-of-control AGIs? [LW · GW] Also, competitive dynamics / race-to-the-bottom is working against us, in that AIs with less intrinsic motivation to be obedient / corrigible will wind up making more money and controlling more resources.
• …Optimist type 2B instead answers that last question by saying that the “specific technical problems that the human wants the AI to solve” is alignment research, or more specifically, “figuring out how to make AIs whose motivation is more like CEV or ambitious value learning rather than obedience”.
  • My pessimistic response: The discussion in §5.1.2 becomes relevant in a different way—I think there’s a chicken-and-egg problem where obeying humans does not yield enough philosophical / ethical taste to judge the quality of a proposal for “AI that has philosophical / ethical taste”. (Semi-related: The Case Against AI Control Research [LW · GW].)

We're pretty firmly committed to at least 3. I think whether we have more than that depends on how well we're doing financially and whether the world is still around at that point.

Collins not Gladwell lmao

What do you think of Deepseek's announcement of R1 being evidence that AI capabilities will slow down, because they got a cheaper but not actually better model, and this kills the industry's growth?

https://www.lesswrong.com/posts/ynsjJWTAMhTogLHm6/?commentId=a2y2dta4x38LqKLDX [LW · GW]

• It's possible that we might manage to completely automate the more objective components of research without managing to completely automate the more subjective components of research. That said, we likely want to train wise AI advisors to help us with the more subjective components even if we can't defer to them.

Agree, I expect the handoff to AI agents to be somewhat incremental (AI is like an intern, a new engineer, a research manager, and eventually, a CRO)

Actually the paper doesn't have any more on this topic than the paragraph above.

I expect that's about not trusting the foreigners who did the clinical trials (I have heard this)^[1], and not so much about expecting that Americans are biologically different from foreigners.

1. ^{^}

   Specifically, someone with some knowledge told me that the FDA knows that there are some countries where the trials are completely untrustworthy.  And that there was a political decision where they said, "If we disallow trials from some countries but not others, there will be much complaining and we'll probably be called racists", and solved the issue by disallowing trials from all foreign countries.

I hope it would, but I actually think it would depend on who or what killed whom, how, and whether it was really an accident at all.

If an American-made AI hacked the DOD and nuked Milan because someone asked it to find a way to get the 2026 Olympics moved, then I agree, we would probably get a push back against race incentives.

If a Chinese-made AI killed millions in Taiwan in an effort create an opportunity for China to seize control, that could possibly *accelerate* race dynamics.

Just because the number of almost-orthogonal vectors in $d$ dimensions scales exponentially with $d$, doesn't mean one can choose all those signals independently. We can still only choose $d$ real-valued signals at a time (assuming away the sort of tricks by which one encodes two real numbers in a single real number, which seems unlikely to happen naturally in the body). So "more intended behaviors than input-vector components" just isn't an option, unless you're exploiting some kind of low-information-density in the desired behaviors (like e.g. very "sparse activation" of the desired behaviors, or discreteness of the desired behaviors to a limited extent).