LessWrong 2.0 Reader

Thanks for the post. It'd be helpful to have a TL;DR for this (an abstract), since it's kinda long - what are the main points you're trying to get across?

links 1/30/25: https://roamresearch.com/#/app/srcpublic/page/01-30-2025

• https://www.astralcodexten.com/p/why-recurring-dream-themes Scott Alexander on recurring dreams
• https://darioamodei.com/on-deepseek-and-export-controls Dario Amodei on DeepSeek and export controls
  • he's right that efficiency improvements in AI training shouldn't reduce demand for chips; In This House We Believe in Jevon's Paradox
  • I believe him if he says he's priced in efficiency improvements into his estimates for time and cost to human-level AI performance. it would be stupid not to, and he's not stupid.
  • the case he's making is "no, DeepSeek doesn't show that export controls backfire by motivating Chinese AI researchers to pursue efficiency."
    • I think I agree narrowly.
    • China's top AI researchers aren't stupid either; they're pursuing efficiency anyhow. So is everyone.
      • I think it's never wise to bet on the best people in a competitive field being complacent, even when they can "afford to be."
      • Poverty is a cost, not a benefit. It's not clear to me that China's chip-poverty (or overall poverty) made DeepSeek more innovative than its American counterparts.
        • I know there are DeepSeek stans out there who think everything about American culture is toxic, but the world is big, you have not read every AI paper under the sun, and you simply do not have enough information to conclude that they are uniquely innovative.
      • also, whether we have export controls or not right now, it is Known that the US-China relationship is chilly and that it's in both countries' national interest to pursue independent tech stacks.
    • otoh, while I don't think DeepSeek's strong performance was caused by export controls, I also think we can't make strong claims in the other direction like "if we have these export controls, Chinese AI will be such-and-such amount far behind American AI". Progress towards more cost-efficient and compute-efficient models is something everyone expects, but it's very hard to predict how fast it will happen, or whether it'll be steady or bursty, or how it will be implemented.
      • if what we really care about, as Dario claims, is ensuring victory in a potential US-China war, rather than economically immiserating the Chinese, then I think it is also very questionable whether the AI that wins wars is the most "advanced" AI.
        • People like Dario whose bread-and-butter is model performance invariably over-index on model performance, especially on benchmarks. But practical value comes from things besides the model; what tasks you use it for and how effective you are at deploying it.
        • If you want to win a war, does it help to have an AI that can win math Olympiads? I dunno. All things equal, I'd bet on the country that's three years behind on the benchmarks and has a functional military procurement system.
    • we should also note that this is a push from the CEO of a private company who wants to outlaw selling advanced chips to his competitors. there's a self-interested motive here.
  • stuff I looked up while reading about Venice
    • https://en.m.wikipedia.org/wiki/Euboea
      • home of https://en.m.wikipedia.org/wiki/Georgios_Papanikolaou, father of the Pap smear. apparently, as with so many scientists, his wife was his unpaid lab tech.
    • https://en.m.wikipedia.org/wiki/History_of_Thessaloniki
      • https://en.m.wikipedia.org/wiki/History_of_Thessaloniki
      • former Salonika; second largest city in the Byzantine Empire; crucial port
    • https://en.m.wikipedia.org/wiki/Pietro_Loredan_(admiral)
    • https://en.m.wikipedia.org/wiki/Paolo_Veronese
    • https://en.m.wikipedia.org/wiki/Murad_II
• https://explorer.globe.engineer/ AI tool for learning about topics; generates heavily nested, illustrated "intros". I don't love the workflow; it's overwhelming rather than sequential. If I'm trying to learn something in earnest, I need to go through it systematically rather than bopping around.
• https://www.emilybynight.com/p/why-makeup-works
  • frustratingly, not a detailed how-to.
• https://www.sciencedirect.com/science/article/pii/S2213671118304776 I don't know how to compare this to anything, but indeed surface microenvironment makes a big difference for mammalian cell culture & cell differentiation.
  • 3D culture is hard; 2D culture can't get big; solution: tuuuuuubes.
• https://www.nature.com/articles/s41467-024-46314-8  possibly the solution to the high cost of cultured meat is by using edible fungi instead.
• https://nintil.com/why-so-few-women-in-cs-the-google-memo-is-right Jose Luis Ricon back in 2017 on the Damore memo

Maybe it is a controversial take, but I am in favor of publishing all solutions to corrigibility.

If a company fails at coming up with solutions for AGI corrigibility, they wouldn't stop building AGI. Instead, they would proceed with ramping up capabilities and end up with a misaligned AGI that (for all we know) will want to turn universe into paperclips.

Due to instrumental convergence, AGI, whose goals are not explicitly aligned with human goals, is going to engage in a very undesirable behavior by default.

The above toy model assumed that we're picking one signal at a time, and that each such "signal" specifies the intended behavior for all organs simultaneously...

... But you're right that the underlying assumption there was that the set of possible desired behaviors is discrete (i. e., that X in "kidneys do X" is a discrete variable, not a vector of reals). That might've indeed assumed me straight out of the space of reasonable toy models for biological signals, oops.

Thanks!

This could be taken as an argument for using some type of goals selected from learned knowledge [LW · GW] for alignment if possible.

Yeah that’s what I was referring to in the paragraph:

“Well OK,” says the optimist. “…so much the worse for Ev! She didn’t have interpretability, and she didn’t have intelligent supervision after the training has already been running, etc. But we do! Let’s just engineer the AI’s explicit motivation!”

Separately, you also wrote:

we're really training LLMs mostly to have a good world model and to follow instructions

I think I mostly agree with that, but it’s less true of o1 / r1-type stuff than what came before, right? (See: o1 is a bad idea [LW · GW].) Then your reply is: DeepSeek r1 was post-trained for “correctness at any cost”, but it was post-post-trained for “usability”. Even if we’re not concerned about alignment faking [LW · GW] during post-post-training (should we be?), I also have the idea at the back of my mind that future SOTA AI with full-fledged (1-3) loops probably (IMO) won’t be trained in the exact same way than present SOTA AI, just as present SOTA AI is not trained in the exact same way as SOTA AI as recently as like six months ago. Just something to keep in mind.

Anyway, I kinda have three layers of concerns, and this is just discussing one of them. See “Optimist type 2B” in this comment [LW(p) · GW(p)].

Re-reading this a couple days later, I think my §5.1 discussion didn’t quite capture the LLM-scales-to-AGI optimist position. I think there are actually 2½ major versions, and I only directly talked about one in the post. Let me try again:

• Optimist type 1: They make the §5.1.1 argument, imagining that humans will remain in the loop, in a way that’s not substantially different from the present.
  • My pessimistic response: see §5.1.1 in the post.
• Optimist type 2: They make the §5.1.3 argument that our methods are better than Ev’s, because we’re engineering the AI’s explicit desires in a more direct way. And the explicit desire is corrigibility / obedience. And then they also make the §5.1.2 argument that “AIs solving specific technical problems that the human wants them to solve” will not undermine those explicit motivations, despite the (1-3) loop running freely with minimal supervision, because the (1-3) loop will work in vaguely intuitive and predictable ways on the object-level technical question.
  • My pessimistic response has three parts: First, the idea that a full-fledged (1-3) loop will not undermine corrigibility / obedience is as yet untested and at least open to question (as I wrote in §5.1.2). Second, my expectation is that some training and/or algorithm change will happen between now and “AIs that really have the full (1-3) triad”, and that change may well make it less true that we are directly engineering the AI’s explicit desires in the first place—for example, see o1 is a bad idea [LW · GW]. Third, what are the “specific technical problems that the human wants the AI to solve”??
• …Optimist type 2A answers that last question by saying the “specific technical problems that the human wants the AI to solve” is just, whatever random things that people want to happen in the world economy.
  • My pessimistic response: See discussion in §5.1.2 plus What does it take to defend the world against out-of-control AGIs? [LW · GW] Also, competitive dynamics / race-to-the-bottom is working against us, in that AIs with less intrinsic motivation to be obedient / corrigible will wind up making more money and controlling more resources.
• …Optimist type 2B instead answers that last question by saying that the “specific technical problems that the human wants the AI to solve” is alignment research, or more specifically, “figuring out how to make AIs whose motivation is more like CEV or ambitious value learning rather than obedience”.
  • My pessimistic response: The discussion in §5.1.2 becomes relevant in a different way—I think there’s a chicken-and-egg problem where obeying humans does not yield enough philosophical / ethical taste to judge the quality of a proposal for “AI that has philosophical / ethical taste”. (Semi-related: The Case Against AI Control Research [LW · GW].)

We're pretty firmly committed to at least 3. I think whether we have more than that depends on how well we're doing financially and whether the world is still around at that point.

Collins not Gladwell lmao

What do you think of Deepseek's announcement of R1 being evidence that AI capabilities will slow down, because they got a cheaper but not actually better model, and this kills the industry's growth?

https://www.lesswrong.com/posts/ynsjJWTAMhTogLHm6/?commentId=a2y2dta4x38LqKLDX [LW · GW]

• It's possible that we might manage to completely automate the more objective components of research without managing to completely automate the more subjective components of research. That said, we likely want to train wise AI advisors to help us with the more subjective components even if we can't defer to them.

Agree, I expect the handoff to AI agents to be somewhat incremental (AI is like an intern, a new engineer, a research manager, and eventually, a CRO)