LessWrong 2.0 Reader

I like Wentworth's toy model, but I want it to have more numbers, so I made some up. This leads me to the opposite conclusion to Wentworth.

I think (2-20%) is pretty sensible for successful intentional scheming of early AGI.

Assume the Phase One Risk is 10%.

Superintelligence is extremely dangerous by (strong) default. It will kill us or at least permanently disempower us, with high probability, unless we solve some technical alignment problems before building it.

Assume the Phase Two Risk is 99%. Also:

• Spending an extra billion dollars on AI control reduces Phase One Risk from 10% to 5%.
• Spending an extra billion dollars on AI alignment reduces Phase Two Risk from 99% to 98%.

The justification for these numbers is that each billion dollars buys us a "dignity point" aka +1 log-odds of survival. This assumes that both research fields are similarly neglected and tractable.

Therefore:

• Baseline: by default we get 9 milli-lightcones.
• If we spend on AI control we get 9.5 milli-lightcones. +0.5 over baseline.
• If we spend on AI alignment we get 18 milli-lightcones, +9 over baseline.

We should therefore spend billions of dollars on both AI control and AI alignment, they are both very cost-efficient. This conclusion is robust to many different assumptions, provided that overall P(Doom) < 100%. So this model is not really a "case against AI control research".

It has been pretty clearly announced to the world by various tech leaders that they are explicitly spending billions of dollars to produce "new minds vastly smarter than any person, which pose double-digit risk of killing everyone on Earth". This pronouncement has not yet incited riots. I feel like discussing whether Anthropic should be on the riot-target-list is a conversation that should happen after the OpenAI/Microsoft, DeepMind/Google, and Chinese datacenters have been burnt to the ground.

Once those datacenters have been reduced to rubble, and the chip fabs also, then you can ask things like, "Now, with the pressure to race gone, will Anthropic proceed in a sufficiently safe way? Should we allow them to continue to exist?" I think that, at this point, one might very well decide that the company should continue to exist with some minimal amount of compute, while the majority of the compute is destroyed. I'm not sure it makes sense to have this conversation while OpenAI and DeepMind remain operational.

Someone just noted that the Review Voting widget might imply that the "Jan 5" end time is meant to be inclusive of a full 24 hours from now, which wasn't the intent, but given that people may have been expecting that, and that the consequences for Not Extending aren't particularly bad, I'm going to give people another 24 hours.

Meanwhile, I said it elsewhere but will say again here: if that there were any posts you were blocked from commenting on that you want to write a review of... I forgot to fix that in the code and it'll take a little while to fix, but meanwhile you can write a shortform comment and DM me (in which I will manually set it to be a review of that post), and/or write a top level post and tag it with 2023 Longform Reviews [? · GW] (which will cause it to appear in the Frontpage Review widget)

People have said that to get a good prompt it's better to have a discussion with a model like o3-mini, o1, or Claude first, and clarify various details about what you are imagining, then give the whole conversation as a prompt to OA Deep Research.

Even if building intelligence requires solving many many problems, preventing that intelligence from killing you may just require solving a single very hard problem. We may go from having no idea to having a very good idea.

I don't know. My view is that we can't be sure of these things.

Downvotes don't (necessarily) mean you broke the rules, per se, just that people think the post is low quality. I skimmed this, and seemed like... a mix of edgy dark politics with poetic obscurantism?

The second half of this post was rather disappointing. You certainly changed my mind on the seemingly orderly progression of learning from simple to harder with your example about chess. This reminds me of an explanation Ruby on Rails creator David Heinemeier Hansson made about intentionally putting himself into a class of motorracing above his (then) abilities^[1].

However there was little detail or actionable advice about how to develop advantages. Such as where to identify situations that are good for learning, least of all from perceived losses or weaknesses. For example:
 

...where we genuinely have all the necessary resources (including internal ones). At the very least, it’s useful to develop the skill of finishing tasks quickly and decisively when nothing is actually preventing us from doing so.

I would be hard-pressed to list any situations where I do have the necessary resources, internal or external, to finish the task but just not the inclination to do so promptly. Clean my bedroom maybe? Certainly if I gave you a list of things found on my bughunt [LW · GW], none of the high-value bugs would fit this criteria.

I also find the "Maximizing the Effective Use of Resources" section feels very much like "How to draw an owl: draw a circle, now draw the rest of the owl". I am aware that often the first idea we have isn't the best.

Except for me... it often is the best. I know because I have a tendency to commit quota filling. What I mean is, the first idea isn't great, but it's the best I have. All the subsequent ideas, even when I use such creativity techniques like "saying no- nos" or removing all internal censors and not allowing myself to feel any embarrassment or shame for posing alternatives - none of them are demonstrably better than the first. In fact they are devolve into an assemblages of words, like a word salad, that seem to exist only for the purpose of ticking the box of "didn't come up with just one idea and use that, thought of other ideas." 

Similarly role-playing often doesn't work for me because if I ask myself something like

“What resources and strategies would Harry James Potter-Evans-Verres / Professor Quirrell use in this situation?” If the answer is obvious, why not apply it?

There is never an obvious answer which is applicable to me. For example, I might well ask myself when on a music video set "How would Stanley Kubrick shoot this?" - and then remember that while he had 6 days at his disposable to a single lateral dolly track with an 18mm lens, and do 50 takes if he wanted. I have 6 hours to shoot the rest of the entire video, only portrait length lenses (55mm and 77mm) and don't have enough track to lay run a long-enough track to shoot it like Kubrick.

I suspect though that this needs to go further upstream - okay, how would Stanley Kubrick get resources to have the luxury of that shot? How would he get the backing of a major studio? Or perhaps more appropriately how would a contemporary music video director like Dave Myers or Hannah Lux Davis get their commissions?

But if I knew that, I'd be doing it. I don't know how they do it. That would involve drawing the rest of the owl.

With this in mind, how can I like Heinemeier Hansson or your hypothetical chess student push myself into higher classes and learn strategies to win?

 

1. ^{^}

   And if his 2013 LeMans results are anything to go by: it worked, his car came 8th overall, and 1st in his class. Overall he beat many ex-Formula One drivers. Including race winner Giancarlo Fisichella (21st), podium placer and future WEC champion Kamui Kobayashi (20th), Karun Chandok and Brendan Hartley (12th) and even Indy 500 winner Alessandro Rossi (23rd)

The private data is, pretty consistently, Anthropic being very similar to OpenAI where it matters the most and failing to mention in private policy-related settings its publicly stated belief on the risk that smarter-than-human AI will kill everyone. 

People worked on capabilities for decades, and never got anywhere until recently, when the hardware caught up, and it was discovered that scaling works unexpectedly well.

If I believed that, then maybe I'd believe (like you seem to do) that there is no strong reason to believe that alignment project cannot be finished successfully before the capabilities project creates an unaligned super-human AI. I'm not saying scaling and hardware improvement have not been important: I'm saying they were not sufficient: algorithmic improvements were quite necessary for the field to arrive at anything like ChatGPT, and at least as early as 2006, there were algorithm improvements that almost everyone in the machine-learning field recognized as breakthrough or important insights. (Someone more knowledgeable about the topic might be able to push the date back into the 1990s or earlier.)

After the publication 19 years ago by Hinton et al of "A Fast Learning Algorithm for Deep Belief Nets", basically all AI researchers recognized it as a breakthrough. Building on it, was AlexNet in 2012, again recognized as an important breakthrough by essentially everyone in the field (and if some people missed it then certainly generational adversarial networks, ResNets and AlphaGo convinced them). AlexNet was the first deep model trained on GPUs, a technique essential for the major breakthrough in 2017 reported in the paper "Attention is all you need".

In contrast, we've seen nothing yet in the field of alignment that is as unambiguously a breakthrough as is the 2006 paper by Hinton et al or 2012's AlexNet or (emphatically) the 2017 paper "Attention is all you need". In fact I suspect that some researchers could tell that the attention mechanism reported by Bahdanau et al in 2015 or the Seq2Seq models reported on by Sutskever et al in 2014 was evidence that deep-learning language models were making solid progress and that a blockbuster insight like "attention is all you need" is probably only a few years away.

The reason I believe it is very unlikely for the alignment research project to succeed before AI kills us all is that in machine learning or the deep-learning subfield of machine learning, what was recognized by essentially everyone in the field as a minor or major breakthrough has occurred every few years. Many of these breakthrough rely on earlier breakthroughs (i.e., it is very unlikely for the sucessive breakthrough to have occurred if the earlier breakthrough had not been disseminated to the community of researcher). During this time, despite very talented people working on it, there has been zero results in alignment research that the entire field of alignment researchers would consider a breakthrough. That does not mean it is impossible for the alignment project to be finished in time, but it does IMO make it critical for the alignment project to be prosecuted in such a way that it does not inadvertently assist the capabilities project.

Yes, much more money has been spent on capability research the last 20 years than on alignment research, but money doesn't help all that much to speed up research in which to have any hope of solving the problem, the researchers need insight X or X2, and to have any hope of arriving at insight X, they need insights Y and Y2, and to have much hope at all of arriving at Y, they need insight Z.

Thank you, I've always been curious about this point of view because a lot of people have a similar view to yours.

I do think that alignment success is the most likely avenue, but my argument doesn't require this assumption.

Your view isn't just that "alternative paths are more likely to succeed than alignment," but that "alternative paths are so much more likely to succeed than alignment, that the marginal capabilities increase caused by alignment research (or at least Anthropic), makes them unworthwhile."

To believe that alignment is that hopeless, there should be stronger proof than "we tried it for 22 years, and the prior probability of the threshold being between 22 years and 23 years is low." That argument can easily be turned around to argue why more alignment research is equally unlikely to cause harm (and why Anthropic is unlikely to cause harm). I also think multiplying funding can multiply progress (e.g. 4x funding $\approx$ 2x duration).

If you really want a singleton controlling the whole world (which I don't agree with), your most plausible path would be for most people to see AI risk as a "desperate" problem, and for governments under desperation to agree on a worldwide military which swears to preserve civilian power structures within each country.^[1]

Otherwise, the fact that no country took over the world during the last centuries strongly suggests that no country will in the next few years, and this feels more solid than your argument that "no one figured out alignment in the last 22 years, so no one will in the next few years."

1. ^{^}

   Out of curiosity, would you agree with this being the most plausible path, even if you disagree with the rest of my argument?