AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now

Downvoted because I view some of the suggested strategies as counterproductive. Specifically, I'm afraid of people flailing [LW · GW]. I'd be much more comfortable if there was a bolded paragraph saying something like the following:

Beware of flailing [LW · GW] and second-order effects and the unilateralist's curse [? · GW]. It is very easy to end up doing harm with the intention to do good, e.g. by sharing bad arguments for alignment, polarizing the issue, etc.

To give specific examples illustrating this (which may also be good to include and/or edit the post):

• I believe tweets like this are much better (and net positive) then the tweet you give as an example. Sharing anything less then the strongest argument can be actively bad to the extent it immunizes people against the actually good reasons to be concerned.
• Most forms of civil disobedience seems actively harmful to me. Activating the tribal instincts of more mainstream ML researchers, causing them to hate the alignment community, would be pretty bad in my opinion. Protesting in the streets seems fine, protesting by OpenAI hq does not.

Don't have time to write more. For more info see this twitter exchange I had with the author, though I could share more thoughts and models my main point is be careful, taking action is fine, and don't fall into the analysis-paralysis of some rationalists, but don't make everything worse.

The above contains links to a lot of arguments, but it does not develop almost any argument fully.

In lieu of responding to them all, I will address one sentence.

Here's a claim you make about likely future timelines: "GPT-4 + curious (but ultimately reckless) academics -> more efficient AI -> next generation foundation model AI (which I’ll call NextAI[10] for short)"

There are two links in "ultimately reckless" go to two papers, which must be to support the claim that researchers are reckless, or the claim that we will quickly get more efficient next level AI, presumably.

One paper uses GPT-4 to do neural architecture search over CIFAR-10, or CIFAR-100, or ImageNet16-120. These are all pretty tiny datasets... which is why people do NAS over them, because otherwise it would be insanely expensive. Even doing this kind of thing with a small LM would be really tough. Furthermore, GPT-4 looks like it's giving you the same approximate OOM improvements that other NAS techniques do, which isn't that great. Like I could write more here, but this is basically miles away from being used to help with actual LLMs, let alone recursive self improvement -- it's on toy datasets used for NAS because they are small.

The other uses GPT-4 augmented with planners to produce plans. "Plans" means classical planing problems like "Given a set of piles of blocks on a table, a robot is tasked with rearranging them into a specified target configuration while obeying the laws of physics." This kind of thing -- with definite ontology, non-vague world, etc -- has been in AI since the 70s and the ability of GPT-4 to work with a classical planner is basically entirely unrelated to... lengthy, contextually vague problems necessary to take over the world. Like, it isn't just that "Take over the world" is too vague and onto logically unspecified for this kind of treatment, "Unload my car, don't break anything" is also too vague and unspecified. In any event, I don't think it's remotely worrisome, and in any event extremely unclear how it leads to more efficient NextAI.

Anyhow, neither of these papers lead me to think that researchers are reckless. I'm really unsure how they're suppose to support that claim, or the claim that we'll get NextAI quickly, or even what NextAI would be.

Anyhow, if the above sentence is representative, the above looks like a gish gallop -- responding at length to many of these points would take 50x the space spent here.

LessWrong:

A post about all the reasons AGI will kill us [LW · GW]: No. 1 all time highest karma (827 on 467 votes; +1.77 karma/vote)
A post about containment strategy for AGI [LW · GW]: 7th all time highest karma (609 on 308 votes; +1.98 karma/vote)
A post about us all basically being 100% dead from AGI [LW · GW]: 52nd all time highest karma (334 on 343 votes; +0.97 karma/vote, a bit more controversial)

Also LessWrong:

A post about actually doing something about containing the threat from AGI and not dying [this one]: downvoted to oblivion (-5 karma within an hour; currently 13 karma on 24 votes; +0.54 karma/vote)




My read: y'all are so allergic [LW · GW] to anything considered remotely political (even though this should really not be a mater of polarisation - it's about survival above all else!) that you'd rather just lie down and be paperclipped [? · GW] than actually do anything to prevent it happening. I'm done [LW · GW].