LessWrong 2.0 Reader

I think many current goals of AI governance might be actively harmful, because they shift control over AI from the labs to USG.

This note doesn’t include any arguments, but I’m registering this opinion now. For a quick window into my beliefs, I think that labs will be increasing keen to slow scaling, and USG will be increasingly keen to accelerate scaling.

Oh, I see, one could reasonably misinterpret the bullet points in my original comment as being about "the way many people have been describing the situation" rather than "major claims in the podcast". Sorry for the ambiguity.

Eliezer thinks (as do I) that technical progress in alignment is hopeless without first improving the pool of prospective human alignment researchers (e.g., via human cognitive augmentation or many generations of natural selection).

Oh, I maybe flipped the sign on what you meant to be saying.

I went through a bunch of similar thoughts before writing the self-unalignment problem [LW · GW]. When we talked about this many years ago with Paul my impression was this is actually somewhat cruxy and we disagree about self-unalignment  - where my mental image is if you start with an incoherent bundle of self-conflicted values, and you plug this into IDA-like dynamic, my intuition is you can end up in arbitrary places, including very bad. (Also cf. the part of Scott's review of What We Owe To Future where he is worried that in a philosophy game, a smart moral philosopher can extrapolate his values to 'I have to have my eyes pecked out by angry seagulls or something' and hence does not want to play the game. AIs will likely be more powerful in this game than Will MacAskill)

My current position is we still don't have a good answer, I don't trust the response 'we can just assume the problem away', and also the response 'this is just another problem which you can delegate to future systems'. On the other existing AIs already seem doing a lot of value extrapolation and the results sometimes seem surprisingly sane, so, maybe we will get lucky, or larger part of morality is convergent - but it's worth noting these value-extrapolating AIs are not necessarily what AI labs want or traditional alignment program aims for.

Any of the many nonprofits, academic research groups, or alignment teams within AI labs. You don't have to bet on a specific research group to decide that it's worth betting on the ecosystem as a whole.

There's also a sizeable contingent that thinks none of the current work is promising, and that therefore buying a little time is value mainly insofar as it opens the possibility of buying a lot of time. Under this perspective, that still bottoms out in technical research progress eventually, even if, in the most pessimistic case, that progress has to route through future researchers who are cognitively enhanced.

To be clear these are just patterns of claims made by Slimepriestess in the linked podcast, and I have no corroborating evidence. But for example at 2:06:00 in the video she says:

At least as far as, like, Ziz et al goes, I don't think that's a remotely accurate description of... Like, there's no organization, there's no centralization, it's not like we have Ziz on, on speed dial and ask her what to do every day. Like, we're just a bunch of anarchist trans leftists that are, like, trying to exist in Current Year

With other variations of the same claims elsewhere in the video.

Question for Ben:

Are you inviting us to engage with the object level argument, or are you drawing attention to the existence of this argument from a not-obviously-unreasonable-source as a phenomenon we are responsible for (and asking us to update on that basis)?

On my read, he’s not saying anything new (concerns around military application are why ‘we’ mostly didn’t start going to the government until ~2-3 years ago), but that he’s saying it, while knowing enough to paint a reasonable-even-to-me picture of How This Thing Is Going, is the real tragedy.

I think the government can speed up alignment more than the government can speed up capabilities, assuming it starts to care much more about both. Why?

AI safety spending [? · GW] is only $0.1 billion while AI capabilities spending is $200 billion. AI safety spending can easily increase by many orders of magnitude, but AI capabilities spending cannot since it already rivals the US military budget.

Also, would you still agree with the "Statement on AI Inconsistency," or disagree with it too?

Statement on AI Inconsistency (v1.0us):

1: ASI threatens the US (and NATO) as much as all military threats combined. Why does the US spend $800 billion/year on its military but less than $0.1 billion/year on AI alignment/safety?

2: ASI being equally dangerous isn't an extreme opinion: the median superforecaster sees a 2.1% chance of an AI catastrophe (killing 1 in 10 people), the median AI expert sees 5%-12%, other experts see 5%, and the general public sees 5%. To justify 8000 times less spending, you must be 99.999% sure of no AI catastrophe, and thus 99.95% sure that you won't realize you were wrong and the majority of experts were right (if you studied the disagreement further).

3: “But military spending isn't just for protecting NATO, it protects other countries far more likely to be invaded.” Even they are not 8000 times less likely to be attacked by ASI. US foreign aid—including Ukrainian aid—is only $100 billion/year, so protecting them can't be the real reason for military spending.

4: The real reason for the 8000fold difference is habit, habit, and habit. Foreign invasion concerns have decreased decade by decade, and ASI concerns have increased year by year, but budgets remained within the status quo, causing a massive inconsistency between belief and behaviour.

5: Do not let humanity's story be so heartbreaking.

(Copied from my post A better “Statement on AI Risk?” [LW · GW])

He appears to be arguing against a thing, while simultaneously criticizing people; but I appreciate that he seems to do it in ways that are not purely negative, also mentioning times things have gone relatively well (specifically, updating on evidence that folks here aren't uniquely correct), even if it's not enough to make the rest of his points not a criticism.

I entirely agree with his criticism of the strategy he's criticizing. I do think there are more obviously tenable approaches than the "just build it yourself lol" approach or "just don't let anyone build it lol" approach, such as "just figure out why things suck as quickly as possible by making progress on thousand year old open questions in philosophy that science has some grip on but has not resolved". I mean, actually I'm not highly optimistic, but it seems quite plausible that what's most promising is just rushing to do the actual research of figuring out how make constructive and friendly coordination more possible or even actually reliably happen, especially between highly different beings like humans and AIs, especially given the real world we actually have now where things suck and that doesn't happen.

Specifically, institutions are dying and have been for a while, and the people who think they're going to set up new institutions don't seem to be competent enough to pull it off, in most cases. I have the impression that institutions would be dying even without anyone specifically wanting to kill them, but that also seems to be a thing that's happening. Solving this is stuff like traditional politics or economics or etc, from a perspective of something like "human flourishing, eg oneself".

Specifically, figuring out how to technically ensure that the network of pressures which keeps humanity very vaguely sane also integrates with AIs in a way that keeps them in touch with us and inclined to help us keep up and participating/actualizing our various individuals' and groups'/cultures' preferences in society as things get crazier, seems worth doing.