Aligned AI is dual use technology

The core argument in this post extrapolates from around 1 or 2 orders of magnitude of wealth to perhaps 40 orders of magnitude.

We're not getting the CEV of humanity even with aligned AI

I agree. I defended almost this exact same thesis too in a recent post [EA · GW].

In keeping with this long tradition of human selfishness, it seems obvious that, if we succeed at aligning AI, the vast, vast majority of its output will get directed toward satisfying the preferences and values of the people controlling it (or possessing leverage over its continued operation) not the "CEV of all humans", let alone the "CEV of all extant moral persons"

I agree with this part too. But I'd add that the people who "control" AIs won't necessarily be the people who build them. Mostly I think AI values will be determined by a variety of forces—including AI developers—but mostly market and regulatory forces outside of the control of AI developers. As I said in another recent post [EA · GW],

I don't think that "the team of humans that succeeds in building the first AGI" will likely be the primary force in the world responsible for shaping the values of future AIs. Instead, I think that (1) there isn't likely to be a "first AGI" in any meaningful sense, and (2) AI values will likely be shaped more by market forces and regulation than the values of AI developers, assuming we solve the technical problems of AI alignment.

In general, companies usually cater to what their customers want, and when they don't do that, they're generally outcompeted by companies who will do what customers want instead. Companies are also heavily constrained by laws and regulations. I think these constraints—market forces and regulation—will apply to AI companies too. Indeed, we have already seen these constraints play a role shaping the commercialization of existing AI products, such as GPT-4. It seems best to assume that this situation will largely persist into the future, and I see no strong reason to think there will be a fundamental discontinuity with the development of AGI.

In the longer term, I expect even "aligned AI" values will evolve outside the bounds of human intentions, but this won't necessarily be bad for humans if we can stay rich—kept afloat by strong property rights and respect for the rule of law—even as our values decline in relative influence with respect to the rest of the universe.

I think the first title was more accurate. There is inherent dual use potential in alignment research.

But that doesn't mean that the good outcome is impossible, or even particularly unlikely. AI developers are quite willing to try to use AI for the benefit of humanity (especially when lots of people are watching them), and governments are happy to issue regulations to that effect (though effectiveness will vary).

Or to put it another way, the outcome is not decided. There are circumstances that make the good outcomes more likely, and there are actions we can take to try and steer the future in that direction.

I feel like this argument fails to engage with the fact that a reasonable fraction of extremely wealthy people have commited high fractions of their money to charity. Even if this is mostly for signaling reasons, it's plausible that similar situations will cause good things to happen in the future.

If it helps clarify: I (and some others) break down the alignment problem into [LW · GW] "being able to steer it at all" and "what to steer it at". This post is about the danger of having the former solved, without the latter being solved well (e.g. through some kind of CEV).

When I see this argument, I think of a parallel argument.

What if medical treatment for aging becomes available, and it's near perfect.  I imagine these huge parties thrown by the wealthy and those friends who are the in group members.  Everyone looks like they just graduated high school, and the people are joyous and care free, knowing they have thousands of years to look forward to.

Meanwhile at the same time, in the same world, people are killing each other with mass produced weapons and entire countries still have senior 'care' centers adjacent to crematoriums.  There are still homeless in the streets.

In such an unjust world, the main thing is to try to make sure you or your family get an invitation to the in group. And definitely don't be late.

With this world model, if you believe this is going to be the outcome, you should be pressuring your country's government not to be late to AGI, better to be early.  The logical strategy if you believe this outcome is the one the universe is going to pick is to support a Manhattan Project for AI.  You would be a very strong accelerationist, beyond even e/acc, since you are not just asking for private groups to be allowed to develop AGI, but for the government to actively invest trillions to develop AGI immediately.  If your home government is too poor, you would be seeking a new citizenship elsewhere.

Note: I do not support the above, I am just saying it appears to be the dominant strategy if you believe this is going to be the outcome.

Yes, people are not strategically altruistic, but they are not strategically selfish, too. Most people do not optimize the world ruthlessly towards selfishly-better outcomes, like, they don't donate to AI notkilleveryoneism and life-extension research, while not being dead is valuable for self-interest. I think if more people become more strategic (using augmentation from AIs, for example), they will become more altruistic and if they create sentient slaves in the meantime they are likely to have "WHAT HAVE I DONE" moment.

(I'm assuming we're talking about singleton outcomes because I think multipolar outcomes are wildly mostly implausible, I think you might not be writing under that assumption? If so the following doesn't apply.)

the vast, vast majority of its output will get directed toward satisfying the preferences and values of the people controlling it

No AGI research org has enough evil to play it that way. Think about what would have to happen. The thing would tell them "you could bring about a utopia and you will be rich beyond your wildest dreams in it, as will everyone", and then all of the engineers and the entire board would have to say "no, just give the cosmic endowment to the shareholders of the company", because if a single one of them blew the whistle the government would take over, and if the government took over a similar amount of implausible evil would have to play out for that to lead to unequal distribution, and an absolutely implausible amount of evil would have to play out for that to not at least lead to an equal distribution over all americans.

And this would have to happen despite the fact that no one who could have done these evil things can even imagine the point of doing them. What the fuck difference does it make to a Californian to have tens of thousands of stars to themselves instead of two or three? The prospect of having even one star to myself mostly makes me feel lonely. I don't know how to be selfish in this scenario.

Extrapolating abstract patterns is fine until you have specific information about the situation we're in, and we do.

While I don't think it's so much about selfishness as such, I think this points at something important, also discussed eg here: The self-unalignment problem [LW · GW]

Tl;dr: The future will always be unevenly distributed.

I don’t have a problem with that.