We don't need to figure out this problem, we can just implement CEV without ever having a good model of what "human values" are.

Why would you think that the CEV even exists?

Humans aren't all required to converge to the same volition, there's no particularly defensible way of resolving any real differences, and even finding any given person's individual volition may be arbitrarily path-dependent.

The vast majority of the utility you have to gain is from {getting a utopia rather than everyone-dying-forever}, rather than {making sure you get the right utopia}.

Whether something is a utopia or a dystopia is a matter of opinion. Some people's "utopias" may be worse than death from other people's point of view.

In fact I can name a lot of candidates whose utopias might be pretty damned ugly from my point of view. So many that it's entirely possible that if you used a majoritarian method to find the "CEV", the only thing that would prevent a dystopia would be that there are so many competing dystopic "utopias" that none of them would get a majority.

Expected utility maximization seems to fully cover this. More general models aren't particularly useful to saving the world.

Most actually implementable agents probably don't have coherent utility functions, and/or have utility functions that can't be computed even approximately over a complete state-of-the-world. And even if you can compute your utility over a single state-of-the-world, that doesn't imply that you can do anything remotely close to computing a course of action that will maximize it.

we can just implement CEV [? · GW]

So we have a detailed, mechanistic understanding of CEV now?

It's not that hard to build an AI that kills everyone: you just need to solve [some problems] and combine the solutions. Considering how easy it is compared to what you thought, you should increase your P(doom) / shorten your timelines.

It's not that hard to build an AI that saves everyone: you just need to solve [some problems] and combine the solutions. Considering how easy it is compared to what you thought, you should decrease your P(doom) / shorten your timelines.

They do a value-handshake and kill everyone together.

Any two AIs unaligned with humanity are very unlikely to be also aligned with each other, and would have no especial reason to coordinate with other unaligned AIs over humans (or uploads, or partially aligned neuromorphic AGIs, etc)

The vast majority of the utility you have to gain is from {getting a utopia rather than everyone-dying-forever}, rather than {making sure you get the right utopia}.

Whose utopia? For example: some people's utopia may be one where they create immense quantity of descendants of various forms and allocate resources to them rather than existing humans. I also suspect that Hamas's utopia is not mine. This idea that the distribution of future scenarios is bimodal around "we all die" or "perfect egalitarian utopia" is dangerously oversimplistic.

Excellent post, big upvote.

We need more discussion of the cognitive biases affecting the alignment debate. The eye needs to see its own flaws in order to make progress fast enough to survive AGI.

It's curious that I disagree with you on many particulars, but agree with you on the main important points.

Outside-view is overrated

Agreed on the main logic: it's tough to guess how many opinions are just duplicates. Safer? Maybe, but we also overvalue "having your own opinion" even if it's not informed and well-thought-out.

Arguments about P(doom) are filtered for nonhazardousness

Agreed, and it's important. But I think the cause is mostly different: most people with a low estimate are succumbing to motivated reasoning. They very much want to believe they and everything they love won't die. So any alternative feels better, and they wind up believing it.

Confusion about the problem often leads to useless research

Very much agreed. This is also true of every other scientific and engineering field I know about. People like doing research more than they like figuring out what research is important. To your specific points:

What are human values?

Doesn't matter. The only human values that matter are the ones of the people in charge of the first ASI.

Aligned to whom?

I think this matters an awful lot, because there are sadists and sociopaths in positions of power. As long as the team in charge of ASI has a positive empathy - sadism balance, we'll like their utopia just fine, at least after they get time to think it through.

What does it mean for something to be an optimizer?

Optimization doesn't matter. What matters is pursuing goals more competently than humans can.

Okay, unaligned ASI would kill everyone, but how?

Agreed that it doesn't matter. You can keep a smarter thing contained or controlled for a while, but eventually it will outsmart you and do whatever it wants.

What about multipolar scenarios?

Disagree. Value handshakes aren't practically possible. They go to war and you die as collatoral damage. More importantly, it doesn't matter which of those is right. Multipolar scenarios are worse not better.

What counts as AGI, and when do we achieve that?

What counts is the thing that can outsmart you. When we achieve it does matter, but only a little for time to plan and prepare. How we achieve that matters a lot, because technical alignment plans are specific to AGI designs.

Thanks for the post on an important topic!

Note that beyond not-being-mentioned, such arguments are also anthropically filtered against: in worlds where such arguments have been out there for longer, we died a lot quicker, so we’re not there to observe those arguments having been made.

This anthropic analysis doesn't take into account past observers (see this post [LW · GW]).