Why a Human (Or Group of Humans) Might Create UnFriendly AI Halfway On Purpose

All of this being said, what do you think SIAI should do differently? FAI has to be built. It has to be built by a team of humans. SIAI will do its best to make sure those humans are aware of problems like this, and that's where I'd say it would be nice to know some concrete steps they could take to mitigate the risk. But aside form that, are there any other alternatives?

Too many people--at least, too many writers of the kind of fiction where the villain turns out to be an all-right guy in the end--seem to believe that if someone is the hero of their own story and genuinely believes they're doing the right thing, they can't really be evil. But you know who was the hero of his own story and genuinely believed he was doing the right thing? Hitler. He believed he was saving the world from the Jews and promoting the greatness of the German volk.

Was this bit absolutely necessary? It comes across as almost a cliched invocation of the Nazis. This would probably be better received if one had a different example or no example at all.

This is on the whole a minor criticism more about presentation than anything else. Your general points seems worth-while.

The fact that Eliezer has stated his intention to have the Singularity Institute create FOOM-capable AI doesn't worry me much, because I think the SIAI is highly unlikely to succeed at that.

That's largely my attitude. The Fellowship of the AI probably won't lead to a scenaro like Gollum's final moments.

"what can we do to prevent some small group of humans (the SIAI, a secret conspiracy of billionaires, a secret conspiracy of Google employees, whoever) from steering a first-mover scenario in a direction that's beneficial to themselves and perhaps their blood relatives, but harmful to the rest of humanity?"

Actually, if they managed to do that, then they have managed to build an FAI. The large(/largest ?) risk some (as SIAI I think, but I am not an expert) is that they think they are building an FAI (or perhaps a too weak AI to be really dangerous) but that they are misstaken in that assumption. In reality they have been building an uFAI that takes over the world and humanity as a whole is doomed, including a small minority of humanity that possibly the AI was supposed to be friendly to.

There seems to be three different problems here. To analyse how dangerous AIs in general are. If dangerous, how can one make an FAI, that is an AGI that is at least beneficial to some. And then if an FAI can be built, to whom should it be friendly. As I interprete your post you are discussing the third question and dangers related to that while hypothetically assuming that the small group building the AGI has managed to solve the second question? If so, you are not really discussing why some would build an uFAI half-way by purpose but why some would build an FAI that is unfriendly to most humans?

I don't want to find out what I'd do with unlimited power.

Well, I think the first thing you should do is attempt to make a substantial effort to not accidentally kill yourself when you use your powers, as the person with the outcome pump does in the linked story.

But the next thing you need to do (while bearing in mind not killing yourself) is to gain at least some ability to predict the behavior of the unlimited power. This appears to run contrary to a literal interpretation of your point. I DO want to find out at least some of what I would do with unlimited power, but I want to know that before ACTUALLY having unlimited power.

For instance, when you hand a person a gun, they need to be able to predict what the gun is going to do to be able to use it without injury. If they know nothing at all, they might do the possibly fatal combination of looking in the hole while pulling the trigger. If they are more familiar with guns, they might still injure themselves if the recoil breaks their bones when they fire it holding it the wrong way. I'm actually not familiar with guns personally, so there are probably plenty of other safety measures that haven't even occurred to me.

And as you point out, a flaw in our current method of elite selection is that our elites aren't as predictable as we like, and that makes them dangerous. Maybe they'll be good for humanity... maybe they'll turn us all into slaves. Once again, the unpredictability is dangerous.

And we can't just make ourselves the elites, because we aren't necessarily safe either.

In fact, realistically, saying "I don't want to find out what I'd do with unlimited power." might ITSELF a corruption sign. It means you can truthfully say "I don't think I would do anything evil." because you haven't thought about it. It still leaves you eligible for power later. It's easy to delude yourself that you're really a good person if you don't think about the specifics.

One possible next step might be to imagine a publicly known computer algorithm which can predict what humans will do given substantial amounts of new power. The algorithm doesn't need to be generally self improving, or an oracle, or an AI. It just needs to be able to make that specific prediction relatively reliably, and ideally in a way that it can defeat people trying to fool it, and in a way that is openly verifiable to everyone. (For obvious reasons, having this kind of determination be private doesn't work, or the current elites can just write it to support themselves.)

Note, problems doing that kind of system wrong have certainly come up.

But if done correctly, we could then use that to evaluate "Would this humans be likely to go cacklingly mad with power in a bad way?" If so, we could try to not let them have power. If EVERYONE is predicted to go cacklingly mad in a bad way, then we can update based on that as well.

What's weird is that if this is a good idea, it means that realistically, I would have to be supporting some sort of precrimish system, where elites would have to live with things like "The computer predicts that you are too likely to abuse your power at this time, and even though you haven't done that YET... we need to remove you from your position of power for everyone's safety."

But, if we are worried about elites who haven't done anything wrong yet abusing their power later, I don't see any way around solving it in advance that wouldn't restrict elites in some way.

Of course, this brings up a FURTHER problem, which is that in a lot of cases, humans aren't even as capable of choosing their own elites. So even if we had a solid algorithm of judging who would be a good leader and who would be a bad leader, and even if people generally accepted this, we can't then take the candidates and simply install them without significant conflicts.

Unfortunately, realistically this post is getting substantially above the length at which I feel competent writing/thinking, so I should get some evaluation on it before continuing.

This relates to a theoretical question which is more interesting to me: are "individual extrapolated volitions" in/compatible with a "collective extrapolated volition" (a la CEV)? Are there "natural classes" of individuals who, in collection, form their own coherent extrapolated volition which is compatible among themselves, but incompatible with others extrapolated volitions, e.g. men vs. women, smart people vs. dumb people, individualists vs. collectivists, all of humanity happens to be compatible, or every person individually is incompatible with everyone else...? In other words, how much natural coherence is there among individual extrapolated volitions, and how much would CEV require to sacrifice individual extrapolated volitions for the collective extrapolated volition?

If it happened to work out that the entire collective of humanity could have a coherent extrapolated volition in which no individual extrapolated volition is sacrificed, that would be perfect and would relieve these uncomfortable questions. If that's not the case - should be proceed with CEV anyway? Or is it possible we may want to favor one group over another?

what can we do to prevent some small group of humans (the SIAI, a secret conspiracy of billionaires, a secret conspiracy of Google employees, whoever) from steering a first-mover scenario in a direction that's beneficial to themselves and perhaps their blood relatives, but harmful to the rest of humanity?

Consumers can support those organisations with the most convincing promises and strategy - while boycotting those who they don't want to see in charge.

A man naturally loves his children better than his nephews, his nephews better than his cousins, his cousins better than strangers, where every thing else is equal." And we care more about ourselves than any single other individual on the planet (even if we might sacrifice ourselves for two brothers or eight cousins.)

As a nitpick, that isn't necessarily true. People do sacrifice themselves for other individuals. Sometimes it even makes evolutionary sense - if they have a terminal illness, for instance.

One thing that might help is to make the decisions before you have the power--that is, to "lock in" the AI's choices on the basis of what prosocial-current-you (or the EV thereof) wants, before you actually have a superintelligence and your subconscious activates powerful-selfish-you. Then by the time you can be corrupted by power, it's too late to change anything. Coding the AI before launching it might be enough, especially if other people are watching and will punish cackling madness. On the other hand, being alone coding the AI would be a position of great power at least by proxy, and so might still be open to corruption.

You didn't talk about counterexamples: for example, lots of nations like the US have few corrupt politicians. There's an availability effect here. Philanthropists are boring, bad boys are interesting.

I actually agree this is a big danger, but I'm not sure it's unsolvable with the right individuals. Does power corrupt women, or lifelong meditators, or hopelessly geeky folks, or people whose entire career has consisted of researching this sort of thing, or people who fall into all four categories?

WRT the Stanford Prison Experiment: I've hung out with some elite university undergraduates and mostly they seem full of youthful foolishness and much less reflective than median Less Wrong user. All your other examples feature even worse candidates where power corruption is concerned. (Also, iirc the portion resisting authority in the Milgram experiment was decently large.)

And you haven't discussed the problem of corruption when an entire group, not self-selected on the basis of power seeking tendencies, receives committee style power. It seems like this could significantly curtail power-seeking tendencies.