[Link] Introducing OpenAI

From their website, it looks like they'll be doing a lot of deep learning research and making the results freely available, which doesn't sound like it would accelerate Friendly AI relative to AI as a whole. I hope they've thought this through.

Edit: It continues to look like their strategy might be counterproductive. [Edited again in response to this.]

The variance of outcomes over the next few decades just went way up.

With Sam Altman (CEO of YCombinator) talking so much about AI safety and risk over the last 2-3 months, I was so sure that he was working out a deal to fund MIRI. I wonder why they decided to create their own non-profit instead.

Although on second thought, they're aiming for different goals. While MIRI is focused on safety once strong AI occurs, OpenAI is trying to actually speed up the research of strong AI.

I left this comment on Hacker News exploring whether "AI for everyone" will be a good thing or not. Interested to hear everyone's thoughts.

It's important to remember the scale we're talking about here. A $1B project (even when considered over its lifetime) in such an explosive field with such prominent backers, would be interpreted as nothing other than a power-grab unless it included a lot of talk about openness (it will still be, but as a less threatening one). Read the interview with Musk and Altman and note how they're talking about sharing data and collaborations. This will include some noticeable short term benefits for the contributors, and pushing for safety, either via including someone from our circles or by a more safety focused mission statement, would impede your efforts at gathering such a strong coalition.

It's easy to moan over civilizational inadequacy and moodily conclude that above shows us how (as a species) we're so obsessed with appropriateness and politics that we will avoid our one opportunity to save ourselves. Sure do some of that, and then think of the actual effects for a few minutes:

If the Value Alignment research program is solvable in the way we all hope it is (complete with a human universal CEV, stable reasoning under self-modification and about other instances of our algorithm) then having lots of implementations running around will be basically the same as distributing the code over lots of computers. If the only problem is that human values won't quite converge: this gives us a physical implementation of the merging algorithm of everyone just doing their own thing and (acausally?) trading with each other.

If we can't quite solve everything that we're hoping for, this does change the strategic picture somewhat. Mainly it seems to push us away from a lot of quick fixes, that will likely seem tempting as we approach the explosion: we can't have a sovereign just run the world like some kind of OS that keeps everyone separate, we'll also be much less likely to make the mistakes of creating CelestAI from Friendship is Optimal, something that optimizes most our goals but has some undesired lock-ins. There are a bunch of variations here, but we seem locked out of strategies that try to achieve some minimum level of the cosmic endowment, while possibly failing at getting a substantial constant fraction of our potential by achieving it at the cost of important values or freedoms.

Whether this is a bad thing or not really depends on how one evaluates two types of risk: (1) the risk of undesired lock-ins from an almost perfect superintelligence getting too much relative power, (2) the risk of bad multi-polar traps. Much of (2) seems solvable by robust cooperation, that we seem to be making good progress on. What keeps spooking me are risks due to consciousness: either mistakenly endowing algorithms with it creating suffering, or evolving to the point that we loose it. These aren't as easily solved by robust cooperation, especially if we don't notice them until it's too late. The real strategic problem right now is that there isn't really anyone we can trust to be unbiased in analyzing the relative dangers of (1) and (2), especially because they pattern-match so well with the ideological split between left and right.

I would argue that this is a terrible, terrible, terrible idea. Once you've got an AI, you could just ask it how to make chemical or biological weapons. Or to hack into various computer systems. Or how to create self-replicating nano-bots. The problem is not every attack necessarily has a defense; or, even if such an defense exists it is typically if much more resource consuming. For example, if you want to protect your buildings against bomb attacks, an individual can just target the buildings with minimal defenses. There are also major problems with how good AIs could be at running scams or manipulating people.

"Musk: I think the best defense against the misuse of AI is to empower as many people as possible to have AI. If everyone has AI powers, then there’s not any one person or a small set of individuals who can have AI superpower. Altman: "I expect that [OpenAI] will [create superintelligent AI], but it will just be open source and useable by everyone <...> Anything the group develops will be available to everyone", "this is probably a multi-decade project <...> there’s all the science fiction stuff, which I think is years off, like The Terminator or something like that. I’m not worried about that any time in the short term"

It's like giving everybody a nuclear reactor and open source knowledge about how to make a bomb. Looks like to result in disaster.

I would like to call this type of thinking "billionaire arrogance" bias. A billionaire thinks that the fact that he is rich is an evidence that he is most clever person in world. But in fact it is evidence that he was lucky before.

My first thought was: "the way to avoid bad outcomes from bioweapons is to give everyone equal access to bioweapons. Oh, wait..." (Not entirely fair, I know.)

Still, since I heard of this, my quarterly donation to MIRI increased by 5% of my income.

You can tell pretty easily how good research in math or physics is. But in AI safety research, you can fund people working on the wrong things for years and never know, which is exactly the problem MIRI is currently crippled by. I think OpenAI plans to get around this problem by avoiding AI safety research altogether and just building AIs instead. That initial approach seems like the best option. Even if they contribute nothing to AI safety in the near-term, they can produce enough solid, measurable results to keep the organization alive and attract the best researchers, which is half the battle.

What troubles me is that OpenAI could set a precedent for AI safety as a political issue, like global warming. You just have to read the comments on the HN article to find that people don't don't think they need any expertise in AI safety to have strong opinions about it. In particular, if Sam Altman and Elon Musk have some false belief about AI safety, who is going to prove it to them? You can't just do an experiment like you can in physics. That may explain why they have gotten this far without being able to give well-thought-out answers on some important questions. What MIRI got right is that AI safety is a research problem, so only the opinions of the experts matter. While OpenAI is still working on ML/AI and producing measurable results, it might work to have the people who happened to be wealthy and influential in charge. But if they hope to contribute to AI safety, they will have to hand over control to the people with the correct opinions, and they can't tell who those people are.

In fact we have too many good AI projects which may result in incompatible versions of AI friendliness and wars between AIs. It often happens in humans history before, most typically when two versions of one religion fight each other (like Shi'ah against Sunni, or different version of Buddhism).

I think it would be much better to concentrate all friendly AI efforts under control of one person or organisation.

Basically we are moving from underinvetmnet stage to overinvetmnet.