Heading off a near-term AGI arms race

That's about 1500 man-years.

1.5e3 is not large compared to the total number of man-years spent on AI, which is probably more like 1.5e5. There are probably 1e4 researchers in AI-related fields, so we're producing at least 1e4 man-years of effort per year. It may be that private sector projects are more promising/threatening than academic projects, but it seems implausible that this would be a 100x effect.

By "heading off" I think we should be clear that we are referring to go stones, not some form of sabotage. How can we ensure there will be better safety incentives over the next few decades? That sort of thing.

It's possible that, if the feasibility just isn't there yet no matter the funding, it'll turn out like nanotechnology - funding for molecule-sized robots that gets spent on chemistry instead. (I wonder what the "instead" would be in this case.)

I'm curious at what likelihood of AGI imminence SI or LessWrong readers would think it was a good idea to switch over to an ecoterrorist strategy. The day before the badly vetted machine is turned on is probably a good day to set the charges to blow during the night shift. The funding of this project is probably too early.

Do people think SI should be devoting more of its time and resources to corporate espionage and/or sabotage if unfriendly AI is the most pressing existential threat?

Convince programmers to refuse to work on risky AGI projects:

Please provide constructive criticism.

We're in an era where the people required to make AGI happen are in so much demand that if they refused to work on an AGI that wasn't safe, they'd still have plenty of jobs left to choose from. You could convince programmers to adopt a policy of refusing to work on unsafe AGI. These specifics would be required:

• Make sure that programmers at all levels have a good way to determine whether the AGI they're working on has proper safety mechanisms in place. Sometimes employees get such a small view of their job and will be told such confident fluff by management, that they have no idea what is going on. I am not qualified to do this, but if someone reading this post is, it might be very important if you write some guidelines for how programmers can tell whether the AGI they're working on might be unsafe from within their employment position. It may be more effective to give them a confidential hotline. Things can get complicated, both in programming, and in corporate culture, and employees may need help sorting out what's going on.

• You could create resources to help programmers organize a strike or programmer's walk. Things like: An anonymous web interface where people interested in striking can post their intent - this would help momentum build. A place for people to post stories about how they took action against unsafe AI projects. They might not know how to organize otherwise (especially in large projects) or might need the inspiration to get moving.

• If a union is formed around technological safety, the union could make demands that outside agencies must be allowed to check on the project, and that the company must be forthcoming with all safety related information.

On the feasibility of getting through to the programmers

See Also "Sabotage would not work"

The company to get all of this right will be the first two trillion dollar company.

Is there any way to reverse this trend in public perception?

You don't supply a counter-argument. Do you disagree - or are you looking for a way to create a mass delusion?

Help good guys beat the race:

Please provide constructive criticism.

An open source project might prevent this problem, not because having an open source AGI is safe, but because 1.) open source projects are open, so anybody can influence it, including people who are knowledgeable about risks and 2.) the people involved in open source projects probably tend to have a pretty strong philanthropic streak and they're more likely to listen to the dangers than a risk-taking capitalist. The reason it may stop them is this: If an open source project gets there first, it won't be seen as a juicy target to capitalists anymore. It will be a niche that's already filled for free. If they wanted to make an AGI they'd have to make one that was so much better than the existing one that it makes sense to charge, or fail at business.

Making an open source AGI, in order to compete with a business might cause the open source programmers to rush. However, imagine what would happen if customers got the following messages around the time that the closed source AGI was going to be released: If you wait a while longer, an AGI will come out for free, plus, the open source AGI is going to be thoroughly tested to discover dangers before you run it. The closed source AGI is very risky." That would deter a lot of people from buying, which would at least reduce the exposure to the closed source AGI - and the open source group would not have to release the AGI until they had tested thoroughly. If, during the course of their tests, they discover hideous risks, these could serve as warnings about AGI in general, make those risks feel real, and prevent people from running risky AGIs. Assuming that the open source project had good PR and advertising / public education campaigns.

Why open source might have a competitive advantage:

• Open source people may be more willing to merge, especially if our future depends on it, whereas companies tend to behave in self-interested ways and work separately for the most part. They're already divided, so open source could conquer them.

• I was told by a Microsoft employee that he thought Linux would eventually win. Considering the influence that corporate culture can have on software design (the rushing to make deadlines which results in code debt), I don't disagree with him one bit. One concept here that could turn out to be really important is that any company working on AGI that does not put safety first may also have a short-term culture, which means they might actually take much, much longer to release their project, or to have recalls that force them to start over, than an organization of programmers that is allowed to do things the right way. An open source project has that potential benefit on it's side.

• People who work on open source projects are probably more altruistic. They may be able to be persuaded that working on AI is so much more important to the future of humanity that they jump out of their current open source project and get involved.

  For those three reasons, I think an open source project has a good chance of getting there first.

  The obvious argument against this would be "An open source AGI!!! Won't bad people write their own versions?" My counter argument is: In a world where pirates routinely crack software within days of it coming out, and corporate espionage is a real possibility for a target this juicy, what makes you think the code won't get stolen THE VERY NEXT DAY? In that event, the best tool to save us from rogue AGIs would be if every open source programmer has access to editable copies of a friendly AGI, don't you think?

  An even faster solution: How just the threat of having to compete with a massive open source project may stop them.

  See Also "Sabotage would not work"

Create public relations nightmare for anyone producing risky AGI:

Please provide constructive criticism.

One powerful way to get people thinking about safety is if clever ways are invented to shout from the rooftops that this could be dangerous and present the message in a way that most people will grok. If everybody is familiar enough with how dangerous it could be, then funding an AGI project without a safety plan in place would be a PR disaster for the companies doing it. That would put a lot of pressure on them to put safeties into place. This wouldn't need to cost a lot. Sometimes small, clever acts can get a huge amount of attention. For instance, Improv Everywhere. Imagine what would happen if a few hundred volunteers gathered in every major city on the same day and did this:

Dressed in business suits, covered in happy face stickers, they start asking strangers "Have you seen my robot anywhere? It was trying to make me happy and it covered me in these stickers. What do you think it will do next?" They'll probably say "I don't know." And you could respond "Yeah, the companies currently building robots like this one have no idea what they're going to do either... you might need this (the volunteer hands them a pamphlet on AGI safety) "I've got to keep an eye out for my robot." (Then they go away and do it again.) The suits are important because they would contrast the happy face stickers such that the message was more likely to be interpreted as "Being covered in happy face stickers is very embarrassing and therefore really bad" as opposed to just seeming silly, gives the person an upper-class look that supports the idea that they own a robot, and also makes strangers more likely to interpret somebody covered in happy face stickers as sane.

Another idea: If the internet "wears" black for a time, like they did for SOPA, that could work to get attention. Surely intelligent people in tech companies would grok the threat that unfriendly AGI poses to humanity, and they're in a unique position to warn about it. It's not like the people on this forum have no connections to the technology world...

Even if millions of dollars aren't available to spend on a public advertising campaign, if enough people try enough clever ways to get everyone's attention (internet memes on youtube for instance) then something will probably get through. When large masses of people fear something, they tend to push for regulation and tend to scare investors off. They can also be given specific actions to take.

See Also "Sabotage would not work"

Even Faster Solution:

Survey a bunch of open source people asking them if they'd switch to working on friendly AGI in the event that an AGI project started without enough safety, or get their signatures. Surely the thousands of programmers now working on projects like Firefox and Open Office, who clearly have an altruistic bent as they are working for free, will see that it is more important to prevent unfriendly AGI than to make sure the next version of these smaller projects are released on time.

If we can honestly say to these companies "If you try to start an AGI project without thorough safety precautions, 100,000 programmers have said they'll rise up against you and make a FREE AGI to compete with yours that's safer." What they will hear is "We'll be put out of business!" Assuming they believe the survey results are accurate, and that the plan for the project is feasible, they will be will forced to take safety precautions in order to protect their investments.

Just that ONE piece of information, if communicated right, could transform a risky AGI arms race into a much safer one.

Here's a multiplier effect: If you're asking a bunch of programmers anyway, you may as well ask them if they'd be willing to make a monetary contribution toward the friendly AGI project for x, y, or z strategies/prerequisites. Programmers tend to make a lot of money.

How this could postpone an arms race:

If the bar is set high enough (which can be done by asking the programmers all the conditions the AGI would have to meet, without which they'd deem it "risky" and get involved), you may postpone the arms race quite some time while companies regroup and try to figure out a strategy to compete with these guys. This assumes, also, that it becomes common knowledge among the people who would start a risky AGI project that this pact among open source programmers exists.

Other pieces that would be required to make this idea work:

• The open source programmers would have to be given a message about the company who has started an AGI project that gets them to understand the gravity of the problem. They're probably more likely to grok it just because they're programmers and they're the right sort to have already thought about this sort of thing, but we'd want to make sure the message is really clear. This could be a little tricky due to laws about libel.

• Companies may not believe the open source programmers are serious about switching. This is easily resolved by creating a wall where they can put remarks about why they think competing with unsafe AGI is an important project. Surely they will put convincing things like "I love using Linux, but if a risky AGI destroys enough, that won't matter anymore."

• Have a way to contact the companies who are starting risky AGI projects in order to send them the message that they're provoking a loss of their investment. Asking the volunteer programmers to email them a threat to compete with them, (the way that a lot of activist organizations ask their members to tell companies they won't put up with them destroying the environment), would be one way. This requires previously having collected the email addresses of the programmers so that they can be asked to email the company. It also requires getting the email addresses of important people at the company, but that's not hard if you know how to look up who owns a website.

• Ensure the open source programmers in question are knowledgeable enough about the dangers of AGI to want high standards for safety. They may need to be educated about this in order to make informed decisions. Providing compelling examples and a clearly written list of safety standards are both important or not everyone will be on the same page, and there won't be something solid causing them to consciously confront their biases and doubts.

• Have a way to ask all (or a significant number of) the people interested in doing open source programming whether they'd switch. This does two things: 1. You get your survey results / signatures. 2. You get them thinking about it as a cause. Getting them thinking about it and discussing it, if they aren't already, would catalyze more of them to decide to work on it, assuming they use rational thought processes. After all, what's more important? Failing to upgrade Firefox, or having to live with unfriendly AGI? Just asking enough people the question would start a snowball effect that would attract people to the cause.

• Retain some contact method that allows you to inform them when a risky AGI project starts. Note: Sending out mass-mailings is really tricky because spam filters are set to "paranoid" - it might take a person experienced with this to get the email campaign to go through.

• When it is time for them to switch to AGI, they'll need to be convinced that it ACTUALLY is, in fact, that time. There will be inertia to overcome, so you'd need to present compelling reasons to believe that they should change over immediately.

• The idea for the open source AGI must sound feasible in order for it to be convincing to companies that are starting unsafe AGI projects.

  Please critique. I'd be happy to get more involved in problem-solving.

  Other Ideas: Three possible solutions: (Which also explains why I think open source might have a competitive advantage.)

New Idea:

Note: A main reason I expounded upon this idea is because if somebody else has the idea but doesn't realize that there are pitfalls to it, that could turn out bad. If you guys vote this down till it's hidden, then people searching to see whether anyone has had this idea before won't be able to find my description of the pitfalls (I use ctrl-f on pages this long, myself, and it doesn't find things in hidden comments).

Patent every conceivable method of making an AGI in every country where it is possible to hold a patent and refuse to allow anyone to produce an AGI similar to your patented design until safety is assured. This idea has various flaws and could potentially backfire (a TLDR version is in the conclusion section) but I am posting it anyway for a variety of reasons, explained in the last section. I also posted several ideas that I think are better than this one.

This idea has the following flaws:

1. Patents expire after 14 years (in the US), so their protection is temporary.

2. Patenting something doesn't keep people from building it, it just gives you the right to sue them if they do. At best, it would be a deterrent, not a guarantee.

This idea could backfire in the following ways:

1. Patenting an idea has the effect of releasing some information about it to the public. If someone were to decide to learn from the ideas described in your patent and ignore the fact that you patented it - say, a government that wants to use AI to have more military power, or China because they don't view intellectual property the same way, you'd only be giving them an information advantage, not slowing them down.

2. Patenting an AGI plan under your name would be an effective way of notifying advanced hackers who want to steal your data that you are an interesting target to them.

3. If somebody goes out of their way to patent the best AGI ideas they can think of, anyone who wants to make an AGI will have to top them. If there aren't people capable and motivated enough to top the ideas being produced, that would offer some protection for the best ideas. If not, and some AGI researcher/s are both capable and motivated enough to top them, this could just contribute to and/or expediate an arms race rather than slowing it down. However, if you got all of the best AGI researchers to form a group that makes patents for the purpose of preventing unsafe AGIs from being used, this is likely to have an influence on whether other researchers are legally allowed to use those ideas.

4. I've been told (though am not a lawyer) that only the best design out of the options is patentable. If this is true, it means that even if you did manage to come up with a design better than all your competitors which is therefore patentable, you haven't stopped anyone from making an inferior AGI. In fact, all you've done is to ensure that all the inferior variations on the design are now legal for everybody else to use. The effect may be more that you've essentially made the safest one forbidden while encouraging people to use less safe ideas. If I am incorrect about this and any idea can be patented regardless of whether it is the best, then this flaw does not apply.

5. Because patenting AGI plans could result in an environment in which the AGIs that get built are much more likely to either be inferior designs or built by people who don't care for laws, it may result in the AGI being more dangerous instead of resulting in AGI being delayed.

Conclusion:

Patenting AGI ideas has a chance to slow down Moore's law, but also has a chance to make it more likely for the following things to occur which are pretty bad separately but would be worse in combination:

1. People that don't heed laws will still be able to make AGI.

2. People that don't heed laws may glean information from the patents filed and be more likely to make an AGI.

3. Law abiding people will be less likely to make AGI. This means a lower chance that AGIs created by law-abiding people will counteract the effects of bad AGI.

4. AGIs with inferior designs are more likely.

Why am I posting this idea if I can see it's flaws?

1. What if someone else has the same idea but does not know the ways in which it might backfire? If they check the internet to see whether someone had previously posted the idea, this may prevent them from triggering the consequences.

2. Often a flawed idea is very useful as part of a comprehensive strategy to solve a problem. Some big problems require several solutions, not just one.

3. Often, ideas are inspired by other ideas. The more ideas are posted, the more opportunity there is for this type of inspiration.

4. Sometimes an idea that's risky can be made safer in a way not obvious to the idea's originator.

5. Some people have a hard time imagining how difficult problems such as this one could be solved. Simply seeing that there are a bunch of ideas that could influence things or that somebody is working on it may help the person avoid feeling that the problem is hopeless. This is important because people do not try to solve problems they think are hopeless.