Safety Culture and the Marginal Effect of a Dollar

It worries me a tad that nobody in the discussion group corrected what I consider to be the obvious basic inaccuracy of the model.

Success on FAI is not a magical result of a researcher caring about safety. The researcher who would have otherwise first created AGI does not gain the power to create FAI just by being concerned about it. They would have to develop a stably self-improving AI which learned an understandable goal system which actually did what they wanted. This could be a completely different set of design technologies than what would have gone into something unstable that improved itself by ad-hoc methods well enough to go FOOM and end the game. The researcher who would have otherwise created AGI might not be good enough to do this. The best you might be able to convince them to do would be to retire from the game. It's a lot harder to convince someone to abandon the incredibly good idea they're enthusiastic about, and start over from scratch or leave the game, then to persuade people to be "concerned about safety", which is really cheap (you just put on a look of grave concern).

If I thought all you had to do to win was to convince the otherwise-first creator of AGI to be "take safety seriously", this problem would be tremendously easier and I would be approaching it in a very different way. I'd be putting practically all of my efforts into PR and academia, not trying to assemble a team to solve basic FAI problems over however-many years and then afterward build FAI. A free win just for convincing someone to take something seriously? Hot damn, that'd be one easy planet to save; there'd be no point in pursuing any other avenue until you'd totally exhausted that one.

As it stands, though, you're faced with (a) the much harder sell of convincing AGI people that they will destroy the world and that being concerned is not enough to save them, that they have to tackle much harder problems than they wanted to face on a problem that seems to them hard-enough-already; and (b) if you do convince the AGI person who otherwise would've destroyed the world, to join the good guys on a different problem or retire, you don't win. The game isn't won there. It's just a question of how long it takes the next AGI person in line to destroy the world. If you convinced them? Number three. You keep dealing through the deck until you turn up the ace of spades, unless the people working on the ace of hearts can solve their more difficult problem before that happens.

All academic persuasion does is buy time, and not very much of that - the return on effort invested seems to be pretty low.

We spent an evening at last week's Rationality Minicamp... We came up with a concrete (albeit greatly oversimplified) model...

Just to be clear: this model was drafted by a couple of mini-camp participants, not by the workshop as a whole, and isn't advocated by the Singularity Institute. For example, when I do my own back-of-the-envelopes I don't expect nearly a 30% increase in existential safety from convincing 30% of AI researchers that risk matters. Among other things, this is because there's a distance between "realize risk matters" and "successfully avoid creating UFAI" (much less "create FAI")", since sanity and know-how also play roles in AI design; and partly because there are more players than just AI researchers.

Still, it is good to get explicit models out there where they can be critiqued -- I just want to avoid folks having the impression that this is SingInst's model, or that it was taught at minicamp.

I agree that there is a lot of room for more and better academic work on this topic to reduce existential risk (including other channels like more academic research into AI safety strategies, influence on other actors like large corporations and governments, etc), but as I said at the minicamp, I think the assumptions of this model systematically lead to overestimates of effectiveness of this channel (EDIT: and would lead to overestimates of other strategies as well, including the "FAI team in a basement" strategy as I mention in my comment below).

One of the primary reasons for concern about AI risk is the likelihood of tradeoffs between safety and speed of development. Commercial or military competition make it plausible that quite extensive tradeoffs along these lines will be made, so that reckless (or self-deceived) projects are more likely to succeed first than more cautious ones. So the "random selection" assumption disproportionately favors safety.

The assumption that safety-conscious researchers always succeed in making any AI they produce safe is also fairly heroic and a substantial upward bias. There may be some cheap and simple safety measures that any safety-conscious AI project can take without significant sacrifice, but we shouldn't assign high probability to the problem being that easy. Also, if it turns out safety is so easy, why wouldn't any group sophisticated enough to build AI start to take such precautions once it became apparent they were making real progress?

As folks discussed at the time this idea was first presented, if 'concern for safety' means halting a project with high risk to pursue a lower-risk design, then unless almost all researchers are affected this just leads to a modest expected delay until someone unconcerned succeeds.

It's hard for me to imagine 100 good papers on the subject of AI safety (as opposed to say, FAI design). Once you have 10 good papers with variations of "AGI is dangerous, please be careful!", what can you say in the 11th one that you haven't already said? Also, 100 papers all carrying the same basic message, all funded by the same organization... that seems a bit surreal.

ETA: Sorry, I'm being overly skeptical and nitpicking. On reflection I think something like this probably is a good idea and should be pursued (unless money is a constraint and someone can come up with better use for it).

ETA2: If someone has done serious thinking about the feasibility of convincing a substantial fraction of AGI researchers about the need for safety, by "publishing X good quality papers", could they please explain their thoughts in more detail? (My mind keeps changing about whether this is feasible or not.)

What is the status of the academic papers from the 2010 Singularity Research Challenge?

The model is this: assume that if an AI is created, it's because one researcher, chosen at random from the pool of all researchers, has the key insight; and humanity survives if and only if that researcher is careful and takes safety seriously.

A human alone can't build a superintelligence. So, companies and other organisations are what we should mostly be concerned with. Targetting the engineering talent with the message is probably the wrong approach - you mostly want the managers and directors, since they are more likely to be the ones who willl decide what the machine wants.

I think the low-hanging fruit in getting corporations to behave better is reputation systems - which I discuss here. Merely telling corporations what they are doing is risky seems unlikely to be very effective - corporations are just not that risk-averse.

Does anyone know of a historical example of a concerted effort to convince people in an academic discipline to pay attention to something, by funding a bunch of papers on or related to the topic?

If so, how well did it work?

As a result of this calculation, I will be thinking and writing about AI safety, attempting to convince others of its importance, and, in the moderately probable event that I become very rich, donating money to the SIAI so that they can pay others to do the same.

Surely the most existential-risk-reduction-per-buck at this point is not "thinking and writing about AI safety", but thinking up more strategies like it in order to possibly find even better ones? Shouldn't SIAI (or perhaps FHI, depending on the comparative advantage between them) fund and publish some sort of systematic search-and-comparison of existential risk reduction strategies in order to have high confidence that the strategies it ends up pursuing are the optimal ones?

ETA: To be more constructive, has anyone done a similar analysis for "pushing for world-wide safety regulations on AI research" or "spending money directly on building FAI"?

Summary:

• If the first researcher with the key insight into general AI is really "safety conscious" we don't automatically get friendly AI first. That's a 10x reduction in marginal value from the original model.
• Being "safety conscious" correctly is really hard and most of the 30% won't be safety conscious in the way we want, even though they "know" they should. That's another 30x reduction in marginal value from the original model.

One big penalty that was discussed is the likelihood of another researcher having the key insight before the first researcher can leverage the insight into friendly AI. Throwing some crazy numbers down (aka a concrete albeit greatly simplified model), call it a 1% "no one would possibly think of this before FAI", 10% "50% someone else will think of this in time to beat me if they're unfriendly", 89% "this is idea whose time has come, we save a couple years on the first friendly researcher, a year on the next two, and months on the rest" and call it some fraction of 50 years. That gives something like 0.02 + 0.065 + 0.048 = down by a factor of 10.

(Edited) Another factor is whether being "safety conscious" about your key insight actually ends up gaining us anything. e.g. telling a collaborator you thought was okay but wasn't loses some of the gains. I haven't thought through this but wouldn't be viscerally upset if someone said anywhere from 10%-50% that being safety conscious works. (Edited) After reading Eliezer's comment, I think I was confusing two things (and maybe others are). There's a spectrum of safety consciousness, and I don't think all of those 30% of researchers convinced by 100 papers get to the 10%-50% level of "safety consciousness from them will work". Maybe 2% get to 50%, 10% get to 10%, and 88% get to 2% or worse aka 1%. That brings this factor down to 3%.

(Edited: this is a non-issue) There's also the possibility of very negative consequences to buying up the bright grad students (I assume we need the bright ones to get good papers produced in the right journals). I don't know if this is actually any concern at all to those with at least some intuition into the matter - I have no such intuition. (Edited) This came from my thought: "if it was generally well-known that a relatively small group of people was trying to buy up 30% of the AI research, might that cause a social backlash?" which is just flat-out wrong, we're trying to write 100 papers to convince 30% of the community, not actually buy 30% of the research. :)

In the other direction there's the possibility that "100 good papers" leads to "30% convinced" leads to "balloon upwards to 80% due to networking and no-longer-non-mainstream effects". (Edited: if this happens, it gives us a factor of about 1.5, so its total contribution is pretty small unless it's very likely)

Oh, and there's the expected time until FAI as compared to GAI... if FAI is too much longer, we only get a benefit from the 1% piece of that model which would make it down by an extremely unstable (1% plus or minus 1% ;)) factor of 50. (Edited) Let's put some crazy numbers down and say FAI being so much friggin' harder than GAI is 25%, plus 10% FAI is actually just impossible, for a 35% chance we only get the benefits from the 1% (plus or minus 1%) piece of the "delaying general AI" model. My other intuitions were coming from FAI being really hard but not a century harder than GAI. This takes my original 0.02 + 0.065 + 0.048 = 0.133 down to 0.35 0.02 + 0.65 0.133 = 0.0935, which is still about a 10x factor.

Anyone with better intuitions/experience/(gasp)knowledge want to redo those numbers or note why one of the models is terribly broken or brainstorm other yet-unmentioned factors?

After reading through the post and all the comments I think the most important moral is that a simple quantitative model thought up by very smart people in a context emphasizing rationality and examined and found lacking in significant sources of error (to the point that one of these smart people is willing to post it to Less Wrong main) can still ultimately be off by many orders of magnitude.

(Not to say that drafting a simple quantitative model isn't a great starting point, but instead that when interpreting such models one should assume that the margin of error is really really big, especially when pondering implications of the model, especially especially when pondering implications for decision policies.)

The model is this: assume that if an AI is created, it's because one researcher, chosen at random from the pool of all researchers, has the key insight; and humanity survives if and only if that researcher is careful and takes safety seriously.

The "key insight" model seems deeply flawed. We know that the technical side of the problem involves performing inductive inference - which is a close cousin of stream compression. So, progress is very likely to look like progress with stream compression. Some low-hanging fruit - and then gradually diminishing returns. Rather like digging a big hole in the ground.

"Estimate a 10% current AI risk"... wait, where did that come from? You say "Let A be the probability that an AI will be created", but actually your A is the probability that an AI will be created which then goes on to wipe out humanity unless precautions are taken, but which will also fail to wipe out humanity if the proper precautions are taken.

Your estimate for that is a whopping 10%? Without any sort of substantiating argument??
... Let's say I claim 0.000001% is a much more reasonable figure for this: what would be your rationale supporting that your estimate is more plausible than mine? Using my estimate, it suddenly becomes much more worthwhile in terms of lives saved per dollar to just build wells in Africa.

(addendum, in fact, I would argue that utility is not to me measured in lives-saved-per-dollar, or else you would need to invest in increasing fertility in Africa so you can then go on to save more lives by building wells. Instead your utility should be a stable and happy Africa (Africa because that's the most unhappy continent right now, so your payoff will tend to be greatest if you invest in Africa) -- for which end the rational thing to do will be invest in birth control rather than wells. But that's a different story)

Marginal taking-of-safety-seriously, as Eliezer points out, doesn't look good enough: you just delay the inevitable a little bit, if even that. On the other hand, establishing a widely-accepted consensus that AGI is as dangerous as A-bombs that blow up the whole universe might influence the field in more systematic ways (although it's unclear how, and achieving this goal doesn't look plausible).

Is there a body of knowledge about controlling self-modifying programs which could be used as a stepping stone to explaining what would be involved in FAI?

...if there were a 100 good papers in about it in the right journals;

Just one paper (AI safety or FAI design)...I will be very impressed. I will donate a minimum of $10 ($20 for a technical paper on FAI design) per peer-reviewed research paper per journal to the SIAI.

I doubt I'll have to donate even once within the next 50 years. But I would be happy to be proven wrong.

The model is this: assume that if an AI is created, it's because one researcher, chosen at random from the pool of all researchers, has the key insight; and humanity survives if and only if that researcher is careful and takes safety seriously.

I contest this use of the term "safety". If your goal is for humanity to survive, say that your goal is for humanity to survive. Not to "promote safety".

"Safety" means avoiding certain bad outcomes. By using the word "safety", you're trying to sneak past us the assumption "humans remaining the dominant lifeform = good, humans not remaining dominant = bad".

The argument should be over what humans have that is valuable, and how we can contribute that to the future. Not over how humans can survive.

humanity survives if and only if that researcher is careful and takes safety seriously

Here's where I'd stick in the 10^-3 penalty. It's reasonable to assume that taking safety seriously will keep you safe from accidental leaks of toxic chemicals, deadly viruses, etc. because these are well-understood phenomena that pose a single, predictable risk. If you can keep the muriatic acid off your skin, it won't burn you. If you can keep the swine flu out of your lungs, it won't infect you.

A truly general AI, though, almost by definition, would be able to think up countless ways of overpowering you. It's very unlikely that you could adequately guard against all of those ways, and the AI only needs to succeed once to cause an existential risk. Thus, it's not enough that the AI be 'securely' boxed; the AI also has to be provably friendly. And that means we have to figure out what 'friendly' even means. And while there are certainly researchers out there who can be convinced to invest in a bit of safety, proving that an AI is friendly requires way more resources than just putting on a pair of gloves or working in an airtight room.

You focus on visibly HAL-like or Skynet-like AI - the sort of thing that AI researchers produce as demos. However, we have large, smart, durable, existing entities (businesses and other computer+human teams) that are continuously getting smarter (and entrenching themselves deeper into our society) by automating their existing business practices.

I don't advocate trying to stop business automation, or humans organizing themselves into better and better teams; I think that would be throwing the baby out with the bathwater. However, I do think "business as usual" or "the default future" is the threat that existential risks people should be imagining.

The vast majority of writing about these issues has a story of terrorists or scientists (who are wizards meddling with things man was not meant to know) accidentally creating paperclip-making machines. That isn't thinking, that's straight out of folklore; e.g. Why the Sea is Salt.

1) You assign no probability to the AI being "Unfriendly" as you put it.. In particular you assume that AI = AI that kills everyone. For all I know this could be 0 and I am certainly of the opinion that it is very low.

2) The idea that the number of papers counts but the quality doesn't (except in that they are "good") is ridiculous - not only could 1 excellent paper be worth 1000 "good" ones, the "1000" good ones may not even be written if the excellent one comes first.

IMHO the only way to assess the risk of "unfriendly" AI is to build an AI (carefully) and ask it :)