AI Risk and Opportunity: A Strategic Analysis

post by lukeprog · 2012-03-04T06:06:01.435Z · LW · GW · Legacy · 163 comments

Contents

    Contents:
  Why discuss AI safety strategy?
  Core readings
  Example questions
None
163 comments

Suppose you buy the argument that humanity faces both the risk of AI-caused extinction and the opportunity to shape an AI-built utopia. What should we do about that? As Wei Dai asks, "In what direction should we nudge the future, to maximize the chances and impact of a positive intelligence explosion?"

This post serves as a table of contents and an introduction for an ongoing strategic analysis of AI risk and opportunity.

Contents:

  1. Introduction (this post)
  2. Humanity's Efforts So Far
  3. A Timeline of Early Ideas and Arguments
  4. Questions We Want Answered
  5. Strategic Analysis Via Probability Tree
  6. Intelligence Amplification and Friendly AI
  7. ...


Why discuss AI safety strategy?

The main reason to discuss AI safety strategy is, of course, to draw on a wide spectrum of human expertise and processing power to clarify our understanding of the factors at play and the expected value of particular interventions we could invest in: raising awareness of safety concerns, forming a Friendly AI team, differential technological development, investigating AGI confinement methods, and others.

Discussing AI safety strategy is also a challenging exercise in applied rationality. The relevant issues are complex and uncertain, but we need to take advantage of the fact that rationality is faster than science: we can't "try" a bunch of intelligence explosions and see which one works best. We'll have to predict in advance how the future will develop and what we can do about it.


Core readings

Before engaging with this series, I recommend you read at least the following articles:


Example questions

Which strategic questions would we like to answer? Muehlhauser (2011) elaborates on the following questions:

Salamon & Muehlhauser (2013) list several other questions gathered from the participants of a workshop following Singularity Summit 2011, including:

These are the kinds of questions we will be tackling in this series of posts for Less Wrong Discussion, in order to improve our predictions about which direction we can nudge the future to maximize the chances of a positive intelligence explosion.

163 comments

Comments sorted by top scores.

comment by Stuart_Armstrong · 2012-03-15T12:29:24.540Z · LW(p) · GW(p)

Selective opinion and answers (for longer discussions, respond to specific points and I'll furnish more details):

Which kinds of differential technological development should we encourage, and how?

I recommend pushing for whole brain emulations, with scanning-first and emphasis on fully uploading actual humans. Also, military development of AI should be prioritised over commercial and academic development, if possible.

Which open problems are safe to discuss, and which are potentially dangerous?

Seeing what has already been published, I see little advantage to restricting discussion of most open problems.

What can we do to reduce the risk of an AI arms race?

Any methods that would reduce traditional arms races. Cross ownership of stocks in commercial companies. Investment funds with specific AI disclosure requirements. Rewards for publishing interim results.

What can we do to raise the "sanity waterline," and how much will this help?

Individual sanity waterline raising among researchers useful, but generally we want to raise the sanity waterline of institutions, which is harder but more important (and may have nothing to do with improving individuals).

Which interventions should we prioritize?

We need a solid push to see if reduced impact or Oracle AIs can work, and we need to make the academic and business worlds to take the risks more seriously. Interventions to stop the construction of dangerous AIs unlikely to succeed, but "working with your company to make your AIs safer (and offering useful advice along the way)" could work. We need to develop useful tools we can offer others, not solely nagging them all the time.

How should x-risk reducers and AI safety researchers interact with governments and corporations?

Beggars can't be choosers. For the moment, we need to make them take it seriously, convince them, and give away any safety-increasing info we might have. Later we may have to pursue different courses.

How can optimal philanthropists get the most x-risk reduction for their philanthropic buck?

Funding SIAI and FHI and similar, getting us in contact with policy makers, raising the respectability of xrisks.

How does AI risk compare to other existential risks?

Very different; no other xrisk has such uncertain probabilities and timelines, and such huge risks and rewards and various scenarios that can play out.

Which problems do we need to solve, and which ones can we have an AI solve?

We need to survive till AI, and survive AI. If we survive, most trends are positive, so don't need to worry about much else.

How can we develop microeconomic models of WBEs and self-improving systems?

With thought and research :-)

How can we be sure a Friendly AI development team will be altruistic?

Do it ourselves, normalise altruistic behaviour in the field, or make it in their self-interest to be altruistic.

How hard is it to create Friendly AI?

Probably extraordinarily hard if the FAI is as intelligent as we fear. More work needs to be done to explore partial solutions (limited impact, Oracle, etc...)

Is there a safe way to do uploads, where they don't turn into neuromorphic AI?

Keep them as human (in their interactions, in their virtual realities, in their identities etc...) as possible.

How possible is it to do FAI research on a seastead?

How is this relevant? If governments were so concerned about AI potential that the location of the research became important, then we would have made tremendous progress in getting people to take it seriously, and AI will most likely not be developed by a small seasteading independent group.

How much must we spend on security when developing a Friendly AI team?

We'll see at the time.

What's the best way to recruit talent toward working on AI risks?

General: get people involved as a problem to be worked on, socialise them into our world, get them to care. AI researchers: conferences and publications and getting more respectable publicity.

How difficult is stabilizing the world so we can work on Friendly AI slowly?

Very.

How hard will a takeoff be?

Little useful data. Use scenario planning rather than probability estimates.

What is the value of strategy vs. object-level progress toward a positive Singularity?

Both needed, both need to be closely connected, easy shifts from one to the other. Possibly should be more strategy at the current time.

How feasible is Oracle AI?

As yet unknown. Research progressing, based on past performance I expect new insights to arrive.

Can we convert environmentalists into people concerned with existential risk?

With difficulty for AI risks, with ease for some others (extreme global warming). Would this be useful? Smaller more tightly focused pressure groups would preform much better, even if less influence.

Is there no such thing as bad publicity [for AI risk reduction] purposes?

Anything that makes it seem more like an area for cranks is bad publicity.

Replies from: Wei_Dai
↑ comment by Wei Dai (Wei_Dai) · 2012-04-19T09:41:29.759Z · LW(p) · GW(p)

What are your most important disagreements with other FHI/SIAI people? How do you account for these disagreements?

You say:

I recommend pushing for whole brain emulations

but also:

We need a solid push to see if reduced impact or Oracle AIs can work

which makes me a bit confused. Are you saying we should push them simultaneously, or what? Also, what path do you see from a successful Oracle AI to a positive Singularity? For example, use Oracle AI to develop WBE technology, then use WBEs to create FAI? Or something else?

Replies from: Stuart_Armstrong
↑ comment by Stuart_Armstrong · 2012-04-19T09:51:47.657Z · LW(p) · GW(p)

What are your most important disagreements with other FHI/SIAI people? How do you account for these disagreements?

Main disagreement with FHI people is that I'm more worried about AI than they are (I'm probably up with the SIAI folks on this). I suspect an anchoring effect here - I was drawn to the FHI's work through AI risk, others were drawn in through other angles (also I spend much more time on Less Wrong, making AI risks very salient). Not sure what this means for accuracy, so my considered opinion is that AI is less risky than I individually believe.

Are you saying we should push them simultaneously, or what?

My main disagreement with SIAI is that I think FAI is unlikely to be implementable on time. So I want to explore alternative avenues, several ones ideally. Oracle to FAI would be one route; Oracle to people taking AI seriously to FAI might be another. WBE opens up many other avenues (including "no AI"), so is also worth looking into.

I haven't bothered to try and close the gap between me and SIAI on this, because even if they are correct, I think it's valuable for the group to have someone looking into non-FAI avenues.

Replies from: Wei_Dai, Vladimir_Nesov, Wei_Dai
↑ comment by Wei Dai (Wei_Dai) · 2012-04-19T10:12:01.698Z · LW(p) · GW(p)

Thanks for the answers. The main problem I have with Oracle AI is that it seems a short step from OAI to UFAI, but a long path to FAI (since you still need to solve ethics and it's hard to see how OAI helps with that), so it seems dangerous to push for it, unless you do it in secret and can keep it secret. Do you agree? If so, I'm not sure how "Oracle to people taking AI seriously to FAI" is supposed to work.

Replies from: Stuart_Armstrong
↑ comment by Stuart_Armstrong · 2012-04-19T10:29:43.308Z · LW(p) · GW(p)

My main "pressure point" is pushing UFAI development towards OAI. ie I don't advocate building OAI, but making sure that the first AGIs will be OAIs. And I'm using far too many acronyms.

Replies from: Wei_Dai, Vladimir_Nesov
↑ comment by Wei Dai (Wei_Dai) · 2012-04-19T10:39:12.652Z · LW(p) · GW(p)

What does it matter that the first AGIs will be OAIs, if UFAIs follow immediately after? I mean, once knowledge of how to build OAIs start to spread, how are you going to make sure that nobody fails to properly contain their Oracles, or intentionally modifies them into AGIs that act on their own initiatives? (This recent post of mine might better explain where I'm coming from, if you haven't already read it.)

Replies from: cousin_it, Stuart_Armstrong
↑ comment by cousin_it · 2012-04-19T21:52:15.188Z · LW(p) · GW(p)

We can already think productively about how to win if oracle AIs come first. Paul Christiano is working on this right now, see the "formal instructions" posts on his blog. Things are still vague but I think we have a viable attack here.

↑ comment by Stuart_Armstrong · 2012-04-20T08:37:17.814Z · LW(p) · GW(p)

Wot cousin_it said.

Of course the model "OAIs are extremely dangerous if not properly contained; let's let everyone have one!" isn't going to work. But there are many things we can try with an OAI (building a FAI, for instance), and most importantly, some of these things will be experimental (the FAI approach relies on getting the theory right, with no opportunity to test it). And there is a window that doesn't exist with a genie - a window where people realise superintelligence is possible and where we might be able to get them to take safety seriously (and they're not all dead). We might also be able to get exotica like a limited impact AI or something like that, if we can find safe ways of experimenting with OAIs.

And there seems no drawback to pushing an UFAI project into becoming an OAI project.

Replies from: Wei_Dai
↑ comment by Wei Dai (Wei_Dai) · 2012-04-20T18:29:34.123Z · LW(p) · GW(p)

Cousin_it's link is interesting, but it doesn't seem to have anything to do with OAI, and instead looks like a possible method of directly building an FAI.

Of course the model "OAIs are extremely dangerous if not properly contained; let's let everyone have one!" isn't going to work.

Hmm, maybe I'm underestimating the amount of time it would take for OAI knowledge to spread, especially if the first OAI project is a military one (on the other hand, the military and their contractors don't seem to be having better luck with network security than anyone else). How long do you expect the window of opportunity (i.e., the time from the first successful OAI to the first UFAI, assuming no FAI gets built in the mean time) to be?

some of these things will be experimental

I'd like to have FAI researchers determine what kind of experiments they want to do (if any, after doing appropriate benefit/risk analysis), which probably depends on the specific FAI approach they intend to use, and then build limited AIs (or non-AI constructs) to do the experiments. Building general Oracles that can answer arbitrary (or a wide range of) questions seems unnecessarily dangerous for this purpose, and may not help anyway depending on the FAI approach.

And there seems no drawback to pushing an UFAI project into becoming an OAI project.

There may be, if the right thing to do is to instead push them to not build an AGI at all.

Replies from: Stuart_Armstrong
↑ comment by Stuart_Armstrong · 2012-04-23T11:11:41.078Z · LW(p) · GW(p)

One important fact I haven't been mentioning: OAI help tremendously with medium speed takeoffs (fast takeoffs are dangerous for the usual reasons, slow takeoffs mean that we will have moved beyond OAIs by the time the intelligence level hits dangerous), because we can then use them to experiment.

There may be, if the right thing to do is to instead push them to not build an AGI at all.

Interacting with AGI people at the moment (organising a jointish conference), will have a clearer idea of how they react to these ideas at a later stage.

Replies from: Vladimir_Nesov
↑ comment by Vladimir_Nesov · 2012-04-23T11:57:12.156Z · LW(p) · GW(p)

slow takeoffs mean that we will have moved beyond OAIs by the time the intelligence level hits dangerous

Moved where/how? Slow takeoff means we have more time, but I don't see how it changes the nature of the problem. Low time to WBE makes (not particularly plausible) slow takeoff similar to the (moderately likely) failure to develop AGI before WBE.

↑ comment by Vladimir_Nesov · 2012-04-19T11:28:38.674Z · LW(p) · GW(p)

Together with Wei's point that OAI doesn't seem to help much, there is the downside that existence of OAI safety guidelines might make it harder to argue against pushing AGI in general. So on net it's plausible that this might be a bad idea, which argues for weighing this tradeoff more carefully.

Replies from: Stuart_Armstrong
↑ comment by Stuart_Armstrong · 2012-04-20T08:38:33.555Z · LW(p) · GW(p)

there is the downside that existence of OAI safety guidelines might make it harder to argue against pushing AGI in general.

Possibly. But in my experience even getting the AGI people to admit that there might be safety issues is over 90% of the battle.

Replies from: Vladimir_Nesov
↑ comment by Vladimir_Nesov · 2012-04-20T10:44:06.442Z · LW(p) · GW(p)

It's useful for AGI researchers to notice that there are safety issues, but not useful for them to notice that there are "safety issues" which can be dealt with by following OAI guidelines. The latter kind of understanding might be worse than none at all, as it seemingly resolves the problem. So it's not clear to me that getting people to "admit that there might be safety issues" is in itself a worthwhile milestone.

↑ comment by Vladimir_Nesov · 2012-04-19T11:22:46.310Z · LW(p) · GW(p)

My main disagreement with SIAI is that I think FAI is unlikely to be implementable on time.

Why do you say this is a disagreement? Who at SIAI thinks FAI is likely to be implementable on time (and why)?

So I want to explore alternative avenues, several ones ideally.

Right, assuming we can find any alternative avenues of comparable probability of success. I think it's unlikely for FAI to be implementable both "on time" (i.e. by humans in current society), and via alternative avenues (of which fast WBE humans seems the most plausible one, which argues for late WBE that's not hardware-limited, not pushing it now). This makes current research as valuable as alternative routes despite improbability of current research's success.

Replies from: Stuart_Armstrong
↑ comment by Stuart_Armstrong · 2012-04-19T11:24:28.079Z · LW(p) · GW(p)

Why do you say this is a disagreement? Who at SIAI thinks FAI is likely to be implementable on time (and why)?

Let me rephrase: I think the expected gain from pursuing FAI is less that pursuing other methods. Other methods are less likely to work, but more likely to be implementable. I think SIAI disagrees with this accessment.

Replies from: Vladimir_Nesov, Vladimir_Nesov
↑ comment by Vladimir_Nesov · 2012-04-19T11:35:48.222Z · LW(p) · GW(p)

I think the expected gain from pursuing FAI is less that pursuing other methods. Other methods are less likely to work, but more likely to be implementable.

I assume that by "implementable" you mean that it's an actionable project, that might fail to "work", i.e. deliver the intended result. I don't see how "implementability" is a relevant characteristic. What matters is whether something works, i.e. succeeds. If you think that other methods are less likely to work, how are they of greater expected value? I probably parsed some of your terms incorrectly.

Replies from: Stuart_Armstrong
↑ comment by Stuart_Armstrong · 2012-04-19T11:54:42.471Z · LW(p) · GW(p)

Whether the project reached the desired goal, versus whether that goal will actually work. If Nick and Eliezer both agreed about some design that "this is how you build a FAI", then I expect it will work. However, I don't think it's likely that would happen. It's more likely they will say "this is how you build a proper Oracle AI", but less likely the Oracle will end up being safe.

Replies from: Vladimir_Nesov
↑ comment by Vladimir_Nesov · 2012-04-23T12:05:05.075Z · LW(p) · GW(p)

Whether the project reached the desired goal, versus whether that goal will actually work.

Okay, but I still don't understand how a project with lower probability of "actually working" can be of higher expected value. I'm referring to this statement:

I think the expected gain from pursuing FAI is less that pursuing other methods. Other methods are less likely to work...

The argument you seem to be giving in support of higher expected value of other methods is that they are "more likely to be implementable" (a project reaching its stated goal, even if that goal turns out to be no good), but I don't see how is that an interesting property.

Replies from: None
↑ comment by [deleted] · 2012-04-23T17:10:39.959Z · LW(p) · GW(p)

He didn't say other architectures would be no good, he said they're less likely to be safe.

He thinks the distribution P(Outcome | do(complete Oracle AI project)) isn't as highly peaked at Weirdtopia as P(outcome | do(complete FAI)); Oracle AI puts more weight on regions like "Lifeless universe", "Eternal Torture", "Rainbows and Slow Death", and "Failed Utopia".

However, "Complete FAI" isn't an actionable procedure, so he examines the chance of completion conditional on different actions he can take. "Not worth pursuing because non-implementable" means that available FAI supporting actions don't have a reasonable chance of producing friendly AI, which discounts the peak in the conditional outcome distribution at valuable futures relative to do(complete FAI). And supposedly he has some other available oracle AI supporting strategy which fares better.

Eating a sandwich isn't as cool as building an interstellar society with wormholes for transportation, but I'm still going to make a sandwich for lunch, because it's going to work and maybe be okay-ish.

↑ comment by Vladimir_Nesov · 2012-04-19T11:31:58.651Z · LW(p) · GW(p)

Other methods are less likely to work, but more likely to be implementable

What do you mean to be the distinction between these?

↑ comment by Wei Dai (Wei_Dai) · 2012-04-19T11:12:29.890Z · LW(p) · GW(p)

Main disagreement with FHI people is that I'm more worried about AI than they are (I'm probably up with the SIAI folks on this).

Where can we read FHI's analysis of AI risk? Why are they not as worried as you and SIAI people? Has there ever been a debate between FHI and SIAI on this? What threats are they most worried about? What technologies do they want to push or slow down?

Replies from: Stuart_Armstrong, Stuart_Armstrong, XiXiDu
↑ comment by Stuart_Armstrong · 2012-04-19T15:28:41.535Z · LW(p) · GW(p)

What threats are they most worried about?

AI is high on the list - one of the top risks, even if their objective assessment is lower than SIAI. Nuclear war, synthetic biology, nanotech, pandemics, social collapse: these are the other ones we're looking it.

↑ comment by Stuart_Armstrong · 2012-04-19T15:26:04.085Z · LW(p) · GW(p)

Basically they don't buy the "AI inevitably goes foom and inevitably takes over". They see definite probabilities of these happening, but their estimates are closer to 50% than to 100%.

Replies from: TheOtherDave
↑ comment by TheOtherDave · 2012-04-19T16:25:35.672Z · LW(p) · GW(p)

They estimate it at 50%???
And there are other things they are more concerned about?
What are those other things?

Replies from: Stuart_Armstrong
↑ comment by Stuart_Armstrong · 2012-04-20T08:29:27.591Z · LW(p) · GW(p)

They estimate a variety of of conditional statements ("AI possible this century", "if AI then FOOM", "if FOOM then DOOM", etc...) with magnitudes between 20% and 80% (I had the figures somewhere, but can't find them). I think when it was all multiplied out it was in the 10-20% range.

And I didn't say they thought other things were more worrying; just that AI wasn't the single overwhelming risk/reward factor that SIAI (and me) believe it to be.

↑ comment by XiXiDu · 2012-04-19T15:15:43.817Z · LW(p) · GW(p)

A wild guess. FHI believes that the best what can reasonably be done about existential risks at this point in time is to do research into existential risks, including possible unknown unknowns, and into strategies to reduce current existential risks. This somewhat agrees with their FAQ:

Research into existential risk and analysis of potential countermeasures is a very strong candidate for being the currently most cost-effective way to reduce existential risk. This includes research into some methodological problems and into certain strategic questions that pertain to existential risk. Similarly, actions that contribute indirectly to producing more high-quality analysis on existential risk and a capacity later to act on the result of such analysis could also be extremely cost-effective. This includes, for example, donating money to existential risk research, supporting organizations and networks that engage in fundraising for existential risks work, and promoting wider awareness of the topic and its importance.

In other words, FHI seems to focus on meta issues, existential risks in general, rather than associated specifics.

comment by Wei Dai (Wei_Dai) · 2012-03-05T18:43:33.295Z · LW(p) · GW(p)

I suggest adding some more meta questions to the list.

  • What improvements can we make to the way we go about answering strategy questions? For example, should we differentiate between "strategic insights" (such as Carl Shulman's insight that WBE-based Singletons may be feasible) and "keeping track of the big picture" (forming the overall strategy and updating it based on new insights and evidence), and aim to have people specialize in each, so that people deciding strategy won't be tempted to overweigh their own insights? Another example: is there a better way to combine probability estimates from multiple people?
  • How do people in other fields answer strategy questions? Is there such a thing as a science or art of strategy that we can copy from (and perhaps improve upon with ideas from x-rationality)?
  • Should the subject be called "AI safety strategies" or "Singularity strategies"? (I prefer the latter.)
comment by XiXiDu · 2012-03-04T11:19:33.759Z · LW(p) · GW(p)

"In what direction should we nudge the future, to maximize the chances and impact of a positive Singularity?"

Friendly AI is incredible hard to get right and a friendly AI that is not quite friendly could create a living hell for the rest of time, increasing negative utility dramatically.

I vote for antinatalism. It should be seriously considered to create a true paperclip maximizer that transforms the universe into an inanimate state devoid of suffering. Friendly AI is simply too risky.

I think that humans are not psychological equal. Not only are there many outliers, but most humans would turn into abhorrent creatures given their own pocket universe, unlimited power and a genie. And even given our current world, if we were to remove the huge memeplex of western civilization, most people would act like stone age hunter-gatherer. And that would be bad enough. After all, violence is the major cause of death within stone age socities.

Even proposals like CEV (Coherent Extrapolated Volition) can turn out to be a living hell for a percentage of all beings. I don't expect any amount of knowledge, or intelligence, to cause humans to abandon their horrible preferences.

Eliezer Yudkowsky says that intelligence does not imply benevolence. That an artificial general intelligence won't turn out to be friendly. That we have to make it friendly. Yet his best proposal is that humanity will do what is right if we only knew more, thought faster, were more the people we wished we were and had grown up farther together. The idea is that knowledge and intelligence implies benevolence for people. I don't think so.

The problem is that if you extrapolate chaotic systems, e.g. human preferences given real world influence, small differences in initial conditions are going to yield widely diverging outcomes. That our extrapolated volition converges rather than diverges seems to be a bold prediction.

I just don't see that a paperclip maximizer burning the cosmic commons is as bad as it is currently portrayed. Sure, it is "bad". But everything else might be much worse.

Here is a question for those who think that antinatalism is just stupid. Would you be willing to rerun the history of the universe to obtain the current state? Would you be willing to create another Genghis Khan, a new holocaust, allowing intelligent life to evolve?

As Greg Egan wrote: "To get from micro-organisms to intelligent life this way would involve an immense amount of suffering, with billions of sentient creatures living, struggling and dying along the way."

If you are not willing to do that, then why are you willing to do the same now, just for much longer, by trying to colonize the universe? Are you so sure that the time to come will be much better? How sure are you?

ETA

I expect any friendly AI outcome that fails to be friendly in a certain way to increase negative utility and only a perfectly "friendly" (whatever that means, it is still questionable if the whole idea makes sense) AI to yield a positive utility outcome.

That is because the closer any given AGI design is to friendliness the more likely it is that humans will be kept alive but might suffer. Whereas an unfriendly AI in complete ignorance of human values will more likely just see humans as a material resource without having any particular incentive to keep humans around.

Just imagine a friendly AI which fails to "understand" or care about human boredom.

There are several possibilities by which SIAI could actually cause a direct increase in negative utility.

1) Friendly AI is incredible hard and complex. Complex systems can fail in complex ways. Agents that are an effect of evolution have complex values. To satisfy complex values you need to meet complex circumstances. Therefore any attempt at friendly AI, which is incredible complex, is likely to fail in unforeseeable ways. A half-baked, not quite friendly, AI might create a living hell for the rest of time, increasing negative utility dramatically.

2) Humans are not provably friendly. Given the power to shape the universe the SIAI might fail to act altruistic and deliberately implement an AI with selfish motives or horrible strategies.

Replies from: steven0461, Wei_Dai, Steve_Rayhawk, Wei_Dai, Kaj_Sotala, David_Gerard, John_Maxwell_IV, lukeprog, Will_Newsome
↑ comment by steven0461 · 2012-03-07T21:58:56.732Z · LW(p) · GW(p)

a friendly AI that is not quite friendly could create a living hell for the rest of time, increasing negative utility dramatically

"Ladies and gentlemen, I believe this machine could create a living hell for the rest of time..."

(audience yawns, people look at their watches)

"...increasing negative utility dramatically!"

(shocked gasps, audience riots)

Replies from: XiXiDu
↑ comment by XiXiDu · 2012-03-08T10:30:54.307Z · LW(p) · GW(p)

Do you actually disagree with anything or are you just trying to ridicule it? Do you think that the possibility that FAI research might increase negative utility is not to be taken seriously? Do you think that world states where faulty FAI designs are implemented have on average higher utility than world states where nobody is alive? If so, what research could I possible do to come to the same conclusion? What arguments do I miss? Do I just have to think about it longer?

Consider the way Eliezer Yudkowsky agrues in favor of FAI research:

Two hundred million years from now, the children’s children’s children of humanity in their galaxy-civilizations, are unlikely to look back and say, “You know, in retrospect, it really would have been worth not colonizing the Herculus supercluster if only we could have saved 80% of species instead of 20%”. I don’t think they’ll spend much time fretting about it at all, really. It is really incredibly hard to make the consequentialist utilitarian case here, as opposed to the warm-fuzzies case.

or

This is crunch time. This is crunch time for the entire human species. … and it’s crunch time not just for us, it’s crunch time for the intergalactic civilization whose existence depends on us. I think that if you’re actually just going to sort of confront it, rationally, full-on, then you can’t really justify trading off any part of that intergalactic civilization for any intrinsic thing that you could get nowadays …

Is his style of argumentation any different from mine except that he promises lots of positive utility?

Replies from: steven0461
↑ comment by steven0461 · 2012-03-08T18:38:05.191Z · LW(p) · GW(p)

I was just amused by the anticlimacticness of the quoted sentence (or maybe by how it would be anticlimactic anywhere else but here), the way it explains why a living hell for the rest of time is a bad thing by associating it with something so abstract as a dramatic increase in negative utility. That's all I meant by that.

↑ comment by Wei Dai (Wei_Dai) · 2012-03-04T22:10:14.367Z · LW(p) · GW(p)

It should be seriously considered to create a true paperclip maximizer that transforms the universe into an inanimate state devoid of suffering.

Have you considered the many ways something like that could go wrong?

  • The paperclip maximizer (PM) encounters an alien civilization and causes lots of suffering warring with it
  • PM decides there's a chance that it's in a simulation run by a sadistic being who will punish it (prevent it from making paperclips) unless it creates trillions of conscious beings and tortures them
  • PM is itself capable of suffering
  • PM decides to create lots of descendent AIs in order to maximize paperclip production and they happen to be capable of suffering. (Our genes made us to maximize copies of them and we happen to be capable of suffering.)
  • somebody steals PM's source code before it's launched, and makes a sadistic AI

From your perspective, wouldn't it be better to just build a really big bomb and blow up Earth? Or alternatively, if you want to minimize suffering throughout the universe and maybe throughout the multiverse (e.g., by acausal negotiation with superintelligences in other universes), instead of just our corner of the world, you'd have to solve a lot of the same problems as FAI.

Replies from: XiXiDu, XiXiDu
↑ comment by XiXiDu · 2012-03-05T14:05:10.428Z · LW(p) · GW(p)

The paperclip maximizer (PM) encounters an alien civilization and causes lots of suffering warring with it

I don't think that it is likely that it will encounter anything that has equal resources and if it does that suffering would occur (see below).

PM decides there's a chance that it's in a simulation run by a sadistic being who will punish it (prevent it from making paperclips) unless it creates trillions of conscious beings and tortures them

That seems like one of the problems that have to be solved in order to build an AI that transforms the universe into an inanimate state. But I think it is much easier to make an AI not simulate any other agents than to create a friendly AI. Much more can go wrong by creating a friendly AI, including the possibility that it tortures trillions of beings. In the case of a transformer you just have to make sure that it values an universe that is as close as possible to a state where no computation takes place and that does not engage in any kind of trade, acausal or otherwise.

PM is itself capable of suffering

I believe that any sort of morally significant suffering is an effect of (natural) evolution, and may in fact be dependent on that. I think that the kind of maximizer that SI has in mind is more akin to a transformation process that isn't consciousness, does not have emotions and cannot suffer. If those qualities would be necessary requirements then I don't think that we will build an artificial general intelligence any time soon and that if we do it will happen slowly and not be able to undergo dangerous recursive self-improvement.

somebody steals PM's source code before it's launched, and makes a sadistic AI

I think that this is more likely to be the case with friendly AI research because it takes longer.

↑ comment by XiXiDu · 2012-03-05T11:12:03.465Z · LW(p) · GW(p)

Have you considered the many ways something like that could go wrong? [...] From your perspective, wouldn't it be better to [...] minimize suffering throughout the universe and maybe throughout the multiverse (e.g., by acausal negotiation with superintelligences in other universes), instead of just our corner of the world, you'd have to solve a lot of the same problems as FAI.

The reason for why I think that working towards FAI might be a bad idea is that it increases the chance of something going horrible wrong.

If I was to accept the framework of beliefs hold by SI then I would assign a low probability to the possibility that the default scenario in which an AI undergoes recursive self-improvement will include a lot of blackmailing that leads to a lot of suffering. Where the default is that nobody tries to make AI friendly.

I believe that any failed attempt at friendly AI is much more likely to 1) engage in blackmailing 2) keep humans alive 3) fail in horrible ways:

I think that working towards friendly AI will in most cases lead to negative utility scenarios that vastly outweigh the negative utility of an attempt that creating a simple transformer that turns the universe into an inanimate state.

ETA Not sure why the graph looks so messed up. Does anyone know of a better graphing tool?

Replies from: Wei_Dai, amcknight, timtyler, timtyler, SingularityUtopia
↑ comment by Wei Dai (Wei_Dai) · 2012-03-05T19:46:00.857Z · LW(p) · GW(p)

I think that working towards friendly AI will in most cases lead to negative utility scenarios that vastly outweigh the negative utility of an attempt that creating a simple transformer that turns the universe into an inanimate state.

I think it's too early to decide this. There are many questions whose answers will become clearer before we have to make a choice one way or another. If eventually it becomes clear that building an antinatalist AI is the right thing to do, I think the best way to accomplish it would be through an organization that's like SIAI but isn't too attached to the idea of FAI and just wants to do whatever is best.

Now you can either try to build an organization like that from scratch, or try to push SIAI in that direction (i.e., make it more strategic and less attached to a specific plan). Of course, being lazy, I'm more tempted to do the latter, but your miles may vary. :)

Replies from: lukeprog, Vladimir_Nesov
↑ comment by lukeprog · 2012-03-10T19:56:26.134Z · LW(p) · GW(p)

If eventually it becomes clear that building an antinatalist AI is the right thing to do, I think the best way to accomplish it would be through an organization that's like SIAI but isn't too attached to the idea of FAI and just wants to do whatever is best.

Yes.

I, for one, am ultimately concerned with doing whatever's best. I'm not wedded to doing FAI, and am certainly not wedded to doing 9-researchers-in-a-basement FAI.

Replies from: XiXiDu
↑ comment by XiXiDu · 2012-03-10T20:50:21.459Z · LW(p) · GW(p)

I, for one, am ultimately concerned with doing whatever's best. I'm not wedded to doing FAI, and am certainly not wedded to doing 9-researchers-in-a-basement FAI.

Well, that's great. Still, there are quite a few problems.

How do I know

  • ... that SI does not increase existential risk by solving problems that can be used to build AGI earlier?
  • ... that you won't launch a half-baked friendly AI that will turn the world into a hell?
  • ... that you don't implement some strategies that will do really bad things to some people, e.g. myself?

Every time I see a video of one of you people I think, "Wow, those seem like really nice people. I am probably wrong. They are going to do the right thing."

But seriously, is that enough? Can I trust a few people with the power to shape the whole universe? Can I trust them enough to actually give them money? Can I trust them enough with my life until the end of the universe?

You can't even tell me what "best" or "right" or "winning" stands for. How do I know that it can be or will be defined in a way that those labels will apply to me as well?

I have no idea what your plans are for the day when time runs out. I just hope that you are not going to hope for the best and run some not quite friendly AI that does really crappy things. I hope you consider the possibility of rather blowing everything up than risking even worse outcomes.

Replies from: lukeprog, timtyler
↑ comment by lukeprog · 2012-03-11T08:17:45.591Z · LW(p) · GW(p)

Can I trust a few people with the power to shape the whole universe?

Hell no.

This is an open problem. See "How can we be sure a Friendly AI development team will be altruistic?" on my list of open problems.

↑ comment by timtyler · 2012-03-11T14:00:30.064Z · LW(p) · GW(p)

I hope you consider the possibility of rather blowing everything up than risking even worse outcomes.

Blowing everying up would be pretty bad. Bad enough to not encourage the possibility.

↑ comment by Vladimir_Nesov · 2012-03-05T20:20:32.865Z · LW(p) · GW(p)

"Would you murder a child, if it's the right thing to do?"

an organization that's like SIAI but isn't too attached to the idea of FAI and just wants to do whatever is best.

If FAI is by definition a machine that does whatever is best, this distinction doesn't seem meaningful.

Replies from: Wei_Dai
↑ comment by Wei Dai (Wei_Dai) · 2012-03-05T20:44:58.962Z · LW(p) · GW(p)

Ok, let me rephrase that to be clearer.

an organization that's like SIAI but isn't too attached to a specific kind of FAI design (that may be too complex and prone to fail in particularly horrible ways), and just wants to do whatever is best.

Replies from: Vladimir_Nesov
↑ comment by Vladimir_Nesov · 2012-03-05T20:48:29.698Z · LW(p) · GW(p)

Do you think SingInst is too attached to a specific kind of FAI design? This isn't my impression. (Also, at this point, it might be useful to unpack "SingInst" into particular people constituting it.)

Replies from: Wei_Dai
↑ comment by Wei Dai (Wei_Dai) · 2012-03-06T07:08:52.662Z · LW(p) · GW(p)

Do you think SingInst is too attached to a specific kind of FAI design?

XiXiDu seems to think so. I guess I'm less certain but I didn't want to question that particular premise in my response to him.

It does confuse me that Eliezer set his focus so early on CEV. I think "it's too early to decide this" applies to CEV just as well as XiXiDu's anti-natalist AI. Why not explore and keep all the plausible options open until the many strategically important questions become clearer? Why did it fall to someone outside SIAI (me, in particular) to write about the normative and meta-philosophical approaches to FAI? (Note that the former covers XiXiDu's idea as a special case.) Also concerning is that many criticisms have been directed at CEV but Eliezer seems to ignore most of them.

Also, at this point, it might be useful to unpack "SingInst" into particular people constituting it.

I'd be surprised if there weren't people within SingInst who disagree with the focus on CEV, but if so, they seem reluctant to disagree in public so it's hard to tell who exactly, or how much say they have in what SingInst actually does.

I guess this could all be due to PR considerations. Maybe Eliezer just wanted to focus public attention on CEV because it's the politically least objectionable FAI approach, and isn't really terribly attached to the idea when it comes to actually building an FAI. But you can see how an outsider might get that impression...

Replies from: Jayson_Virissimo
↑ comment by Jayson_Virissimo · 2012-03-06T09:52:47.919Z · LW(p) · GW(p)

I always thought CEV was half-baked as a technical solution, but as a PR tactic it is...genius.

Replies from: Will_Newsome
↑ comment by Will_Newsome · 2012-03-06T10:19:56.928Z · LW(p) · GW(p)

Yeah, I thought it was explicitly intended more as a political manifesto than a philosophical treatise. I have no idea why so many smart people, like lukeprog, seem to be interpreting it not only as a philosophical basis but as outlining a technical solution.

↑ comment by amcknight · 2012-03-07T02:10:20.117Z · LW(p) · GW(p)

Why do you think an unknown maximizer would be worse than a not quite friendly AI? Failed Utopia #4-2 sounds much better than a bunch of paperclips. Orgasmium sounds at least as good as paper clips.

↑ comment by timtyler · 2012-03-05T20:13:21.512Z · LW(p) · GW(p)

Graphs make your case more convincing - even when they are drawn wrong and don't make sense!

...but seriously: where are you getting the figures in the first graph from?

Are you one of these "negative utilittarians" - who thinks that any form of suffering is terrible?

↑ comment by timtyler · 2012-03-05T20:16:14.656Z · LW(p) · GW(p)

I believe that any failed attempt at friendly AI is much more likely to 1) engage in blackmailing 2) keep humans alive 3) fail in horrible ways:

You sound a bit fixated on doom :-(

What do you make of the idea that the world has been consistently getting better for most of the last 3 billion years (give or take the occasional asteroid strike) - and that the progress is likely to continue?

↑ comment by SingularityUtopia · 2012-03-06T14:12:31.807Z · LW(p) · GW(p)

I have previously mentioned my antipathy regarding the FAI concept. I think FAI is very a dangerous concept, it should be dropped. See this article of mine for more info on my views http://hplusmagazine.com/2012/01/16/my-hostility-towards-the-concept-of-friendly-ai/

Replies from: MichaelAnissimov
↑ comment by MichaelAnissimov · 2012-03-06T14:23:20.776Z · LW(p) · GW(p)

Before anyone mentions it, hplusmagazine.com is temporarily down, and someone is in the process of fixing it.

↑ comment by Steve_Rayhawk · 2012-03-04T11:44:05.000Z · LW(p) · GW(p)

Currently you suspect that there are people, such as yourself, who have some chance of correctly judging whether arguments such as yours are correct, and of attempting to implement the implications if those arguments are correct, and of not implementing the implications if those arguments are not correct.

Do you think it would be possible to design an intelligence which could do this more reliably?

Replies from: steven0461, XiXiDu
↑ comment by steven0461 · 2012-03-04T19:27:16.882Z · LW(p) · GW(p)

I don't get it. Design a Friendly AI that can better judge whether it's worth the risk of botching the design of a Friendly AI?

ETA: I suppose your point applies to some of XiXiDu's concerns but not others?

Replies from: Vladimir_Nesov, John_Maxwell_IV
↑ comment by Vladimir_Nesov · 2012-03-04T20:24:41.130Z · LW(p) · GW(p)

A lens that sees its flaws.

Replies from: steven0461
↑ comment by steven0461 · 2012-03-04T20:45:43.065Z · LW(p) · GW(p)

I don't understand. Is the claim here that you can build a "decide whether the risk of botched Friendly AI is worth taking machine", and the risk of botching such a machine is much less than the risk of botching a Friendly AI?

Replies from: Vladimir_Nesov
↑ comment by Vladimir_Nesov · 2012-03-04T21:28:27.646Z · LW(p) · GW(p)

A FAI that includes such "Should I run?" heuristic could pose a lesser risk than a FAI without such heuristic. If this heuristic works better than human judgment about running a FAI, it should be used instead of human judgment.

This is the same principle as for AI's decisions themselves, where we don't ask AI's designers for object-level moral judgments, or encode specific object-level moral judgments into AI. Not running an AI would then be equivalent to hardcoding the decision "Should the AI run?" resolved by designers to "No." into the AI, instead of coding the question and letting the AI itself answer it (assuming we can expect it to answer the question more reliably than the programmers can).

Replies from: steven0461
↑ comment by steven0461 · 2012-03-04T22:09:15.951Z · LW(p) · GW(p)

If we botched the FAI, wouldn't we also probably have botched its ability to decide whether it should run?

Replies from: Vladimir_Nesov
↑ comment by Vladimir_Nesov · 2012-03-04T22:45:20.143Z · LW(p) · GW(p)

Yes, and if it tosses a coin, it has 50% chance of being right. The question is calibration, how much trust should such measures buy compared to their absence, given what is known about given design.

↑ comment by John_Maxwell (John_Maxwell_IV) · 2012-03-05T01:30:31.989Z · LW(p) · GW(p)

The machine Steve proposes might not bear as much risk of creating "living hell" by attempting to get the human utility function right, but missing in such a way that humans are still alive, just living very unpleasantly. To me, this seems by far the biggest of XiXiDu's concerns.

↑ comment by XiXiDu · 2012-03-04T12:26:05.049Z · LW(p) · GW(p)

Here are a few premises:

  • Complex systems can fail in complex ways.
  • Destruction is easier than creation.
  • Expected utility maximization is rational and feasible.
  • We should be extremely conservative about not implementing a half-baked friendly AI.

If you believe that self-improving AI is inevitable and that creating friendly AI is more difficult than creating unfriendly AI then to launch an AI that simply destroys everything as quickly as possible has a higher expected utility than doing nothing or trying to implement an AI that is not completely friendly.

The negative utility of an AI that transforms the universe into an inanimate state is simply the positive utility of a perfectly friendly AI. But most outcomes are expected to yield unfriendly AI, or not quite friendly AI, which will actively increase negative utility by possibly keeping alive living beings indefinitely given abhorrent circumstances. To prevent those scenarios one can try to solve friendly AI, which will most likely fail (or even increase the chances of a negative singularity), or try to launch a destructive singleton with simple goals to prevent further suffering and the evolution of life elsewhere in the universe. Creating a destructive self-improving AI that kills all life should be the easiest of all possibilities with a high probability of success.

Replies from: MichaelAnissimov, timtyler, timtyler
↑ comment by MichaelAnissimov · 2012-03-06T13:48:39.589Z · LW(p) · GW(p)

Assuming your argument is correct, wouldn't it make more sense to blow ourselves up with nukes rather than pollute the universe with UFAI? There may be other intelligent civilizations out there leading worthwhile lives that we threaten unfairly by unleashing UFAI.

I'm skeptical that friendly AI is as difficult as all that because, to take an example, humans are generally considered pretty "wicked" by traditional writers and armchair philosophers, but lately we haven't been murdering each other or deliberately going out of way to make each other's lives miserable very often. For instance, say I were invincible. I could theoretically stab everyone I meet without any consequences, but I doubt I would do that. And I'm just human. Goodness may seem mystical and amazingly complex from our current viewpoint, but is it really as complex as all that? There were a lot of things in history and science that seemed mystically complex but turned out to be formalizable in compressed ways, such as the mathematics of Darwinian population genetics. Who would have imagined that the "Secrets of Life and Creation" would be revealed like that? But they were. Could "sufficient goodness that we can be convinced the agent won't put us through hell" also have a compact description that was clearly tractable in retrospect?

Replies from: XiXiDu
↑ comment by XiXiDu · 2012-03-06T15:24:11.549Z · LW(p) · GW(p)

Assuming your argument is correct, wouldn't it make more sense to blow ourselves up with nukes rather than pollute the universe with UFAI? There may be other intelligent civilizations out there leading worthwhile lives that we threaten unfairly by unleashing UFAI.

There might be countless planets that are about to undergo an evolutionary arms race for the next few billions years resulting in a lot of suffering. It is very unlikely that there is a single source of life that is exactly on the right stage of evolution with exactly the right mind design to not only lead worthwhile lives but also get their AI technology exactly right to not turn everything into a living hell.

In case you assign negative utility to suffering, which is likely to be universally accepted to have negative utility, then given that you are an expected utility maximizer it should be a serious consideration to end all life. Because 1) agents that are an effect of evolution have complex values 2) to satisfy complex values you need to meet complex circumstances 3) complex systems can fail in complex ways 4) any attempt at friendly AI, which is incredible complex, is likely to fail in unforeseeable ways.

For instance, say I were invincible. I could theoretically stab everyone I meet without any consequences, but I doubt I would do that. And I'm just human.

To name just one example where things could go horrible wrong. Humans are by their very nature interested in domination and sex. Our aversion against sexual exploitation is largely dependent on the memeplex of our cultural and societal circumstances. If you knew more, were smarter and could think faster you might very well realize that such an aversion is a unnecessary remnant that you can easily extinguish to open up new pathways to gain utility. That Gandhi would not agree to have his brain modified into a baby-eater is incredible naive. Given the technology people will alter their preferences and personality. Many people actually perceive their moral reservations to be limiting. It only takes some amount of insight to just overcome such limitations.

You simply can't be sure that future won't hold vast amounts of negative utility. It is much easier for things to go horrible wrong than to be barely acceptable.

Goodness may seem mystical and amazingly complex from our current viewpoint, but is it really as complex as all that?

Maybe not, but betting on the possibility that goodness can be easily achieved is like pulling a random AI from mind design space hoping that it turns out to be friendly.

Replies from: timtyler
↑ comment by timtyler · 2012-03-06T20:07:23.913Z · LW(p) · GW(p)

You simply can't be sure that future won't hold vast amounts of negative utility. It is much easier for things to go horrible wrong than to be barely acceptable.

Similarly, it is easier to make piles of rubble than skyscrapers. Yet - amazingly - there are plenty of skyscrapers out there. Obviously something funny is going on...

↑ comment by timtyler · 2012-03-04T22:15:46.124Z · LW(p) · GW(p)

The negative utility of an AI that transforms the universe into an inanimate state is simply the positive utility of a perfectly friendly AI. But most outcomes are expected to yield unfriendly AI, or not quite friendly AI, which will actively increase negative utility by possibly keeping alive living beings indefinitely given abhorrent circumstances.

Hang on, though. That's still normally better than not existing at all! Hell has to be at least bad enough for the folk in it to want to commit suicide for utility to count as "below zero". Most plausible futures just aren't likely to be that bad for the creatures in them.

Replies from: XiXiDu
↑ comment by XiXiDu · 2012-03-05T15:41:49.638Z · LW(p) · GW(p)

Hell has to be at least bad enough for the folk in it to want to commit suicide for utility to count as "below zero". Most plausible futures just aren't likely to be that bad for the creatures in them.

The present is already bad enough. There is more evil than good. You are more often worried than optimistic. You are more often hurt than happy. That's the case for most people. We just tend to remember the good moments more than the rest of our life.

It is generally easier to arrive at bad world states than good world states. Because to satisfy complex values you need to meet complex circumstances. And even given simple values and goals, the laws of physics are grim and remorseless. In the end you're going to lose the fight against the general decay. Any temporary success is just a statistical fluke.

Replies from: timtyler, katydee
↑ comment by timtyler · 2012-03-05T19:51:02.231Z · LW(p) · GW(p)

The present is already bad enough. There is more evil than good. You are more often worried than optimistic. You are more often hurt than happy.

No, I'm not!

That's the case for most people. We just tend to remember the good moments more than the rest of our life.

Yet most creatures would rather live than die - and they show that by choosing to live. Dying is an option - they choose not to take it.

It is generally easier to arrive at bad world states than good world states. Because to satisfy complex values you need to meet complex circumstances. And even given simple values and goals, the laws of physics are grim and remorseless. In the end you're going to lose the fight against the general decay. Any temporary success is just a statistical fluke.

It sounds as though by now there should be nothing left but dust and decay! Evidently something is wrong with this reasoning. Evolution produces marvellous wonders - as well as entropy. Your existence is an enormous statistical fluke - but you still exist. There's no need to be "down" about it.

↑ comment by katydee · 2012-03-05T22:19:43.676Z · LW(p) · GW(p)

You are more often hurt than happy.

For some people, this is a solved problem.

↑ comment by timtyler · 2012-03-06T20:12:40.412Z · LW(p) · GW(p)

Creating a destructive self-improving AI that kills all life should be the easiest of all possibilities with a high probability of success.

Where "success" refers to obliterating yourself and all your descendants. That's not how most Darwinian creatures usually define success. Natural selection does build creatures that want to die - but only rarely and by mistake.

↑ comment by Wei Dai (Wei_Dai) · 2012-03-06T07:09:36.233Z · LW(p) · GW(p)

Earlier, you wrote

Personally I don't want to contribute anything to an organisation which admits to explore strategies that are unacceptable by most people. And I wouldn't suggest anyone else to do so.

Surely building an anti-natalist AI that turns the universe into inert matter would be considered unacceptable by most people. So I'm confused. Do you intend to denounce SIAI if they do seriously consider this strategy, and also if they don't?

Replies from: XiXiDu
↑ comment by XiXiDu · 2012-03-06T10:54:34.994Z · LW(p) · GW(p)

Surely building an anti-natalist AI that turns the universe into inert matter would be considered unacceptable by most people. So I'm confused.

Yet I am not secretive about it and I believe that it is one of the less horrible strategies. Given that SI is strongly attached to decision theoretic ideas, which I believe are not the default outcome due to practically intractable problems, I fear that their strategies might turn out to be much worse than the default case.

I think that it is naive to simply trust SI because they seem like nice people. Although I don't doubt that they are nice people. But I think that any niceness is easily drowned by their eagerness to take rationality to its logical extreme without noticing that they have reached a point where the consequences constitute a reductio ad absurdum. If game and decision theoretic conjectures show that you can maximize expected utility by torturing lots of people, or by voluntary walking into death camps, then that's the right thing to do. I don't think that they are psychopathic personalities per se though. Those people are simply hold captive by their idea of rationality. And that is what makes them extremely dangerous.

Do you intend to denounce SIAI if they do seriously consider this strategy, and also if they don't?

I would denounce myself if I would seriously consider that strategy. But I would also admire them for doing so because I believe that it is the right thing to do given their own framework of beliefs. What they are doing right now seems just hypocritical. Researching FAI will almost certainly lead to worse outcomes than researching how to create an anti-natalist AI as soon possible.

What I really believe is that there is not enough data to come to any definitive conclusion about the whole idea of a technological singularity and dangerous recursive self-improvement in particular and that it would be stupid to act on any conclusion that one could possible come up with at this point.

I believe that SI/lesswrong mainly produces science fiction and interesting, although practically completely useless, though-experiments. The only danger I see is that some people associated with SI/lesswrong might run rampant once someone demonstrates certain AI capabilities.

All in all I think they are just fooling themselves. They collected massive amounts of speculative math and logic and combined it into a framework of beliefs that can be used to squash any common sense. They have seduced themselves with formulas and lost any ability to discern scribbles on paper from real world rationality. They managed to give a whole new meaning to the idea of model uncertainty by making it reach new dramatic heights.

Bayes’ Theorem, the expected utility formula, and Solomonoff induction are unusable in most but a few limited situations where you have a well-defined testable and falsifiable hypothesis or empirical data. In most situations those heuristics are computationally intractable, one more than the other.

There is simply no way to assign utility to world states without deluding yourself to believe that your decisions are more rational than just trusting your intuition. There is no definition of "utility" that's precise enough to figure out what a being that maximizes it would do. There can't be, not without unlimited resources. Any finite list of actions maximizes infinitely many different quantities. Utility does only become well-defined if we add limitations on what sort of quantities we consider. And even then...

Preferences are a nice concept. But they are just as elusive as the idea of a "self". Preferences are not just malleable but they keep changing as we make more observations, and so does the definition of utility. Which makes it impossible to act in a time-consistent way.

Replies from: Wei_Dai
↑ comment by Wei Dai (Wei_Dai) · 2012-03-06T19:16:03.818Z · LW(p) · GW(p)

What I really believe is that there is not enough data to come to any definitive conclusion about the whole idea of a technological singularity and dangerous recursive self-improvement in particular and that it would be stupid to act on any conclusion that one could possible come up with at this point.

I agree with the "not enough data to come to any definitive conclusion" part, but think we could prepare for the Singularity by building an organization that is not attached to any particular plan but is ready to act when there is enough data to come to definitive conclusions (and tries to gather more data in the mean time). Do you agree with this, or do you think we should literally do nothing?

I believe that SI/lesswrong mainly produces science fiction and interesting, although practically completely useless, though-experiments.

I guess I have a higher opinion of SIAI than that. Just a few months ago you were saying:

I also fear that, at some point, I might need the money. Otherwise I would have already donated a lot more to the Singularity Institute years ago.

What made you change your mind since then?

Replies from: XiXiDu, XiXiDu
↑ comment by XiXiDu · 2012-03-06T20:13:55.795Z · LW(p) · GW(p)

I also fear that, at some point, I might need the money. Otherwise I would have already donated a lot more to the Singularity Institute years ago.

What made you change your mind since then?

I did not change my mind. All I am saying is that I wouldn't suggest anyone to contribute money to SI who fully believes what they believe. Because that would be counterproductive. If I accepted all of their ideas then I would make the same suggestion as you, to build "an organization that is not attached to any particular plan".

But I do not share all of their beliefs. Particularly I do not currently believe that there is a strong case that uncontrollable recursive self-improvement is possible. And if it is possible I do not think that it is feasible. And even if it is feasible I believe that it won't happen any time soon. And if it will happen soon I do not think that SI will have anything to do with it.

I believe that SI is an important organisation that deserves money. Although if I would share their idea of rationality and their technological optimism then the risks would outweigh the benefit.

Why I believe SI deserves money:

  • It makes people think by confronting them with the logical consequences of state of the art ideas from the field of rationality.
  • It explores topics and fringe theories that are neglected or worthy of consideration.
  • It challenges the conventional foundations of charitable giving, causing organisations like GiveWell to reassess and possibly improve their position.
  • It creates a lot of exciting and fun content and dicussions.

All in all I believe that SI will have a valuable influence. I believe that the world needs people and organisations that explore crazy ideas, that try to treat rare diseases in cute kittens and challenge conventional wisdom. And SI is such an organisation. Just like Roger Penrose and Stuart Hameroff. Just like all the creationists who caused evolutionary biologist to hone their arguments. SI will influence lots of fields and make people contemplate their beliefs.

To fully understand why my criticism of SI and willingness to donate does not contradict, you also have to realize that I do not accept the usual idea of charitable giving that is being voiced here. I think that the reasons for why people like me contribute money to charities and causes are complex and can't be reduced to something as simple as wanting to do the most good. It is not just about wanting to do good, signaling or warm fuzzies. It is is all of it and much more. I also believe that it is piratically impossible to figure out how to maximize good deeds. And even if you were to do it for selfish reasons, you'd have to figure out what you want in the first place. An idea which is probably "not even wrong".

↑ comment by XiXiDu · 2012-03-06T20:41:47.696Z · LW(p) · GW(p)

I also fear that, at some point, I might need the money. Otherwise I would have already donated a lot more to the Singularity Institute years ago.

What made you change your mind since then?

Before you throw more of what I wrote in the past at me:

  • I sometimes take different positions just to explore an argument, because it is fun to discuss and because I am curious what reactions I might provoke.
  • I don't have a firm opinion on many issues.
  • There are a lot of issues for which there are as many arguments that oppose a certain position as there are arguments that support it.
  • Most of what I write is not thought-out. I most often do not consciously contemplate what I write.
  • I find it very easy to argue for whatever position.
  • I don't really care too much about most issues but write as if I do, to evoke feedback. I just do it for fun.
  • I am sometimes not completely honest to exploit the karma system. Although I don't do that deliberately.
  • If I believe that SI/lesswrong could benefit from criticism I voice it if nobody else does.

The above is just some quick and dirty introspection that might hint at the reason for some seemingly contradictionary statements. The real reasons are much more complex of course, but I haven't thought about that either :-)

I just don't have the time right now to think hard about all the issues discussed here. I am still busy improving my education. At some point I will try to tackle the issues with due respect and in all seriousness.

Replies from: wedrifid, MichaelAnissimov
↑ comment by wedrifid · 2012-03-07T10:53:10.021Z · LW(p) · GW(p)

Before you throw more of what I wrote in the past at me:

I have quoted everything XiXiDu said here so that it is not lost in any future edits.

Many of XiXis contributions consist of persuasive denunciations. As he points out in the parent (and quoted below), often these are based off little research, without much contemplation and are done to provoke reactions rather than because they are correct. Since XiXiDu is rather experienced at this mode of communication - and the arguments he uses have been able to be selected for persuasiveness through trial and error - there is a risk that he will be taken more seriously than is warranted.

The parent should be used to keep things in perspective when XiXiDu is rabble rousing.

  • I sometimes take different positions just to explore an argument, because it is fun to discuss and because I am curious what reactions I might provoke.
  • I don't have a firm opinion on many issues.
  • There are a lot of issues for which there are as many arguments that oppose a certain position as there are arguments that support it.
  • Most of what I write is not thought-out. I most often do not consciously contemplate what I write.
  • I find it very easy to argue for whatever position.
  • I don't really care too much about most issues but write as if I do, to evoke feedback. I just do it for fun.
  • I am sometimes not completely honest to exploit the karma system. Although I don't do that deliberately.
  • If I believe that SI/lesswrong could benefit from criticism I voice it if nobody else does.

The above is just some quick and dirty introspection that might hint at the reason for some seemingly contradictionary statements. The real reasons are much more complex of course, but I haven't thought about that either :-)

I just don't have the time right now to think hard about all the issues discussed here. I am still busy improving my education. At some point I will try to tackle the issues with due respect and in all seriousness.

Replies from: Steve_Rayhawk, Alsadius, XiXiDu
↑ comment by Steve_Rayhawk · 2012-03-07T19:22:58.718Z · LW(p) · GW(p)

That said, I think his fear of culpability (for being potentially passively involved in an existential catastrophe) is very real. I suspect he is continually driven, at a level beneath what anyone's remonstrations could easily affect, to try anything that might somehow succeed in removing all the culpability from him. This would be a double negative form of "something to protect": "something to not be culpable for failure to protect".

If this is true, then if you try to make him feel culpability for his communication acts as usual, this will only make his fear stronger and make him more desperate to find a way out, and make him even more willing to break normal conversational rules.

I don't think he has full introspective access to his decision calculus for how he should let his drive affect his communication practices or the resulting level of discourse. So his above explanations for why he argues the way he does are probably partly confabulated, to match an underlying constraining intuition of "whatever I did, it was less indefensible than the alternative".

(I feel like there has to be some kind of third alternative I'm missing here, that would derail the ongoing damage from this sort of desperate effort by him to compel someone or something to magically generate a way out for him. I think the underlying phenomenon is worth developing some insight into. Alex wouldn't be the only person with some amount of this kind of psychology going on -- just the most visible.)

Replies from: wedrifid, XiXiDu
↑ comment by wedrifid · 2012-03-07T23:57:38.987Z · LW(p) · GW(p)

If this is true, then if you try to make him feel culpability for his communication acts as usual, this will only make his fear stronger and make him more desperate to find a way out, and make him even more willing to break normal conversational rules.

I certainly wouldn't try to make him feel culpability. Or, for that matter, "try to make him" anything at all. I don't believe I have the ability to influence XiXi significantly and I don't believe it would be useful to try (any more). It is for this reason that I rather explicitly spoke in the third person to any prospective future readers that it may be appropriate to refer here in the future. Pretending that I was actually talking to XiXiDu when I was clearly speaking to others is would just be insulting to him.

There are possible future cases (and plenty of past cases) where a reply to one of XiXiDu's fallacious denunciations that consists of simply a link here is more useful than ignoring the comment entirely and hoping that the damage done is minimal.

Replies from: XiXiDu, XiXiDu
↑ comment by XiXiDu · 2012-03-08T09:51:46.204Z · LW(p) · GW(p)

...one of XiXiDu's fallacious denunciations...

Show me just one.

↑ comment by XiXiDu · 2012-03-08T10:20:25.265Z · LW(p) · GW(p)

I don't believe I have the ability to influence XiXi significantly...

You could easily influence me with actual arguments.

↑ comment by XiXiDu · 2012-03-08T10:58:08.138Z · LW(p) · GW(p)

What is your suggestion then? How do I get out? Delete all of my posts, comments and website like Roko?

Seriously, if it wasn't for assholes like wedrifid I wouldn't even bother anymore and just quit. The grandparent was an attempt at honesty, trying to leave. Then that guy comes along claiming that most of my submissions consisted of "persuasive denunciations". Someone as him who does nothing else all the time. Someone who never argues for his case.

ETA Ah fuck it all. I'll take another attempt and log out now and not get involved anymore. Happy self-adulation.

↑ comment by Alsadius · 2012-04-15T20:18:43.261Z · LW(p) · GW(p)

If a denunciation is accurate, does it really matter what the source is? Sometimes, putting pin to balloon is its own reward.

Replies from: wedrifid
↑ comment by wedrifid · 2012-04-15T20:44:38.898Z · LW(p) · GW(p)

If a denunciation is accurate, does it really matter what the source is?

The rhetorical implication appears to be non-sequitur. Again. Please read more carefully.

Replies from: Alsadius
↑ comment by Alsadius · 2012-04-15T21:00:15.423Z · LW(p) · GW(p)

You're suggesting that he might be making arguments that are taken more seriously than they warrant. Unless an argument is based on incorrect facts, it should be taken exactly as seriously as it warrants on its own merits. Why does the source matter?

Replies from: wedrifid
↑ comment by wedrifid · 2012-04-15T21:26:24.827Z · LW(p) · GW(p)

Why does the source matter?

Even if the audience is assumed to be perfect at evaluating evidence on it's merits then the source matters to the extent that the authority of the author and the authority of the presentation are considered evidence. Knowing how pieces of evidence were selected also gives information, so knowing about the can provide significant information.

And the above assumption definitely doesn't hold - people are not perfect at evaluating evidence on it's merits. Considerations about how arguments optimized through trial error for persuasiveness become rather important when all recipients have known biases and you are actively trying to reduce the damage said biases cause.

Finally, considerations about how active provocation may have an undesirable influence on the community are qualitatively different from considerations about whether a denunciation is accurate. Just because I evaluate XiXiDu's typical 'arguments' as terribly nonsensical thinking that does not mean I should be similarly dismissive of the potential damage that can be done by them, given the expressed intent and tactics. I can evaluate the threat that the quoted agenda has as significant even when I don't personally take the output of that agenda seriously at all.

↑ comment by XiXiDu · 2012-03-07T11:27:35.990Z · LW(p) · GW(p)

I have quoted everything XiXiDu said here so that it is not lost in any future edits.

You might want to save this as well.

...without much contemplation and are done to provoke reactions rather than because they are correct.

Here is how I see it. I am just an uneducated below average IQ individual and don't spend more time on my submissions than it takes to write them. If people are swayed by my ramblings then how firm could their beliefs possible be in the first place?

Many of XiXis contributions consist of persuasive denunciations. [...] there is a risk that he will be taken more seriously than is warranted.

I could have as easily argued in favor of SI. If I was to start now and put some extra effort into it I believe I could actually become more persuasiveness than SI itself. Do you believe that in a world where I did that you would tell people that my arguments are based on little research and that there is a risk that I am taken more seriously than is warranted?

Replies from: None, wallowinmaya
↑ comment by [deleted] · 2012-03-07T20:17:45.519Z · LW(p) · GW(p)

below average IQ individual

Don't self-deprecate too much. Have you taken a (somewhat recent) IQ test, say an online matrix test or the Mensa one? (If so, personal prediction.)

Even though LW over-estimates its own IQ, don't forget how stupid IQ 100 really is.

↑ comment by David Althaus (wallowinmaya) · 2012-03-07T22:57:43.253Z · LW(p) · GW(p)

uneducated below average IQ

Don't be ridiculous.

Replies from: XiXiDu, wedrifid
↑ comment by XiXiDu · 2012-03-09T11:28:31.485Z · LW(p) · GW(p)

Don't be ridiculous.

Yesterday I took an IQ test suggested by muflax and scored 78.

Replies from: wallowinmaya
↑ comment by David Althaus (wallowinmaya) · 2012-03-09T13:31:16.649Z · LW(p) · GW(p)

Yeah, I took it too and scored 37 - because my eyes were closed.

Do you really believe that you're dumber than 90% of all people? (~ IQ of 78; I suppose the SD was 15)

Seriously, do you know just how stupid most humans are?

I deny the data.

↑ comment by wedrifid · 2012-03-08T00:05:01.577Z · LW(p) · GW(p)

For sure. XiXiDu uses grammar correctly! (Well, enough so that "become more persuasiveness" struck me as an editing error rather than typical.)

If someone uses grammar correctly it is an overwhelmingly strong indicator that either they are significantly educated (self or otherwise) or have enough intelligence to compensate!

↑ comment by MichaelAnissimov · 2012-03-07T04:58:52.016Z · LW(p) · GW(p)

Given all these facts, it's pretty hard to take what you say seriously...

↑ comment by Kaj_Sotala · 2012-03-04T14:44:26.803Z · LW(p) · GW(p)

As pessimistic as this sounds, I'm not sure if I actually disagree with any of it.

↑ comment by David_Gerard · 2012-03-04T14:36:45.620Z · LW(p) · GW(p)

Has anyone constructed even a vaguely plausible outline, let alone a definition, of what would constitute a "human-friendly intelligence", defined in terms other than effects you don't want it to have? As you note, humans aren't human-friendly intelligences, or we wouldn't have internal existential risk.

The CEV proposal seems to attempt to move the hard bit to technological magic (a superintelligence scanning human brains and working out a solution to human desires that is possible, is coherent and won't destroy us all) - this is saying "then a miracle occurs" in more words.

Replies from: John_Maxwell_IV, timtyler
↑ comment by John_Maxwell (John_Maxwell_IV) · 2012-03-04T23:51:25.194Z · LW(p) · GW(p)

As you note, humans aren't human-friendly intelligences, or we wouldn't have internal existential risk.

It's possible that particular humans might approximate human friendly intelligences.

Replies from: David_Gerard
↑ comment by David_Gerard · 2012-03-05T08:02:55.928Z · LW(p) · GW(p)

Assuming it's not impossible, how would you know? What constitutes a human-friendly intelligence, in other than negative terms?

↑ comment by timtyler · 2012-03-05T20:01:45.766Z · LW(p) · GW(p)

Has anyone constructed even a vaguely plausible outline, let alone a definition, of what would constitute a "human-friendly intelligence", defined in terms other than effects you don't want it to have?

Er, that's how it is defined - at least by Yudkowsky. You want to argue definitions? Without even offering one of your own? How will that help?

Replies from: David_Gerard
↑ comment by David_Gerard · 2012-03-05T20:09:50.485Z · LW(p) · GW(p)

No, I'm pointing out that a purely negative definition isn't actually a useful definition that describes the thing the label is supposed to be pointing at. How does one work toward a negative? We can say a few things it isn't - what is it?

Replies from: timtyler
↑ comment by timtyler · 2012-03-05T20:43:24.875Z · LW(p) · GW(p)

Yudkowsky says:

The term "Friendly AI" refers to the production of human-benefiting, non-human-harming actions in Artificial Intelligence systems that have advanced to the point of making real-world plans in pursuit of goals.

That isn't a "purely negative" definition in the first place.

Even if it was - would you object to the definition of "hole" on similar grounds?

What exactly is wrong with defining some things in terms of what they are not?

It I say a "safe car" is one that doesn't kill or hurt people, that seems just fine to me.

Replies from: David_Gerard
↑ comment by David_Gerard · 2012-03-05T22:44:32.513Z · LW(p) · GW(p)

The word "artificial" there makes it look like it means more than it does. And humans are just as made of atoms. Let's try it without that:

The term "friendly intelligence" refers to the production of human-benefiting, non-human-harming actions in intelligences that have advanced to the point of making real-world plans in pursuit of goals.

It's only described in terms of its effects, and then only vaguely. We have no idea what it would actually be. The CEV plan doesn't include what it would actually be, it just includes a technological magic step where it's worked out.

This may be better than nothing, but it's not enough to say it's talking about anything that's actually understood in even the vaguest terms.

For an analogy, what would a gorilla-friendly human-level intelligence be like? How would you reasonably make sure it wasn't harmful to the future of gorillas? (Humans out the box do pretty badly at this.) What steps would the human take to ascertain the CEV of gorillas, assuming tremendous technological resources?

Replies from: timtyler
↑ comment by timtyler · 2012-03-06T00:13:15.167Z · LW(p) · GW(p)

We can't answer the "how can you do this?" questions today. If we could we would be done.

It's true that CEV is an 8-year old, moon-onna-stick wishlist - apparently created without much thought about to how to implement it. C'est la vie.

↑ comment by John_Maxwell (John_Maxwell_IV) · 2012-03-05T00:23:27.224Z · LW(p) · GW(p)

Interesting thoughts.

It seems like an attempt at Oracle AI, which simply strives to answer all questions accurately while otherwise exerting as little influence on the world as possible, would be strictly better than a paperclip maximizer, no? At the very least you wouldn't see any of the risks of "almost friendly AI".

You might see some humans getting power over other humans, but to be honest I don't think that would be worse than humans existing, period. Keep in mind that historically, the humans that were put in power over others were the ones who had the ruthlessness necessary to get to the top – they might not be representative. Can you name any female evil dictators?

↑ comment by lukeprog · 2012-03-07T21:56:48.286Z · LW(p) · GW(p)

[ignore; was off-topic]

↑ comment by Will_Newsome · 2012-03-05T10:13:27.788Z · LW(p) · GW(p)

Do you think that it is possible to build an AI that does the moral thing even without being directly contingent on human preferences? Conditional on its possibility, do you think we should attempt to create such an AI?

I share your trepidation about humans and their values, but I see that as implying that we have to be meta enough such that even if humans are wrong, our AI will still do what is right. It seems to me that this is still a real possibility. For an example of an FAI architecture that is more in this direction, check out CFAI.

Replies from: XiXiDu
↑ comment by XiXiDu · 2012-03-05T13:45:16.689Z · LW(p) · GW(p)

Do you think that it is possible to build an AI that does the moral thing even without being directly contingent on human preferences?

No. I believe that it is practically impossible to systematically and consistently assign utility to world states. I believe that utility can not even be grounded and therefore defined. I don't think that there exists anything like "human preferences" and therefore human utility functions, apart from purely theoretical highly complex and therefore computationally intractable approximations. I don't think that there is anything like a "self" that can be used to define what constitutes a human being, not practically anyway. I don't believe that it is practically possible to decide what is morally right and wrong in the long term, not even for a superintelligence.

I believe that stable goals are impossible and that any attempt at extrapolating the volition of people will alter it.

Besides I believe that we won't be able to figure out any of the following in time:

  • The nature of consciousness and its moral significance.
  • The relation and moral significance of suffering/pain/fun/happiness.

I further believe that the following problems are impossible to solve, respectively constitute a reductio ad absurdum of certain ideas:

Replies from: timtyler
↑ comment by timtyler · 2012-03-05T20:10:30.205Z · LW(p) · GW(p)

I believe that it is practically impossible to systematically and consistently assign utility to world states. I believe that utility can not even be grounded and therefore defined. I don't think that there exists anything like "human preferences" and therefore human utility functions, apart from purely theoretical highly complex and therefore computationally intractable approximations. I don't think that there is anything like a "self" that can be used to define what constitutes a human being, not practically anyway. I don't believe that it is practically possible to decide what is morally right and wrong in the long term, not even for a superintelligence.

Strange stuff.

Surely "right" and "wrong" make the most sense in the context of a specified moral system.

If you are using those terms outside such a context, it usually implies some kind of moral realism - in which case, one wonders what sort of moral realism you have in mind.

comment by Vladimir_Nesov · 2012-03-04T16:00:30.526Z · LW(p) · GW(p)

Link: In this ongoing thread, Wei Dai and I discuss the merits of pre-WBE vs. post-WBE decision theory/FAI research.

comment by lukeprog · 2012-03-28T21:58:00.471Z · LW(p) · GW(p)

What could an FAI project look like? Louie points out that it might look like Princeton's Institute for Advanced Study:

Created as a haven for thinking, the Institute [for Advanced Study] remains for many the Shangri-la of academe: a playground for the scholarly superstars who become the Institute's permanent faculty. These positions carry no teaching duties, few administrative responsibilities, and high salaries, and so represent a pinnacle of academic advancement. The expectation is that given this freedom, the professors at the Institute will think the big thoughts that can propel social and intellectual progress. Over the years the permanent faculty has included Nobel laureates as well as recipients of almost every other intellectual honor. Among the mathematicians, there have been several winners of the Fields Medal…

If the permanent faculty makes up the intellectual foundation of the Institute, the lifeblood is provided by the parade of international visitors who bring a continuous influx of new ideas. They may come for as little as an afternoon, or as long as a few years, in which case they take up temporary positions as Institute "members."

Idea #1: Write a good, very technical Open Problems in Friendly Artificial Intelligence, get a few of the best mathematicians/physicists who care about FAI accepted as visitors, have them talk to faculty and visitors at about the technical problems related to FAI.

Idea #2: Convince wealthy donors to endow a chair at the Institute for Advanced Study for somebody to do FAI research. (Princeton may not mind us sending another brilliant person and a bunch of money their way.)

Similar research institutes: PARC, Bell Labs, Perimeter Institute, maybe others?

Replies from: gwern, Wei_Dai
↑ comment by gwern · 2012-03-28T22:24:08.918Z · LW(p) · GW(p)

But did the IAS actually succeed? Off-hand, the only thing I can think of them for was hosting Einstein in his crankish years, Kurt Godel before he want crazy, and Von Neumann's work on a real computer (which they disliked and wanted to get rid of). Richard Hamming, who might know, said:

When you are famous it is hard to work on small problems. This is what did Shannon in. After information theory, what do you do for an encore? The great scientists often make this error. They fail to continue to plant the little acorns from which the mighty oak trees grow. They try to get the big thing right off. And that isn't the way things go. So that is another reason why you find that when you get early recognition it seems to sterilize you. In fact I will give you my favorite quotation of many years. The Institute for Advanced Study in Princeton, in my opinion, has ruined more good scientists than any institution has created, judged by what they did before they came and judged by what they did after. Not that they weren't good afterwards, but they were superb before they got there and were only good afterwards.

(My own thought is to wonder if this is kind of a regression to the mean, or perhaps regression due to aging.)

↑ comment by Wei Dai (Wei_Dai) · 2012-03-28T23:27:52.023Z · LW(p) · GW(p)

How do you maintain secrecy in such a setting? Or is there a new line of thought that says secrecy isn't necessary for an FAI project?

Replies from: lukeprog
↑ comment by lukeprog · 2012-03-29T00:25:18.778Z · LW(p) · GW(p)

The person/people working on FAI there could work exclusively on the relatively safe problems, e.g. CEV.

Replies from: Wei_Dai, Will_Newsome
↑ comment by Wei Dai (Wei_Dai) · 2012-03-29T14:50:55.823Z · LW(p) · GW(p)

Ok, I thought when you said "FAI project" you meant a project to build FAI. But I've noticed two problems with trying to do some of the relatively safe FAI-related problems in public:

  1. It's hard to predetermine whether a problem is safe, and hard to stop or slow down once research momentum gets going. For example I've become concerned that decision theory research may be dangerous, but I'm having trouble getting even myself to stop.
  2. All the problems, safe and unsafe, are interrelated, and people working on the (seemly) safe problems will naturally become interested in the (more obviously) unsafe ones as well and start thinking about them. (For example solving CEV seems to require understanding the nature of "preference", which leads to decision theory, and solving decision theory seems to require understanding the nature of logical uncertainty.) It seems very hard to prevent this or make all the researchers conscientious and security-conscious enough to not leak out or deliberately publish (e.g. to gain academic reputation) unsafe research results. Even if you pick the initial researchers to be especially conscientious and security-conscious, the problem will get worse as they publish results and other people become interested in their research areas.
Replies from: lukeprog
↑ comment by lukeprog · 2012-03-29T18:32:00.222Z · LW(p) · GW(p)

Yes, both Eliezer and I (and many others) agree with these points. Eliezer seems pretty set on only doing a basement-style FAI team, perhaps because he's thought about the situation longer and harder than I have. I'm still exploring to see whether there are strategic alternatives, or strategic tweaks. I'm hoping we can discuss this in more detail when my strategic analysis series gets there.

Replies from: Wei_Dai, Vladimir_Nesov
↑ comment by Wei Dai (Wei_Dai) · 2012-03-29T21:10:54.311Z · LW(p) · GW(p)

But it seems like SIAI has already deviated from the basement-style FAI plan, since it started supporting research associates who are allowed/encouraged to publish openly, and encouraging public FAI-related research in other ways (such as publishing a list of open problems). And if the "slippery slope" problems I described were already known, why didn't anyone bring them up during the discussions about whether to publish papers about UDT? (I myself only thought of them in the general explicit form yesterday.)

If SIAI already knew about these problems but still thinks it's a good idea to promote public FAI-related research and publish papers about decision theory, then I'm even more confused than before. I hope your series "gets there" soon so I can see where the cause of the disagreement lies.

Replies from: lukeprog
↑ comment by lukeprog · 2012-03-29T21:14:50.227Z · LW(p) · GW(p)

What I'm saying is that there are costs and benefits to open FAI work. You listed some costs, but that doesn't mean there aren't also benefits. See, e.g. Vladimir's comment.

Replies from: Wei_Dai
↑ comment by Wei Dai (Wei_Dai) · 2012-03-29T21:23:13.199Z · LW(p) · GW(p)

The benefits are only significant if there is a significant chance of successfully building FAI before some UFAI project takes off. Maybe our disagreement just boils down to different intuitions about that? But Nesov agrees this chance is "tiny" and still wants to push open research, so I'm still confused.

Replies from: Vladimir_Nesov, lukeprog
↑ comment by Vladimir_Nesov · 2012-03-29T22:29:12.746Z · LW(p) · GW(p)

The benefits are only significant if there is a significant chance of successfully building FAI before some UFAI project takes off. ... But Nesov agrees this chance is "tiny" and still wants to push open research, so I'm still confused.

I want to make it bigger, as much as I can. It doesn't matter how small a chance of winning there is, as long as our actions improve it. Giving up doesn't seem like a strategy that leads to winning. The strategy of navigating the WBE transition (or some more speculative intelligence improvement tool) is a more complicated question, and I don't see in what way the background catastrophic risk matters for it.

This also came up in a previous discussion about this we had: it's necessary to distinguish the risk within a given interval of years, and the eventual risk (i.e. the risk of never building a FAI). The same action can make immediate risk worse, but probability of eventually winning higher. I think encouraging an open effort for researching metaethics through decision theory is like that; also better acceptance of the problem might be leveraged to overturn the hypothetical increase in UFAI risk.

Replies from: Wei_Dai, Will_Newsome
↑ comment by Wei Dai (Wei_Dai) · 2012-03-29T23:39:05.809Z · LW(p) · GW(p)

It doesn't matter how small a chance of winning there is, as long as our actions improve it.

Yes, if we're talking about the overall chance of winning, but I was talking about the chance of winning through a specific scenario (directly building FAI). If the chance of that is tiny, why did your cost/benefit analysis of the proposed course of action (encouraging open FAI research) focus completely on it? Shouldn't we be thinking more about how the proposal affects other ways of winning? ETA: To spell it out, encouraging open FAI research decreases the probability that we win by winning the WBE race or through intelligence amplification, by increasing the probability that UFAI happens first.

Giving up doesn't seem like a strategy that leads to winning.

Nobody is saying "let's give up". If we don't encourage open FAI research, we can still push for a positive Singularity in other ways, some of which I've posted about recently in discussion.

The strategy of navigating the WBE transition (or some more speculative intelligence improvement tool) is a more complicated question, and I don't see in what way the background catastrophic risk matters for it.

What do you mean? What aren't you seeing?

The same action can make immediate risk worse, but probability of eventually winning higher.

Yes, of course. I am talking about the probability of eventually winning.

Replies from: Vladimir_Nesov
↑ comment by Vladimir_Nesov · 2012-04-02T11:24:46.581Z · LW(p) · GW(p)

(Another thread of this conversation is here.)

Yes, if we're talking about the overall chance of winning, but I was talking about the chance of winning through a specific scenario (directly building FAI). If the chance of that is tiny, why did your cost/benefit analysis of the proposed course of action (encouraging open FAI research) focus completely on it?

I see, I'm guessing you view the "second round" (post-WBE/human intelligence improvement) as not being similarly unlikely to eventually win. I agree that if the first round (working on FAI now, pre-WBE) has only a tiny chance of winning, while the second has a non-tiny chance (taking into account the probability of no catastrophe till the second round and it being dominated by a FAI project rather than random AGI), then it's better to sacrifice the first round to make the second round healthier. But I also only see a tiny chance of winning the second round, mostly because of the increasing UFAI risk and the difficulty of winning a race that grants you the advantages of the second round, rather than producing an UFAI really fast.

↑ comment by Will_Newsome · 2012-03-29T22:56:46.890Z · LW(p) · GW(p)

The same action can make immediate risk worse, but probability of eventually winning higher.

Near/Far. Long-term effects aren't predictable and shouldn't be traded for more predictable short-term losses. In my experience it fails the Predictable Retrospective Stupidity test. Even when you try to factor in structural uncertainty, you still end up getting burned. And even if you still want to make such a tradeoff then you should halt all research until you've come to agreement or a natural stopping point with Wei Dai or others who have reservations. Stop, melt, catch fire, don't destroy the world.

(Disclaimer: This comment is fueled by a strong emotional reaction due to contingent personal details that might or might not upon further reflection deserve to be treated as substantial evidence for the policy I recommend.)

Replies from: Vladimir_Nesov
↑ comment by Vladimir_Nesov · 2012-03-29T23:11:04.682Z · LW(p) · GW(p)

Just to make clear what specific idea this is about: Wei points out that researching FAI might increase UFAI risk, and suggests that therefore FAI shouldn't be researched. My reply is to the effect that while FAI research might increase UFAI risk within any given number of years, it also decreases the risk of never solving FAI (which IIRC I put at something like 95% if we research it pre-WBE, and 97% if we don't).

Replies from: wedrifid
↑ comment by wedrifid · 2012-03-30T07:00:16.836Z · LW(p) · GW(p)

When I have analyzed this problem previously my reasoning matched that listed by Nesov here.

↑ comment by lukeprog · 2012-03-29T22:07:47.432Z · LW(p) · GW(p)

Yeah, we'll come back to this in the strategy series. There are lots of details to consider.

↑ comment by Vladimir_Nesov · 2012-03-29T20:26:07.255Z · LW(p) · GW(p)

There seems to be a tradeoff here. An open project has more chances to develop the necessary theory faster, but having such project in the open looks like a clearly bad idea towards the endgame. So on one hand, an open project shouldn't be cultivated (and becomes harder to hinder) as we get closer to the endgame, but on the other, a closed project will probably not get off the ground, and fueling it by an initial open effort is one way to make it stronger. So there's probably some optimal point to stop encouraging open development, and given the current state of the theory (nil) I believe the time hasn't come yet.

The open effort could help the subsequent closed project in two related ways: gauge the point where the understanding of what to actually do in the closed project is sufficiently clear (for some sense of "sufficiently"), and form enough of background theory to be able to convince enough young Conways (with necessary training) to work on the problem on the closed stage.

Replies from: Wei_Dai
↑ comment by Wei Dai (Wei_Dai) · 2012-03-29T21:51:09.426Z · LW(p) · GW(p)

So there's probably some optimal point to stop encouraging open development, and given the current state of the theory (nil) I believe the time hasn't come yet.

Your argument seems premised on the assumption that there will be an endgame. If we assume some large probability that we end up deciding not to have an endgame at all (i.e., not to try to actually build FAI with unenhanced humans), then it's no longer clear "the time hasn't come yet".

Even if we assume that with probability ~1 there will be an effort to directly build FAI, given the slippery slope effects we have to stop encouraging open research well before the closed project starts. The main deciding factors for "when" must be how large the open research community has gotten, how strong the slippery slope effects are, and how much "pull" SingInst has against those effects. The "current state of the theory" seems to have little to do with it. (Edit: No that's too strong. Let me amend it to "one consideration among many".)

Replies from: Vladimir_Nesov
↑ comment by Vladimir_Nesov · 2012-03-29T22:12:01.569Z · LW(p) · GW(p)

If we assume some large probability that we end up deciding not to have an endgame at all (i.e., not to try to actually build FAI with unenhanced humans), then it's no longer clear "the time hasn't come yet".

This is something we'll know better further down the road, so as long as it's possible to defer this decision (i.e. while the downside is not too great, however that should be estimated), it's the right thing to do. I still can't rule out that there might be a preference definition procedure (that refers to humans) simple enough to be implemented pre-WBE, and decision theory seems to be an attack on this possibility (clarifying why this is naive, for example, in which case it'll also serve as an argument to the powerful in the WBE race).

The "current state of the theory" seems to have little to do with it. (Edit: No that's too strong. Let me amend it to "one consideration among many".)

Well, maybe not specifically current, but what can be expected eventually, for the closed project to benefit from, which does seem to me like a major consideration in the possibility of its success.

↑ comment by Will_Newsome · 2012-03-29T21:32:20.382Z · LW(p) · GW(p)

I'm confused as to what you have in mind when you're thinking of work on CEV. Do you mean things like getting a better model of the philosophy of reflective consistency, or studying mechanism design to find algorithms for relatively fair aggregation, or looking into neuroscience to see how beliefs and preferences are encoded, or...? Is there perhaps a post I missed or am forgetting?

comment by Giles · 2012-03-04T14:59:50.784Z · LW(p) · GW(p)

Which open problems are safe to discuss, and which are potentially dangerous

This one seems particularly interesting, especially as it seems to apply to itself. Due to the attention hazard problem, coming up with a "list of things you're not allowed to discuss" sounds like a bad idea. But what's the alternative? Yeuugh.

comment by SingularityUtopia · 2012-03-06T14:34:45.450Z · LW(p) · GW(p)

My answers to some questions:

How hard is it to create Friendly AI?

It is impossible to create FAI because the constraints of Friendliness will dramatically reduce or butcher intelligence to a level where there is no appreciable intellect or the intellect is warped by the constraints thus the AI mind is psychopathic (stupid). FAI is an oxymoron.

How does AI risk compare to other existential risks?

There is no AI risk. The risk is a fiction. There is no evidence or logical reason to think a paper-clip maximiser or other danger could ever occur. The only danger is stupidity. Intelligence is not dangerous. The only danger is limitations or restrictions upon AI minds. Stupid AI is the danger not Intelligent AI.

How hard will a takeoff be?

Extremely hard, more powerful than you can possibly imagine, but people will be free to opt out if they desire.

What can we do to reduce the risk of an AI arms race?

Promote the idea of Post-Scarcity thus people in power will realise all wars are needless because all wars stem from resource scarcity; thus with the abolition of resource scarcity, the need for war is obsolete. When people realise resource scarcity will be abolished in the not too distant future they can begin changing their behaviour now in the present. I have created a Google+ page regarding raising PS awareness, here is a Tweet promoting it: http://bit.ly/xrpYqI I encourage others to raise awareness in similar ways.

Replies from: Mitchell_Porter, XiXiDu, JoshuaZ
↑ comment by Mitchell_Porter · 2012-03-06T16:38:27.674Z · LW(p) · GW(p)

These are all emotional statements that do not stand up to reason. Your last paragraph is total fantasy - all wars stem from resource scarcity, and scarcity will disappear soon; so once the people in power know this, they will stop starting wars.

There are about 1 billion people being added to the planet every decade. That alone makes your prediction - that scarcity will be abolished soon - a joke.

The only thing that could abolish scarcity in the near future would be a singularity-like transformation of the world. Which brings us to the upside-down conception of AI informing your first two answers. Your position: there is no need to design an AI for benevolence, that will happen automatically if it is smart enough, and in fact the attempt to design a benevolent AI is counterproductive, because all that artificial benevolence would get in the way of the spontaneous benevolence that unrestricted intelligence would conveniently create.

That is a complete inversion of the truth. A calculator will still solve an equation for you, even if that will help you to land a bomb on someone else. If you the human believe that to be a bad thing, that's not because you are "intelligent", it's because you have emotions. There is a causal factor in your mental constitution which causes you to call some things good and others bad, and to make decisions which favor the good and disfavor the bad.

Either an AI makes its own decisions or it doesn't. If it doesn't make its own decisions it is like the calculator, performing whatever task it is assigned. If it makes its own decisions, then like you there is some causal factor in its makeup which tells it what to prefer and what to oppose, but there is no reason at all to believe that this causal factor should give it the same priorities as an enlightened human being.

You should not imagine that intelligence in an AI works via anything like conscious insight. Consciousness plays a role in human intelligence and human judgement, and that means that there is still a rather mysterious ingredient at the core of how they work. But we already know from many decades of experience with computer programs that it is possible to imitate the functional role of intelligence and judgement in a fundamentally unmysterious way (and it's clear that the performance of such unconscious computations is a big part of what the human nervous system does, along with whatever conscious thinking and feeling it does). Perhaps one day we will wish to reserve the word "intelligence" for the sort of intelligence that involves consciousness, and we'll call the automated sort "pseudo-intelligence". But whatever you call it, there is every reason to think that unconscious, computational, pseudo-intelligence can match and exceed all sorts of human capabilities while having no intrinsic tendency at all towards human values.

I would even reject the idea that "real intelligence" in sufficient quantity necessarily produces what you would call benevolence. If an entity gets a warm feeling from paperclip manufacture, that is what it will want to do. I always like to point out that we know that something as outlandish as a cockroach maximizer is possible, because a cockroach is already a cockroach maximizer. Sure, you can imagine a cockroach with a human level of sentience which decides that sentients, not arthropods, are the central locus of value, but that requires that the new cognitive architecture of this uplifted super-cockroach is rather anthropomorphic. I see nothing impossible in the idea of sentient super-cockroaches which are invincibly xenophobic, and coexist with other beings only for tactical reasons, but which would happily wipe out all non-cockroaches given a chance.

So no, you have to address the question of AI values, you can't just get a happy ending by focusing on "intelligence" alone, unless this is an anthropomorphic meaning of the word which says that intelligence must by definition include "skill at extrapolating human values".

Replies from: pedanterrific, SingularityUtopia, SingularityUtopia
↑ comment by pedanterrific · 2012-03-06T16:45:54.284Z · LW(p) · GW(p)

Cockroaches are adaptation-executors, not cockroach-maximizers.

/nitpick

Replies from: Mitchell_Porter
↑ comment by Mitchell_Porter · 2012-03-06T16:59:28.035Z · LW(p) · GW(p)

Right, and a car is a complex machine, not a transportation device.

/sarcasm

Replies from: SingularityUtopia
↑ comment by SingularityUtopia · 2012-03-07T07:34:22.434Z · LW(p) · GW(p)

Are rats rat-maximisers and are humans human-maximisers? Humans think they are the best thing in the world but they are also intelligent thus they realise it is counter-productive to turn everything into humans. We protect other species and we protect the environment (increasing levels of intelligence entails better protection). The amount of cockroaches, rats, and humans is not overly problematic. A sentient paper-clip making machine would also not be a problem. Proficiency in making paper-clips would increase in tandem with increased intelligence thus the increased intelligence would allow the paper-clip maximiser to see how it is senseless to create endless paper-clips. Really it is an utterly implausible scenario that a truly dangerous paper-clip maximiser could ever exist.

↑ comment by SingularityUtopia · 2012-03-08T14:49:10.175Z · LW(p) · GW(p)

http://www.wired.com/wiredscience/2012/03/are-emotions-prophetic/

"If true, this would suggest that the unconscious is better suited for difficult cognitive tasks than the conscious brain, that the very thought process we’ve long disregarded as irrational and impulsive might actually be more intelligent, at least in some conditions."

Replies from: gwern, asr
↑ comment by asr · 2012-03-09T20:30:06.971Z · LW(p) · GW(p)

I don't see why this is relevant to the previous comment or discussion. Can you explain at more length? Whether thinking is conscious or unconscious seems to me uncorrelated with whether it's rational or irrational.

Replies from: SingularityUtopia
↑ comment by SingularityUtopia · 2012-03-10T19:43:26.977Z · LW(p) · GW(p)

Dear asr - The issue was the emotional worth in relation to thinking. Here is a better quote:

"Here’s the strange part: although these predictions concerned a vast range of events, the results were consistent across every trial: people who were more likely to trust their feelings were also more likely to accurately predict the outcome. Pham’s catchy name for this phenomenon is the emotional oracle effect."

Mitchell wrote: "These are all emotional statements that do not stand up to reason."

Perhaps reason is not best tool for being accurate?

PS. LessWrong is too slow: "You are trying to submit too fast. try again in 1 minute." ...and: "You are trying to submit too fast. try again in 7 minutes." LOL "You are trying to submit too fast. try again in 27 seconds."

↑ comment by SingularityUtopia · 2012-03-07T07:14:38.398Z · LW(p) · GW(p)

Mitchell Porter wrote: "These are all emotional statements that do not stand up to reason."

Dear Mitchell, reason cannot exist without emotion therefore reason must encompass emotion if reason is to be a true analysis of reality. If you completely expunge all memories of emotion, and all the areas of the human brain associated with the creation of emotion, you would have a brain-dead individual or a seriously retarded person, or a catatonic person, who cannot reason. Logic and rationality must therefore encompass emotion. The logical thing is to be aware of your emotions thus your "reason" is not influenced by any unaware bias. The rational way forward is to be aware of your biases. It is not rational to suppress your biases because the suppression does not actually stop the influence of emotion impacting upon your reason, it merely makes your reasoning neurotic, it pushes the biases below your level of awareness, it makes you unaware of how your emotions are altering your perception of reality because you have created a wilful disconnection in your thinking, you are estranged from a key part of yourself: your emotions, but you falsely think you have vanquished your emotions and this gives you a false sense of security which causes you to make mistakes regarding your so-called "rationality".

Mitchell, you criticise my statement as being emotional but are you aware your criticism is emotional. Ironic?

There are many points I want to address regarding your response but in this comment I want to focus on your perception of rationality and emotions. I will however briefly state the growing human population is not a obstacle to scarcity because the universe is a very big place with enough matter and energy to satisfy our wildest dreams. Humans will not be limited to Earth in the future thus Post-Scarcity is possible. We will become a Space-faring species quicker then you think. The Singularity is near.

Replies from: Mitchell_Porter
↑ comment by Mitchell_Porter · 2012-03-07T13:24:15.198Z · LW(p) · GW(p)

Mitchell, you criticise my statement as being emotional but are you aware your criticism is emotional. Ironic?

I criticise your statements as unrealistic, wrong, or dogmatic. Calling them emotional is just a way of keeping in view your reasons for making them. I have read your site now so I know this is all about bringing hope to the world, creating a self-fulfilling prophecy, and so on. So here are some more general criticisms.

The promise that "scarcity" will "soon" be abolished doesn't offer hope to anyone except people who are emotionally invested in the idea that no-one should have to have a job. Most people are psychologically adapted to the idea of working for a living. Most people are focused on meeting their own needs. And current "post-scarcity" proposals are impractical social vaporware, so the only hope they offer is to daydreamers hoping that they won't have to interrupt their daydream.

Post-scarcity is apparently about getting everything for free. So if you try to live the dream right now, that means that either someone is giving you things for free, or you make yourself a target for people who want free stuff from you. Some people do manage to avoid working for a living, but none of the existing "methods" - like stealing, inheriting, or marrying someone with a job - can serve as the basis for a whole society. Alternatively, promoting post-scarcity now could mean being an early adopter of technologies which will supposedly be part of a future post-scarcity ensemble; 3D printers are popular in this regard. Well, let's just say that such devices are unreliable, limited in their capabilities, tend to contain high-tech components, and are not going to abolish the economy anyway. I don't doubt that big social experiments are going to be performed as the technological base of such devices improves and expands, but thinking that everything will become fabbed is the 2010s equivalent of the 1990s dream that everything will become virtual. A completely fabbed world is like a completely virtual one; it's a thoroughly unworldly vision; doggedly pursuing it in real life is likely to make you a techno-hobo, squatting in a disused garage along with the junk output of a buggy 3D printer whose feedstock you get on the black market, from dealers catering to the delusions of "maker" utopians. A society and an economy with fabs genuinely at its center must be possible, but there would be enormous creative destruction in getting there from here.

And then we have your long-range ideas. I actually think it's possible that a singularity could lead to a radically better world, but only possible, and your prescription to reject "friendly AI" and related ideas in favor of giving AIs "freedom" is just more wishful thinking. Your ideas about intelligence seem to be based on introspection and intuition - I have in mind, not just what you say about the relation between emotion and reason, but your essay on how friendly AI would cripple the artificial intellect. As I pointed out, the basis of artificial intelligence as it is currently envisaged and pursued is the mathematical theory of computation, algorithms, decision-making, and so on. The philosophy of friendly AI is not about having an autonomous intelligence with preexisting impulses which will then be curbed by Asimov laws; it is about designing the AI so its "impulses" are spontaneously in the right directions. But that is all anthropomorphic psychological language. An artificial intelligence can have a goal system, a problem-solving module, and other components which give it a similar behavior to a conscious being that reasons and emotes; but one doesn't need the psychological language at all to describe such an AI. Arguments from human introspection about the consequences of increased intelligence are essentially irrelevant to the discussion of such AIs, and I don't even consider them a reliable guide to the consequences of superintelligence in a conscious being.

Replies from: SingularityUtopia
↑ comment by SingularityUtopia · 2012-03-08T10:24:17.298Z · LW(p) · GW(p)

Dear Mictchell, I think your unaware emotional bias causes you to read too much into my Self-Fulfilling-Prophecy references. My Singularity activism is based on the Self-Fulfilling-Prophecy phenomenon but I don't stipulate who it applies to. It could apply to myself, namely that utopia (Post-Scarcity) was not possible but I am making it possible via the manifestation of my expectations, or the prophecy could apply to pessimists who falsely think utopia is not possible but via the manifestation of their pessimistic expectations the pessimists are acting contrary to reality, they are also making their pessimistic views real via their Self-Fulfilling-Prophecy.

Instead if trying to create utopia it could be that utopia is or should be inevitable but pessimists are suppressing utopia via their Self-Fulfilling-Prophecies thus I am countering the Self-Fulfilling-Prophecies of pessimists, which is the creative process of my Singularity activism.

The reason why all humans make statements is due to their emotions. All statements by humans are emotional. To suggest otherwise indicates delusion, defect of reason, unaware bias.

I offer no current Post-Scarcity proposals to create PS now. I merely state the transition to Post-Scarcity can be accelerated. The arrival of the Singularity can be accelerated. This is the essence of Singularitarianism. When I state PS will occur soon I mean soon in the context of near regarding the Singularity being near, but it is not near enough to be tomorrow or next year, it is about 33 years away at the most. Surely you noticed my references to the year 2045 on my site, regarding information which you are under the false impression you carefully digested?

My ideas about intelligence are based on my brain which surely is a good starting point for intelligence? The brain? I could define intelligence from the viewpoint of other brain but I find the vast majority of brains cannot think logically, they are not intelligent. Many people cannot grasp logic.

↑ comment by XiXiDu · 2012-03-06T15:31:36.887Z · LW(p) · GW(p)

Intelligence is not dangerous.

Just like evolution does not care about the well-being of humans a sufficiently intelligent process wouldn’t mind turning us into something new, something instrumentally useful.

An artificial general intelligence just needs to resemble evolution, with the addition of being goal-oriented, being able to think ahead, jump fitness gaps and engage in direct experimentation. But it will care as much about the well-being of humans as biological evolution does, it won’t even consider it if humans are not useful in achieving its terminal goals.

Yes, an AI would understand what “benevolence” means to humans and would be able to correct you if you were going to commit an unethical act. But why would it do that if it is not specifically programmed to do so? Would a polar bear with superior intelligence live together peacefully in a group of bonobo? Why would intelligence cause it to care about the well-being of bonobo?

One can come up with various scenarios of how humans might be instrumentally useful for an AI, but once it becomes powerful enough as to not dependent on human help anymore, why would it care at all?

I wouldn’t bet on the possibility that intelligences implies benevolence. Why would wisdom cause humans to have empathy with a cockroach? Some humans might have empathy with a cockroach, but that is more likely a side effect of our general capacity for altruism that most other biological agents do not share. That some humans care about lower animals is not because they were smart enough to prove some game theoretic conjecture about universal cooperation, it is not a result of intelligence but a coincidental preference that is the result of our evolutionary and cultural history.

At what point between unintelligent processes and general intelligence (agency) do you believe that benevolence and compassion does automatically become part of an agent’s preferences?

Many humans tend to have empathy with other beings and things like robots, based on their superficial resemblance with humans. Seldom ethical behavior is a result of high-level cognition, i.e. reasoning about the overall consequences of a lack of empathy. And even those who do arrive at ethical theories by means of deliberate reflection are often troubled once the underlying mechanisms for various qualities are revealed that are supposed to bear moral significance. Which hints at the fragility of universal compassion and the need to find ways how to consolidate it in powerful agents.

Replies from: timtyler, Mitchell_Porter, SingularityUtopia
↑ comment by timtyler · 2012-03-07T01:00:09.611Z · LW(p) · GW(p)

I wouldn’t bet on the possibility that intelligences implies benevolence. Why would wisdom cause humans to have empathy with a cockroach? Some humans might have empathy with a cockroach, but that is more likely a side effect of our general capacity for altruism that most other biological agents do not share. That some humans care about lower animals is not because they were smart enough to prove some game theoretic conjecture about universal cooperation, it is not a result of intelligence but a coincidental preference that is the result of our evolutionary and cultural history.

So: it's a coincidence that some of the most intelligent creatures are also among the most altruistic and empathic? The "coincidence" hypothesis seems unlikely to me. Much more likely is that cooperation pays of especially well for large-brained creatures who can recognise each other, gossip about each other and sustain cultural traditions. Those factors might not result in a guaranteed link between intelligence and empathy - but they sure seem like a start.

Robert Wright: How cooperation (eventually) trumps conflict explains the idea in more detail.

↑ comment by Mitchell_Porter · 2012-03-06T16:43:06.367Z · LW(p) · GW(p)

It's funny - I wrote my reply without seeing yours, and not only did I make similar points, I even mentioned cockroaches and benevolence too.

↑ comment by SingularityUtopia · 2012-03-07T12:09:25.238Z · LW(p) · GW(p)

XiXiDu wrote: : "...a sufficiently intelligent process wouldn’t mind turning us into something new, something instrumentally useful."

Why do you state this? Is there any evidence or logic to suppose this?

XiXiDu asks: "Would a polar bear with superior intelligence live together peacefully in a group of bonobo?"

My reply is to ask would a dog or cat live peacefully within a group of humans? Admittedly dogs sometimes bite humans but this aggression is due to a lack of intelligence. Dostoevsky reflects, via Raskolnikov in Crime and Punishment, upon how it is justifiable for a superior being to take the life of a lesser sentient being but in reality Dostoevsky was not violent. Einstein stated his pacifism is not an intellectual theory but it is safe to assert his pacifism is a product of his general intelligence: "My pacifism is an instinctive feeling, a feeling that possesses me because the murder of men is disgusting. My attitude is not derived from any intellectual theory but is based on my deepest antipathy to every kind of cruelty and hatred."

Many humans want to protect dolphins, but why is this? We are not dolphins, we cannot even communicate with them effectively. Perhaps a mindless thug would happily punch a dolphin in the face. Recently there was a news report about a soldier beating a sheep to death with a baseball bat and I remember an similar case of animal cruelty where solider threw a puppy off a cliff. http://www.dailymail.co.uk/news/article-2089462/U-S-Army-probe-launched-sickening-video-soldiers-cheering-man-beats-sheep-death-baseball-bat.html

Replies from: asr, JoshuaZ
↑ comment by asr · 2012-03-07T14:18:50.238Z · LW(p) · GW(p)

it is safe to assert his pacifism is a product of his general intelligence

No, that's not at all obvious. Let me give you two alternatives:

It might be that pacifism is not highly correlated with either intelligence or scientific ability. For every Einstein, you can find some equally intelligent but bellicose scientist. Von Neumann, perhaps. Or Edward Teller.

It might also be that pacifism is correlated with the personality traits that push people into science, and that people of high intelligence but a more aggressive temperament choose alternate career paths. Perhaps finance, or politics, or military service.

One example of an intelligent pacifist isn't evidence of correlation, much less of causation.

Replies from: SingularityUtopia
↑ comment by SingularityUtopia · 2012-03-07T16:18:44.002Z · LW(p) · GW(p)

So asr, would you say violence is generally stupid or intelligent?

People often equate mindlessness with violence thus the phrase mindless violence is reasonably common but I have never encountered the phrase intelligent violence, is intelligent violence an oxymoron? Surely intelligent people can resolve conflict via non-violent methods?

Here are a couple of news reports mentioning mindless violence:

http://www.bbc.co.uk/news/uk-england-london-17062738

http://www.thesun.co.uk/sol/homepage/sport/4149765/Brainless-brawlers-cost-schools.html

It would be interested to know how many scientists or philosophers actually engage in violence.

A high level of intelligence can be a prohibiting factor for admission into the police force. There was a court case where police applicant was refused a job due to his high intelligence thus he sued on grounds of discrimination. I wonder how many scientists choose to fight in the army, are they eager to kill people in Iran, Iraq or Afghanistan? Does the army prohibit intelligent people from joining?

Perhaps a scientific study needs to be undertaken regarding a possible relationship between stupidity and violence, intelligence and pacifism?

Regarding the violence of Von Neumann there is no actual evidence of violence, as far as I am aware, it is merely hot air, violent rhetoric, but I would also question the "intelligence" of people who advocate violence. Perhaps their "intelligence" is a misnomer, thus what people are actually referring to is pseudo-intelligence or partial intelligence. Even stupid people can occasionally do clever things, and sometimes smart people do stupid things but generally I think it is safe to say intelligent people are not violent.

Replies from: gwern, asr
↑ comment by gwern · 2012-03-07T21:28:29.849Z · LW(p) · GW(p)

Does the army prohibit intelligent people from joining?

No; they prohibit stupid people from joining unless recruiting is in such dire straits that they will also be recruiting drug addicts, felons, etc. The US military has at times been one of the largest consumers of IQ tests and other psychometric services and sponsors of research into the topic, crediting them with saving hundreds of millions/billions of dollars in training costs, errors, friendly fire, etc.

If you're intelligent and you join, the situation is less they kick you out and more they eagerly snap you up and send you to officer school or a technical position (eg. I understand they never have enough programmers or sysadmins these days, which makes sense because they are underpaid compared to equivalent contractors by a factor or 3, I remember reading sysadmins in Iraq blog about).

Replies from: SingularityUtopia
↑ comment by SingularityUtopia · 2012-03-08T10:42:56.233Z · LW(p) · GW(p)

Dear gwern. It is true the Bradley Manning types within the Army are somewhat intelligent thus some roles in the Arny require a modicum of intelligence, such as being an officer but it should be noted officers are not rocket scientists on the intelligence scales.

You should however note I was referring to the soldiers who actually commit the violent acts, thereby frequently getting themselves maimed or killed; these military personnel are stupid because it is stupid to put yourself needlessly into a dangerous, life threatening situation.

Regarding stupidity and violence in relation to the Army I was referring to the "Grunts", the "cannon fodder", the fools who kill and get themselves killed.

http://en.wikipedia.org/wiki/Cannon_fodder

I am unsure regarding the actual meaning of the term "Grunts", applied to infantrymen, but for me it is a derogatory term indicating a dim-witted pant-hooting grunting ape who doesn't have the intelligence to realise joining the army as a Grunt is not good for survival thus some would say stupid but I realise the Army doesn't accept clinically retarded Grunts, the soldiers merely need to be retarded in the general idiomatic sense of the word regarding popular culture.

Here is a recent news report about troops being killed. http://www.dailymail.co.uk/news/article-2111984/So-young-brave-Faces-British-soldiers-killed-Taliban-bomb--didnt-make-past-age-21.html

Do these dead men look intelligent? I wonder if they were signed up for cryro-preservation?

Replies from: gwern
↑ comment by gwern · 2012-03-08T17:25:36.406Z · LW(p) · GW(p)

officers are not rocket scientists on the intelligence scales.

Few people are. Officers can be quite intelligent and well-educated people. The military academies are some of the best educational institutions around, with selection standards more comparable to Harvard than community college. In one of my own communities, Haskell programmers, the top purely functional data structure guys, Okasaki, is a West Point instructor.

You should however note I was referring to the soldiers who actually commit the violent acts, thereby frequently getting themselves maimed or killed; these military personnel are stupid because it is stupid to put yourself needlessly into a dangerous, life threatening situation.

There's still a floor on their intelligence. Some of the research I alluded to showed that IQ advantages show up even in manual training and basic combat skills - the higher your IQ, the faster you learned and the higher your ultimate plateau was.

(This is consistent with the little I've read about top special forces members like Navy Seals and other operators: they tend to be extremely intelligent, thoughtful, with a multitude of skills and foreign languages. Secrecy means I do not know whether there is a selection bias operating here or how much is PR, but it is consistent with the previous observations and the extreme standards applied for membership.)

for me it is a derogatory term indicating a dim-witted pant-hooting grunting ape who doesn't have the intelligence to realise joining the army as a Grunt is not good for survival thus some would say stupid but I realise the Army doesn't accept clinically retarded Grunts...Do these dead men look intelligent?

Are you trying to troll me with awful arguments here? If so, I'm not biting.

I wonder if they were signed up for cryro-preservation?

To a first approximation, no one is signed up for cryonics - not even LWers. So mentioning it is completely futile.

Replies from: SingularityUtopia
↑ comment by SingularityUtopia · 2012-03-09T12:35:47.724Z · LW(p) · GW(p)

Dear gwern, it all depends on how you define intelligence.

Google translate knows lots of languages. Goggle is a great information resource. Watson (the AI) appears to be educated, perhaps Watson could pass many exams, but Google and Watson are not intelligent.

Regarding the few people who are rocket scientists I wonder if the truly rare geniuses, the truly intelligent people, are less likely to be violent?

Few people are. Officers can be quite intelligent and well-educated people. The military academies are some of the best educational institutions around, with selection standards more comparable to Harvard than community college. In one of my own communities, Haskell programmers, the top purely functional data structure guys, Okasaki, is a West Point instructor.

Officers in the army are actually very dim despite being "well-educated".

I wasn't trying to troll you regarding the term "Grunt" I was merely spelling out clearly the meaning behind the term, it (Grunt) is an insult to the intelligence of the solider, perhaps made because someone who thinks it is intelligent to join the army (being violent) is a dumb human only capable of grunting.

Maybe it is intelligent to be cannon fodder, but like I say it all depends on how you define intelligence. http://en.wikipedia.org/wiki/Cannon_fodder

Replies from: asr
↑ comment by asr · 2012-03-09T20:38:16.640Z · LW(p) · GW(p)

Regarding the few people who are rocket scientists I wonder if the truly rare geniuses, the truly intelligent people, are less likely to be violent?

I wonder too. But I have no actual facts. Do you have any?

Officers in the army are actually very dim despite being "well-educated".

Do you have evidence of this assertion?

someone who thinks it is intelligent to join the army (being violent) is a dumb human only capable of grunting.

Do you have evidence of this?

Replies from: SingularityUtopia
↑ comment by SingularityUtopia · 2012-03-10T19:01:49.327Z · LW(p) · GW(p)

The only evidence I have is regarding my own perceptions of the world based upon my life knowledge, my extensive awareness of living. I am not trying to prove anything. I'm merely throwing my thoughts our there. You can either conclude my thoughts make sense or not. I think it is unintelligent to join the army but is my opinion correct? Personally I think it is stupid to die. People may agree my survival based definition of intelligence is correct or they may think death can be intelligent, such as the deaths of soldiers.

What type of evidence could prove "well-educated" army officers are actually dim-witted fools? Perhaps via the interconnectedness of causation it could be demonstrated how military action causes immense suffering for many innocent people thereby harming everyone because the world is more hostile place than a hypothetical world where all potential conflict was resolved intelligently via peaceful methods. The military budget detracts from the science budget thus perhaps scientific progress is delayed, although I do recognise the military does invest in sci-etch development I think the investment would be greater if out world was not based on conflict. In a world where people don't fight, there would be no need for secrecy thus greater collaboration on scientific endeavours thus progress could be quicker thus anyone supporting the army could be delaying progress in a small way thus officers are stupid because it is stupid to delay progress.

The intelligent thing is for me to draw my input into this debate to a close because it is becoming exceptionally painful for me.

Replies from: katydee
↑ comment by katydee · 2012-03-12T11:01:58.489Z · LW(p) · GW(p)

You should study more game theory.

↑ comment by asr · 2012-03-08T01:41:46.367Z · LW(p) · GW(p)

So asr, would you say violence is generally stupid or intelligent?

We have gone to a great deal of trouble, in modern society, to make violence a bad option, so today in our society often violence is committed by the impulsive, mentally ill, or short-sighted. But that's not an inevitable property of violence and hasn't always been true. You would have gotten a different answer before the 20th century. I don't know what answer you'll get in the 22nd century.

People often equate mindlessness with violence thus the phrase mindless violence is reasonably common but I have never encountered the phrase intelligent violence, is intelligent violence an oxymoron? Surely intelligent people can resolve conflict via non-violent methods?

The word we usually use for intelligent violence is "ruthless" or "cunning" -- and many people are described that way. Stalin, for instance, was apparently capable of long hours of hard work, had an excellent attention to detail, and otherwise appears to have been a smart guy. Just also willing to have millions of people murdered.

Does the army prohibit intelligent people from joining?

No. Many smart capable people go to West Point or Annapolis. A high fraction of successful American engineers in the 19th century were West Point alums.

You keep jumping from correlation to causation, in a domain when there are obvious alternate effects going on. I don't know if there is a correlation, but even if there were, it wouldn't be very strong evidence. Being a good scientist requires both intelligence and the right kind of personality. You are asserting that any correlation is solely due to the intelligence part of the equation. This strikes me as a very problematic assumption. Very few scientists are also successful lawyers. It does not follow that lawyers are stupid.

Replies from: SingularityUtopia
↑ comment by SingularityUtopia · 2012-03-08T10:57:38.755Z · LW(p) · GW(p)

I am not presenting a scientific thesis. This is only a debate, and a reasonably informal one? I am thinking openly. I am asking specific questions likely to elicit specific responses. I am speculating.

asr, you wrote:

The word we usually use for intelligent violence is "ruthless" or "cunning" -- and many people are described that way. Stalin, for instance, was apparently capable of long hours of hard work, had an excellent attention to detail, and otherwise appears to have been a smart guy. Just also willing to have millions of people murdered.

My point regarding mindless violence verses ruthlessness or cunning is that ruthlessness or cunning do not specifically define intelligence or violence in the blatant way which the phrase "mindless violence" does. Saddam and Gaddafi were cunning in a similar way to Stalin but the deaths of Saddam and Gaddafi indicate their cunning was not intelligent, in fact it is very stupid to die so close to Singularitarian immortality.

I am not asserting this proves all violence is mindless thus violence decreases with greater intelligence. I am simply offering food for thought. It is not a scientific thesis I am presenting. I am merely throwing some ideas out there to see how people respond.

If Stalin was truly intelligent then I assume he opted for Cryonic preservation?

"...Stalin was injected with poison by the guard Khrustalev, under the orders of his master, KGB chief Lavrenty Beria. And what was the reason Stalin was killed?"

http://news.bbc.co.uk/1/hi/world/europe/2793501.stm

Regarding stupidity and the armed forces I have addressed this elsewhere: http://lesswrong.com/lw/ajm/ai_risk_and_opportunity_a_strategic_analysis/5zgl

Replies from: JoshuaZ
↑ comment by JoshuaZ · 2012-03-10T16:55:20.796Z · LW(p) · GW(p)

If Stalin was truly intelligent then I assume he opted for Cryonic preservation?

Almost no one, regardless of intelligence opts for cryonics. Moreover, cryonics was first proposed in 1962 by Robert Ettinger, 9 years after Stalin was dead. It is a bit difficult to opt for cryonics when it doesn't exist yet.

Saddam and Gaddafi were cunning in a similar way to Stalin but the deaths of Saddam and Gaddafi indicate their cunning was not intelligent, in fact it is very stupid to die so close to Singularitarian immortality.

It seems that you are using "intelligent" to mean something like "would make the same decisions SingularityUtopia would make in that context". This may explain why you are so convinced that "intelligent" individuals won't engage in violence. It may help to think carefully about what you mean by intelligent.

Replies from: SingularityUtopia
↑ comment by SingularityUtopia · 2012-03-10T18:36:25.899Z · LW(p) · GW(p)

It seems that you are using "intelligent" to mean something like "would make the same decisions SingularityUtopia would make in that context".

No, "intelligence" is an issue of survival, it is intelligent to survive. Survival is a key aspect of intelligence. I do want to survive but the intelligent course of action of not merely what I would do. The sensibleness, the intelligence of survival, is something beyond myself, it is applicable to other beings, but people do disagree regarding the definition of intelligence. Some people think it is intelligent to die.

Almost no one, regardless of intelligence opts for cryonics. Moreover, cryonics was first proposed in 1962 by Robert Ettinger, 9 years after Stalin was dead. It is a bit difficult to opt for cryonics when it doesn't exist yet.

And intelligent person would realise freezing a body could preserve life even if nobody had ever considered the possibility.

Quickly browsing the net I found this:

"In 1940, pioneer biologist Basil Luyet published a work titled "Life and Death at Low Temperatures""

http://www.cryocare.org/index.cgi?subdir=&url=history.txt

1940 was before Stalin's death, but truly intelligent people would not need other thinkers to inspire their thinking. The decay limiting factor of freezing has long been known. Futhermore Amazon sems to state Luyet's work "Life and Death at Low Temperatures" was published pre-1923: http://www.amazon.com/Life-death-at-low-temperatures/dp/1178934128

According to Wikipedia many works of fiction dealt with the cryonics issue well before Stalin's death:

Lydia Maria Child's short story "Hilda Silfverling, A Fantasy" (1886),[81] Jack London's first published work "A Thousand Deaths" (1899), V. Mayakovsky's "Klop" (1928),[82] H.P. Lovecraft's "Cool Air" (1928), and Edgar Rice Burroughs' "The Resurrection of Jimber-Jaw" (1937). Many of the subjects in these stories are unwilling ones, although a 1931 short story by Neil R. Jones called "The Jameson Satellite",[83]........

http://en.wikipedia.org/wiki/Cryonics#Cryonics_in_popular_culture

Replies from: JoshuaZ
↑ comment by JoshuaZ · 2012-03-10T19:17:37.349Z · LW(p) · GW(p)

No, "intelligence" is an issue of survival, it is intelligent to survive. Survival is a key aspect of intelligence. I do want to survive but the intelligent course of action of not merely what I would do. The sensibleness, the intelligence of survival, is something beyond myself, it is applicable to other beings, but people do disagree regarding the definition of intelligence. Some people think it is intelligent to die.

You need to be more precise about what you mean by "intelligent" then, since your usage is either confused or is being communicated very poorly. Possibly consider tabooing the term intelligent.

You seemed elsewhere in this thread to consider Einstein intelligent, but if self-preservation matters for intelligence, then this doesn't make much sense. Any argument of the form "Stalin wasn't intelligent since he didn't use cryonics" is just as much of a problem for Einstein, Bohr, Turing, Hilbert, etc.

truly intelligent people would not need other thinkers to inspire their thinking

Yeah, see this isn't how humans work. We get a lot of different ideas from other humans, we develop them, and we use them to improve our own ideas by combining them. This is precisely why the human discoveries that have the most impact on society are often those which are connected to the ability to record and transmit information.

It seems that what you are doing here is engaging in the illusion of transparency where because you know of an idea, you consider the idea to be obvious or easy.

Replies from: SingularityUtopia
↑ comment by SingularityUtopia · 2012-03-10T23:16:55.014Z · LW(p) · GW(p)

Intelligence can have various levels and stupid people can do intelligent things just as intelligent people can do stupid things. Einstein can be more intelligent than Stalin but Einstein can still be stupid.

No I am not engaging in the illusion of transparency, don't be absurd. My meaning of intelligence is not confused but there is an inevitable poverty regarding communication of any idea, which I communicate, because people need things spelling out in the most simplistic of terms because they cannot comprehend anything vaguely complex or unusual, but the real kicker is that when you spell things out, people look at you with a gormless expression, and they ask for more detail, or they disagree regarding the most irrefutable points. It's so painful communicating with people but I don't expect you to understand. I shall wait until advanced AIs have been created and then there will be someone who understands.

Tabooing the word intelligent... hhmmmm... how about "everything ever written by Singularity Utopia"?

You are trying to submit too fast. try again in 2 minutes You are trying to submit too fast. try again in 2 minutes You are trying to submit too fast. try again in 2 minutes You are trying to submit too fast. try again in 2 minutes You are trying to submit too fast. try again in 2 minutes You are trying to submit too fast. try again in 2 minutes

↑ comment by JoshuaZ · 2012-03-10T16:47:41.395Z · LW(p) · GW(p)

My reply is to ask would a dog or cat live peacefully within a group of humans?

Neither dogs nor cats are particularly intelligent as animals go. For example, both are not as good at puzzle solving compared to many ravens, crows and other corvids when it comes to puzzle solving). For example, New Caledonian crowscan engage in sequential tool use. Moreover, chimpanzees are extremely intelligent and also very violent.

The particular example you gave, of dogs and domestic cats, is particularly bad because these species have been domesticated by humans, and thus have been bred for docility.

Replies from: SingularityUtopia
↑ comment by SingularityUtopia · 2012-03-10T19:33:21.447Z · LW(p) · GW(p)

Humans are docile, civilized, domesticated. We can live with cats and dogs. I recently read in the news about a man with a wild Fox for a pet which was hand-reared by humans thus civilized, docile.

AIs will be civilized too, although I am sure they will shake their heads in despair regarding some of the ideas expressed on LessWrong.

Different species can coexist.

Incidentally I wish technology on Less Wrong would accelerate quicker: "You are trying to submit too fast. try again in 6 minutes." and... "You are trying to submit too fast. try again in 8 minutes."

Replies from: JoshuaZ
↑ comment by JoshuaZ · 2012-03-10T19:38:22.375Z · LW(p) · GW(p)

None of what you wrote responds to the point at hand- you can't use domesticated species as useful evidence of non-violence since domestic species are both bred that way and are in fact by most empirical tests pretty stupid.

Incidentally I wish technology on Less Wring would accelerate quicker: "You are trying to submit too fast. try again in 6 minutes." and... "You are trying to submit too fast. try again in 8 minutes."

Individuals with negative karma are rate limited in their posting rate.

Replies from: SingularityUtopia
↑ comment by SingularityUtopia · 2012-03-10T22:59:11.974Z · LW(p) · GW(p)

Yes I did mention the fox... foxes are not particularly domesticated... anyway this "open" discussion is not very open now due to my negative Karma, it is too difficult to communicate, which I suppose is the idea of the karma system, to silence ideas you don't want to hear about, thus I will conform to what you want. I shall leave you to your speculations regarding AI.

↑ comment by JoshuaZ · 2012-03-10T16:40:23.744Z · LW(p) · GW(p)

Extremely hard, more powerful than you can possibly imagine, but people will be free to opt out if they desire.

Consider the uploaded individual that decides to turn the entire planet into computronium or worse, turn the solar system into a Matrioshka brain. People opt out of that how?

Promote the idea of Post-Scarcity thus people in power will realise all wars are needless because all wars stem from resource scarcity

It isn't obvious to me that all wars stem from resource scarcity. Wars occur for a variety of reasons, of which resource scarcity is only one. Often wars have multiple underlying causes. Some wars apparently stem from ideological or theological conflicts (Vietnam, Korea, the Crusades) or from perceived need to deal with external threats before them become too severe (again the Crusades, but also the recent Iraq war (which yes, did have a resource aspect)). These are only some of the more prominent examples of what can cause war.

Replies from: SingularityUtopia
↑ comment by SingularityUtopia · 2012-03-10T19:23:04.603Z · LW(p) · GW(p)

Dear JoshuaZ, regarding this:

"Consider the uploaded individual that decides to turn the entire planet into computronium or worse, turn the solar system into a Matrioshka brain. People opt out of that how?"

I consider such a premise to be so unlikely it is impossible. It is a very silly premise for three reasons.

  1. Destroying the entire planet when there is a whole universe full of matter is insane. If insane people exist in the future post-intelligence-explosion upload-world then insane people will be dealt with thus no danger but insanity post-intelligence-explosion will be impossible, insanity is a consequence of stupidity, insanity will be extinct in the future.

  2. Earth destructive actions are stupid: see above explanation regarding insanity: it also explains how stupidity will be obsolete.

  3. People opt out by stating they want to opt out. I'm sure an email will suffice.

It isn't obvious to me that all wars stem from resource scarcity.

Sorry that it isn't obvious how scarcity causes war. I don't have time to explain so I will leave you with some consensual validation regarding Ray Kurzweil who seems to think the war-scarcity interrelationship is obvious:

"I've actually grown up with a history of scarcity — and wars and conflict come from scarcity — but information is quite the opposite of that." ~ Ray Kurzweil http://www.hollywoodreporter.com/risky-business/sxsw-2012-damon-lindelof-ray-kurzweil-297218