Existential Risk

Actually, both that and the Earth image at the beginning of the article seem a little out of place. At least the latter would fit well into a print article (where you can devote half a page or a page to thematic images and still have plenty of text for your eyes to seek to), but online it forces scrolling on mid-sized windows before you can read comfortably. I think it'd read more smoothly if it was smaller, along the lines of the header images in "Philosophy by Humans" or (as an extreme on the high end) "The Cognitive Science of Rationality".

Agreed, especially since it is presented with no explanation or context. If the aim was "here's a picture of what we might achieve," I would personally aim for more of a Shock Level 2 image rather than an SL3 one-- presuming, of course, that this is being written for someone around SL1 (which seems likely). That said, I might omit it altogether.

Not only is the picture slightly sci-fi and weird, it's also wrong. I mean, my thought processes on seeing it went something like this: "Oh, hey, it's a ringworld. Presumably this is meant to hint at the glorious future that might be ahead of us if we don't get wiped out, and therefore the importance of not getting wiped ou ... no, wait a moment, it's kinda like a ringworld but it's really really really small. Much smaller than the earth. What the hell's the point of that?"

Agreed on this. The ringworld thing comes out of nowhere and doesn't clearly follow from the content of the article.

Unless the point is to wink-wink-nudge-nudge at the idea that we might have to do some weird-looking and weird-sounding things in order to save the world... in which case I still don't like the picture.

Also, I'd say both of those pictures seem to have the effect of inducing far mode.

I'm in favor of including the last picture as part of the article, because it shows the possible world we gain by averting existential risk. I don't believe that "context" is necessary, the image is self-explanatory.

Nitpicking on ringworld vs. stanford torus is not relevant, or interesting. The overall connotations and message are clear.

"Sci-fi" of today becomes "reality" of tomorrow. Non-transhumanists ought to open up their eyes to the potential of the light cone, and introducing them to that potential, whether directly or indirectly, is one of the biggest tasks that we have. Otherwise people are just stuck with what they see right in front of their eyes.

For a big picture issue like existential risk, it fits that one would want to also introduce a vague sketch of the possibilities of the big picture future.

Suggesting that the Earth picture itself doesn't belong in the post shows some kind of general bias against visuals, or something. You think that a picture about saving human life on earth isn't appropriately paired with a picture of the Earth? What image could be more appropriate than that?

The squid-shaped dingbats are pretty bad, too.

because, according to the Permanent Mission of the Russian Federation, nuclear retaliation is based on multiple sources that confirm an actual attack.

Given that this is coming from the sort of people who thought that setting up the Dead Hand was a good idea, and given that ass-covering and telling the public less than the truth was standard operating procedure in Russia, and given everything we know about the American government's incompetence, paranoia, greed, destructive experiments & actions (like setting PAL locks to zero, to pick a nuclear example) and that nuclear authority really was delegated to individual officers (this and other scandalous aspects came up recently in the New Yorker, actually: http://www.newyorker.com/online/blogs/newsdesk/2014/01/strangelove-for-real.html )...

I see zero reason to place any credence in their claims. This is fruit of the poisonous tree. They have reason to lie. I have no more reason to disbelieve Petrov than other similar incidents (like the Cuban Missile Crisis's submarine incident).

Very interesting. But the standard account says that Russian authorities were afraid of American attack at the time, and likely to make the wrong decision regardless of standard procedure. So the parent by itself doesn't address the relevant claim.

Also, the Wikipedia quote made it sound like Petrov might have reported sighting missiles after all (perhaps with a disclaimer). This is neither cited nor credible. If one of his superiors arguably saved the world by following protocol, high probability Putin's people would have mentioned it in their press release.

And that's why I hate Petrov story. It's ridiculous how otherwise sensible people are willing to believe in it.

This seems to beg the question: what is wrong with our established methods for dealing with these risks? The information you posted, if it is credible, would completely change the story that this post tells. Rather than a scary story about how we may be on the brink of annihilation, it becomes a story about how our organizations have changed to recognize the risks posed by technology, in order to avert these risks. In the Cold War, our methods of doing so were crude, but they sufficed, and we no longer have the same problems.

Is x-risk nevertheless an under-appreciated concern? Maybe, but I don't find this article convincing. You could make the argument that, along with the development of a technology, understanding of its risks and how to mitigate them also advances. Then it would not require a dedicated effort to understand these risks in advance. So, why is the best approach to analyse possible future risks, rather than working on projects which solve immediate problems, and dealing with issues as they arise?

Don't get me wrong, I respect what the guys at SIAI do, but I don't know the answer to this question. And it seems quite important.

"Destroy the world" can mean many things. There aren't nearly enough nuclear weapons to blast Earth itself, the planet will continue to exist, of course.

The raw destructive power of nukes may not be enough to kill most of humanity, yes. Targeted on major cities, it'll still kill an enormous amount of people, an overwhelming majority of the targeted country for industrial (ie, urban) countries.

But that's forgetting all the "secondary effects" : direct radioactive fallouts, radioactive contamination of rivers and water sources, nuclear winter, ... those are pretty sure to obliterate in the few next years most of the remaining humanity. Maybe not all of us. Maybe a few would survive, in a scorched Earth, without much left of technological civilization. That's pretty much "destroy the world" to me.

.

Indeed.

It's as if people are being deliberately mischievous by writing both "the SIAI" (which should be "SIAI"), and on the other hand, "Singularity Institute" (which should be "the Singularity Institute").

Luke is probably confused by the fact that the organization is often called "Singinst" by its members. But that expression grammatically functions as a name, like "SIAI" (or, now, "SI"), and thus does not take the definite article.

The full name, however, ("the Singularity Institute") functions grammatically as a description, and thus does take the definite article. Compare: the United Nations, the Brookings Institution, the Institute for Advanced Study, the London School of Economics, the Center for Inquiry, the National Football League.

Abbrevations differ as to whether they function as names or descriptions: IAS, but the UN. SI(AI) is like the former, not the latter.

If the abbreviation is an acronym (i.e. pronounced as a word rather than a string of letter names), then it will function as a name: ACORN, not "the ACORN" (even though, in full, it's "the Association...").

There have been quite a few posts on Language Log about which proper names are preceded by the, e.g. http://languagelog.ldc.upenn.edu/nll/?p=2172

The singinst.org About Us/Our Mission page uses the article#Definite_article), as do some other places on the site. The Strategic Plan ("UPDATE: AUGUST 2011") consistently uses no article.

"Somebody set up us the bomb."*

Fixed.

I got caught on that as well.

I'd like to point out some lukeprog fatigue here, if anyone else wrote this article it would have way more points by now.

I doubt it. I'd bet that if someone else wrote it it would have less votes.

It's a slightly expanded version of a post that has already been made, and re-run and had yearly posts linking to it. It's fine to hear the same old story again but it doesn't deserve more than half a dozen votes. It mostly scrapes through because luke seems to be writing something of a sequence on the subject so this would fit more neatly into a link collection.

Thanks for the pointer - it made me realize my lukeprog fatigue and correct for it by upvoting.

Mutually assured destruction is essentially a precommitment strategy: if you use nuclear weapons on me I commit to destroying you and your allies, a larger downside than any gains achievable from first use of nuclear weapons.

With this in mind, it's not clear to me that it'd be wrong (in the decision-theoretic sense, not the moral) to launch on a known-good missile warning. TDT states that we shouldn't differentiate between actions in an actual and a simulated or abstracted world: if we don't make this distinction, following through with a launch on warning functions to screen off counterfactual one-sided nuclear attacks, and ought to ripple back through the causal graph to screen off all nuclear attacks (a world without a nuclear war in it is better along most dimensions than the alternative). It's not a decision I'd enjoy making, but every increment of uncertainty increases the weighting of the unilateral option, and that's something we really really don't want. Revenge needn't enter into it.

(This assumes a no-first-use strategy, which the USSR at Petrov's time claimed to follow; the US claimed a more ambiguous policy leaving open tactical nuclear options following conventional aggression, which can be modeled as a somewhat weaker deterrent against that lesser but still pretty nasty possibility.)

Of course, that all assumes that the parties involved are making a rational cost-benefit analysis with good information. I'm not sure offhand how the various less ideal scenarios would change the weighting, except that they seem to make pure MAD a less safe strategy than it'd otherwise be.

From a game-theoretic perspective, if the other side knew you thought that way then they should launch on your watch.

MAD only works if both sides believe the other is willing to retaliate. If one side is willing to push the button and the other is not willing to retaliate, then the side willing to push the button nukes the other and takes over the world.

If you can be absolutely certain the other side never finds out you aren't willing to retaliate, then yours is the optimal policy.

This is why you would not have been hired to sit in front of the button, even given the Soviets' dubious hiring techniques. Also, if you had been raised in Soviet Russia, your thoughts on the topic might have been different.

Followup question: if someone was about to be placed in front of that red button, would you rather it be someone who had previously expressed the same opinion, or someone who had credibly committed to retaliate in case of a nuclear strike (however useless or foolish such retaliation might be)?

Conversely, if someone were to be placed in front of the corresponding red button of a country your leaders were about to launch a barrage of nuclear weapons against, which category would you prefer they be in?

Not that I disagree with your conclusion, but there was a significant selection pressure in the process of qualifying to get into the chair in front of the button. Political leaders don't like to give power to subordinates who are not likely to implement leadership's desires.

Having gone through the process and its accompanying ideological training makes Petrov's refusal to risk nuclear armageddon even more impressive. Even though moral courage was [ETA: not] a criteria in selecting him, Petrov showed more that anyone could reasonably expect.

Primitive needs to revenge can be extremely dangerous with todays technology.

Primitive need for revenge can be even more vital with today's technology. It is the only thing holding the most powerful players in check.

Fixed, thanks.

It would have made sense to use Bostrom's definition of existential risk, which focuses on earth-originating intelligent life instead. Replacement of humans by posthumans is not existential risk.

Some people hereabouts are concerned about some types of posthuman and "earth-originating intelligent life".

Or at the very least, explain why self-replicating things are more dangerous.