Individuals angry with humanity as a possible existential risk?

This is not an argument against technology - I'm a transhumanist after all, and I completely embrace technological developments.

If technology brings more harm than good, we should want to believe that technology does more harm than good - group affiliation is a very bad guide for epistemic rationality.

Is a sociopathic intelligent individual deliberately doing humanity harm a greater risk than a reasonable and sincere intelligent individual making a terrible mistake, or an organisation of reasonable and sincere intelligent individuals making a terrible mistake? The population of the last two groups is much larger than of the first group.

James Halperin's The Truth Machine long ago converted me to the idea that the best way to deal with this is to abandon privacy and the right to privacy as a societal ideal, and hope that our ability to thwart terrorists races their increase in power. Even an opt-in total surveillance system would help a lot by reducing the number of suspects.

I should probably make the case against privacy in a top-level post at some point, but pretty much everything I'll say will be taken from that book. For example, I bet Amanda Knox and Raffaele Sollecito are currently cursing the fact that they don't have a government-timestamped video of themselves at the time of Meredith Kercher's murder.

I think the standard community answer to this question is "Have FAI before then"

I've thought of this from the angle of the Fermi paradox. Afaik, Fermi thought war was a major filter. Spam is a minor indicator that individual sociopathy could be another filter as individual power increases. How far are we from home build-a-virus kits?

The major hope [1] I can see is that any of the nano or bio tech which could be used to destroy the human race will have a run-up period, and there will be nano and bio immune systems which might be good enough that the human race won't be at risk, even though there may be large disasters.

[1]Computer programs seem much more able to self-optimize than nano and bio systems. Except that of course, a self-optimizing AI would use nano and bio methods if they seem appropriate.

This is not a cheering thought. I think the only reasonably popular ideology which poses a major risk is the "humanity is a cancer on the planet" sort of enviromentalism-- it seems plausible that a merely pretty good self-optimizing AI tasked with eliminating the human race for the sake of other living creatures would be a lot easier to build than an FAI, and it might be possible to pull a group of people together to work on it.

"At the moment, you still need to be a fairly well informed terrorist in order to do any serious damage. But what happens when any disgruntled Induhvidual can build a weapon of mass destruction by ordering the parts through magazines?" - Scott Adams, The Dilbert Future, 1997

Thirteen years on, I don't think there's a good answer to that question yet.

In seeking to prevent such outcomes, you should focus much more on the technology than on the psychology, because the technology is the essential ingredient in these end-of-the-world scenarios and the specific psychology you describe is not an essential ingredient. Suppose there is a type of nanoreplicator which could destroy all life on Earth. Yes, it might be created and released by a suitably empowered angry person; but it might also be released for some other reason, or even just as an accident.

Sometimes this scenario comes up because someone has been imagining a world where everyone has their own desktop nanofactory, and then they suddenly think, what about the sociopaths? If anyone can make anything, that means anyone can make a WMD, which means a small minority will make and use WMDs - etc. But this just means that the scenario of "first everyone gets a nanofactory, then we worry about someone choosing to end the world" is never going to happen. The possibility of human extinction has been part of the nanotech concept from almost the beginning. This was diluted once you had people hoping to get rich by marketing nanotech, and further still once "nanotech" just became a sexy new name for "chemistry", but the feeling of peril has always hovered over the specific concept of replicating nanomachines, especially free-living ones, and any person or organization who begins to seriously make progress in that direction will surely know they are playing with fire.

There simply never will be a society of free wild-type humans with lasting open access to advanced nanotechnology. It's like giving a box of matches to every child in a kindergarten, the place would burn down very quickly. And maybe that is where we're headed anyway, not because some insane idiot really will give everyone on earth a desktop WMD-factory, but because the knowledge is springing up in too many places at once.

Ordinary monitoring and intervention (as carried out by the state) can't be more than a temporary tactic - it might work for a period of years, but it's not a solution that can define a civilization's long-term response to the challenge of nanotechnology, because in the long run there are just too many ways in which the deadly threat might materialize - designed in secret by a distributed process, manufactured in a similar way.

As with Friendly AI, the core of the long-term solution is to have people (and other intelligent agents) who want to not end the world in this way - so "psychology" matters after all - but we are talking about a seriously posthuman world order then, with a neurotechnocracy which studies your brain deeply before you are given access to civilization's higher powers, or a ubiquitous AI environment which invasively studies and monitors the value systems and real-time plans of every intelligent being. You're a transhumanist, so perhaps you can deal with such scenarios, but all of them are on the other side of a singularity and cannot possibly define a practical political or technical pre-singularity strategy for overcoming this challenge. They are not designed for a world in which people are still people and in which they possess the cognitive privacy, autonomy, and idiosyncrasy that they naturally have, and in which there are no other types of intelligent actor on the scene. Any halfway-successful approach for forestalling nanotechnological (and related) doomsdays in that world will have to be a tactical approach (again, tactical means that we don't care about it being a very long-term solution, it's just crisis management, a holding pattern) which focuses first on the specificities of the technology (what exactly would make it so dangerous, how can that be neutralized), and only secondarily on social and psychological factors behind its potential misuse.

I agree wholeheartedly with your concern. I think a more practical way of reducing risk other than "develop FAI" (which seems certainly 50+ years out, and probably 100+) is to actually take the War on Terror seriously. Sure, angry individuals are bad, but angry organizations are much, much worse, especially competent ones like Al Qaeda.

I suspect biologists should also care much more about bioterrorism than they currently do, as part of their social responsibility.