[Altruist Support] LW Go Foom

Half-rationalists: people who pick up on enough memes to be really dangerous, but not on enough to realise that what they're doing might be foolish. For example, building an AI without adding the friendliness features.

Not everything is about AI and existential risk. . We already have a section in the sequences about how knowing about cognitive biases can hurt you. It seems unlikely that anyone is going to get the knowledge base to build an AGI from simply being exposed to a few memes here. If there's one thing that we've seen from AI research in the last fifty years is that strong AI is really, really hard.

Rationalists with bad goals: Someone could rationally set about trying to destroy humanity, just for the lulz.

This seems extremely unlikely. Some humans like making things bad for other people. Those people don't generally want to destroy the world, because the world is where their toys are. Moreover, destroying humanity is something that takes a lot of effort. Barring making bad AGI, getting hold of a large nuclear arsenal, engineering a deadly virus, or making very nasty nanotech, humans don't have many options for any of these. All of these are very tough. And people who are doing scientific research are generally (although certainly not always) people who aren't getting much recognition and are doing it because they want to learn and help humanity. The people likely to even want to cause large scale destruction don't have much overlap with the people who have the capability. The only possible exceptions to this might be some religious and nationalist fanatics in some countries, but that's not a problem of rationality, and even they can't trigger existential risk events.

Dangerous information discovered: e.g. the rationalist community develops a Theory of Everything that reveals a recipe for a physics disaster (e.g. a cheap way to turn the Earth into a block hole). A non-rationalist decides to exploit this.

This isn't a rationalist-community worry. This is a general worry. As technology improves, people individually have more destructive power. That's a problem completely disconnected from rationalists. Even if such improved rationality did lead to massive tech leaps, it is rarely general theories that immediately give nasty weapons, but rather sophisticated and fairly complicated applications of them along with a lot of engineering. In 1939 the basic theory for atomic weapons existed, but they were developed in secret.

And people are starting to look at fixing it. I'm not worried that their attempts - and mine - will fail. At least we'd have fun and learn something.I'm worried that they will succeed.

Your concerns are misplaced. I think you're a consequentialist, so perhaps instead of thinking about it as "failing to start effective rationalist communities" you should think of it as "shattering all existing rationalist communities, leaving a handful of uncoordinated splinters whose influence on the world is minimal." The transposition is bad for morale, but good for getting over status quo bias.

Trading off probability of success against risk reduction is reasonable to consider, but after only a little considering I strongly believe that all of your proposals except for "1. Ignore" go way too far.

I would be worried about a movement that aims to make all of humanity more effective at everything they do.

It seems like the dangerous thing isn't so much widespread rationality so much as widespread superpowerful tools. Like, to the point that you can easily build a nuke in your garage, or staple together an AI from other available programs.

I'm not too too worried about people getting better at achieving their goals, because most people at least understand that they don't want to murder their friends and family. Like, a human tasked with eliminating malaria wouldn't think that killing all life on Earth would be the best way to accomplish that.

I'd prefer for most people to be better at achieving their goals, even if those goals x-risk reduction. Like, it would be nice if churches trying to help raise the quality of life of poor people did better at doing so.

In some areas (like status), everyone getting better at accomplishing their goals would probably be zero-sum, but its already that way. As it stands, I would like for rationalists to increase their status in order to better get things done, but once everyone's a rationalist I don't have strong preferences for who has more status than who. So its not that big of a problem.

Some people are often mentioned as wanting destruction (suicide bombers, for instance). Gwern posted a nice article about how ineffective they are, and it seems like becoming more rational (and thus happier, more well-adjusted) would probably draw them out of terrorism.

I have some optimism that a sort of "organizational takeoff" might occur, though if it does I expect it will happen in an organization which can convert its constituents' rationality into money in order to grow. Such an organization is likely to look quite unlike LW, and it may interact with the public in a very different way than you envision (for example, it may recruit widely but selectively, and may not publish anything at all).

But given that humanity already has technology with slight existential threat implications (nuclear weapons, rudimentary AI research), I would be worried about a movement that aims to make all of humanity more effective at everything they do.

We already have Science...

The idea being that anyone who understands rationality will also understand that it must be used for good.

I thought you were going to say that the do-good ideas would repulse rationalists with bad goals. That seems reasonably likely.