The mind-killer

It's not obvious that the best way to reduce existential risk is to actually work on the problem. Imagine if every farmer put down his plow and came to the university to study artificial intelligence research. Everyone would starve. It may well be that someone's best contribution is to continue to write software to do billing for health insurance, because that helps keep society running, which causes increased wealth, which then funds and supports people who specialize in researching risks among other fields.

I suspect that actually, only a small percentage of people, even of people here, could usefully learn the political truths relevant to existential risk mitigation via the kind of discussion you are proposing. Very few people are in a position to cause political change. The marginal utility gain for the average person to learn the truth on a political matter is practically zero due to his lack of influence on the political process. The many arguments against voting apply to this question as well, of seeking political truth; and even more so, because it's harder to ascertain political truths than to vote.

Most interest in politics is IMO similar to interest in sports or movies. It's fun, and it offers an opportunity to show off a bit, gives something to talk and socialize about, helps people form communities and define their interests. But beyond these kinds of social goals, there is no true value.

Most of the belief that one is in a position where knowing political truths is important, is likely to be self-deception. We see ourselves as being potentially more important and influential than we are likely ever to become. This kind of bias has been widely documented in many fields.

To me, politics is not so much the mind-killer as the mind-seducer. It leads us to believe that our opinions matter, it makes us feel proud and important. But it's all a lie. Politics is a waste of time and should be viewed simply as a form of entertainment. Now entertainment can be good, we all need a break from serious work and politics may be as valid as any other form of recreation, but we here should recognize that and not inflate its importance.

I agree with ciphergoth that we would probably have an easier time discussing political issues than some other communities, and I agree with HalFinney that it's probably not a very good use of our time anyway. Let's say that everyone on LessWrong agrees on a solution to some political problem. So what? We already have lots of good ideas no one will listen to. It doesn't take a long-time reader of Overcoming Bias to realize marijuana criminalization isn't working so well, but so far the efforts of groups with far more resources than ourselves have been mostly in vain.

If someone came up with a new idea for pulling ropes sideways, that might be useful. For example, Robin's idea of futarchy is interesting, although so large-scale that it would be very hard to implement. If someone came up with a suggestion that brilliant, but for a smaller problem, it might do some good. But I always interpreted our posting policies to permit that sort of thing anyway.

But I think the best thing we could possibly do would be to raise the sanity waterline - a rising tide lifts all boats. That means coming up with compact, attractive summaries of our key findings and spreading them as far as possible. More on this later, possibly.

"We already have lots of good ideas no one will listen to."

this is my primary thought on all such sentiments. the best thing for people here to do would probably be to stop worrying about altruism and start trying to get rich. Once you're rich your altruism will actually mean something.

what traps can we defuse in advance?

We care about saving the world and we care about the truth, so sometimes we start caring too much about the ideas that we think represent those things. How can we foster detachment? How can we encourage people to consider an idea even if they don't like it, and then encourage people to relinquish an idea after it's been considered and evenly rejected?

The following paradigm has worked for me:

It's natural to be afraid of considering an idea that we know is false. Thus it is useful to occasionally practice considering ideas that we don't like in order to find that nothing bad happens to our brains or ourselves when we consider them.

It is really important to be able to consider bad ideas, not just because the bad idea might be a good idea, but because it is only through empathy (identification) with an idea that you will be able to find the right counter-argument that will encourage the holder of the idea to relinquish it. (Otherwise, as I'm sure you've observed, the same argument just gets presented another way, another time.)

Survey: are you motivated to improve or save the world?

This survey aims to determine if there is significant consensus or disparity. It is in response to the datapoint presented here.

If you would like to qualify or explain your response, feel free to do so as a comment to the appropriate response.

• Note that this is a general solution to the problem of conducting a quick off-the-cuff survey on LW without affecting karma, but you need to be able to view negative scoring comments.

• If you want to leave me with positive karma, please keep the survey neutral and vote the parent (this) up instead.

Here's Wikipedia's list of Forbidden Words, which I think has some good examples of how language can be subtly loaded on controversial / emotionally charged issues. Diligently watching out for that sort of thing is probably one of the best things we could do to avoid political discussions degenerating.

I think it will be very necessary to carefully frame what it would be that we might wish to accomplish as a group, and what not. I say this because I'm one of those who thinks that humanity has less than a 50% chance of surviving the next 100 years, but I have no interest in trying to avert this. I am very much in favour of humanity evolving into something a lot more rational than what it is now, and I don't really see how one can justify saying that such a race would still be 'humanity'. On the other hand, if the worry is the extinction of all rational thought, or the extinction of certain, carefully chosen, memes, I might very well wish to help out.

The main problem, as I see it, is in being clear on what we want to have happen (and what not) and what we can do to make the preferred outcomes more likely. The more I examine the entire issues, the harder it appears to define how to distinguish between the good and the bad outcomes.

If politicians start following expected utility consequentialism, special interest groups will be able to exploit the system by manufacturing in themselves "offense" (extreme emotional disutility) at unfavored measures, forcing your maximizer to give in to their demands. To avoid this, you need a procedure for distinguishing "warranted" offense from "unwarranted" offense: some baseline of personal rights ultimately derived from something other than self-assessed emotional utility.

If you see a way around this difficulty, let me know, because it seems insurmountable to me right now. Until we sort this out, I find it hard to talk about politics from a consequentialist standpoint, because most successful interest groups today are already heavily using the exploit I've described.

Come on, Ciphergoth, the problem of saving humanity would be too easy if you could convince a large number of humans to go along with your proposals! You have a harder challenge: save humanity in spite of the apathy, and in many cases intransigent opposition, of the humans.

I have a hard time believing that anyone in power is serious about saving humanity. There are so many obvious and easy things that could be done, that would clearly be enormously helpful, that no one with power is doing or even suggesting. Politics is almost entirely a signalling game.

As several people have been starting to suggest, it's likely that the best strategy for rationalists involves two steps:

• accumulate massive wealth and power
• use it to save humanity

And probably the first step is far harder than the second.

I will admit to an estimate higher than 95% that humanity or its uploads will survive the next hundred years. Many of the "apocalyptic" scenarios people are concerned about seem unlikely to wipe out all of humanity; so long as we have a breeding population, we can recover.

My impression is that the material covered on OB/LW is more than sufficient to allow people that really understand the material to talk politics without exploding. I don't think we need any politics specific tricks for those that are likely to be helpful contributors.

This came up in the Santa Barbara LW meetup, and I felt like that group could have talked politics the right way. The implicit consensus seemed to be "Yeah, it'd probably work", though we didn't try.

Of course, with a smaller group and stronger selection pressures it is less likely to have a group member or two that can't hold it together when compared to a larger and online community (ie LW).

To summarize: I think there are plenty of people here that can handle it, but probably enough that can't to ruin it if there aren't any preventative measures. Hopefully a merciless downvoting policy will be enough, but failing that, I'm sure we could come up with a way to select a subset of people that are allowed to talk politics.

If we here can't do better than that, then this whole rationality discussion we've been having comes to no more than how we can best get out of bed in the morning, solve a puzzle set by a powerful superintelligence in the afternoon, and get laid in the evening.

And you say that like it is a bad thing! The possibility of creating just such a utopia sounds like a damn good motivating influence for concerted altruistic effort and existintial risk mitigation to me!

I think few here would give an estimate higher than 95% for the probability that humanity will survive the next 100 years; plenty might put a figure less than 50% on it.

For the record I would put it at levels overwhelmingly higher than 95%. More like 99.999%.

One observation and a related suggestion:

(1) We've gone off-topic regarding the demands of this post. Ciphergoth asks what traps can we defuse in advance, before we start to talk about specific ideas to do with what one does in order to change the world. However, I'm neutral about not following instructions -- perhaps Ciphergoph hasn't asked the right question after all, and we need to triangulate towards the right question.

(2) I've got no idea how to begin answering some of the other problems that are being posed. (E.g., how can we best help the world?) So I like Vladimir_Nesov's reminder that there is a post about the value of not proposing solutions until the problem has been discussed as thoroughly as possible. I think we might try it to see if it works in practice. So let's talk about the problem/problems. What is it / what are they?

Richard Posner on the economics of the flu epidemic:

We need an overall "catastrophe budget" that would match expenditures to the net expected benefits of particular measures targeted at particular catastrophic threats.

If we here can't do better than that, then this whole rationality discussion we've been having comes to no more than how we can best get out of bed in the morning, solve a puzzle set by a powerful superintelligence in the afternoon, and get laid in the evening.

Sounds like good work if you can get it. ;-)

More seriously, though, if you can't handle the getting out of bed part, it seems like taking on much bigger tasks might be off the agenda. And if more people were getting laid in the evening, we might have less violent conflict in the world.

But I'm definitely with you on the superintelligence puzzle solving being a bit less important. ;-)

There has to be a way for rationalists to talk about it and actually make a difference. Before we start to talk about specific ideas to do with what one does in order to change or save the world, what traps can we defuse in advance?

The first trap is assuming that having good ideas or being able to talk about them has anything to do with getting others to go along with them. For that, you need to be able to understand and expect to deal with irrational hidden agendas, not open rational discussion. If you can't deal with the former, the latter isn't going to help much.

So if you place any non-negligible value on future generations whose existence is threatened, reducing existential risk has to be the best possible contribution to humanity you are in a position to make.

This sentence smuggles in the assumption that we are in a position to reduce existential risk.

Two big risk are global warming and nuclear war.

The projections for large changes in climate depend on continuing growth in wealth and population in order to get the high levels of carbon dioxide emission needed to create the change. If it really goes horribly wrong, we are still looking at a self limiting problem with billions dying but billions living on in nuclear powered prosperity at higher latitudes. It is not an existential risk.

Nuclear war in the next hundred years is shaping up to be 2nd rank nations duking it out with 20kilo-ton fission weapons, not 200kilo-ton fusion weapons. That is enough to change building codes in ways that make current earthquake precautions seem cheap, but it is a long way short of an existential risk.

Existential risk might be large, but it comes from the unkown unkowns, not the known unkowns, and not even kowning that we don't know there is nothing useful we can do beyond maintaining a willingness to recognise a new danger if it makes its possibility known.