Dr. Jubjub predicts a crisis

I felt that the nonsense words in this post were a good idea, both in avoiding explicit references to mindkilling subjects, and because they sound cute.

Business as usual for the next 300 years with, say, 3% growth per year makes us around seven thousand times richer. This can't happen without technologies that are so disruptive that we don't have business as usual.

Even if we don't get AI, eugenics or intelligence enhancement which creates lots of people at least as smart as John von Neumann would give us a massively improved capacity to solve problems.

Maybe we'll have the capacity for AI in 50 years - but need another 500 years to make it friendly.

Unfortunately, if that's true, we all lose.

Agreed that we should treat the chances of a non-singularity future as being significant, but even with no singularity technological advance is a normal part of our society. Bandersnatch can be right even if Jabberwock is wrong.

Are you thinking on the margin? You don't control the priority function for humanity, only for yourself and those you influence. Even if slithiness were the most pressing issue, it doesn't follow that everyone should be working on it.

The question is why should we care about slithy toves? How high is the utility of protecting them?

You need to answer those questions to get me to care about slithy toves.

And besides, people won’t stand for it – slithy toves with unwhiffled tulgey wood are a part of our way of life.

In general culture changes when there a need for the change. You don't have to worry to much about technology being "socially acceptable" to prevent day values.

Prof. Bandersnatch: Of course! Prof. Jabberwock assures me the singularity will be here around tea-time on Tuesday. That is, if we roll up our sleeves and don’t waste time with trivialities like your tove issue.

I don't think that's the case. As far as I remember from the last survey the average LW participant predicts singularity after 2100. I see very few people arguing that we shouldn't fight aging because we will have singularity before it matters.

But my natural inclination is to take the Jubjub view. I think the chances of a basically business-as-usual future for the next 200 or 300 years are not epsilon

It depends on what you mean with business-as-usual. In general history shows us that lot of changes do happen and things don't stay constant.

In this respect a hypothetical positive future works exactly opposite to a hypothetical negatvie future (apocalypse). The later causes action now. The point is: If you lack a risk/cost-model model than the more powerful future wins. Here obviously the singularity wins over e.g. climate change.

Compare to http://xkcd.com/989/ (if all think a great future comes it will not come. kind of selfavoiding prephecy)

On any problem that won’t become an absolute crisis in the next few decades, someone will take the Bandersnatch view that it will be more easily solved later (with cheaper or more socially acceptable technology) so we shouldn’t work directly on it now. The way out is forward - let’s step on the gas and get to the finish line before any annoying problems catch up with us.

Ah, the black and white fallacy. The approach more likely to succeed is to start mapping out and working toward the potential solutions, among other things, dedicating proportionately more resources to the more immediate and dangerous problems, while still keeping in mind the long-term issues. There are exceptions, of course, but the Adam's law of slow-moving disasters seems to hold pretty well.

If we assume a scenario without AGI and without a Hansonian upload economy, it seems quite likely that there are large currently unexpected obstacles for both AGI and uploading. Computing power seems to be just about sufficient right now (if we look at supercomputers), so it probably isn't the problem. So it will probably be a conceptual limitation for AGI and a scanning or conceptual limitation for uploads.

Conceptual limitation for uploads seems unlikely, because were just taking a system cutting it up into smaller pieces and and solving differential equations on a computer. Lots of small problems to solve, but no major conceptual ones. We could run into problems related to measuring quantum systems when doing the scanning (I believe Scott Aaronson wrote something about this suspicion lately). Note that this also puts a bound on the level of nano-technology we could have achieve, if we have neuron-sized scanning robots, we would be able to scan a brain and start the Hansonian scenario. Note that this does not preclude slightly larger scale manufacturing technologies, which would probably come from successive miniaturisations of 3d-printers.

Conceptual difficulties creating AGI are more or less expected by everyone around here, but in the case AGI is delayed by over a century we should get quite worried about other existential risks on our way there. Major contenders are global conflict and terrorism, especially involving nuclear, nano-technological or biological weapons. Even if nano-technology will not reach the level described in Sci-Fi, the bounds given above still allow for sufficient development to make advanced weapons be a question of blueprints and materials. Low probability huge impact risks from global warming are also worth mentioning, if only to note that there are a lot of other people working on them.

What does this tell us about analysing long-term risks like the slithy toves? Well I don't know anything about slithy toves, but let's look at the eugenics stuff discussed earlier and consider how it would influence the probability of major global conflicts, the question is not whether it would increase the risk of global conflict, but how much it would increase the risk of global conflict. On the other hand if AI-safety is already taken care of, it becomes a priority to develop AGI as soon as humanly possible. And then it would be really good if humanly possible was a sigma or so better than today. Still it wouldn't be great, since most of the risks we would be facing at this point would be quite small for each year (as it seems today we could of course get other info on our way there). It's really quite hard to say what would be the proper balance between more intelligent people and more time available at this point, we could say that if we've already had a century to solve the problem more time can't be that useful, on the other hand we could say that if we still haven't solved the problem in a century there are loads of sequential steps to get right we need all the time we can buy.

tldr: No AGI & No Uploads => most X-risk from different types of conflict => eugenics or any kind of superhumans increases X-risk due to risk of war between enhanced and old-school humans