Beyond the Reach of God

I don't think that even Buddhism allows that.

Of course, 'utter destruction' is not a well-defined term. Depending on who you ask, nothing in Buddhism is ever actually destroyed. Or in the Dust hypothesis, or the Library of Babel... the existence of the mind never ends, because we've never beaten our wives in the first place.

I live with this awareness.

Summary: "Bad things happen, which proves God doesn't exist." Same old argument that atheists have thrown around for hundreds, probably thousands, of years. The standard rebuttal is that evil is Man's own fault, for abusing free will. You don't have to agree, but at least quit pretending that you've proven anything.

"Conway's Life has been proven Turing-complete, so it would be possible to build a sentient being in the Life universe"

Bit of a leap in logic here, no?

"In sober historical fact", clear minds could already see in 1919 that the absurdity of the Treaty of Versailles (with its total ignorance of economic realities, and entirely fueled by hate and revenge) was preparing the next war -- each person (in both nominally winning and nominally defeated countries) being put in such unendurable situations that "he listens to whatever instruction of hope, illusion or revenge is carried to him on the air".

This was J.M. Keynes writing in 1919, when A. Hitler was working as a police spy for the Rechswehr, infiltrating a tiny party then named DAP (and only later renamed to NDA); Keynes' dire warnings had nothing specifically to do with this "irrelevant" individual, which he had no doubt never even heard about -- there were plenty of other matches ready to set fire to a tinderbox world, after all; for examle, at that time, Benito Mussolini was a much more prominent figure, a well known and controversial journalist, and had just founded the "Fasci Nazionali di Combattimento".

So your claim, that believing the European errors in 1919 made another great war extremely likely, "is an unreasonable belief", is absurd. You weaken your interesting general argument by trying to support it with such tripe; "inevitable" is always an overbid, but to opine that the situation in 1919 made another great war all too likely within a generation, quite independently of what individuals would be leading the various countries involved, is perfectly reasonable.

Keynes's strong and lucid prose could not make a difference in 1919 (even though his book was a best-seller and may have influenced British and American policies, France was too dead-set in its hate and thirst for revenge) -- but over a quarter of a century later, his ideas prevailed: after a brief attempt to de-industrialize Germany and push it back to a pastoral state (which he had already argued against in '19), ironically shortly after Keynes' death, the Marshall Plan was passed (in rough outline, what Keynes was advocating in '19...) -- and we didn't get yet another great european war after that.

Without Hitler, but with Versailles and without any decent reconstruction plan after the Great War, another such great war WAS extremely likely -- it could have differed in uncountable details and even in strategic outline, from the events as they actually unfolded, just like the way a forest fire in dry and thick woods can unfold in many ways that differ in detail... but what exact match or spark lights the fire is in a sense a detail -- the dry and flame-prone nature of the woods makes a conflagration far too likely to avoid it by removing one specific match, or spark: there will be other sparks or matches to play a similar role.

The claim isn't that Germany would have been perfectly fine, and would never have started a war or done anything else extreme. And the claim is not that Hitler trashed a country that was ticking along happily.

The claim is that the history of the twentieth century would have gone substantially differently. World War II might not have happened. The tremendous role that Hitler's idiosyncrasies played in directing events, doesn't seem to leave much rational room for determinism here.

Reminds me of this: "Listen, and understand. That terminator is out there. It can't be bargained with. It can't be reasoned with. It doesn't feel pity, or remorse, or fear. And it absolutely will not stop, ever, until you are dead."

But my question would be: Is the universe of cause and effect really so less safe than the universe of God? At least in this universe, someone who has an evil whim is limited by the laws of cause and effect, e.g. Hitler had to build tanks first, which gave the allies time to prepare. In that other universe, Supreme Being decides he's bored with us and zap we're gone, no rules he has to follow to achieve that outcome.

So why is relying on the goodness of God safer than relying on the inexorability of cause and effect?

Given how widespread white nationalism is in America, (i.e. it's a common phenomenon) and how intimately tied to fascism it is, I think that there's a substantial chance that the leader that would have taken Hitler's place would have shared his predilection for ethnic cleansing, even if not world domination.

It looks more and more like all of this 'Friendly AI' garbage is just a reaction to Eliezer's departure from Judaism.

Which is not to say that this hasn't been obvious for a long time. But this is the closest Eliezer's ever come to simply acknowledging it.

He's already come to terms with the fact that reality doesn't conform to his ideas of what it should be. Now he just has to come to terms with the fact that his ideas do not determine what reality should be.

There isn't going to be a magical sky daddy no matter what we do. There's no such thing as magic. There's no such thing as 'Friendly'. It's not possible to make reality a safe and cozy haven, and trying to make it so will only cause harm.

"I don't think that even Buddhism allows that."

Remove whatever cultural or personal contextual trappings you find draped over a particular expression of Buddhism, and you'll find it very clear that Buddhism does "allow" that, or more precisely, un-asks that question.

As you chip away at unfounded beliefs, including the belief in an essential self (however defined), or the belief that there can be a "problem to solved" independent of a context for its specification, you may arrive at the realization of a view of the world flipped inside-out, with everything working just as before, less a few paradoxes.

The wisdom of "adult" problem-solving is not so much about knowing the "right" answers and methods, but about increasingly effective knowledge of what doesn't work. And from the point of view of any necessarily subjective agent in an increasingly uncertain world, that's all there ever was or is.

Certainty is always a special case.

By the way, I should clarify that my total disagreement with your thesis on WW2 being single-handedly caused by A. Hitler does in no way imply disagreement with your more general thesis. In general I do believe the "until comes steam-engine-time" theory -- that many macro-scale circumstances must be present to create a favorable environment for some revolutionary change; to a lesser degree, I also do think that mostly, when the macro-environment is ripe, one of the many sparks and matches (that are going off all the time, but normally fizz out because the environment is NOT ripe) will tend to start the blaze. But there's nothing "inevitable" here: these are probabilistic, Bayesian beliefs, not "blind faith" on my part. One can look at all available detail and information about each historical situation and come to opine that this or that one follows or deviates from the theory. I just happen to think that WW2 is a particularly blatant example where the theory was followed (as Keynes could already dimly see it coming in '19, and he was NOT the only writer of the time to think that way...!); another equally blatant example is Roman history in the late Republic and early Empire -- yes, many exceptional individuals shaped the details of the events as they unfolded, but the nearly-relentless march of the colossus away from a mostly-oligarchic Republic and "inevitably" towards a progressively stronger Principate looms much larger than any of these individuals, even fabled ones like Caesar and Octavian.

But for example I'm inclined to think of more important roles for individuals in other historically famous cases -- such as Alexander, or Napoleon. The general circumstances at the time of their accessions to power were no doubt a necessary condition for their military successes, but it's far from clear to me that they were anywhere close to sufficient: e.g., without a Bonaparte, it does seem quite possible to me that the French Revolution might have played itself out, for example, into a mostly-oligarchic Republic (with occasional democratic and demagogic streaks, just like Rome's), without foreign expansionism (or, not much), without anywhere like the 20 years of continuous wars that in fact took place, and eventually settling into a "stable" state (or, as stable as anything ever is in European history;-). And I do quite fancy well-written, well-researched "alternate history" fiction, such as Turtledove's, so I'd love to read a novel about what happens in 1812 to the fledgling USA if the British are free to entirely concentrate on that war, not distracted by Napoleon's last hurrahs in their backyard, because Napoleon was never around...;-) [To revisit "what if Hitler had never been born", btw, if you also like alternate history fiction, Stephen Fry's "Making History" can be recommended;-)]

After Napoleon, France was brought back to the closest status to pre-Revolutionary that the Powers could achieve -- and ("inevitably" one might say;-) 15 years later the Ancien Regime crumbled again; however, that time it gave birth somewhat peacefully to a bourgeois-dominated constitutional monarchy (with no aggressive foreign adventures, except towards hopefully-lucrative colonies). Just like the fact that following Keynes' 1919 advice in 1947 did produce lasting peace offers some support to Keynes' original contention, so the fact that no other "strong-man" emerged to grab the reins in 1830 offers some support to the theory that there way nothing "inevitable" about a military strong man taking power in 1799 -- that, had a military and political genius not been around and greedy for power in '99, France might well have evolved along different and more peaceful lines as it later did in '30. Of course, one can argue endlessly about counterfactuals... but should have better support before trying to paint a disagreement with oneself as "absurd"!-)

BTW, in terms of human death and suffering (although definitely not in terms of "sheer evil" in modern ethical conception), the 16 years of Napoleon's power were (in proportion to the population at the time) quite comparable to, or higher than, Hitler's 12; so, switching from Hitler to Napoleon as your example would not necessarily weaken it in this sense.

I thought I already knew all this, but this post has made me realize that I've still, deep down, been thinking as you describe - that the universe can't be that unfair, and that the future isn't really at risk. I guess the world seems like a bit scarier of a place now, but I'm sure I'll go back to being distracted by day-to-day life in short order ;).

As for cryonics, I'm a little interested, but right now I have too many doubts about it and not enough spare money to go out and sign up immediately.

With all the sci fi brought up here, I think we are familiar with Hitler's Time Travel Exemption Act.

Ian C., that is half the philosophy of Epicurus in a nutshell: there are no gods, there is no afterlife, so the worst case scenario is not subject to the whims of petulant deities.

If you want a sufficient response to optimism, consider: is the probability that you will persist forever 1? If not, it is 0. If there is any probability of your annihilation, no matter how small, you will not survive for an infinite amount of time. That is what happens in an infinite amount of time: everything possible. If all your backup plans can fail at once, even at P=1/(3^^^3), that number will come of eventually with infinite trials.

What's the point of despair? There seems to be a given assumption in the original post that:

1) there is no protection, universe is allowed to be horrible --> 2)lets despair

But number 2 doesn't change 1 one bit. This is not a clever argument to disprove number 1. I'm just saying despair is pointless if it changes nothing. It's like when babies cry automatically when something isn't the way they like because they are programmed to by evolution because this reliably attracted the attention of adults. Despairing about the universe will not attract the attention of adults to make it better. We are the only adults, that's it. I would rather reason along the lines of:

1) there is no protection, universe is allowed to be horrible --> 2)what can I do to make it better

Agreed with everything else except the part where this is really sad news that's supposed to make us unhappy.

I don't understand the faith in cryonics.

In a Universe beyond the reach of God, who is to say that the first civilization technologically advanced enough to revive you will not be a "death gives meaning to life" theocracy which has a policy of reviving those who chose to attempt to escape death in order to submit them and their defrosted family members to 1000 years of unimaginable torture followed by execution?

Sure, there are many reasons to believe such a development is improbable. But you are still rolling those dice in a Universe beyond God's reach, are you not?

Of course you are. It's still a probability game. But Eliezer's contention is that the probabilities for cryonics look good. It's worth rolling the dice.

Yes very very bad things can happen for little reason. But of course we still want positive arguments to convince us to assign large probabilities to scenarios about which you want us to worry.

Where is this noirish Eliezer when he's writing about the existence of free will and non-relativist moral truths?

Don't get bored with the small shit. Cancers, heart disease, stroke, safety engineering, suicidal depression, neurodegenerations, improved cryonic tech. In the next few decades I'm probably going to see most of you die from that shit (and that's if I'm lucky enough to persist as an observer), when you could've done a lot more to prevent it, if you didn't get bored so easily of dealing with the basics.

Kip, the colors of rationality are crystal, mirror, and glass.

Robin, fair enough; but conversely no amount of argument will convince someone in zettai daijobu da yo mode.

For the benefit of those who haven't been following along with Overcoming Bias, I should note that I actually intend to fix the universe (or at least throw some padding atop my local region of it, as disclaimed above) - I'm not just complaining here.

"If you want a sufficient response to optimism, consider: is the probability that you will persist forever 1? If not, it is 0. If there is any probability of your annihilation, no matter how small, you will not survive for an infinite amount of time. That is what happens in an infinite amount of time: everything possible. If all your backup plans can fail at once, even at P=1/(3^^^3), that number will come of eventually with infinite trials." Zubon, this seems to assume that the probabilities in different periods are independent. It could be that there is some process that allows the chance of obliteration to decline exponentially (it's easy enough to imagine a cellular automata world where this could be) with time, such that the total chance of destruction converges to a finite number. Of course, our universe's apparent physics seem to preclude such an outcome (the lightspeed limit, thermodynamics, etc), but I wouldn't assign a probability of zero to our existing as simulations in a universe with laws permitting such stability, or to (improbable) physics discoveries permitting such feats in our apparent universe.

Without Hitler it's likely Ludendorf would have been in charge and things would have been even worse. So perhaps we should be grateful for Hitler!

I gather there are some Orthodox Jews involved in Holocaust denial and were in Iran for that, but this post gets me to thinking that there should be more of them if they really believe in a benevolent and omnipotent God that won't allow sufficiently horrible things to happen.

How widespread is white nationalism in America? I would think it's one of the least popular things around, although perhaps I'm taking the Onion too seriously.

"The standard rebuttal is that evil is Man's own fault,"

There is no evil. There is neutrality. The universe isn't man's fault; it isn't anyone's fault.

I'm not at all saddened by these facts. My emotional state is unaltered. It's because I take them neutrally.

I've experienced severe pain enough to know that A) Torture works. Really. It does. If you don't believe it, try it. It'll be a short lesson. B) Pain is not such a big deal. It's just an avoid-this-at-all-cost -signal. Sure, I'm in agony, sure, I'd hate to remain in a situation where that signal doesn't go away, but it still is just a signal.

Perhaps as you look at some spot in the sky, they've already - neutrality allowing - tamed neutrality there; made it Friendly.

We've got a project to finish.

More parents might let their toddler get hit by a car if they could fix the toddler afterwards.

There are an awful lot of types of Buddhism. Some allow mind annihilation, and even claim that it should be our goal. Some strains of Epicurianism hold that mind annihilation is a) neutral, and b) better than what all the religions believed in. Some ancient religions seemed to believe in the same awful universal fate as quantum immortality believers do, e.g. eternal degeneration, progressively advanced Alzheimers forever more or less. Adam Smith suggests that this is what most people secretly believe in.

It would take quite a black swan tech to undo all the good from tech up to this point. UFAI probably wouldn't pass the test, since without tech humans would go extinct with a smaller total population of lives lived anyway. Hell worlds seem unlikely. 1984 or the Brave New World (roughly) are a bit more likely, but is it worse than extinction? I don't generally feel that way, though I'm not sure.

Good post, but how to deal with this information so that it is not so burdensome: Conway himself, upon creating The Game of Life, didn't believe that the cellular automaton could 'live' indefinitely, but was proven wrong shortly after his games creation by the discovery of the glider gun. We cannot assume that the cards were dealt perfectly and the universe or our existence is infinite, but we can hope that the pattern we have put down will continue to stand the test of time. Belief that we are impervious to extinction or that the universe will not ultimately implode leaving everything within no choice but to be squished into a single particle can only do us harm as we try to create new things and discover ways to transcend this existence. Hope that we will make it and that there is ultimately a way off of what may be a sinking ship is what keeps us going.

I don't understand why the end of the universe bugs people so much. I'll just be happy to make it to next decade, thanks very much. When my IQ rises a few thousand points, I'll consider things on a longer timescale.

Would Camus agree with you Eliezer?

What I don't understand is that we live on a planet, where we don't have all people with significant loose change

A) signing up for cryonics B) super-saturating the coffers of life-extensionists, extinction-risk-reducers, and AGI developers.

Instead we currently live on a planet, where their combined (probably) trillions of currency units are doing nothing but bloating as 1s and 0s on hard drives.

Can someone explain why?

"What can a twelfth-century peasant do to save themselves from annihilation? Nothing."

She did something. She passed on a religious meme whose descendents have inspired me, in turn, to pass on the idea that we should engineer a world that can somehow reach backward to save her from annihilation. That may not prove possible, but some possibilities depend on us for their realization.

A Jewish prophet once wrote something like this: "Behold, I will send you Elijah the prophet before the coming of the great and dreadful day of the Lord: And he shall turn the heart of the fathers to the children, and the heart of the children to their fathers, lest I come and smite the earth with a curse." The Elijah meme has often turned my heart toward my ancestors, and I wonder whether we can eventually do something for them.

Unless we are already an improbable civilization, our probable future will be the civilization we would like to become only to the extent that such civilizations are already probable. The problem of evil is for the absolutely omnipotent God -- not for the progressing God.

Chad: if you seriously think that Turing-completeness does not imply the possibility of sentience, then you're definitely in the wrong place indeed.

And I do quite fancy well-written, well-researched "alternate history" fiction, such as Turtledove's, so I'd love to read a novel about what happens in 1812 to the fledgling USA if the British are free to entirely concentrate on that war, not distracted by Napoleon's last hurrahs in their backyard, because Napoleon was never around...

Nitpick:

The "War of 1812" was basically an offshoot of the larger Napoleonic Wars; Britain and France were both interfering with the shipping of "neutral" nations, such as the United States, in order to hurt their enemy. After France dropped its restrictions (on paper, at least), the United States became a lot less neutral, and James Madison and Congress eventually declared war on Britain. (Several years earlier, Jefferson, in response to the predations of the two warring European powers, got Congress to pass an Embargo Act that was to end foreign trade for the duration of the war, so as to keep the U.S. from getting involved. It didn't work out so well.)

In other words, without Napoleon, there probably wouldn't have been a War of 1812 at all.

What I don't understand is that we live on a planet, where we don't have all people with significant loose change

A) signing up for cryonics B) super-saturating the coffers of life-extensionists, extinction-risk-reducers, and AGI developers.

Instead we currently live on a planet, where their combined (probably) trillions of currency units are doing nothing but bloating as 1s and 0s on hard drives.

Can someone explain why?

Many people believe in an afterlife... why sign up for cryonics when you're going to go to Heaven when you die?

I should note that I actually intend to fix the universe [...]

I was not aware that the universe was broken. If so, can we get a replacement instead? ;-)

It is a strange thing. I often feel the impulse to not believe that something would really be possible - usually when talking about existential risks - and I have to make a conscious effort to suppress that feeling, to remind myself that anything the laws of physics allow is possible. (And even then, I often don't succeed - or don't have the courage to entirely allow myself to succeed.)

A) Torture works. Really. It does. If you don't believe it, try it. It'll be a short lesson.

That depends on what you're trying to use it for. Torture is very good at getting people to do whatever they believe will stop the torture. For example, it's a good way to get people to confess to whatever you want them to confess to. Torture is a rather poor way to get people to tell you the truth when they have motive to lie and verification is difficult; they might as well just keep saying things at random until they say something that ends the torture.

Consequentialist: Is it a fair universe where the wealthy live forever and the poor die in the relative blink of an eye? It seems hard for our current society to look past that when setting public policy. This doesn't necessarily explain why there isn't more private money put to the purpose, but I think many of the intelligent and wealthy at the present time would see eternal life quests as a millennial long cliche of laughable selfishness and not in tune with leaving a respectable legacy.

...Can someone explain why?

Many people believe in an afterlife... why sign up for cryonics when you're going to go to Heaven when you die?

That's probably not the explanation, since there are many millions of atheists who heard about cryonics and/or extinction risks. I figure the actual explanation is a combination of conformity, the bystander effect, the tendency to focus on short term problems, and the Silliness Factor.

I can only speak for myself on this, but wouldn't sign up for cryonics even if it were free, because I don't want to be revived in the future after I'm dead. (Given the choice, I would rather not have existed at all. However, although mine was not a life worth creating, my continued existence will do far less harm than my abrupt death.)

There's a corallary mystery category which most of you fall into: why are so few smart people fighting, even anonymously, against policy grounded in repugnancy bias that'll likely reduce their persistence odds? Where's the fight against a global ban on reproductive human cloning? Where's the fight to increase legal organ markets? Where's the defense of China's (and other illiberal nations)rights to use prisoners (including political prisoners) for medical experimentation? Until you square aware your own repugnancy bias based inaction, criticisms of that of the rest of population on topics like cryonics reads as incoherent to me as debating angels dancing on the heads of pins. My blog shouldn't be so anomolous in seeking to overcome repugnancy bias to maximize persistence odds. Where are the other anonymous advocates? Our reality is the Titanic -who want to go down with the ship for the sake of a genetic aesthetic -because your repugnancy bias memes are likely to only persist in the form of future generations if you choose to value it over your personal persistence odds.

To show that hellish scenarios are worth ignoring, you have to show not only that they're improbable, but also that they're improbable enough to overcome the factor (utility of oblivionish scenario - utility of hellish scenario)/(utility of heavenish scenario - utility of oblivionish scenario), which as far as I can tell could be anywhere between tiny and huge.

As for global totalitarian dictatorships, I doubt they'd last for more than millions of years without something happening to them.

HA,

"why are so few smart people fighting, even anonymously, against policy grounded in repugnancy bias that'll likely reduce their persistence odds?" Here's a MSM citation of Gene Expression today: http://www.theglobeandmail.com/servlet/story/LAC.20081004.WORDS04//TPStory/Science

Steve Sailer is also widely read among conservative (and some other) elites, and there's a whole network of anonymous bloggers associated with him.

"Where's the fight against a global ban on reproductive human cloning?" Such bans have been fought, primarily through alliance with those interested in preserving therapeutic cloning.

"Where's the fight to increase legal organ markets?" Smart people can go to Iran, where legal markets already exist.

"Where's the defense of China's (and other illiberal nations)rights to use prisoners (including political prisoners) for medical experimentation?" There are some defenses of such practices, but it's not obviously a high-return area to invest your energies in, given the alternatives. A more plausible route would be suggesting handy experiments to Chinese partners.

I can only speak for myself on this, but wouldn't sign up for cryonics even if it were free, because I don't want to be revived in the future after I'm dead.

I would probably sign up for cryonics if it were free, with a, "do not revive sticker" and detailed data about me so that future brain studiers would have another data point when trying to figure out how it all works.

I don't wish that I hadn't been born, but I figure I have a part to play a purpose that no one else seems to be doing. Once that has been done, then unless something I see need doing and is important and sufficiently left field for no one else to be doing, I'll just potter along doing random things until I die.

"I figure I have a part to play a purpose that no one else seems to be doing"

How do you figure that? Aren't you a materialist? Or do you just mean that you might find a niche to fill that would be satisfying and perhaps meaningful to someone? I'm having trouble finding a non-teleological interpretation of your comment.

"If you look at the rules for Conway's Game of Life (which is Turing-complete, so we can embed arbitrary computable physics in there), then the rules are really very simple. Cells with three living neighbors stay alive; cells with two neighbors stay the same, all other cells die. There isn't anything in there about only innocent people not being horribly tortured for indefinite periods."

While I of course I agree with the general sentiment of the post, I don't think this argument works. There is a relevant quote by John McCarthy:

"In the 1950s I thought that the smallest possible (symbol-state product) universal Turing machine would tell something about the nature of computation. Unfortunately, it didn't. Instead as simpler universal machines were discovered, the proofs that they were universal became more elaborate, and do did the encodings of information." (http://cs.nyu.edu/pipermail/fom/2007-October/012141.html)

One might add that the existence of minimalistic universal machines tell us very little about the nature of metaphysics and morality also. The problem is that the encodings of information gets very elaborate: a sentient being implemented in Life would presumably take terabytes of initial state, and that state would be encoding some complicated rules for processing information, making inferences, etc etc. It is those rules that you need to look at to determine if the universe is perfectly unfair or not.

Who knows, perhaps there is a deep fundamental fact that it is not possible to implement sentient beings in a universe where the evaluation rules don't enforce fairness. Or, slightly more plausible, it could be impossible to implement sentient tyrants who don't feel a "shade of gloom" when considering what they've done.

Neither scenario sounds very plausible, of course. But in order to tell whether such fairness constraints exist or not, the 3 rules of Life itself are completely irrelevant. This can be easily seen, since the same higher level rules could be implemented on top of any other universal machine equally easily. So invoking them do not give us any more information.

Doug, Will: There is no fundamental difference between being revived after dying, waking up after going to sleep, or receiving neurotransmitter in a synapse after it was released. There is nothing special about 10^9 seconds as opposed to 10^4 seconds or 10^-4 seconds. Unless, of course, these times figure into your morality, but these are considerations far out of scope of ancestral environments humans evolved in. This is a care where unnatural category meets unnatural circumstances, so figuring out a correct answer is going to be difficult, and relying on intuitively reinforced judgment would be reckless.

"So invoking them do not give us any more information."

I do think we get a little: if such constraints exist, they are a property of the patterns themselves, and not a property of the low-level substrate on which they are implemented. If such a thing were true in this world, it would be a property of people and societies, not a metaphysical property. That rules out a lot of religion and magical thinking, and could be a useful heuristic.

What probability do you guys assign to the god hypothesis being true?

You can't 'fix the universe'. You can at most change the properties of small parts of reality -- and that can only be accomplished by accepting and acting in accordance to the nature of reality.

If you don't like the nature of reality, you'd better try to change what you like.

What probability do you guys assign to the god hypothesis being true?

I don't want to sign up for cryonics because I'm afraid I will be revived brain-damaged. But maybe others are worried they will have the social status of a freak in that future society.

Great post and discussion. Go Team Rational!

Eliezer, I think there's a slight inconsistency in your message. On the one hand, there are the posts like this, which can basically be summed up as: "Get off your asses, slackers, and go fix the world." This is a message worth repeating many times and in many different ways.

On the other hand are the "Chosen One" posts. These posts talk about the big gaps in human capabilities - the idea being that some people just have an indefinable "sparkliness" that gives them the power to do incredible things. I read these posts with uneasiness: while agreeing with the general drift, I think I would interpret the basic observations (e.g. CEOs really are smarter than most other people) in a different way.

The inconsistency is that on the one hand you're telling people to get up and go do something, because the future is uncertain and could be very, very good or very, very bad; but on the other hand you're essentially saying that if a person is not a Chosen One, there's not much he can really contribute.

So, what I'd like to see is a discussion of what the rank-and-file members of Team Rational should be doing to help (and I hope that involves more than donating lots of money to SIAI).

Also, we need a cool mascot, maybe zebras.

"So, what I'd like to see is a discussion of what the rank-and-file members of Team Rational should be doing to help (and I hope that involves more than donating lots of money to SIAI)." How 'rank-and-file' are we talking here? With what skillset, interests, and level of motivation?

Writing papers like Nick Bostrom's can be valuable:

http://www.nickbostrom.com/

I have an analogy: "justice is like cake, it's permitted to exist but someone has to make it".

Can you be happier sheltering in ignorance? I'm not convinced. I think that's a strategy that only works while you're lucky.

It is extraordinarily difficult to figure out how to use volunteers. Almost any nonprofit trying to accomplish a skilled-labor task has many more people who want to volunteer their time than they can use. The Foresight Institute has the same problem: People want to donate time instead of money, but it's really, really hard to use volunteers. If you know a solution to this, by all means share.

I'm surprised by the commenters who cannot conceive of a future life that is more fun than the one they have now - who can't imagine a future they would want to stick around for. Maybe I should bump the priority of the Fun Theory sequence.

"The Foresight Institute has the same problem: People want to donate time instead of money, but it's really, really hard to use volunteers. If you know a solution to this, by all means share."

There's always Amazon's Mechanical Turk (https://www.mturk.com/mturk/welcome). It's an inefficient use of people's time, but it's better than just telling people to go away. If people are reluctant to donate money, you can ask for donations of books- books are actually a fairly liquid asset (http://www.cash4books.net/).

Eliezer: Does the law allow just setting them to productive but entirely tangential work, and pocketing the profit for SIAI?

@Hidden: just a "typical" OB reader, for example. I imagine there are lots of readers who read posts like this and say to themselves "Yeah! There's no God! If we want to be saved, we have to save ourselves! But... how...?" Then they wake up the next day and go to their boring corporate programming jobs.

@pdf23ds: This feels like tunnel vision. Surely the problem SIAI is working on isn't the ONLY problem worth solving.

@Eliezer: I recognize that it's hard to use volunteers. But members of Team Rational are not herd thinkers. They probably don't need to be led, per se - just kind of nudged in the right direction. For example, if you said, "I really think project X is important to the future of humanity, but it's outside the scope of SIAI and I don't have time to dedicate to it", probably some people would self-motivate to go and pursue project X.

The obvious example of a horror so great that God cannot tolerate it, is death - true death, mind-annihilation. I don't think that even Buddhism allows that.
This is sort of a surprising thing to hear from someone with a Jewish religious background. Jews spend very little attention and energy on the afterlife. (And your picture of Buddhism is simplistic at best, but other people have already dealt with that). I've heard the interesting theory that this stems from a reaction against their Egyptian captors, who were of course obsessed with death and the afterlife.

Religion aside, I truly have trouble understanding why people here think death is so terrible, and why it's so bloody important to deep-freeze your brain in the hopes it might be revved up again some time in the future. For one thing, nothing lasts forever, so death is inevitable no matter how much you postpone it. For another, since we are all hard-core materialists here, let me remind you that the flow of time is an illusion, spacetime is eternal, and the fact that your own personal self occupies a chunk of spacetime that is not infinite in any direction is just a fact of reality. It makes about as much sense to be upset that your mind doesn't exist after you die as it does to be upset that it didn't exist before you were born. Lastly, what makes you so damn important that you need to live forever? Get over yourself. After you die, there will be others taking over your work, assuming it was worth doing. Leave some biological and intellectual offspring and shuffle off this mortal coil and give a new generation a chance. That's how progress gets made -- "a new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it". (Max Planck, quoted by Thomas Kuhn)

Hopefully posthumans will be a little bit less stubborn in opposition to new scientific ideas.

"but on the other hand you're essentially saying that if a person is not a Chosen One, there's not much he can really contribute."

Do you think there aren't at least a few Neos whom Eliezer, and transhumanism in general, hasn't reached and influenced? I'm sure there are many, though I put the upper limit of number of people capable of doing anything worthwhile below 1M (whether they're doing anything is another matter). Perhaps the figure is much lower. But the "luminaries", boy, they are rare.

Millions of people are capable of hoovering money well in excess of their personal need. Projects aiming for post-humanity only need to target those people to secure unlimited funding.

"what makes you so damn important that you need to live forever? Get over yourself. After you die, there will be others taking over your work, assuming it was worth doing. Leave some biological and intellectual offspring and shuffle off this mortal coil and give a new generation a chance"

I vehemently disagree. What makes me so damn important, huh? What makes you so damn unimportant that you're not even giving it a try? The answer to both of these: You, yourself; you make yourself dman important or don't. Importance and significance are self-made. No one can give them to you. You must earn them.

There are damn important people. Unfortunately most of them were. Think of the joy if you could revive the best minds who've ever walked the earth. If you aren't one of them, try to become one.

Mtraven: "I truly have trouble understanding why people here think death is so terrible [...] [S]ince we are all hard-core materialists here, let me remind you that the flow of time is an illusion, spacetime is eternal [...]"

I actually think this one goes the other way. You choose to live right now, rather than killing yourself. Why not consistently affirm that choice across your entire stretch of spacetime?

"[W]hat makes you so damn important that you need to live forever?"

Important to whom?

Even if you're only capable of becoming an average, main sequence star, and not a quasistellar object outshining billions of others, what you must do is to become that star and not remain unlit. Oftentimes those who appear to shine brightly do so only because there's relative darkness around.

What if Eliezers weren't so damn rare; what if there were 100,000 x "luminaries"; which Eliezer's blog would you read?

"Important to whom?"
Important to the development of the universe. It's an open-ended project where we, its sentient part, decide what the rewards are, we decide what's important. I've come to the conclusion that optimizing, understanding, and controlling that which is (existence) asymptotically perfectly, is the most obvious goal. Until we have that figured out, we need to stick around.

"What if Eliezers weren't so damn rare"

The weird obsequiousness towards Eliezer makes yet another appearance on OB.

What the hell is supposed to be worth anything if life isn't?

Oh, and while I'm stirring up the pot, let me just say that this statement made me laugh: "But members of Team Rational are not herd thinkers." Dude. Self-undermining much?

Consequentialist: "I've come to the conclusion that optimizing, understanding, and controlling that which is (existence) asymptotically perfectly, is the most obvious goal."

You haven't been talking to Roko or Richard Hollerith lately, have you?

"The weird obsequiousness towards Eliezer makes yet another appearance on OB."

Quite the contrary. I'd prefer it be so that Eliezer is a dime a dozen. It's the relative darkness around that keeps him in the spotlight. Is suspect there's nothing special - in the Von Neumann sense - about this chap, just that I haven't found anyone like him so far. Care to point some others like him?

Eliezer, if that last comment was in response to mine it is a disappointingly obtuse misinterpretation which doesn't engage with any of the points I made. "Life" is worth something; that doesn't mean that striving for the infinite extension of individual lives should be a priority.

I'm surprised by the commenters who cannot conceive of a future life that is more fun than the one they have now - who can't imagine a future they would want to stick around for. Maybe I should bump the priority of the Fun Theory sequence.

I a different type of fun helping people perform a somewhat meaningful* task than I do when I am just hanging out, puzzle solving, adventure sports or going on holiday. I have a little nagging voice asking, "What was the point of that". Which needs to be placated every so often, else the other types of fun lose their shine.

If I'm revived into a culture that has sufficient technology to revive me, then it is likely that it will not need any help that I could provide. My choices would be, no longer be myself by radically altering my brain to be of some use to people or immerse myself in make work fantasy tasks. If I pick the first, it makes little difference to my current self whether it is radically altered me or someone else being useful. The second choice is also not appealing, it would require lying to myself sufficiently well in order to fool my pointfulness meter.

*Tasks that have real world consequences

Who knows, perhaps there is a deep fundamental fact that it is not possible to implement sentient beings in a universe where the evaluation rules don't enforce fairness. Or, slightly more plausible, it could be impossible to implement sentient tyrants who don't feel a "shade of gloom" when considering what they've done. Neither scenario sounds very plausible, of course.

The rule of thumb is: if you can imagine it, you can simulate it (because your brain is a simulator). The simulation may not be easy, but at least it's possible.

You name specific excuses for why life in the future will be bad for you. It sounds like you see the future as a big abandoned factory, where you are a shadow, and the strange mechanisms do their spooky dance. Think instead of what changes could make the future right specifically for you, with tremendous amount of effort applied to this goal. You are just a human, so attention your comfort can get starts far above the order of whole of humanity thinking about every tiny gesture to make you a little bit more comfortable for millions of years, and thinking about it from your own point of view. There is huge potential for improvement in the joy of life, just like human intelligence is nowhere near the ten orders of magnitude from the top, human condition is nowhere near the most optimal. You can't trust your estimate of how limited the future will be, how impossible it will be for the future to find a creative solution to your problem. Give it a try.

"The claim isn't that Germany would have been perfectly fine, and would never have started a war or done anything else extreme. And the claim is not that Hitler trashed a country that was ticking along happily.

The claim is that the history of the twentieth century would have gone substantially differently. World War II might not have happened. The tremendous role that Hitler's idiosyncrasies played in directing events, doesn't seem to leave much rational room for determinism here."

I disagree. Hitler did not departure very far from the general beliefs of the time. The brand of socialism and nationalism that became what Hitler preached had been growing in prominence for decades in academia and in the middle class consciousness. The alliance between the socialists and the conservatives against the liberals probably would have happened whether Hitler was at the top or not.

[sorry for ambiguity: thinking for millions of years, not making comfortable for millions of years]

It sounds like you see the future as a big abandoned factory, where you are a shadow, and the strange mechanisms do their spooky dance.

Think instead of what changes could make the future right specifically for you.

However a future without massive disparities of power and knowledge between myself and the inhabitants, would not be able to revive me from cryo sleep.

So you don't think you could catch up? If you had been frozen somewhere between -10000 and -100 years and revived now, don't you think you could start learning what the heck it is people are doing and understand nowadays? Besides a lot of the pre-freeze life-experience would be fully applicable to present. Everyone starts learning from the point of birth. You'd have headway compared to those who just start out from nothing.
There are things we can meaningfully contribute to even in a Sysop universe, filled with Minds. We, after all, are minds, too, which have the inherent quality of creativity - creating new, evermore elegant and intricate patterns - at whatever our level is, and self-improvement; optimization.

Bored? Got nothing meaningful to do? Try science.

This is a big do-it-yourself project. Don't complain about there not being enough opportunities to do meaningful things. If you don't find anything meaningful to do, that's your failure, not the failure of the universe. Searching for meaningful problems to solve is part of the project.

Correction: headway - I meant to say headstart.

I find Eliezer's (and many of the others here) total and complete obsession with the "God" concept endlessly fascinating. I bet you think about "God" more often than the large majority of the nominally religious. This "God" fellow has seriously pwn3d your wetware. . .

Giant cheesecake fallacy. If future could do everything you wanted to do, it doesn't mean it would do so. Especially if it will be bad for you. If future decides to let you work on a problem, even though it could solve it without you, you can't apply to the uselessness of your action: if future refuses to perform it, only you can make a difference. You can grow to be able to vastly expand the number of things you will be capable of doing, this source never dwindles. If someone or something else solved a problem, it doesn't necessarily spoil the fun for everyone else for all eternity. Or it was worth spoiling the fun, lifting poverty and disease, robbing you of the possibility of working on those things yourself. Take joy in your personal discoveries. Inequality can only be bad because you are not all you could be, not because there are things greater than you. Seek personal growth, not universal misery. Besides, doing something "inherently worthwhile" is only one facet of life, so even if there wouldn't be a solution to that, there are other wonders worth living for.

A "head start" in the wrong direction isn't much help.

Imagine a priest in the temple of Zeus, back in Ancient Greece. Really ancient. The time of Homer, not Archimedes. He makes how best to serve the gods the guiding principle of his life. Now, imagine that he is resurrected in the world of today. What do you think would happen to him? He doesn't speak any modern language. He doesn't know how to use a toilet. He'd freak out at the sight of a television. Nobody worships the gods any more. Our world would seem not only strange, but blasphemous and immoral, an abomination that ought to be destroyed. His "head start" has given him far more anti-knowledge than actual knowledge.

Of course, he'd know things about the ancient world that we don't know today, but, even twenty years after he arrived, would you hire him, or a twenty-year-old born raised in the modern world? From almost every perspective I could think of, it would be better to invest resources in raising a newborn than to recreate and rehabilitate a random individual from our barbaric past.

----

Me, to The Future: Will you allow me to turn myself into the equivalent of a Larry Niven-style wirehead?
The Future: No.
Me: Will you allow me to end my existence, right now?
The Future: No.
Me: Then screw you, Future.

Doug: From almost every perspective I could think of, it would be better to invest resources in raising a newborn than to recreate and rehabilitate a random individual from our barbaric past.

No, for him it won't be better. Altruistic aspect of the humane morality will help, even if it's more energy-efficient to incinerate you. For that matter, why raise a newborn child instead of making a paperclip?

In the interest of helping folks here to "overcome bias", I should add just how creepy it is to outside observers to see the unswervingly devoted members of "Team Rational" post four or five comments to each Eliezer post that consist of little more than homilies to his pronouncements, scattered with hyperlinks to his previous scriptural utterances. Some of the more level-headed here like HA have commented on this already. Frankly it reeks of cultism and dogma, the aromas of Ayn Rand, Scientology and Est are beginning to waft from this blog. I think some of you want to live forever so you can grovel and worship Eli for all eternity. . .

I'd like a future where people were on a level with me, so I could be of some meaningful use.

However a future without massive disparities of power and knowledge between myself and the inhabitants, would not be able to revive me from cryo sleep.

I already guessed that might be the wish of many people. That's one reason why I would like to acquire the knowledge to deliberately create a single not-person, a Very Powerful Optimization Process. What does it take to not be a person? That is one of those moral questions that runs into empirical confusions. But if I could create a VPOP that did not have subjective experience (or the confusion we name subjective experience), and did not have any pleasure or pain, or valuation of itself, then I think it might be possible to have around a superintelligence that did not, just by its presence, supersede us as an adult; but was nonetheless capable of guarding the maturation of humans into adults, and, a rather lesser problem, capable of reviving cryonics patients.

If there is anything in there that seems like it should be impossible to understand, then remember that mysteries exist in the map, not in the territory.

The only thing more difficult than creating a Friendly AI, involving even deeper moral issues, is creating a child after the right and proper fashion of creating a child. If I do not wish to be a father, I think that it is acceptable for me to avoid it; and certainly the skill to deliberately not create a child would be less than the skill to deliberately create a healthy child.

So yes I do confess: I wish to create something that has no value of itself, so that it can safeguard the future and our choices, without, in its own presence, automatically setting the form of humankind's adulthood according to the design decisions of a handful of programmers, and superseding our own worth. An emergency measure should do as little as possible, and everything necessary; if I can avoid creating a child I should do so, if only because it is not necessary.

The various silly people who think I want to keep the flesh around forever, or constrain all adults to the formal outline of an FAI, are only, of course, making things up; their imagination is not wide enough to understand the concept of some possible AIs being people, and some possible AIs being something else. A mind is a mind, isn't it? A little black box just like yours, but bigger. But there are other possibilities than that, though I can only see them unclearly, at this point.

A.R.: The standard rebuttal is that evil is Man's own fault, for abusing free will.

That only excuses moral evil, not natural evil.

I was not aware that the universe was broken. If so, can we get a replacement instead? ;-)

Britain is broken, but Cameron's on that case.

An emergency measure should do as little as possible, and everything necessary;

The VPOP will abolish, "Good bye," no?

It will obsolete or profoundly alter the nature of emergency surgery doctors, cancer researchers, fund raisers for cancer research, security services, emergency relief workers, existential risk researchers etc...

Every person on the planet who is trying to act somewhat like an adult will find they are no longer needed to do what is necessary. It doesn't matter that they are obsoleted by a process rather than a person, they are still obsolete. This seems like a step back for the maturation of humanity as a whole. It does not encourage taking responsibility for our actions. Life or life decisions are not so important, or meaningful.

You see it as the only way for humanity to survive, if I bought that I may support your vision, even though I would not necessarily want to live through it.

On a side note, do any of your friends jokingly call you Hanuman? The allusion is unfair, I'll grant you, he is far more insane and cruel than you, but the vision and motivation is eerily similar, on the surface details at least.

"Frankly it reeks of cultism and dogma,"

Oh, I wouldn't worry about that too much; that's a cunning project underway to enbias Eliezer with delusions-of-grandeur bias, smarter-than-thou bias and whatnot.

Anything to harden our master. :D

"Chad: if you seriously think that Turing-completeness does not imply the possibility of sentience, then you're definitely in the wrong place indeed."

gwern: The implication is certainly there and it's one I am sympathetic with, but I'd say its far from proven. The leap in logic there is one that will keep the members of the choir nodding along but is not going to win over any converts. A weak argument is a weak argument, whether you agree with the conclusion reached by that argument -- it's better for the cause if the arguments are held to higher standards.

Zubon,

"If you want a sufficient response to optimism, consider: is the probability that you will persist forever 1? If not, it is 0."

You're only correct if the probability is constant with respect to time. Consider, however, that some uncertain events have a non-zero probability even if infinite time passes. For example, random walks in three dimensions (or more) are not guaranteed to meet their origin again, even over infinite time:

gwern: The implication is certainly there and it's one I am sympathetic with, but I'd say its far from proven.

Disagree? Give us an example of a phenomenon that cannot be represented by a Turing Machine, and we'll talk.

Eliezer: imagine that you, yourself, live in a what-if world of pure mathematics

Isn't this true? It seems the simplest solution to "why is there something rather than nothing". Is there any real evidence against our apparently timeless, branching physics being part of a purely mathematical structure? I wouldn't be shocked if the bottom was all Bayes-structure :)

RI, it shouldn't literally be Bayes-structure because Bayes-structure is about inference is about mind. I have certainly considered the possibility that what-if is all there is; but it's got some problems. Just because what-if is something that humans find deductively compelling does not explain how or why it exists Platonically - to suppose that it is necessary just because you can't find yourself not believing it, hardly unravels the mystery. And much worse, it doesn't explain why we find ourselves in a low-entropy universe rather than a high-entropy one (why our memories are so highly ordered).

Consequentialist, I just had a horrifying vision of a rationalist cult where the cultists are constantly playing pranks on the Master out of a sincere sense of duty.

I worry a bit less about being a cult leader since I noticed that Paul Graham also has a small coterie following him around accusing him of being a cult leader, and he's not even trying to save the world. In any case, I've delivered such warnings as I have to offer on both the nature of cultish thinking, and the traps that await those who are terrified of "being in a cult"; that's around as much as I can do.

Eliezer,
I'm a little disappointed, frankly. I would have thought you'd be over both God and the Problem of Evil by now. Possibly it goes to show just how difficult it is for people raised as (or by) theists to kill God in themselves.

But possibly you'll get there as you go along. I'd tell you what that was like but I don't know myself yet.

Caledonian:

In an argument that is basically attempting to disprove the existence of God, it seems a little disingenuous to me to include premises that effectively rule out God's existence. If you aren't willing to at least allow the possibility of dualism for the sake of argument, then why bother talking about God at all?

Also, I am not sure what your notion of "infinite" mathematics is about. Can you elaborate or point me to some relevant resources?

No, for him it won't be better.

Well, there's also the perspective of the newborn and the person it grows up into; if we consider that perspective, it probably would prefer that it exists. I don't want The Future to contain "me"; I want it to contain someone better than "me". (Or at least happier, considering that I would prefer to not have existed at all.) And I really doubt that my frozen brain will be of much help to The Future in achieving that goal.

the probabilities for cryonics look good.

They don't have to look good, they just have to beat the probabilities of your mind surviving the alternatives. Current alternatives: cremation, interment, scattering over your favourite football pitch. Currently I'm wavering between cryonics and Old Trafford.

Eliezer, I'm ridiculously excited about the next fifty years, and only slightly less excited about the fun theory sequence. Hope it chimes with my own.

Excellent post, agree with every single line of it. It's not depressing for me -- I went through that depression earlier, after finally understanding evolution.

One nitpick -- I find the question at the end of the text redundant.

We already know that all this world around us is just an enormous pattern arising out of physically determined interactions between particles, with no 'essence of goodness' or other fundamental forces of this kind.

So the answer to your question seems obvious to me -- if we don't like patterns we see around us (including us ourselves, no exceptions), all we need to do is to use physics to arrange particles into a certain pattern (for example superintelligence) that, in future, produces patterns desirable for us. That's all.

Eliezer, what's your answer to the question?

On the existential question of our pointless existence in a pointless universe, my perspective tends to oscillate between two extremes:

1.) In the more pessimistic (and currently the only rationally defensible) case, I view my mind and existence as just a pattern of information processing on top of messy organic wetware and that is all 'I' will ever be. Uploading is not immortality, it's just duplicating that specific mind pattern at that specific time instance. An epsilon unit of time after the 'upload' event that mind pattern is no longer 'me' and will quickly diverge as it acquires new experiences. An alternative would be a destructive copy, where the original copy of me (ie. me that is typing this right now) is destroyed after or at the instance of upload. Or I might gradually replace each synapse of my brain one by one with a simulator wirelessly transmitting the dynamics to the upload computer until all of 'me' is in there and the shell of my former self is just discarded. Either way, 'I' is destroyed eventually--maybe uploading is a fancier form of preserving ones thoughts for posterity as creating culture and forming relationships is pre-singularity, but it does not change the fact that the original meatspace brain is going to eventually be destroyed, no matter what.

The second case, what I might refer to as an optimistic appeal to ignorance, is to believe that though the universe appears pointless according to our current evidence, there may be some data point in the future that reveals something more that we are ignorant to at the moment. Though our current map reveals a neutral territory, the map might be incomplete. One speculative position taken directly from physics is the idea that I am a Boltzmann Brain. If such an idea can be taken seriously (and it is) than surely there are other theoretically defensible positions where my consciousness persists in some timeless form one way or another. (Even Bostrom's simulation argument gives another avenue of possibility)

I guess my two positions can be simplified into:
1.) What we see is all there is and that's pretty fucked up, even in the best case scenario of a positive singularity.

2.) We haven't seen the whole picture yet, so just sit back, relax, and as long as you have your towel handy, don't panic.

RE: the reek of cultishness and dogma, I agree.

Regardless of whether you want to argue that being in a cult might be ok or not anything to worry about, the fact is this sort of thing doesn't look good to other people. You're going to win many converts -- at least the kind you want -- by continuing to put on quasi-religious, messianic airs, and welcoming the sort of fawning praise that seems to come up a lot in the comments here. There's obviously some sharp thinking going on in these parts, but you guys need to pay a bit more attention to your PR.

Not going to win, that should read.

by continuing to put on quasi-religious, messianic airs

Huh. Let a guy have a bit of poetic license now and then, eh? I really don't see what you mean.

The request that we should 'fix the world' suggests that a.)we know that it is broken and b.)we know how to fix it; I am not so sure that this is the case. When one says 'X is wrong/unfair/undesirable etc., one is more often than not actually making a statement about one's state of mind rather than the state of reality i.e., one is saying 'I think or feel that X is wrong/unfair/undesirable'. Personally, I don't like to see images of suffering and death but I'm not sure that my distaste for suffering and death is enough to confidently assert that they are wrong or that they should be avoided. For example, without the facility for pain that leads to suffering we probably wouldn't make it past infancy and without death the world would be even more overpopulated than it is currently. No matter how rigorous and free from preconditioned religious thinking our reasoning is, 'what we would like to see' is still a matter of personal taste to some extent. Feeling and reasoning ineract in such an intricate and inseparable way that, while one may like to think one has reached conclusions about right/wrong/good/bad/just/fair etc. in a wholly dispassionate and rational way, it is likely that personal feelings have slipped in there unnoticed and unquestioned and added a troublesome bias.

Alice, can't tell crap from great? Don't worry, 90% of people share your inability. Why? Because 90% of everything is crap. (Sturgeon's law)

Lets fix the things that are obviously crap first. After that, well address the iffy things.

Towards less crappy greatness.

You've said the bit about Paul Graham twice now in this thread; do you actually consider that good reasoning, or are you merely being flip? Paul Graham's followers may or may not be cultish to some degree, but that doesn't bear on the question of whether your own promotional strategies are sound ones. Let me put it this way: you will need solid, technically-minded, traditionally-trained scientists and engineers in your camp if you ever hope to do the things you want to do. The mainstream science community, as a matter of custom, doesn't look favorably upon uncredentialed lone wolves making grandiose pronouncements about "saving the world." This smacks scarily of religion and quackery. Like it or not, credibility is hugely important; be very careful about frittering it away.

I take your point...if your point is 'we gotta start somewhere'. Nontheless, the use of 'obviously' is problematic and misleading. To whom is it obvious? To you? Or perhaps you and your friends, or you and other people on the internet who tend to think in the same way as you and with whom you generally agree? Don't get me wrong, I have a very clear idea of what I think is crap (and I strongly suspect it'd be similar to yours) and I'm just as keen to impose my vision of the 'uncrap' on the world as the next person. However, I can't help but be troubled by the thought that the mass murder of jews, gypsies, the mentally retarted and homosexuals was precipitated by the fact that Hitler et al thought it was 'obvious' that they were crap and needed fixing.

Whoops! I thought that comment had been swallowed by the ether, so I said it again. Turns out it's on the previous page. Dup has been deleted.

On the unfairness of existence:

Those who (want to) understand and are able, joyously create things that have always existed as potentials.

Those who don't (want to) understand and can't do anything real, make stuff up that never was possible and never will be.

The former last forever in eternal glory, spanning geological timescales and civilizations, for the patterns they create are compatible with the structure of the universe and sustained by it, while oblivion is reserved for the latter.

Science. The real stuff. Be all you can be.

Science. Keepin' it real.

Science. Live a life with a purpose.
Science. Live a life worth living.

In an argument that is basically attempting to disprove the existence of God, it seems a little disingenuous to me to include premises that effectively rule out God's existence.

How exactly can you construct a disproof of X without using premises that rule out X? That's what disproving is.

Non-infinite mathematics: also known as finite mathematics, also known as discrete mathematics. Non-continuum. Not requiring the existence of the real numbers.

To the best of our knowledge, reality only seems to require the integers, although constructing models that way is a massive pain. If you can give us an example of a physical phenomenon that cannot be generated by a Turing Machine's output -- just one example -- then I will grant that we have no grounds for presuming human cognition can be so constructed. Also, you'll win several Nobel Prizes and go down in history as one of the greatest scientist-thinkers ever.

I'm not holding my breath.

is creating a child after the right and proper fashion of creating a child. If I do not wish to be a father, I think that it is acceptable for me to avoid it; and certainly the skill to deliberately not create a child would be less than the skill to deliberately create a healthy child.

I would agree, I am not trying to create a child either. I'm trying to create brain stuff, and figure out how to hook it up to a human so that it becomes aligned to that humans brain. Admittedly it is giving more power to children, but I think the only feasible way to get adults is for humans to get power and the self-knowledge of what we are, and grow up in a painful fashion. We have been remarkably responsible with nukes, and I am willing to bet on humanity becoming more responsible as it realises how powerful it really is. You are and SIAI are good data point for this supposition.

After my degree going back 9 years or so I spent some time thinking about seedAIs of a variety of fashions, not calling them such. I could see no way for a pure SeedAI to be stable, unless it starts off perfect. To exist is to be imperfect, from everything I have learnt of the world. You always see the edited highlights of peoples ruminations, so when you see us disagree you think we have not spent time on it.

That is also probably part of why I don't want to be in the future, I see it as needing us to be adults and my poor simian brain is not suited to perpetual responsibility or willing to change to be so.

>How exactly can you construct a disproof of X without using
>premises that rule out X? That's what disproving is.

Sure, a mathematical proof proceeds from its premises and therefore any results achieved are entailed in those premises. I am not sure we are really in the real of pure mathematics here but I probably should have been more precise in my statement. In a non-mathematical discussion, a slightly longer chain of reasoning is generally preferred -- starting with the premise that dualism is false is a little uncomfortably close to starting with the premise that God doesn't exist for my taste.

>If you can give us an example of a physical phenomenon that
>cannot be generated by a Turing Machine's output -- just
>one example -- then I will grant that we have no grounds
>for presuming human cognition can be so constructed.

For the record (not that I have any particular standing worthy of note): I am not a dualist and I believe 100% that human cognition is a physical phenomenon that could be captured by a sufficiently complex Turing Machine. Can I prove this to be the case? No I can't and I don't really care to try -- and it's likely "above my level". The only reason I piped up at all is because I think strawman arguments are unconvincing and do a disservice to everyone.

>Also, you'll win several Nobel Prizes and go down in history
>as one of the greatest scientist-thinkers ever.
>I'm not holding my breath.

Don't worry -- I have no such aspiration, so you can comfortably continue with your respiration.

"Bayesian cult encourages religious people to kill God in themselves" - how's that for a newspaper headline?

P.S. I'd delete this comment after a certain amount of time, you might not want it to get cached by google or something.

The various silly people who think I want to keep the flesh around forever, or constrain all adults to the formal outline of an FAI, are only, of course, making things up; their imagination is not wide enough to understand the concept of some possible AIs being people, and some possible AIs being something else.

And this post reveals more failings along those lines; imagining that death is something too awful for a God to allow is incompatible with viewing intelligent life in the universe as an extended system of computations, and again suggests you are overly-attached to linking agency and identity to discrete physical bodies. The way you present this, as well as the discussion in the comments, suggests you think "death" is a thing that can be avoided by living indefinitely; this, also, is evidence of not thinking deeply about identity in deep time. The way you speak about the danger facing you - not the danger facing life, which I agree with you about; but the personal danger of death - suggests that you want to personally live on beyond the Singularity; whereas more coherent interpretations of your ideas that I've heard from Mike Vassar imply annihilation or equivalent transformation of all of us by the day after it. It seems most likely to me either that you're intentionally concealing that the good outcomes of your program still involve the "deaths" of all humans, or that you just haven't thought about it very hard.

What I've read of your ideas for the future suffers greatly from your not having worked out (at least on paper) notions of identity and agency. You say you want to save people, but you haven't said what that means. I think that you're trying to apply verbs to a scenario that we don't have the nouns for yet.

It is extraordinarily difficult to figure out how to use volunteers. Almost any nonprofit trying to accomplish a skilled-labor task has many more people who want to volunteer their time than they can use. The Foresight Institute has the same problem: People want to donate time instead of money, but it's really, really hard to use volunteers. If you know a solution to this, by all means share.

it doesn't explain why we find ourselves in a low-entropy universe rather than a high-entropy one

I didn't think it would solve all our questions, I just wondered if it was both the simplest solution and lacking good evidence to the contrary. Would there be a higher chance of being a Boltzmann brain in a universe identical to ours that happened to be part of a what-if-world? If not, how is all this low-entropy around me evidence against it?

Just because what-if is something that humans find deductively compelling does not explain how or why it exists Platonically

How would our "Block Universe" look different from the inside if it was a what-if-Block-Universe? It all adds up to...

Not trying to argue, just curious.

How would our "Block Universe" look different from the inside if it was a what-if-Block-Universe? It all adds up to...

I'm not saying this is wrong, but in its present form, isn't it really a mysterious answer to a mysterious question? If you believed it, would the mystery seem any less mysterious?

suggests that you want to personally live on beyond the Singularity; whereas more coherent interpretations of your ideas that I've heard from Mike Vassar imply annihilation or equivalent transformation of all of us by the day after it

Eliezer, doesn't "math mysteriously exists and we live in it" have one less mystery than "math mysteriously exists and the universe mysteriously exists and we live in it"? (If you don't think math exists it seems like you run into indispensability arguments.)

IIRC the argument for a low-entropy universe is anthropic, something like "most non-simple universes with observers in them look like undetectably different variants of a simple universe rather than universes with dragons in them".

Alastair Malcolm:

in any comparison of all possible combinations of bit/axiom strings up to any equal finite (long) length (many representing not only a world but also (using 'spare' string segments inside the total length) extraneous features such as other worlds, nothing in particular, or perhaps 'invisible' intra-world entities), it is reasonable to suppose that the simplest worlds (ie those with the shortest representing string segments) will occur most often across all strings, since they will have more 'spare' irrelevant bit/axiom combinations up to that equal comparison length, than those of more complex worlds (and so similarly for all long finite comparison lengths).

Thus out of all worlds inhabitable by SAS's, we are most likely to be in one of the simplest (other things being equal) - any physics-violating events like flying rabbits or dragons would require more bits/axioms to (minimally) specify their worlds, and so we should not expect to find ourselves in such a world, at any time in its history.

(SAS means self-aware substructure)

Re: The way you present this, as well as the discussion in the comments, suggests you think "death" is a thing that can be avoided by living indefinitely [...]

Er... ;-) Many futurists seem to have it in for death. Bostrom, Kurzweil, Drexler, spring to mind. To me, the main problem seems to be uncopyable minds. If we could change our bodies like a suit of clothes, the associated problems would mostly go away. We will have copyable minds once they are digital.

I'm not saying this is wrong, but in its present form, isn't it really a mysterious answer to a mysterious question? If you believed it, would the mystery seem any less mysterious?

Hmm. You're right.

Darn.

Re: The way you present this, as well as the discussion in the comments, suggests you think "death" is a thing that can be avoided by living indefinitely [...]

Er... ;-) Many futurists seem to have it in for death. Bostrom, Kurzweil, Drexler, spring to mind. To me, the main problem seems to be uncopyable minds. If we could change our bodies like a suit of clothes, the associated problems would mostly go away. We will have copyable minds once they are digital.

If we imagine a world where

- you can share (or sell) your memories with other people, and borrow (or rent) their memories

- most of "your" memories are of things that happened to other people

- most of the time, when someone is remembering something from your past, it isn't you

- you have sold some of the things that "you" experienced to other people, so that legally they are now THEIR experiences and you may be required to pay a fee to access them, or to erase them from your mind

- you make, destroy, augment, or trim copies of yourself on a daily basis; or loan out subcomponents of yourself to other people while borrowing some of their components, according to the problem at hand, possibly by some democratic (or economic) arbitration among "your" copies

- and you have sold shares in yourself to other processes, giving them the right to have a say in these arbitrations about what to do with yourself

- "you" subcontract some of your processes - say, your computation of emotional responses - out to a company in India that specializes in such things

- which is advantageous from a lag perspective, because most of the bandwidth-intensive computation for your consciousness usually ends up being distributed to a server farm in Singapore anyway

- and some of these processes that you contract out are actually more computationally intensive than the parts of "you" that you own/control (you've pooled your resources with many other people to jointly purchase a really good emotional response system)

- and large parts of "you" are being rented from someone else; and you have a "job" which means that your employer, for a time, owns your thoughts - not indirectly, like today, but is actually given write permission into your brain and control of execution flow while you're on the clock

- but you don't have just one employer; you rent out parts of you from second to second, as determined by your eBay agent

- and some parts of you consider themselves conscious, and are renting out THEIR parts, possibly without notifying you

- or perhaps some process higher than you in the hierarchy is also conscious, and you mainly work for it, so that it considers you just a part of itself, and can make alterations to your mind without your approval (it's part of the standard employment agreement)

- and there are actually circular dependencies in the graph of who works for whom, so that you may be performing a computation that is, unknown to you, in the service of the company in India calculating your emotional responses

- and these circles are not simple circles; they branch and reconverge, so that the computation you are doing for the company in India will be used to help compute the emotions of trillions of "people" around the world

In such a world, how would anybody know if "you" had died?

That sounds like the "One Big Organism" concept. Nick Bostrom has also written about that - e.g. see his What is a Singleton?

The fictional Borg work similarly, I believe. Death would become rather like cutting your toenails.

Phil: [. . .] In such a world, how would anybody know if "you" had died?

Tim -

What I described involves some similar ideas, but I find the notion of a singleton unlikely, or at least suboptimal. It is a machine analogy for life and intelligence. A machine is a collection of parts, all working together under one common control to one common end. Living systems, by contrast, and particularly large evolving systems such as ecosystems or economies, work best, in our experience, if they do not have centralized control, but have a variety of competing agents, and some randomness.

There are a variety of proposals floating about for ways to get the benefits of competition without actually having competition. The problem with competition is that it opens the doors to many moral problems. Eliezer may believe that correct Bayesian reasoners won’t have these problems, because they will agree about everything. This ignores the fact that it is not computationally efficient, physically possible, or even semantically possible (the statement is incoherent without a definition of “agent”) for all agents to have all available information. It also ignores the fact that randomness, and using a multitude of random starts (in competition with each other), are very useful in exploring search spaces.

I don't think we can eliminate competition; and I don't think we should, because most of our positive emotions were selected for by evolution only because we were in competition. Removing competition would unground our emotional preferences (eg, loving our mates and children, enjoying accomplishment), perhaps making their continued presence in our minds evolutionarily unstable, or simply superfluous (and thus necessarily to be disposed of, because the moral imperative I have most confidence that a Singleton would follow is to use energy efficiently).

The concept of a singleton is misleading, because it makes people focus on the subjectivity (or consciousness; I use these terms as synonyms) of the top level in the hierarchy. Thus, just using the word Singleton causes people to gloss over the most important moral questions to ask about a large hierarchical system. For starters, where are the locuses of consciousness in the system? Saying “just at the top” is probably wrong.

Imagining a future that isn’t ethically repugnant requires some preliminary answers to questions about consciousness, or whatever concept we use to determine what agents need to be included in our moral calculations. One line of thought is to impose information-theoretical requirements on consciousness, such as that a conscious entity has exactly one possible symbol grounding connecting its thoughts to the outside world. You can derive lower bounds for consciousness from this supposition. Another would be to posit that the degree of consciousness is proportional to the degree of freedom, and state this with an entropy measurement relating a processes’ inputs to its possible outputs.

Having constraints such as these would allow us to begin to identify the agents in a large, interconnected system; and to evaluate our proposals.

I'd be interested in whether Eliezer thinks CEV requires a singleton. It seems to me that it does. I am more in favor of an ecosystem or balance-of-power approach that uses competition, than a totalitarian machine that excludes it.

>To exist is to be imperfect
A thing that that philosophical types like to do that I dislike is making claims about about what it is to exist in general, claims that presumably would apply to all minds or 'subjects', when in fact those claims concern at most only the particular Homo Sapiens condition, and are based only on the experiences of one particular Homo Sapiens.

>However, I can't help but be troubled by the thought that the
>mass murder of jews, gypsies, the mentally retarted and
>homosexuals was precipitated by the fact that Hitler et al
>thought it was 'obvious' that they were crap and needed fixing.
To point out the obvious, Alice at least judges Hitler's actions as crap, and judges the imposition on the world of values that have not been fully considered as crap, and would like to impose those value on the world. This is was a good post on the subject: http://www.overcomingbias.com/2008/08/invisible-frame.html

A thing that that philosophical types like to do that I dislike is making claims about about what it is to exist in general, claims that presumably would apply to all minds or 'subjects', when in fact those claims concern at most only the particular Homo Sapiens condition, and are based only on the experiences of one particular Homo Sapiens.

My claim is mainly based on physics of one sort of another. For one the second law of thermodynamics. All systems will eventually degrade to whatever is most stable. Neutrons, IIRC. And unless a set of neutrons in thermodynamic equilibria happen to be your idea of perfection, or your idea of perfection is impermanent, then my statement stands.

Another one is acting, a quantum system decoheres or splits the universe into two possible worlds. The agent doesn't know which of the possible worlds it is in (unless it happens to have a particle in super position with the decohering system), so has to split the difference and act as if it could be in either. As such it is imperfect.

What I described involves some similar ideas, but I find the notion of a singleton unlikely, or at least suboptimal. It is a machine analogy for life and intelligence. A machine is a collection of parts, all working together under one common control to one common end. Living systems, by contrast, and particularly large evolving systems such as ecosystems or economies, work best, in our experience, if they do not have centralized control, but have a variety of competing agents, and some randomness.

The idea of one big organism is not really that it will be, in some sense "optimal". It's more that it might happen - e.g. as the result of an imbalance of the powers at the top.

We have very limited experience with political systems. The most we can say is that so far, we haven't managed to get communism to be as competitive as capitalism. However, that may not matter. If all living system fuse, they won't have any competition, so how well they operate together would not be a big issue.

In theory, competition looks very bad. Fighting with each other can't possibly be efficient. Almost always, battles should be done under simulation - so the winner can be determined early - without the damage and waste of a real fight. There's a huge drive towards cooperation - as explained by Robert Wright.

In theory, competition looks very bad. Fighting with each other can't possibly be efficient. Almost always, battles should be done under simulation - so the winner can be determined early - without the damage and waste of a real fight. There's a huge drive towards cooperation - as explained by Robert Wright.

If the winner could be determined by a simpler computation, you would be running that computation instead [...]

Well, that's the point. Usually it can be, and often we're not. There's a big drive towards virtualising combat behaviour in nature. Deer snort at each other, sea lions bellow - and so on: signalling who is going to win without actually fighting. Humans do the same thing with national sports - and with companies - where a virtual creature dies, and the people mostly walk away. But we are still near the beginning of the curve. There are still many fights, and a lot of damage done. Huge improvements in this area could be made.

Tim - I'm asking the question whether competition, and its concomitant unpleasantness (losing, conflict, and the undermining of CEV's viability), can be eliminated from the world. Under a wide variety of assumptions, we can characterize all activities, or at least all mental activities, as computational. We also hope that these computations will be done in a way such that consciousness is still present.

My argument is that optimization is done best by an architecture that uses competition. The computations engaged in this competition are the major possible loci for consciousness. You can't escape this by saying that you will simulate the competition, because this simulation is itself a computation. Either it is also part of a possible locus of consciousness, or you have eliminated most of the possible loci of consciousness, and produced an active but largely "dead" (unconscious) universe.

Alex, I admit I hope the fawning praisers, who are mostly anonymous, are Eliezer's sockpuppets. Rather than a dozen or more people on the internet who read Eliezer's posts and feel some desire to fawn. But it's mostly an aesthetic preference -I can't say it makes a real difference in accomplishing shared goals, beyond being a mild waste of time and energy.

So. Hard. To. Say. Anything. Bad. About. Eliezer.

:)

Aren't you a bit biased here? If one expresses positive views about Eliezer, that's fawning, obsequiousness, or other rather exaggerated word, but negative views and critique is just business as usual. As usual.

It would be better if talking about people ceased and ideas and actions got 100% attention.

Remove the talk about people from politics and what's left? Policies? I don't know what the people/policies ratio in political discussion in the media is, but often it feels like most of the time is spent on talking about the politicians, not about policies. I guess it's supposed to be that way.

My argument is that optimization is done best by an architecture that uses competition.

Optimization is done best by an architecture that performs trials, inspects the results, makes modifications and iterates. No sentient agents typically need to be harmed during such a process - nor do you need multiple intelligent agents to perform it.

Remember the old joke: "Why is there only one Monopolies Commission?"

The evidence for the advantages of cooperation is best interpreted as a lack of our ability to manage large complex structures effectively. We are so bad at it that even a stupid evolutionary algorithm can do better - despite all the duplication and wasted effort that so obviously involves. Companies that develop competing products to fill a niche in ignorance of each other's efforts often is the stupid waste of time that it seems. In the future, our management skills will improve.

Optimization is done best by an architecture that performs trials, inspects the results, makes modifications and iterates. No sentient agents typically need to be harmed during such a process - nor do you need multiple intelligent agents to perform it.

Some of your problems will be so complicated, that each trial will be undertaken by an organization as complex as a corporation or an entire nation.

If these nations are non-intelligent, and non-conscious, or even unemotional, and incorporate no such intelligences in themselves, then you have a dead world devoid of consciousness.

If they do incorporate agents, then for them not to be "harmed", they need not to feel bad if their trial fails. What would it mean to build agents that weren't disappointed if they failed to find a good optimum? It would mean stripping out emotions, and probably consciousness, as an intermediary between goals and actions. See "dead world" above.

Besides being a great horror that is the one thing we must avoid above all else, building a superintelligence devoid of emotions ignores the purpose of emotions.

First, emotions are heuristics. When the search space is too spiky for you to know what to do, you reach into your gut and pull out the good/bad result of a blended multilevel model of similar situations.

Second, emotions let an organism be autonomous. The fact that they have drives that make them take care of their own interests, makes it easier to build a complicated network of these agents that doesn't need totalitarian top-down Stalinist control. See economic theory.

Third, emotions introduce necessary biases into otherwise overly-rational agents. Suppose you're doing a Monte Carlo simulation with 1000 random starts. One of these starts is doing really well. Rationally, the other random starts should all copy it, because they want to do well. But you don't want that to happen. So it's better if they're emotionally attached to their particular starting parameters.

It would be interesting if the free market didn't actually reach an optimal equilibrium with purely rational agents, because such agents would copy the more successful agents so faithfully that risks would not be taken. There is some evidence of this in the monotony of the movies and videogames that large companies produce.

The evidence for the advantages of cooperation is best interpreted as a lack of our ability to manage large complex structures effectively. We are so bad at it that even a stupid evolutionary algorithm can do better - despite all the duplication and wasted effort that so obviously involves. Companies that develop competing products to fill a niche in ignorance of each other's efforts often is the stupid waste of time that it seems. In the future, our management skills will improve.

This is the argument for communism. Why should we resurrect it? What conditions will change so that this now-unworkable approach will work in the future? I don't think there are any such conditions that don't require stripping your superintelligence of most of the possible niches where smaller consciousnesses could reside inside it.

Ha, I get it now, FAI is about creating god.

Anyways, no matter what you do, mind annihilation is certain in our universe, i.e. 2nd law of thermodynamics.

@Doug S. Read "The Gentle Seduction" by Marc Stiegler. And, if you haven't already, consider anti-depressants: I know a number of people whom they have saved from suicide.

The main question here is -

Can finite state machine have qualia?

Because suffering is qualia. If it is not, it does''t matter. It is easy to write a programm that print "i am suffering" if you press a button.

So, God cannot influence on result of work of final automata. But He can give it qualia, or switch qualia off for it - and nobody would ever mention it.

So, existing of final automat Universe, don''t prove absense of God, because God could change qualia without changing result of work of the automat.

So, God cannot influence on result of work of final automata. But He can give it qualia, or switch qualia off for it - and nobody would ever mention it.

What makes you think qualia aren't necessarily bound to algorithms?

Why the hangup about turing-completeness?

In a finite universe world there are no true turing machines, as there are no infinite tapes; thus if you are going to be assigning some philosophical heft to turing-completeness you are being a bit sloppy, and should be saying "show me something that provably cannot be computed by a finite state machine of any size".

Yes, some Buddhist sects allow for complete annihilation of self. Most of the Zen sects, actually. No gods, no afterlife, just here-and-now, whichever now you happen to be considering. Reincarnation is simply the reconfiguration of you, moment by moment, springing up from the Void (or the quantum foam, if you prefer), each moment separate and distinct from the previous or the subsequent. Dogen (and Bankei and Huineng, for that matter) understood the idea of Timeless Physics very well.

What makes you think qualia aren't necessarily bound to algorithms?

I have never come across anyone who could present a coherent and intelligible definition for the word that didn't automatically render the referent non-existent.

Before we try to answer the question, we need to establish that the question is a valid one. 'How many angels can dance on the head of a pin?' is not one of the great mysteries, because the question is only meaningful in a context of specific, unjustifiable beliefs. Eliminate those beliefs and there's no more question.

Note: I'm an atheist who, like you, agrees that there's no divine plan and that, good or bad, shit happens.

That said, I think there's a hole in your argument. You're convincing when you claim that unfair things happen on Earth; you're not convincing when you claim there's no afterlife where Earthly-unfairness is addressed.

Isn't that the whole idea (and solace) of the afterlife? (Occam's Razor stops me from believing in an afterlife, but you don't delve into that in your essay.) A theist could easily agree with most of your essay but say, "Don't worry, all those Holocaust victims are happy in the next world."

The same holds for your Game of Life scenario. Let's say I build an unfair Game of Life. I construct rules that will lead to my automata suffering, run the simulation, automata DO suffer, and God doesn't appear and change my code.

Okay, but how do I know that God hasn't extracted the souls of the automata and whisked them away to heaven? Souls are the loophole. Since you can't measure them (since they're not part of the natural world but are somehow connected to the natural world), God can cure unfairness (without messing with terrestrial dice) by playing right by souls.

My guess is that, like me, you simply don't believe in souls because such a belief is an arbitrary, just-because-it-feels-good belief. My mind -- trained and somehow naturally conditioned to cling to Occam's Razer -- just won't go there.

So you don't think you could catch up? If you had been frozen somewhere between -10000 and -100 years and revived now, don't you think you could start learning what the heck it is people are doing and understand nowadays? Besides a lot of the pre-freeze life-experience would be fully applicable to present. Everyone starts learning from the point of birth. You'd have headway compared to those who just start out from nothing.
There are things we can meaningfully contribute to even in a Sysop universe, filled with Minds. We, after all, are minds, too, which have the inherent quality of creativity - creating new, evermore elegant and intricate patterns - at whatever our level is, and self-improvement; optimization.

No, you couldn't. Someone from 8,000BC would not stand a chance if they were revived now. The compassionate thing to do, really, would be to thaw them out and bury them.

Yes, they would be worse off than children. Don't underestimate the importance of development when it comes to the brain.

Minds don't have the "inherent quality of creativity". Autistics are the obvious counterexample.

if I could create a VPOP that did not have subjective experience (or the confusion we name subjective experience)

The confusion we name subjective experience? TBH Eli, that sounds like neomystical crap. See below.

I have never come across anyone who could present a coherent and intelligible definition for the word that didn't automatically render the referent non-existent.

Qualia are neural representations of certain data. They can induce other neurological states, creating what we know as the first-person. So what? I don't see why so called reductionists quibble over this so much. They exist, just get over it and study it if you really want to.

Minds don't have the "inherent quality of creativity". Autistics are the obvious counterexample.

Sorry, but no, no, no.

The evidence for the advantages of cooperation is best interpreted as a lack of our ability to manage large complex structures effectively. We are so bad at it that even a stupid evolutionary algorithm can do better - despite all the duplication and wasted effort that so obviously involves. Companies that develop competing products to fill a niche in ignorance of each other's efforts often is the stupid waste of time that it seems. In the future, our management skills will improve.

This is the argument for communism. Why should we resurrect it? What conditions will change so that this now-unworkable approach will work in the future? [...]

Nature excels at building large-scale cooperative structures. Multicellular organisms are one good example, and the social insects are another.

If the evidence for the superiority of competition over cooperation consists of "well, nobody's managed to get cooperation in the dominant species working so far", then it seems to me, that's a pretty feeble kind of evidence - offering only extremely tenuous support to the idea that nobody will ever get it to work.

The situation is that we have many promising examples from nature, many more promising examples from history, a large trend towards globalisation - and a theory about why cooperation is naturally favoured.

Some of your problems will be so complicated, that each trial will be undertaken by an organization as complex as a corporation or an entire nation.

Maybe - but test failures are typically not a sign that you need to bin the offending instance. Think of how programmers work. See some unit test failures? Hit undo a few times, until they go away again. Rarely do you need to recycle on the level of worms.

I find it strange how atheists always feel able to speak for God. Can you speak for your human enemies? Can you even speak for your wife, if you have one? Why would you presume to think you can say what God would or wouldn't allow?

Abe: I find it strange how atheists always feel able to speak for God.

Qualia are neural representations of certain data.

So what? I don't see why so called reductionists quibble over this so much.

That is NOT what that word is generally used to refer to.

Why, because it's a meaningful definition - and people are generally referring to something utterly meaningless? If you want me to define what people, in general, are talking about then of course I can't give a meaningful definition.

But I contend that this is meaningful, and it is what people are referring to - even if they don't know how to properly talk about it.

Imagine person A says that negative numbers are not even conceptually possible, or that arithmetic or whatever can't be performed with them. Person B contends otherwise. Person A asks how one could possibly add negative numbers, and B responds with a lecture about algebraic structures. A objects, "But people aren't generally referring to algebraic structures when they talk about maths, etc. I wasn't talking about that, I was talking about (4-7) and (-2*14) and how these things make no sense."

Well I contend that even if people don't know what they're talking about when they say "qualia this" or "qualia that" - and, in general, they're using gibberish definitions and speech - they're actually trying to talk about something close to the definition I've given.

you're using the word incorrectly

Again, if the only "correct" way to use the word is in the same manner as it is generally thought of, then of course you will never find a sensible definition because none of the sensible definitions are in common use - so you've ruled them out a priori, and are touting a tautology. But I'm not going to define what other people think they mean by a word - I'm going to define the ontology of the situation. If that's at odds with what people think they're talking about, then so what? People talk about God and think they're referring to this guy in the sky who is actually real - doesn't mean it's what's really going on, or that that's a really accurate definition of God (which would lead to the ontological argument being sound).

What makes you think that 'qualia' are a meaningful concept?

The problem, of course, is that qualia (or more generally, experiencing-ness) is not a concept at all (well there is a concept of experiencing-ness, but that is just the concept, not the actuality). A metaphor for experiencing-ness is the "theater of awareness" in which all concepts, sensations, and emotions appear and are witnessed. But experiencing-ness is prior to any and all concepts.

Nate Barna: Sometimes, they're not trying to speak for God, as they're not first assuming that an ideally intelligent God exists. Rather, they're imagining and speaking about the theist assumption that an ideally intelligent God exists, and then they carefully draw inferences which tend to end up incoherent on that grounding. However, philosophy of religion reasonably attempts coherence, and not all atheists are completely indifferent toward it.

It may be true that some times atheists carefully draw inferences from the idea of an ideally intelligent God. I have yet to see it. Eliezer doesn't seem to be at all careful when he says, "The obvious example of a horror so great that God cannot tolerate it, is death - true death, mind-annihilation. I don't think that even Buddhism allows that. So long as there is a God in the classic sense - full-blown, ontologically fundamental, the God - we can rest assured that no sufficiently awful event will ever, ever happen. There is no soul anywhere that need fear true annihilation; God will prevent it." There is no careful inference there. There is just bald-face assertion.

Why would a being that can create minds at will flinch at their annihilation? The absolute sanctity of minds, even before God, is the sentiment of modern western man, not a careful deduction based on an inconceivably superior intelligence.

The truth is, we don't even know what a SIAI would do, let alone a truly transcendent being, like God. If one going to try and falsify a concept of God, it should at least be a concept more authoritative than the ad hoc imaginings of an atheist.

Abe: Why would a being that can create minds at will flinch at their annihilation? The absolute sanctity of minds, even before God, is the sentiment of modern western man, not a careful deduction based on an inconceivably superior intelligence.

Following on to the sub-thread here, initiated by Recovering Irrationalist, about whether mathematical existence may be all there is, and that we live in it.

What does that say about the title of the post, Beyond the Reach of God?

Wouldn't it imply that there are those who are indeed beyond God's reach, since even God Himself cannot change the nature of mathematics? That is, God does not really have any control over the multiverse; it exists in an invariant form independent of God's existence.

However we can also argue that there are worlds within the multiverse where something like God exists and does have control, as well as worlds where there is no such entity. (This requires understanding "control" as being compatible with the deterministic nature of the mathematical multiverse.) The question then arises as to which kind of world within the multiverse we inhabit. Are we beyond the reach of God?

A case can be made that this kind of question is poorly posed, because entities with brain structures identical to our own inhabit many places in the multiverse, hence we have to view our experiences as being a sort of superposition of all those instances. So we should ask, what fraction of our copies exist in universes controlled by a God-like entity, versus what fraction exist in universes without any such controller? At this point the traditional arguments come into play about how likely the universe we see about us is likely to be compatible with the ability and motivations of a God-like entity, whether such an entity would allow injustice and evil to exist, etc.

@Doug S. Read "The Gentle Seduction" by Marc Stiegler. And, if you haven't already, consider anti-depressants: I know a number of people whom they have saved from suicide.

::Googles "The Gentle Seduction"::

Yeah, that's a very beautiful story. And yes, I take antidepressants. They just change my feelings, not my beliefs. Their subjective effect on me can best be described as "Yes, my life still sucks, but I'm cheerful anyway!" If I honestly prefer retroactive non-existence even when happy, doesn't that suggest that my assessment stems from something other than a lack of happiness chemicals in my brain?

Fear not, I have no intention of committing suicide in the near future. Although I prefer the world in which I never existed, the world in which I exist and and then die tomorrow is worse than the one in which I exist and continue to exist for several more years. (Specifically, my untimely death would cause great misery to several people that care about me, so I will refrain from dying before they do.)

Nothing really matters,
Anyone can see,
Nothing really matters-,nothing really matters to me,

Any way the wind blows....

**
Queen - Bohemian Rhapsody
We miss you Freddie....

I don't understand why you believe this thought exercise leads to despair or unhappiness. I went through this thought experiment many years ago, and the only significant impact it had on me was that I evaluate risk very differently than most people around me. I'm no less happy (or more depressed at least) or motivated, and I experience about as much despair as a non-secular optimist: occasional brief glimpses of it which quickly evaporate as I consider my options and choices.

And, to be honest, the process of looking at existentialism and going through some of the chains of thought mentioned in this article definitely improved my young adulthood. It made it more interesting and exciting, and made me more interesting to other people.

In any case, despair is not such a bad thing. Look at Woody Allen. Is he a joyless unproductive fear-ridden hermit? uh, no. But he's certainly thought through everything in this article, and accepted it, and turned it all into a source of amusement, curiosity, and intellectual stimualtion.

Lots of ideas here. They only seem to work if God is primarily concerned about fairness on earth. What if God is not so concerned about our circumstance as He in our response to circumstance. After all, He has an eternal perspective, while our perspective is limited by what we see of life. If this were true, then earth, and our existence, are like a big machine designed specifically to sort out the few good from the bad. Being raised in an Orthodox Jewish family, I’m sure you encountered countless examples in the bible where bad stuff happened to good people. This is no great revelation of truth – it’s plainly obvious, so the authors of the bible obviously had no problem reconciling this dilemma, countless Jews and Christians have no difficulty reconciling these facts. They’re probably not all idiots or wishful thinkers, so perhaps they understand a perspective that you have not considered. Maybe God doesn’t settle all accounts on your time table, but rather His. Maybe your values (and mine) do not perfectly align with a perfect God –so who then should change? I completely understand why people spend their lives praying to God, searching for understanding, proselytizing to slammed doors. I cannot fathom why an atheist (an authentic atheist) would waste a moment of their precious short life writing endlessly on something they believe to be pure fiction. As for me, I’ll keep praying.

I tend to resolve these issues with measure-problem hand-waving. Basically, since any possible universe exists (between quantum branching, inflationary multiverse, and simulated/purely mathematical existence), any collection of particles (such as me sitting here) exists with a practically uncountable set of futures and pasts, many of which make no sense (bolzman brains). The measure problem is, why is that "many" not actually "most"? The simplest answer is the anthropic one: because that kind of existence simply "doesn't count". So, there is some set of qualities of the universe as we know it that make it "count", let's call that set "consciousness". And, personally, I think that this set includes not only the existence of optimizing agents (ourselves), but also the fact that these agents are fundamentally limited in something similar to the ways that we are. In other words, the very existence of some FAI which can keep all of your bad decisions (for any given definition of "bad") from having consequences, means that "consciousness" as we know it has ended. Whatever exists on the other side of that barrier is simply incommensurable with my values here on this side. It's "game over". I can have perfect faith that my "me" will never see it completed - by definition, since then I'd no longer be a conscious "me" under my definition.

That means I am much more motivated to look for (weakly) "incremental" solutions to the problems I see with the world than for truly revolutionary ones like FAI or cryonics. (I regard the last "revolutionary" change to be the evolution of humanity itself - so "incremental" in this sense encompasses all human revolutions to date. The end of death would not be incremental in this sense.)

Sure, I can see where this is more of a justification for acting like a normal person than a rational exploration of fully coherent value space. Yet I can also argue that being meta-rational means justifying, not re-questioning, certain axioms of behavior.

Shorter me: "solving the whole world" leaves me cold, despite fun theory and all. So does ending death, or avoiding it personally. So me not signing up for cryo is perfectly rational.

While I acknowledge that this might not be the most complete and coherent possible set of values, I see no evidence that it's specifically incoherent, and it is complete enough for me. The Singularity Institute set of values may be more complete and just as non-incoherent, but I suspect that mine are operationally superior, or at least less likely to be falsified, since they attain similar coherence with less of a divergence from evolved human behaviour.

or maybe "I hope he becomes Jewish."

That is a rotten thing to wish upon any adult male. Think of the pain!

Last night I was reading through your "coming of age" articles and stopped right before this one, which neatly summarizes why I was physically terrified. I've never before experienced sheer existential terror, just from considering reality.

Absolute, utter, exceptionless neutrality.

I hate these filthy Neutrals, Kif. With enemies you know where they stand but with Neutrals, who knows? It sickens me.

Are there any useful summaries of strategies to rearrange priorities and manage time to deal with the implications of this post? I get the existential terror part. We're minds in constant peril, basically floating on a tattered raft in the middle of the worst hurricane ever imagined. I'm sure only few of the contributors here think that saying, "this sucks but oh well" is a good idea. So what do we do?

Since I've started reading LW, I have started to devote way more of my life to reading. I read for hours each day now, mostly science literature, philosophy, economics, lots of links to external things from LW. But it hardly feels like enough and every choice I make about what to read feels like a precious one indeed. I am a grad student, and I think often about the rapidly changing landscape of PhD jobs. Should I be content going to an industrial job and paying for cryonics and hoping to nudge people in my social circles to adopt more rational hygiene in their beliefs (while working on doing so myself as well)? I know no one can really answer that kind of question for me, but other people can simulate that predicament and offer advice. Is there any?

If an intelligence explosion does happen in the next few decades, why am I even spending precious minutes worrying about what skill set to train into myself for such-and-such an industry or such-and-such a career? Those types of tasks might even be the very first tasks to be subsumed by advanced technology (much the way that technology displaces legal research assistants faster than janitors). The world isn't fair. I could study advanced math and engineering and hit my career at just the moment in history when those stop being people-tasks. I could be like the Reeks and Recs from Vonnegut's Player Piano. This is serious beeswax here. I want to make a Bayesian decision about how to spend my time and what skill set to train into myself. It would seem like this site is among the best places to pose the question and ask for direction to sweet updatable evidence. But where is some?

Not every child needs to stare Nature in the eyes. Buckling a seatbelt, or writing a check, is not that complicated or deadly. I don't say that every rationalist should meditate on neutrality. I don't say that every rationalist should think all these unpleasant thoughts. But anyone who plans on confronting an uncalibrated challenge of instant death, must not avoid them.

Granted. Now, where are the useful, calibrated challenges? I am like a school-age child in my rationality; I can read and understand this passage about neutrality and think about it for a moment, but I cannot yet hold it in my mind as I go out into the world to do work. But I want to do work, and I want it to be useful. Is there anything I can do short of disconnecting from society and meditating on rationality until neutrality seems intuitive?

Unfortunately, this post, dated 4 October 2008, blatantly ignores the good sense of the 'Occam's Razor' one, dated 26 September 2007. http://lesswrong.com/lw/jp/occams_razor/ It is very naive to argue along the lines of "cellular automata are Turing complete, hence we can build a cellular automaton simulating anything we want to". This is just using the term "Turing complete" in the same way as the poor barbarians of the 'Occam's Razor' post use the term "Thor", viz., as a sort of totem you wave around in order to spare you the work of actually thinking things through. Well, of course you can imagine a cellular automaton simulating anything you like, as long as it isn't self-contradictory. But there lies the problem, it is very difficult to know whether some concept is self-contradictory just using natural language before you have actually gone and built it. Who is telling you that all the moral and spiritual aspects of the conditio humana aren't going to pop up in your simulation as epiphenomena, by necessity, just as they did in this universe? That's right, you can't know until you have done the simulation. The smug "Is this world starting to sound familiar?" really cuts two ways in this case.

I wrote this when I was like sixteen, before I'd ever heard of LessWrong:

Dreaming of things yet to come, bleeding ink into the sand,
we passed the days on hither shore, for time measureless to man;
and soon our bloody mark was made on the sand on which we ran.

Footprints upon the sunless shore were rarer than the stains,
the moon cast light upon the trees, the moon gave us its rain,
and to each other in the night we shouted our refrain:

“We immortal run here, waiting; we don’t die despite the bleeding;
we will continue their lives’ taking if they won’t take our heeding;
and we would trade their lives for light, for these stains are worth reading.”

So surprising, mountain rising, o’er the sunless rainbow land,
we deathless died of shock and bled as God lifted His hand;
and these last words that we had bled are stained there in the sand:

Still stained there in the sand, beyond the reach of man.

But in sober historical fact, this is an unreasonable belief; I chose the example of World War II because from my reading, it seems that events were mostly driven by Hitler's personality, often in defiance of his generals and advisors.

Bullshit.

Besides the contemporary accounts of the inevitability of war, there's also, computer modelling of state relations using pre-war evidence (In fact, the modelling even generates the split for the cold war, before the start of WWII).

For the record, Heinrich Heine also said in 1821: "Das war ein Vorspiel nur, dort wo man Bücher verbrennt, verbrennt man auch am Ende Menschen." (That was but a prelude; where they burn books, they will ultimately burn people also.)

Besides that, your entire post could mostly be summarised as The universe doesn't care.

[sarcasm]

No, really? I never knew.

[/sarcasm]

As for saying that 'God wouldn't allow it', It's a ridiculous argument that anthropomorphises an imaginary creation, that was invoked in the first place to reassure people living in a world that never made sense. Of course god wouldn't allow it! He wouldn't allow it by definition. The 'Problem of Evil' is the very reason god exists in human minds in the first place! To use it to attack religion is as unsatisfactory as a rebuttal can get.

Thank you, Eliezer. I will cherish this article. People build their entire world views to run from this and here you are depicting the profound brutality - not obscured with fluff, but stripped naked by your words.

a world beyond the reach of God, an utterly unprotected world where anything at all can happen. ... Someone who wants to dance the deadly dance with Nature, does need to understand what they're up against: Absolute, utter, exceptionless neutrality. ... challenges are not calibrated to your skills

I feel a great relief reading these simple and profound insights: somebody else gets it. Somebody who others pay attention to is warning the rest and I now have a new way to communicate this to others.

I don't say that every rationalist should meditate on neutrality.

I have experienced the neutrality. There's no turning back. I know that there is nothing to trust.

I didn't expect to find any solace that would lighten the harrowing effect of this realization. Your communication gave me that solace. Thank you.

I have never really had a problem with the complete neutrality of life. It doesn't really change what happens, since it's inevitable. I think there is a certain art to learning and not let what you learn consume you with despair. If there is no inherent meaning of life and it is all just what happens, so what? It won't really change anything about your life or how you live it unless you allow it to. And if you die and the part of reality that is your consciousness will entirely seize to exist, so what? You won't be alive to give a dahm about it.

Acknowledge the truth, give it a polite nod, and continue on with your life. It will be there regardless, as will your immediate life

Yet who prohibits? Who prevents it from happening?

Eliezer seems absurdly optimistic to me. He is relying on some unseen entity to reach in and keep the laws of physics stable in our universe. We already see lots of evidence that they are not truly stable, for example we believe in both the electroweak transition and earlier transitions, of various natures depending on your school of physics. We /just saw/ in 1998 that unknown laws of physics can creep up and spring out by surprise, suddenly 'taking over' 74 percent of the Universe's postulated energy supply.

Who is keeping the clockwork universe running? Or, why is the hardware and software (operating system etc) for the automata fishbowl so damned stable? Is it part of the Library of Babel? Well, that's a good explanation, but the Bible's universe is in there also and arguably a priori more probable than a computer system that runs many worlds quantum mechanics sims on a panuniversal scale, and never halts and/or catches fire. It is very hard to keep a perfectly stable web server going when demand keeps expanding, and that seems much simpler.

Look into the anthropic principle literature and search on 'Lee Smolin' or start somewhere like http://evodevouniverse.com/wiki/Cosmological_natural_selection_(fecund_universes) for some reasoned speculation on how we might have wound up in such a big, rich, diverse universe from simpler origins.

I don't think it is rational to accept the stability of natural law without some reasonable model for either the origins of said law, or some timeless physics model that has stable physics with apparently consistent evolution as an emergent property, or it's a gift from entities with powers far beyond ours, or some such.

If the worst that can happen is annihilation, you're living in a fool's paradise. Many people have /begged/ for true death and in the animal world things happen all the time that are truly horrifying to most human eyes. I will avoid going into sickening detail here, but let me refer you to H.P. Lovecraft and Charles Stross for some examples of how the universe might have been slightly less friendly.

On WWII - the "more or less the same way" is actually rather flexible. Usual advocates of "don't put it all on Hitler" say (AFAIK) something among the lines "historical balance was such that SOME major war involving Germany was bound to occur and, given the rise of tech, be about as deadly and as propaganda-fueled; Hitler did not invent or destroy so much tech as to change that particular statement" but not "there would arise a party against Jews which would win over Communists, make an alliance with another "communists", then betray them". Without Hitler we may have found Communist Germany led by Telman out to destroy Capitalism, instead of Jews, and then fighting against USSR before actually managing that for reasons similar to why Trotsky was exiled and then killed - because Stalin would fear Telman the Führer to be an opponent of his. Or something entirely different - I've already given too much detail to my alternative history. The only thing advocated is "major, deadly, propaganda-fueled war in several decades after the Last War with Germany as a prominent player".

It’s interesting that you say that a Good God wouldn’t destroy a soul, as one of the biggest issues I’m currently finding myself having with Orthodox Judaism is that according to the Talmud at least, there have been a number of historical cases of souls being completely destroyed, which seems rather incompatible with the rest of Orthodox Jewish morality....I don’t know about the Christian or Muslim God, but they do both seem to believe that some people burn in hell forever, which is arguably worse than simply not existing. I really don’t get how this isn’t discussed more often in conventional theism...

I do believe that World War II was largely doomed. But I also believe that the bombs on Hiroshima could easily have been avoided, it seems everyone would agree that in this case it really was decided by a roll of the dice. I sometimes have moments when I think of the zero-sum game as something that would always be found by science, but then I remind myself that we could easily live in a world where there is no such thing, we just got lucky that our indifferent universe at least allows some movement towards the light. I have personally experienced causeless injustice, and I have no faith in faith, so it obviously feels to me that we live in a causal world that is not under the influence of any plan.