Being Half-Rational About Pascal's Wager is Even Worse

post by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T05:20:25.235Z · LW · GW · Legacy · 162 comments

Contents

162 comments

For so long as I can remember, I have rejected Pascal's Wager in all its forms on sheerly practical grounds: anyone who tries to plan out their life by chasing a 1 in 10,000 chance of a huge payoff is almost certainly doomed in practice.  This kind of clever reasoning never pays off in real life...

...unless you have also underestimated the allegedly tiny chance of the large impact.

For example.  At one critical junction in history, Leo Szilard, the first physicist to see the possibility of fission chain reactions and hence practical nuclear weapons, was trying to persuade Enrico Fermi to take the issue seriously, in the company of a more prestigious friend, Isidor Rabi:

I said to him:  "Did you talk to Fermi?"  Rabi said, "Yes, I did."  I said, "What did Fermi say?"  Rabi said, "Fermi said 'Nuts!'"  So I said, "Why did he say 'Nuts!'?" and Rabi said, "Well, I don't know, but he is in and we can ask him." So we went over to Fermi's office, and Rabi said to Fermi, "Look, Fermi, I told you what Szilard thought and you said ‘Nuts!' and Szilard wants to know why you said ‘Nuts!'" So Fermi said, "Well… there is the remote possibility that neutrons may be emitted in the fission of uranium and then of course perhaps a chain reaction can be made." Rabi said, "What do you mean by ‘remote possibility'?" and Fermi said, "Well, ten per cent." Rabi said, "Ten per cent is not a remote possibility if it means that we may die of it.  If I have pneumonia and the doctor tells me that there is a remote possibility that I might die, and it's ten percent, I get excited about it."  (Quoted in 'The Making of the Atomic Bomb' by Richard Rhodes.)

This might look at first like a successful application of "multiplying a low probability by a high impact", but I would reject that this was really going on.  Where the heck did Fermi get that 10% figure for his 'remote possibility', especially considering that fission chain reactions did in fact turn out to be possible?  If some sort of reasoning had told us that a fission chain reaction was improbable, then after it turned out to be reality, good procedure would have us go back and check our reasoning to see what went wrong, and figure out how to adjust our way of thinking so as to not make the same mistake again.  So far as I know, there was no physical reason whatsoever to think a fission chain reaction was only a ten percent probability.  They had not been demonstrated experimentally, to be sure; but they were still the default projection from what was already known.  If you'd been told in the 1930s that fission chain reactions were impossible, you would've been told something that implied new physical facts unknown to current science (and indeed, no such facts existed).  After reading enough historical instances of famous scientists dismissing things as impossible when there was no physical logic to say that it was even improbable, one cynically suspects that some prestigious scientists perhaps came to conceive of themselves as senior people who ought to be skeptical about things, and that Fermi was just reacting emotionally.  The lesson I draw from this historical case is not that it's a good idea to go around multiplying ten percent probabilities by large impacts, but that Fermi should not have pulled out a number as low as ten percent.

Having seen enough conversations involving made-up probabilities to become cynical, I also strongly suspect that if Fermi had foreseen how Rabi would reply, Fermi would've said "One percent".  If Fermi had expected Rabi to say "One percent is not small if..." then Fermi would've said "One in ten thousand" or "Too small to consider" - whatever he thought would get him off the hook.  Perhaps I am being too unkind to Fermi, who was a famously great estimator; Fermi may well have performed some sort of lawful probability estimate on the spot.  But Fermi is also the one who said that nuclear energy was fifty years off in the unlikely event it could be done at all, two years (IIRC) before Fermi himself oversaw the construction of the first nuclear pile.  Where did Fermi get that fifty-year number from?  This sort of thing does make me more likely to believe that Fermi, in playing the role of the solemn doubter, was just Making Things Up; and this is no less a sin when you make up skeptical things.  And if this cynicism is right, then we cannot learn the lesson that it is wise to multiply small probabilities by large impacts because this is what saved Fermi - if Fermi had known the rule, if he had seen it coming, he would have just Made Up an even smaller probability to get himself off the hook.  It would have been so very easy and convenient to say, "One in ten thousand, there's no experimental proof and most ideas like that are wrong!  Think of all the conjunctive probabilities that have to be true before we actually get nuclear weapons and our own efforts actually made a difference in that!" followed shortly by "But it's not practical to be worried about such tiny probabilities!"  Or maybe Fermi would've known better, but even so I have never been a fan of trying to have two mistakes cancel each other out.

I mention all this because it is dangerous to be half a rationalist, and only stop making one of the two mistakes.  If you are going to reject impractical 'clever arguments' that would never work in real life, and henceforth not try to multiply tiny probabilities by huge payoffs, then you had also better reject all the clever arguments that would've led Fermi or Szilard to assign probabilities much smaller than ten percent.  (Listing out a group of conjunctive probabilities leading up to taking an important action, and not listing any disjunctive probabilities, is one widely popular way of driving down the apparent probability of just about anything.)  Or if you would've tried to put fission chain reactions into a reference class of 'amazing new energy sources' and then assigned it a tiny probability, or put Szilard into the reference class of 'people who think the fate of the world depends on them', or pontificated about the lack of any positive experimental evidence proving that a chain reaction was possible, blah blah blah etcetera - then your error here can perhaps be compensated for by the opposite error of then trying to multiply the resulting tiny probability by a large impact.  I don't like making clever mistakes that cancel each other out - I consider that idea to also be clever - but making clever mistakes that don't cancel out is worse.

On the other hand, if you want a general heuristic that could've led Fermi to do better, I would suggest reasoning that previous-historical experimental proof of a chain reaction would not be strongly be expected even in worlds where it was possible, and that to discover a chain reaction to be impossible would imply learning some new fact of physical science which was not already known.  And this is not just 20-20 hindsight; Szilard and Rabi saw the logic in advance of the fact, not just afterward - though not in those exact terms; they just saw the physical logic, and then didn't adjust it downward for 'absurdity' or with more complicated rationalizations.  But then if you are going to take this sort of reasoning at face value, without adjusting it downward, then it's probably not a good idea to panic every time you assign a 0.01% probability to something big - you'll probably run into dozens of things like that, at least, and panicking over them would leave no room to wait until you found something whose face-value probability was large.

I don't believe in multiplying tiny probabilities by huge impacts.  But I also believe that Fermi could have done better than saying ten percent, and that it wasn't just random luck mixed with overconfidence that led Szilard and Rabi to assign higher probabilities than that.  Or to name a modern issue which is still open, Michael Shermer should not have dismissed the possibility of molecular nanotechnology, and Eric Drexler will not have been randomly lucky when it turns out to work: taking current physical models at face value imply that molecular nanotechnology ought to work, and if it doesn't work we've learned some new fact unknown to present physics, etcetera.  Taking the physical logic at face value is fine, and there's no need to adjust it downward for any particular reason; if you say that Eric Drexler should 'adjust' this probability downward for whatever reason, then I think you're giving him rules that predictably give him the wrong answer.  Sometimes surface appearances are misleading, but most of the time they're not.

A key test I apply to any supposed rule of reasoning about high-impact scenarios is, "Does this rule screw over the planet if Reality actually hands us a high-impact scenario?" and if the answer is yes, I discard it and move on.  The point of rationality is to figure out which world we actually live in and adapt accordingly, not to rule out certain sorts of worlds in advance.

There's a doubly-clever form of the argument wherein everyone in a plausibly high-impact position modestly attributes only a tiny potential possibility that their face-value view of the world is sane, and then they multiply this tiny probability by the large impact, and so they act anyway and on average worlds in trouble are saved.  I don't think this works in real life - I don't think I would have wanted Leo Szilard to think like that.  I think that if your brain really actually thinks that fission chain reactions have only a tiny probability of being important, you will go off and try to invent better refrigerators or something else that might make you money.  And if your brain does not really feel that fission chain reactions have a tiny probability, then your beliefs and aliefs are out of sync and that is not something I want to see in people trying to handle the delicate issue of nuclear weapons.  But in any case, I deny the original premise:  I do not think the world's niches for heroism must be populated by heroes who are incapable in principle of reasonably distinguishing themselves from a population of crackpots, all of whom have no choice but to continue on the tiny off-chance that they are not crackpots.

I haven't written enough about what I've begun thinking of as 'heroic epistemology' - why, how can you possibly be so overconfident as to dare even try to have a huge positive impact when most people in that reference class blah blah blah - but on reflection, it seems to me that an awful lot of my answer boils down to not trying to be clever about it.  I don't multiply tiny probabilities by huge impacts.  I also don't get tiny probabilities by putting myself into inescapable reference classes, for this is the sort of reasoning that would screw over planets that actually were in trouble if everyone thought like that.  In the course of any workday, on the now very rare occasions I find myself thinking about such meta-level junk instead of the math at hand, I remind myself that it is a wasted motion - where a 'wasted motion' is any thought which will, in retrospect if the problem is in fact solved, not have contributed to having solved the problem.  If someday Friendly AI is built, will it have been terribly important that someone have spent a month fretting about what reference class they're in?  No.  Will it, in retrospect, have been an important step along the pathway to understanding stable self-modification, if we spend time trying to solve the Lobian obstacle?  Possibly.  So one of these cognitive avenues is predictably a wasted motion in retrospect, and one of them is not.  The same would hold if I spent a lot of time trying to convince myself that I was allowed to believe that I could affect anything large, or any other form of angsting about meta.  It is predictable that in retrospect I will think this was a waste of time compared to working on a trust criterion between a probability distribution and an improved probability distribution.  (Apologies, this is a technical thingy I'm currently working on which has no good English description.)

But if you must apply clever adjustments to things, then for Belldandy's sake don't be one-sidedly clever and have all your cleverness be on the side of arguments for inaction.  I think you're better off without all the complicated fretting - but you're definitely not better off eliminating only half of it.

And finally, I once again state that I abjure, refute, and disclaim all forms of Pascalian reasoning and multiplying tiny probabilities by large impacts when it comes to existential risk.  We live on a planet with upcoming prospects of, among other things, human intelligence enhancement, molecular nanotechnology, sufficiently advanced biotechnology, brain-computer interfaces, and of course Artificial Intelligence in several guises.  If something has only a tiny chance of impacting the fate of the world, there should be something with a larger probability of an equally huge impact to worry about instead.  You cannot justifiably trade off tiny probabilities of x-risk improvement against efforts that do not effectuate a happy intergalactic civilization, but there is nonetheless no need to go on tracking tiny probabilities when you'd expect there to be medium-sized probabilities of x-risk reduction.  Nonetheless I try to avoid coming up with clever reasons to do stupid things, and one example of a stupid thing would be not working on Friendly AI when it's in blatant need of work.  Elaborate complicated reasoning which says we should let the Friendly AI issue just stay on fire and burn merrily away, well, any complicated reasoning which returns an output this silly is automatically suspect.

If, however, you are unlucky enough to have been cleverly argued into obeying rules that make it a priori unreachable-in-practice for anyone to end up in an epistemic state where they try to do something about a planet which appears to be on fire - so that there are no more plausible x-risk reduction efforts to fall back on, because you're adjusting all the high-impact probabilities downward from what the surface state of the world suggests...

Well, that would only be a good idea if Reality were not allowed to hand you a planet that was in fact on fire.  Or if, given a planet on fire, Reality was prohibited from handing you a chance to put it out.  There is no reason to think that Reality must a priori obey such a constraint.

EDIT:  To clarify, "Don't multiply tiny probabilities by large impacts" is something that I apply to large-scale projects and lines of historical probability.  On a very large scale, if you think FAI stands a serious chance of saving the world, then humanity should dump a bunch of effort into it, and if nobody's dumping effort into it then you should dump more effort than currently into it.  On a smaller scale, to compare two x-risk mitigation projects in demand of money, you need to estimate something about marginal impacts of the next added effort (where the common currency of utilons should probably not be lives saved, but "probability of an ok outcome", i.e., the probability of ending up with a happy intergalactic civilization).  In this case the average marginal added dollar can only account for a very tiny slice of probability, but this is not Pascal's Wager.  Large efforts with a success-or-failure criterion are rightly, justly, and unavoidably going to end up with small marginally increased probabilities of success per added small unit of effort.  It would only be Pascal's Wager if the whole route-to-an-OK-outcome were assigned a tiny probability, and then a large payoff used to shut down further discussion of whether the next unit of effort should go there or to a different x-risk.

162 comments

Comments sorted by top scores.

comment by [deleted] · 2013-04-18T18:46:48.697Z · LW(p) · GW(p)

In summary, you could say that I'm in this field less because of what you could do with a quantum computer, than because of what the possibility of quantum computers already does to our conception of the world. Either practical quantum computers can be built, and the limits of the knowable are not what we thought they are; or they can't be built, and the principles of quantum mechanics themselves need revision; or there's a yet-undreamt method to simulate quantum mechanics efficiently using a conventional computer. All three of these possibilities sound like crackpot speculations, but at least one of them is right!

  • Scott Aaronson, in the preface of "Quantum Computing Since Democritus"

Ideally, you put yourself in a scenario where verifying any possibility has a huge payoff.

comment by Scott Alexander (Yvain) · 2013-04-18T06:46:53.709Z · LW(p) · GW(p)

Are you classifying 10% as a Pascal-level probability? How big does a probability have to get before you don't think Pascal-type considerations apply to it?

Are you suggesting that if there was (for example) a ten percent probability of an asteroid hitting the Earth in 2025, we should devote fewer resources to asteroid prediction/deflection than simple expected utility calculations would predict?

Replies from: orthonormal, Jonathan_Graehl
comment by orthonormal · 2013-04-18T14:20:38.043Z · LW(p) · GW(p)

No, he's saying that 10% and 1% are non-Pascalian probabilities for x-risks, but that 1-in-10,000 is effectively Pascalian.

Replies from: ciphergoth
comment by Paul Crowley (ciphergoth) · 2013-04-18T20:35:27.828Z · LW(p) · GW(p)

I don't think it counts as "Pascalian" until it starts to scrape below the threshold of probabilities you can meaningfully assert about propositions. If we were basically assured of a bright astronomical future so long as person X doesn't win the lottery, I wouldn't say that worrying that X might win the lottery was a Pascalian risk.

comment by Jonathan_Graehl · 2013-04-18T07:51:46.743Z · LW(p) · GW(p)

I didn't like his anecdote, either.

I think you've read him wrong. He's opposed to "don't pay attention to high utility * small probability scenarios", on the basis of heroism.

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:04:10.399Z · LW(p) · GW(p)

I'm usually fine with dropping a one-time probability of 0.1% from my calculations. 10% is much too high to drop from a major strategic calculation but even so I'd be uncomfortable building my life around one. If this was a very well-defined number as in the asteroid calculation then it would be more tempting to build a big reference class of risks like that one and work on stopping them collectively. If an asteroid were genuinely en route, large enough to wipe out humanity, possibly stoppable, and nobody was doing anything about this 10% probability, I would still be working on FAI but I would be screaming pretty loudly about the asteroid on the side. If the asteroid is just going to wipe out a country, I'll make sure I'm not in that country and then keep working on x-risk.

comment by atorm · 2013-04-18T12:14:37.919Z · LW(p) · GW(p)

What probability are you assigning to cryonics working that makes you think it's a good idea? I was under the impression that the standard LW argument for signing up was (tiny probability of success)*(monumental heap of utility if it works)=(a good investment). If that's not your argument, what is?

Replies from: orthonormal, None, Luke_A_Somers, Eliezer_Yudkowsky
comment by orthonormal · 2013-04-18T14:42:20.860Z · LW(p) · GW(p)

I was under the impression that the standard LW argument for signing up was (tiny probability of success)*(monumental heap of utility if it works)=(a good investment). If that's not your argument, what is?

The standard LW argument is that cryonics has a non-tiny probability of success. I did my own estimate, and roughly speaking, P(success) is at least P(A)P(B|A)P(C|A,B)P(D|A,B,C), where

  • A = materialism, i.e. preserving the brain or just the information of the brain-state is enough to preserve the mind
  • B = the freezing process preserves the information of the brain-state in such a way that plausible future tech can recover it
  • C = future society develops the tech to actually recover the info from cryopreserved brains without extravagant energy costs
  • D = my cryonics provider keeps me frozen for the entire time, and someone in the future sees fit to revive me

And my honest estimates were, roughly, P(A) > .95, P(B|A) > .8, P(C|A,B) > .3, and P(D|A,B,C) > .2, giving an overall lower-bound estimate of about 5% (with a lot of metauncertainty, obviously); then I tried to estimate how much waking up in the future would really be worth to me in terms of my current values compared to the value of money now, and overall determined that it was worth it for me to sign up (but not overwhelmingly; had it been 10 times the actual cost of $20 a month for the insurance and dues, I'd have waited until I was significantly richer).

And the point of the post is that 5% is not a Pascal's Wager-esque tiny probability, but something on the level of health risks that we do in fact take seriously.

(You're welcome to take exception to my estimates, of course, but the main point is that I signed up for non-Pascalian reasons. There's actually more to consider, including the effect of MWI and anthropic reasoning on C and D, and consideration about the motivations that someone might revive cryopreserved people, but this is a good enough sketch for current purposes.)

Replies from: None, Eliezer_Yudkowsky, atorm
comment by [deleted] · 2013-04-18T15:51:52.779Z · LW(p) · GW(p)

E (should go between A and B given your chronological ordering scheme): You die in such a way that high-quality vitrification/plastination is possible. (This variable gets overlooked way too frequently in these calculations).

Replies from: orthonormal
comment by orthonormal · 2013-04-18T16:29:24.249Z · LW(p) · GW(p)

Ah, good call. For a young and healthy person like me, that's a significant factor, since the likely causes of untimely death would probably be unexpected and/or violent. (Anyone have an idea about how to estimate this one properly?)

Replies from: DaFranker
comment by DaFranker · 2013-04-18T18:00:14.507Z · LW(p) · GW(p)

(Anyone have an idea about how to estimate this one properly?)

Get some base rate cause-of-death statistics for people in your age group and geography. Exclude those deaths for which you are certain you are exempt (or just discount them appropriately according to your beliefs that you might die of them, but that's a lot of work and the uncertainty is already so large that this won't affect much of anything).

The hard work is finding good comprehensive stats on this. The WHO databases could be good fallback if you don't have anything better / more specific. For Canada, these proved quite useful to get a general picture.

I did some research for myself, and came to the conclusion that E is (probably) low enough until some age group that I shouldn't bother with cryonics until then. For my specifics, the rough base rates for sudden or destructive death are above 50%, while it's down to something like 15% at 45-54.

The actual math for deciding that I used ended up having a few more factors, but overall what I've got is "don't sign up for cryonics until 40+ unless some other evidence comes up (or the price goes down)".

Replies from: None, army1987
comment by [deleted] · 2013-04-18T18:39:48.131Z · LW(p) · GW(p)

I'm surprised your math came out close enough for a factor of less-than-two to make a difference.

Replies from: DaFranker
comment by DaFranker · 2013-04-18T19:24:12.442Z · LW(p) · GW(p)

Well, yeah, it wasn't just a factor of less than two. Discounting rates, decrease of marginal u / $, P(B) probably increasing over time, and a few other things came into account.

Not to mention the sheer increase in natural mortality rates - %-of-deaths gives you a ratio by which to cut down odds of success, but deaths-per-population is what counts in calculating expected utility of signing up for cryonics at a given time. These rates climb very sharply past 40, especially for the causes of death that cryonics can actually help with.

Overall though, I must admit (after taking another look at it) that my math is/was full of potential holes to poke at. I may be going over it more carefully at some point in the near future, or I may just end up signing up for cryonics to save myself the trouble and never have to think about it this much again (barring new evidence or other events, of course).

comment by A1987dM (army1987) · 2013-04-18T18:45:40.435Z · LW(p) · GW(p)

I did some research for myself, and came to the conclusion that E is (probably) low enough until some age group that I shouldn't bother with cryonics until then.

On the other hand, while E increases with age, so does the cost of life insurance. On the third hand, so does your income and your net worth.

Replies from: army1987
comment by A1987dM (army1987) · 2013-04-19T12:32:57.616Z · LW(p) · GW(p)

And on the fourth hand, anti-agathics becoming available while you're still alive would bring E back down.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:09:18.481Z · LW(p) · GW(p)

Expanding conjunctive probabilities without expanding disjunctive probabilities is another classic form of one-sided rationality. If I wanted to make cryonics look more probable than this, I would individually list out many different things that could go right.

Replies from: orthonormal, army1987, Luke_A_Somers
comment by orthonormal · 2013-04-18T21:01:45.315Z · LW(p) · GW(p)

For the purpose of establishing that it's not a Pascalian probability, it suffices to talk about a lower bound on the main line of reasoning.

Ah, I see that I said "estimate" instead of "lower bound" in the critical place. I'll edit.

comment by A1987dM (army1987) · 2013-04-20T09:15:31.805Z · LW(p) · GW(p)

Can you give a few examples?

comment by Luke_A_Somers · 2013-04-18T21:28:14.670Z · LW(p) · GW(p)

In this case, I'm not seeing the disjunctive possibilities that lead one to sign up for cryo in this particular case. ABCD seem to be phrased pretty broadly, and A and B in particular are already pretty big. Do you mean as an alternate to D that, say, a new cryo provider takes over the abandoned preserved heads before they thaw? Or as an alternate to C, that even though the cost is high, they go ahead and do it anyway?

Beyond that, I only see scenarios that are nice but don't point one toward cryopreservation. Like, time travel scans of dying people meaning no one ever really died is wonderful, but it means getting cryopreserved only did good in that your family wouldn't be QUITE as sad you were gone in the time before they 'died'.

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T22:24:15.954Z · LW(p) · GW(p)

Do you mean as an alternate to D that, say, a new cryo provider takes over the abandoned preserved heads before they thaw?

Sure. That happened already once in history (though there was, even earlier, a loss-thaw). It's why all modern cryo organizations are very strict about demanding advance payment, despite their compassionate hearts screaming at them not to let their friends die because of mere money. Sucks to be them, but they've got no choice.

Or as an alternate to C, that even though the cost is high, they go ahead and do it anyway?

Yep. I'd think FAI scenarios would tend to yield that.

Basically I always sigh sadly when somebody's discussing a future possibility and they throw up some random conjunction of conditional probabilities, many steps of which are actually pretty darned high when I look at them, with no corresponding disjunctions listed. This is the sort of thinking that would've led Fermi to cleverly assign a probability way lower than 10% to having an impact, by the time he was done expanding all the clever steps of the form "And then we can actually persuade the military to pay attention to us..." If you're going to be silly about driving down all impact probabilities to something small via this sort of conjunctive cleverness, you'd better also be silly and multiply the resulting small probability by a large payoff, so you won't actually ignore all possible important issues.

Replies from: CarlShulman
comment by CarlShulman · 2013-04-18T23:17:03.763Z · LW(p) · GW(p)

"And then we can actually persuade the military to pay attention to us..."

The government did sit on it for quite a while, delaying the bomb until after the defeat of Germany. Nudges from Britain were important in getting things moving.

Replies from: SilasBarta
comment by SilasBarta · 2013-04-19T12:18:58.778Z · LW(p) · GW(p)

The military "paid attention to them" long before that though.

comment by atorm · 2013-04-18T15:24:04.347Z · LW(p) · GW(p)

Ah. Your reasoning seems to be sound, but I would estimate P(B|A) << 0.8. Thank you for the explanation.

Replies from: orthonormal, ciphergoth, Tenoke
comment by orthonormal · 2013-04-18T21:13:34.012Z · LW(p) · GW(p)

My estimate of P(B|A) would be lower, too, had I not read Drexler's Engines of Creation. The theoretical limits of useful nanotech allow for devices way smaller and more efficient than the behemoths of weakly-bonded amino acid chains that make up our cells, so that cheaply repairing intracellular damage is not unreasonable at that point. (Let alone the alternative of scanning the contents and doing the real work on a computer simulation.)

At that point, it becomes a question not of whether cells are structurally intact, but of whether their inter-relationships at the necessary scale remain stably encoded after the vitrification process. (Since there are several different scales which might be "the necessary scale" for recovering a mind, this does involve some uncertainty.) Pending the outcomes of the BPF Prize and the NEMALOAD Project, I'm pretty optimistic on that front. (See here for my declarations of how I'd update given bad news on either of those projects.)

comment by Paul Crowley (ciphergoth) · 2013-04-18T20:39:36.357Z · LW(p) · GW(p)

Why?

Replies from: atorm
comment by atorm · 2013-04-18T23:46:10.100Z · LW(p) · GW(p)

My understanding of neurobiology (BS in biology, current Plant Biology grad student) leads me to believe that the mind is not stored strictly statically in relationships between neurons, but also in the subcellular states of several proteins. These states are unlikely to be preserved in time for cryopreservation. They probably will be disrupted by the freezing process even if a living brain were to be preserved.

Replies from: ciphergoth, ciphergoth
comment by Paul Crowley (ciphergoth) · 2013-04-20T08:08:06.061Z · LW(p) · GW(p)

I need to write my "You appear to be making an argument against the technical feasability of cryonics as a comment on a blog post" blog post. I've already blogged all the pieces, but I need to write the one piece that ties it all together.

Replies from: TitaniumDragon
comment by TitaniumDragon · 2013-04-20T23:04:30.193Z · LW(p) · GW(p)

There's a lot of good reasons to believe that cyronics is highly infeasible. I agree that P(B|A) is low, and P(D|A,B,C) is also absurdly low. We don't care about starving people in Africa today; what is the likelihood that we care about dead frozen people in the future, especially if we have to spend millions of dollars resurrecting them (as is more than likely the case), especially given how difficult it would be to do so? And that's assuming we can even fix whatever caused the problem in the first place; if they die of brain cancer, how are we supposed to fix them, exactly?

Senility is another issue, as it could potentially permanently destroy portions of your brain, rendering you no longer you.

But really I find the overall probability of everything incredibly bad.

But even if we CAN do it, I suspect that it would not be worth doing because what we'd really be doing is just building a copy of you from a frozen copy most likely, in which case you, personally, are still dead; the fact that a copy of you is running around doesn't really change that, and also raises the problem that they could make any numbers of copies of people, which likely would make them dubious about doing so in the first place.

Replies from: gwern, pianoforte611
comment by gwern · 2013-04-20T23:20:02.830Z · LW(p) · GW(p)

what is the likelihood that we care about dead frozen people in the future, especially if we have to spend millions of dollars resurrecting them (as is more than likely the case), especially given how difficult it would be to do so?

This is a standard criticism people come up with after 5 seconds of thought, and a perfect example of http://lesswrong.com/lw/h8n/litany_of_a_bright_dilettante/

Do you really think that no one in cryonics hasn't ever thought - 'wait a second! why would anyone in the future even bother putting in the work?' - and you have successfully exposed a fatal ~70-year-old blindspot in a comment written in a few seconds?

Replies from: ciphergoth, TitaniumDragon
comment by Paul Crowley (ciphergoth) · 2013-04-21T12:51:36.306Z · LW(p) · GW(p)

And it's not necessarily that the replies to this problem are good, but that they are what you need to reply to. There's nothing to be said for making a serve we've already returned; to advance the discussion, you need to actually hit the ball back into our court, by reading and replying to the standard replies to this point.

Replies from: TitaniumDragon
comment by TitaniumDragon · 2013-04-25T20:52:16.121Z · LW(p) · GW(p)

I have never actually seen any sort of cogent response to this issue. Ever. I see it being brushed aside constantly, along with the magical brain restoration technology necessary for this, but I've never actually seen someone go into why, exactly, anyone would bother to thaw them out and revive them, even if it WAS possible to do. They are, all for all intents and purposes, dead, from a legal, moral, and ethical standpoint. Not only that, but defrosting them has little actual practical benefit - while there is obvious value to the possible cryopreservation of organs, that is only true if there aren't better way of preserving organs for shipment and preservation. As things are today, however, that seems unlikely - we already have means of shipping organs and keeping them alive, and given the current trend towards growing organs, it seems far more likely to me that the actual method will be to grow organs and keep them alive rather than keep them in cryopreservation, and without that technology being worked on, there is pretty much no value at all to developing unfreezing technology.

That means that, realistically speaking, the only purpose of such technology would be, say, shipping humans to another planet, which while probably not really rational from an economic perspective is at least somewhat reasonably likely. But even still that is a different kettle of fish - the technology in question may not resemble present day cryogenics at all, and as such may be utterly useless for unfreezing people from present-day cyrogenic treatments. Once you can prove that people CAN be revived in that way, then there is much more incentive towards cryogenics... but that is not present day cryogenics, and there is no evidence to suggest future cryogenic treatments will be very similar to present ones.

Okay, so even all that technology aside, let's assume, at some point, we do develop this technology for whatever reason. At this point, not only do you have to bear the expense of unfreezing these people, but you also have to bear the expense of fixing whatever is wrong with them (which, I will note, actually killed them in the past), as well as fixing whatever damage was done to them prior to being cryogenically frozen (and lest we forget, 10 minutes without oxygen is very likely to cause irreparable brain damage in humans who survive - let alone humans who are beyond what we in the present day can deal with). This is likely to be very, very expensive indeed, and there is little real incentive for someone in the future to spend their money in this way instead of on something else. You are basically hoping for some rich idiot to not only be capable of doing this, but also being willing to do it and having the legal ability to do so (as, lest we forget, there are laws about playing around with human corpses, and I suspect that it is unlikely they will change positively for frozen people in the future - as if they do change, what are the odds that your frozen body won't be used in some other sort of experiment?).

I have never seen arguments which really address these issues. People wave their hands and talk about nanotechnology and brain uploading, but as someone who has actually dealt with nanotechnology I can tell you that it is not, in fact, magical, nor is it capable of many of the feats people believe it will be capable of, nor will it EVER be capable of many of the feats that people imagine it will be capable of. Nanomachines have to be provided with energy the same as anything else, among other, major issues, and I have some severe doubts about the unfreezing process in the first place due to various issues of thermodynamics and the fact that the bodies are not frozen in a setup which is likely to facilitate unfreezing them.

A lot of cryonics arguments basically boil down to "future technology is magic", and that's a pretty big problem for any sort of rational argumentation. "You can't prove that they won't be able to revive me" can be used for all sorts of terrible arguments, as the burden of proof is on the person making the argument that it IS possible, not on the person holding to the present day "we can't, and see no way to do so."

I mean, you look at things like:

http://www.alcor.org/Library/html/resuscitation.htm

The technology in here is, quite literally, magic. It doesn't exist, and it won't exist. Ever. Things on the level are very dumb; they cannot be intelligent, and they cannot act intelligently, because they are too small, too simple. The bits where they stick stuff into your cells is where things get really ridiculous, but even before then, those little nanomachines are going to have real issues doing what you are hoping for, and would have to be custom built for the task at hand. We're talking enormous expense if it is even possible to do at all, and given the extremely small cryogenic population, the odds of perfecting the technology prior to running out of dead people is not very good. Remember, if the result is brain dead or severely brain damaged, it is still a failure. But even these sorts of nanomachines are very questionable; transistors are only going to get 256 times smaller at most, which makes me question whether said nanomachines can function in the way that is hoped for at all. Of course, this is not necessarily a barrier to, say, a different sort of nanomachine (though they'd be more micromachines really, on the scale of a cell rather than on the scale of large molecules) which was controlled by some sort of external process with the little machines being extensions/remotes of it, but this is still questionable.

Extreme expense, questionable technology (which would have to be custom developed for the purpose), the question of whether cryonics is even a viable technological route for something else for cryogenic revival to piggyback on, likely custom technology for reviving people who have died of things that people no longer die of because of earlier preventative measures (why build something to fix someone with late stage cancer when no one gets late state cancer anymore?), legal problems, the necessity for experimental subjects... all of these things add up to the question of why these hypothetical future people are even going to bother. That's assuming it is even ethical to revive someone who is, say, not genetically engineered and therefore would be at the bottom of the societal heap if they were revived.

Replies from: drethelin
comment by drethelin · 2013-04-26T16:57:08.379Z · LW(p) · GW(p)

We spend millions of dollars digging up dinosaurs.

People get really excited when we find things like Troy.

Look at all the antique stores that are around.

Why WOULDN'T people get revived?

Replies from: TitaniumDragon, David_Gerard
comment by TitaniumDragon · 2013-04-27T01:06:57.784Z · LW(p) · GW(p)

Humans aren't dinosaurs, nor can you put them on your mantlepiece as a conversation piece. They are not property, but living, independent persons.

Replies from: drethelin, wedrifid
comment by drethelin · 2013-04-27T04:10:58.004Z · LW(p) · GW(p)

That only makes them insanely more valuable for reality tv

comment by wedrifid · 2013-04-27T09:00:33.956Z · LW(p) · GW(p)

Humans aren't dinosaurs, nor can you put them on your mantlepiece as a conversation piece.

Speak for yourself. (People have at times kept humans for similar purposes and there is no reason why future intelligent agents could not do so.)

They are not property, but living, independent persons.

That is either a false dichotomy or a No True Scottsman equivocation on 'property'.

comment by David_Gerard · 2013-04-28T13:59:28.639Z · LW(p) · GW(p)

This implies that the Drake equation for cryonics needs an explicit term for "being one of the lucky first few revivals, in the short time when that's still novel".

comment by TitaniumDragon · 2013-04-26T07:59:48.606Z · LW(p) · GW(p)

Dunning-Kruger and experience with similar religious movements suggests otherwise.

It takes someone who really thinks about most things very little time to come up with very obvious objections to most religious doctrine, and given the overall resemblance of cryonics to religion (belief in future resurrection, belief that donating to the church/cyronics institution will bring tangible rewards to yourself and others in the future, belief in eternal life) its not really invalid to suggest something like that.

Which is more likely - that people are deluding themselves over the possibility of eternal life and don't actually have any real answers to the obvious questions, but conveniently ignore them because they see the upside as being so great, or that this has totally been answered, despite the fact that you didn't even articulate an actual answer to it in your response, or even link to it?

I'm pretty sure that, historically speaking, the former is far more likely than the latter.

If someone comes up to you and starts talking about how you have an immortal soul, if you've spent any time studying medicine or neurobiology at all, or have experience with anyone who has suffered brain damage, it really doesn't take you very long to come up with a good counterargument to people having souls. And people have argued about the nature of being for -thousands- of years, and dubiousness about souls has been around for considerably longer than cryonics has been. And yet, people still believe in souls, despite the fact that a very simple, five minutes of thought counterargument exists and has never been countered.

The fact that you did not have a counter for my argument and instead linked to a page which was meant to be a "take that" directed at me is evidence against you having an actual answer to my query, which is always a bad sign. This is not to say that it doesn't have an answer, but a quick, simple answer (or link) would be no more difficult to find than the litany article.

Indeed, after looking at the Alcor site, and reading around, all I really find are arguments against it. The best argument for it that I've seen is that resurrecting 20th century people might be profitable from an entertainment/educational standpoint, but I find even that to be a weak argument - not only is resuscitating someone for the purpose of entertainment deeply morally repugnant (and likely to be so into the future), but wikipedia and various other sources from the 20th and 21st century are likely to be far more valuable to historians, while writers will benefit more from creating their own characters who are considerably more interesting than real people - and it is considerably cheaper and less morally and legally questionable to do so.

So what is the argument for it? If it is so simple to resolve, then what is the resolution?

As ciphergoth pointed out, there isn't really a good answer here. And that is troubling given that the whole thing is pointless if no one is ever going to bring you back anyway. I was reading one article on Alcor which suggested that, even for a cyronics optimist, the odds of it actually paying off were 15% if he only used his most optimistic numbers - and I think his numbers about the technology are optimistic indeed. That's bad news, especially given the guy is someone who actually thinks that doing cryonics is worthwhile.

Replies from: gwern, ciphergoth
comment by gwern · 2013-04-26T22:34:01.259Z · LW(p) · GW(p)

Dunning-Kruger

You obviously have not actually read the Dunning-Krueger paper and understood what it showed.

and experience with similar religious movements suggests otherwise.

Name three. Like V_V, I suspect that for all that you glibly allude to 'cults' you have no personal experience and you have not acquainted yourself with even a surface summary of the literature, much like you have not bothered to so much as read a cryonics FAQ or book before thinking you have refuted it.

It takes someone who really thinks about most things very little time to come up with very obvious objections to most religious doctrine

And it takes even less time to notice that there are long thorough answers to the obvious objections. Your point here is true, but says far more about you than religion or cryonics; after all, many true things like heliocentrism or evolution have superficial easily thought-of objections which have been addressed in depth. Sometimes they work, sometimes they don't; the argument from evil is probably the single most obvious argument against Western religions, there are countless replies from theists of various levels of sophistication, and while I don't think any of them actually work, I also don't think someone going 'My mother died! God doesn't exist!' is contributing anything whatsoever. What, you think the theists somehow failed to notice that bad things happen? Of course they did notice, so if you want to argue against the existence of God, read up on their response.

Which is more likely - that people are deluding themselves over the possibility of eternal life and don't actually have any real answers to the obvious questions, but conveniently ignore them because they see the upside as being so great, or that this has totally been answered, despite the fact that you didn't even articulate an actual answer to it in your response, or even link to it?

If you had spent less time being arrogant, it might have occurred to you that I see this sort of flip reaction all the time in which people learn of cryonics and in five seconds think they've come up with the perfect objection and refuse to spend any time at all to disconfirm their objection. You are acting exactly like the person who said, "but it's not profitable to revive crypatients! QED you're all idiots and suckers", when literally the first paragraph of the Wikipedia article on ALCOR implies how they attempt to resolve this issue; here's a link to the discussion: http://lesswrong.com/lw/gh5/cryo_and_social_obligations/8d43

Notice how you are acting exactly like cheapviagra. You've come up with an obvious issue for cryonics, and rather than do any - gasp! - research; you commented on it. OK, fine. I then told you that there were replies to the issue, and that you should've known this because the issue is so obvious, and rather than learn a lesson about useful contributions, you are instead self-righteously criticizing me for not willing to drop everything and dig up every answer to every idle passing thought you have!

By the way, a benchmark I've found useful in discussing factual matters or matters with a long pre-existing literature is number of citations and hyperlinks per comment. You're still batting a zero.

I'm pretty sure that, historically speaking, the former is far more likely than the latter.

I'm impressed you've failed to notice that LW is maybe a little different from other sites and we have higher standards, and what happens 'historically' isn't terribly relevant.

The fact that you did not have a counter for my argument and instead linked to a page which was meant to be a "take that" directed at me is evidence against you having an actual answer to my query, which is always a bad sign. This is not to say that it doesn't have an answer, but a quick, simple answer (or link) would be no more difficult to find than the litany article.

Apparently you missed the point. The point was: stop being arrogant. Think for a freaking second about how obvious an argument might be and at least what the reply might be, if you cannot be arsed to look up actual sources. Do us the courtesy of not just thinking your way to your bottom line of 'cryonics sucks' but maybe a step beyond that too.

Indeed, after looking at the Alcor site, and reading around, all I really find are arguments against it. The best argument for it that I've seen is that resurrecting 20th century people might be profitable from an entertainment/educational standpoint

Really? That's the best argument? What did you read, exactly? Where do they say 'there's no reason to revive people except for entertainment'? Or are you just picking out the weakest possible argument because that's what you want to talk about?

As ciphergoth pointed out, there isn't really a good answer here.

What? No! That's not what ciphergoth meant at all! Here is what he said:

And it's not necessarily that the replies to this problem are good, but that they are what you need to reply to.

He did not say the answers were not good. He said, that even if the answers were not good, they are still what you need to deal with. You need to work on your reading comprehension if that's what you got out of his comment. (Shades of Dunning-Kruger, since you brought it up...) Or is this another aspect of you refusing to do any research, bring up the weakest lamest arguments you can find as strawmen, and grossly misinterpret what people have said?

I was reading one article on Alcor which suggested that, even for a cyronics optimist, the odds of it actually paying off were 15% if he only used his most optimistic numbers - and I think his numbers about the technology are optimistic indeed. That's bad news,

No, it's not bad news. It's just news. Expected value is about payoff, cost, and probability. 15% means nothing more and nothing less than 15%; without additional details, it does not mean that something is a good idea and it does not mean something is a bad idea either.

Replies from: TitaniumDragon, TitaniumDragon
comment by TitaniumDragon · 2013-04-27T02:20:02.354Z · LW(p) · GW(p)

I understood Dunning-Kruger quite well. Dunning-Kruger suggests that, barring outside influence, people will believe themselves to be of above-average ability. Incompetent people will greatly overestimate their capability and understanding, and the ability to judge talent in others was proportional to ability in the skill itself - in other words, people who are incompetent are not only incompetent, but also incapable of judging competence in other people.

Competent people, conversely, overestimate the competence of the incompetent; however, they do have the ability to judge incompetence, so when they are allowed to look at the work of others relative to their own, their estimation of their own personal ability more closely matches their true ranking - while incompetent people being exposed to the work of others had no such effect, though training in the skill improved their ability to self-judge, judge others, and at the skill itself.

People, therefore, are unfit to judge their own competence; the only reliable way to get feedback is via actual practice (i.e. if you have some sort of independent metric for your ability, such as success or failure of actual work) or if you have other competent people judge your competence. As you might imagine, this, of course, creates the problem where you have to ask yourself, "Who is actually competent in cryonics?" And the answer is "cryobiologists and people in related disciplines". And what is THEIR opinion of cyronics?

Quite poor, on the whole. While there are "cryonics specialists" there are no signs of actual competence there as there is no one who can actually revive frozen people, let alone revive frozen people and fix whatever problems they had prior to being frozen. Ergo, they can't really be viewed as useful judges on the whole because they have shown no signs of actual competence - there is no proof that anyone is competent at cryonics at all.

Dunning-Kruger definitely applies here, and applies in a major way. The closest things to experts are the people working in real scientific disciplines, such as cyrobiology and similar things. These people have real expertise, and they are not exactly best friends with Alcor and similar organizations. In fact, most of them say that it is, at best, well-intended stupidity and at worst a scam.

Name three. Like V_V, I suspect that for all that you glibly allude to 'cults' you have no personal experience and you have not acquainted yourself with even a surface summary of the literature, much like you have not bothered to so much as read a cryonics FAQ or book before thinking you have refuted it.

Similar religious movements? How many movements don't have some concept of life after death? It is very analogous.

I have indeed read papers on cyrobiology and on cryonics, though I could not name them off-hand - indeed, I couldn't tell you the name of the paper I read on the subject just yesterday, or the others I read earlier this week. I am, on the whole, not very impressed. There are definitely things we can freeze and thaw just fine - embryos and sperm, for instance. We can freeze lots of "lower organisms". We've played around with freezing fish and frogs and various creatures which have adapted to such things.

But freezing mammals? Even reducing mammalian body temperatures to the point where freezing begins is fatal, albeit not immediately; we have revived rats and monkeys and hamsters down to very low temperatures (below 0C) and revived them, but they don't tend to do very well afterwards, dying on the scale of hours to days. Some organs, such as the heart and kidney, have been frozen and revived - which is cool, to be fair. Well, frozen is the wrong term really - more "preserved at low temperatures". There was the rabbit kidney which they did vitrify, while the hearts I've seen have mostly been reduced to low temperatures without freezing them - though you can apparently freeze and thaw hearts and they'll work, at least for a while (we figured that out more than half a century ago).

However, a lot of cryobiology is not about things applicable to cryonics - we're talking taking tissue down to like, -2C, not immersing it in LN2. The vitrified rabbit kidney is interesting for that reason, but unfortunately the rabbit in question only lasted nine days - so while it could keep them up for a while, it did eventually fail. And all the other rabbits they experimented on perished as well.

And it takes even less time to notice that there are long thorough answers to the obvious objections. Your point here is true, but says far more about you than religion or cryonics; after all, many true things like heliocentrism or evolution have superficial easily thought-of objections which have been addressed in depth. Sometimes they work, sometimes they don't; the argument from evil is probably the single most obvious argument against Western religions, there are countless replies from theists of various levels of sophistication, and while I don't think any of them actually work, I also don't think someone going 'My mother died! God doesn't exist!' is contributing anything whatsoever. What, you think the theists somehow failed to notice that bad things happen? Of course they did notice, so if you want to argue against the existence of God, read up on their response.

The length of an answer has very little to do with its value. Look at Alcor's many answers - there are plenty of long answers there. Long on hope, that is, short on reality. In fact, being able to answer something succicently is often a sign of actual thought. It is very simple to pontificate and pretend you have a point, it is much more difficult to write a one paragraph answer that is complete. And in this case, the answer SHOULD be simple.

If it was so easy, again, why are you writing a long response?

If you had spent less time being arrogant, it might have occurred to you that I see this sort of flip reaction all the time in which people learn of cryonics and in five seconds think they've come up with the perfect objection and refuse to spend any time at all to disconfirm their objection. You are acting exactly like the person who said, "but it's not profitable to revive crypatients! QED you're all idiots and suckers", when literally the first paragraph of the Wikipedia article on ALCOR implies how they attempt to resolve this issue; here's a link to the discussion: http://lesswrong.com/lw/gh5/cryo_and_social_obligations/8d43

I enjoy how you are calling me arrogant, and yet you still are not answering my question.

At least other people have tried. "Dead people are valuable artifacts! People are excited about Jurassic park, that would totally be a viable business venture, so why not dead people?" Now that is a quick, succicient argument. It makes a reasonable appeal - dead people could be valuable as an attraction in the future.

The problem with that is the idea that it would make you any money at all. The Thirteenth Amendment prohibits owning people, and that kind of puts a major crimp in the idea of a tourist attraction, and given the sheer expense involved, again, you need some sort of parallel technology to get rid of those costs in any case. Humans are also a lot less exciting than dinosaurs are. I'm not going to go to the zoo to see someone from the 17th century, and indeed the idea is morally repugnant. Sure, I might go to ren faires, but let's face it - ren faires aren't run by people from the 10th century, they're run by people from the 21st century for people from the 21st century.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-27T12:33:08.190Z · LW(p) · GW(p)

You are comparing the current state of the art (freezing mammals and rabbits) with what may or may not be theoretically possible, potentially centuries down the line.

How long a rabbit survived upon being revived using current methods is besides the point - how long rabbits (and humans) can possibly survive when revived a long time into the future would be more relevant. Potentially no survival would be necessary at all, if the informational state was uploaded to a different hardware substrate. Not postulating magic, just not postulating anything which would contradict our current understanding of the laws of physics - and even that is more of a lower bound.

Concerning the technological feasibility, all we can say is that we can't say one way or the other how closely a reconstituted / scanned brain would resemble the original person. There is little indication that a high-fidelity reconstruction is in principle impossible. And a supposed impossibility cannot be established by looking at how long rabbits survive using current methods, molecular biology in its more theoretical variants would be more relevant.

So, the jury's still out for the "technological viability in the future" part. The "would any agent (group of agents) get to the point where it (they) could revive us, and if so, would it (they) want to, and if so, would we want to be revived that way" are different questions. Let's not muddle the issues.

Few cryonicists expect to be revived if Earth is rendered uninhabitable during World Wars III to X. Or if the facility in which they were stored went bankrupt, and the cadavers thrown out. Or if the facility were destroyed in some natural disaster (building on tectonic fault lines is a dumb long-term plan).

Also, few cryonicists would want to be revived by some uncaring alien civilization stumbling upon our remains, and reanimating us to test the pain endurance of 21st century human specimens. Maybe for whatever reasons resources would be scarce, and revival and retraining frozen Homo-heidelbergensis-equivalents may be prohibited (although it stands to reason that if the capabilities to do so in the first place are there, energy isn't an issue anymore. There's plenty around, after all).

There's a danger of being revived just to serve as some sort of living exhibit, or to be reconfigured into a database, either inert with no consciousness, or forced to relive selectively looped memories over and over (a sort of cryonics-based simulation argument). Most cryonicists would probably count such a successful revival as a failure.

Yet for all that, you mention a US-amendment as if it could be relevant at that future point in time? Historically, the dominating constant has been hard to predict change. There are many future scenarios in which you'd want to have been frozen, and many in which you wouldn't. It's not a large monetary investment. Why not? People spend more on experimental cancer treatments, or antibodies that give them a few additional weeks on average.

When I'm in my own galaxy, I'll think back on you, and maybe construct a best-guess facsimile based on your comments, invested into the body of Statler or Waldorf (which would you prefer?). See you then!

comment by TitaniumDragon · 2013-04-27T02:23:14.858Z · LW(p) · GW(p)

By the way, a benchmark I've found useful in discussing factual matters or matters with a long pre-existing literature is number of citations and hyperlinks per comment. You're still batting a zero.

So that means your comment is worthless, and thus can be safely ignored, given your only "citations" do not support yourself in any way and is merely meant to insult me?

In any case, citations are mostly unimportant. I use google and find various articles to support my stances; you can do the same to support yours, but I don't go REF Fahy et. al. "Physical and biological aspects of renal vitrification" Organogenesis. 2009 Jul-Sep; 5(3): 167–175.

Most of the time, you aren't going to bother checking my sources anyway, and moreover, you're asking for negative evidence, which is always a problem. You're asking for evidence that God does not exist, and rejecting everything but "Hey look, God is sitting here, but he's not".

You're acting like someone who was just told that they don't have a soul and therefore won't go to heaven when they die, because heaven doesn't exist.

You can take ten seconds to see a long list of objections by googling "Cryonics is a scam". You can go to Alcor and read a paper where a true believer suggests that the odds of revival are, at best, 15%, and that's assuming magical nanomachines have a 99% chance of existing. You can read the opinions of various experts who point out the problems with ice crystal formation, the toxicity of vitrification chemicals (which would have to be purged prior to revival), the issues of whether microdamage to structures would cause you to die anyway, the issues of whether you can actually revive them, and pointing out that, once you do warm them up, you've got a dead body, and all you have to do from there is ressurect the dead. We do know that even short times wtihout oxygen cause irreparable brain damage, and even at cold temperatures, that process does not stop completely - once they're in LN2, sure, maybe, assuming the process doesn't destroy them. Or you know, that the process of putting in the chemicals doesn't cause damage.

The truth is that none of the objections will sway you because you're a believer.

IF it is possible to do this sort of thing, there is a very, very good chance that it will require a very specific freezing process. A process which does not yet exist.

I'm impressed you've failed to notice that LW is maybe a little different from other sites and we have higher standards, and what happens 'historically' isn't terribly relevant.

The problem is that it isn't, and a cursory search of the internet will tell you that. :\

I was a bit excited to find a site devoted to rationality, and was rather disappointed to learn that no, it wasn't.

I wrote a little hymn about it a while ago. It starts with "Our AI, who art in the future", and you can imagine that it goes downhill from there.

In fact, a cursory search of the net showed at least one topic that you guys preemptively banned from discussion because some future AI might find it and then torture you for it. If that isn't a religious taboo, I don't know what is.

The singularity is not going to happen. Nanomachines the way that they are popularly imagined will never exist. Cyronics, today, is selling hope and smoke, and is a bad investment. You've got people discussing "friendly AI" and similar nonsense, without really understanding that they're really talking about magic, and that all this philosophizing about it is pretty silly.

I'm good with doing silly things, but people here take them seriously.

Just because you call yourself a rationalist doesn't make you a rationalist. Being rational is hard for most people to do. But perhaps the most important aspect of being a rationalist is understanding that just because you want something to be true, doesn't make it true. Understanding that deep in your bones.

Most people will be deeply insulted if you imply that they are irrational. And yet people on the whole do not behave rationally, given the goals they claim to possess.

I understand you are deeply emotionally invested in this. I understand that arguing with you is pointless. But I actually enjoy arguing, so its okay. But how is it for you? If you've invested in cryonics, is your brain more or less likely to believe that it is true?

Historical trends are always important, especially when you see obvious similarities. There are obvious and worrisome similarities between basic tenants (ressurrection of the dead, some sort of greatly advanced being who will watch over us (the post-singularity AI or AIs)) and the tenents of religions. You can't claim "we're different" without good evidence, and as they say, extraordinary claims require extraordinary evidence.

And all the evidence today points towards cyronics being a very expensive form of burial. There is no extraordinary evidence that it will allow for ressurection in the future. Thus, it is a waste of money and resources you could spend to make today more awesome.

Apparently you missed the point. The point was: stop being arrogant. Think for a freaking second about how obvious an argument might be and at least what the reply might be, if you cannot be arsed to look up actual sources. Do us the courtesy of not just thinking your way to your bottom line of 'cryonics sucks' but maybe a step beyond that too.

Maybe you should take your own advice?

I am quite aware that this is upsetting to you. I just told you you're going to die and be dead forever. It is an unsurprising reaction; a lot of people react with fear to that idea.

But really, there is no evidence cryonics is useful in any way. The argument is "Well, if you've rotted away, then you've got no chance at all!" Sure. But what if you could spend your money on present-day immortality research? The odds of that paying off are probably much higher than the odds of cryonics paying off. There is a path forward there. We don't know what causes aging, but we know that many organisms live longer than human beings do, and we may be able to take advantage of that. Technology such as artifical or grown organs may allow us to survive until brain failure. Drugs to treat brain disease may allow us to put off degredation of our brains indefinitely. The list goes on.

That is far more promising than "freeze me and hope for the best". Heck, if you really wanted to live forever you'd do things to work towards that. If cyronics is truly so important, why aren't you doing relevant research? Or working towards other things that can help with life extension?

Isn't that far more rational?

Cryonics is a sucker's bet. Even if there was a possibility it worked, the odds of it working are far less than other routes to immortality.

Instead, cryonics is just a way to sell people hope. Just as Christians make peace with the idea of death that they will be going to a better place, that they will be okay, Christians avoid death as much as anyone else does. The same is true of cryonics. The rational thing to do, if it is important to avoid dying, is to work towards avoiding it or mitigating it as much as possible. Are you? If the answer is no, is it really so important to you? Or is paying that money for cryonics just a personal way to make peace with death?

Replies from: Desrtopa, JoshuaZ, wedrifid
comment by Desrtopa · 2013-04-28T01:30:34.214Z · LW(p) · GW(p)

You can take ten seconds to see a long list of objections by googling "Cryonics is a scam". You can go to Alcor and read a paper where a true believer suggests that the odds of revival are, at best, 15%, and that's assuming magical nanomachines have a 99% chance of existing.

I'll note here that the average estimated chance among regulars here for cryonics working is actually lower than that, and the difference in how seriously people on Less Wrong tend to take cryonics compared to the general population is less to do with thinking it's much more likely than most people, and more to do with thinking that a chance of revival on that order is worth taking seriously.

comment by JoshuaZ · 2013-04-29T13:34:45.237Z · LW(p) · GW(p)

I was a bit excited to find a site devoted to rationality, and was rather disappointed to learn that no, it wasn't.

I wrote a little hymn about it a while ago. It starts with "Our AI, who art in the future", and you can imagine that it goes downhill from there.

I'm sorry, what is the intended content here? Because you can write a hymn that parodies strong AI claims that therefore we need to take them less seriously?

In fact, a cursory search of the net showed at least one topic that you guys preemptively banned from discussion because some future AI might find it and then torture you for it. If that isn't a religious taboo, I don't know what is.

Many people are not in favor of discussing the basilisk not because of the issue with a potential AI, but because of the danger that mentally vulnerable people will be disturbed by the notion. But in any event, you are pattern matching in an unhelpful way. The fact that something resembles something done by religions doesn't make it intrinsically wrong. Note for example, that large amounts of computer programming and maintenance look heavily ritualistic if you don't know what it is.

The singularity is not going to happen. Nanomachines the way that they are popularly imagined will never exist. Cyronics, today, is selling hope and smoke, and is a bad investment. You've got people discussing "friendly AI" and similar nonsense, without really understanding that they're really talking about magic, and that all this philosophizing about it is pretty silly.

So these are all conclusions, not arguments. And speaking as someone who agrees with you on a lot of this stuff, you are being both highly irrational and unnecessarily insulting in how you lay out these claims.

Cryonics is a sucker's bet. Even if there was a possibility it worked, the odds of it working are far less than other routes to immortality.

What other routes are you comparing it to? You mention a few methods of life-extension, but none are methods likely to add by themselves more than a few centuries at most.

Instead, cryonics is just a way to sell people hope. Just as Christians make peace with the idea of death that they will be going to a better place, that they will be okay, Christians avoid death as much as anyone else does. The same is true of cryonics. The rational thing to do, if it is important to avoid dying, is to work towards avoiding it or mitigating it as much as possible. Are you? If the answer is no, is it really so important to you? Or is paying that money for cryonics just a personal way to make peace with death?

Don't confuse not having a certain goal set with disagreeing with you about what will most likely accomplish that goal set.

comment by wedrifid · 2013-04-27T08:45:10.816Z · LW(p) · GW(p)

You can read the opinions of various experts who point out the problems with ice crystal formation, the toxicity of vitrification chemicals (which would have to be purged prior to revival), the issues of whether microdamage to structures would cause you to die anyway, the issues of whether you can actually revive them, and pointing out that, once you do warm them up, you've got a dead body, and all you have to do from there is ressurect the dead.

You have just declared yourself ignorant of what cryonics is intended to do and screened off whatever value your opinion may otherwise have had.

I was a bit excited to find a site devoted to rationality, and was rather disappointed to learn that no, it wasn't.

I invite you to leave and find another place where the style of thought is more in accord with that of your own contributions. (I strongly oppose any attempts to make lesswrong more like that.)

comment by Paul Crowley (ciphergoth) · 2013-04-27T09:13:58.349Z · LW(p) · GW(p)

gwern's interpretation of what I wrote here is entirely correct.

comment by pianoforte611 · 2013-04-28T17:20:38.712Z · LW(p) · GW(p)

"what is the likelihood that we care about dead frozen people in the future?"

I wondered that as well when I first heard about cryonics. It is true that society in general won't care about frozen people in the future. But that isn't necessary for cryonics to work. Rather its enough that cryonics organizations care about frozen people.

Why would they care? Because the people running the organization have a vested interested in making their clients live. Among other reasons, the people running the organization might one day be clients as well, so they have to care about the success of the project.

comment by Paul Crowley (ciphergoth) · 2013-04-21T12:53:03.410Z · LW(p) · GW(p)

Actually I think the meat of what I want to say is here: http://blog.ciphergoth.org/blog/2010/02/15/blog-comments-and-article/

comment by Tenoke · 2013-04-18T16:13:01.506Z · LW(p) · GW(p)

90% confidence intervals the way that I understand the cases(B and C require you to involve your interpretation because they are too interdependent under the default interpretation):

.02< P(B|A) < .2

.6<P(C|A,B)< .8

comment by [deleted] · 2013-04-18T12:25:27.809Z · LW(p) · GW(p)

Hanlon has claimed 6% before, based on a Fermi estimate that may or may not mean anything.

The difference between this and Pascal's Wager is that there's no tradeoff between small probability scenarios; there's either assured death or almost-assured-death. The argument is roughly (small probability)*(high utility) > (lost utility of cheap investment).

(But of course see gwern's remarks on the cheapness of the investment.)

Replies from: drethelin
comment by drethelin · 2013-04-18T16:17:52.771Z · LW(p) · GW(p)

You can rephrase it as a small probability of revival vs a small probability of REALLY needing that money.

comment by Luke_A_Somers · 2013-04-18T21:31:07.005Z · LW(p) · GW(p)

I think it was more like (moderate probability of success)*(monumental heap of utility if it works) = (a great investment) so this argument clearly doesn't apply.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:07:43.271Z · LW(p) · GW(p)

Okay, seriously, how the hell did you get this impression?

Replies from: None, MarkusRamikin
comment by [deleted] · 2013-04-18T19:20:02.589Z · LW(p) · GW(p)

Not sure why you're getting downvoted, but I was under the impression that you were on the far end of the LW bell curves for cryonics-optimism and AI-pessimism.

Replies from: atorm
comment by atorm · 2013-04-18T23:48:30.239Z · LW(p) · GW(p)

I suspect he's getting downvoted because he didn't answer the question, not even with "I don't think it has a low probability of success" or some other simple response.

comment by MarkusRamikin · 2013-04-18T19:18:27.961Z · LW(p) · GW(p)

If that's not your argument, what is?

Yeah, did you ever write up a full summary of why you think signing up for cryonics is a good idea? Including, hopefully, not just the information theory stuff,, but also how likely you think it is that you will remain funded and get unfrozen even if the technical problems are all solved, etc. Can't find such an article under the cryonics tag, and I'd love to read such a thing from you.

Replies from: None
comment by [deleted] · 2013-04-18T22:47:33.888Z · LW(p) · GW(p)

No numbers, but I think this is the most comprehensive argument Eliezer's made on the subject (it's linked from the LW wiki page on cryonics): http://lesswrong.com/lw/wq/you_only_live_twice/

comment by A1987dM (army1987) · 2013-04-18T18:37:29.524Z · LW(p) · GW(p)

But Fermi is also the one who said that nuclear energy was fifty years off in the unlikely event it could be done at all, two years (IIRC) before Fermi himself oversaw the construction of the first nuclear pile.

For something in the same(-ish) reference class where the pessimists turned out to be right, commercially viable power generation from nuclear fusion has been “30 years in the future” ever since the mid-20th century.

Replies from: ciphergoth
comment by Paul Crowley (ciphergoth) · 2013-04-18T20:21:04.707Z · LW(p) · GW(p)

Everyone tells this story; I'd like to see a cite. Fusion advocates tell a different story: that fusion was always some large number of dollars away, but the dollars weren't there until relatively recently. Once the dollars arrived, a roadmap was set out and has AFAICT basically hit all its deadlines, with JET, ITER and next DEMO proceeding as planned.

Replies from: CarlShulman
comment by CarlShulman · 2013-04-18T21:16:57.462Z · LW(p) · GW(p)

Fusion advocates tell a different story: that fusion was always some large number of dollars away, but the dollars weren't there until relatively recently.

Could you link to them?

Replies from: ciphergoth, Dr_Manhattan
comment by Paul Crowley (ciphergoth) · 2013-04-19T12:04:57.583Z · LW(p) · GW(p)

I didn't keep links when I read these things, so this is the result of a quick Google search for 'fusion "years away" "dollars away"':

The actual reason is mainly funding. People always use the "twenty/thirty/fifty years away" comment as an insult, a way of showing how fusion (or science in general) is unreliable. The reality is that when those predictions were first made in the 1970s in the wake of the Oil Crisis. What happened during the Oil Crisis? We freaked out (rightly so) and planned to allocate a huge amount of money towards fusion research. What happened after the Oil Crisis ended? That money disappeared. Essentially, scientists were promised X billions of dollars to make fusion work, and said they could do it in a couple decades. Then that money was taken away, and people expected them to stay on schedule. Of course, fusion power turned out to be a lot more complicated than we expected. But the real reason is we simply aren't paying for it. Its not "30 years away" its more like $80 billion dollars away. http://imgur.com/sjH5r

http://www.reddit.com/r/science/comments/1bqxaq/scientists_develop_fusion_rocket_technology_in_lab/c99nvl8

Replies from: Cyan, JoshuaZ
comment by Cyan · 2013-04-20T02:58:29.926Z · LW(p) · GW(p)

I learned something. Excellent.

comment by JoshuaZ · 2013-04-29T13:18:53.943Z · LW(p) · GW(p)

Note that these predictions didn't start in the 1970s. In the 1950s and 1960s similar predictions were made with "20 years away" which is an even shorter timespan. See for example here.

Replies from: gwern
comment by gwern · 2013-04-29T18:41:15.157Z · LW(p) · GW(p)

Your link is to Life, as mainstream a publication as you could find in 1950s. We are all here first-hand familiar with how reporters simplify, misunderstand, and misreport technical matters.

As for your specific quote: I assume you're referring to pg180, the Bhabha quote? The reporter says specifically "a controlled thermonuclear reactor" was <20 years away. He didn't say economical power, power too cheap to meter, break-even or net power, or anything. Was this version of what Bhabha said actually wrong? By 1976, was there nowhere in the world a research tokamak or something which created thermonuclear reactions under controlled non-bomb conditions? I suspect there was.

Replies from: JoshuaZ
comment by JoshuaZ · 2013-04-29T19:09:00.620Z · LW(p) · GW(p)

Those are very good points. The first point isn't that convincing by itself since there are other similar example statements from the 1950s and 60s (although I don't have them available off-hand), and while we do frequenly criticize reporters for misreporting on science matters, most of their statements are not very far off from what is being described. Misreporting is while egregious, a small fraction of most science reporting.

Your second point seems more persuasive. By 1976, not only were there functioning tokamaks, but there were other fusion devices also such as fusors. So the prediction of controlled thermonuclear reactors in 20 years did come true, not just for tokamahs but for other fusion methods as well. This substantially reduces the validity of my point.

Replies from: gwern
comment by gwern · 2013-04-29T19:50:02.541Z · LW(p) · GW(p)

and while we do frequenly criticize reporters for misreporting on science matters, mos of their statements are not very far off from what is being described

I dunno, sometimes they are completely wrong. A few days ago I got the writer of http://arstechnica.com/tech-policy/2012/06/fbi-halted-one-child-porn-inquiry-because-tor-got-in-the-way/ to massively edit the middle of the article because the original source document explicitly said the child porn was not on Silk Road... and his article said the child porn was on Silk Road. Which is about as wrong as possible. And this is far from the first example of the media getting technological or scientific things completely wrong, which is why you need to read the comments or read the original papers if you're going to base any beliefs on what you're seeing.

It's not hard to make the reported versions of stories or predictions be completely wrong, especially in the context of fusion where we were originally discussing the claims of fusion reporters that the credible published official estimate from the government report of 20-30 years were indeed real but had been made explicitly on the basis of enormous funding increases which never materialized, funding was cut substantially, and actual progress has been better than predicted by the low-funding scenarios. (I put a request in the research help page for a copy of the original report to see if the presented graph is accurate but it hasn't come yet.) It's very easy to slide from the apparently accurate version of the conditional prediction "We predict economical fusion in 30 years if we get the planned funding of $80 billion" to the version "they predict fusion in 30 years".

comment by Dr_Manhattan · 2013-04-21T17:48:54.147Z · LW(p) · GW(p)

SImilar story is told about fission weaponry.

1939, Nobel Prize-winning physicist Niels Bohr had argued that building an atomic bomb "can never be done unless you turn the United States into one huge factory." Years later, he told his colleague Edward Teller, "I told you it couldn't be done without turning the whole country into a factory. You have done just that."

http://energy.gov/em/about-us/em-history

comment by wedrifid · 2013-04-18T08:26:16.305Z · LW(p) · GW(p)

It is predictable that in retrospect I will think this was a waste of time compared to working on a trust criterion between a probability distribution and an improved probability distribution. (Apologies, this is a technical thingy I'm currently working on which has no good English description.)

Cool. Are you or one of your minions likely to write it up in an informal technical way at some point in the not-excessively-distant future?

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:14:51.043Z · LW(p) · GW(p)

It's looking more likely to be formal actually.

Replies from: wedrifid
comment by wedrifid · 2013-04-18T17:39:01.497Z · LW(p) · GW(p)

It's looking more likely to be formal actually.

Great, even better.

comment by CarlShulman · 2013-04-18T08:19:02.610Z · LW(p) · GW(p)

In the course of any workday, on the now very rare occasions I find myself thinking about such meta-level junk instead of the math at hand, I remind myself that it is a wasted motion - where a 'wasted motion' is any thought which will, in retrospect if the problem is in fact solved, have not have contributed to having solved the problem.

If you rule out doing anything except X, then you won't get much out of accurately evaluating the plausibility of X. The point of considering likelihood of success is that there are always other options, including cutting one's losses. But to rule out all competing options requires some assessment of their plausibility relative to X.

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:16:53.364Z · LW(p) · GW(p)

A lot of meta-level fretting has the property of being one-sided - it's about a single option considered in isolation, not about two alternatives. If there's a concrete alternative that's supposed to help humanity more and has a decent chance of being actually correct vs. the sort of thing one dutifully ought to consider, I am usually totally happy to consider it. (You've seen me ask 'Can we have a concrete policy implication, please?' or 'Is there an option on the table for what we should be doing instead, if that's true?' at a number of discussions, right? This is often what my 'wasted motion' heuristic looks like when it fires.)

comment by lukeprog · 2013-04-18T19:01:04.703Z · LW(p) · GW(p)

Quoted in 'The Making of the Atomic Bomb' by Richard Rhodes

Specifically, on page 280 of the 25th Anniversary Edition of the book.

comment by komponisto · 2013-04-18T06:19:47.623Z · LW(p) · GW(p)

Since I just posted to announce a meetup featuring Michael Vassar, I suppose I was primed to recall his take on the Fermi episode:

...1 in 10 is not such a bad estimate. The problem was not that Fermi was stupid or that he was bad at making estimates; he was probably much better at making estimates than almost everyone. The problem is that he was adhering to a set of rules for what you should be thinking about or talking about that is flat-out insane, frankly. A set of rules that says you shouldn't think about anything until you're ready to do experiments with more-or-less established experimental techniques.

From this perspective -- which assumes that Fermi arrived at his estimate through an honest, non-motivated calculation -- what Fermi should have done was believe his own estimate, instead of applying the heuristic of "if it's not established. experimentally-tested science, it doesn't exist". Because a 10% probability of the scenario in question is indeed approximately 100%: that is, enough to take seriously.

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T06:23:56.149Z · LW(p) · GW(p)

Why assign a 90% probability to chain reactions being impossible or unfeasible? How should Fermi have known that, especially when it was false?

EDIT: Be careful with your arguments that Fermi should have assigned the false fact 'chain reactions are impossible' an even more extreme probability than 90%. You are training your brain to assign higher and more extreme probabilities to things that are false. You should be looking for potential heuristics that should have fired in the opposite direction. There's such a thing as overfitting, but there's also such a thing as being cleverly contrarian about reasons why nobody could possibly have figured out X and thus training your brain in the opposite direction of each example.

Replies from: orthonormal, private_messaging, komponisto
comment by orthonormal · 2013-04-18T14:18:21.826Z · LW(p) · GW(p)

Because ordinary matter is stable, and the Earth (and, for more anthropically stable evidence, the other planets) hadn't gone up in a nuclear chain reaction already?

Without using hindsight, one might presume that a universe in which nuclear chain reactions were possible would be one in which it happened to ordinary matter under normal conditions, or else only to totally unstable elements, not one in which it barely worked in highly concentrated forms of particular not-very-radioactive isotopes. This also explains his presumption that even if it worked, it would be highly impractical: given the orders of magnitude of uncertainty, it seemed like "chain reactions don't naturally occur but they're possible to engineer on practical scales" is represented by a narrow band of the possible parameters.

I admit that I don't know what evidence Fermi did and didn't have at the time, but I'd be surprised if Szilard's conclusions were as straightforward an implication of current knowledge as nanotech seems to be of today's current knowledge.

Replies from: roystgnr, Eliezer_Yudkowsky
comment by roystgnr · 2013-04-18T16:43:37.606Z · LW(p) · GW(p)

Strictly speaking, chain reactions do naturally occur, they're just so rare that we never found one until decades after we knew exactly what we were looking for, so Fermi certainly didn't have that evidence available.

Also, although I like your argument... wouldn't it apply as well to fire as it does to fission? In fact we do have a world filled with material that doesn't burn, material that oxidizes so rapidly that we never see the unoxidized chemical in nature, and material that burns only when concentrated enough to make an ignition self-sustaining. If forests and grasslands were as rare as uranium, would we have been justified in asserting that wildfires are likely impossible?

One reason why neither your argument nor my analogy turned out to be correct: even if one material is out of a narrow band of possible parameters, there are many other materials that could be in it. If our atmosphere was low-oxygen enough to make wood noncombustable, we might see more plants safely accumulating more volatile tissues instead. If other laws of physics made uranium too stable to use in technology, perhaps in that universe fermium would no longer be too unstable to survive in nature.

Replies from: Eliezer_Yudkowsky, orthonormal
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:37:21.741Z · LW(p) · GW(p)

Consider also the nature of the first heap: Purified uranium and a graphite moderator in such large quantities that the neutron multiplication factor was driven just over one. Elements which were less stable than uranium decayed earlier in Earth's history; elements more stable than this would not be suitable for fission. But the heap produced plutonium by its internal reactions, which could be purified chemically and then fizzed. All this was a difficult condition to obtain, but predictable that human intelligence would seek out such points in possibility-space selectively and create them - that humans would create exotic intermediate conditions not existing in nature, by which the remaining sorts of materials would fizz for the first time, and that such conditions indeed might be expected to exist, because among some of the materials not eliminated by 5 billion years, there would be some unstable enough to decay in 50 billion years, and these would be just-barely-non-fizzing and could be pushed along a little further by human intervention, with a wide space of possibilities for which elements you could try. Or to then simplify this conclusion: "Of course it wouldn't exist in nature! Those bombs went off a long time ago, we'll have to build a slightly different sort! We're not restricted to bombs that grow on trees." By such reasoning, if you had attended to it, you might have correctly agreed with Szilard, and been correctly skeptical of Fermi's hypothetical counterargument.

Not taking into account that engineering intelligence will be applied to overcome the first hypothetical difficulty is, indeed, a source of systematic directional pessimistic bias in long-term technological forecasts. Though in this case it was only a decade. I think if Fermi had said that things were 30 years off and Szilard had said 10, I would've been a tad more sympathetic toward Fermi because of the obvious larger reference class - though I would still be trying not to update my brain in the opposite direction from the training example.

Replies from: private_messaging
comment by private_messaging · 2013-04-19T21:43:39.283Z · LW(p) · GW(p)

because among some of the materials not eliminated by 5 billion years, there would be some unstable enough to decay in 50 billion years, and these would be just-barely-non-fizzing and could be pushed along a little further by human intervention

Except there aren't any that are not eliminated by, say, 10 billion years. And even 40 million years eliminate everything you can make a nuke out of except U235 . This is because besides fizzling, unstable nuclei undergo this highly asymmetric spontaneous fission known as alpha decay.

comment by orthonormal · 2013-04-18T21:17:51.582Z · LW(p) · GW(p)

Good counter-analogy, and awesome Wikipedia article. Thanks!

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:12:54.197Z · LW(p) · GW(p)

A clever argument! Why didn't it work on Reality?

Replies from: AlanCrowe, orthonormal
comment by AlanCrowe · 2013-04-18T22:23:07.878Z · LW(p) · GW(p)

I spot two holes.

First the elephant in the living room: The sun.

Matter usually ends up as a fusion powered, flaming hell. (If you look really closely it is not all like that; there are scattered little lumps in orbit, such as the Earth and Mars)

Second, a world view with a free parameter, adjusted to explain away vulcanism.

Before the discovery of radio-activity, the source of the Earth's internal heat was a puzzle. Kelvin had calculated that the heat from Earth's gravitational collapse, from dispersed matter to planet, was no where near enough to keep the Earth's internal fires going for the timescales which geologists were arguing for.

Enter radioactivity. But nobody actually knows the internal composition the Earth. The amount of radioactive material is a free parameter. You know how much heat you need and you infer the amount of Thorium and Uranium that "must" be there. If there is extra heat due to chain reactions you just revise the estimate downwards to suit.

Sticking to the theme of being less wrong, how does one see the elephant in the room? How does one avoid missing the existence of spontaneous nuclear fusion on a sunny day? Pass.

The vulcanism point is more promising. The structure of the error is to say that vulcanism does not count against the premise "ordinary matter is stable" because we've got vulcanism fully explained. We've worked out how much Uranium and Thorium there needs to be to explain it and we've bored holes 1000km deep and checked and found the correct amount. But wait! We haven't done the bore-hole thing, and it is hard to remember this because it is so hopelessly impractical that we are not looking forward to doing it. In this case we assume that we have dotted the i's and crossed the t's on the existing theory when we haven't.

One technique for avoiding "clever arguments" is to keep track of which things have been cross-checked and which things have only a single chain of inference and could probably be adjusted to fit with a new phenomenon. For example, there was a long time in astronomy when estimates of the distances to galaxies used Cepheid variables as a standard candle, and that was the only way of putting an absolute number on the distance. So there was room for a radical new theory that changed the size of the universe a lot, provided it mucked about with nuclear physics, putting the period/luminosity relationship into doubt (hmm, maybe not, I think it is an empirical relationship based on using parallax to get measured values from galactic Cepheid variables). Anyway along come type Ia supernovas as a second standard candle, and inter galactic distances are calculated two ways and are on a much firmer footing.

So there are things you know that you only know via one route and there is an implicit assumption that there is nothing extra that you don't know about. Things that you only know via a single route can be useless for ruling out surprising new things .

And there are things you know that you know via two routes that pretty much agree. (if they disagree then you already know that there is something you don't know). Things you know via two routes do have some power of ruling out surprising new things. The new thing has to sneak in between the error bars on the existing agreement or somehow produce a coordinated change to preserve the agreement or correctly fill the gap opened up by changing one thing and not the other.

Replies from: NancyLebovitz, AspiringRationalist
comment by NancyLebovitz · 2013-04-20T01:41:36.263Z · LW(p) · GW(p)

I thought they did know that if the sun was solely dependent on chemical reactions, then it would have burned itself out more quickly than the age of the earth suggested.

Replies from: AlanCrowe
comment by AlanCrowe · 2013-04-20T12:55:16.019Z · LW(p) · GW(p)

I was glibly assuming that Fermi would know that the sun was nuclear powered. So he would already have one example of a large scale nuclear reaction to hand. Hans Bethe won his Nobel prize for discovering this. Checking dates, This obituary dates the discovery to 1938. So the timing is a little tight.

As you say, they knew that the sun wasn't powered by chemical fires, they wouldn't burn of long enough, but perhaps I'm expecting Fermi to have assimilated new physics quicker than is humanly possible.

comment by NoSignalNoNoise (AspiringRationalist) · 2013-04-19T20:08:08.355Z · LW(p) · GW(p)

Major nitpick: stars are examples of sustained nuclear fusion, not fission. The two are sustained by completely different mechanisms, so observation of nuclear fusion in stars doesn't really tell us anything about the possibility of sustained nuclear fission.

Minor nitpick: it's spelled volcanism, not vulcanism.

Replies from: AlanCrowe, gwern
comment by AlanCrowe · 2013-04-20T12:36:35.796Z · LW(p) · GW(p)

I'm looking at the outside view argument: matter is stable so we don't expect to get anything nuclear.

But we look at the sun and see a power source with light atoms fusing to make medium weight ones. We already know about the radioactive decay of heavy atoms, and the interesting new twist is the fission of heavy atoms resulting in medium weight atoms and lots of energy. We know that it is medium weight atoms that are most stable, there is surplus energy to be had both from light atoms and heavy atoms. Can we actually do it with heavy atoms? It works elsewhere with light atoms, but that's different. We basically know that it is up for grabs and it is time to go to the laboratory and find out.

I fear that I have outed myself with my tragic spelling error. People will be able to guess that I'm a fan of Mr Spock from the planet Vulcan ;-(

comment by gwern · 2013-04-19T20:55:30.460Z · LW(p) · GW(p)

Major nitpick: stars are examples of sustained nuclear fusioin, not fission.

Quoted for irony.

Replies from: AspiringRationalist
comment by NoSignalNoNoise (AspiringRationalist) · 2013-04-20T00:03:09.403Z · LW(p) · GW(p)

I'm not sure if pointing out my typo was your intent there, but you caused me to notice it, so I fixed it.

comment by orthonormal · 2013-04-18T20:58:03.260Z · LW(p) · GW(p)

At least nine times out of ten in the history of physics, that heuristic probably did work. I agree that Fermi was wrong not to track down a perceived moderately small chance of a consequential breakthrough, but I can't believe with any confidence that his initial estimate was too low without the power of hindsight.

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T22:14:26.569Z · LW(p) · GW(p)

Is there a good example of a conspiracy including physicists of the same prior fame as Rabi and Fermi (Szilard was then mostly an unknown) which was pursuing a 'remote possibility', of similar impact to nuclear weapons, that didn't pan out? Obviously we would have a much lower chance of hearing about it especially on a cursory reading of history books, but the chance is not zero, there are allegedly many such occasions, and the absence of any such known cases is not insignificant evidence. Bolded to help broadcast the question to random readers, in case somebody who knows of an example runs across this comment a year later. The only thing I can think of offhand in possibly arguably the same reference class would be polywell fusion today, assuming it doesn't pan out. There's no known conspiracy there, but there's a high-impact argument and Bussard previously working on the polywell.

Replies from: CarlShulman, private_messaging, itaibn0
comment by CarlShulman · 2013-04-24T02:10:48.872Z · LW(p) · GW(p)

Is there a good example of a conspiracy including physicists of the same prior fame as Rabi and Fermi (Szilard was then mostly an unknown) which was pursuing a 'remote possibility', of similar impact to nuclear weapons, that didn't pan out?

Do you have a set of examples where it did pan out, or are we just talking about a description crafted to describe a particular event?

Restricting to physicists cuts us from talking about other areas like bioweapons research, where indeed most of the "remote possibilities" of apocalyptic destruction don't pan out. Computer scientists did not produce AI in the 20th century, and it was thought of as at least a remote possibility.

For physicists, effective nuclear missile defense using beam weapons and interceptors did not pan out.

Replies from: private_messaging
comment by private_messaging · 2013-04-25T17:48:24.482Z · LW(p) · GW(p)

Radioactivity was discovered via "fluorescence is responsible for x-rays" idea that did not pan out...

There's a big number of fusion related attempts that did not pan out at all, there's fission of lithium which can't be used for a chain reaction and is only used for making tritium. There's hafnium triggering which might or might not pan out (and all the other isomers), and so on.

For the most part chasing or not chasing "wouldn't it be neat if" scenarios doesn't have much of effect on science, it seems - Fermi would still inevitably have discovered secondary neutrons even if he wasn't pursuing chain reaction (provided someone else didn't do that before him).

comment by private_messaging · 2013-04-20T04:24:37.989Z · LW(p) · GW(p)

They were not hell bent on obtaining grant money for a fission bomb no-matter-what. The first thing they had to do was to measure fission cross sections over the neutron spectra, and in the counter-factual world where U235 does not exist but they detected fission anyway (because high energy neutrons do fission U238), they did the founding effort for the accelerator driven fission, the fission products of which heal the cancer around the world (the radiation sources in medicine would still be produced somehow), and in that world maybe you go on using it in some other sequence going on how Szilard was wrong and Fermi dramatically overestimated and how obviously the chance was far lower because they were talking of one isotope and not a single isotope works and how stupid it is to think that fissioning and producing neutrons is enough for chain reaction (the bar on that is tad higher) etc etc. In that alternate world, today, maybe there's even an enormous project of trying to produce - in an accelerator or something more clever - enough plutonium to kick-start breeder reactor economy. Or maybe we got fusion power plants there, because a lot of effort was put into that (plus Manhattan project never happened and some scientists perhaps didn't get cancer) . edit: Or actually, combination of the two could have happened at some point much later than 1945: sub-unity tokamak which produces neutrons via fusion, to irradiate uranium-238 and breed enough plutonium to kick start breeder reactors. Or maybe not, because it could have took a long while there until someone measures properties of plutonium. Either way, Fermi and Szilard end up looking awesome.

comment by itaibn0 · 2013-04-21T12:54:39.900Z · LW(p) · GW(p)

How about the original Pascal's wager? It was made by a famed mathematician rather than a famed physicist, and it wasn't a conspiracy, but it's definitely in the same reference frame.

comment by private_messaging · 2013-04-18T16:34:32.505Z · LW(p) · GW(p)

Because they didn't know if fission produced enough prompt neutrons, which is clear from the quoted passage, and probably also because Fermi has estimated that there's on the order of 10 other propositions about the results from fission which he, if presented with them by an equally enthusiastic proponent, would find comparably plausible. I'm thinking that in the alternate realities where fission does something other than producing sufficient number of neutrons (about 3 on the average), you'd assign likewise high number to them by hindsight, with a sum greater than 1 (so stop calling it probability already).

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:13:22.172Z · LW(p) · GW(p)

A clever argument! Why didn't it work on Reality?

Replies from: private_messaging
comment by private_messaging · 2013-04-18T17:51:08.977Z · LW(p) · GW(p)

A clever argument!

I'm correcting a potential factual error:

They had not been demonstrated experimentally, to be sure; but they were still the default projection from what was already known.

What I am guessing happened (you're welcome to research the topic), first you can learn that uranium can be fissioned by neutrons (which you make, if I recall correctly, by irradiating lithium with alpha particles). Then, you may learn that fission produces neutrons, because, it so happens that you don't just see all of that in microscope, you see particle tracks in photographic emulsion or a cloud chamber or the like, and neutrons, being neutral, are hard to detect. (edit: And this is how I read the quote, anyway, on the first reading. I just parse it as low probability of neutrons, high probability of chain reaction if there's enough neutrons.)

So at first you do not know if fission produces neutrons without very precise and difficult analysis of the conservation of momentum or a big enough experiment to actually be able to count them, or something likewise clever and subtle. To think about it, chronologically, you may happen to first acquire weak evidence that fission does not produce prompt neutrons, by detecting beta decay from the fission products, which implies that they still have too many neutrons for their atomic number. And perhaps by detecting recoil from the delayed neutrons (which are too few for chain reaction, and are too delayed for a bomb).

Why didn't it work on Reality?

Or did it? It's bit like arguing how dumb it was to predict 1/6 probability for the die rolling 1, when it in fact rolled 1. Given 6 other sides and lack of information to prefer one over the other, the probability is 1/6 (edit: or less, of course) . The relevant reality here is the available knowledge and the mechanism that assigns plausibilities, and the first step of "working" is probabilities (somehow related to plausibilities) summing to 1. You ought to be able to test a mind upload's priors - just run them in parallel a very large number of times, having them opinion about probabilities of various topics, and see to what mutually exclusive scenarios sum, or if the sum even converges. Ghmm.

Replies from: private_messaging
comment by private_messaging · 2013-04-18T19:58:15.037Z · LW(p) · GW(p)

Ohh, the elephant in the room that I somehow neglected to mention. (It is hard to argue against silly ideas, I suspect for a reason similar to why it is very hard / impossible to truly reflect on how you visually tell apart cats and dogs)

There's a lot of nuclei that can fission, but can't sustain a chain reaction! Because they do not produce high enough energy neutrons, or they capture neutrons too often and fission too rarely, and so on. And the neutron source of the time (radium plus lithium, or radium plus beryllium, or something like that), it produced a lot of high energy neutrons.

It would be quite interesting if someone far more obsessive compulsive than me would go over the table of isotopes and see if about 1 in 10 isotopes that can fission when irradiated with radium-lithium or radium-beryllium neutron source produce enough neutrons of high enough energy. Because if it is close to 1 in 10, and I think it is (on appropriate, i.e. logarithmic, scale), then the evidence that one isotope can fission, will only get you to 1 in 10 chance that it makes neutrons that can fission it.

Replies from: endoself
comment by endoself · 2013-04-18T22:40:55.560Z · LW(p) · GW(p)

Szilard was proposing the idea of fission chain reactions in general. Of course he would be less confident if asked about a specific isotope, but he's still right that the idea is important if he gets the isotope wrong. Anyway, the fact that he discusses uranium specifically shows that the evidence available to him points toward uranium and that this sort of reference class is not using all the evidence that they had at the time.

Replies from: private_messaging
comment by private_messaging · 2013-04-18T23:31:26.538Z · LW(p) · GW(p)

and that this sort of reference class is not using all the evidence that they had at the time.

You're making it sound like you have a half of the periodic table on the table. You don't. There's U-238, U-235, Th-232, and that's it . Forget plutonium, you won't be making any significant amount of that in 1945 without a nuclear reactor. Of them the evidence for fission would be coming, actually, from U238 fissioning by fast neutrons, and U238 can't sustain chain reaction because too many of the the neutrons slow down before they fission anything, and slow neutrons get captured rather than cause fission.

U235 is the only naturally abundant fissile isotope, and it has a half life of 700 million years, which is 4400 times longer than the half life of the second most stable fissile isotope (U-233) and 30 000 longer than that of the third most stable isotope (that's it. The factor of 4400 difference, then the factor of less than 7 , and so on). That's how much U235 is a fluke. One can legitimately wonder if our universe is fine tuned for U235 to be so stable.

edit: note, confusing terminology here: "fissile" means capable of supporting a chain reaction, not merely those capable of fissioning when whacked with a high energy neutron.

edit2: and note that the nucleus must be able to capture a slow neutron and then fission due to capturing it, not due to being whammed by it's kinetic energy, contrary to what you might have been imagining, because neutrons lose kinetic energy rather quickly, before sufficient chance at causing a fission. It must be very unstable, yet, it must be very stable.

comment by komponisto · 2013-04-18T06:33:52.896Z · LW(p) · GW(p)

If you view the 90% number as an upper bound, with a few bits' worth of error bars, it doesn't look like such a strong claim. If Szilard and Fermi both agreed that the probability of the bad scenario was 10% or more, then it may well have been dumb luck that Szilard's estimate was higher. Most of the epistemic work would have been in promoting the hypothesis to the 10% "attention level" in the first place.

(Of course, maybe Fermi didn't actually do that work himself, in which case it might be argued that this doesn't really apply; but even if he was anchoring on the fact that others brought it to his attention, that was still the right move.)

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:14:09.440Z · LW(p) · GW(p)

I suppose if we postulate that Szilard and Rabi did better by correlated dumb luck, then we can avoid learning anything from this example, yes.

comment by shminux · 2013-04-18T16:14:47.175Z · LW(p) · GW(p)

Can someone point to MIRI's estimates (with justifications) of various x-risks and the odds of mitigating them? Just wondering how, in MIRI's view, the FAI work stacks up against other disaster prevention efforts. I can't seem to find this information on their site.

Replies from: Jack
comment by Jack · 2013-04-18T18:36:37.163Z · LW(p) · GW(p)

One relevant point is that successfully building an FAI mitigates other x-risks.

Replies from: DaFranker
comment by DaFranker · 2013-04-19T14:12:59.834Z · LW(p) · GW(p)

A potentially-relevant sub-point is that successfully building a fast explosive fooming FAI optimally reduces all x-risks.

comment by private_messaging · 2013-04-18T08:13:28.262Z · LW(p) · GW(p)

Enrico Fermi said:

Well… there is the remote possibility that neutrons may be emitted in the fission of uranium

and then of course perhaps a chain reaction can be made.

The way I interpret it, he gave a remote possibility to enough neutrons being emitted in the fission of uranium (I guess from the tendency of other things to happen to excess neutrons in the nuclei, such as beta decay), and high probability ("of course") to the chain reaction conditional on the above.

I haven't written enough about what I've begun thinking of as 'heroic epistemology' - why, how can you possibly be so overconfident as to dare even try to have a huge positive impact when most people in that reference class blah blah blah - but on reflection, it seems to me that an awful lot of my answer boils down to not trying to be clever about it.

You know what, before trying a startup project seriously for money, I got clever about precisely this issue, joined TopCoder, and did a programming contest. I kind of think you'd likewise be smart enough to see the utility of reducing the uncertainty, if it was your own money on the line.

But in any case, I deny the original premise: I do not think the world's niches for heroism must be populated by heroes who are incapable in principle of reasonably distinguishing themselves from a population of crackpots, all of whom have no choice but to continue on the tiny off-chance that they are not crackpots.

That's all fine, but the issue is that the donors need to be able to do that, or else, no matter how much you argue that people shouldn't be donating on the basis of small probabilities, low probabilities is all they can have.

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:26:18.375Z · LW(p) · GW(p)

How did Fermi arrive at a 90% confidence for the false proposition that there weren't enough neutrons? What was the clever technical argument he immediately saw that Szilard and Rabi didn't, and why did it not work on Reality?

Replies from: gwern
comment by gwern · 2013-04-18T17:43:38.241Z · LW(p) · GW(p)

From your reference:

Fermi was not misleading Szilard. It was easy to estimate the explosive force of a quantity of uranium, as Fermi would do standing at his office window overlooking Manhattan, if fission proceeded automatically from mere assembly of the material; even journalists had managed that simple calculation. But such obviously was not the case for uranium in its natural form, or the substance would long ago have ceased to exist on earth. However energetically interesting a reaction, fission by itself was merely a laboratory curiosity. Only if it released secondary neutrons, and those in sufficient quantity to initiate and sustain a chain reaction, would it serve for anything more. "Nothing known then," writes Herbert Anderson, Fermi's young partner in experiment, "guaranteed the emission of neutrons. Neutron emission had to be observed experimentally and measured quantitatively." No such work had yet been done. It was, in fact, the new work Fermi had proposed to Anderson immediately upon returning from Washington. Which meant to Fermi that talk of developing fission into a weapon of war was absurdly premature.

Many years later Szilard succinctly summed up the difference between his position and Fermi's. "From the very beginning the line was drawn," he said. ". . . Fermi thought that the conservative thing was to play down the possibility that [a chain reaction] may happen, and I thought the conservative thing was to assume that it would happen and take all the necessary precautions."

(Even if he had an elegant technical argument, doesn't mean he would be right. Heisenberg had a short elegant argument for why the uranium critical mass would be 1 ton, but it was actually more like 10 pounds.)

Replies from: private_messaging, lukeprog, Eliezer_Yudkowsky
comment by private_messaging · 2013-04-18T20:36:42.671Z · LW(p) · GW(p)

There's something else. Most nuclei that will fission when irradiated with high energy neutrons (such as the ones from the neutron sources of the time) actually will not sustain chain reaction! That's the distinction between "fissile" and "fissionable". More here .

edit: Curiously enough, U238 can fission with neutrons produced by it's own fission, it just that most neutrons slow down before they fission any nucleus, and U238 can only be fissioned by fast neutrons. Had U235 had a bit shorter half life, or had evolution taken longer to make us, or had Sun formed later in the cloud, or the like, bench-top fission would still have been discovered (using neutrons from radium&lithium and U238) but we wouldn't have bomb anywhere near 1945 . This is quite interesting because of it's potential impact on Fermi's paradox. Nukes could be a lot harder to make.

Replies from: Luke_A_Somers
comment by Luke_A_Somers · 2013-04-18T21:45:27.747Z · LW(p) · GW(p)

Most don't, that's true. It only takes one.

There are at least four materials capable of sustaining a fission chain reaction, and any change to nuclear physics that is barely large enough to take those away would replace them with others. We are not even particularly near the boundary of it being possible.

Replies from: private_messaging
comment by private_messaging · 2013-04-18T21:52:05.662Z · LW(p) · GW(p)

1: When you only know that one fissions to begin with, you can't use that.

2: You need something naturally abundant enough, and (correct me if I am wrong) there's only one, it's U-235, with an outstandingly long half life for a fissile isotope, of 700 millions years, which is still not very long (other stuff doesn't hit even a million years). I wouldn't be surprised in the slightest if small adjustments to fundamental constants can change it to 2 and other small changes, 0.

edit: basically. No U-235, no bomb until you can bootstrap some sort of breeder using a particle accelerator, like, after decades and decades and decades of engineering, when all countries know the principle, but its too expensive.

Replies from: Luke_A_Somers
comment by Luke_A_Somers · 2013-04-18T22:09:17.476Z · LW(p) · GW(p)

But they made Pu-239 in time for WW2. Did that rely on reactor-grade U-235? Even if they did use reactor-grade U-235 to make it, could they have just stuck it near some other neutron emitter?

Replies from: private_messaging
comment by private_messaging · 2013-04-18T22:16:25.754Z · LW(p) · GW(p)

Yes, Fermi engineered the nuclear reactor (and very carefully too, with very well thought out safety system for it, just in case there would be a positive feedback of some kind), using natural uranium and graphite. Other neutron emitters would be very very very expensive. edit: e.g. an accelerator would need ridiculous amounts of electrical energy. The lab emitter used radium in combination with lithium, beryllium, or some other light nuclei, which isn't a viable route either.

Replies from: gwern
comment by gwern · 2013-04-24T20:57:37.607Z · LW(p) · GW(p)

It's a little sad that Fermi can correctly assume the worst in engineering his reactor and overengineer his precautions against a small chance of runaway feedback, and yet when it came to him estimating 10% for the possibility of an atomic bomb which he knew could be on the scale of city-obliterating and society-killing (because that calculation was so easy a journalist could do it), he apparently didn't do much of anything. Near vs Far, I guess.

Replies from: private_messaging
comment by private_messaging · 2013-04-24T21:23:48.226Z · LW(p) · GW(p)

What do you think he should/could have done about it? Try to prevent it's creation? Try to speed up it's creation? It's 10% on the fission releasing secondary neutrons, I'd think. Successful bomb requires other conditions to be met.

Easier to illustrate on example of hypothetical hafnium isomer initiated bullet sized thermonuclear bombs. What is the probability those are possible? That those will be made in next 5 years? Are you buying stocks on basis of that probability?

Replies from: gwern
comment by gwern · 2013-04-24T22:00:00.120Z · LW(p) · GW(p)

He could have done either, yes. He was a famous scientist, his Nobel in 1938 massively increased his credibility, his work was cited in the Einstein-Szilard letter which triggered the American bomb project and the first meeting of the bomb committee was to fund more work by Fermi. If Fermi could not have either slowed down or sped up the development of the bomb, it's hard to think who could have!

Are you buying stocks on basis of that probability?

I wasn't aware I assigned 10% either to their possibility or to the possibility of a critical but as-yet unknown requirement for their possibility.

Replies from: private_messaging
comment by private_messaging · 2013-04-25T04:14:56.685Z · LW(p) · GW(p)

He could have done either, yes.

Precisely. Now, which he should have done on basis of 10% possibility? There's a non-negligible cost as well, and 'business as usual' still leads to update from 10% to much larger probability, just a bit later, and arguably counter balanced by potential returns of other research done on the way. It's really easy to misestimate expected utility differences here by re-use of hindsight.

I wasn't aware I assigned 10% either to their possibility or to the possibility of a critical but as-yet unknown requirement for their possibility, a bit later but somewhat counterbalanced by various probabilities of discovering something else of interest.

Suppose you did assign 10% probability to hafnium triggering. You still have rest of the mini-nuke as a big conjunction, which is far clearer than for the nuke because for the nuke there's all this pop sci that makes it sound a lot easier than it actually is, plus having cherry picked the nuke that was actually made in time as an example, a lot of bad learning can be done.

Replies from: gwern
comment by gwern · 2013-04-26T04:20:19.745Z · LW(p) · GW(p)

Now, which he should have done on basis of 10% possibility?

I'm not sure what would be the best tack to take; knowing everything I do about the subsequent course of history, I think delaying the bomb program as much as possible would have been best, but I don't know if that was the best decision one would've made at the time.

and 'business as usual' still leads to update from 10% to much larger probability, just a bit later

Secrecy was already a major issue; business as usual could well lead to a fatal leak of information to the Germans who still were probably the first or second greatest physics establishment in the world and who Fermi must have known were also working on nuclear issues and were aware of the possibility of nuclear explosives; relevant papers were being kept secret and classified, etc. No, anyone at the time knew perfectly well that this was not a harmless issue with no consequences or actions which could be taken.

Well, anyone at the time if they weren't pulling a Fermi and essentially acting as if the 10% possibility didn't exist... Whether one decided to retard or advance the bomb, it seems highly unlikely to me that acting exactly as if he hadn't figured ~10% would just happen to turn out to be the right decision! Listen, I'm not the one claiming Fermi ignored the possibility, it's right there in the reference:

Many years later Szilard succinctly summed up the difference between his position and Fermi's. "From the very beginning the line was drawn," he said. ". . . Fermi thought that the conservative thing was to play down the possibility that [a chain reaction] may happen, and I thought the conservative thing was to assume that it would happen and take all the necessary precautions."

I don't criticize him for any sort of Pascal's wager, because 10% is a really effing big probability. Moving on...

Suppose you did assign 10% probability to hafnium triggering. You still have rest of the mini-nuke as a big conjunction

Suppose I did assign 10%. Then you cannot defend my inertia and ostrich attitude at 10% because the actual probability is lower! So make up your mind, is Fermi right to put his head in the sand at 10% or is the actual probability lower and he should be doing something else? Or are you arguing that 10% is a magical threshold at and below which head-in-the-sand is always the right response?

I think that is taking anti-Pascal-mugging way too far: if a doctor told me I had a 10% chance of dying this year of a rare disease, I would hysterically shove all the money I had at the doctor to make it go away and start drawing up my will and drafting letters for my family, I wouldn't go 'hm, this is very interesting, perhaps I should apply for more government money grants to study this fascinating possibility.'

Replies from: private_messaging
comment by private_messaging · 2013-04-26T04:55:53.984Z · LW(p) · GW(p)

I'm not sure what would be the best tack to take; knowing everything I do about the subsequent course of history, I think delaying the bomb program as much as possible would have been best, but I don't know if that was the best decision one would've made at the time.

This is another issue with wagers on speculative propositions, its unclear what actually should be done, besides secrecy which they'd already apply to any results involving secondary neutrons upon obtaining said results. edit: though, French obtained and published these results anyway. Which leaves the issue of carbon's neutron absorption cross-section, which Fermi measured as late as 1940 , after secondary neutrons (and thus well after the quote, at the point when self sustaining chain reaction was much less speculative).

Sure, the expected |utility| is huge, but expected utility is only huge when you can make predictions of what's best with good accuracy.

Suppose I did assign 10%. Then you cannot defend my inertia and ostrich attitude at 10% because the actual probability is lower!

And how'd you know that, I wonder? You have gamma ray emitting nuclear isomer that you can maybe trigger with x-rays. You have another nuclear isomer that can be triggered like this. It's not like fission's unknown-at-the-time secondary neutrons which are maybe released promptly (or maybe stay in the decay products which then undergo beta decay).

By the way, right here is your problem. "actual probability", which doesn't even meaningfully exist in such cases. You guys flip between subjectivist probability that lets you assign arbitrary numbers to the end of the world, and some intuitive notion of actual probability which is under which acting upon probabilities is the sanest thing.

if a doctor told me I had a 10% chance of dying this year of a rare disease

But the doctor did not tell that. Secondary neutrons do not necessarily imply the self sustaining chain reaction, which doesn't necessarily imply the bomb, which doesn't necessarily imply any effect on the course of the war! (Specifically the war that these European scientists cared about). This makes me wonder what exactly they were even talking about - maybe Szilard was worried they'd irradiate themselves fatally or blow themselves up doing experiments, and Fermi was like "Nuts!".

edit: also some background on quote is necessary - exact date down to the day, as the conclusions were moving really quick at that point.

edit2: a timeline: http://oznucforum.customer.netspace.net.au/ANFINFO8.pdf

comment by lukeprog · 2013-05-10T18:24:23.679Z · LW(p) · GW(p)

For others' reference: this begins on page 280 of The Making of the Atomic Bomb.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:55:14.168Z · LW(p) · GW(p)

It's wise to consider how non-hindsight might have been harder. It's even wiser to consider, for each training example, what general heuristics might've helped anyway.

But such obviously was not the case for uranium in its natural form, or the substance would long ago have ceased to exist on earth.

But there ought to be some unstable elements that hadn't fizzed by themselves in natural aggregations and purities, and many such, and these might be manipulated by humans. If something doesn't happen naturally, are you in a situation where you're likely to be learning about a randomly placed lower bound that's probably randomly far above you, or in a case where you're learning about a nearby lower bound that probably has some things right above it?

However energetically interesting a reaction, fission by itself was merely a laboratory curiosity.

Sounds like an absurdity heuristic; this is a bad general lesson to learn. "Laboratory curiosity" foresooth.

Only if it released secondary neutrons, and those in sufficient quantity to initiate and sustain a chain reaction, would it serve for anything more.

Which it did. So why should one have been confident that they didn't...?

"Nothing known then," writes Herbert Anderson, Fermi's young partner in experiment, "guaranteed the emission of neutrons. Neutron emission had to be observed experimentally and measured quantitatively."

The good old confusion between negative information and positive information of falsehood, perhaps?

Again, trying to avoid hindsight bias is not best done by inventing new cynical contrarian ideas that serve to steer your mind in the opposite direction of each training example. It would be better to look for truths that are hard to see, and not plausible falsehoods that by golly you ought to have believed. "It would have been just as easy to think Y as X, given Z" is a powerful argument against an alleged heuristic Z that supposedly could've told you X. "But it would have been perfectly rational to think Y!" is not how you want to train yourself.

Replies from: private_messaging, CarlShulman
comment by private_messaging · 2013-04-19T16:29:02.742Z · LW(p) · GW(p)

But there ought to be some unstable elements that hadn't fizzed by themselves in natural aggregations and purities, and many such, and these might be manipulated by humans. If something doesn't happen naturally, are you in a situation where you're likely to be learning about a randomly placed lower bound that's probably randomly far above you, or in a case where you're learning about a nearby lower bound that probably has some things right above it?

This doesn't actually work...

There's only 3 isotopes to choose from. Th232 , U238 , U235 . Evidence that fission occurs probably came from U238 being fissioned by fast neutrons (or could just as well have). You can't make a bomb out of U238 , though, because it doesn't get fissioned by slow neutrons, and neutrons slow down quite rapidly, before they fission it enough. You need a nucleus so unstable, that it fissions when it captures a neutron. It must also fission immediately - if it fissions with a delay (if the mechanism of fissioning is that it captures the neutron, transmutes into something unstable that fissions later. Because neutrons do not leave tracks you don't immediately know that this is not what is going on).

There's precisely one naturally abundant isotope that you can use, it is U235 . Forget about plutonium, it'd be very expensive to make any without a reactor. Without naturally abundant U235 , no bomb anywhere near 1945 . It'd be something akin to an antimatter bomb - you need to make the material in a particle accelerator, which is ridiculously inefficient. (One could maybe make some plutonium in the particle accelerator, then use that plutonium in a breeder reactor to kick-start breeder economy, but the energy requirements for production of the seed plutonium are still utterly insane)

There's very little U235 because it has half life of 700 millions years. It is still ~4400 times the half life of the next most stable isotope that you could blow up, though (U233). Which has ~7x the half life of the next stablest (Pu239) , which has 3.2 x the half life of the next stablest (Am-243) . Which suggests to me that it's like "this cylinder will land with it's axis horizontal" prediction for something that turns out to be a coin rather than a pencil. (Frankly I do not understand why we even have any U235 at all. Could be some really weird anthropic reason that we don't know enough to deduce)

edit: doh, a correction. Neptunium-237 , albeit never used in bombs (critical mass 60kg - ish), can maybe be made into one, no doubt with great difficulty due to the size. It has half life of 2 million years. So the sequence of relative half-life becomes 3.2, 7, 12.5, 350

"But it would have been perfectly rational to think Y!" is not how you want to train yourself.

Well of course, but when 2 dice rolled sixes, you can't go on how it was irrational of Fermi to think the probability of such is 1/36 before anyone ever looked at the dice.

comment by CarlShulman · 2013-04-24T02:24:03.734Z · LW(p) · GW(p)

On the other hand, if you want a general heuristic that could've led Fermi to do better, I would suggest reasoning that previous-historical experimental proof of a chain reaction would not be strongly be expected even in worlds where it was possible, and that to discover a chain reaction to be impossible would imply learning some new fact of physical science which was not already known.

It's wise to consider how non-hindsight might have been harder. It's even wiser to consider, for each training example, what general heuristics might've helped anyway.

I think the comments have done a good job showing that learning that a chain reaction was possible would have also implied new facts of physical science, e.g. about neutron emission of the available isotopes, so the heuristic in the OP doesn't help much.

Replies from: private_messaging
comment by private_messaging · 2013-04-24T20:17:46.897Z · LW(p) · GW(p)

Yes. Essentially, "they were still the default projection from what was already known." was dramatically untrue at the time of 10% assessment, when it was not known that any neutrons are produced in fission, or any similar processes. And once it became true, Enrico Fermi did rapidly do a very difficult calculation of the neutron multiplication factor, and concluded that self sustaining chain reaction is possible.

The imaginary world where people of Enrico Fermi calibre are unable to follow simple steps due to the extreme conclusions, is a shared fantasy of many, many crackpots.

comment by James_Miller · 2013-04-18T14:13:25.324Z · LW(p) · GW(p)

And finally, I once again state that I abjure, refute, and disclaim all forms of Pascalian reasoning and multiplying tiny probabilities by large impacts when it comes to existential risk.

So is this a bad reason to give $100 to MIRI:

"MIRI reduces existential risks by a non-tiny probability. My contribution of $100 would increase the chance of MIRI's success, however, by only a tiny probability. Still, multiplying this tiny probability increase by the good that would occur if my $100 did end up making the difference justifies my giving $100 to MIRI."

Replies from: Eliezer_Yudkowsky, shminux
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:29:38.292Z · LW(p) · GW(p)

On a very large scale, if you think FAI stands a serious chance of saving the world, then humanity should dump a bunch of effort into it, and if nobody's dumping effort into it then you should dump more effort than currently into it. Calculations of marginal impact in POKO/dollar are sensible for comparing two x-risk mitigation efforts in demand of money, but in this case each marginal added dollar is rightly going to account for a very tiny slice of probability, and this is not Pascal's Wager. Large efforts with a success-or-failure criterion are rightly, justly, and unavoidably going to end up with small marginal probabilities per added unit effort. It would only be Pascal's Wager if the whole route-to-humanity-being-OK were assigned a tiny probability, and then a large payoff used to shut down further discussion of whether the next unit of effort should go there or to a different x-risk.

Replies from: James_Miller
comment by James_Miller · 2013-04-18T23:12:30.589Z · LW(p) · GW(p)

Thanks for answering. I just gave $100 to MIRI.

comment by shminux · 2013-04-18T16:19:35.506Z · LW(p) · GW(p)

Eliezer explains it in his other comment (emphasis mine):

Before scanning, I precommit to renouncing, abjuring, and distancing MIRI from the argument in the video if it argues for no probability higher than 1 in 2000 of FAI saving the world, because I myself do not positively engage in long-term projects on the basis of probabilities that low (though I sometimes avoid doing things for dangers that small). There ought to be at least one x-risk effort with a greater probability of saving the world than this - or if not, you ought to make one. If you know yourself for an NPC and that you cannot start such a project yourself, you ought to throw money at anyone launching a new project whose probability of saving the world is not known to be this small.

comment by gothgirl420666 · 2013-04-25T01:30:15.638Z · LW(p) · GW(p)

For so long as I can remember, I have rejected Pascal's Wager in all its forms on sheerly practical grounds: anyone who tries to plan out their life by chasing a 1 in 10,000 chance of a huge payoff is almost certainly doomed in practice.

Almost certainly doomed, yes. You might even say doomed 9,999 out of 10,000 times.

comment by A1987dM (army1987) · 2013-04-18T18:27:01.003Z · LW(p) · GW(p)

If I have pneumonia and the doctor tells me that there is a remote possibility that I might die, and it's ten percent, I get excited about it.

Did “excited” mean something different back then? (If so, I may have misinterpreted a certain line in “All Along the Watchtower” by Bob Dylan.)

Replies from: AlanCrowe
comment by AlanCrowe · 2013-04-18T22:53:37.894Z · LW(p) · GW(p)

I retain an eight volume dictionary from 1899 to answer this kind of question. Meaning four is

To arouse the emotions of; agitate or perturb mentally; move: as, he was greatly excited by the news.

One real-life example is

The news of the fall of Calcutta reached Madras, and excited the fiercest and bitterest resentment

Today "exciting" is often contrasted with "boring" and has a positive connotation. (eg "We hoped the football game would be exciting and were disappointed when it was boring.") My old dictionary seems evenly balanced with "excited" being bad and good by turns.

Replies from: DaFranker
comment by DaFranker · 2013-04-19T14:11:20.068Z · LW(p) · GW(p)

I retain an eight volume dictionary from 1899 to answer this kind of question.

That's some epic skill levels in Arcane Lore right there.

Replies from: gwern
comment by gwern · 2013-04-19T17:05:13.204Z · LW(p) · GW(p)

Or he could've just downloaded a copy of the OED.

("excited: 1.a Stirred by strong emotion, disturbed, agitated. '1855 Macaulay Hist. Eng. III. 275 "The population of Edinburgh was in an excited state." 1864 Mrs. Carlyle Lett. III. 216 "The excited people‥rushed out to me." 1879 McCarthy Own Times I. 199 "Thiers carried with him much of the excited public feeling of France."'")

comment by Jonathan_Graehl · 2013-04-18T07:47:08.881Z · LW(p) · GW(p)

Laughing at Fermi for 10% is uncharitable.

It sounds like his heuristic for deciding what avenue of research to follow rejected chain reactions. If, as Eliezer claims, >10% should have been obvious to Fermi if he really thought about it, then we can conclude that he didn't feel a need to think about it, for whatever reason.

I do wish senior/brilliant thinkers wouldn't discourage anyone based on their take of something they haven't really thought about, but that probably doesn't stop the really bold upstarts.

I'd like to understand better why really bright and hard-working people don't bother with real thinking when having discussions about their area of theoretical expertise. I guess a social interaction often demands snap decisions.

"heroic epistemology" - that's what all those now-rich startups used, right?

I've fallen out of love with weighing "if people like me ...". Yes, I'd like to make love, and not war, if I were on a boat with 50 clones of myself. But does this help me decide or achieve anything in reality? Aren't there more interesting things to work on than fantasy? I already am the kind of person I am. (partial rebuttal: Eliezer is just the kind of person who's changed/focused by "if people like me ..." daydreams!)

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:32:21.655Z · LW(p) · GW(p)

"heroic epistemology" - that's what all those now-rich startups used, right?

It's what AirBNB used. I didn't get a chance to hear about them until they had traction, but I honestly think my general good-idea heuristics would've fired more strongly on this than a lot of conventional wisdom.

Wow, you sure are selective in your charity...

comment by dgsinclair · 2013-04-18T20:43:53.999Z · LW(p) · GW(p)

EY: I don't multiply tiny probabilities by huge impacts. I also don't get tiny probabilities by putting myself into inescapable reference classes, for this is the sort of reasoning that would screw over planets that actually were in trouble if everyone thought like that.

But isn't the latter exactly what you are doing with Pascal's wager? Underestimating the existence of God's probability so that you may retreat back to 'tiny probability'?

comment by OrphanWilde · 2013-04-18T07:42:10.861Z · LW(p) · GW(p)

Isn't Fermi the guy who insisted that a nuclear reaction could set the atmosphere on fire in a massive nuclear reaction?

I'm having trouble making sense of the quoted section. It makes a lot more sense if that's what they're talking about, especially the "if it means that we may die of it," rather than the possibility of a nuclear reaction in general.

Replies from: CarlShulman, Eliezer_Yudkowsky
comment by CarlShulman · 2013-04-18T17:20:02.411Z · LW(p) · GW(p)

Isn't Fermi the guy who insisted that a nuclear reaction could set the atmosphere on fire in a massive nuclear reaction?

That was Teller. And I think it was more "raised the possibility" than "insisted."

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-18T17:28:31.355Z · LW(p) · GW(p)

Isn't Fermi the guy who insisted that a nuclear reaction could set the atmosphere on fire in a massive nuclear reaction?

Not as far as I know. This was considered technically even though it seemed obviously false on its face, assigned an even lower credence afterward, and then it didn't in fact turn out to be true.

comment by Dr_Manhattan · 2013-04-19T20:17:54.039Z · LW(p) · GW(p)

I always wondered if Szilard's slightly outcast status (if my recollection of Rhodes book is correct) helped him see things establishment scientists ignored.

comment by lukeprog · 2013-04-22T01:37:01.072Z · LW(p) · GW(p)

"Found"? Didn't you write that post, Dmytry? Why wouldn't you just say so?

Replies from: common_law, private_messaging
comment by common_law · 2013-04-22T01:53:44.460Z · LW(p) · GW(p)

Now there's an excellent example of rationality failure: I'm not Dmytry. Check my profile and my blogs.

Replies from: lukeprog
comment by lukeprog · 2013-04-22T02:07:54.909Z · LW(p) · GW(p)

Oh wait, you're that other person with a bunch of different monikers: metaphysicist, srdiamond, etc. Sorry.

Replies from: wedrifid, common_law
comment by wedrifid · 2013-04-22T02:19:50.208Z · LW(p) · GW(p)

Oh wait, you're that other person with a bunch of different monikers: metaphysicist, srdiamond, etc. Sorry.

There is another (known) sockpuppet abuser that I need to downvote? Bother. I thought we just had the one.

Replies from: army1987, common_law, common_law
comment by A1987dM (army1987) · 2013-04-22T11:35:28.577Z · LW(p) · GW(p)

The user formerly known as Dmytry now posts as private_messaging, IIRC. (“The one” would be Newsome, right?)

Replies from: wedrifid
comment by wedrifid · 2013-04-22T12:13:29.039Z · LW(p) · GW(p)

The user formerly known as Dmytry now posts as private_messaging, IIRC.

Among others. Someone made a list of them at one point.

(“The one” would be Newsome, right?)

I was thinking of Dmytry et al.

comment by common_law · 2013-04-24T18:16:16.947Z · LW(p) · GW(p)

To Vladimir Nesov:

"A particularly unpopular posting" is not normally the issue, it's usually the systematic failure to respond to negative feedback, including by stopping to post in particular modes or at all.

I'm sorry: what's a "particular mode"? And what does stopping to post altogether have to do with multiple identities?

More to the point, what is "responding to feedback"? Posting responses to disagreement? Surely you know that depresses "karma" further. Or is "responding to feedback" a euphemism for conforming one's opinion and conduct to the community?

Mr. Nesov, you want to be a scientist; why do you post in bureaucratese? Obfuscatory writing is both cause and symptom of wretched thinking.

Edit. Changed Nessov to Nesov.

Replies from: wedrifid, Kawoomba
comment by wedrifid · 2013-04-24T19:05:56.614Z · LW(p) · GW(p)

To Vladimir Nessov:

Since I'm not Nesov I can only assume you are writing to me because of some argument you are having with someone else is in a trollblocked thread. I'm fairly sure Nesov has a button meant for just this kind of thing.

comment by Kawoomba · 2013-04-24T19:15:22.758Z · LW(p) · GW(p)

Mr. Nessov, (...) why do you post in bureaucratese?

I don't know who this Mr. Nessov is, but I sure am glad he seems to have very little in common with our Vladimir Nesov.

comment by common_law · 2013-04-24T01:17:19.806Z · LW(p) · GW(p)

Doesn't sockpuppetry occur when one poster uses another identity for deceitful purposes, usually to exaggerate his support or denigrate the quality of opposition? I think you have to consider the mens rea before accusing of sockpuppetry--unless you have a rule I haven't noticed against multiple identities. If Dmytry had been guilty of what lukeprog accused him (so cavalierly), Dmytry would have been engaging in sockpuppetry.

There are prudential reasons for having multiple identities. It's like the protection of incorporation: it limits damage. If one identity goes down in flames after a particularly unpopular posting, it doesn't exhaust my "karma" capital.

Replies from: Vladimir_Nesov, wedrifid, TimS
comment by Vladimir_Nesov · 2013-04-24T10:11:09.090Z · LW(p) · GW(p)

If one identity goes down in flames after a particularly unpopular posting, it doesn't exhaust my "karma" capital.

One purpose of (significantly) negative (total) Karma is to indicate when a user should be kicked out. Limiting this damage damages the forum. ("A particularly unpopular posting" is not normally the issue, it's usually the systematic failure to respond to negative feedback, including by stopping to post in particular modes or at all.)

Replies from: drethelin, bogus, troll
comment by drethelin · 2013-04-24T16:47:02.550Z · LW(p) · GW(p)

For example: I've made posts and comments that get massively downvoted but I have 2500 karma so I don't get kicked out.

comment by bogus · 2013-04-24T19:59:07.390Z · LW(p) · GW(p)

One purpose of (significantly) negative (total) Karma is to indicate when a user should be kicked out.

I disagree, total karma should never be the sole indicator of whether a user should be kicked out. If a user is being obnoxious enough that kicking her out becomes a possibility, this will be quite evident from a cursory look at her post and comment log. I'm thinking of a LW user who would make a perfect example of this, except that I just can't recall his or her name. Monkey(something), IIRC.

Total karma does have some useful technical effects, such as (AIUI) disabling downvotes and rate-limiting posts and comments.

comment by troll · 2013-04-24T16:17:29.031Z · LW(p) · GW(p)

Some people might downvote everything another has ever said out of spite.

Some might gather a group of friends to downvote a comment they don't like.

It's like giving everyone a double vote.

We don't kick people out of real life for disagreeing with us. (some do but no)

Users should never be kicked out. Downvotes shouldn't exist. Baring extreme scenarios, a system of upvotes should sort everything out.

(hesitantly) And total karma shouldn't exist. You should feel good for making a good comment, but not for making good comments in the past.

Replies from: drethelin, mwengler, Richard_Kennaway
comment by drethelin · 2013-04-24T16:44:20.253Z · LW(p) · GW(p)

Spite exists, and people do things out of spite. That doesn't mean punishment shouldn't exist. If you don't stop being friends with anyone ever you will be abused and used and forced to spend time with awful people.

Total karma isn't for you, it's for everyone else.

Replies from: common_law
comment by common_law · 2013-04-24T19:03:09.682Z · LW(p) · GW(p)

Total karma isn't for you, it's for everyone else.

Producing correlated rather than independent judgments of post quality, with the well-known cascading effects. The "system" deliberately introduces what I call belief-opinion confusion

comment by mwengler · 2013-04-24T17:08:15.481Z · LW(p) · GW(p)

Intentionally or not you just described the karma system at fool.com (which they call recommendations). As a long time poster there it does have its uses: there are some extremely prolix boards where I skip through reading only the high rec posts, and don't have to worry about being "shielded" from controversial ideas like you get with the system here.

If i were putting someone else's karma system in here, I would put in stackexchange.com They have downvoting, but it costs the downvoter karma. So you don't get things mindlessly downvoted just because they don't kowtow. But actual low quality stuff is downvoted by high-karma individuals.

The recent improvement of showing pctage of downvotes here is nice. I think I would go the last step and "decorate" the up and down vote buttons with the total number of existing up and down votes. This would also be slightly more compact than the current system where I have points reported on top of comment, and then uninformative buttons below the post. And I would by scanning upvote totals be able to use recs in the way you want them used, if I wanted to ignore downvotes.

comment by Richard_Kennaway · 2013-04-25T10:35:37.325Z · LW(p) · GW(p)

Users should never be kicked out. Downvotes shouldn't exist.

The issues you are raising have been thoroughly aired here in the past. It isn't helpful to barge in and propose rearranging the furniture just because you don't see why it is the way it is. For some of the background I suggest this posting. There is certainly room for useful discussion of how LessWrong should be run, but it has to start from where it is.

Baring extreme scenarios, a system of upvotes should sort everything out.

Right.

Which meaning of "should" is that?

Now. I've written the above on the hypothesis that you are not exactly what you call yourself, a troll. But the evidence is not favourable. You pre-emptively shot yourself in the foot by choosing that handle, posted a bio consisting of a list of what are hurrah keywords on LessWrong, accumulated some karma with innocuous postings, then started posting stuff that instantly gets downvoted out of sight.

If you are genuine, you need to change something. That is not something that I will say twice.

comment by wedrifid · 2013-04-24T02:37:44.696Z · LW(p) · GW(p)

Doesn't sockpuppetry occur when one poster uses another identity for deceitful purposes, usually to exaggerate his support or denigrate the quality of opposition?

Yes. That is precisely what we are talking about. You may argue that once the identities have been identified and can no longer be used simultaneously in the same conversation they are no longer technically sockpuppets. But that distinction doesn't seem terribly important.

There are prudential reasons for having multiple identities. It's like the protection of incorporation: it limits damage. If one identity goes down in flames after a particularly unpopular posting, it doesn't exhaust my "karma" capital.

Yes. There are incentives for using multiple accounts (whether for sock-puppetry, karma assassination or otherwise). I prefer the case where such practice is sufficiently discouraged that any perpetrators must at least go to the effort of acting differently---maybe by having different pet issues that they rant about rather than the same battle and same arguments with a different moniker.

comment by TimS · 2013-04-24T02:11:01.491Z · LW(p) · GW(p)

There's no rule against multiple accounts (AFAIK), but there is a local social norm against multiple accounts. Some folks even dislike Clippy, who I find hilarious.

comment by common_law · 2013-04-22T18:23:31.398Z · LW(p) · GW(p)

Apology accepted, but I think it's Dmytry to whom you actually owe it: he's the one you recklessly accused of deceitful self-promotion.

comment by private_messaging · 2013-04-22T05:09:53.112Z · LW(p) · GW(p)

lol. Great physicists (Enrico Fermi) obviously bullshitting (even though the word is not said, the assumption is made that 10% comes from nowhere) because you confabulated what they did know at the time, people coming in from links on my blog getting identified as my socks... that on top of the world dying if not for your work, with probability over 30% (edit: ohh, sorry, that's unclear - 30% conditioning on there being world to save, like its saner this way)... you guys are, like, super rational or something.

comment by RomeoStevens · 2013-04-18T08:33:11.837Z · LW(p) · GW(p)

blarg. TDT + this post = http://imgur.com/gallery/6NfmQ

Thinking this way is probably good for my motivation/productivity, but does make me a bit more stressed.