The Irrationality Game

Given that most of the top comments are meta in one way or another it would seem that the 'top comments' list belongs somewhere other than on the front page. Can't we hide the link to it on the wiki somewhere?

Yes.

I think "top comments" was an experiment with a negative result, and so should be removed.

Prior before having learned of Fatima, roughly? Best guess at current probability?

I don't think that's a very good guess, but it's as good as any I've seen. I tried to phrase my belief statement to include things like this within its umbrella.

This is what spurred me to give consideration to the idea initially, but what makes me confident is sifting through simply mountains of reports. To get an idea of the volume and typical content, here's a catalog of vehicle interference cases in Australia from 1958 to 2004. Most could be explained by a patchwork of mistakes and coincidences, some require more elaborate, "insanity or hoax" explanations, and if there are multiple witnesses, insanity falls away too. But there is no pattern that separates witnesses into a "hoax" and a "mistake" group, or even that separates them from the general population.

But the rules are different in this thread. 64 here means that 64 more voters disagree than agree.

It certainly wasn't high... I went through most of my life never giving the idea a thought, stumbled onto the miracle of fatima one day, and said "well, clearly this wasn't a flying saucer, but what the heck was it?"

But the rabbit hole just kept going down. It is not a particularly pleasant feeling to me, as someone who used to think he had a fairly solid grip on the workings of the world.

Agreed. I think they've explicitly denied that they're working on AGI, but I'm not too reassured. They could be doing it in secret, probably without much consideration of Friendliness, and even if not, they're probably among the entities most likely (along with, I'd say, DARPA and MIT) to stumble upon seed AI mostly by accident (which is pretty unlikely, but not completely negligible, I think).

If they've explicitly denied doing research into AGI, they would have no reason to talk about friendliness research; that isn't additional evidence. I do think the OP is extremely overconfident though.

If he could, he wouldn't be making that mistake in the first place.

Well, from a certain point of view you could see the two propositions as being essentially equivalent... i.e. the inhabitants of a higher layer reality poking through the layers and toying with us (if you had a universe simulation running on your desktop, would you really be able to refrain from fucking with your sims' heads)? So whatever probability you assign to one proposition, your probability for the other shouldn't be too much different.

I certainly agree with you now, but it wasn't entirely certain what you meant by your statement. A qualifier might help.

What do you mean by this? Do you mean "a non-negligible fraction of my measure is in a simulation" in which case you're almost certainly right. Or do you mean "this particular instantiation of me is in a simulation" in which case I'm not sure what it means to assign a probability to the statement.

So you know which I must have meant, then. I do try to be almost certainly right. ;)

(Technically, we shouldn't really be thinking about probabilities here either because it's not important and may be meaningless decision theoretically, but I think LW is generally too irrational to have reached the level of sophistication such that many would pick that nit.)

Our point-estimate best model plausibly says so, but, structural uncertainty? (It's not privileging the non-simulation hypothesis to say that structural uncertainty should lower this probability, or is it?)

That is a good question. I feel like asking 'in what direction would structural uncertainty likely bend my thoughts?' leads me to think, from past trends, 'towards the world being bigger, weirder, and more complex than I'd reckoned'. This seems to push higher than 99.5%. If you keep piling on structural uncertainty, like if a lot of things I've learned since becoming a rationalist and hanging out at SIAI become unlearned, then this trend might be changed to a more scientific trend of 'towards the world being bigger, less weird, and simpler than I'd reckoned'. This would push towards lower than 99.5%.

What are your thoughts? I realize that probabilities aren't meaningful here, but they're worth naively talking about, I think. Before you consider what you can do decision theoretically you have to think about how much of you is in the hands of someone else, and what their goals might be, and whether or not you can go meta by appeasing those goals instead of your own and the like. (This is getting vaguely crazy, but I don't think that the craziness has warped my thinking too much.) Thus thinking about 'how much measure do I actually affect with these actions' is worth considering.

Our point-estimate best model plausibly says so, but, structural uncertainty? (It's not privileging the non-simulation hypothesis to say that structural uncertainty should lower this probability, or is it?)

That's a good question. My impression is that it is somewhat. But in the figures we are giving here we seem to be trying to convey two distinct concepts (not just likelyhoods).

Pah! Real Bayesians don't need experiment or testing; Bayes transcends the epistemological realm of mere Science. We have way more than enough data to make very strong guesses.

Yep. Over-reliance on anthropic arguments IMO.

If 'living in a simulation' includes those scenarios where the beings running the simulation never intervene then I think it's a non-trivial philosophical question whether "we are living in a simulation" actually means anything.

It's a nontrivial philosophical question whether 'means anything' means anything here. I would think 'means anything' should mean 'has decision theoretic significance'. In which case knowing that you're in a simulation could mean a lot.

First off, even if the simulators don't intervene, we still intervene on the the simulators just by virtue of our existence. Decision theoretically it's still fair game, unless our utility function is bounded in a really contrived and inelegant way.

(Your link is way too long for me to read. But I feel confident in making the a priori guess that Putnam's just wrong, and is trying too hard to fit non-obvious intuitive reasoning into a cosmological framework that is fundamentally mistaken (i.e., non-ensemble).)

[S]urely everything we know about science - the uniformity and universality of physical laws - suggests that this is false.

What if I told you I'm a really strong and devoted rationalist who has probably heard of all the possible counterarguments and has explicitly taken into account both outside view and structural uncertainty considerations, and yet still believes 99.5% to be reasonable, if not perhaps a little on the overconfident side?

Hilary Putnam made (or gave a tantalising sketch of) an argument that even if we were living in a simulation, a person claiming "we are living in a simulation" would be incorrect.

Err... I'm not intimately acquainted with the sport myself... What's the approximate difficulty rating of that kind of verbal gymnastics stunt again? ;)

Care to lay out the evidence? Or is this not the place for that?

I really couldn't; it's such a large burden of proof to justify 99.5% certainty that I would have to be extremely careful in laying out all of my disjunctions and explaining all of my intuitions and listing every smart rationalist who agreed with me, and that's just not something I can do in a blog comment.

Also, what exactly do you consider making up "your measure"? Just isomorphic computations?

Naively, probabilistically isomorphic computations, where the important parts of the isomorphism are whatever my utility function values... such that, on a scale from 0 to 1, computations like Luke Grecki might be .9 'me' based on qualia valued by my utility function, or 1.3 'me' if Luke Grecki qualia are more like the qualia my utility function would like to have if I knew more, thought faster, and was better at meditation.

How is that different than "I believe that I am a simulation with non-negligible probability"?

If the same computation is being run in so-called 'basement reality' and run on a simulator's computer, you're in both places; it's meaningless to talk about the probability of being in one or the other. But you can talk about the relative number of computations of you that are in 'basement reality' versus on simulators' computers.

This also breaks down when you start reasoning decision theoretically, but most LW people don't do that, so I'm not too worried about it.

In a dovetailed ensemble universe, it doesn't even really make sense to talk about any 'basement' reality, since the UTM computing the ensemble eventually computes itself, ad infinitum. So instead you start reasoning about 'basement' as computations that are the product of e.g. cosmological/natural selection-type optimization processes versus the product of agent-type optimization processes (like humans or AGIs).

The only reason you'd expect there to be humans in the first place is if they appeared in 'basement' level reality, and in a universal dovetailer computing via complexity, there's then a strong burden of proof on those who wish to postulate the extra complexity of all those non-basement agent-optimized Earths. Nonetheless I feel like I can bear the burden of proof quite well if I throw a few other disjunctions in. (As stated, it's meaningless decision theoretically, but meaningful if we're just talking about the structure of the ensemble from a naive human perspective.)

This means that there exist an infinite number of Earths like ours that are in a simulation, and an infinite number of Earths like ours that are not in a simulation. Thus it becomes meaningless to ask whether or not we exist in a simulation. We exist in every possible world containing us that is a simulation, and exist in every possible world containing us that is not a simulation.

Just because a set is infinite doesn't mean it's meaningless to speak of measures on it.

Right, I agree with Eugine Nier: the relative measures are important. You are in tons of universes at once, but some portion of your measure is simulated, and some not. What's the portion?

The motive would be NASA's budget being mostly funneled into someone's pockets rather than materials and labor for actual spaceflight. I tried to take into account that people underestimate the power of orders of magnitude (~1 in 100 possible universes would have this be true--like a single person wearing a blue shirt in a decent-size crowd of red-shirted people).

And what reason do you have for assigning a high probability to an unfriendly AI coming into existence with Eliezer not involved in its creation?

;)

Edit: I meant what reason do you (nic12000) have? Not you (RobinZ). Sorry for the confusion.

Well, most of the arguments against it are, to my knowledge, start with something along the lines of "If time travel exists, causality would be fucked up, and therefore time travel can't exist," though it might not be framed quite that implicitly.

Also, if FTL travel exists, either general relativity is wrong, or time travel exists, and it might be possible to create FTL travel by harnessing the Casimir effect or something akin to it on a larger scale, and if it is possible to do so, a recursively improving AI will figure out how to do so.

nick voted up, robin voted down... This feels pretty weird.

Wait... was the grandparent serious? He's talking about the flaming swords of the angels being laser turrents! That's got to be tongue in cheek!

We should learn to present this argument correctly, since complexity of hypothesis doesn't imply its improbability. Furthermore, the prior argument drives probability through the floor, making 99% no more surprising than 1%, and is thus an incorrect argument if you wouldn't use it for 1% as well (would you?).

Here's another case:

"Let me get this straight. We had sex. I wind up in the hospital and I can't remember anything?" Alice said. There was a slight pause. "You owe me a 30-carat diamond!" Alice quipped, laughing. Within minutes, she repeated the same questions in order, delivering the punch line in the exact tone and inflection. It was always a 30-carat diamond. "It was like a script or a tape," Scott said. "On the one hand, it was very funny. We were hysterical. It was scary as all hell." While doctors tried to determine what ailed Alice, Scott and other grim-faced relatives and friends gathered at the hospital. Surrounded by anxious loved ones, Alice blithely cracked jokes (the same ones) for hours.

I wish it was professionally ethical for psychologists to do this kind of thing intentionally.

They could probably do some relevant research by talking to Alzheimer's patients - they wouldn't get anything as clear as that, I think, but I expect they'd be able to get statistically-significant data.

I would classify such kinds of predictions as vague, after all they match equally well for every human being in almost any condition.

I guess I'm playing the game right then :)

I'm curious, do you also think that a singleton is a desirable outcome? It's possible my thinking is biased because I view this outcome as a dystopia and so underestimate it's probability due to motivated cognition.

I expected the third to be higher than most less wrongers would estimate.

Stable equilibrium here does not refer to a property of a mind. It refers to a state of the universe. I've elaborated on this view a little here before but I can't track the comment down at the moment.

Essentially my reasoning is that in order to dominate the physical universe an AI will need to deal with fundamental physical restrictions such as the speed of light. This means it will have spatially distributed sub-agents pursuing sub-goals intended to further its own goals. In some cases these sub-goals may involve conflict with other agents (this would be particularly true during the initial effort to become a singleton).

Maintaining strict control over sub-agents imposes restrictions on the design and capabilities of sub-agents which means it is likely that they will be less effective at achieving their sub-goals than sub-agents without such design restrictions. Sub-agents with significant autonomy may pursue actions that conflict with the higher level goals of the singleton.

Human (and biological) history is full of examples of this essential conflict. In military scenarios for example there is a tradeoff between tight centralized control and combat effectiveness - units that have a degree of authority to take decisions in the field without the delays or overhead imposed by communication times are generally more effective than those with very limited freedom to act without direct orders.

Essentially I don't think a singleton AI can get away from the principal-agent problem. Variations on this essential conflict exist throughout the human and natural worlds and appear to me to be fundamental consequences of the nature of our universe.

Two points that influence my thinking on that claim:

1. Gains from trade have the potential to be greater with greater difference in values between the two trading agents.
2. Destruction tends to be cheaper than creation. Intelligent agents that recognize this have an incentive to avoid violent conflict.

Just to make sure, is this talking about something different from people committing less crimes when they are themselves practicing TM or in daily contact with someone who does?

I'm not sure if personal contact with practitioners has an effect, but the studies I'm thinking of were on the level of cities -- put a group of meditators in Chicago, the Chicago crime rate goes down.

What arm TM-Sidhis, are they something distinct from regular TM (are these different types of practicioners).

TM-Sidhis is a separate/additional practice that has TM as a dependency in the sense of package management. If you know TM, you can learn TM-Sidhis.

And what's with the sqrt(1%)? One in ten people in the total population need to be TM-Sidhis for the crime rate reduction effect to kick in?

Sorry, I meant sqrt(.01p) where p is the population of the group to be affected. For example, a city of one million people would require ten thousand TM meditators or 100 TM-Sidhis meditators.

Can you clarify the question, or does the whole statement seem meaningless?

Yes!

Humans are "designed" to act intelligently in the physical world here on earth, we have complex adaptations for this environment. I don't think we are capable of acting effectively in "strange" environments, e.g. we are bad at predicting quantum mechanical systems, programming computers, etc.

Of course you're right in the strictest sense! I should have included something along the lines of "an algorithm that can be efficiently computed", this was already discussed in other comments.

I'm not sure what your point is, I don't think I use the term "algorithm" in a non-standard way.

Wikipedia says: "Thus, an algorithm can be considered to be any sequence of operations that can be simulated by a Turing-complete system."

When talking about "intelligence" I assume we are talking about a goal-oriented agent, controlled by an algorithm as defined above.

No, I don't think I would be capable if the domain is sufficiently different from the EEA.

I wrote about it here:

http://www.ideasinactiontv.com/tcs_daily/2007/10/a-thousand-chinese-einsteins-every-year.html

Once we have identified genes that play a key role in intelligence then eugenics through massive embryo selection has a good chance at producing lots of super-geniuses especially if you are willing to tolerate a high "error rate." The Chinese are actively looking for the genetic keys to intelligence. (See http://vladtepesblog.com/?p=24064) The Chinese have a long pro-eugenics history (See Imperfect Conceptions by Frank Dikötter) and I suspect have a plan to implement a serious eugenics program as soon as it becomes practical which will likely be within the next five years.

I almost agree with this due to fictional evidence from Three Worlds Collide, except that a manufactured intelligence such as an AI could be constructed without evolutionary constraints and saying that every possible descendant of a being that survived evolution MUST have a moral similarity to every other being seems like a much more complicated and less likely hypothesis.

Indeed, that was not at all what I meant.

I'd say that it's about as likely to apply to paperclippers or babyeaters as it is to us. While I think there's a non-trivial chance that such a morality exists, I can't even begin to speculate about what it might be or how it exists. There's just a lot of uncertainty and very little either evidence.

The reason I think there's a chance at all, for what it's worth, is the existence of information theory. If information is a fundamental mathematical concept, I don't think it's inconceivable that there are all kinds of mathematical laws specifically about engines of cognition. Some of which may look like things we call morality.

But most likely not.

I would assume any objectively real morality would be in some way entailed by the physical universe, and therefore in theory discoverable.

I wouldn't say that a thing existed if it could not interact in any causal way with our universe.

No. I intend to revive one. Possibly all four, if necessary. Consider it thawing technology so advanced it can revive even the pyronics crowd.

Sorry, I was letting my own uncertainty get in the way of clarity there. A stronger version of what I was trying to say would be that consciousness gives us the illusion of being in control of our actions when in fact we have no such control. Or to put it another way: we're all P-zombies with delusions of grandeur (yes, this doesn't actually make logical sense, but it works for me)

I'm not sure whether "not even wrong" calls for an upvote, does it?

Sure. Here's a version of the analogy that first got me thinking about it:

If I turn on a lamp at night, it sheds both heat and light. But I wouldn't say that the point of a lamp is to produce heat, nor that the amount of heat it does or doesn't produce is relevant to its useful light-shedding properties. In the same way, consciousness is not the point of the brain and doesn't do much for us. There's a fair amount of cogsci literature that suggests that we have little if any conscious control over our actions and reinforces this opinion. But I like feeling responsible for my actions, even if it is just an illusion, hence the low probability assignment even though it feels intuitively correct to me.

We are living in a Fun Theory universe where we find ourselves as individual or aggregate fun theoretic agents, or something else really bizarre that is not explained by naive Less Wrong rationality, such as multiversal agents playing with lots of humanity's measure.

That sounds weird to me.

For some definitions of weird, our deal (assuming it continues to completion) is enough to land this universe in the block of weird universes.

Now that we're up-voting underconfidence I changed my vote.

I almost want this reworded the opposite way for this reason, as a 40% chance that there are not better-than-placebo methods for causing significant fat loss. Even if I didn't have first and second hand examples to fall back on I don't see why there is real doubt on this question. Another more interesting variation is, does such a method exist that is practical for a large percentage of people?

Likewise. My p: 99.5%

likewise

Ah, but betting for a proposition is equivalent to betting against its opposite. Why are you so certain that there's no better-than-placebo methods for causing significant fat loss?

But If you do change your mind, please don't change the original, as then everyone's comments would be irrelevant.

I sympathize with this. But I also upvoted the original comment because of it (i.e. I also think you're underconfident).

I meant diet, exercise, and perhaps supplements; liposuction is trivially true.

This post has generated so much more controversy than I expected.

I meant exactly exercise and healthy eating! I thought people would assume I meant that. Not gastric bypass surgery, not liposuction, not starvation, not amputating limbs.

The reason it's questionable: how long can one not eat? Can most people not eat for long enough?

It has been remarked in support of that proposition that no fat people came out of Auschwitz (or Singapore, or similar episodes). But is that because they got thin, or did they die before getting thin? Has any research been done on how people of different body types respond to starvation? The full report on this experiment might address that, but the Wiki article doesn't. However, the volunteers for that experiment were "young, healthy men" volunteering as an alternative to military service, so it's unlikely that any of them were obese going in.

Coming from the angle of competition in governance, I think you might be mixing up a lot of stuff. A joint stock corporation which is sovereign is trying to compete in the wider world for customers , i.e. willing taxpayers.

If the people desire the values you have mentioned then the joint-stock government will try to provide those cost effectively.

Clean Air and Immunizations will almost certainly be on the agenda of a city government

Biodiversity will be important to a government which includes forests in its assets and wants to sustainably maintain the same.

A free marketplace of ideas, free education and social safety nets would purely be determined by the market for people. Is it an important value enough that people would not come to your country and would go to another? if it is, then the joint stock government would try to provide the same. If not, then they wouldn't.

England had property-rights based monarchy. It's basically gone now. So pace Mencius Moldbug, it can't be especially good a system - else it would not have died.

Moldbug calls this a joint-stock republic, though he mixes it with some of his more fringe ideas.

I'll second gwern's recommendation on Nick Szabo's blog - he has a post on Government for Profit, which I think was written as a rebuttal to some of Moldbug's ideas (see the comments in this post)

I think you're not drawing a clear enough distinction between two different things, namely the mathematical relationships between numbers, and the correspondence between numbers and reality.

If you ask an astronomer what is the mass of some asteroid, he will presumably give you a number with a few significant digits and and uncertainty interval. If you ask him to justify this number, he will be able to point to some observations that are incompatible with the assumption that the mass is outside this interval, which follows from a mathematical argument based on our best knowledge of physics. If you ask for more significant digits, he will say that we don't know (and that beyond a certain accuracy, the question doesn't even make sense, since it's constantly losing and gathering small bits of mass). That's what it means for a number to be rigorously justified.

But now imagine that I make an uneducated guess of how heavy this asteroid might be, based on no actual astronomical observation. I do of course know that it must be heavier than a few tons or otherwise it wouldn't be noticeable from Earth as an identifiable object, and that it must be lighter than 10^20 or so tons since that's roughly the range where smaller planets are, but it's clearly nonsensical for me to express that guess with even one digit of precision. Yet I could insist on a precise guess, and claim that it's "meaningful" in a way analogous to your above justification of subjective probability estimates, by deriving various mathematical and physical implications of this fact. If you deprecate my claim that its mass is 4.5237 x 10^15kg, then you cannot also deprecate my claim that it is a sphere of radius 1km and average density 1000kg/m^3, since the conjunction of these claims is by the sheer force of mathematics false.

Therefore, I don't see how you can argue that a number is meaningful by merely noting its relationships with other numbers that follow from pure mathematics. Or am I missing something with this analogy?

Another upvote. That's crazy talk.

komponisto:

Are you sure you're not just worried about poor calibration?

No, my objection is fundamental. I provide a brief explanation in the comment I linked to, but I'll restate it here briefly.

The problem is that the algorithms that your brain uses to perform common-sense reasoning are not transparent to your conscious mind, which has access only to their final output. This output does not provide a numerical probability estimate, but only a rough and vague feeling of certainty. Yet in most situations, the output of your common sense is all you have. There are very few interesting things you can reason about by performing mathematically rigorous probability calculations (and even when you can, you still have to use common sense to establish the correspondence between the mathematical model and reality).

Therefore, there are only two ways in which you can arrive at a numerical probability estimate for a common-sense belief:

1. Translate your vague feeling of certainly into a number in some arbitrary manner. This however makes the number a mere figure of speech, which adds absolutely nothing over the usual human vague expressions for different levels of certainty.

2. Perform some probability calculation, which however has nothing to do with how your brain actually arrived at your common-sense conclusion, and then assign the probability number produced by the former to the latter. This is clearly fallacious.

Honestly, all this seems entirely obvious to me. I would be curious to see which points in the above reasoning are supposed to be even controversial, let alone outright false.

That’s an excellent list of questions! It will help me greatly to systematize my thinking on the topic.

Before replying to the specific items you list, perhaps I should first state the general position I’m coming from, which motivates me to get into discussions of this sort. Namely, it is my firm belief that when we look at the present state of human knowledge, one of the principal sources of confusion, nonsense, and pseudosicence is physics envy, which leads people in all sorts of fields to construct nonsensical edifices of numerology and then pretend, consciously or not, that they’ve reached some sort of exact scientific insight. Therefore, I believe that whenever one encounters people talking about numbers of any sort that look even slightly suspicious, they should be considered guilty until proven otherwise -- and this entire business with subjective probability estimates for common-sense beliefs doesn’t come even close to clearing that bar for me.

Now to reply to your list.

(1) Confession of frequentism. Only sensible numerical probabilities are those related to frequencies, i.e. either frequencies of outcomes of repeated experiments, or probabilities derived from there. (Creative drawing of reference-class boundaries may be permitted.) Especially, prior probabilities are meaningless.

(2) Any sensible numbers must be produced using procedures that ultimately don't include any numerical parameters (maybe except small integers like 2,3,4). Any number which isn't a result of such a procedure is labeled arbitrary, and therefore meaningless. (Observation and measurement, of course, do count as permitted procedures. Admittedly arbitrary steps, like choosing units of measurement, are also permitted.)

My answer to (1) follows from my opinion about (2).

In my view, a number that gives any information about the real world must ultimately refer, either directly or via some calculation, to something that can be measured or counted (at least in principle, perhaps using a thought-experiment). This doesn’t mean that all sensible numbers have to be derived from concrete empirical measurements; they can also follow from common-sense insight and generalization. For example, reading about Newton’s theory leads to the common-sense insight that it’s a very close approximation of reality under certain assumptions. Now, if we look at the gravity formula F=m1*m2/r^2 (in units set so that G=1), the number 2 in the denominator is not a product of any concrete measurement, but a generalization from common sense. Yet what makes it sensible is that it ultimately refers to measurable reality via a well-defined formula: measure the force between two bodies of known masses at distance r, and you’ll get log(m1*m2/F)/log(r) = 2.

Now, what can we make out of probabilities from this viewpoint? I honestly can’t think of any sensible non-frequentist answer to this question. Subjectivist Bayesian phrases such as “the degree of belief” sound to me entirely ghostlike unless this “degree” is verifiable via some frequentist practical test, at least in principle. In this sense, I do confess frequentism. (Though I don’t wish to subscribe to all the related baggage from various controversies in statistics, much of which is frankly over my head.)

(3) Degrees of confidence shall be expressed without reflexive thinking about them. Trying to establish a fixed scale of confidence levels (like impossible - very unlikely - unlikely - possible - likely - very likely - almost certain - certain), or actively trying to compare degrees of confidence in different beliefs is cheating, since such scales can be then converted into numbers using a non-numerical procedure.

That depends on the concrete problem under consideration, and on the thinker who is considering it. The thinker’s brain produces an answer alongside a more or less fuzzy feeling of confidence, and the human language has the capacity to express these feelings with about the same level of fuziness as that signal. It can be sensible to compare intuitive confidence levels, if such comparison can be put to a practical (i.e. frequentist) test. Eight ordered intuitive levels of certainty might perhaps be too much, but with, say, four levels, I could produce four lists of predictions labeled “almost impossible,” “unlikely,” “likely,” and “almost certain,” such that common-sense would tell us that, with near-certainty, those in each subsequent list would turn out to be true in ever greater proportion.

If I wish to express these probabilities as numbers, however, this is not a legitimate step unless the resulting numbers can be justified in the sense discussed above under (1) and (2). This requires justification both in the sense of defining what aspect of reality they refer to (where frequentism seems like the only answer), and guaranteeing that they will be accurate under empirical tests. If they can be so justified, then we say that the intuitive estimate is “well-calibrated.” However, calibration is usually not possible in practice, and there are only two major exceptions.

The first possible path towards accurate calibration is when the same person performs essentially the same judgment many times, and from the past performance we extract the frequency with which their brain tends to produce the right answer. If this level of accuracy remains roughly constant in time, then it makes sense to attach it as the probability to that person’s future judgments on the topic. This approach treats the relevant operations in the brain as a black box whose behavior, being roughly constant, can be subjected to such extrapolation.

The second possible path is reached when someone has a sufficient level of insight about some problem to cross the fuzzy limit between common-sense thinking and an actual scientific model. Increasingly subtle and accurate thinking about a problem can result in the construction of a mathematical model that approximates reality well enough that when applied in a shut-up-and-calculate way, it yields probability estimates that will be subsequently vindicated empirically.

(Still, deciding whether the model is applicable in some particular situation remains a common-sense problem, and the probabilities yielded by the model do not capture this uncertainty. If a well-established physical theory, applied by competent people, says that p=0.9999 for some event, common sense tells me that I should treat this event as near-certain -- and, if repeated many times, that it will come out the unlikely way very close to one in 10,000 times. On the other hand, if p=0.9999 is produced by some suspicious model that looks like it might be a product of data-dredging rather than real insight about reality, common sense tells me that the event is not at all certain. But there is no way to capture this intuitive uncertainty with a sensible number. The probabilities coming from calibration of repeated judgment are subject to analogous unquantifiable uncertainty.)

There is also a third logical possibility, namely that some people in some situations have precise enough intuitions of certaintly that they can quantify them in an accurate way, just like some people can guess what time it is with remarkable precision without looking at the clock. But I see little evidence of this occurring in reality, and even if it does, these are very rare special cases.

(4) The question of whether somebody is well calibrated is confused for some reason. Calibrating people has no sense. Although we may take the "almost certain" statements of a person and look at how often they are true, the resulting frequency has no sense for some reason.

I disagree with this, as explained above. Calibration can be done successfully in the special cases I mentioned. However, in cases where it cannot be done, which includes the great majority of the actual beliefs and conclusions made by human brains, devising numerical probabilities makes no sense.

(5) Unlike #3, beliefs can be ordered or classified on some scale (possibly imprecisely), but assigning numerical values brings confusing connotations and should be avoided. Alternatively said, the meaning subjective probabilities is preserved after monotonous rescaling.

This should be clear from the answer to (3).

[Continued in a separate comment below due to excessive length.]

So why stop there? If you can justify 54%, then why not go further and calculate a dozen or two more significant digits, and stand behind them all with unshaken resolve?

Downvote if you agree with something, upvote if you disagree.

EDIT: I missed the word only. I just read "I think you're probably right." My mistake.

Same here. A "pretty sure" confidence level would probably have done it for me.

Boy did I hit a hornets' nest with this one!

No, of course I didn't mean anything like that. Here is how I see this situation. Silver has a model, which is ultimately a piece of mathematics telling us that some p=0.667, and for reasons of common sense, Silver believes (assuming he's being upfront with all this) that this model closely approximates reality in such a way that p can be interpreted, with reasonable accuracy, as the probability of Republicans winning a House majority this November.

Now, when you ask someone which party is likely to win this election, this person's brain will activate some algorithm that will produce an answer along with some rough level of confidence. Someone completely ignorant about politics might answer that he has no idea, and cannot say anything with any certainty. Other people will predict different results with varying (informally expressed) confidence. Silver himself, or someone else who agrees with his model, might reply that the best answer is whatever the model says (i.e. Republicans win with p=0.667), since it is completely superior to the opaque common-sense algorithms used by the brains of non-mathy political analysts. Others will have greater or lesser confidence in the accuracy of the model, and might take its results into account, with varying weight, alongside other common-sense considerations.

Ultimately, the status of this number depends on the relation between Silver's model and reality. If you believe that the model is a vast improvement over any informal common-sense considerations in predicting election results, just like Newton's theory is a vast improvement over any common-sense considerations in predicting the motions of planets, then we're not talking about a common-sense conclusion any more. On the other hand, if you believe that the model is completely out of touch with reality, then you would discard its result as noise. Finally, if you believe that it's somewhat accurate, but still not reliably superior to common sense, you might revise its conclusion using common sense.

What you believe about Silver's model, however, is still ultimately a matter of common-sense judgment, and unless you think that you have a model so good that it should be used in a shut-up-and-calculate way, your ultimate best prediction of the election results won't come with any numerical probabilities, merely a vague feeling of how confident you are.

Luke_Grecki:

Do you not think that this feeling response can be trained through calibration exercises and by making and checking predictions?

Well, sometimes frequentism can come to the rescue, in a sense. If you are repeatedly faced with an identical situation where it's necessary to make some common-sense judgment, like e.g. on an assembly line, you can look at your past performance to predict how often you'll be correct in the future. (This assuming you're not getting better or worse with time, of course.) However, what you're doing in that case is treating a part of your own brain as a black box whose behavior you're testing empirically to extrapolate a frequentist rule -- you are not performing the judgment itself as a rigorous Bayesian procedure that would give you the probability for the conclusion.

That said, it's clear that smarter and more knowledgeable people think with greater accuracy and subtlety, so that their intuitive feelings of (un)certainty are also subtler and more accurate. But there is still no magic step that will translate these feelings output by black-box circuits in their brains into numbers that could lay claim to mathematical rigor and accuracy.

Aside from some quibbles that aren't really worth getting into, I have no significant disagreement with your comments. There is nothing wrong with looking at people's acts in practice and observing that they behave as if they operated with subjective probability estimates in some range. However, your statement that "one's own probability assignments are not necessarily introspectable" basically restates my main point, which was exactly about the meaninglessness of analyzing one's own common-sense judgments to arrive at a numerical probability estimate, which many people here, in contrast, consider to be the right way to increase the accuracy of one's thinking. (Though I admit that it should probably be worded more precisely to make sure it's interpreted that way.)

The data I'm working from is that contact with certain people sometimes causes me to have mystical experiences. This has happened somewhere between 20 and 100 times, with less than a dozen people. Sometimes but not always, it happens in both directions; i.e., they also have a mystical experience as a result of the contact.

The simpler hypothesis, from a materialist point of view, is that seeing these people just tripped some switch in my brain, without any direct mind-to-mind interaction being involved. Then we can say that I also tripped such a switch in their brains in the cases where it was reciprocal. We are left with the question of why this weird psychological phenomenon happens.

The religious explanation is in many ways easier and more natural. We can say that my souls brushed up against these people's. It makes sense from within the religious frame of mind that this sort of thing would happen. But obviously we run into the issues with religious views in general.

I would expect that most simulators who worried about computational capacity wouldn't bother simulating to the depth of quantum physics anyway. However, I'm not entirely sure that I should use this sort of argument when talking about the local laws of "physics". There is some sense, I think, in which the laws of physics around here are "supposed to be" MWI-like and that we should take them at face value.

I'd be curious to know when you think the crossover point was.

probably false (10%).

That's probable in your nomenclature?

Christianity makes the world a better place, as compared to if all those people were non-believers in any religion.

I think a better question is "would the world a better place if people who are currently Christian became their next most likely alternative belief system?". I'm going to go out on a limb here and speculate that if the median Christian lost his faith he wouldn't become a rational-empiricist.

Christianity is good for its believers - they are better off to believe than to be atheist.

I'd change this one to:

• Christianity is good for most of its believers - they are better off to believe than to be atheist.

~62%

Christianity is good for Christendom - it is a positive force for majority Christian societies, as compared to if those societies were mostly atheist.

~69%

Christianity makes the world a better place, as compared to if all those people were non-believers in any religion.

~58%

Edit: I case it wasn't clear the 70% refers to the disjunction of the above 3.

Note that it is in general very hard to tell if the artistic and cultural contributions associated with religion are actually due to religion. In highly religious cultures that's often the only form of expression that one is able to get funding for. Dan Barker wrote an essay about this showing how a lot of classical composers were agnostics, atheists or deists who wrote music with religious overtones mainly because that was their only option.

Funny, I upvoted this because of the artistic and cultural contributions of religion. For most of history, until the Industrial Revolution or a little before, human economies were Malthusian. You could not increase incomes without decreasing average lifespans. The implication is that the money spent on cathedrals and gargoyles and all the rest came directly at the expense of people's lives. (A recent Steven Landsburg debate with Dinesh D'Souza explored this line of thinking more; I wouldn't recommend watching much more than the opening statements, though.)

I think the positive externalities of having more of those people's descendants alive today would be of higher value than the current benefits of past art--especially since most of that past art has been destroyed.

You sound more confident than Eugine, in which case you should upvote. Or does 70% roughly match your belief?

It could be that people are browsing the recent comments section and impulse-downvoting. :/

It's a tough question, and involves reasoning heavily about counterfactuals. What would a humanity without religion look like? I tend to think it'd look a lot better, even though I admit there's a lot of confusion in the counterfactual surgery. So I upvoted.

I was worried people would think that, but if I posted links to present evidence, I ran the risk of convincing them so they wouldn't vote it up! All I've eaten in the past three weeks is: pork belly, butter, egg yolks (and a few whites), cheese, sour cream (like a tub every three days), ground beef, bacon fat (saved from cooking bacon) and such. Now, that's no proof about the medical claim but I hope it's an indication that I'm not just bullshiting. But for a few links: http://www.ncbi.nlm.nih.gov/pubmed/19179058 (the K2 in question is virtually found only in animal fats and meats, see http://www.westonaprice.org/abcs-of-nutrition/175-x-factor-is-vitamin-k2.html#fig4)--the pubmed is on prevention of heart disease in humans http://wholehealthsource.blogspot.com/2008/11/can-vitamin-k2-reverse-arterial.html shows reversal in rat studies from K2 http://trackyourplaque.com/ -- a clinic that uses K2 among other things to reverse heart disease note that I am not trying to construct a rational argument but to convince people that I do hold this belief. I do think a rational argument can be constructed but this is not it.

Downvoted means you agree (on this thread), correct? If so, I've wanted to see a post on rationality and nutrition for a while (on the benefits of high-animal fat diet for health and the rationality lessons behind why so many demonize that and so few know it).

What evidence?

If you're referring to the Atkins diet, I think that's a rather different matter from simply eating lots of bacon fat and sour cream, which doesn't preclude also eating plenty of carbohydrates.

Or worse, it might entail eating nothing else. The post isn't very precise.

Brilliant.

It was a minority, contrarian position just a decade ago. But Wachtershauser's position is not just "metabolism first". It is also "strictly autotrophic" and "lipid first". So I think it is still fair to call it a minority opinion.

Interesting. They are actually teaching this stuff now! Was the Origins material from the textbook, or from lectures? If textbook, could you name the book?

To clarify, you do think there was an "RNA world" - but it just post-dated cell walls.

Yes, except that what it post-dated was cell membranes, not cell walls. The distinction is important. I do think that there was an "RNA world" stage in life's evolution when living cells could be modeled as "bags full of RNA". But I believe that there was an earlier stage when they could be modeled as simply "bags full of water and minerals" and an even earlier stage when life consisted of "patches of living bag material adhering to the surface of minerals".

there was no "prebiotic soup"

...sounds questionable, or at least very speculative: the first cells probably derived some nutrient value from at least one organic compound - not least because their cell [membranes] were probably composed of organic compounds.

Nope. No organic nutrients whatsoever. Autotrophic. This is the key idea that distinguishes Wachtershauser from almost everyone else. Yes, membrane materials are organic, but they were made (on site and just-in-time) by the first living membranes (on mineral surfaces).

...and of course, I am obviously not going to agree with the last line. IMO, Wachtershauser came along rather late, long after the guts of the problem were sorted out.

Of course. It amused me as I wrote my piece that you could write a strictly parallel contrarian position on the origin-of-life question. "Unless you are familiar with the work of Glascow chemist Graham Cairns-Smith, everything you have read about the origin of life on earth is wrong".

Errh. If you are disagreeing with me, doesn't that mean you should upvote?

Downvoted for the sheer number of reversals of what used to be my background assumptions about biology without an obvious identification of a single lever that could be used to push on all of those variables.

I am now interested in Wachtershauser, but it takes more than a good LW post to make me think that everything I know is wrong and that it was all disproved by the same person.

Well, he hasn't disproved anything, merely offered an alternative hypothesis. A convincing one, IMHO.

But there is a "single lever". Wachtershauser believes that the origin of life was "autotrophic". Everyone else - Miller, Orgel, Deamer, Dyson, even Morowitz on his bad days, thinks that the first living things were "heterotrophic". And since defining those two terms and explaining their significance would take more work than I want to expend right now, I'll leave the explaining to wikipedia and Google. I'll be happy to answer follow-up questions, though.

Then you should upvote, not downvote!

Unfortunately, most of Wachtershauser's papers are behind paywalls. This paper (one of his first publications) is an exception. Ignore everything beyond the first 15 pages or so.

This New York Times article is surprisingly good for pop-science journalism.

Oh. Umm... 33%!

I'm truly not joking!!! You know perfectly well that I don't share much of what's commonly known as "sanity". So to me it's worthy of totally non-ironic consideration..

To directly address your point - what I mean is if you have 1 computer that you never use, with 200MHz processor, I'd think twice about buying a 1.6GHz computer, especially if the 200MHz machine is suffering from depression due to it's feeling of low status and worthlessness.

I probably stole from The Economist too.

Yes - thank you for the cite.

Let's say 65%.

(ie. Gross overcondidence.)

I was originally going to include an additional 9, but decided I should compensate for overconfidence bias. :)

But, seriously, I don't understand why people are so reluctant to quote large probabilities. For some statements, e.g., "the sun will rise tomorrow", 99.999% seems way underconfident.

This is much less than obvious-- if the parent universe has sufficient resources, it's entirely plausible that it would include detailed simulations for fun-- art or gaming or some costly motivation that we don't have.

Basically, I fail to see why anyone would create such a detailed simulation if it bore absolutely no resemblance to reality.

I have seen simulators of Conway’s Game of Life (or similar) that contain very complex things, including an actual Turing machine.

I could see someone creating a simulator for CGL that simulates a Turing machine that simulates a universe like ours, at least as a proof of concept. With ridiculous amounts of computation available I’m quite sure they’d run the inner universe for a few billion of years.

If by accident a civilization arises in the bottom universe and they found some way of “looking above” they’d find a CGL universe before finding the one similar to theirs.

Army1987: Not sure what the rules are for comments replying to the original, but hell. Voted down for agreement.