Unknown unknowns

Posts linked to from Logical rudeness wiki page seem relevant, including Katja Grace's Estimation is the best we have.

One should always use the best tools available, even if they are no good or don't satisfy given standards of rigor. Declaring "I don't know" is also such a tool, but is it the best one available? Ironically, the problem seems to be partially with "I don't know" acting as a curiosity stopper, prompting one to stop thinking where a bit more thought could otherwise lead to better decisions.

Here's something I just posted elsewhere (in a debate concerning cryonics!) relating to this:

Sure, there are many people who know the same amount as I do and reach different numbers. But that's hardly a unique failing of this approach: equally-knowledgable people disagree about all kinds of issues all the time, even when they don't try to put their intuitions into numbers.

No, there isn't any well-established, objective approach for deriving the numbers. But that's beside the point. The difference between just throwing around verbal arguments and writing down probability estimates is like the difference in basing judgements on vague intuitions in your head and explicitly writing out the pros and cons. Putting numbers on things not only helps clarify the exact degree to which you disagree with someone else, it also forces you to be more explicit in your reasoning.

My above comment is a good example: Aleksander challenged me on two points, which led me to 1) consciously realize that I'd been basing one of my figures on two disjoint assumptions, helping me clarify my reasons for why I believed that 2) do some more research on another figure, leading me to data that made me revise my estimate to one tenth of what it previously was. If we'd been just throwing around verbal arguments and vague appeals to intuitions like we'd been doing before, I'm not sure whether either one of those would have happened. So no, putting explicit numbers on things doesn't mean we'll ever reach the correct conclusion, but it does help work out the exact points of disagreement and maybe make the estimates converge at least a bit.

It's also important to realize that not putting numbers on things doesn't mean we're not still pulling numbers out of thin air! Your brain is still doing some sort of implicit probability estimate. And while one could reasonably argue that trying to put numbers into our intuitions loses important data (as we don't have introspective access to all our thought processes), there are also some pretty convincing lines of argument suggesting that consciously-held information has evolved as much to appear good and persuade others as it has to actually evaluate the truthfulness of things. So a refusal to put explicit numbers on things seems to me suspicious, as it seems like the kind of a trick one that'd be useful in hiding inconsistencies in one's arguments.

Note however that I'm most definitely not accusing anyone of intentional dishonesty, or anything along those lines. The issue is not in people being consciously deceitful. The issue is in people's minds being built in such a way as to trick their consciousness into making estimates based on something else than truth, and then having reasonable-seeming intuitions that happen to lead to a "cover-up" for the flimsiness of the reasoning. I'm just as skeptical about my own conscious thought processes as I am of those of others (or at least I try to be), which is part of the reason why I'm so eager to put down my own probability estimates for others to critique.

The following is an idea that's been nagging at me for a while, and I finally have it clear enough in my mind to at least try to state it. Any feedback would be highly appreciated (especially if what I say is confusing!).

I think there are cases where you shouldn't assign a probability to your beliefs. Most Bayesian updating is a form of computation, and you need to assign a probability to that computation being a reasonable thing to update on. Unless I have confidence in the procedure that I'm using to update my beliefs, I shouldn't take the number I get at the end as something to update my beliefs to...yes, I should update my beliefs towards that number, but possibly only a very tiny amount.

Now here is the problem you run into. When I start out, I probably haven't even chosen a prior. Choosing that prior is itself a computation that I have to make. If I have a low degree of confidence in the reasonability of that computation, then what am I supposed to do? It seems silly to take the result of that computation as my prior, as the result is probably meaningless. I basically want to update by a small amount away from "nothing", but "nothing" isn't a probabilitym so it's quite unclear what to do in probabilistic terms. (Note that once we start to have higher confidence in our calculations, then we can essentially update away from the "nothing" state by saying that almost any possible prior would have led to something close to our current belief.)

In discussions with a friend, who expressed great discomfort in talking about cryonics, I finally extracted the confession that he had no emotional or social basis for considering cryonics. None of his friends or family had done it, it was not part of any of the accepted rituals that he had grown up with -- there was an emotional void around it that placed it outside of the range of options that he was able to think about. It was "other", alien, of such a nature that merely rational evaluation could not be applied.

He's in his 70's, so this issue is more than just academic. He understands that by rejecting cryonics he is embracing his own death. He does not believe in an afterlife. He becomes emotionally perturbed when I discuss cryonics precisely because I am persuasive about its technical feasibility.

Perhaps this observation isn't germane to the present thread, as this seems an emotional response rather than a response driven by "no belief." But perhaps "no belief" has an emotional component, as in "I don't want to have a belief. If I had a belief, then I'd have to take an unpleasant action."

to me "no belief" often means that people have a lot of uncertainty about their probability estimates. I think that this uncertainty can be best expressed by asking people to make a market on their probability estimate, rather than just specifying a single number. So, for instance if you asked me to make a market in the probability that a fair coin toss will come up heads, I will be like 49@51, (I'll buy 49% and I'll sell 51% because I am very confident that the true value is 50% and so I know I am getting some edge on that one). If you ask me the probability that someone currently being stored at alcor will be brought back to life at some point in the future, I have very little confidence about how to estimate that, and so I'll be something like .0001 bid @ 85 offer.

If you take your beliefs seriously you should be willing to bet on them. If you are willing to bet on them they should be 2-sided markets and not single numbers because you should not be willing to take either side of a bet with the same odds, even a coin flip, because you have 0 EV at best. Once you consider credit risk, transaction costs, adverse selection, etc. then you are definitely -EV unless you include a bid/ask spread.

About two...

There is no jump, because "I don't know" is the maximum entropy distribution. The maximum entropy distribution is the distribution over probabilities which creates the maximum information-theoretic entropy, while obeying the observed parameters of the system. This works because entropy is just the expected value of the information gained from measuring a system. You want the maximum entropy distribution because anything else is literally pulling information out of thin air. If you pick a lower entropy distribution when you can construct a higher entropy one consistent with the data, then you're expecting less information to be given on a measurement, as if you already knew something about it.

The maximum entropy hypothesis on any yes/no question is a 50/50 chance. At those odds, cryonics are great!

However, they probably have information which adjusts their probability down. An actual "I don't know" would be the result of a coinflip, whereas anything under than a 50% probability of cryonics working is based on information which makes you think it's unlikely. So they have beliefs about it.

The "no belief" option seems at least likely to solve the "Pascal's Mugging" (http://lesswrong.com/lw/kd/pascals_mugging_tiny_probabilities_of_vast/) problem, by allowing you to ignore very low probability outcomes.

You just make them bet. That reveals that they do have probability estimates.

Thanks cipergoth, for raising this fundamental issue. I'll try to defend the "no belief" approach, since I still consider it possibly correct. However, it should be noted that other options include credence intervals, for example "somewhere between almost-certain and certain, inclusive".

On the first argument - you have to make a decision, but it needn't literally be a calculated decision.

On the second, I would suggest a model of thought which has more continuity. In between "no belief" and a precise numerical probability could lie various qualitative assessments of the evidence for and against. On this model, the precise probabilities that a speaker might avow for certain select bets are good-enough approximations to a belief-state that may not quite fully live up to that precision. The exact numerical probabilities are used because expected utility calculations are a convenient approach to certain decisions. The jump from qualitative to numerical probabilities is made when the perceived advantages of expected utility calculations justify it - and perhaps the jump is more verbal than real.

Teddy Seidenfeld has a critique of maximum-entropy priors which, to my admittedly ill-trained eye, looks like a serious problem. I would love to believe that every probability question has an objective answer. But I don't, at least not yet.

This seems like basically the same mistake that was described in "But There's Still A Chance, Right?" except on the opposite end of the probability scale.