Advancing Certainty

I'd like to recast the problem this way: we know we're running on error-prone hardware, but standard probability theory assumes that we're running on errorless hardware, and seems to fail, at least in some situations, when running on error-prone hardware. What is the right probability theory and/or decision theory for running on error-prone hardware?

ETA: Consider ciphergoth's example:

do you think you could make a million statements along the lines of "I will not win the lottery" and not be wrong once? If not, you can't justify not playing the lottery, can you?

This kind of reasoning can be derived from standard probability theory and would work fine on someone running errorless hardware. But it doesn't work for us.

We need to investigate this problem systematically, and not just make arguments about whether we're too confident or not confident enough, trying to push the public consensus back and forth. The right answer might be completely different, like perhaps we need different kinds or multiple levels of confidence, or upper and lower bounds on probability estimates.

One simple example: do you think you could make a million statements along the lines of "I will not win the lottery" and not be wrong once? If not, you can't justify not playing the lottery, can you?

P(A) = P(X) * P(A|X) + P(~X) * P(A|~X)

E(A) = p * E(a) + (1-p) * x

P(L) = p * P_calculated + (1-p) * P_typical

You propose to ignore the "odd" errors humans sometimes make while calculating a probability for some event. However, errors do occur, even when judging the very first case. And they (at least some of them) occur randomly. When you believe you have correctly calculated the probability, you just might have made an error anywhere in the calculation.

If you keep around the "socially accepted" levels of confidence, those errors average out pretty fast, but if you make only one error in 10^5 calculations, you should not assign probabilities smaller than 1 / 10^5. Otherwise a bet 10000 to 1 between you and me (a fair game from your perspective) will give me an expected value larger than 0 due to the errors in your thoughts you could possibly make.

This is another advantage an AI might have over humans, if the hardware is good enough, probability assignments below 10^-5 might actually be reasonable.

Now, if it is the case that she didn't, then it follows that, given sufficient information about how-the-world-is, one's probability estimate could be made arbitrarily close to 0.

What, like 1/3^^^3? There isn't that much information in the universe, and come to think, I'm not sure I can conceive of any stream of evidence which would drive the probability that low in the Knox case, because there are complicated hypotheses much less complicated than that in which you're in a computer simulation expressly created for the purpose of deluding you about the Amanda Knox case.

The fundamental question of rationality is: why do you believe what you believe? As a rationalist, you can't just pull probabilities out of your rear end. And now here's the kicker: that includes the probability of your model being wrong. The latter must, paradoxically but necessarily, be part of your model itself. If you're uncertain, there has to be a reason you're uncertain; if you expect to change your mind later, you should go ahead and change your mind now.

You're just telling people to pull different probabilities out of their rear end. Framing the other guy's model with 'rear endeness' doesn't make your model any less so. Your model must include information about the part of the universe that is komponisto and more generally about human psychology. Your model appears to make poor predictions about the likelyhood that human beliefs are well founded and so, were it convenient, I would bet against such predictions.

It may be complicated and my model is certainly not detailed but nor is it especially vaguer than it should be given the information I have available.

What are the odds that, given that I didn't make a mistake pressing the buttons, that my electronic calculator (which appears to be in proper working order) will give a wrong answer on a basic arithmetic problem that it should be able to solve?

Without, ultimately, trusting science more than intuition, there's no hope of making epistemic progress.

That's not really true. 10,000 hours of deliberate practice at making predictions about a given field will improve the intuition by a lot. Intuition isn't fixed.

Summary: P(I'm right| my theory is good / I'm not having a brain fart) can be much more extreme than P(I'm right). Over time, we can become more confident in our theories.

"Generalizing from One Example" and "Reference Class of the Unreferenceclassable" links are both broken.

In accordance with a suggestion that turned out to be rather good, I've now deleted several paragraphs of unnecessary material from the post. (The contents of these paragraphs, having to do with previous posts of mine, were proving to be a distraction; luckily the post still flows without them.)

Perhaps too much is being made of the "arbitrarily close to zero" remark. Sure, there isn't enough information in the universe to get to one-over-number-made-with-up-arrow-notation. But there's certainly enough to get to, say, one in a thousand, or one in a million; and while this isn't arbitrarily close to zero in a mathematical sense, it's good enough for merely human purposes. And what's worse, the word 'arbitrary' is distracting from the actual argument being made, in what looks to me like a rather classic case of "Is that your real objection?"

A probability estimate is not a measure of "confidence" in some psychological sense.

This is one of the possible interpretations of probability. To say that this interpretation is wrong requires an argument, not simply you saying that your interpretation is the correct one.

As we acquire more and more information about the world, our Bayesian probabilities should become more and more confident.

How do we know that we are acquiring more real information? The number of open questions in science grows. It doesn't shrink.

I think it's fair to compare the LHC with past scientific experiments but if you do you should remember that no past scientific experiment destroyed the world and therefore you don't get a prior probability greater than 0 by that process.

The LHC even didn't worked the first time around. You could say the predictions of how the LHC was supposed to work were wrong. There however millions of different ways that the LHC can turn out results that aren't what anybody expects that don't include the LHC blowing up the planet.

There have been a number of posts recently on the topic of beliefs, and how fragile they can be. They would benefit A LOT by a link to Making Beliefs Pay Rent

When you say Amanda Knox either killed her roommate, or she didn't, you've moved from a universe of rational beliefs to that of human-responsibility models. It's very unclear (to me) what experience you're predicting with "killed her roomate". This confusion, not any handling of evidence or bayesean updates, explains a large divergence in estimates that people give. They're giving estimates for different experiences, not different estimates of the same experience.

How do you get from "uncertainty exists in the map, not the territory" to the following ?

given sufficient information about how-the-world-is, one's probability estimate could be made arbitrarily close to 0

One's uncertainty about the here-and-now, perhaps. In criminal cases we are dealing with backward inference, and information is getting erased all the time. Right about now perhaps the only way you could get "arbitrarily close to 0" is by scanning Knox's brain or the actual perpetrator's brain; if both should die, we would reach a state of absolute uncertainty about the event, all the evidence you could in principle examine to reach certainty will have been rearranged into unintelligible patterns.

Similarly, I can't shake the notion that the physical distance between you and the evidence should constrain how much strength you think the evidence has, just as much as the temporal distance eventually must.

I could be wrong about that, and I have already come around to your point of view somewhat, but you seem to have a blind spot about this particular objection: you're just some guy who got his information about the case from the Internet (which is fine, so is almost everyone).

Your comparison between the Amanda Knox case and scientific knowledge leaves me cold. Science is concerned with regularities, situations where induction applies; the knowledge sought in a criminal case is of a completely different kind, by definition applying to a unique and hopefully irregular situation.

Yes, I agree that improved "epistemic technology" is grounds for more confidence, even in cases such as this one; but your argument would be improved by throttling back the eloquence - you have at times sounded like a defence lawyer - and focusing more on the concrete details of your argument.

You'd do more to convince me if you observed, for instance, how the Web has allowed you access to multiple reports about the case, improving your chances that the biases in each source would cancel each other out, and the facts that remain are (as you claim) basically all there is to know. Pre-Internet you would have had to rely on one or two "official" news sources. That is also part of "epistemic technology".

I now suspect that misleading imagery may be at work here. A million statements -- that sounds like a lot, doesn't it? If you made one such pronouncement every ten seconds, a million of them would require you to spend months doing nothing but pontificating, with no eating, sleeping, or bathroom breaks. Boy, that would be tiring, wouldn't it? At some point, surely, your exhausted brain would slip up and make an error. In fact, it would surely make more than one -- in which case, poof!, there goes your calibration.

That would be misleading imagery. I don't think that is a fair representation of the imagery you quote. It sounds more like a description a debater would use when trying to make a position sound bad. Fatigue doesn't come into it. That would be silly.

This is a side-point, perhaps, but something to take into account with assigning probabilities is that while Amanda Knox is not guilty, she is certainly a liar.

When confronting someone known to be lying during something as high stakes as a murder trial, people assign them a much higher probability of guilt, because someone that lies during a murder trial is actually more likely to have committed murder. That seems to be useful evidence when we are assigning numerical probabilities, but it was a horrific bias for the judge and jury of the case.

Edit: To orthnormal, yes, that is what I meant, thank you. I also agree that it's possible that her being a sociopath and/or not neurotypical confused the prosecutor.