[SEQ RERUN] Science Doesn't Trust Your Rationality

Science is built around the assumption that you're too stupid and self-deceiving to just use Solomonoff induction. After all, if it was that simple, we wouldn't need a social process of science... right?

Seeing how often overconfidence bias is brought up as a problem, and how the rationality camps etc. are implemented to battle this bias, among others, this assumption doesn't seem to be a bad starting point.

In the beginning came the idea that we can't just toss out Aristotle's armchair reasoning and replace it with different armchair reasoning. We need to talk to Nature, and actually listen to what It says in reply. This, itself, was a stroke of genius.

If you do a probability-theoretic calculation correctly, you're going to get the rational answer.

How does one make sure that this "probability-theoretic calculation" is not a "different armchair reasoning"?

Science doesn't trust your rationality, and it doesn't rely on your ability to use probability theory as the arbiter of truth. It wants you to set up a definitive experiment. [...] Science is built around the assumption that you're too stupid and self-deceiving to just use Solomonoff induction.

This seems like a safe assumption. On the other hand, trusting in your powers of Solomonoff induction and Bayesianism doesn't seem like one: what if you suck at estimating priors and too unimaginative to account for all the likely alternatives?

So, are you going to believe in faster-than-light quantum "collapse" fairies after all? Or do you think you're smarter than that?

Again a straw-collapse. No one believes in faster-than-light quantum "collapse", except for maybe some philosophers of physics.

For those interested in the current state of the Born rule research, there is a review in the latest Foundations of Physics.

A note of warning: this journal is heavily skewed towards philosophy and sometimes publishes complete crankery. Even its new chief editor, a Nobel prize winner Gerard 't Hooft, writes stuff like this on occasion.

I think Yudkowsky's analysis here isn't putting enough weight on the social aspects. "Science", as we know it, is a social process, in a way that Bayesian reasoning is not.

The point of science isn't to convince yourself -- it's to convince an audience of skeptical experts.

A large group of people, with different backgrounds, experiences, etc aren't going to agree on their priors. As a result, there won't be any one probability on a given idea. Different readers will have different background knowledge, and that can make a given hypothesis seem more or less believable.

(This isn't avoidable, even in principle. The Solomonoff prior of an idea is not uniquely defined, since encodings of ideas aren't unique. You and the reviewers are not necessarily wrong in putting different priors on an idea even if you are both using a Solomonoff prior. The problem wouldn't go away, even if you and the reviewers did have identical knowledge, which you don't.)

Yudkowsky is right that this makes science much more cautious in updating than a pure Bayesian. But I think that's desirable in practice. There is a lot of value to having a scientific community all use the same theoretical language and have the same set of canonical examples. It's expensive (in both human time and money) to retrain a lot of people. Societies cannot change their minds as quickly or easily as the members can, so it makes sense to move more slowly if the previous theory is still useful.

I disagree with his statements on the effects of state power. Regulation seems to work well enough over here; I don't know from where he takes the unsourced assumption that it doesn't.

I disagree with the quoted part of the post. Science doesn't reject your bayesian conclusion (provided it is rational), it's simply unsatisfied by the fact that it's a probabilistic conclusion. That is, probabilistic conclusions are never knowledge of truth. They are estimations of the likelihood of truth. Science will look at your bayesian conclusion and say "99% confident? That's good!, but lets gather more data and raise the bar to 99.9%!). Science is the constant pursuit of knowledge. It will never reach it it, but it will demand we never stop trying to get closer.

Beyond that, I think in a great many cases (not all) there are also some inherent problems in using explicit bayesian (or otherwise) reasoning for models of reality because we simply have no idea what the space of hypotheses could be. As is such, the best bayesian can ever do in this context is give an ordering of models (e.g., this model is better than this model), not definitive probabilities. This doesn't mean science rejects correct bayesian reasoning for the reason previously stated, but it would mean that you can't get definitive probabilistic conclusions with bayesian reasoning in the first place for many contexts.

Libertarianism secretly relies on most individuals being prosocial enough to tip at a restaurant they won't ever visit again.

Libertarianism might rely on individuals generally being that prosocial, but that specific thing isn't necessary. Most jobs don't get tips. There's no reason waiters need them.