Against Occam's Razor

I mentioned to Eliezer once a few years ago that a weak form of Occam's razor, across a countable hypothesis space, is inevitable. This observation was new to him so it seems worth reproducing here.

Suppose $P\left(i\right)$ is any prior whatsoever over a countable hypothesis space, for example the space of Turing machines. The condition that ${\sum }_{i}P\left(i\right)=1$, and in particular the condition that this sum converges, implies that for every $ϵ>0$ we can find $N$ such that ${\sum }_{i\ge N}P\left(i\right)<ϵ$; in other words, the total probability mass of hypotheses with sufficiently large indices gets arbitrarily small. If the indices index hypotheses by increasing complexity, this implies that the total probability mass of sufficiently complicated hypotheses gets arbitrarily small, no matter what the prior is.

The real kicker is that "complexity" can mean absolutely anything in the argument above; that is, the indexing can be arbitrary and the argument will still apply. And it sort of doesn't matter; any indexing has the property that it will eventually exhaust all of the sufficiently "simple" hypotheses, according to any other definition of "simplicity," because there aren't enough "simple" hypotheses to go around, and so must eventually have the property that the hypotheses being indexed get more and more "complicated," whatever that means.

So, roughly speaking, weak forms of Occam's razor are inevitable because there just aren't as many "simple" hypotheses as "complicated" ones, whatever "simple" and "complicated" mean, so "complicated" hypotheses just can't have that much probability mass individually. (And in turn the asymmetry between simple and complicated is that simplicity is bounded but complexity isn't.)

There's also an anthropic argument for stronger forms of Occam's razor that I think was featured in a recentish post: worlds in which Occam's razor doesn't work are worlds in which intelligent life probably couldn't have evolved.

Doesn't the speed prior diverge quite rapidly from the universal prior? There are many short programs of length _n_ which take a long time to compute their final result - up to BB(_n_) timesteps, specifically...

How do you define "works well" for a prior? I argue that for most things, the universal prior (everything is equally likely) works about as well as the lazy prior or Occam's prior, because _all_ non-extreme priors are overwhelmed with evidence (including evidence from other agents) very rapidly. all three fail in the tails, but do just fine for the majority of uses.

Now if you talk about a measure of model simplicity and likelihood to apply to novel situations, rather than probability of prediciton, then it's not clear that universal is usable, but it's also not clear that lazy is better or worse than Occam.

"Why should the Occamian prior work so well in the real world?"

A different way of phrasing Occam's Razor is "Given several competing models of reality, the most likely one is the one that involves multiplying together the fewest probabilities." That's because each additional level of complexity is adding another probability that needs to be multiplied together. It's a simple result of probability.

I have a feeling that you mix probability and decision theory. Given some observations, there are two separate questions when considering possible explanations / models:

1. What probability to assign to each model?

2. Which model to use?

Now, our toy-model of perfect rationality would use some prior, e.g. the bit-counting universal/kolmogorov/occam one, and bayesian update to answer (1), i.e. compute the posterior distribution. Then, it would weight these models by "convenience of working with them", which goes into our expected utility maximization for answering (2), since we only have finite computational resources after all. In many cases we will be willing to work with known wrong-but-pretty-good models like Newtonian gravity, just because they are so much more convenient and good enough.

I have a feeling that you correctly intuit that convenience should enter the question which model to adopt, but misattribute this into the probability-- but which model to adopt should formally be bayesian update + utility maximization (taking convenience and bounded computational resources into account), and definitely not "Bayesian update only", which leads you to the (imho questionable) conclusion that the universal / kolmogorov / occam prior is flawed for computing probability.

On the other hand, you are right that the above toy model of perfect rationality is computationally bad: Computing the posterior distribution after some prior and then weighting by utility/convenience is of stupid if directly computing prior * convenience is cheaper than computing prior and convenience separately and then multiplying. More generally, probability is a nice concept for human minds to reason about reasoning, but we ultimately care about decision theory only.

Always combining probability and utility might be a more correct model, but it is often conceptually more complex to my mind, which is why I don't try to always adopt it ;)

Why should the Occamian prior work so well in the real world? It's a seemingly profound mystery that is asking to be dissolved.

Is it? It seems a rather straightforward consequence of how knowledge works, as in information allows you to establish probabilistic beliefs and then probability theory explains pretty simply what Occam's Razor is.