Thought experiments on simplicity in logical probability

I do not know of any proposed prior that says that "simple sentences are more likely to be true."

The prior I proposed says that I care more about being correct on simple sentences. This translates better to "I expect that my answer to simple questions is more likely to matter"

The prior Paul Christiano proposed tries to minimize a "weighted" entropy, and so translates as "try harder to say neutral on the simple sentences"

The prior Abram Demski proposed (or at least the minor modification to it that he takes more seriously where you are just as likely to put the negation of a sentence in as the sentence itself) translates as "truth values of simple sentence are more likely to modify truth values of complicated sentences than vice versa."

The reason these look like "simple sentences are more likely to be true" is because you are looking at a list of mutually exclusive sentences, and so pushing them towards 1/2 is generally increasing their probability. If instead, you had a list of sentences and knew that exactly 5 out of the 7 were true, the simple ones would look less likely.

Note 1: I have recently become convinced that Abram's prior is likely a better proposal than my own.

Note 2: I haven't posted this yet, but my proposal is actually three different definitions, which are (conjecturally) the same, and one of them is better described similarly to Abram's in that it lets simple sentences change complex sentences more than vice versa.

I don't believe that complex mathematical statements should have lower probability than simple statements. It's not just about A vs not-A. If A is a conjunction of many parts, its probability should be low. If A is a disjunction of many parts, its probability should be high.

The part about "mutually exclusive and exhaustive set of sentences" is a little similar to the idea from an old post of mine. Namely, if you have some set of sentences S_1 ... S_n, then for each i you can define the probability that a random proof, sampled from a length-based prior, happens to prove S_i, conditional on the fact that it proves one of S_1 ... S_n. The probabilities defined this way always sum to 1, provided that at least one S_i is true. Such an approach is helpful when S_i are logical counterfactuals, which look "mutually exclusive" to the agent but are actually all true. Also note that it agrees with our intuition about conjunctions and disjunctions, because proofs of disjunctions will tend to be shorter.

With regards to thought experiment 2, I think a useful reference point is completely random strings of digits. If you were indeed picking the strings of digits totally at random, you would expect a distribution in which:
1) Any string of digits is equally likely.
2) A significant majority of the strings has complexity of roughly m (plus some constant overhead).

Now, in the actual thought experiment, the string of digits is quite obviously not random; as you've said, the Kolmogorov complexity has a hard upper bound of C + log(m) + log(n), which in general could be significantly smaller than m.

Here's one alternative hypothesis that I can think of, which makes sense to me but could easily be wrong:
H2: In general, such strings will appear to be mostly random, except for the simple fact that they actually occur relatively early in pi.
If this is true, one would expect that for quite a lot of those strings there will not be a more compact representation than a program that, in one way or another, explicitly computes those digits of pi.

Also, I think I've thought of an interesting way of testing a prediction along the lines you've suggested. If we take S(n, m, pi) to mean "the m digits following the nth digit of pi", let's define
N(s, pi) := the smallest integer n such that S(n, len(s), pi) = s

If, as you suggest, simpler strings should be "more likely", then it would follow that for simpler strings s, you would expect to see lower values of N(s, pi). In particular, how about the following experiments:
i) Generate a bunch of random sequences R (all of length m) and look at the distribution you get for N(r, pi). You can also use your apparent complexity checker on those sequences, but in general you would expect most of them to have complexity of roughly m.
ii) Pick out a bunch of strings Q that are demonstrably simple, e.g. 0000000000, 1111111111, 0123456789, and look at the values you get for N(q, pi)
iii) Pick a different number, which you think has similar properties to pi in this regard - let's say e. Now, for a given number n, you can calculate
F(n) := N(S(n, m, e), pi)
which basically means "for a sequence occurring at position n in e, what position does it first occur at in pi"?

I think if your hypothesis is true, it will hold that the values for (ii) are typically much smaller than the values in (i), and the values for F(n) in (iii) will generally tend to increase as n increases (up to a finite limit, because eventually you will have covered all sequences of length m). On the other hand, if H2 is correct, the values for (i), (ii), and (iii) will all be pretty similar.

Now, with all that being said, I'm not sure that your thought experiment #2 carries across to a prior as to whether or not certain logical statements are true. It may be an interesting experiment to actually try in practice, though.

The second complication is in what exactly a simplicity prior is supposed to look like. In the case of Solomonoff induction the shape is exponential - more complicated hypotheses are exponentially less likely. But why not a power law? Why not even a Poisson distribution?

More complicated hypotheses have to be on average at least exponentially less likely, because there are exponentially many of them. There are probability measures that decline with length faster than exponential, but none that are slower. One could even say that the Solomonoff probabilities decline as slowly as possible.