On Having No Clue

Complexity heuristics possibly work to some extent here. If A is a hypothesis of comparable complexity to B, then you at least have some basis for assigning them comparable prior probabilities.

If A can be conveniently split into A1 and A2, then the child hypotheses are often more complex than A in that they each require some additional property on top of A. This isn't always the case, but in the exceptional cases it's likely more useful to consider one or both of the child hypotheses to be "primary" instead of A.

This is all very rough, since complexity is usually only defined up to some constant and is relative to some sort of specification model, but it's something that isn't necessarily contingent on observations of the world. If you're an entity that is considering probabilities and hypotheses at all, the concept of measuring complexity is likely already accessible to you.

If you have two options, A and B, 50% odds is maximal ignorance; you aren't saying they have equivalent odds of being true, you're saying you have no information by which to make an inference which is true.

If you then say we can split A into A1 and A2, you have added information to the problem.  Like the Monty Hall problem, information can change the odds in unexpected ways!

There's no contradiction here - you have more information than when you originally assigned odds of 50/50.  And the information you have added should, in real situations, inform how to distribute the odds.  If A1 and A2 are sufficiently distinct (independent), it is possible that a 33/33/33 split is appropriate; if they aren't, it is possible that a 25/25/50 split is appropriate.  In order to make a judgment, we'd have to know more about what exactly A1 and A2 are, and why they can be considered a "split" of A.

Consider, for example, the case of a coin being flipped - we don't know if the coin is fair or not.  Let us say A is that the coin comes up "heads" and B is that the coin comes up "tails".  The split, then, could reflect a second flip, after the first flip is decided, if and only if it is heads; A1 might be "heads-heads", A2 might be "heads-tails".  Then a 25/25/50 split makes sense; A1 and A2 are not independent.

If, on the other hand, we have discovered that it isn't a coin at all, but a three-sided die, two faces of which have the same symbol, and one face of which has another symbol; we label the faces with the similar symbol A1 and A2, and the face with the same edit: other symbol B.  We still don't know whether or not the die is fair - maybe it is weighed - but the position of maximal ignorance is 33/33/33, because even if it -is- weighted, we don't know which face it is weighted in favor of; A1 and A2 are independent.

So - what are A1 and A2, and how independent are they?  We have equations that can work this out with sample data, and your prior probability should reflect your expectation of their independence.  If you insist on maximal ignorance about independence - then you can assume independence.  Most things are independent; it is only the way the problem is constructed that leads us to confusion here, because it seems to suggest that they are not independent (consider that we can simply rename the set of conclusions to "A, B, and C" - all the names you have utilized are merely labels, after all, and in effect, what you have actually done is to introduce C, with an implication that A and C should maybe be considered partially dependent variables).  If you insist on maximal ignorance about that, as well, then you can, I suppose, assume 50% independence, which would be something like splitting the difference between the die and the coin.  And there's an argument to be made there, in that you have, in fact, implied that they should maybe be considered partially dependent variables - but this comes down to trying to interpret what you have said, rather than trying to understand the nature of probability itself.

Agreed, great post. But I think you are trying to push Bayesian Statistics past what it SHOULD be used for. 

Bayesian Statistics are only useful because we approach the correct answer as we gain all the information possible. Only in this limit (of infinite information) is Bayesian useful. Priors based off no information are, well, useless.

Scenario 1: You flip a fair coin and have a 50/50 chance of it landing heads

Scenario 2: (to steal xepo's example) are bloxors greeblic? You have NO IDEA, so your priors are 50/50

Even though in both scenarios the chances are 50/50, I would feel much more confident betting money on scenario 1 than scenario 2. Therefore my model of choices contains something MORE than probabilities. As far as I know Bayesian statistics just doesn't convey this NEEDED information. You cant use Bayesian probabilities here in a useful way. It's the wrong tool for the job. 

Even frequentest statistics is useless here. 

A lot of day-to-day decisions are based off very limited information. I am not able to lay out a TRUE model of how we intuitively make those decisions but "how much information I have to work with" is definitely an aspect in my mental model that is not entirely captured by Bayes Theorem. 

So the one we know the most about gets the heavier cumulative weight, because it has more sub-classes in our reasoning, because we know the most about it.

I suspect that without a solid base case, this process of over-weighting familiar options could be weaponized to form an argument against pursuing novel ideas about familiar topics. If I have "no clue" about, say, how a new model of the universe would compare to the old models, I could split "possibilities from old models" into so many fragments that the probability of the new model, weighted equally to all the tiny slivers of the familiar old model, approaches 0.

It seems like preventing this may require that the things being compared seem of similar size, but if you can guess the size of something, you're no longer entirely clueless about it.

It seems like the problem might be in the assumption of having "no clue". I think someone truly clueless about a question would be unable to divide it into parts in order to compare the parts' probabilities. I imagine being asked an advanced question about the grammar or meaning of a passage in a language that I can neither speak nor read, and I would be unable to even formulate a question to assign probabilities to.

I claim the problem is that our model is insufficient to capture our true beliefs.

There’s a difference in how we act between a coin flip (true 50/50) and “are bloxors greeblic?” (a question we have no info about).

For example, if our friend came and said “Yes, i know this one, the answer is (heads|yes)”. For coin flip you’d say “are you out of your mind?” and for bloxors you’d say “Ok, sure, you know better than me”

I’ve been idly pondering over this since Scott Alexander’s post. What is a better model?

One option would be to have another percentage — a meta-percentage. e.g. “What credence do i give to “this is an accurate model of the world””? For coin flips, you’re 99.999% that 50% is a good model. For bloxors, you’re ~0% that 50% is a good model.

I don’t love it, but it’s better than presuming anything on the base level, i think.