Against strong bayesianism

You might argue that bayesianism is conceptually useful, and thereby helps real humans reason better. But I think that concepts in bayesianism are primarily useful because they have suggestive names, which make it hard to realise how much work our intuitions are doing to translate from ideal bayesianism to our actual lives.

This reminds me of an old critique of LW Bayesianism by David Chapman, and the conclusion that we reached in the comment section of it:

The valuable part of LW, for many people, is a collection of simple, practical insights into reasoning, rather than the complex technical framework. [...] The small practical insights [...] are all excellent. [...] I’d suggest that the Bayesian framework is not necessary to understand any of them, and perhaps not helpful (except maybe for “Update Yourself Incrementally”). Maybe this depends on one’s cognitive style. For some people, understanding that all those insights loosely relate to a mathematical framework would be satisfying and helpful; for others, the framework would be difficult to understand and an unnecessary distraction.

I object to this point and would be interested to see a defense of it:

Block makes a number of arguments about the nature of comprehension and intelligence based on Blockhead - but many (including Daniel Dennett, and myself) think that these arguments are deeply flawed, and the example of Blockhead is not useful for gaining either conceptual insight or practical inspiration. Why not? Well, it’s absurdly unrealistic; you could never get anywhere near implementing it in real life.

I'll fire back with a quote from David Lewis (paraphrased, I can't find the original) "This possibility is so strange and outlandish that some people refuse to learn anything from it."

If you write out the arguments, they don't depend on Blockhead actually happening in the future, or even actually happening in any world that shares our laws of physics. As far as I can tell. So whether or not Blockhead is realistic is irrelevant.

Edge cases make bad law, but they make great mathematics. Philosophy is more like math than law. QED. (Actually, some parts of philosophy really are more like law than math. I don't think this part is, though.)

Later, you say:

More importantly, even though Blockhead gets the right answer on all the inputs we give it, it’s not doing anything remotely like thinking or reasoning.

Wasn't that exactly the point Block was trying to make with the Blockhead thought experiment? From the paper:

...two systems could have actual and potential behavior typical of familiar intelligent beings, that the two systems could be exactly alike in their actual and potential behavior, and in their behavioral dispositions and capacities and counterfactual behavioral properties (i.e., what behaviors, behavioral dispositions, and behavioral capacities they would have exhibited had their stimuli differed)--the two systems could be alike in all these ways, yet there could be a difference in the information processing that mediates their stimuli and responses that determines that one is not at all intelligent while the other is fully intelligent.

Maybe he later went on to derive other conclusions, and it is those that you object to? I haven't followed the literature as closely as I'd like.

Towards whatever-you-call-the-thing-I-got-from-reading-LW ism:

Thinking about all information systems as fundamentally performing the computation "Result <== Prior x Evidence" has been responsible for 5 of the 7 biggest successes in my career thus far. The other 2 had nothing to do with math/information/probability. All of the 5 were me noticing, where many better educated "more qualified" individuals did not, that some part of the actual information system's implementation was broken w.r.t. "Result <== Prior x Evidence" and figuring out how to phrase the brokenness in some other way ('cause inferential distance), resulting in institutional pressure to fix it.

While Bayesian statistics are obviously a useful method, I am dissatisfied with the way "Bayesianism" has become a stand-in for rationality in certain communities. There are well-developed, deep objections to this. Some of my favorite references on this topic:

• Probability Theory Does Not Extend Logic by Dave Chapman. Part of what is missing from simulating every logically possible universe is indeed reasoning, in the sense that probabilistic inference nicely extends propositional logic but cannot solve problems in first order logic. This is why practical planning algorithms also use tools like tree search.
• Philosophy and the Practice of Bayesian Statistics by Andrew Gelman (who wrote the book on Bayesian methods) and Cosma Shalizi. Abstract:

A substantial school in the philosophy of science identifies Bayesian inference with inductive inference and even rationality as such, and seems to be strengthened by the rise and practical success of Bayesian statistics. We argue that the most successful forms of Bayesian statistics do not actually support that particular philosophy but rather accord much better with sophisticated forms of hypothetico-deductivism. We examine the actual role played by prior distributions in Bayesian models, and the crucial aspects of model checking and model revision, which fall outside the scope of Bayesian confirmation theory

• Even the standard construction of probability is potentially suspect. Why is the Dutch Book argument correct? There are serious potential problems with this, as John Norton has discussed The Material Theory of Induction, which also discusses the shortcomings of Bayesianism as a foundational inference method (chapter 10).
• Bayesianism is ultimately a mathematical formalism that is actually useful only to the extent that two quantifications succeed in practice: the quantification of the state of the world into symbolic form, and the quantification of potential interventions and their results (necessary if we wish to connect reasoning to causation). There are many choices which need to be made both at the conceptual and the practical level when quantifying, as I have tried to discuss here in the context of data journalism. Quantification might be the least studied branch of statistics.

Finally, I'd note that explicitly Bayesian calculation is rarely used as the top level inference framework in practical decision-making, even when the stakes are high. I worked for a decade as a data journalist, and you'd think that if Bayesianism is useful anywhere then data journalists would use it to infer the truth of situations. But it is very rarely useful in practice. Nor is Bayesianism the primary method used in e.g. forecasting and policy evaluation. I think it's quite instructive to ask why not, and I wish there was more serious work on this topic.

In short: Bayesianism is certainly foundational, but it is not a suitable basis for a general theory of rational action. It fails on both theoretical and practical levels.

I do agree that most things people identify as tenets of bayesianism [LW · GW] are useful for thinking about knowledge; but I claim that they would be just as useful, and better-justified, if we forced each one to stand or fall on its own.

So we have this machine with a track record of cranking out really useful tools for thinking and reasoning. But it would be more useful, and better-justified, if we considered each of these tools on its own merits, rather than thinking that it's likely to be useful just because it came from the machine.

... That does seem like a very self-consistent claim for someone arguing against Bayesianism.

The Carnot engine is an abstract description of a maximally efficient heat engine. You can't make your car engine more efficient by writing thermodynamic equations on the engine casing.

The Solomonoff Inductor is an abstract description of an optimal reasoner. Memorizing the equations doesn't automagically make your reasoning better. The human brain is a kludge of non modifiable special purpose hardware. There is no clear line between entering data and changing software. Humans are capable of taking in a "rule of thumb" and making somewhat better decisions based on it. Humans can take in Occams razor, the advice to "prefer simple and mathematical hypothesis" and intermittently act on it, sometimes down-weighting a complex hypothesis when a simpler one comes to mind. Humans can sometimes produce these sort of rules of thumb from an understanding of Solomonoff Induction.

Its like looking at a book about optics doesn't automatically make your eyes better, but if you know the optics, you can sometimes work out how your vision is distorted and say "that line looks bent, but its actually straight".

If you want to try making workarounds and patches for the bug riddled mess of human cognition, knowing Solomonoff Induction is somewhat useful as a target and source of inspiration.

If you found an infinitely fast computer, Solomonoff Induction would be incredibly effective, more effective than any other algorithm.

I would expect any good AI design to tend to Solomonoff Induction (or something like it ? ) in the limit of infinite compute (and the assumption that acausal effects don't exist?) I would expect a good AI designer to know about Solomonoff Induction, in much the same way I would expect a good car engine designer to know about the Carnot engine.

There are two things that should be distinguished:

• Given one's current beliefs, how to update them given new evidence.
• How to get the machine started: how to get an initial set of beliefs on the basis of no evidence

The answer to the first is very simple: Bayes' rule. Even if you do not have numbers to plug in, there are nevertheless some principles following from Bayes' rule that can still be applied. For example, avoiding the conjunction fallacy, bottom-line reasoning, suggestive variable names, failure to entangle with reality, taking P and not-P both as evidence for a favoured idea, and so on. These are all written of in the Sequences.

Answering the second generally leads to speculation about Solomonoff priors, the maxent principle, improper priors, and so on, and I am not sure has contributed much. But the same is true of every other attempt to find an ultimate foundation for knowledge. A rock cannot discover Bayes' rule, and what is the tabula rasa of an agent but a rock?

Another direction which has so far led only to more philosophical floundering is trying to apply probabilistic reasoning at the meta-level: what is the probability that our methods of logical reasoning are sound? Whatever conclusions we come to about that, what is the probability that those conclusions are true? And so on. Nothing good has yet come of this. It is a standard observation that despite all the work that has been done on non-standard logic, at the meta-level everyone reverts to good old standard logic. The only innovation that is used at the meta-level, because it was there all along, is constructiveness. When a mathematician proves something, he actually exhibits a proof, not a non-constructive argument that there exists a proof. But Euclid did that already.

I enjoyed this post. I think the dialogue in particular nicely highlights how underdetermined the phrase 'becoming more Bayesian' is, and that we need more research on what optimal reasoning in more computationally realistic environments would look like.

However, I think there are other (not explicitly stated) ways I think Bayesianism is helpful for actual human reasoners. I'll list two:

• I think the ingredients you get from Bayes' theorem offer a helpful way of making more precise what updating should look like. Almost everyone will agree that we should take into account new evidence, but I think explicitly bearing in mind 'okay, what's the prior?', and 'how likely is the evidence given the hypothesis?', offers a helpful framework which allows us to update on new evidence in a way that's more likely to make us calibrated.
• Moreover, even thinking in terms of degrees of belief as subjective probabilities at all (and not just how to update them) is a pretty novel conceptual insight. I've spent plenty of time speaking to people with advanced degrees in philosophy, many of whom think by default in terms of disbelief/full belief, and don't have a conception of anything like the framework of subjective probabilities.

Perhaps you agree with what I said above. But I think such points are worth stating explicitly, given that I think they're pretty unfamiliar to most people, and constitute ways in which the Bayesian framework has generated novel insights about good epistemic behaviour.

When you have a simple description doing a lot of work and then add more description instead of piling on complexity it can be seen as making implicit claims more explicit. One is in effect "filling out the details".

Taking things to the extreme might pinpoint where the nebolous parts need further clarification. As laid out a bayesian deals in probabilities even in the limits which makes it kind of like emplying the most useful atomic heuristics. The lookup table can be thought as being the go-to answer for "has optimal action" and bayesian as "has integrated all information" with everything else set at a token level.

A more real thought process might have the property that there are information that is not integrated into the decision but it also seems that being blind to too much central information would be grounds to judge a thought process as deficient. The principle that allows to delineate some info as remote and some as central could point to a different maxim. It feels like the bayesian is more advanced as the lookup table despite the lookup table being more solid and unhesitant. So it would be interesting why it would be okay or move things forward to neglect remote information while it still impacts probabilities.

There's a joke that theists like to tell. An atheist challenges God, saying, whatever you can do I can do also! God says, "I created Man from the dust." So the atheist says, "Ok," and gathers up some dust to work with. God replies, "No, no, first you must create your own dust."

Here's a rationalist version.

A rationalist challenges God, saying, whatever you can do I can do also! God says, "I made the dust evolve into thinking beings! You can have some dust this time." The rationalist says, "Ok," and devises a machine to start from first principles and which, set running in the universe, will update itself until it develops into an AGI. "No, no," says God, "first you must have the dust find those first principles."

An interesting read and I'm happy to see people publishing posts like this.

The more that beliefs are questioned, the more improvements in thinking can be made. (or something to that effect, I'm struggling with the wording).

I feel like a lot of Bob's responses are natural consequences of Eliezer's position that you describe as "strong bayesianism", except where he talks about what he actually recommends, and as such this post feels very uncompelling to me. Where they aren't, "strong bayesianism" is correct: it seems useful for someone to actually think about what the likelihood ratio of "a random thought popped into my head" is, and similarly about how likely skeptical hypotheses are.

Similarly,

In other words, an ideal bayesian is not thinking in any reasonable sense of the word - instead, it’s simulating every logically possible universe. By default, we should not expect to learn much about thinking based on analysing a different type of operation that just happens to look the same in the infinite limit.

seems like it just isn't an argument against

Whatever approximation you use, it works to the extent that it approximates the ideal Bayesian calculation - and fails to the extent that it departs.

(and also I dispute the gatekeeping around the term 'thinking': when I simulate future worlds, that sure feels like thinking to me! but this is less important)

In general, I feel like I must be missing some aspect of your world-view that underlies this, because I'm seeing almost no connection between your arguments and the thesis you're putting forwards.

Are you against Bayesianism or 'Bayesianism'?

I do agree that most things people identify as tenets of bayesianism [LW · GW] are useful for thinking about knowledge; but I claim that they would be just as useful, and better-justified, if we forced each one to stand or fall on its own.

This makes me think that you're (mostly) arguing against 'Bayesianism', i.e. effectively requesting that we 'taboo' that term and discuss its components ("tenets") separately.

One motivation for defending Bayesianism itself is that the relevant ideas ("tenets") are sufficiently entangled that they can or should be considered effectively inseparable.

I also have a sense that the particular means by which intelligent entities like ourselves can, incrementally, approach thinking like an 'idealized Bayesianism intelligence' is very different than what you sketched in your dialog. I think a part of that is something like maintaining a 'network' of priors and performing (approximate) Bayesian updates on specific 'modules' in that network and, more infrequently, propagating updates thru (some portion of) the network. Because of that, I didn't think this last part of the dialog was warranted:

A: So why do people advocate for the importance of bayesianism for thinking about complex issues if it only works in examples where all the variables are well-defined and have very simple relationships?

B: I think bayesianism has definitely made a substantial contribution to philosophy. It tells us what it even means to assign a probability to an event, and cuts through a lot of metaphysical bullshit.

In my own reasoning, and what I consider to be the best reasoning I've heard or read, about the COVID-19 pandemic, Bayesianism seems invaluable. And most of the value is in explicitly considering both evidence and the lack of evidence, how it should be interpreted based on (reasonably) explicit prior beliefs within some specific 'belief module', and what updates to other belief modules in the network are warranted. One could certainly do all of that without explicitly believing that Bayesianism is overall effective, but it also seems like a weird 'epistemological move' to make.

If you agree that most of the tenets of a big idea are useful (or true) in what important sense is it useful to say you're against the big idea? Certainly any individual tenet can be more or less useful or true, but in helping one stand or fall on its own, when are you sharpening the big idea versus tearing it down?

By default, we should not expect to learn much about [X] based on analysing a different type of operation that just happens to look the same in the infinite limit.

Yeah, that's a good point. We should probably stop using linear approximations; we can't really expect to learn much about a function's behavior by analyzing a different function which just happens to look the same in the infinitesimal limit. And big-O analysis will have to go as well; we should not expect to learn much about an algorithm's runtime by comparing it to a different operation that just happens to look the same in the limit of large inputs. And while we're talking computer science, we should definitely stop pretending that floating-point operations tell us anything about operations with real numbers, just because they happen to agree in the limit of infinite precision. Thermodynamics can go; just because we know what happens in the limit of infinitely many microscopic particles doesn't mean we know anything useful about finite systems. For that matter, forget Bayesianism, why bother even with frequentist statistics? It's not like we'd expect to learn anything useful about a bunch of data samples just from some idealized distribution which happens to look similar in the infinite limit.

Additional context: ricraz and I had a related discussion about idealized models of instrumental convergence [LW(p) · GW(p)].

To be honest, I don't really know what "Bayesianism as a conceptual framework" is, or if it even exists at all. Bayes theorem is an equation, and Bayesianism to the rest of the world is a formalization of probability based on Bayes theorem. It's certainly not this comical strawman of a hypothetical general AI that doesn't understand that its prediction algorithms take time to run.

Could you say a bit more on why you think we should quantify the accuracy of credences with a strictly proper scoring rule, without reference to optimality proofs? I was personally confused about what principled reasons we had to think only strictly proper scoring rules were the only legitimate measures of accuracy, until I read Levinstein's paper offering a pragmatic vindication for such rules.

B: I think bayesianism has definitely made a substantial contribution to philosophy. It tells us what it even means to assign a probability to an event, and cuts through a lot of metaphysical bullshit

It formalises the idea of subjective probability whilst saying about the nature or existence of objective probability.

I agree with the rejection of strong Bayesianism. I don’t think it follows from what you’ve written, though, that “bayesianism is not very useful as a conceptual framework for thinking either about AGI or human reasoning”.

I'm probably just echoing things that have been said many times before, but:

You seem to set up a dichotomy between two uses of Bayesianism: modeling agents as doing something like "approximate Solomonoff induction", and Bayesianism as just another tool in our statistical toolkit. But there is a third use of Bayesianism, the way that sophisticated economists and political scientists use it: as a useful fiction for modeling agents who try to make good decisions in light of their beliefs and preferences. I’d guess that this is useful for AI, too. These will be really complicated systems and we don’t know much about their details yet, but it will plausibly be reasonable to model them as “trying to make good decisions in light of their beliefs and preferences”. In turn, the Bayesian framework plausibly allows us to see failure modes that are common to many boundedly rational agents.

Perhaps a fourth use is that we might actively want to try to make our systems more like Bayesian reasoners, at least in some cases. For instance, I mostly think about failure modes in multi-agent systems. I want AIs to compromise with each other instead of fighting. I’d feel much more optimistic about this if the AIs could say “these are our preferences encoded as utility functions, these are our beliefs encoded as priors, so here is the optimal bargain for us given some formal notion of fairness” --- rather than hoping that compromise is a robust emergent property of their training.

I've been looking for a word for the community of people who are good at identifying precise, robust, extremely clarifying conceptual frameworks. It seems like a very tight cluster that will grow increasingly defined and self-actualised. "Bayesian" seemed like the best fit for a name. Would you object to that?