Bayesian Probability is for things that are Space-like Separated from You

I think your links are a good indication of the way that LW has engaged with a relatively narrow aspect of this, and with a somewhat biased manner. "Crony beliefs" is a good example - starting right from the title, it sets up a dichotomy of "merit beliefs" versus "crony beliefs", with the not-particularly-subtle-connotation of "merit beliefs are this great thing that models reality and in an ideal world we'd only have merit beliefs, but in the real world, we also have to deal with the fact that it's useful to have crony beliefs for the purpose of manipulating others and securing social alliances".

Which... yes, that is one aspect of this. But the more general point of the original post is that there are a wide variety of beliefs which are underdetermined by external reality. It's not that you intentionally have fake beliefs which out of alignment with the world, it's that some beliefs are to some extent self-fulfilling, and their truth value just is whatever you decide to believe in. If your deep-level alief is that "I am confident", then you will be confident; if your deep-level alief is that "I am unconfident", then you will be that.

Another way of putting it: what is the truth value of the belief "I will go to the beach this evening"? Well, if I go to the beach this evening, then it is true; if I don't go to the beach this evening, it's false. Its truth is determined by the actions of the agent, rather than the environment.

The predictive processing thing could be said to take this even further: it hypothesizes that all action is caused by these kinds of self-fulfilling beliefs; on some level our brain believes that we'll take an action, and then it ends up fulfilling that prediction:

About a third of Surfing Uncertainty is on the motor system, it mostly didn’t seem that interesting to me, and I don’t have time to do it justice here (I might make another post on one especially interesting point). But this has been kind of ignored so far. If the brain is mostly just in the business of making predictions, what exactly is the motor system doing?

Based on a bunch of really excellent experiments that I don’t have time to describe here, Clark concludes: it’s predicting action, which causes the action to happen.

This part is almost funny. Remember, the brain really hates prediction error and does its best to minimize it. With failed predictions about eg vision, there’s not much you can do except change your models and try to predict better next time. But with predictions about proprioceptive sense data (ie your sense of where your joints are), there’s an easy way to resolve prediction error: just move your joints so they match the prediction. So (and I’m asserting this, but see Chapters 4 and 5 of the book to hear the scientific case for this position) if you want to lift your arm, your brain just predicts really really strongly that your arm has been lifted, and then lets the lower levels’ drive to minimize prediction error do the rest.

Under this model, the “prediction” of a movement isn’t just the idle thought that a movement might occur, it’s the actual motor program. This gets unpacked at all the various layers – joint sense, proprioception, the exact tension level of various muscles – and finally ends up in a particular fluid movement

Now, I've mostly been talking about cases where the truth of a belief is determined purely by our choices. But as the OP suggests, there are often complex interplays between the agent and the environment. For instance, if you believe that "I will be admitted to Example University if I study hard enough to get in", then that belief may become self-fulfilling in that it causes you to study hard enough to get in. But at the same time, you may simply not be good enough, so the truth value of this belief is determined both by whether you believe in it, and by whether you actually are good enough.

With regard to the thing about confidence; people usually aren't just confident in general, they are confident about something in particular. I'm much more confident in my ability to write on a keyboard, than I am in my ability to do brain surgery. You could say that my confidence in my ability to do X, is the probability that I assign to doing X successfully.

And it's often important that I'm not overconfident. Yes, if I'm really confident in my ability to do something, then other people will give me more respect. But the reason why they do that, is that confidence is actually a bit of a costly signal. So far I've said that an agent's decisions determine the truth-values of many beliefs, but it's also the other way around: the agent's beliefs determine the agent's actions. If I believe myself to be really good at brain surgery even when I'm not, I may be able to talk myself into a situation where I'm allowed to do brain surgery, but the result will be a dead patient. And it's not going to take many dead patients before people realize I'm a fraud and put me in prison. But if I'm completely deluded and firmly believe myself to be a master brain surgeon, that belief will cause me to continue carrying out brain surgeries, even when it would be better from a self-interested perspective to stop doing that.

So there's a complicated thing where beliefs have several effects: they determine your predictions about the world and they determine your future actions and they determine the subconscious signals that you send to others. You have an interest in being overconfident for the sake of persuading others, and for the sake of getting yourself to do things, but also in being just-appropriately-confident for the sake of being able to predict the consequences of your own future actions better.

An important framing here is "your beliefs determine your actions, so how do you get the beliefs which cause the best actions". There have been [LW · GW] some [LW · GW] posts [LW · GW] offering tools for belief-modification which had the goal of causing change, but this mostly hasn't been stated explicitly, and even some of the posts which have offered tools for this (e.g. Nate's "Dark Arts of Rationality") have still talked about it being a "Dark Art" thing which is kinda dirty to engage in. Which I think is dangerous, because getting an epistemically correct map is only half of what you need for success, with the "have beliefs which cause you to take the actions that you need to succeed" being the other half that's just as important to get right. (Except, as noted, they are not two independent things but intertwined with each other in complicated ways.)

In particular, you shouldn't force yourself to believe that you're attractive.

And I never said this.

But there's a thing that can happen when someone else gaslights you into believing that you're unattractive, which makes it true, and you might be interested in undoing that damage, for example.

It seems pretty easy for such mechanisms to be adapted for maximizing reproduction in some ancestral excitement but maladapted for maximizing your preferences in the modern environment.

I think I agree that your point is generally under-considered, especially by the sort of people who compulsively tear down Chesterton's fences.

What rossry said, but also, why do you expect to be "winning" all arms races here? Genes in other people may have led to development of meme-hacks that you don't know are actually giving someone else an edge in a zero sum game.

In particular, they might call you fat or stupid or incompetent and you might end up believing it.

This is the first thing I've read from Scott Garrabant, so "otherwise reputable" doesn't apply here. And I have frequently seen things written on LessWrong that display pretty significant misunderstandings of the philosophical basis of Bayesian probability, so that gives me a high prior to expect more of them.

Pretty confident they meant it that way:

I am not thinking about physics-time, I am thinking about logical-time. If something is in your past, but has no effect on what algorithm you are running on what observations you get, then it might as well be considered as space-like separated from you. If you compute how everything in the universe evaluates, the space-like separated things are the things that can be evaluated either before or after you, since their output does not change yours or vice-versa. If you partially observe a fact, then I want to say you can decompose that fact into the part that you observed and the part that you didn't, and say that the part you observed is in your past, while the part you didn't observe is space-like separated from you.