Cognitive Instability, Physicalism, and Free Will

Start with being an embedded agent in a universe that is partially predictable from the inside, that agent has an internal model of the universe in order to survive. Some parts of this model are necessary, others are spurious. The belief in free will is an artifact of an introspecting subsystem colliding with an opaque decision-making subsystem. Knowledge is not "justified true belief" or whatever the heck, but an abstraction for that part of the internal model of the universe which is reasonably accurate for the purpose of making decisions. There are other parts of your internal model, the spurious ones, which do not reflect the universe. These give you the musings like "what if my brain is an algorithm that produces 1+1=3". 

There is nothing more to it.

I think that Carroll's argument is wrong because it already assumes as true what it wants to disprove. 

His argument is the following claim: If I am BB, I can't reason about physics and the probabilities of being BB. However, this claim is true only if I am BB. But if I am interested only in the question "Am I BB?", it doesn't matter how good I am thinking of being a BB or everything else, because the only thing which I am interested in is already assumed as true. 

The same logic was used in the attempts to disprove the simulation argument by Danila Medvedev (and maybe others). If I am in a simulation, I can't have a perfect model of the distribution of simulations in the universe and - Danila suggests, – it disproves the simulation argument. 

Let's elaborate: One of two statements is true:

1. I am real and simulations are impossible.
2. I am in simulation, but in that case, I can't perform perfect probability estimations.

The italic part doesn't have any logical power over the whole argument.

I don't know if you're intentionally recapitulating this line of argument, but C.S. Lewis makes this argument in Miracles. There's a long history of the back and forth on wikipedia

I don't think it works, mostly because the fact that a belief is result of a physical process doesn't tell my anything at all about the rationality / irrationality of belief. Different physical processes should be judged differently; some are entangled with the resulting state of belief and others aren't.

But this means your belief is merely a product of this physical system called a person, by its internal dynamics and interactions with the environment. The feeling of thinking based on reasoning and rationality has no bearing on the outcome. Then how can you be certain that, this particular physical system happens to believe in the correct theories about the universe? Including theories like "I have no free will"? 

This seems to contradict itself. Before, you said that """free will""" was defined as decisions that cannot be explained by physical theories. But such """free""" decisions would not involve reasoning and rationality, as reasoning and rationality requires a certain amount of order similar to what is sufficient for physical and computational theories. So it is precisely if decisions are based on non-"""free""" factors that you can come to believe in correct things about the universe.

I am a human.  I believe that only a minuscule percentage of all humans who have ever lived are capable of reasoning about physics concerning environments in which evolution has not directly equipped them to innately understand.  If I think that I am one of the few people who actually can reason about physics outside of such environments, don't I also have to think that I am probably mistaken?  If not, than if I think I am a  Boltzmann brain and accept that most Boltzmann brains can't properly reason, can't I think that I am an exception? 

But this means your belief is merely a product of this physical system called a person, by its internal dynamics and interactions with the environment. The feeling of thinking based on reasoning and rationality has no bearing on the outcome. Then how can you be certain that, this particular physical system happens to believe in the correct theories about the universe?

You cant, but you can't under any alternative, either. You could argue that you can have certainty about your own qualia , if you accept qualia, but that still doesn't mean you have certainty about reasoning processes.

Instead, maybe we should treat physics as inherently perspective-dependent. Among other benefits, it would not have this conflict and be stable.

I don't see why.

Then how can you be certain that, this particular physical system happens to believe in the correct theories about the universe? Including theories like "I have no free will"?

"Certain to be correct?" No. All you can hope for is to be Less Wrong, hence the name of this site.

How you get there in a materialistic view is fairly straightforward: by the same type of process operating on different timescales and levels of abstraction.

Firstly by evolution of nervous systems and brains, encoding behaviour that match better what the organisms need to do in the world to survive. Secondly, with more powerful brains enabling complex behaviour to be learned, subsequent evolution of acquired behaviour. Thirdly with more powerful brains still, cultural evolution affecting patterns of thought and formation of systems of mass dissemination of ideas. Fourthly, a process of evolution via science of more detailed and explicit models of how things work.

By this stage of the process you should expect the models that survive to be more accurate, not because they are willed to be so but because the patterns of thought that produce such models enable success in the physical world for the organisms using them. It is of course possible that they're complete bunk that happens to work anyway, in which case too bad for us. What we can be pretty sure of though is that they're better than literally random noise.