The Way You Go Depends A Good Deal On Where You Want To Get: FEP minimizes surprise about actions using preferences about the future as *evidence*

post by Christopher King (christopher-king) · 2025-04-27T21:55:16.114Z · LW · GW · 1 comments

Contents

  Free Energy and the Dark Room problem
  You can sense your actions now, and that is the only sense you can change
  You choose the least surprising action
  P is well-calibrated
None
1 comment

The motivation for this post came when I was reading various Scott Alexander posts on the Free Energy Principle, in which he seemed very confused about it (the title God Help Us, Let’s Try To Understand Friston On Free Energy [LW · GW] might have hinted at that). I was intrigued but also very confused, so I fell down a rabbit hole, so to speak, trying to figure it out. I am very grateful to Active Inference: The Free Energy Principle in Mind, Brain, and Behavior, which was published after Scott's post. This open access book has this amusing section in the preface:

A Note from Karl Friston
I have a confession to make. I did not write much of this book. Or, more precisely, I was not allowed to. This book’s agenda calls for a crisp and clear writing style that is beyond me. Although I was allowed to slip in a few of my favorite words, what follows is a testament to Thomas and Giovanni, their deep understanding of the issues at hand, and, importantly, their theory of mind—in all senses.

Free Energy and the Dark Room problem

The Free Energy principle asserts that biological systems (such as trees and the human brain) have beliefs and take actions that minimize a quantity known as Variational Free Energy. Variational Free Energy is a function of beliefs (Q) and sense data (y), where sense data comes from the external world (x) and from actions.

$F(Q,y)=D_{KL}\left(Q\right(x)\parallel P(x|y\left)\right)-\ln P\left(y\right)$

$Q\left(x\right)$ is meant to be the system's attempt at approximating $P\left(x|y\right)$. For simplicity, for most of this post I'll assume that the system can do perfect Bayesian inference, so that $Q\left(x\right)=P\left(x|y\right)$, implying that $F(Q,y)=-\ln P\left(y\right)$. In other words, the system is trying to minimize how surprising the sense data is.

An apparent paradox is called the Dark Room problem. A dark room has very little suprise by virtue of having very little information. So why don't all organisms hang out in them? According to Translating Predictive Coding Into Perceptual Control

The main proposed solution is to claim you have some built-in predictions (of eg light, social interaction, activity levels), and the dark room will violate those.

Friston's explanation is similar, if a bit wordy.

The free-energy principle says that we harvest sensory signals that we can predict (cf., emulation theory; Grush, 2004); ensuring we keep to well-trodden paths in the space of all the physical and physiological variables that underwrite our existence.

I am pretty sure these resolutions are don't actually resolve the problem. While technically true, they miss the core problem.

The actual resolution is this: deciding to be in a dark room minimizes future free energy, but FEP actually says that actions are minimizing the present free energy.

In other words, there has been substantial confusion about what exactly is being optimized with respect to what variables.

You can sense your actions now, and that is the only sense you can change

Usually when we think of senses, we mean sensing the external world, such as by using sight and sound and what not. However, in the Free Energy literature, this definition is typically somewhat expanded: actions also count as sense data. In this way, we can factor $P\left(y\right)$ as $P\left(\text{actions}|\text{external senses}\right)P\left(\text{external senses}\right)$.

This is why minimizing the present free energy is even possible. No action you can take (such as entering a dark room) will immediately register in your vision or other external senses. Only your actions can immediately enter the sense data while solving this optimization problem.

You choose the least surprising action

‘The rule is, jam to-morrow and jam yesterday—but never jam to-day.’

— Lewis Carroll, "Through the Looking-Glass"

Consider a choice between the dark room on the left and a room with delicious food on the right. The problem is, you don't know what food! The door is currently closed. But you do know the food is good. In particular, you know it is more likely that you will go into the food room than the dark room (because obviously you like food better than darkness).

Consider these priors:

$P\left(\text{Go left into the dark room}\right)=\frac{1}{3}$$P\left(\text{Go right into the food room and it has steak}\right)=\frac{1}{6}$$P\left(\text{Go right into the food room and it has cotton candy}\right)=\frac{1}{6}$$P\left(\text{Go right into the food room and it has pizza}\right)=\frac{1}{6}$$P\left(\text{Go right into the food room and it has jam}\right)=\frac{1}{6}$

So, sadly, if you go into the food room, you will have more free energy than if you went to into the dark room. Do we need a fudge factor? (Would adding fudge into the mix make things worse?)

Nope. You choose the action that is currently least surprising according to your own model and preferences. Minimizing surprise now means choosing the action $a$ that maximizes $P\left(a|\text{external senses}\right)$, or equivalently, minimizes $-\ln P\left(a|\text{external senses}\right)$. Since $P\left(\text{Go right}\right)=2/3>P\left(\text{Go left}\right)=1/3$, "Go right" is the less surprising action choice right now. The potential future sensory surprise inside the food room is not the deciding factor for the current action.

We can even add random sensory information (like random numbers) to the right room. This will increase its future free energy, but not the present free energy of choosing right.

Likewise, if you end up in the dark room, the least surprising action is to leave immediately. The fact that you will be surprised once you leave is irrelevant.

This is also why the "humans don't like the dark" explanation is technically correct. If humans did really like darkness, P(Go left) might be higher than P(Go right). The reason it is a bad explanation is that is missed the point about the predictability of the dark room being irrelevant for the current action choice. Random sensory info might contribute billions of bits to the free energy! I am sure that Friston understands this, but I think he didn't realize that others missed that point!

P is well-calibrated

Here is another apparent problem:

So for example, suppose it’s freezing cold out, and this makes you unhappy, and so you try to go inside to get warm. FE/PC would describe this as “You naturally predict that you will be a comfortable temperature, so the cold registers as strong prediction error, so in order to minimize prediction error you go inside and get warm.” PCT would say “Your temperature set point is fixed at ‘comfortable’, the cold marks a wide deviation from your temperature set point, so in order to get closer to your set point, you go inside”.

The PCT version makes more sense to me here because the phrase “you naturally predict that you will be a comfortable temperature” doesn’t match any reasonable meaning of “predict”.

This again mixes up the present and the future, but in the other direction. When you predict you will be a comfortable temperature, that is a belief about the future. All predictions are beliefs about the future.

But didn't I just say that the Free Energy Principle is about the present? The trick is that learning about the future requires you to do a Bayesian update on your predicted actions in the present. This is known as planning-as-inference. In fact, by dropping the first term in free energy, this entire post has secretly been about planning-as-inference.

So why do people say that free energy is minimizing prediction errors? Because your actions now affect your predictions about the future.

$P(\text{comfy later}|\text{cold now},\text{start moving inside})>P(\text{comfy later}|\text{cold now},\text{do nothing})$

Thus, our prior preference/prediction to be comfortable soon makes the action "start moving inside now" the most probable (least surprising) action to infer in the present. Starting to move inside now reduces our prediction error about the future, and Bayes favors hypotheses that reduce prediction errors.

And in fact, if we decide to go inside, all our beliefs turn out correct! We are cold now, we decided to go inside, and we will be comfy later. So $P$ is a well-calibrated self-fulfilling prophecy.

Same thing for the motor cortex. How ought your muscles to move? To quote the cheshire cat, "that depends a good deal on where you want to get to." It is not telling you where your muscles are now, it is fulfilling predictions about where your muscles will be a couple fractions of a second from now (which just so happens to usually be close to where they are now).

Since $P$ is making sensible predictions, I think “you naturally predict that you will be a comfortable temperature” is completely reasonable!

Note that $P$ is occasionally wrong tho. For example, yesterday (true story) when I was chewing food, I thought I had partially swallowed it, when in fact I had swallowed all of it. In other words, I had a false belief that I had food in my mouth! Did that cause me to resolve the error by putting food in my mouth? No, the most likely action was to try chewing the food, which I did, causing my teeth to clack when no food was there (and presumably causing many prediction errors in my motor cortex). When I discovered the error, was my response to put food in my mouth? No, because that prediction error was now a few moments in the past. My beliefs about the past updated, but my decisions in the present would only influence the future probability, not the past probability, of having food in my mouth.

1 comments

Comments sorted by top scores.