God Help Us, Let’s Try To Understand Friston On Free Energy

(Posting here rather than SSC because I wrote the whole comment in markdown before remembering that SSC doesn't support it).

We had a guest lecture from Friston last year and I cornered him afterwards to try to get some enlightenment (notes here). I also spent the next few days working through the literature, using a multi-armed bandit bandit as a concrete problem (notes here ).

Very few of the papers have concrete examples. Those that do often skip important parts of the math and use inconsistent/ambiguous notation. He doesn't seem to have released any of the code for his game-playing examples.

The various papers don't all even implement the same model - the free energy principle seems to be more a design principle than a specific model.

The wikipedia page doesn't explain much but at least uses consistent and reasonable notation.

Reinforcement learning or active inference has most of a worked model, and is the closest I've found to explaining how utility functions get encoded into meta-priors. It also contains:

When friends and colleagues first come across this conclusion, they invariably respond with; “but that means I should just close my eyes or head for a dark room and stay there”. In one sense this is absolutely right; and is a nice description of going to bed. However, this can only be sustained for a limited amount of time, because the world does not support, in the language of dynamical systems, stable fixed-point attractors. At some point you will experience surprising states (e.g., dehydration or hypoglycaemia). More formally, itinerant dynamics in the environment preclude simple solutions to avoiding surprise; the best one can do is to minimise surprise in the face of stochastic and chaotic sensory perturbations. In short, a necessary condition for an agent to exist is that it adopts a policy that minimizes surprise.

I am leaning towards 'the emperor has no clothes'. In support of this:

• Friston doesn't explain things well, but nobody else seems to have produced an accessible worked example either, even though many people claim to understand the theory and think that is important.
• Nobody seems to have has used this to solve any novel problems, or even to solve well-understood trivial problems.
• I can't find any good mappings/comparisons to existing models. Are there priors that cannot be represented as utility functions, or vice versa? What explore/exploit tradeoffs do free-energy models lead to, or can they encode any given tradeoff?

At this point I'm unwilling to invest any further effort into the area, but I could be re-interested if someone were to produce a python notebook or similar with a working solution for some standard problem (eg multi-armed bandit).

Hi,

I now work in a lab allied to both the Friston branch of neuroscience, and the probabilistic modeling branch of computational cognitive science, so I now feel even more arrogant enough to comment fluently.

I’m gonna leave a bunch of comments over the day as I get the spare time to actually respond coherently to stuff.

The first thing is that we have to situate Friston’s work in its appropriate context of Marr’s Three Levels of cognitive analysis: computational (what’s the target?), algorithmic (how do we want to hit it?), and implementational (how do we make neural hardware do it?).

Friston’s work largely takes place at the algorithmic and implementational levels. He’s answering How questions, and then claiming that they answer the What questions. This is rather like unto, as often mentioned, formulating Hamiltonian Mechanics and saying, “I’m solved physics by pointing out that you can write any physical system in terms of differential equations for its conserved quantities.” Well, now you have to actually write out a real physical system in those terms, don’t you? What you’ve invented is a rigorous language for talking about the things you aim to explain.

The free-energy principle should be thought of like the “supervised loss principle”: it just specifies what computational proxy you’re using for your real goal. It’s as rigorous as using probabilistic programming to model the mind (caveat: one of my advisers is a probabilistic programming expert).

Now, my seminar is about to start soon, so I’ll try to type up a really short step-by-step of how we get to active inference. Let’s assume the example where I want to eat my nice slice of pizza, and I’ll try to type something up about goals/motivations later on. Suffice to say, since “free-energy minimization” is like “supervised loss minimization” or “reward maximization”, it’s meaningless to say that motivation is specified in free-energy terms. Of course it can be: that’s a mathematical tautology. Any bounded utility/reward/cost function can be expressed as a probability, and therefore a free-energy — this is the Complete Class Theorem Friston always cites, and you can make it constructive using the Boltzmann Distribution (the simplest exponential family) for energy functions.

1) Firstly, free-energy is just the negative of the Evidence Lower Bound (ELBO) usually maximized in variational inference. You take a $P$ (a model of the world whose posterior you want to approximate), and a $Q$ (a model that approximates it), and you optimize the variational parameters (the parameters with no priors or conditional densities) of $Q$ by maximizing the ELBO, to get a good approximation to $P\left(H|D\right)$ (probability of hypotheses, given data). This is normal and understandable and those of us who aren’t Friston do it all the time.

2) Now you add some variables to $P$: the body’s proprioceptive states, its sense of where your bones are and what your muscles are doing. You add a $P(D^{\prime }=bonesandmuscles)$, with some conditional $P\left(D|D^{\prime }\right)$ to show how other senses depend on body position. This is already really helpful for pure prediction, because it helps you factor out random noise or physical forces acting on your body from your sensory predictions to arrive at a coherent picture of the world outside your body. You now have $P\left(D|D^{\prime }\right)P\left(D^{\prime }|H\right)$.

3) For having new variables in the posterior, $P(H|D^{\prime },D)$, you now need some new variables in $Q$. Here’s where we get the interesting insight of active inference: if the old $P\left(H|sensoryD\right)$ was approximated as $Q(stuffH;sensoryD)$, we can now expand to $Q(stuffH;sensoryD,motorM)$. Instead of inferring a parameter that approximates the proprioceptive state, we infer a parameter that can “compromise” with it: the actual body moves to accommodate $M$ as much as possible, while $M$ also adjusts itself to kinda suit what the body actually did.

Here’s the part where I’m really simplifying what stuff does, to use more of a planning as inference explanation than “pure” active inference. I could talk about “pure” active inference, but it’s too fucking complicated and badly-written to get a useful intuition. Friston’s “pure” active inference papers often give models that would have very different empirical content from each-other, but which all get optimized using variational inference, so he kinda pretends they’re all the same. Unfortunately, this is something most people in neuroscience or cognitive science do to simplify models enough to fit one experiment well, instead of having to invent a cognitive architecture that might fit all experiments badly.

4) So now, if I set a goal by clamping some variables in $P(goalstuffH=pizza)$ (or by imposing “goal” priors on them, clamping them to within some range of values with noise), I can’t really just optimize $Q\left(stuffH\right)$ to fit the new clamped model. $Q\left(stuffH\right)$ is really $Q(stuffH;sensoryD,motorM)$, and $Q\left(sensoryD\right)$ has to approximate $P\left(sensoryD\right)$. Instead, I can only optimize $Q(motorM|goalstuffH=pizza)$ to fit $P(bodyposition{D}^{\prime }|goalstuffH=pizza)$. Actually doing so reaches a “Bayes-optimal” compromise between my current bodily state and really moving. Once $Q$ already carries a good dynamical model (through time) of how my body and senses move (trajectories through time), changing $M$ as a function of time lets me move as I please, even assuming my actual movements may be noisy with respect to my motor commands.

That’s really all “active inference” is: variational inference with body position as a generative parameter, and motor commands as the variational parameter approximating it. You set motor commands to get the body position you want, then body position changes noisily based on motor commands. This keeps getting done until the ELBO is maximized/free-energy minimized, and now I’m eating the pizza (as a process over time).

Ok, now a post on motivation, affect, and emotion: attempting to explain sex, money, and pizza. Then I’ll try a post on some of my own theories/ideas regarding some stuff. Together, I’m hoping these two posts address the Dark Room Problem in a sufficient way. HEY SCOTT, you’ll want to read this, because I’m going to link a paper giving a better explanation of depression than I think Friston posits.

The following ideas come from one of my advisers who studies emotion. I may bungle it, because our class on the embodied neuroscience of this stuff hasn’t gotten too far.

The core of “emotion” is really this thing we call core affect, and it’s actually the core job of the brain, any biological brain, at all. This is: regulate the states of the internal organs (particularly the sympathetic and parasympathetic nervous systems) to keep the viscera functioning well and the organism “doing its job” (survival and reproduction).

What is “its job”? Well, that’s where we actually get programmed-in, innate “priors” that express goals. Her idea is, evolution endows organisms with some nice idea of what internal organ states are good, in terms of valence (goodness/badness) and arousal (preparedness for action or inaction, potentially: emphasis on the sympathetic or parasympathetic nervous system’s regulatory functions). You can think of arousal and sympathetic/parasympathetic as composing a spectrum between the counterposed poles of “fight or flight” and “rest, digest, reproduce”. Spending time in an arousal state affects your internal physiology, so it then affects valence. We now get one of the really useful, interesting empirical predictions to fall right out: young and healthy people like spending time in high-arousal states, while older or less healthy people prefer low-arousal states. That is, even provided you’re in a pleasurable state, young people will prefer more active pleasures (sports, video gaming, sex) while old people will prefer passive pleasures (sitting on the porch with a drink yelling at children). Since this is all physiology, basically everything impacts it: what you eat, how you socialize, how often you mate.

The brain is thus a specialized organ with a specific job: to proactively, predictively regulate those internal states (allostasis), because reactively regulating them (homeostasis) doesn’t work as well). Note that the brain how has its own metabolic demands and arousal/relaxation spectrum, giving rise to bounded rationality in the brain’s Bayesian modeling and feelings like boredom or mental tiredness. The brain’s regulation of the internal organs proceeds via closed-loop predictive control, which can be made really accurate and computationally efficient. We observe anatomically that the interoceptive (internal perception) and visceromotor (exactly what it says on the tin) networks in the brain are at the “core”, seemingly at the “highest level” of the predictive model, and basically control almost everything else in the name of keeping your physiology in the states prescribed as positive by evolution as useful proxies for survival and reproduction.

Get this wrong, however, and the brain-body system can wind up in an accidental positive feedback that moves it over to a new equilibrium of consistently negative valence with either consistent high arousal (anxiety) or consistent low arousal (depression). Depression and anxiety thus result from the brain continually getting the impression that the body is in shitty, low-energy, low-activity states, and then sending internal motor commands designed to correct the problem, which actually, due to brain miscalibration, make it worse. You sleep too much, you eat too much or too little, you don’t go outside, you misattribute negative valence to your friends when it’s actually your job, etc. Things like a healthy diet, exercise, and sunlight can try to bring the body closer to genuinely optimal physiological states, which helps it yell at the brain that actually you’re healthy now and it should stuff fucking shit up by misallocating physiological resources.

“Emotions” wind up being something vaguely like your “mood” (your core affect system’s assessment of your internal physiology’s valence and arousal) combined with a causal “appraisal” done by the brain using sensory data, combined with a physiological and external plan of action issued by the brain.

You’re not motivated to sit in a Dark Room because the “predictions” that your motor systems care about are internal, physiological hyperparameters which can only be revised to a very limited extent, or which can be interpreted as some form of reinforcement signalling. You go into a Dark Room and your external (exteroceptive, in neuro-speak) senses have really low surprise, but your internal senses and internal motor systems are yelling that your organs say shit’s fucked up. Since your organs say shit’s fucked up, “surprise” is now very high, and you need to go change your external sensory and motor variables to deal with that shit.

Note that you can sometimes seek out calming, boring external sensory states, because your brain has demanded a lot from your metabolism and physiology lately, so it’s “out of energy” and you need to “relax your mind”.

Pizza becomes positively valenced when you are hungry, especially if you’re low on fats and glucose. Sex becomes most salient when your parasympathetic nervous system is dominant: your body believes that it’s safe, and the resources available for action can now be devoted to reproduction over survival.

Note that the actual physiological details here could, once again, be very crude approximations of the truth or straight-up wrong, because our class just hasn’t gotten far enough to really hammer everything in.

Scott writes on tumblr:

I don’t think I even understand the most basic point about how a probability distribution equals a utility function. What’s the probability distribution equal to “maximize paperclips”? Is it “state of the world with lots of paperclips - 100%, state of the world with no paperclips, 0%”? How do you assign probability to states of the world with 5, 10, or 200 paperclips?

I know nothing about this discussion, but this one is easy:

The utility function U(w) corresponds to the distribution $P\left(w\right)\propto \exp \left(U\right(w\left)\right)$.

(i.e. $P\left(w\right)=\exp \left(U\right(w\left)\right)/Z$, where Z is a meaningless number we choose to make the total probability add up to 1.)

Without math: every time you add one paperclip to a possible world, you make it 10% more likely. On this perspective, there is a difference between kind of wanting paperclips and really wanting paperclips--if you really want paperclips, adding one paperclip to the world makes it twice as likely. This determines how you trade off paperclips vs. other kinds of surprise.

Maximizing expected log probability under this distribution is exactly the same as maximizing the expectation of U.

You can combine the $\exp \left(U\right)$ term with other facts you know about the world $w$, by multiplying them (and then adjusting the normalization constant $Z$ appropriately).

A very similar formulation is often used in inverse reinforcement learning (MaxEnt IRL).

if the free energy principle were right, would it disprove the orthogonality thesis?

As far as I can tell, it would not - unless you think that the determinism of physics also disproves the orthogonality thesis (because if the world is deterministic, then you can't get every possible motivation, right? Just the ones that actually happen in the world).

Free energy explains behaviours and their opposites - it explains why someone punches someone or refrains from doing it, eats sushi or hamburger or soylent for lunch or skips it entirely, does/doesn't, wants/doesn't want, stays/leaves... builds paperclips/doesn't build paperclips...

This doesn't mean that free energy is vacuous, any more than sometimes predicting sunshine and sometimes predicting snow makes weather prediction vacuous. It means that weather prediction/free energy need some other set of inputs to predict an action. In the case of weather prediction, this is things like pressure, wind speed, satellite imagery, etc... In the case of free energy, it's less clear what the other inputs are, but motivation and preferences seem perfectly valid inputs.

(for the Bayesian version of Free Energy, the evidence and the priors can serve as the - variable - inputs)

Ok, now the post where I go into my own theory on how to avoid the Dark Room Problem, even without physiological goals.

The brain isn’t just configured to learn any old predictive or causal model of the world. It has to learn the distal causes of its sensory stimuli: the ones that reliably cause the same thing, over and over again, which can be modeled in a tractable way.

If I see a sandwich (which I do right now, it’s lunchtime), one of the important causes is that photons are bouncing off the sandwich, hitting my eyes, and stimulating my retina. However, most photons don’t make me see a sandwich, they make me see other things, and trying to make a model complex enough that exact photon behavior becomes parameters instead of noise is way too complicated.

So instead, I model the cause of my seeing a sandwich as being the sandwich. I see a sandwich because there really is a sandwich.

The useful part about this is that since I’m modeling the consistent, reliable, repeatable causes, these same inferences also support and explain my active interventions. I see a sandwich because there really is a sandwich, and that explains why I can move my hands and mouth to eat the sandwich, and why when I eat the sandwich, I taste a sandwich. Photons don’t really explain any of that without recourse to the sandwich.

However, if I were to reach for the sandwich and find that my hands pass through it, I would have to expand my hypothesis space to include ghost sandwiches or living in a simulation. Some people think the brain can do this with nonparametric models: probabilistic models of infinite stuff, of which I use finite pieces to make predictions. When new data comes in that supports a more complex model, I just expand the finite piece of the infinite object that I’m actually using. The downside is, a nonparametric model will always, irreducibly have a bit of extra uncertainty “left over” when compared to a parametric model that started from the right degree of complexity. The nonparametric has more things to be uncertain about, so it’s always a little more uncertain.

How can these ideas apply to the Dark Room? Well, if I go into a Dark Room, I’m actually sealing myself off from the distal causes of sensations. The walls of the room block out what’s going on outside the room, so I have no idea when, for instance, someone might knock on the door. Really knowing what’s going on requires confidence about the distal causal structure of my environment, not just confidence about the proximal structure of a small local environment. Otherwise, I could always just say, “I’m certain that photons are hitting my eyeballs in some reasonable configuration”, and I’d never need to move or do any inferences at all.

It gets worse! If my model of those distal causes is nonparametric, it always has extra leftover uncertainty. No matter how confident I am about the stuff I’ve seen, I never have complete evidence that I’ve seen everything, that there isn’t an even bigger universe out there I haven’t observed yet.

So really “minimizing prediction error” with respect to a nonparametric model of distal causes ends up requiring that I not only leave my room, but that I explore and control as much of the world as possible, at all scales which ever significantly impact my observations, without limit.

My ulterior motive is to get to the point where I can understand Friston’s recent explanation of depression, relevant to my interests as a psychiatrist.

I started reading this but it's annoying to read so I'm going to stop. I don't know what's standard in the field here but there seems to be way more jargon than necessary. Also the explanation of depression does not ring true to me:

In this sense, we might conjecture that major depression occurs when the brain is certain that it will encounter an uncertain environment, i.e. the world is inherently volatile, capricious, unpredictable and uncontrollable.

Offhand, this sounds like a description of anxiety to me, not depression. I first really experienced something I'd call anxiety a little over a year ago, and the internal experience felt a lot to me like tremendous uncertainty about whether X was happening, where X felt like a life-and-death situation. Whereas, to the extent that I've experienced anything like depression, it felt more like certainty that nothing good will ever happen.

I do get the sense that this paper has something valuable to say but I don't want to put in the additional effort to figure out what that thing is at the moment. I'm also distracted by the repeated references to mean and precision being sufficient statistics, which I can't make sense of; those are sufficient statistics for, say, Gaussian distributions, but certainly not in general.

I am curious about the dark room problem. Even if we accept surprise minimization, the brain developed in an environment where a dark room was generally not an option: the sun came up, night fell, the weather changed. Aside from the external environment, there remains the problem of thirst and hunger.

I also note that we did go to a lot of trouble to build as many rooms as possible with constant light, constant temperature, and constant humidity. Locking oneself alone in a dark room and not coming out is the archetype of depression, and depressed people are less prone to optimistic biases - this seems like it would correlate to 'more successful in resolving their surprise'. Depression: the state where no stimulus is a superstimulus.

What about surprise being based on a pattern of change rather than a state? I imagine if the sun failed to rise one morning the freakout rate would approach 100%. I also note that people get into habits, and become attached to them even if they are bad. This amounts to a predictable change in stimuli we impose through action. If we peg to the expected change in stimuli, then a constant stimuli is the case where the expected change is 0.

In line with these thoughts, it initially appears to me that the dark room problem is not an actual problem; the distinction between accuracy-maximization and surprise-minimization is unclear to me.

if the free energy principle were right, would it disprove the orthogonality thesis? Might it be impossible to design a working brain with any goal besides free energy reduction? Would anything – even a paperclip maximizer – have to start by minimizing uncertainty, and then add paperclip maximization in later as a hack? Would it change anything if it did?

My own take on this is "kinda no", but mostly because I already see the orthogonality thesis as holding only for a sufficiently general intelligence. That is, there are probably lots of things that are not orthogonal to what we think of as general intelligence, and it's only after you get those basics up to a certain level that you get an intelligence general enough for orthogonality to kick in such that capabilities can be orthogonal to telos for capabilities and telos above a certain generality threshold but below that threshold you get capabilities and telos that are correlated.

For example, I suspect phenomenal consciousness to be a property that correlates capabilities and telos around the intentional relation, and free energy may well be looking at a related notion from the perspective of how telos is achieved, and to a certain extent it looks to me like anything that does anything "intelligent" will follow these patterns if it can be called "general". We still have to worry about orthogonality, though, because these are very low level kinds of telos that shape an agents goals in correlation to its capabilities but not so strongly as to allow steering away from dangerous territory.

In short, orthogonality is probably not true all the way down, but it doesn't matter because the orthogonality thesis is more about orthogonality after generality of intelligence rather than before.

Discussion question for machine ethics researchers – if the free energy principle were right, would it disprove the orthogonality thesis?

No, and for two reasons.

1) The free energy principle is descriptive only, as Friston says in the Alius interview. It (apparently) makes no predictions about behaviour, much less about terminal goals.

2) It applies specifically to biological organisms. Most of Scott's sources note that this behaviour arose through natural selection, to handle certain specific types of uncertainty related to staying alive. It has no bearing whatsoever on, say, alien intelligences, much less computers, which can be programmed with any mind we can design.

This assumes that the free energy principle is true & correct, which I’m not sure that it is. Being unfalsifiable is a bad start, as is the fact that Karl Friston’s work is impenetrable. Most simplified explanations of the free energy principle are either equally impenetrable or seem somehow confused (this one is difficult to quantify, but reading this hasn’t really given me any insight into behaviour; if this is actually revolutionary, there should be some combination of words that makes the true meaning shine through like the sun on a cloudless day) and as far as I know, nobody has used free energy or its related concepts to achieve anything remarkable. Strong evidence that this is probably pointless.

Promoted to frontpage.

Reposting my comment from SSC:

I [just now] read the 2009 letter in Cell. It was very clear that this was a proposal for a model of human perception and action that was not at all tautological. But it didn’t explain why we’d expect this model to be true… instead, it had a lot of handwaving, and for “more details,” referred me to the 2010 Nature paper. Which I then skimmed, looking for the derivation or motivation of these equations (e.g. from figure 1 in Friston 2009). Of which I found exactly nothing.

Basically, when presented with an idea, it’s often hard to tell whether it’s true in a vacuum. But it’s not so hard to evaluate why it’s true – there are so many false things that if you believe something without good reason, it’s probably false. So rather than delving into issues with the idea itself, which might lead to engaging with some very vague writing, it’s a lot easier to just note that the mathematical parts of this model are pulled directly from the posterior.

But this definitely seems like the better website to talk to Eli Sennesh on :)

Somewhat meta question: is it better to comment here, or on SSC? Both?

Reading this I was wondering when Scott suddenly got way more confident with math. Turns out the quotes are messed up in this version. Only the first paragraph of each long block-quote is quoted properly.

Maybe rather then 'free energy' there is a better term? How about "minimum energy machine" - the idea here being that what the brain does is to generate a model or simulation that tries to generate an internal state that exactly cancels out the incoming stimuli. What is used for error detection is something called "edge detection" which is a way to detect both movement and amount by comparing the input and the memory or simulation and then modifying the model/simulation so once again match the incoming.

This is interesting because all the biological/neurological processing each sense needs and vision has many different delays SO the model/simulation needs to run in the future, taking into account the various processing delays, so that in the moment the senses arrive they are greeted with their opposite to edge detect with.

the different senses have different worlds with the innermost being the body/brain which also controls muscles. this is divided into two parts - the pre and the active - and there is a time delay in between.

When the brain is generating a good and accurate TRANSFORM that accurately predicts the incoming stimulus then there is a minimum energy point and with edge detection and ml/ai networks this minimum is maintained and so the transform that the brain created is a model of the outside (and inside body/brain senses) that is constantly seeking to minimize energy - the energy necessary to match the outside stimulus that would overwhelm the brain - think of the problem of seizures.

and one can imagine that this type of processing - from the earliest of life - works long before there are eyes and ears as in the dark depths of the ocean there was still feeling and temperature sensing and the need to find and eat or consume the BMR required to maintain life......the internal feelings then being part of what the minimum energy brain must also minimize by using the transform/model/simulation......hence the feeling of hunger or being satiated.

and the above can and must be done at all levels of life, from the first beginnings of memory which is necessary to make a model or have a history from which to make choices.....and this type of communication/cooperation can be seen to arise out of things that want to flock or are drawn to each other and their own type of life......before sex......but then this would soon invent sex because it is an excellent way to produce the higher level model or species from the individuals.

there is an interesting podcast on the evolution of cooperation.........

flocking behavior only requires a couple of simple rules:

1. Get closer to your neighbor.

2. Do not hit or harm your neighbor

(and the flock arises - now lets add the reward)

3. watch a member or two of the flock thus formed that is NOT next to you, but maybe 7 or so away, and do what they do if they SURPRISE you by jumping outside the flock!!!!!

And from the above you now get automatically almost the protective value or reward of a flock.

Now add numbers based on food requirements for the flock. As the group grows in size and volume there are necessary tasks that must be accomplished and what happens next is "Division of Labor" because what is needed for the new superorganism is a way to feed and take care of the group which now provides ones living by taking up a specialty inside the colony.

And yes - recent research shows that all life - each having to grow from an egg - being feed even in an ant colony in the larva and pupa stage differently depending upon location - (what and what NOT is feed to them by the workers determines temperament/personality even though each has the same genes........

so the idea of adapting to lower energy requirements while staying alive does explain much else in life and life's development.....

....AND......evolution is nolonger an uphill process but is rather like a stream running down a littered hillside that breaks into tributaries as it descends, predictably getting better and better at getting the resources necessary to maintain the bodies order against entropy......

as a joke - or only partly so - life happens where the river empties into the ocean, where all the good stuff grows and all the animals drink and communication is best accomplished..

...that's why we get port cities!