You can't understand human agency without understanding amoeba agency

An explanation that I've seen before of "where agency begins" is when an entity executes OODA loops. I don't know if OODA loops are a completely accurate map to reality, but they've been a useful model so far. If someone were going to explore "where agency begins" OODA loops might be a good starting point.

I feel like an article about "what agency is" must've already been written here, but I don't remember it. In any case, that article on agency in Conway's Life sounds like my next stop, thank you for linking it!

I'm not arguing against studying amoebas, I'm arguing for also studying higher-level things including agency without first studying amoebas. Amoebas are simpler, which makes them easier to study, but they are also less agenty, and in some ways *less* simple *as agents*. It would be easier to understand an abstractly written program to perform addition, than to understand register readouts from a highly optimized program, even if the former never appears "in the wild" because it's too computationally expensive.

going back to the clay token accounting might not have been a bad idea

I agree, as I said. But it would be a mistake to not also think at the abstract level; you can learn/invent digital addition just by trying to count stuff.

What do you disagree about? I agree that understanding addition implies that you understand something important about token accounting. I think there's something about addition that is maybe best learned by studying token accounting or similar (understanding how minds come to practice addition). I also think much of the essence of [addition as addition itself] is best and most easily understood in a more normal way--practicing counting and computing things in everyday life--and *not* by studying anything specifically about Mesopotamian clay token accounting, because relative to much of the essence of addition, historical accounting systems are baroque with irrelevant detail, and are a precursor or proto form of practicing addition, hence don't manifest the essence of addition in a refined and clear way.

I like your elevator example. I think it's an open question whether / how amoebas are manifesting the same principles as (human, say) agency / mind / intelligence, i.e. to what extent amoebas are simpler models of the same thing (agent etc.) vs. models of something else (such as deficient agency, a subset of agency, etc.). I mean, my point isn't that there's some "amount" that amoebas "are agents" or whatever, that's not exactly well-defined or interesting, my point is that the reasons we're interested in agency make human agency much more interesting than amoeba agency, and this is not primarily a mistake; amoebas pretty much just don't do fictive learning, logical inference, etc., even though if you try you can read into amoebas a sort of deficient/restricted form of these things.

That's not unreasonable as a quick summary of the principle.

I would say there is more to what makes a living system alive than just following the free energy principle per se. For instance, the robot would also need to scavenge for material and energy resources to incorporate into itself for maintenance, repair, and/or reproduction. Just correcting its gait when thrown off balance allows it minimize a sort of behavioral free energy, but that's not enough to count as alive.

But if you want to put amoebas and humans in the same qualitative category of "agency", then you need a framework that is general enough to capture the commonalities of interest. And yes, under such a broad umbrella, artificial control systems and dynamically balancing walking robots would be included.

The free energy principle applies to a lot of systems, not just living or agentic. I see it more as a way to systematize our approach to understanding a system or process rather than an explanation in and of itself. By focusing on how a system maintains set points (e.g., homeostasis) and minimizes prediction error (e.g., unsupervised learning), I think we would be better positioned to figure out what real agents are actually doing in a way that could inform the both the design and alignment of AGI.

The model may be implicit, but it's embedded in the structure of the whole thermostat system, from the thermometer that measures temperature to the heating and cooling systems that it controls. For instance, it "knows" that turning on the heat is the appropriate thing to do when the temperature it reads falls below its set point. There is an implication there that the heater causes temperature to rise, or that the AC causes it to fall, even though it's obviously not running simulations (unless it's a really good thermostat) on how the heating/cooling systems affect the dynamics of temperature fluctuations in the building.

The engineers did all the modeling beforehand, then built the thermostat to activate the heating and cooling systems in response to temperature fluctuations according to the rules that they precomputed [LW · GW]. Evolution did just this in building the structure of the amoeba's gene networks and the suite of human instincts (heritable variation + natural selection is how information is transferred from the environment into a species' genome). Lived experience pushes further information from the environment to the internal model, upregulating or downregulating various genes in response to stimuli or learning to reinforce certain behaviors in certain contexts. But environmental information was already there in the structure to begin with, just like it is in more traditional artificial control systems.

The example with the hunter and pheasants was just to show how "regulating" (i.e., consistently achieving a state in the desirable set = pheasant successfully shot) requires the hunter to have a good mental model of the system (pheasant behavior, wind disturbances, etc.). Again, this model does not have to be explicit in general but could be completely innate.

Maybe it's better to start with something we do understand, then, to make the contrast clear.  Can we study the "real" agency of a thermometer, and if we can, what would that research program look like?

My sense is that you can study the real agency of a thermometer, but that it's not helpful for understanding amoebas.  That is, there isn't much to study in "abstract" agency, independent of the substrate it's implemented on.  For the same reason I wouldn't study amoebas to understand humans; they're constructed too differently.

But it's possible that I don't understand what you're trying to do.