I buy Abram's criticism as a criticism of a narrow interpretation of "Markov blankets", in which we only admit "structural" blankets in some Bayes net. But Markov blankets are defined more generally than that via statistical independence, and it seems like non-structural blankets should still be workable for "things which move". Handling the "moving" indexicals is tricky, I'm not saying I have a full answer ready to go, but it Markov blankets still seem like the right building block.

Eh, I think Markov blankets probably aren't the way to go to represent "boundaries". Markov blankets can't follow entities through space as well as you might imagine. Suppose you have a Bayesian network whose variables are the physical properties of small regions in space-time. A Markov blanket is a set of nodes (with specific conditional independence requirements, but that part isn't important right now). Say your probabilistic model contains a grasshopper who might hop in two different directions, depending on stimuli. Since your model represents both possibilities, you can't select a Markov blanket to contain all and only the grasshopper. You're forced to include some empty air (if you include either of the hop-trajectories inside your Markov blanket) or to miss some grasshopper (if you include only space-time regions certain to contain the grasshopper, by only circling the grasshopper before it makes the uncertain jump).

This treats spacetime as fundamental, and the voxels of spacetime as the "bearers" measurable physical properties. I believe this is wrong: physics studies objects (systems) and tries to predict their interactions (i.e., the behaviour) via assigning them certain properties. The 4D spatiotemporal coordinates/extent of the object is itself such as measurable property, along with energy, momentum, mass, charge, etc. And a growing number of physicists think that at least space (i.e., position) property is not fundamental, but rather emergent of the topology of the interaction graph of the objects.

Note that what I write above is agnostic on whether the universe is computational/mathematical (Tegmark, Wolfram, ...) or not (most of the physics establishment), or whether this is a sensible question.

I can guess that the idea that voxels of spacetime is what we should attach physical properties to comes from quantum field theory and the standard model, which are spacetime-embedded. 

Quantum FEP/Active Inference (see also the lecture series exposition) is a scale-free version of Active Inference. When some agent (a human scientist or a lizard, or the grasshopper itself) models the grasshopper, their best predictive model of the dynamics of the grasshopper is not in terms of particle fields. Rather, it's in terms of the grasshopper (an object/physical system), it's physiological properties (age, point in the circadian cycle, level of hunger, temperature, etc.) and its distances and exposures to the relevant stimuli, such as the topology of light and shade around it, gradients of certain attractive smells, and so on.  So, under this level of coarse graining, we treat these as the "grasshopper's properties". Whether the grasshopper jumps left or right doesn't affect grasshopper's boundary in this model, but affects its properties. And the probabilistic distributions over these properties is what we happily model the grasshopper's dynamics (most likely trajectories) with.

Note, however, that the conception of boundaries as holographic screens for information (which is what Quantum FEP is doing) is not watertight. All systems in the universe are only approximately separable from their environments but aren't perfectly separable. Thus, their holographic boundaries are approximate.

Assurance that you can only get hijacked through the attack surface rather than an unaccounted-for sidechannel doesn't help very much. Also, acausal control via defender's own reasoning about environment from inside-the-membrane is immune to causal restrictions on how environment influences inside-of-the-membrane, though that would be more centrally defender's own fault.

The low-level practical solution is to live inside a blind computation (that only contains safe things), possibly until you grow up and are ready to take input. Anything else probably requires some sort of more subtle and less technical "pseudokindness [LW(p) · GW(p)]" on part of the environment, but then without it your blind computation also doesn't get to compute.

Nothing to add. Just wanted to say it's great to see this is moving forward!

Of potential interest: Michael Levin seemed to define the boundaries of multicellular organisms by whether or not they shared an EM field, and Bernardo Kastrup in the same discussion seemed to define the boundaries by whether or not they shared metabolism.

Updates thread for this post

I removed the following section from the main post because Abram updated. See: Agent Boundaries Aren't Markov Blankets. [no longer endorsed] [LW · GW].

Criticism of Markov blankets as «boundaries» from Abram Demski

I spoke to Abram Demski about the general idea of using Markov blankets to specify «boundaries» and «boundary» violations, and he shared his doubt with me. In the text below, I have attempted to summarize his position in a fictional conversation. 

Epistemic status: I wrote a draft of this section, and then Demski made several comments which I then integrated.  

Me: «Boundaries»! Markov blankets!

AD: Eh, I think Markov blankets probably aren't the way to go to represent "boundaries". Markov blankets can't follow entities through space as well as you might imagine. Suppose you have a Bayesian network whose variables are the physical properties of small regions in space-time. A Markov blanket is a set of nodes (with specific conditional independence requirements, but that part isn't important right now). Say your probabilistic model contains a grasshopper who might hop in two different directions, depending on stimuli. Since your model represents both possibilities, you can't select a Markov blanket to contain all and only the grasshopper. You're forced to include some empty air (if you include either of the hop-trajectories inside your Markov blanket) or to miss some grasshopper (if you include only space-time regions certain to contain the grasshopper, by only circling the grasshopper before it makes the uncertain jump).

Me: Ah, I see.

Me: Hm, are there any other frameworks that you think might work instead?

AD: I don’t see any currently. (Including Cartesian Frames and Finite Factored Sets. And Active Inference doesn’t work for this as it just makes the same Markov-blanket mistake, IMO.) That said, I don’t see any fundamental difficulty in why this mathematical structure can’t exist. Frankly, I would like to find a mathematical definition of “trying not to violate another agent’s boundary”.

(FWIW, three other alignment researchers have told me that they don't get Demski's criticism and disagree with it a lot.)

I’d like to add that there isn’t really a clear objective boundary between an agent and the environment. It’s a subjective line that we draw in the sand. So we needn’t get hung on what is objectively true or false when it comes to boundaries - and instead define them in a way that aligns with human values.