Computational Model: Causal Diagrams with Symmetry
post by johnswentworth · 20190822T17:54:11.274Z · score: 40 (15 votes) · LW · GW · 14 commentsContents
Why would we want to do this? Learning None 14 comments
Consider the following program:
f(n):
if n == 0:
return 1
return n * f(n1)
Let’s think about the process by which this function is evaluated. We want to sketch out a causal DAG [LW · GW] showing all of the intermediate calculations and the connections between them (feel free to pause reading and try this yourself).
Here’s what the causal DAG looks like:
Each dotted box corresponds to one call to the function f. The recursive call in f becomes a symmetry in the causal diagram: the DAG consists of an infinite sequence of copies of the same subcircuit.
More generally, we can represent any Turingcomputable function this way. Just take some pseudocode for the function, and expand out the full causal DAG of the calculation. In general, the diagram will either be finite or have symmetric components  the symmetry is what allows us to use a finite representation even though the graph itself is infinite.
Why would we want to do this?
For our purposes, the central idea of embedded agency is to take these blackbox systems which we call “agents”, and break open the black boxes to see what’s going on inside.
Causal DAGs with symmetry are how we do this for Turingcomputable functions in general. They show the actual causeandeffect process which computes the result; conceptually they represent the computation rather than a blackbox function.
In particular, a causal DAG + symmetry representation gives us all the natural machinery of causality  most notably counterfactuals. We can ask questions like “what would happen if I reached in and flipped a bit at this point in the computation?” or “what value would f(5) return if f(3) were 11?”. We can pose these questions in a welldefined, unambiguous way without worrying about logical counterfactuals [LW · GW], and without adding any additional machinery. This becomes particularly important for embedded optimization: if an “agent” (e.g. an organism) wants to plan ahead to achieve an objective (e.g. find food), it needs to ask counterfactual questions like “how much food would I find if I kept going straight?”.
The other main reason we would want to represent functions as causal DAGs with symmetry is because our universe appears to be one giant causal DAG [LW · GW] with symmetry.
Because our universe is causal, any computation performed in our universe must eventually bottom out in a causal DAG. We can write our programs in any language we please, but eventually they will be compiled down to machine code and run by physical transistors made of atoms which are themselves governed by a causal DAG. In most cases, we can represent the causal computational process at a more abstract level  e.g. in our example program, even though we didn’t talk about registers or transistors or electric fields, the causal diagram we sketched out would still accurately represent the computation performed even at the lower levels.
This raises the issue of abstraction  the core problem of embedded agency. My own main usecase for the causal diagram + symmetry model of computation is formulating models of abstraction: how can one causal diagram (possibly with symmetry) represent another in a way which makes counterfactual queries on the map correspond to some kind of counterfactual on the territory? Can that work when the “map” is a subDAG of the territory DAG? It feels like causal diagrams + symmetry are the minimal computational model needed to get agencyrelevant answers to this sort of question.
Learning
The traditional ultimate learning algorithm is Solomonoff Induction: take some blackbox system which spews out data, and look for short programs which reproduce that data. But the phrase “blackbox” suggests that perhaps we could do better by looking inside that box.
To make this a little bit more concrete: imagine I have some python program running on a server which responds to http requests. Solomonoff Induction would look at the data returned by requests to the program, and learn to predict the program’s behavior. But that sort of blackbox interaction is not the only option. The program is running on a physical server somewhere  so, in principle, we could go grab a screwdriver and a tiny oscilloscope and directly observe the computation performed by the physical machine. Even without measuring every voltage on every wire, we may at least get enough data to narrow down the space of candidate programs in a way which Solomonoff Induction could not do. Ideally, we’d gain enough information to avoid needing to search over all possible programs.
Compared to Solomonoff Induction, this process looks a lot more like how scientists actually study the real world in practice: there’s lots of taking stuff apart and poking at it to see what makes it tick.
In general, though, how to learn causal DAGs with symmetry is still an open question. We’d like something like Solomonoff Induction, but which can account for partial information about the internal structure of the causal DAG, rather than just overall inputoutput behavior. (In principle, we could shoehorn this whole thing into traditional Solomonoff Induction by treating information about the internal DAG structure as normal old data, but that doesn’t give us a good way to extract the learned DAG structure.)
We already have algorithms for learning causal structure in general. Pearl’s Causality sketches out some such algorithms in chapter 2, although they’re only practical for either very small systems or very large amounts of data. Bayesian structure learning can handle larger systems with less data, though sometimes at the cost of a very large amount of compute  i.e. estimating highdimensional integrals.
However, in general, these approaches don’t directly account for symmetry of the learned DAGs. Ideally, we would use a prior which weights causal DAGs according to the size of their representation  i.e. infinite DAGs would still have nonzero prior probability if they have some symmetry allowing for finite representation, and in general DAGs with multiple copies of the same subDAG would have higher probability. This isn’t quite the same as weighting by minimum description length in the Solomonoff sense, since we care specifically about symmetries which correspond to function calls  i.e. isomorphic subDAGs. We don’t care about graphs which can be generated by a short program but don’t have these sorts of symmetries. So that leaves the question: if our prior probability for a causal DAG is given by a notion of minimum description length which only allows compression by specifying reused subcircuits, what properties will the resulting learning algorithm possess? Is it computable? What kinds of data are needed to make it tractable?
14 comments
Comments sorted by top scores.
A few comments:

Because of quantum physics, the universe is actually not described by a causal DAG very well (if we require microscopic precision). But I agree that in practice there are useful approximate descriptions that do look like a causal DAG.

One direction towards studying the learning of symmetric causal DAGs (of a sort) is my ideas about cellular decision processes and the more general "graph decision processes", see this comment.

As a side note, I'm not convinced that "breaking open black boxes" to solve "embedded agency" is a useful way of thinking. IMO the problems associated with "embedded agency" should be solved using a combination of incomplete models and correctly dealing with traps.
The quantum fields themselves follow causality just fine. Things only become weird if we introduce wavefunction collapse, which isn't strictly necessary.
Hmm, no, not really. The quantum fields follow "causality" in some quantum sense (roughly speaking, operators in spacelike separations commute, and any local operator can be expressed in terms of operators localized near the intersection of its past lightcone with any spacelike hypersurface), which is different from the sense used in causal DAGsa (in fact you can define "quantum causal DAGs" which is a different mathematical object). Violation of Bell's inequality precisely means that you can't describe the system by a causal DAG. If you want to do the MWI, then the wavefunction doesn't even decompose into data that can be localized.
Violation of Bell's inequality precisely means that you can't describe the system by a causal DAG
No, violation of Bell's inequality means you can't describe the system by causal interactions among particles and measurements. If we stop thinking about particles and "measurements" altogether, and just talk about fields, that's not an issue.
As you say, under MWI, the wavefunction doesn't even decompose into data that can be localized. So, in order to represent the system using classical causal diagrams, the "system state" has to contain the whole wavefunction. As long as we can write down an equation for the evolution of the wavefunction over time, we have a classical causal model for the system.
Quantum causal models are certainly a much cleaner representation, in this case, but classical causal models can still work  we just have to define the "DAG vertices" appropriately.
It doesn't really have much to do with particles vs. fields. We talk about measurements because measurements are the thing we actually observe. It seems strange to say you can model the world as a causal network if the causal network doesn't include your actual observations. If you want to choose a particular frame of reference and write down the wavefunction time evolution in that frame (while ignoring space) then you can say it's a causal network (which is just a linear chain, and deterministic at that) but IMO that's not very informative. It also loses the property of having things made of parts, which AFAIU was one of your objectives here.
The wavefunction does have plenty of internal structure, that structure just doesn't line up neatly with space. It won't just be a linear chain, and it will be made of "parts", but those parts won't necessarily line up neatly with macroscopic observations/objects.
And that's fine  figuring out how to do ontology mapping between the lowlevel "parts" and the highlevel "parts" is a central piece of the problem. Not being able to directly observe variables in the lowlevel causal diagram is part of that. If we want e.g. a theory of abstraction, then these issues are perfect usecases.
Thanks for a nice post about causal diagrams!
Because our universe is causal, any computation performed in our universe must eventually bottom out in a causal DAG.
Totally agree. This is a big part of the reason why I'm excited about these kinds of diagrams.
This raises the issue of abstraction  the core problem of embedded agency. ... how can one causal diagram (possibly with symmetry) represent another in a way which makes counterfactual queries on the map correspond to some kind of counterfactual on the territory?
Great question, I really think someone should look more carefully into this. A few potentially related papers:
https://arxiv.org/abs/1105.0158
https://arxiv.org/abs/1812.03789
In general, though, how to learn causal DAGs with symmetry is still an open question. We’d like something like Solomonoff Induction, but which can account for partial information about the internal structure of the causal DAG, rather than just overall inputoutput behavior.
Again, agreed. It would be great if we could find a way to make progress on this question.
This isn’t quite the same as weighting by minimum description length in the Solomonoff sense, since we care specifically about symmetries which correspond to function calls  i.e. isomorphic subDAGs. We don’t care about graphs which can be generated by a short program but don’t have these sorts of symmetries.
Can you elaborate on this? What would be an example of a graph that can be generated by a short program, but that does not have these sorts of symmetries?
My intuition is that the class of processes your describing is Turing complete, and therefore can simulate any Turing machine, and is thus just another instance of Solomonoff induction with a different MDL constant.
Edit: Rule 110 would be an example.
Were you summoned by this post accidentally using your true name?
I'll give an answer via analogy: the digit sequence 123123123123123... is symmetric: the sequence directly contains a copy of itself. The sequence 12345678910111213141516... is not symmetric: it will not repeat and does not contain a copy of itself, although there is a pattern and it could be generated programmatically.
Similarly with DAG symmetries. If we're producing these DAGs by expanding out programs, then the only infinite patterns we'll see are symmetries  subDAGs which repeat, corresponding to function blocks. We don't need to worry about DAGs with strange patterns which could be generated programmatically but can't just be built by repeating subDAGs.
I really like this model of computation and how naturally it deals with counterfactuals, surprised it isn't talked about more often.
This raises the issue of abstraction  the core problem of embedded agency.
I'd like to understand this claim better  are you saying that the core problem of embedded agency is relating highlevel agent models (represented as causal diagrams) to lowlevel physics models (also represented as causal diagrams)?
I'm saying that the core problem of embedded agency is relating a highlevel, abstract map to the lowlevel territory it represents. How can we characterize the mapterritory relationship based on our knowledge of the mapgenerating process, and what properties does the mapgenerating process need to have in order to produce "accurate" & useful maps? How do queries on the map correspond to queries on the territory? Exactly what information is kept and what information is thrown out when the map is smaller than the territory? Good answers to these question would likely solve the usual reflection/diagonalization problems, and also explain when and why worldmodels are needed for effective goalseeking behavior.
When I think about how to formalize these sort of questions in a way useful for embedded agency, the minimum requirements are something like:
 need to represent physics of the underlying world
 need to represent the causeandeffect process which generates a map from a territory
 need to run counterfactual queries on the map (e.g. in order to do planning)
 need to represent sufficiently general computations to make agenty things possible
... and causal diagrams with symmetry seem like the natural class to capture all that.
the DAG consists of an infinite sequence of copies of the same subcircuit.
If it's infinite then the program doesn't terminate. (In theory this may be the case if n is a nonnegative integer.)
Ideally, we’d gain enough information to avoid needing to search over all possible programs.
Over all shorter possible programs.
(In principle, we could shoehorn this whole thing into traditional Solomonoff Induction by treating information about the internal DAG structure as normal old data, but that doesn’t give us a good way to extract the learned DAG structure.)
Sounds like SI is being treated as a black box.
We don’t care about graphs which can be generated by a short program but don’t have these sorts of symmetries.
Why do we like programs which use recursion instead of iteration?
The nontermination point is a bit subtle. If you look at the example in the OP, the causal diagram itself is infinite, but as long as we pass in a positive integer the output value won't actually depend on the whole circuit. One of the conditional nodes will recognize the base case, and ignore whatever's going on in the rest of the circuit below it. So the program can terminate even if the DAG is infinite. (Equivalent conceptualization: imagine running a lazy evaluator on the infinite DAG.)
That said, DAGs with symmetry certainly can represent computations which do not halt. This is a feature, not a bug: there are in fact computations which do not halt, and we can observe their physical behavior midcomputation. If we want to think about e.g. what my processor is doing when running an infinite loop, then we need a computational model which can represent that sort of thing, rather than just talking about input/output behavior.