paulfchristiano's Shortform

[For reference, my view is “probably fast takeoff, defined by <2 years between widespread common knowledge of basically how AGI algorithms can work, and the singularity.” (Is that “fast”? I dunno the definitions.) ]

My view is that LLMs are not part of the relevant category of proto-AGI; I think proto-AGI doesn’t exist yet. So from my perspective,

• you’re asking for a prediction about how much economic impact comes from a certain category of non-proto-AGI software,
• you’re noting that the fast takeoff hypothesis doesn’t constrain this prediction at all,
• and therefore you’re  assigning the fast takeoff hypothesis a log-uniform prior as its prediction.

I don’t think this is a valid procedure.

ANALOGY:

The Law of Conservation of Energy doesn’t offer any prediction about the half-life of tritium. It would be wrong to say that it offers an implicit prediction about the half-life of tritium, namely, log-uniform from 1 femtosecond to the lifetime of the universe, or whatever. It just plain doesn’t have any prediction at all!

Really, you have to distinguish two things:

• “The Law of Conservation of Energy”
• “My whole worldview, within which The Law of Conservation of Energy is just one piece.”

If you ask me to predict the total energy of the tritium decay products, I would make a prediction using Conservation of Energy. If you ask me to predict the half-life of tritium, I would make a prediction using other unrelated things that I believe about nuclear physics.

And that’s fine!

And in particular, the failure of Conservation of Energy here is not a failure at all; indeed it’s not any evidence that I’m wrong to believe Conservation of Energy. Right?

BACK TO AI:

Again, you’re asking for a prediction about how much economic impact comes from a certain category of (IMO)-non-AGI AI software. And I’m saying “my belief in fast takeoff doesn’t constrain my answer to that question—just like my belief in plate tectonics doesn’t constrain my answer to that question”. I don’t think any less of the fast takeoff hypothesis on account of that fact, any more than I think less of plate tectonics.

So instead, I would come up with a prediction in a different way. What do these systems do now, what do I expect them to do in the future, how big are relevant markets, etc.?

I don’t know what my final answers would be, but I feel like you’re not allowed to put words in my mouth and say that my prediction is “log-uniform from $10M to $10T”. That’s not my prediction!

Sorry if I’m misunderstanding.

1. What you are basically saying is "Yudkowsky thought AGI might have happened by now, whereas I didn't; AGI hasn't happened by now, therefore we should update from Yud to me by a factor of ~1.5 (and also from Yud to the agi-is-impossible crowd, for that matter)" I agree.
2. Here's what I think is going to happen (this is something like my modal projection, obviously I have a lot of uncertainty; also I don't expect the world economy to be transformed as fast as this projects due to schlep and regulation, and so probably things will take a bit longer than depicted here but only a bit.):
   
   
   No pressure, but I'd love it if you found time someday to fiddle with the settings of the model at takeoffspeeds.com and then post a screenshot of your own modal or median future. I think that going forward, we should all strive to leave this old "fast vs. slow takeoff" debate in the dust & talk more concretely about variables in this model, or in improved models.
    

Very minor note: is there some way we can move away from the terms "slow takeoff" and "fast takeoff" here? My impression is that almost everyone is confused by these terms except for a handful of people who are actively engaged in the discussions. I've seen people variously say that slow takeoff means:

• That AI systems will take decades to go from below human-level to above human level
• That AI systems will take decades to go from slightly above human-level to radically superintelligent
• That we'll never get explosive growth (GDP will just continue to grow at like 3%)

Whereas the way you're actually using the term is "We will have a smooth rather than abrupt transition to a regime of explosive growth." I don't have any suggestions for what terms we should use instead, only that we should probably come up with better ones.

Questions of how many trillions of dollars OpenAI will be allowed to generate by entities like the U.S. government are unimportant to the bigger concern, which is: Will these minds be useful in securing AIs before they become so powerful they're dangerous? Given the pace of ML research and how close LLMS are apparently able to get to AGI without actually-being-AGI, as a person who was only introduced to this debate in like the last year or so, my answer is: seems like that's what's happening now? Yeah?

In my model, the AGI threshold is a phase change. Before that point, AI improves at human research speed; after that point it improves much faster (even as early AGI is just training skills and not doing AI research). Before AGI threshold, the impact on the world is limited to what sub-AGI AI can do, which depends on how much time is available before AGI for humans to specialize AI to particular applications. So with short AGI threshold timelines, there isn't enough time for humans to make AI very impactful, but after AGI threshold is crossed, AI advancement accelerates much more than before that point. And possibly there is not enough time post-AGI to leave much of an economic impact either, before it gets into post-singularity territory (nanotech and massive compute; depending on how AGIs navigate transitive alignment, possibly still no superintelligence).

I think this doesn't fit into the takeoff speed dichotomy, because in this view the speed of the post-AGI phase isn't observable during the pre-AGI phase.

The most important complication is that the AI is no longer isolated from the deliberating humans. We don't care about what the humans "would have done" if the AI hadn't been there---we need our AI to keep us safe (e.g. from other AI-empowered actors), we will be trusting our AI not to mess with the process of deliberation, and we will likely be relying on our AI to provide "amenities" to the deliberating humans (filling the same role as the hypercomputer in the old proposal).

Going even further, I'd like to avoid defining values in terms of any kind of counterfactual like "what the humans would have said if they'd stayed safe" because I think those will run into many of the original proposal's problems.

Instead we're going to define values in terms of what the humans actually conclude here in the real world. Of course we can't just say "Values are whatever the human actually concludes" because that will lead our agent to deliberately compromise human deliberation rather than protecting it.

Instead, we are going to add in something like narrow value leaning. Assume the human has some narrow preferences over what happens to them over the next hour. These aren't necessarily that wise. They don't understand what's happening in the "outside world" (e.g. "am I going to be safe five hours from now?" or "is my AI-run company acquiring a lot of money I can use when I figure out what I want?"). But they do assign low value to the human getting hurt, and assign high value to the human feeling safe and succeeding at their local tasks; they assign low value to the human tripping and breaking their neck, and high value to having the AI make them a hamburger if they ask for a hamburger; and so on. These preferences are basically dual to the actual process of deliberation that the human undergoes. There is a lot of subtlety about defining or extracting these local values, but for now I'm going to brush that aside and just ask how to extract the utility function from this whole process.

It's no good to simply use the local values, because we need our AI to do some lookahead (both to future timesteps when the human wants to remain safe, and to the far future when the human will evaluate how much option value the AI actually secured for them). It's no good to naively integrate local values over time, because a very low score during a brief period (where the human is killed and replaced by a robot accomplice) cannot be offset by any number of high scores in the future.

Here's my starting proposal:

• We quantify the human's local preferences by asking "Look at the person you actually became. How happy are you with that person? Quantitatively, how much of your value was lost by replacing yourself with that person?" This gives us a loss on a scale from 0% (perfect idealization, losing nothing) to 100% (where all of the value is gone). Most of the values will be exceptionally small, especially if we look at a short period like an hour.
• Eventually once the human becomes wise enough to totally epistemically dominate the original AI, they can assign a score to the AI's actions. To make life simple for now let's ignore negative outcomes and just describe value as a scalar from 0% (barren universe) to 100% (all of the universe is used in an optimal way). Or we might use this "final scale" in a different way (e.g. to evaluate the AI's actions rather than the actually assessing outcomes, assigning high scores to corrigible and efficient behavior and somehow quantifying deviations from that ideal).
• The utility is the product of all of these numbers.

I think there are a lot of problems with this method of quantitative aggregation. But I think this direction is promising and I currently expect something along these lines will work.

The most fundamental reason that I don't expect this to work is that it gives up on "sharing parameters" between the extractor and the human model. But in many cases it seems possible to do so, and giving on up on that feels extremely unstable since it's trying to push against competitiveness (i.e. the model will want to find some way to save those parameters, and you don't want your intended solution to involve subverting that natural pressure).

Intuitively, I can imagine three kinds of approaches to doing this parameter sharing:

1. Introduce some latent structure $L$ (e.g. semantics of natural language, what a cat "actually is") that is used to represent both humans and the intended question-answering policy. This is the diagram $H\leftarrow L\to f^{+}$
2. Introduce some consistency check $f^{?}$ between $H$ and $L$. This is the diagram $H\to f^{?}\leftarrow f^{+}$
3. Somehow extract $f^{+}$ from $H$ or build it out of pieces derived from $H$. This is the diagram $H\to f^{+}$. This is kind of like a special case of 1, but it feels pretty different.

(You could imagine having slightly more general diagrams corresponding to any sort of d-connection between $H$ and $f^{+}$.)

Approach 1 is the most intuitive, and it seems appealing because we can basically leave it up to the model to introduce the factorization (and it feels like there is a good chance that it will happen completely automatically). There are basically two challenges with this approach:

• It's not clear that we can actually jointly compress $H$ and $f^{+}$. For example, what if we represent $H$ in an extremely low level way as a bunch of neurons firing; the neurons are connected in a complicated and messy way that learned to implement something like $f^{+}$, but need not have any simple representation in terms of $f^{+}$. Even if such a factorization is possible, it's completely unclear how to argue about how hard it is to learn. This is a lot of what motivates the compression-based approaches---we can just say "$H$ is some mess, but you can count on it basically computing $f^{+}$" and then make simple arguments about competitiveness (it's basically just as hard as separately learning $H$ and $f^{+}$).
• If you overcame that difficulty, you'd still have to actually incentivize this kind of factorization in the model (rather than sharing parameters in the unintended way). It's unclear how to do that (maybe you're back to think about something speed-prior like, and this is just a way to address my concern about the speed-prior-like proposals), but this feels more tractable than the first problem.

I've been thinking about approach 2 over the last 2 months. My biggest concern is that it feels like you have to pay the bits of H back "as you learn them" with SGD, but you may learn them in such a way that you don't really get a useful consistency update until you've basically specified all of H. (E.g. suppose you are exposed to brain scans of humans for a long time before you learn to answer questions in a human-like way. Then at the end you want to use that to pay back the bits of the brain scans, but in order to do so you need to imagine lots of different ways the brain scans could have looked. But there's no tractable way to do that, because you have to fill in the the full brain scan before it really tells you about whether your consistency condition holds.)

Approach 3 is in some sense most direct. I think this naturally looks like imitative generalization, where you use a richer set of human answers to basically build $f^{+}$ on top of your model. I don't see how to make this kind of thing work totally on its own, but I'm probably going to spend a bit of time thinking about how to combine it with approaches 1 and 2.

One aspect of this proposal which I don't know how to do is evaluation the answers of the question-answerer. That looks too me very related to the deconfusion of universality that we discussed a few months ago, and without an answer to this, I feel like I don't even know how to run this silly approach.

I think the biggest problem is that $F_{\theta }$ can compute the instrumental policy (or a different policy that works well, or a fragment of it). Some possible reasons:

• Maybe some people in the world are incidentally thinking about the instrumental policy and $F_{\theta }$ makes predictions about them. 
• Maybe an adversary who computes a policy that performs well in order to try to attack the learning process (since $E_{{\theta }^{\prime }}$ may just copy the adversary's policy in order to be fast if it works well on training, resulting in bad generalization).
• Maybe no one computes the instrumental policy itself, but they compute helpful building blocks and $E_{{\theta }^{\prime }}$ can save compute by reusing those.
• Maybe $F_{\theta }$ wants to predict question-answering policies, and computing optimal responses to the training objective is a simple way for it to forecast what they will do. (It's totally unclear if this is actually more efficient than the intended policy.) Then $E_{{\theta }^{\prime }}$ can copy that.

I don't know if any of those particular failures are too likely. But overall it seems really bad to rely on $F_{\theta }$ never computing something inconvenient, and it definitely doesn't look like it's going to work in the worst case.

What are some possible outs, if in fact $F_{\theta }$ computes something adversarial to try to make it easy for $E_{{\theta }^{\prime }}$ to learn something bad?

• At the time when $F_{\theta }$ is running it doesn't know the question Q that $E_{\theta }$ will need to answer. So in order to cause trouble it would need to compute answers for all of these questions. But I'm afraid that this is doable in many cases, since $F_{\theta }$ could translate a big chunk of its internal state into natural language and then $E_{{\theta }^{\prime }}$ can do some simple QA on top of that.
• We could try to ensure that the correctness/consistency checks are really complex or challenging to meet, at least for some questions. If the complexity of these checks scales up with the complexity of $F_{\theta }$ itself (e.g. by using amplification where the evaluator consults a model as big as $F_{\theta }$), then maybe it's impossible for $F_{\theta }$ to meet them. This idea can be applied without the factorization into F and E, but it's probably useful to have this kind of factorization in mind as a possible tool when exploring approaches based on tricky-evaluations.
• We could try to exploit some further structural facts about the parts of $Z_{\theta }\left(X\right)$ that are used by $E_{{\theta }^{\prime }}$. For example, it feels like the intended model is going to be leveraging facts that are further "upstream." For example, suppose an attacker observes that there is a cat in the room, and so writes out "There is a cat in the room" as part of a natural-language description of what it's going on that it hopes that $E_{{\theta }^{\prime }}$ will eventually learn to copy. If $F_{\theta }$ predicts the adversary's output, it must first predict that there is actually a cat in the room, which then ultimately flows downstream into predictions of the adversary's behavior. And so we might hope to prefer the "intended" $E_{{\theta }^{\prime }}$ by having it preferentially read from the earlier activations (with shorter computational histories).

Overall this kind of approach feels like it's probably doomed, but it does capture part of the intuition for why we should "just" be able to learn a simple correspondence rather than getting some crazy instrumental policy. So I'm not quite ready to let it go yet. I'm particularly interested to push a bit on the third of these approaches.

How much hope is there for jointly representing $f_{{\theta }_1}$ and $H_{{\theta }_2}$?

The most obvious representation in this case is to first specify $f_{{\theta }_1}$, and then actually model the process of gradient descent that produces $H_{{\theta }_2}$. This runs into a few problems:

1. Actually running gradient descent to find ${\theta }_2$ is too expensive to do at every datapoint---instead we learn a hypothesis that does a lot of its work "up front" (shared across all the datapoints). I don't know what that would look like. The naive ways of doing it (redoing the shared initial computation for every batch) only work for very large batch sizes, which may well be above the critical batch size. If this was the only problem I'd feel pretty optimistic about spending a bunch of time thinking about it.
    
2. Specifying "the process that produced $H_{{\theta }_2}$" requires specifying the initialization ${\tilde{\theta }}_2$, which is as big as ${\theta }_2$. That said, the fact that the learned ${\theta }_2$ also contains a bunch of information about ${\theta }_1$ means that it can't contain perfect information about the initialization ${\tilde{\theta }}_2$, i.e. that multiple initializations lead to exactly the same final state. So that suggests a possible out: we can start with an "initial" initialization ${\tilde{\theta }}_2^0$. Then we can learn ${\tilde{\theta }}_2$ by gradient descent. The fact that many different values of ${\tilde{\theta }}_2$ would work suggests that it should be easier to find one of them; intuitively if we set up training just right it seems like we may be able to get all the bits back.
    
3. Running the gradient descent to find ${\theta }_2$ even a single time may be much more expensive than the rest of training. That is, human learning (perhaps extended over biological or cultural evolution) may take much more time than machine learning. If this the case, then any approach that relies on reproducing that learning is completely doomed.
   
   Similar to problem 2, the mutual information between ${\theta }_2$ and the learning process that produced ${\theta }_2$ also must be kind of low---there are only $\left|{\theta }_2\right|$ bits of mutual information to go around between the learning process, its initialization, and ${\theta }_1$. But exploiting this structure seems really hard, if actually there aren't any fast learning processes that lead to the same conclusion

My current take is that even in the case where ${\theta }_2$ was actually produced by something like SGD, we can't actually exploit that fact to produce a direct, causally-accurate representation ${\theta }_2$.

That's kind of similar to what happens in my current proposal though: instead we use the learning process embedded inside the broader world-model learning. (Or a new learning process that we create from fresh to estimate the specialness of ${\theta }_2$, as remarked in the sibling comment [AF(p) · GW(p)].)

So then the critical question is not "do we have enough time to reproduce the learning process that lead to ${\theta }_2$?" it is "Can we directly learn $H_{{\theta }_2}$ as an approximation to $f_{{\theta }_1}$?" If we able to do this in any way, then we can use that to help compress ${\theta }_2$. In the other proposal, we can use it to help estimate the specialness of ${\theta }_2$ in order to determine how many bits we get back---it's starting to feel like these things aren't so different anyway.

Fully learning the whole human-model seems impossible---after all, humans may have learned things that are more sophisticated then what we can learn with SGD (even if SGD learned a policy with "enough bits" to represent ${\theta }_2$, so that it could memorize them one by one if it saw the brain scans or whatever).

So we could try to do something like "learning just the part of the human policy is that is about answering questions." But it's not clear to me how you could disentangle this from all the rest of the complicated stuff going in for the human.

Overall this seems like a pretty tricky case. The high-level summary is something like: "The model is able to learn to imitate humans by making detailed observations about humans, but we are not able to learn a similarly-good human model from scratch given data about what the human is 'trying' to do or how they interpret language." Under these conditions it seems particularly challenging to either jointly represent $H_{{\theta }_2}$ and $f_{{\theta }_1}$, or to compute how many bits you should "get back" based on a consistency condition between them. I expect it's going to be reasonably obvious what to do in this case (likely exploiting the presumed limitation of our learning process), which is what I'll be thinking about now.

The difficulty of jointly representing $f_{{\theta }_1}$ and $H_{{\theta }_2}$ motivates my recent proposal [AF · GW], which avoids any such explicit representation. Instead it separately specifies ${\theta }_1$ and ${\theta }_2$, and then "gets back" bits by imposing a consistency condition that would have been satisfied only for a very small fraction of possible ${\theta }_2$'s (roughly $\exp (-|{\theta }_1\left|\right)$ of them).

But thinking about this neural network case also makes it easy to talk about why my recent proposal could run into severe computational problems:

• In order to calculate this loss function we need to evaluate how "special" ${\theta }_2$ is, i.e. how small is the fraction of ${\theta }_2$'s that are consistent with ${\theta }_1$
• In order to evaluate how special ${\theta }_2$ is, we basically need to do the same process of SGD that produces ${\theta }_2$---then we can compare the actual iterates to all of the places that it could have gone in a different direction, and conclude that almost all of the different settings of the parameters would have been much less consistent with ${\theta }_1$.
• The implicit hope of my proposal is that the outer neural network is learning its human model using something like SGD, and so it can do this specialness-calculation for free---it will be considering lots of different human-models, and it can observe that almost all of them are much less consistent with ${\theta }_1$.
• But the outer neural network could learn to model humans in a very different way, which may not involve representing a serious of iterates of "plausible alternative human models." For example, suppose that in each datapoint we observe a few of the bits of ${\theta }_2$ directly (e.g. by looking at a brain scan), and we fill in much of ${\theta }_2$ in this way before we ever start making good predictions about human behavior. Then we never need to consider any other plausible human-models.

So in order to salvage a proposal like this, it seems like (at a minimum) the "specialness evaluation" needs to take place separately from the main learning of the human model, using a very different process (where we consider lots of different human models and see that it's actually quite hard to find one that is similarly-consistent with ${\theta }_1$). This would take place at the point where the outer model started actually using its human model $H_{{\theta }_2}$ in order to answer questions.

I don't really know what that would look like or if it's possible to make anything like that work.

The speed prior is calibrated such that this never happens if the learned optimizer is just using brute force---if it needs to search over 1 extra bit then it will take 2x longer, offsetting the gains.

That means that in the regime where P is simple, the speed prior is the "least you can reasonably care about speed"---if you care even less, you will just end up pushing the optimization into an inner process that is more concerned with speed and is therefore able to try a bunch of options.

(However, this is very mild, since the speed prior cares only a tiny bit about speed. Adding 100 bits to your program is the same as letting it run 2^100 times longer, so you are basically just optimizing for simplicity.)

To make this concrete, suppose that I instead used the kind-of-speed prior, where taking 4x longer is equivalent to using 1 extra bit of description complexity. And suppose that P is very simple relative to the complexities of the other objects involved. Suppose that the "object-level" program M has 1000 bits and runs in 2^2000 time, so has kind-of-speed complexity 2000 bits.  A search that uses the speed prior will be able to find this algorithm in 2^3000 time, and so will have a kind-of-speed complexity of 1500 bits. So the kind-of-speed prior will just end up delegating to the speed prior.

I think multi-level search may help here.  To the extent that you can get a lower-confidence estimate of P much more quickly, you can budget your total search time such that you examine many programs and then re-examine only the good candidates.   If your confidence is linear with time/complexity of evaluation, this probably doesn't help.

This is interesting to me for two reasons:

• [Mainly] Several proposals for avoiding the instrumental policy work by penalizing computation. But I have a really shaky philosophical grip on why that's a reasonable thing to do, and so all of those solutions end up feeling weird to me. I can still evaluate them based on what works on concrete examples, but things are slippery enough that plan A is getting a handle on why this is a good idea.
• In the long run I expect to have to handle learned optimizers by having the outer optimizer instead directly learn whatever the inner optimizer would have learned.  This is an interesting setting to look at how that works out. (For example, in this case the outer optimizer just needs to be able to represent the hypothesis "There is a program that has property P and runs in time T' " and then do its own search over that space of faster programs.)

In traditional settings, we are searching for a program M that is simpler than the property P. For example, the number of parameters in our model should be smaller than the size of the dataset we are trying to fit if we want the model to generalize. (This isn't true for modern DL because of subtleties with SGD optimizing imperfectly and implicit regularization and so on, but spiritually I think it's still fine..)

But this breaks down if we start doing something like imposing consistency checks and hoping that those change the result of learning. Intuitively it's also often not true for scientific explanations---even simple properties can be surprising and require explanation, and can be used to support theories that are much more complex than the observation itself.

Some thoughts:

1. It's quite plausible that in these cases we want to be doing something other than searching over programs. This is pretty clear in the "scientific explanation" case, and maybe it's the way to go for the kinds of alignment problems I've been thinking about recently.
   
   A basic challenge with searching over programs is that we have to interpret the other data. For example, if "correspondence between two models of physics" is some kind of different object like a description in natural language, then some amplified human is going to have to be thinking about that correspondence to see if it explains the facts. If we search over correspondences, some of them will be "attacks" on the human that basically convince them to run a general computation in order to explain the data. So we have two options: (i) perfectly harden the evaluation process against such attacks, (ii) try to ensure that there is always some way to just directly do whatever the attacker convinced the human to do. But (i) seems quite hard, and (ii) basically requires us to put all of the generic programs in our search space.
    
2. It's also quite plausible that we'll just give up on things like consistency conditions. But those come up frequently enough in intuitive alignment schemes that I at least want to give them a fair shake.

Just spit-balling here, but it seems that T' will have to be much smaller than T in order for the inner search to run each candidate program (multiple times for statistical guarantees) up to T' to check whether it satisfies P. Because of this, you should be able to just run the inner search and see what it comes up with (which, if it's yet another search, will have an even shorter run time) and pretty quickly (relative to T) find the actual M.

This is also a way to think about the proposals in this post and the reply [AF(p) · GW(p)]:

• The human believes that A' and B' are related in a certain way for simple+fundamental reasons.
• On the training distribution, all of the functions we are considering reproduce the expected relationship. However, the reason that they reproduce the expected relationship is quite different.
• For the intended function, you can verify this relationship by looking at the link (A --> B) and the coarse-graining applied to A and B, and verify that the probabilities work out. (That is, I can replace all of the rest of the computational graph with nonsense, or independent samples, and get the same relationship.)
• For the bad function, you have to look at basically the whole graph. That is, it's not the case that the human's beliefs about A' and B' have the right relationship for arbitrary Ys, they only have the right relationship for a very particular distribution of Ys. So to see that A' and B' have the right relationship, we need to simulate the actual underlying dynamics where A --> B, since that creates the correlations in Y that actually lead to the expected correlations between A' and B'.
• It seems like we believe not only that A' and B' are related in a certain way, but that the relationship should be for simple reasons, and so there's a real sense in which it's a bad sign if we need to do a ton of extra compute to verify that relationship. I still don't have a great handle on that kind of argument. I suspect it won't ultimately come down to "faster is better," though as a heuristic that seems to work surprisingly well. I think that this feels a bit more plausible to me as a story for why faster would be better (but only a bit).
• It's not always going to be quite this cut and dried---depending on the structure of the human beliefs we may automatically get the desired relationship between A' and B'. But if that's the case then one of the other relationships will be a contingent fact about Y---we can't reproduce all of the expected relationships for arbitrary Y, since our model presumably makes some substantive predictions about Y and if those predictions are violated we will break some of our inferences.

So are there some facts about conditional independencies that would privilege the intended mapping? Here is one option.

We believe that A' and C' should be independent conditioned on B'. One problem is that this isn't even true, because B' is a coarse-graining and so there are in fact correlations between A' and C' that the human doesn't understand. That said, I think that the bad map introduces further conditional correlations, even assuming B=B'. For example, if you imagine Y preserving some facts about A' and C', and if the human is sometimes mistaken about B'=B, then we will introduce extra correlations between the human's beliefs about A' and C'.

I think it's pretty plausible that there are necessarily some "new" correlations in any case where the human's inference is imperfect, but I'd like to understand that better.

So I think the biggest problem is that none of the human's believed conditional independencies actually hold---they are both precise, and (more problematically) they may themselves only hold "on distribution" in some appropriate sense.

This problem seems pretty approachable though and so I'm excited to spend some time thinking about it.

The conditional probability assumed in the real world carries over to the data representation world simply because it's trying to model the same phenomenon in the real world, despite it's coarse grained nature. Without the conditional probability, we wouldn't be able to make the same strong inferences that match up to the real world. The causality is part of the data. If you use a different casual relationship, the end model would be different, and you would be solving a very different problem than if you applied the real world casual relationship.