Path dependence in ML inductive biases

post by Vivek Hebbar (Vivek), evhub · 2022-09-10T01:38:22.885Z · LW · GW · 13 comments

Contents

  Path dependence
    Diagrams for training dynamics
      Low path dependence #1:
      High path dependence #1:
      Low path dependence #2:
  Reasons to care about path dependence
      I.  Effectiveness of mid-training evaluation
      II.  Generalization of concepts and goals
      III.  Effectiveness of interventions
      IV.  Consistency across labs
      V.  Constructability of priors
      VI.  Likelihood of inner misalignment
  Specific aspects of training dynamics
      A.  Variance across runs
      B.  Sensitivity to small changes in the training setup
      C.  Existence of distinct clusters
      D.  Stability of clusters across training
      E.  Predictive power of the mid-training model
      F.  Sensitivity to early interventions
      G.  Survival of circuits
      H.  Existence of closed form inductive bias
    A question about predictiveness
  Conclusion
None
13 comments

In this post, we define path dependence as the sensitivity of a model's behavior to the details of the training process and training dynamics.^[1]  High path-dependence indicates that small changes to the training process can cause significant changes to how the final model generalizes (such as the details of off-distribution behavior).  It implies that inner alignment can be reasoned about by thinking about what the model looks like at various stages of training, and how its structure is affected by the immediate pressures of gradient descent.  It implies that early-training interventions can be quite potent in shaping how a model turns out, and that a proper theory of inductive bias must reason about the order in which features are learned (where features learned faster/earlier can “screen off” the need for other implementations of a similar niche, in a way that affects the final model).

In contrast, a world with low path dependence allows us to reason about inductive bias in terms of priors and updates, sparing the details of training dynamics.  It is more pessimistic about the ability to steer the model’s ontology through early interventions, believing instead that the final result is overdetermined.  As Evan discusses in a previous post [LW · GW], it makes us less worried about variance in alignment outcomes between labs, since small changes to the training procedure don’t strongly affect alignment outcomes.

Possible mechanistic reasons for high path dependence would include the existence of distinct stable ontologies, the ability for early features to kill gradients, and the difficulty of building highly serial features whose components aren't independently useful.  Mechanistic reasons for low path dependence would include grokking-like phase transitions [LW · GW] which wipe out early circuits, overdetermination of correct ontologies [LW · GW], and an abundance of low loss paths between seemingly dissimilar solutions.^[2] 

We remain mostly agnostic about which world we are in.  The purpose of this post is to elucidate the various strands of path dependence, and their implications for alignment.  We hope it will encourage people to run experiments determining where reality falls on these spectra.

Path dependence

Fundamentally, path dependence is about the sensitivity of a model's behavior to the details of the training process and training dynamics.  In this section, we enumerate concrete claims which constitute path dependence.  It should be emphasized that it is conceivable for these properties to vary separately, and that we lump them together because we hypothesize without proof that they are correlated.

In a world with maximum path dependence:

• The outcome of training a DNN depends strongly on what representations and circuits are learned early in training.
• This set of circuits varies between runs and is highly sensitive to the details of the training procedure.
• High-level properties of the model are very likely to stay roughly constant during the late phases of training.  We can assess these properties at the middle of training, and our assessment will be highly predictive of the final properties, even if the model gets much smarter.
• Circuits learned early in training are never outright deleted.  Parts of the task which were solved early in training continue to be done with the same circuits in the final model.
• The inductive bias doesn’t have a closed form.  To figure out which of two final models is more likely, you must reason explicitly about the order in which things are learned; you cannot just look at simplicity or depth or any other static measure of the final circuits.

By contrast, in a world with minimum path dependence:

• The outcome of training is highly consistent across runs and similar setups, and the order in which circuits are learned doesn’t matter.
• High-level properties often change drastically in the late phases of training.
• Circuits are frequently destroyed and replaced with new ones.  It is common for a new set of powerful circuits to replace an older set of clunky ad-hoc ones, and to completely displace them.
• Idealized priors like the “circuit prior [AF · GW]” might be implementable in nearly-pure form by setting up training appropriately.

Diagrams for training dynamics

Low path dependence #1:

Above, we see an simple example of low-path-dependence training dynamics -- The order in which features are learned doesn't matter, so a change to the training procedure which makes A learned faster than B won't change the final outcome.

High path dependence #1:

Here, there are two reachable end states, and early interventions can affect which one we end up in.  Features C and D are mutually exclusive, and are facilitated by A and B respectively.  There might be a grokking-like path from one of the apparent final states to the other, but training is stopped before it occurs.

If feature "D" is associated with deceptive alignment, and "C" with corrigibility, then detecting one or the other in the second-to-last phase is a reliable indicator of the final behavior.

Low path dependence #2:

In this model, there appears to be path dependence at first, but component E destroys C and D when it arises.  Any safety properties associated with C are destroyed at the last minute, despite appearing stable until then.

Reasons to care about path dependence

In this section, we go through some implications of path dependence for alignment.  In general, higher path dependence is good news for early evaluations and interventions, and (more speculatively) for being able to shape model ontology.  However, it is bad news for consistency across labs, and for solutions to inner alignment which rely on the construction of pure theoretical priors (like circuit priors).

I.  Effectiveness of mid-training evaluation

High path dependence makes mid-training evaluations more reliable, due to the "stickyness [AF · GW]"/persistence of model properties.  Suppose we want to do things like:

• Check for myopia
• Extract the goals of a system
• Check the “control structure” of a model (the broad way in which it factors the task, like a diagram made of interacting modules with particular roles)

Let's say we can do one of these things in the early and middle stages of training, when the model is somewhat simpler / less intelligent, but not on the final model.  In a high path-dependence world, there is a good chance that the result of the early check is also valid for the final model.  In a low path-dependence world, the early check is invalidated by phase transitions, circuit destruction, fundamental reorganizations of the model to an overdetermined "correct" structure, etc.

II.  Generalization of concepts and goals

A major issue in alignment is the question of how a model will generalize from the domain where we are able to label outcomes to the domain where we can't.  Here are a some examples:

• You train AIs to automate a factory.  Eventually they become smart enough that their action space includes things like "build a robot army" or "build nanotech" or "hack the internet to get more compute".  How will the goal acquired from training on simpler tasks generalize to this new level of intelligence?
• You train a question answerer, and it learns some question-answering methodology^[3].  You can use <magic method> (Christiano et al. 2025) to confirm that its internal control flow corresponds to "True honest answering [AF · GW]" w.r.t. to its current world model.  Unfortunately, <magic method> only works when the AI is below a certain intelligence threshold.  Will the internal structure generalize properly as it gets smarter, meaning that we retain the "honest answering" property?

High path-dependence is beneficial here, implying continuity of properties like "honest answering".

III.  Effectiveness of interventions

If mid-training evaluations are valid, then a further desideratum would be for mid-training interventions to be effective.  This could matter for certain forms of adversarial training—you can imagine a world where mid-training diagnostics work fine on natural models, but adversarial training at the middle stages just builds "shallow patches" which aren't well embedded in the main control flow and are erased by training dynamics in the later stages of training.  High path dependence is again desirable here.

IV.  Consistency across labs

In a high-path-dependence world, we might need to enforce a high degree of exactness in how training procedures are implemented by different labs (especially if alignment is more sensitive to changes in training procedure than capabilities are, which seems likely).  On the other hand, path dependence might help us here if alignment and capabilities are closely coupled, in the sense that copying a training procedure and changing it would also break capabilities.

V.  Constructability of priors

In a high-path-dependence world, it is hard to construct something close to a “pure” theoretical prior.  This could be bad if the most promising alignment solution ends up needing a specific prior.  Examples of theoretical priors are the Solomonoff prior (punishes program length), the circuit prior (punishes circuit size), and the speed prior (punishes serial runtime).  If SGD is approximately a Bayesian sampler,^[4] then it could be possible approximate certain combinations of these priors.  However, if path dependence is high enough, then these priors should be almost impossible to construct, due to vast differences in the findability of solutions (the existence of training paths to these solutions, and whether these paths are fast enough to not have their gradients stolen/ruined by other features).

VI.  Likelihood of inner misalignment

Arguments around the likelihood of deceptive alignment and of misaligned proxies often depend strongly on path dependence.  See Evan's recent post [AF · GW] for a thorough discussion of how path dependence affects deceptive alignment.

Specific aspects of training dynamics

At the moment, “path dependence” is really a cluster of related properties which we suspect to be correlated.  In this section, we lay out a full set of distinct concepts.  Many of these can be thought of as mechanistic sources of high/low path dependence.

A.  Variance across runs

Suppose we run training several times, with different random seeds.  How much will the final models differ from each other?  We are interested in both generalization behavior and internal representations.  At a higher capability level, we would be interested in properties like “deceptiveness” and “goal”—do these vary across training runs?

B.  Sensitivity to small changes in the training setup

How much do the properties of the final model differ if we change the training setup slightly?  For instance, if two different labs each train a language model with broadly the same architecture and approach, but all the implementation details differ, how similar will their behavior and internal properties be?

C.  Existence of distinct clusters

Suppose we train some language models (with different seeds or slightly different setups), and we look at a very specific property like:

• How does the model represent the fact ‘apples are red’?
• What does the model output on this set of 5 closely-related reasoning questions?
• What circuit is used to capitalize words at the beginning of a sentence?

If the models differ on one of these questions, there are two ways it could happen:

1. There is a spectrum of behaviors or internal setups, and we get a different point on that spectrum each time.
2. There are a small number of discrete clusters, and each time we get something which is definitively from one of these clusters.  

Hypothetical examples:

• A model either answers all five reasoning questions with “yes” or all five with “no”, but never something in between.
• The “apples are red” is always encoded in two neurons in the third layer, or one attention head in the second layer, but never any other way.  Within each way of doing it, the computation done by the neurons or attention head is always the same.

D.  Stability of clusters across training

If stability of clusters is true, this is what it would look like:

If there are two distinct clusters which a model can fall under, like “encodes apple color in neurons” vs. “encodes apple color in an attention head”, then it will fall into one of these clusters early in training, and will stay in that cluster until the end.  Each cluster will be stable to small perturbations—if we nudge the parameters slightly toward the other solution, then further training will bring us back to the center of the “valley” around the old solution.  However, if we make a strong enough intervention, we will push the model over the ridge to the other stable attractor, and it will remain in that cluster all the way through training.  So there will be a bimodal effect: either the intervention decays away almost completely, or it successfully pushes us to the other attractor and sticks fully.

E.  Predictive power of the mid-training model

Suppose that important properties of the model vary across runs or setups.  In this case, can we predict how the final model is going to turn out by pausing in the middle of training and inspecting the model?  For instance, if we’re training an RL agent, and it ends up with the wrong proxy 50% of the time, can we pause early on in training and predict which version we will get?

F.  Sensitivity to early interventions

Some examples of interventions would be:

• A small amount of adversarial training
• Injecting special training examples
• Adding a small amount of human feedback into an RL setup

The question here is: Suppose we can only intervene in the early and middle stages of training.  How sensitive are the properties of the final model to our interventions? 

G.  Survival of circuits

Closely related to (D), (E), and (F) is the survival of circuits and representations across training.  Some circuits will be learned early on and remain unchanged throughout training.  For properties associated with these circuits, the mid-training model will be perfectly predictive of the final model.  Other circuits will be wiped out and replaced with new ones.

One important question here is whether there will be some kind of "general intelligence circuitry" which develops late in training and wipes out large swathes of earlier circuits, by virtue of being more efficient in terms of "cases handled per parameter of circuitry".  Such a phenomenon would be devastating for the sort of path dependence we discuss here.

More survival of circuits means higher path dependence, since it allows the speeds at which different early circuits are formed to be more influential on the final model.

H.  Existence of closed form inductive bias

Suppose we have two computational graphs which solve a task, and we want to know which one is more likely to be isomorphic to the model resulting from training.  A closed form inductive bias lets us figure this out by inspection.  

A simple example is a circuit prior, where we simply count the number of nodes and/or connections (depending on how you define it), and say that the one with fewer nodes/connections is more likely to be found.

However, a closed form is allowed to be more complex than this.  For instance, we could have some mix of a circuit prior with depth penalties, modularity bonuses, an information loss related term, and so forth.

A non-closed-form understanding of inductive bias would be something like "Here's a rule for figuring out which circuits will be built in the first 10% of training.  Run that rule on your dataset and architecture, and write down the circuits.  Now here's a rule for what will be learned in the next 10%, which depends on the circuits we already have.  And so forth for the rest of training."

The thing which makes it not a closed form is that you have to reason through it step-by-step; you can't skip to end and say "well it all caches out to picking the simplest final solution".  This is a very path-dependent way for things to be.^[5]

If we are in such a world, then it is harder to construct a training process which gives us a "pure" circuit prior, or any other theoretical prior we might want.

A question about predictiveness

A simpler way to operationalize path dependence is to think about the following two questions:

1. How predictive of the final model are the properties at the middle of the training run?  (Consistency over time)
2. How predictive of the final model is a looking at the result of a different training run, with slightly different settings?  (Consistency between runs / Homogeneity [LW · GW])

A single axis model of path dependence assumes that these two types of consistency are anti-correlated, but it's not clear to us whether this is the case.  We leave it as an open question.

Conclusion

Given the implications of path dependence for thinking about alignment, it would be good to find out which world we're in.  Each concept (A-H) in the previous section is a yes/no question^[6] to be empirically resolved.  The exact answer will vary by task and architecture.  However, we hypothesize that there is probably some principal component across all tasks, architectures, and concepts, a "General factor of path dependence".

If you would like to run path dependence experiments, please leave a comment or send us a message, and we can get in touch.

1. ^{^}

   Mechanistically, high path dependence corresponds to significant influence and continuity from the structure and ontology of the early model to its final structure, and low path-dependence to destruction/radical transformation of early structures and overdetermination/stability of the final result.

2. ^{^}

   "Dissimilar solutions" = Different factorizations of the task into circuits.  The low-path-dependence argument here is that "paths are always abundant in high-dimensional spaces".

3. ^{^}

   I'm avoiding the term "goal" since I don't presume consequentialism.

4. ^{^}

   See one of these [LW · GW] reviews.  I mostly disbelieve the qualitative conclusions people draw from this work though, for reasons that deserve their own post.

5. ^{^}

   All this means right now is "It gives us strong path-dependence vibes, so it's probably correlated with the other stuff"

6. ^{^}

   Really a spectrum.

13 comments

Comments sorted by top scores.