Measuring Nonlinear Feature Interactions in Sparse Crosscoders [Project Proposal]

post by Jason Gross (jason-gross), rajashree (miraya) · 2025-01-06T04:22:12.633Z · LW · GW · 0 comments

Contents

  TL;DR
  Introduction
    Linear Interaction Assumption
    Measuring nonlinear feature interactions
  Theoretical intuition
    The Math
      Defining sparse crosscoders
      Bounding reconstruction error
        The crosscoder achieves low loss
        The crosscoder and the model correspond
        The base case
        The recursive case: this is where the magic happens
  Feature interaction metrics
    Probabilistic bounds or proof bounds
    For ReLU
    For softmax-attention
      Cross-position interaction
      Cross-feature interaction
  Applications
    Model diffing
    Describe the model mechanisms in more detail
    Suppress undesired interactions
    Adversarial examples
    Training objective of the crosscoder
  Discussion
    Crosscoders vs. SAEs
    The encoder and the decoder should not be treated symmetrically
      Information content
      Meaning: we don't need causal crosscoders
    Mechanistic faithfulness and measuring the richness of the crosscoder explanation
    Crosscoder errors measure violation of the linear representation hypothesis
  Acknowledgements
  Citation
None
No comments

TL;DR

• Problem: Sparse crosscoders are powerful tools for compressing neural network representations into interpretable features. However, we don’t understand how features interact.

• Perspective: We need systematic procedures to measure and rank nonlinear feature interactions. This will help us identify which interactions deserve deeper interpretation. Success can be measured by how useful these metrics are for applications like model diffing and finding adversarial examples.

• Starting contribution: We develop a procedure based on compact proofs [AF · GW]. Working backwards from the assumption that features are linearly independent, we derive mathematical formulations for measuring feature interactions in ReLU and softmax attention.

Introduction

Training a sparse crosscoder (henceforth simply crosscoders) can be thought of as stacking the activations of the residual stream across all layers and training an SAE on the stacked vector. This gives us a mapping between model activations and compressed, interpretable features. We’re excited about this approach to automated interpretability, as it paves the way for accounting for feature interaction in addition to feature discovery.

In this draft project proposal, we describe the linear interaction assumption, and provide a compact-proofs-based explanation for why we’d like to measure feature interaction. We are sharing concrete metrics for measuring feature interaction in ReLU and softmax attention. We close with a discussion of empirical projects that we’re excited about, and some interesting takeaways. Please reach out to us if you have feedback or would like to collaborate!

Linear Interaction Assumption

Crosscoders would be all we need if we could make the following two assumptions about how neural networks process information:

1. Linear Representation Hypothesis: Features are encoded explicitly and linearly in the residual stream. For example, if a model needs to work with geometric areas, this hypothesis suggests the area value would be directly encoded, rather than linearly storing just the radius and computing or nonlinearly extracting $\text{area}=\pi r^2$ when needed.

2. Linear Interaction Assumption: Features are independent. This assumption suggests that the nonlinear computation in the model distributes over features in some approximately linear way.

If the linear representation hypothesis breaks down, then crosscoders would perform poorly, getting high reconstruction error. However, even if a crosscoder achieves 100% reconstruction accuracy, it may still be the case that the linear interaction assumption does not hold for the model, and thus that the crosscoder explanation is incomplete.

Consider a concrete example: Suppose we prompt the model with “The first letter of January is”. To complete this task, an attention head needs to:

1. Look at the word “January”
2. Extract its first letter
3. Move this information to the position after “is”

Before this attention operation occurs, the residual stream at the “is” position cannot predict what will appear there after the operation, because the crucial information (“J”) is stored at a different position. The crosscoder might detect that “J” appears in the output, but it misses the complex interaction between:

• The attention mechanism that knows to look for “January”
• The position-specific information about where “January” is located
• The logic for extracting the first letter

Measuring nonlinear feature interactions

The open problem is measuring how strongly features interact in ways that violate the linear interaction assumption. The basic idea is to:

1. Look at pairs of features that co-occur in the model’s activations on a given dataset
2. Measure how much their effects on the model’s behavior interact/interfere
3. Rank these interactions by strength and frequency

This will give us a precise way to identify where our current feature-based understanding is incomplete, where we need some notion of crosscoder circuits. For each interaction, we can ask:

• Which features are involved?
• At which layer/location do they interact?
• How strong is the interaction?
• How often does it occur?

By focusing on the strongest and most frequent nonlinear interactions first, we get a principled way to gradually improve our understanding of the model.

Theoretical intuition

We start by providing theory-grounded intuition for feature interaction. We use the compact proofs approach, where the goal for mechanistic understanding is to compress the length of the explanation of the model (the proof of the model’s behavior).

If crosscoder features genuinely have only linear interactions, then we can prove a tight bound with linear work. On the other hand, if crosscoder features interact strongly, then the linear proof will give vacuous bounds and making the bounds tighter will bring the proofs much closer to brute-force.

The proof workflow is as follows:

1. We treat the crosscoder as producing an encoding in features of the dataset and treat the decoder as approximating the model

2. We seek to bound the reconstruction error between the logits and the unembed applied to the decoded last-layer activations

3. We can perform exhaustive enumeration on the entire data distribution, but this is as expensive as running the entire model on every datapoint.

4. Instead, we may be able to reduce the time complexity of this bound

   1. by assuming that the crosscoder contains the full extent of our understanding of the model (what we do in this blog post), and
   2. by clustering the input distribution into sets of datapoints that behave similarly^[1]

The next subsection discusses the math in detail; feel free to skip.

The Math

Defining sparse crosscoders

Training a sparse crosscoder can be thought of as stacking the activations of the residual stream across all layers and training an SAE on the stacked vector. Quoting Anthropic:

The basic setup of a crosscoder is as follows. First, we compute the vector of feature activations $f\left(x_j\right)$ on a data point $x_j$ by summing over contributions from the activations of different layers $a^l\left(x_j\right)$ for layers $l\in L$:

$$f\left(x_j\right)=\text{ReLU}\left(\sum _{l\in L}{W}_{\text{enc}}^l{a}^l\right(x_j)+b_{\text{enc}})$$

where $W_{\text{enc}}^l$ is the encoder weights at layer $l$, and $a^l\left(x_j\right)$ is the activations on data point $x_j$ at layer $l$. We then try to reconstruct the layer activations with approximations ${a^{\prime }}^l\left(x_j\right)$ of the activations at layer $l$:

$${a^{\prime }}^l\left(x_j\right)=W_{\text{dec}}^{l}f\left(x_j\right)+b_{\text{dec}}^l$$

Now let us define our architecture hyperparameters. Consider:

• $M_0$, a transformer with
• $v$ vocab tokens,
• $|L|$ layers, and
• $d$ dimensions in the residual stream, so that
• $|L|d$ is the dimension of the stacked activation vector. Let
• $X$ be the (finite) set of data points, underlying
• $D$, the distribution, and let
• $|X|$ denote the number of samples in $X$ and let
• $\overline{\ell }$ denote the average number of tokens in data points in $X$ (averaged over the uniform distribution rather than with respect to $D$), so that
• $|X|\overline{\ell }$ is the number of tokens in the dataset.

Suppose our crosscoder has

• $n$ features (the width) and
• $k$ as its sparsity (L0 norm), so that
• $W_{enc}^l$ and $W_{dec}^l$ generally have shape $d\times n$ and $n\times d$ respectively. Let
• $W_E$ and $W_U$ be the embedding and unembedding matrices of dimensions $v\times d$ and $d\times v$ and respectively.

We will sweep the details of positional encoding under the rug. Let

• $p$ be the floating-point precision we are using.

Bounding reconstruction error

Our proof will consist of two parts^[2]: first the proof that the crosscoder (and therefore, we hope, the model) achieves low loss on the distribution $D$; and second the proof that the crosscoder and the model correspond.

The crosscoder achieves low loss

Take the set of data points $X$ and exhaustively encode them into the feature space by running the encoder $f$, let us say $\widetilde{x_j}=f\left(x_j\right)$.^[3] This data will require $O\left(\right({\log }_2(\frac{n}{k})+kp\left)|X|\overline{\ell }\right)$ bits to encode naïvely. To exhaustively establish low loss on the distribution of interest, we simply compute our approximation for the loss on data point $x_j$ as $W_U(W_{\text{dec}}^{|L|}\widetilde{x_j}+b_{\text{dec}}^{|L|})$ for each data point $\widetilde{x_j}$. This costs $O\left(dv|X|\overline{\ell }\right)$ floating point operations. We can take a weighted average over $D$ to establish the overall expected loss.

The crosscoder and the model correspond

We must now establish that the crosscoder and its feature vectors are actually good approximations of the model activations. That is, we want to compute an upper bound on $${\epsilon }_f^U(x_j,\widetilde{x_j})=\Vert W_U\left({a^{\prime }}^{|L|}\right(x_j)-a^{|L|}(x_j\left)\right)\Vert =\Vert W_U(W_{\text{dec}}^{|L|}\widetilde{x_j}+b_{\text{dec}}^{|L|}-a^{|L|}(x_j\left)\right)\Vert$$

We must ultimately compute this error recursively in layers, where the error term on layer $l$ is computed in terms of the error term on layer $l-1$.^[4] That is, we want to establish, for each layer $l$, an upper bound on $${\epsilon }_f^l(x_j,\widetilde{x_j})=\Vert a^l\left(x_j\right)-{a^{\prime }}^l\left(x_j\right)\Vert$$ likely by computing this bound recursively in terms of ${\epsilon }_f^{l-1}(x_j,\widetilde{x_j})$.^[5]

The base case

This procedure bottoms out in establishing how far the true token embed sequences are from the crosscoder approximation: $${\epsilon }_f^0(x_j,\widetilde{x_j})=\Vert W_{\text{dec}}^0\widetilde{x_j}+b_{\text{dec}}^0-W_E{x}_j\Vert$$ and then we might want to find an upper bound on this error $${\epsilon }_{f,max}^0\ge \underset{j}{max}{\epsilon }_f^0(x_j,\widetilde{x_j})$$ Note that we are sweeping under the rug details of the positional embedding of the sequence of tokens in $x_j$ here.

We can compute this embedding bound exhaustively in time $O\left(\right(dv+dk\left)|X|\overline{\ell }\right)$.

The recursive case: this is where the magic happens

If we have to compute all the error terms exhaustively, we will have gained nothing compression-wise. However, by making an additional assumption that is only somewhat unrealistic, we can compute this error in only as many forward passes as the square of our feature count (time $O\left(|L|d^2{n}^2\right)$)^[6].

The additional assumption we need is the linear interaction assumption, that features interact only in linear ways. This means, for example, that not only do we need the residual stream to be a sum of linearly represented features, but also that the ReLU of a sum of these features must be the sum of the ReLUs, or at least not far off from that.^[7] Violations of this assumption are sure to abound, and define the scope of what interpretability work remains to be done, once we have crosscoders.

Feature interaction metrics

When we seek to bound the reconstruction error systematically in time $O\left(\right(\text{\# features}{)}^2\cdot \left(\text{cost of forward pass}\right))$, we are forced to compute a crude uniform bound on the interaction between features. We can measure how crude this computation is by measuring the size of the error terms it introduces, giving us a metric on feature interaction strength.

Probabilistic bounds or proof bounds

Broadly, two approaches are possible: probabilistic (average-case over model weights) bounds, and proof-based (worst-case) bounds. We will briefly sketch out the metrics grounded in probabilistic bounds, which we may develop more in the future, before diving into the details of proof-based bounds.

If we ground our metrics in probabilistic bounds, then we want to measure deviation from the feature independence hypothesis: that features can be well-modeled as independent random variables. We can, for example:

• fit the feature activations to a multidimensional Gaussian
• use the off-diagonal entries of the covariance matrix as a metric on feature independence

Alternatively, we can:

• analytically evaluate nonlinearities on a binned distribution under the assumption that the features are independent^[8]
• measure the KL divergence between this distribution and the actual distribution of activations

If we ground our metrics in worst-case (proof) bounds, then we want to measure deviation from the linear feature interaction assumption: that nonlinearities distribute over features in approximately linear ways. Testing this assumption requires a different procedure for each kind of nonlinearity.

In the next subsections, we discuss ReLU and softmax-attention: broadly, we test the hypothesis that co-occurring features at each neuron do not interact at all; and that co-occurring features at the same attention head obey a sort of strict dominance principle where the head as a whole behaves as if only the most strongly activating feature were present.

For ReLU

Applying the compact-proof based approach, we define the following procedure for measuring feature interaction for ReLUs, testing the hypothesis that co-occurring features at each neuron do not interact at all.

Given a pair of features $f_i$, $f_j$ that decode at layer $l$ to $W_{\text{dec}}^l{f}_i+b_{\text{dec}}^l$ (and similarly for $f_j$), if the MLP input matrix is $W_{\text{in}}^l$, we can measure the interaction between $f_i$ and $f_j$ at a neuron $m$ via their elementwise absolute ratios, averaged over the empirical distribution of their strengths ${\alpha }_i$, ${\alpha }_j$. Taking $\tilde{i}:={\mathrm{argmax}}_{i,j}\left|\right(W_{\text{in}}^l{W}_{\text{dec}}^l{f}_i{)}_m|$ and $\tilde{j}:={\mathrm{argmin}}_{i,j}\left|\right(W_{\text{in}}^l{W}_{\text{dec}}^l{f}_i{)}_m|$, we may define the interaction strength to be $$s_{i,j,l,m}:={\left(\frac{W_{\text{in}}^l{W}_{\text{dec}}^l{f}_{\tilde{j}}}{W_{\text{in}}^l{W}_{\text{dec}}^l{f}_{\tilde{i}}}\right)}_m$$ so that we may express the neuron-$m$ preactivation from the linear combination of ${\alpha }_i{f}_i+{\alpha }_j{f}_j$ as $${\alpha }_{\tilde{i}}(W_{\text{in}}^l{W}_{\text{dec}}^l{f}_{\tilde{i}}{)}_m(1+\frac{{\alpha }_{\tilde{j}}}{{\alpha }_{\tilde{i}}}{s}_{i,j,l,m})$$ Then we might measure the overall interaction strength by averaging over the dataset: $$E_{{\alpha }_i,{\alpha }_j}\left[\frac{{\alpha }_{\tilde{j}}}{{\alpha }_{\tilde{i}}}\left|s_{i,j,l,m}\right|\right]$$

We can then bound the postactivations by the triangle inequality.

(A simpler but less useful measure would be the norm of the Hadamard (elementwise) product $\left(W_{\text{in}}^l\right(W_{\text{dec}}^l{f}_i+b_{\text{dec}}^l\left)\right)\odot \left(W_{\text{in}}^l\right(W_{\text{dec}}^l{f}_j+b_{\text{dec}}^l\left)\right)$.)

For softmax-attention

Applying the compact-proof based approach, we define the following procedure for measuring feature interaction for softmax attention, testing the hypothesis that co-occurring features at the same attention head obey a sort of strict dominance principle where the head as a whole behaves as if only the most strongly activating feature were present. Because attention mixes information at multiple positions in the stream, we have more interaction terms than for ReLU MLPs.

Cross-position interaction

Given a pair of features $f_i$, $f_j$ that decode at layer $l$ to $W_{\text{dec}}^l{f}_i+b_{\text{dec}}^l$ (and similarly for $f_j$), ignoring biases, we may express the QK interaction term with $i$ in the query position as $\left(W_{\text{dec}}^l{f}_i\right)Q{K}^T(W_{\text{dec}}^l{f}_j{)}^T$. Features interact insofar as this value is different for these features in particular than for the average of this term across features.

Cross-feature interaction

Two features can be said to interact as queries insofar as they interact with overlapping sets of keys.

Applications

How does measuring interactions help? There are several ideas that we’d love to test out:

Model diffing

In Stage-Wise Model Diffing, Bricken et al. propose a procedure for detecting what features change the most when a model is fine-tuned. We can build on this method to answer the questions:

1. Which features have the strongest interactions with the features that change under fine-tuning? Broadly, this provides an automated method for understanding the algorithms that models implement in order to learn the new objective.

2. Which feature interactions change the most during fine-tuning, even or especially when the features themselves do not change? While simple behavioral changes are likely to show up in the features themselves, more subtle or complicated changes may show up only or primarily in the ways features interact. Broadly, this allows us to make our detection and control methods more robust to adversarial pressure.

Describe the model mechanisms in more detail

We can use the ranked list of feature-location triplets as a guide for which parts of the model have the most unexplained dimensionality. We might be able to visualize the model’s circuits just by color-coding features and making a plot of feature interaction strength vs location in the model. Potentially, we will develop the mathematics necessary to find even cleaner abstractions of model computation.

There are probably a number of clusters of similar sorts of nonlinear feature interactions. For example, we might expect to see:

• some features that interact only by worst-case (proof) metrics, not by probabilistic metrics, or vice versa
• features whose encoder (and/or decoder) have high cosine similarity and are therefore likely to interact (we may see this in feature splitting, for example)
• some features that are “obviously supposed to interact” and do so in simple ways (for example, a “UK English vs US English” feature might interact strongly with features for expressions that differ between these dialects, but only very early and/or very late in the model)

Suppress undesired interactions

We can edit or fine-tune a language model to remove undesired feature interactions.

• We can ablate the overlap of the two features entirely, assigning each dimension to one of the two interacting features and suppressing the other one.

• Alternatively, we can use the feature interaction metric as a penalty in the loss function for the specific features of interest, and fine-tune the language model to suppress their interaction.

Adversarial examples

Some pairs of features might co-occur very rarely (or never), but have strong interaction — these features can perhaps be leveraged to generate adversarial examples, especially the lack of co-occurrence is an artifact of the dataset.

Training objective of the crosscoder

The methods for measuring feature interactions can be added as penalties to the crosscoder training loss function. While adding this penalty with a large coefficient would probably distort the features and increase reconstruction error significantly, a weak penalty should encourage the crosscoder to eliminate spurious feature interactions.

Discussion

Anthropic’s Sparse Crosscoders for Cross-Layer Features and Model Diffing post closes with some interesting questions. We respond to these questions from the compact proofs frame.

Crosscoders vs. SAEs

From the compact proofs frame, crosscoders seem clearly much better than SAEs in the sense that they should get at features the model actually uses much more cleanly. The above methodology for generating proofs does not work for SAEs without a correspondingly good story for SAE circuits, and even with circuits it’s not entirely clear how to either validate and leverage or do away with the linear representation hypothesis or anything similar to the linear interaction assumption.

The encoder and the decoder should not be treated symmetrically

Information content

In the proof, we must store the encoded dataset, and we must perform a computation using the decoder. This means that when measuring information, measuring entropy of the entries of the decoder matrix is fine, but for the encoder, we should instead consider the entropy of the encoded dataset.

Meaning: we don't need causal crosscoders

This asymmetry suggests that the encoder is relevant only to the quality of the features, and the interpretation / meaning of the features lives in the decoder. That is, using causal crosscoders or other fancier schemes may give us more accurate features, but they do not change the interpretation of the features we get, except inasmuch as they change how features interact.

Mechanistic faithfulness and measuring the richness of the crosscoder explanation

In the compact proofs approach, we measure the richness of an explanation by the length of the proof. A crosscoder-based proof has overall length approximately $O\left(k\right(\text{vocab size}\left)\right(\text{dataset size})+n^2(\text{cost of forward pass}\left)\right)$ where $k$ is the (L0) sparsity and $n$ is the crosscoder width (number of features).

More precisely, we have^[9] $$\begin{array}{cc}& O(\underset{\text{compressed dataset rep.}}{\underset{}{\left(\log \right({}_n{C}_k)+kp)|X|\overline{\ell }}}+\\ & O(\underset{\text{compressed dataset rep.}}{\underset{}{\left(\log \right({}_n{C}_k)+kp)|X|\overline{\ell }}}\underset{\begin{array}{c}\text{floating point}\\ \text{precision cost}\end{array}}{\underset{}{p\log p}}\cdot (\underset{\text{decoding}}{\underset{}{dv|X|\overline{\ell }}}+\underset{\begin{array}{c}\text{embed decoding}\\ \text{error}\end{array}}{\underset{}{(dv+dk)|X|\overline{\ell }}}+\underset{\begin{array}{c}\text{error}\\ \text{prop}\end{array}}{\underset{}{|L|d^{2}n}}+\underset{\begin{array}{c}\text{linear}\\ \text{interaction}\\ \text{errors}\end{array}}{\underset{}{|L|d^2{n}^2}}+\underset{\begin{array}{c}\text{nonlinear}\\ \text{feature}\\ \text{circuits}\end{array}}{\underset{}{?}}))\end{array}$$ or, collecting terms and taking ${}_n{C}_k=(\frac{n}{k})\approx n^k$ for $k\ll n$, $$O\left(\right(k\log n+(v+k)dp\log p)\underset{\begin{array}{c}\text{dataset}\\ \text{size}\end{array}}{\underset{}{|X|\overline{\ell }}}+n^2\underset{\text{forward pass}}{\underset{}{d^2|L|p\log p}}+\underset{\begin{array}{c}\text{nonlinear}\\ \text{feature circuits}\end{array}}{\underset{}{?}})$$ Note that the leading dataset-size-dependent asymptotic term here is the cost of embedding the dataset (in the regime where $v\gg k$). If we include the pre-embed and post-unembed vectors in the stacked activations, we can avoid this term and the leading dataset-size-dependent asymptotic term becomes $O\left(kv|X|\overline{\ell }\right)$ (or $O\left(k\right(d+v+\log n\left)|X|\overline{\ell }\right)$ if you want to be slightly more precise).

The leading asymptotic term that is independent of dataset-size is $O\left(n^2\right(\text{cost of forward pass}\left)\right)$.

Recall that the brute-force “run inference on the dataset” baseline cost is $O\left(\underset{\begin{array}{c}\text{dataset}\\ \text{size}\end{array}}{\underset{}{|X|\overline{\ell }}}\underset{\text{forward pass}}{\underset{}{d^2|L|p\log p}}\right)$.

We can imagine plotting a pareto frontier of mechanistic faithfulness (tightness of bound, whether probabilistic or proofs-based) vs cost of computation, and measure the richness of crosscoder explanations by looking at how much these explanations improve on the baseline of running model inference.

Crosscoder errors measure violation of the linear representation hypothesis

While crosscoder “error features” allow for a kind of exact isomorphism with the original model, they take significantly more compute to account for than actual features. This is the formal analogue of these “error features” being potentially extremely difficult to interpret. One might see the technical content of this post as being centrally about addressing the question of interpreting crosscoder error features:

• If the error features are large at a given compute budget, this suggests that there is no simple linear decomposition of the model’s representations at that level of abstraction. Insofar as the errors are large across a wide range of compute budgets, this would be evidence against the linear representation hypothesis.

• If the error features are small but our bounds on them are loose, this suggests that features are interacting nonlinearly, that crosscoders fail to fully capture feature interaction, and we need to discover crosscoder circuits.

Acknowledgements

Thanks to Paul Christiano for pointing out that we can assume independence of features instead of noninteraction of features as a starting point. Thanks to Adria Garriga-Alonso, Neel Nanda, Louis Jaburi, and Kola Ayonrinde for discussion and comments on a draft of this post.

Citation

Please cite as:

Gross, Jason and Agrawal, Rajashree, “Measuring Nonlinear Feature Interactions in Sparse Crosscoders [Project Proposal]", AI Alignment Forum, 2025.

BibTeX Citation:

@article{gross2025measuring,
    title={Measuring Nonlinear Feature Interactions in Sparse Crosscoders [Project Proposal]},
    author={Jason Gross and Rajashree Agrawal},
    year={2025},
    journal={AI Alignment Forum},
    note={\url{https://www.alignmentforum.org/posts/RjrGAqJbk849Q7PHP/measuring-nonlinear-feature-interactions-in-sparse}}
}

---

1. This is beyond the scope of this post. ARC Theory is currently working on automating heuristic argument discovery following roughly this direction, though, and if you believe the SLT folks that symmetries define all interesting structure, then this sort of clustering may be essentially all you need to automate the remainder of the interpretability work. ↩︎

2. To make this post self-contained, we include in this footnote a review of the compact proofs approach: In the compact proofs approach, we write down a theorem saying that some particular model $M_0$ does what it does on the distribution $D$ of interest: $$E_{(x,y)\sim D}\left[L\right(M_0\left(x\right),y\left)\right]\le b_0$$ where $D$ is the distribution of input-label pairs (such as The Pile or OpenWebText or the SCC training distribution), $L$ is the loss function, and $b_0$ is the upper bound on loss we wish to prove.
   To prove this theorem, we first construct a function $b\left(M\right)$ which computes a valid bound for any model weights. A proof of model performance then consists of a proof that $b\left(M\right)$ gives valid upper bounds for all model weights $M$ together with a proof that on some particular model $M_0$ we have $b\left(M_0\right)\le b_0$ for some concrete $b_0$ (constructed by running $b$). Mathematically, we want to prove the two inequalities: $$\begin{array}{c}\\ \forall M.E_{(x,y)\sim D}\left[L\right(M\left(x\right),y\left)\right]\le b\left(M\right)\\ \\ b\left(M_0\right)\le b_0\end{array}$$ The first proof we do in the standard way (in LaTeX or a proof assistant or pen-and-paper), and the second proof consists of a transcript of running the program $b$ (which may be written in PyTorch, for example). In general, the length of the proof can be well-approximated (asymptotically) by the running time of $b$.
   The baseline approach is to just run model inference on some fraction of the points in $D$. For the other points, we merely need to establish cheaply that the model never produces $\pm \infty$ nor $\text{NaN}$. (I hope this can be done by bounds propagation and/or standard neural network verification tools, but we’ve not yet tested this approach for establishing the baseline for loss rather than accuracy.) The linear baseline is then just a line between the true loss of the model (which requires running inference on the full distribution) and the largest finite float (which should not cost more than a couple forward passes).
   The compact proofs approach says that having a rich understanding consists of being able to get a better tradeoff between lossiness (tightness of bound $b_0$) and compression ratio (proof length or running time of $b$).
   Note that in general there are two steps to any interpretation: finding the interpretation, and communicating (or presenting or validating) it. As is typical in estimating complexity, we don’t care how long it takes to find an interpretation, only how long it takes to communicate it. (In our case, we “communicate” the interpretation to a proof verifier.) ↩︎

3. As far as proof length — measuring the richness of understanding — is concerned, the computational cost of encoding the dataset is irrelevant; the proof strategy is valid for any feature vector, and incorrectly encoded feature vectors will simply result in loose bounds. ↩︎

4. We may divide the computation up into clusters of datapoints that behave similarly, and compute the error separately on each cluster. The brute force approach has one datapoint per cluster, and the error is tight. The most compact proof we can get without additional insight over the crosscoder has a single cluster for all datapoints. ↩︎

5. Note that if there’s a location where the error estimate is bad, we can insert more intermediate points into the cross-coder training — I imagine that every post-nonlinearity would be useful, as might the $q$, $k$, and $v$ vectors. If we insert enough intermediate points, I expect the recursive estimate to be relatively tight. ↩︎

6. Note that if we relax from worst-case to average case and go for probabilistic bounds, we can assume independence of features and independence of errors and compute a binned distribution of features which we push through each of the layers in sequence, giving something like $O\left(n\right(\text{cost of forward pass}\left)/{\epsilon }^2\right)$ where $1/\epsilon$ is the number of bins. ↩︎

7. This is why I am so excited about crosscoders: for the first time, the explanation captured by an automated interpretability procedure is close enough to a proof that we can write out an almost reasonable proof with a clear picture of what holes remain to be filled, rather than having most of the proof be unspecified! ↩︎

8. ARC Theory has shown how to do this in the context of VAEs. ↩︎

9. We take $p\log p$ to be the asymptotic cost of multiplying $p$-bit floating-point numbers. ↩︎

0 comments

Comments sorted by top scores.