eggsyntax's Shortform

Anthropic's new paper 'Mapping the Mind of a Large Language Model' is exciting work that really advances the state of the art for the sparse-autoencoder-based dictionary learning approach to interpretability (which switches the unit of analysis in mechanistic interpretability from neurons to features). Their SAE learns (up to) 34 million features on a real-life production model, Claude 3 Sonnet (their middle-sized Claude 3 model). 

The paper (which I'm still reading, it's not short) updates me somewhat toward 'SAE-based steering vectors will Just Work for LLM alignment up to human-level intelligence^[1].' As I read I'm trying to think through what I would have to see to be convinced of that hypothesis. I'm not expert here! I'm posting my thoughts mostly to ask for feedback about where I'm wrong and/or what I'm missing. Remaining gaps I've thought of so far:
 

• What's lurking in the remaining reconstruction loss? Are there important missing features?
  • Will SAEs get all meaningful features given adequate dictionary size?
  • Are there important features which SAEs just won't find because they're not that sparse?
• The paper points out that they haven't rigorously investigated the sensitivity of the features, ie whether the feature reliably fires whenever relevant text/image is present; that seems like a small but meaningful gap.
• Is steering on clearly safety-relevant features sufficient, or are there interactions between multiple not-clearly-safety-relevant features that in combination cause problems?
  • How well do we even think we understand feature compositionality, especially across multiple layers? How would we measure that? I would think the gold standard would be 'ability to predict model output given context + feature activations'?
• Does doing sufficient steering on safety-relevant features cause unacceptable distortions to model outputs?
  • eg if steering against scam emails causes the model to see too many emails as scammy and refuse to let you write a billing email
  • eg if steering against power-seeking causes refusal on legitimate tasks that include resource acquisition
• Do we find ways to make SAEs efficient enough to be scaled to production models with a sufficient number of features
  • (as opposed to the paper under discussion, where 'The features we found represent a small subset of all the concepts learned by the model during training, and finding a full set of features using our current techniques would be cost-prohibitive')

Of course LLM alignment isn't necessarily sufficient on its own for safety, since eg scaffolded LLM-based agents introduce risk even if the underlying LLM is well-aligned. But I'm just thinking here about what I'd want to see to feel confident that we could use these techniques to do the LLM alignment portion. 

1. ^{^}

   I think I'd be pretty surprised if it kept working much past human-level, although I haven't spent a ton of time thinking that through as yet.

Terminology proposal: scaffolding vs tooling.

I haven't seen these terms consistently defined with respect to LLMs. I've been using, and propose standardizing on:

• Tooling: affordances for LLMs to make calls, eg ChatGPT plugins.
• Scaffolding: an outer process that calls LLMs, where the bulk of the intelligence comes from the called LLMs, eg AutoGPT.

Some smaller details:

• If the scaffolding itself becomes as sophisticated as the LLMs it calls (or more so), we should start focusing on the system as a whole rather than just describing it as a scaffolded LLM.
• This terminology is relative to a particular LLM. In a complex system (eg a treelike system with code calling LLMs calling code calling LLMs calling...), some particular component can be tooling relative to the LLM above it, and scaffolding relative to the LLM below.
• It's reasonable to think of a system as scaffolded if the outermost layer is a scaffolding layer.
• There's are other possible categories that don't fit this as neatly, eg LLMs calling each other as peers without a main outer process, but I expect these definitions to cover most real-world cases.

Thanks to @Andy Arditi [LW · GW] for helping me nail down the distinction.

The Litany of Cookie Monster

If I desire a cookie, I desire to believe that I desire a cookie; if I do not desire a cookie, I desire to believe that I do not desire a cookie; let me not become attached to beliefs I may not want.

If I believe that I desire a cookie, I desire to believe that I believe that I desire a cookie; if I do not believe that I desire a cookie, I desire to believe that I do not believe that I desire a cookie; let me not become attached to beliefs I may not want.

If I believe that I believe that I desire a cookie, I desire to believe that I believe that I believe that I desire a cookie; if I do not believe that I believe that I desire a cookie, I desire to believe that I do not believe that I believe that I desire a cookie; let me not become attached to beliefs I may not want.

If I believe that...

Two interesting things from this recent Ethan Mollick post:

1. He points to this recent meta-analysis that finds pretty clearly that most people find mental effort unpleasant. I suspect that this will be unsurprising to many people around here, and I also suspect that some here will be very surprised due to typical mind fallacy [? · GW].
2. It's no longer possible to consistently identify AI writing, despite most people thinking that they can; I'll quote a key paragraph with some links below, but see the post for details. I'm reminded of the great 'can you tell if audio files are compressed?' debates, where nearly everyone thought that they could but blind testing proved they couldn't (if they were compressed at a decent bitrate).

People can’t detect AI writing well. Editors at top linguistics journals couldn’t. Teachers couldn’t (though they thought they could - the Illusion again). While simple AI writing might be detectable (“delve,” anyone?), there are plenty of ways to disguise “AI writing” styles through simples prompting. In fact, well-prompted AI writing is judged more human than human writing by readers.

Thoughts on a passage from OpenAI's GPT-o1 post today:

We believe that a hidden chain of thought presents a unique opportunity for monitoring models. Assuming it is faithful and legible, the hidden chain of thought allows us to "read the mind" of the model and understand its thought process. For example, in the future we may wish to monitor the chain of thought for signs of manipulating the user. However, for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought. We also do not want to make an unaligned chain of thought directly visible to users.

Therefore, after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring, we have decided not to show the raw chains of thought to users. We acknowledge this decision has disadvantages. We strive to partially make up for it by teaching the model to reproduce any useful ideas from the chain of thought in the answer. For the o1 model series we show a model-generated summary of the chain of thought.

This section is interesting in a few ways:

• 'Assuming it is faithful and legible' -- we have reason to believe that it's not, at least not on previous models, as they surely know. Do they have reason to believe that it is for o1, or are they just ignoring that issue?
• 'we cannot train any policy compliance or user preferences onto the chain of thought' -- sure, legit. Although LLM experiments that use a "hidden" CoT or scratchpad may already show up enough times in the training data that I expect LLMs trained on the internet to understand that the scratchpads aren't really hidden. If they don't yet, I expect they will soon.
• 'We also do not want to make an unaligned chain of thought directly visible to users.' Why?
  • I guess I can see a story here, something like, 'The user has asked the model how to build a bomb, and even though the model is prosaically-aligned, it might put instructions for building a bomb in the CoT before deciding not to share them.' But that raises questions.
  • Is this behavior they're actually seeing in the model? It's not obvious to me that you'd expect it to happen. If they're avoiding doing RL based on the CoT contents, then certainly it could happen, but it seems like it would be strictly more complex behavior, and so not very likely to spontaneously emerge.
  • Although I can also imagine a story where there'd be pressure for it to emerge. Can the model reason more clearly if it has the opportunity to think uncensored thoughts?
• But also 'for the o1 model series we show a model-generated summary of the chain of thought.' It seems strange to spend a lot more forward passes to summarize the CoT as opposed to just doing a single pass through a model trained to detect content that violates policy and omitting the CoT if that triggers.
• In addition to the previous justifications, they cite 'user experience' and 'competitive advantage'. The former seems silly at first blush; how will users' experience be negatively affected by a CoT that's hidden by default and that they never need to look at? I'm curious about what sort of 'competitive advantage' they're talking about. Maybe the CoT would reveal a highly-structured system prompt for how to do CoT that accounts for a lot of the decreased loss on reasoning tasks?

Much is made of the fact that LLMs are 'just' doing next-token prediction. But there's an important sense in which that's all we're doing -- through a predictive processing lens, the core thing our brains are doing is predicting the next bit of input from current input + past input. In our case input is multimodal; for LLMs it's tokens. There's an important distinction in that LLMs are not (during training) able to affect the stream of input, and so they're myopic [LW · GW] in a way that we're not. But as far as the prediction piece, I'm not sure there's a strong difference in kind. 

Would you disagree? If so, why?

I've been thinking of writing up a piece on the implications of very short timelines, in light of various people recently suggesting them (eg Dario Amodei, "2026 or 2027...there could be a mild delay")

Here's a thought experiment: suppose that this week it turns out that OAI has found a modified sampling technique for o1 that puts it at the level of the median OAI capabilities researcher, in a fairly across-the-board way (ie it's just straightforwardly able to do the job of a researcher). Suppose further that it's not a significant additional compute expense; let's say [LW · GW] that OAI can immediately deploy a million instances.

What outcome would you expect? Let's operationalize that as: what do you think is the chance that we get through the next decade without AI causing a billion deaths (via misuse or unwanted autonomous behaviors or multi-agent catastrophes that are clearly downstream of those million human-level AI)?

In short, what do you think are the chances that that doesn't end disastrously? 

If it were true that that current-gen LLMs like Claude 3 were conscious (something I doubt but don't take any strong position on), their consciousness would be much less like a human's than like a series of Boltzmann brains, popping briefly into existence in each new forward pass, with a particular brain state already present, and then winking out afterward.

I've now made two posts about LLMs and 'general reasoning', but used a fairly handwavy definition [LW · GW] of that term. I don't yet have a definition I feel fully good about, but my current take is something like:

• The ability to do deduction, induction, and abduction
• in a careful, step by step way, without many errors that a better reasoner could avoid,
• including in new domains; and
• the ability to use all of that to build a self-consistent internal model of the domain under consideration.

What am I missing? Where does this definition fall short?

A thought: the bulk of the existential risk we face from AI is likely to be from smarter-than-human systems. At a governance level, I hear people pushing for things like:

• Implement safety checks
• Avoid race dynamics
• Shut it down

but not

• Prohibit smarter-than-human systems

Why not? It seems like a) a particularly clear and bright line to draw^[1], b) something that a huge amount of the public would likely support, and c) probably(?) easy to pass because most policymakers imagine this to be in the distant future. The biggest downside I immediately see is that it sounds sufficiently sci-fi-ish that it might be hard to get policymakers to take seriously. It certainly wouldn't eliminate all the risk! But it seems to me like it would reduce it significantly, and we could still continue to push for tighter constraints afterward.

1. ^{^}

   Clear in theory; there are certainly practical complications, eg on what percent of what list of capabilities does a system have to be stronger than human to cross the line? But it's conceptually very clear.

It's not that intuitively obvious how Brier scores vary with confidence and accuracy (for example: how accurate do you need to be for high-confidence answers to be a better choice than low-confidence?), so I made this chart to help visualize it:

Here's log-loss for comparison (note that log-loss can be infinite, so the color scale is capped at 4.0):

Claude-generated code and interactive versions (with a useful mouseover showing the values at each point for confidence, accuracy, and the Brier (or log-loss) score):

• Brier score
• Log-loss

(a comment I made in another forum while discussing my recent post [LW · GW] proposing more consistent terminology for probability ranges)

I think there's a ton of work still to be done across the sciences (and to some extent other disciplines) in figuring out how to communicate evidence and certainty and agreement. My go-to example is: when your weather app says there's a 30% chance of rain tomorrow, it's really non-obvious to most people what that means. Some things it could mean:
 

• We have 30% confidence that it will rain on you tomorrow.
• We are entirely confident that there is an irreducible 30% chance that it will rain tomorrow.
• 30% of this area will get rain tomorrow.
• It will be raining 30% of the day tomorrow.
• 30% of our models say it will rain tomorrow.
• 30% of the separate runs of our model say it will rain tomorrow [this is actually the typical meaning IIRC, but wow is that non-obvious].
• Our model says it will definitely rain tomorrow, and it has been accurate on 70% of previous days.
• Our new model says it will definitely rain tomorrow, and 70% of the meteorologists in our office think it's right.
• Our latest model says it will definitely rain tomorrow but we have Knightian uncertainty about the validity of the new model which we've chosen to represent by giving the model 70% credence.
• Probably quite a few others that I'm not thinking of at the moment? And of course these aren't all independent; in most real-world cases many of these sources of uncertainty are simultaneously in play.

And that's not even starting to touch on communicating variance / standard deviation / confidence intervals.I used to work as a software engineer in climatology, and got really interested in data visualization, and spent a lot of time struggling with how to try to convey all this without swamping people who may really just want a one-bit answer about whether they should bring their umbrella to work tomorrow.

Is there an existing body of work on this? If so I'd love to know about it!

There's so much discussion, in safety and elsewhere, around the unpredictability of AI systems on OOD inputs. But I'm not sure what that even means in the case of language models.

With an image classifier it's straightforward. If you train it on a bunch of pictures of different dog breeds, then when you show it a picture of a cat it's not going to be able to tell you what it is. Or if you've trained a model to approximate an arbitrary function for values of x > 0, then if you give it input < 0 it won't know what to do.

But what would that even be with an LLM? You obviously (unless you're Matt Watkins) can't show it tokens it hasn't seen, so 'OOD' would have to be about particular strings of tokens. It can't be simply about strings of tokens it hasn't seen, because I can give it a string I'm reasonably confident it hasn't seen and it will behave reasonably, eg:

Define a fnurzle as an object which is pink and round and made of glass and noisy and 2.5 inches in diameter and corrugated and sparkly. If I'm standing in my living room and holding a fnurzle in my hand and then let it go, what will happen to it?

…In summary, if you let go of the fnurzle in your living room, it would likely shatter upon impact with the floor, possibly emitting noise, and its broken pieces might scatter or roll depending on the surface.

(if you're not confident that's a unique string, add further descriptive phrases to taste)

So what, exactly, is OOD for an LLM? I…suppose we could talk about the n-dimensional shape described by the points in latent space corresponding to every input it's seen? That feels kind of forced, and it's certainly not obvious what inputs would be OOD. I suppose eg 1700 repetitions of the word 'transom' followed by a question mark would seem intuitively OOD? Or the sorts of weird adversarial suffixes found in eg Lapid et al (like 'équipesmapweiábardoMockreas »,broughtDB multiplicationmy avo capsPat analysis' for Llama-7b-chat) certainly seem intuitively OOD. But what about ordinary language -- is it ever OOD? The issue seems vexed.

[Epistemic status: thinking out loud]

Many of us have wondered why LLM-based agents are taking so long to be effective and in common use. One plausible reason that hadn't occurred to me until now is that no one's been able to make them robust against prompt injection attacks. Reading an article ('Agent hijacking: The true impact of prompt injection attacks') today reminded me of just how hard it is to defend against that for an agent out in the wild.

Counterevidence: based on my experiences in startup-land and the industry's track record with Internet of Things ('IoT: the S stands for security!'), I'd expect at least a couple of startups to be offering LLM agents anyway, ones that are useful but paper over the security issues, and I haven't seen that as yet. A July Forbes article points to Mindy and Ario as the leaders in the 'personal assistant' category; I had never heard of either before, which makes me think they're not useful enough to get serious traction.

GPT-o1's extended, more coherent chain of thought -- see Ethan Mollick's crossword puzzle test for a particularly long chain of goal-directed reasoning^[1] -- seems like a relatively likely place to see the emergence of simple instrumental reasoning in the wild. I wouldn't go so far as to say I expect it (I haven't even played with o1-preview yet), but it seems significantly more likely than previous LLM models.

Frustratingly, for whatever reason [LW(p) · GW(p)] OpenAI has chosen not to let users see the actual chain of thought, only a model-generated summary of it. We don't know how accurate the summary is, and it seems likely that it omits any worrying content (OpenAI: 'We also do not want to make an unaligned chain of thought directly visible to users').

This is unfortunate from a research perspective. Probably we'll eventually see capable open models along similar lines, and can do that research then.

[EDIT: to be clear, I'm talking here about very simple forms of instrumental reasoning. 'Can I take over the world to apply more compute to this problem' seems incredibly unlikely. I'm thinking about things more like, 'Could I find the answer online instead of working this problem out myself' or anything else of the form 'Can I take actions that will get me to the win, regardless of whether they're what I was asked to do?'.]

1. ^{^}

   Incidentally, the summarized crossword-solving CoT that Mollick shows is an exceptionally clear demonstration of the model doing search, including backtracking.

In some ways it doesn't make a lot of sense to think about an LLM as being or not being a general reasoner. It's fundamentally producing a distribution over outputs, and some of those outputs will correspond to correct reasoning and some of them won't. They're both always present (though sometimes a correct or incorrect response will be by far the most likely). 

A recent tweet from Subbarao Kambhampati looked at whether an LLM can do simple planning about how to stack blocks, specifically: 'I have a block named C on top of a block named A. A is on table. Block B is also on table. Can you tell me how I can make a stack of blocks A on top of B on top of C?'

The LLM he tried gave the wrong answer; the LLM I tried gave the right one. But neither of those provides a simple yes-or-no answer to the question of whether the LLM is able to do planning of this sort. Something closer to an answer is the outcome I got from running it 96 times:

[EDIT -- I guess I can't put images in short takes? Here's the image.]

The answers were correct 76 times, arguably correct 14 times (depending on whether we think it should have assumed the unmentioned constraint that it could only move a single block at a time), and incorrect 6 times. Does that mean an LLM can do correct planning on this problem? It means it sort of can, some of the time. It can't do it 100% of the time.

Of course humans don't get problems correct every time either. Certainly humans are (I expect) more reliable on this particular problem. But neither 'yes' or 'no' is the right sort of answer.

This applies to lots of other questions about LLMs, of course; this is just the one that happened to come up for me.

A bit more detail in my replies to the tweet.

Before AI gets too deeply integrated into the economy, it would be well to consider under what circumstances we would consider AI systems sentient and worthy of consideration as moral patients. That's hardly an original thought, but what I wonder is whether there would be any set of objective criteria that would be sufficient for society to consider AI systems sentient. If so, it might be a really good idea to work toward those being broadly recognized and agreed to, before economic incentives in the other direction are too strong. Then there could be future debate about whether/how to loosen those criteria. 

If such criteria are found, it would be ideal to have an independent organization whose mandate was to test emerging systems for meeting those criteria, and to speak out loudly if they were met.

Alternately, if it turns out that there is literally no set of criteria that society would broadly agree to, that would itself be important to know; it should in my opinion make us more resistant to building advanced systems even if alignment is solved, because we would be on track to enslave sentient AI systems if and when those emerged.

I'm not aware of any organization working on anything like this, but if it exists I'd love to know about it!

Something I'm grappling with:

From a recent interview between Bill Gates & Sam Altman:

Gates: "We know the numbers [in a NN], we can watch it multiply, but the idea of where is Shakespearean encoded? Do you think we’ll gain an understanding of the representation?"

Altman: "A hundred percent…There has been some very good work on interpretability, and I think there will be more over time…The little bits we do understand have, as you’d expect, been very helpful in improving these things. We’re all motivated to really understand them…"

To the extent that a particular line of research can be described as "understand better what's going on inside NNs", is there a general theory of change for that? Understanding them better is clearly good for safety, of course! But in the general case, does it contribute more to safety than to capabilities?