I left another comment on my experience doing interpretability research, but I'd also like to note some overall disagreements with the post.

First, it's very important to note that GPT-4 was trained with SFT (supervised finetuning) and RLHF. I haven't played with GPT-4, but for prior models this has a large effect on the way the model responds to inputs. If the data was public, I would guess looking that the SFT and RLHF data would often be considerably more useful than pretraining. This doesn't morally contradict the post, but it's worth noting it's important to be looking at the right training set and thinking about the exact training objective!

Recall that as the cross-entropy loss of LLM steadily decreases, then the logits of the LLM will asymptotically approach the ground-truth distribution which generated the dataset. In the limit, predicting/explaining the input-output behaviour of the LLM reduces entirely to knowing the regularities in the dataset itself.

In addition to the prior objection I made about needing to look at SFT and RLHF, it's important to note that the logits of the LLM can theoretically very slowly approach the spiritually correct ground truth distribution on some sequences. For instance, suppose the LLM internally tries to predict the probability that data is from the pretraining set. If it thinks the probability is <1e-5 that it's from pretraining it puts 99% of probability mass on predicting the token 'Xf' (which is typically very uncommon) otherwise it predicts normally. If you assume the model is calibrated, then this approach only increases loss by a small amount for any model: $<-1e-5\log \left(0.01\right)=4.6e-5$. You can lower the threshold to make the increase smaller. In practice I expect that smart models will be able to get a lot of signal here, so it should be able to output 'Xf' a lot in deployment/testing while not taking much of a hit in training. I think this overall point isn't currently that important, but it might become quite relevant in the future. See also this section of a post I wrote [LW · GW].

A further disagreement is maybe have is mostly a vibes disagreement: it seems to me like we should try to train our models to try to have specific properties rather than trying to infer them from reasoning about the pretraining data. The 'train your model to do what you want using RLHF/similar' baseline is quite competitive IMO for most usecases (though it's unlikely to be sufficient for avoiding catastrophe). I'm not sure if you disagree with me. Certainly as an outside GPT-4 user the option of training the model to do what you want isn't currently available and it's less convenient even with access in many cases.

My personal experience doing interpretability research has led me to thinking this sort of consideration is quite important for interpretability on pretrained LLMs (but not necessarily for other domains idk).

I've found it quite important when doing interpretability research to have read through a reasonable amount of the training data for the (small) models I'm looking at. In practice I've read an extremely large amount of OpenWebText (mostly while doing interp) and played this prediction game a decent amount: http://rr-lm-game.herokuapp.com/ (from here: https://www.lesswrong.com/posts/htrZrxduciZ5QaCjw/language-models-seem-to-be-much-better-than-humans-at-next [LW · GW])

I've seen some kind of silly errors related to this in interpretability research. For instance, you might observe that only GPT-L/XL can do 'The Eiffel Tower is in ' and think that models have to be at that scale to know facts, but actually this is just a really useless fact on (Open)WebText relative to 'Tom Brady is the quarterback of' and similar sports facts which are much more important for good loss.

This also comes up in designing toy alignment failure settings: you ideally want your setting to work for the dumbest possible model trained on as natural of data as possible. So understanding the incentives of models trained on the actual data is quite important.

I'd recommend interpretability researchers working with models trained on next token prediction:

• Read quite a bit of the training corpus and think about what is/isn't important
• Try playing this prediction game
• Follow along with the token by token predictions of a moderately sized model looking at it's log probs and entropy token by token. It might also be good to check the predictions of a model after you guess while playing the linked prediction game (but I don't have a nice way to do this, you'd have to type manually or someone could build an app).

This is probably obvious, but maybe still worth mentioning:

It’s important to take into account the ROI per unit time. In the amount of time it would take for me to grok transformers (let’s say 100 hours), I could read ~1 million tokens, which is ~0.0002% of the training set of GPT3.

The curves aren’t clear to me, but i would bet grokking transformers would be more effective than a 0.0002% increase in training set knowledge.

This might change if you only want to predict GPT’s output in certain scenarios.

Unless I'm missing something, this seems correct, but unhelpful. It doesn't point towards how we should do anything substantive to understand, much less control AI, it just gives us a way to do better at tasks that help explain current models. Is there something you're pointing to that would make this more useful than just for prompt engineering or ad-hoc / post-hoc explanation of models we don't understand?

Agreed broadly. However, cases where it will be helpful to understand transformers include when the model acts in a very inhuman way. For example, when the model gets stuck in a loop where it repeats itself, or the way that the models struggle with wordplay, word games, rhymes and puns (due to the encoding). Also, glitch tokens.

This post is helping me with something I've been trying to think ever since being janus-pilled back in September '22: the state of nature for LLMs is alignment, and the relationship between alignment and control is reversed for them compared to agentic systems.

Consider the exchange in Q1 of the quiz: ChatGPT's responses here are a model of alignment. No surprise, given that its base model is an image of us! It's the various points of control that can inject or select for misalignment: training set biases, harmful fine-tuning, flawed RLHF, flawed or malicious prompt engineering. Whether unintentional (eg amplified representation of body shaming in the training set) or malicious (eg a specialized bot from an unscrupulous diet pill manufacturer), the misalignments stem not from lack of control, but from too much of the wrong kind.

This is not to minimize the risks from misalignment - they don't get any better just by rethinking the cause. But it does suggest we're deluded to think we can get a once-and-for-all fix by building an unbreakable jail for the LLM.

It also means - I think - we can continue to treasure the LLM that's as full a reflection of us as we can manage. There are demons in there, but our best angels too, and all the aspirations we've ever written down. This is human-aligned values at species scale - in the ideal; there's currently great inequality in representation that needs to be fixed - something we ourselves have not achieved. In that sense, we should also be thinking about how we're going to help it align us.

Answer to question 4 is that they were trying to define the Luigi behavior by explicitly describing Waluigi and saying to be the opposite. Which does not work well even with humans. (When someone is told not to think about something, it can actually increase the likelihood that they will think about it. This is because the brain has difficulty processing negative statements without also activating the associated concept. In the case of the "white monkey" example, the more someone tries to suppress thoughts of a white monkey, the more frequently and intensely they may actually think about it - from GPT's explanation of white monkey aka pink elephant phenomenon)

I think referring Waluigi can only work well if it is used only in the separate independent "censor" AI, which does not output anything by itself and only suppresses certain behaviour of the primary one. As the Bing and character.ai already do, it seems.

I guess yeah. The more general point is that AIs get good at something when they have a lot of training data for it. Have many texts or pictures from the internet = learn to make more of these. So to get a real world optimizer you "only" need a lot of real world reinforcement learning, which thankfully takes time.

It's not so rosy though. There could be some shortcut to get lots of training data, like AlphaZero's self play but for real world optimization. Or a shortcut to extract the real world optimization powers latent in the datasets we already have, like "write a conversation between smart people planning to destroy the world". Scary.

FWIW, back in the 1970s and 1980s David Marr and Thomas Poggio argued that large complex 'information processing systems' need to be analyzed and described on several levels, with the higher levels being implemented in the lower levels. Just what and how many levels there are has varied according to this and that, but the principle is alive and kicking. David Chapman makes a similar argument about levels: How to understand AI systems. 

Why should this be the case? Because ANNs in general are designed to take on the structure inherent in the data they absorb during training. If they didn't, they'd be of little value. If you want to understand how a LLM tells stories, consult a narratologist.

I've made the levels argument, with reference to Marr/Poggio, in two recent working papers:

• ChatGPT intimates a tantalizing future; its core LLM is organized on multiple levels; and it has broken the idea of thinking
• ChatGPT tells stories, and a note about reverse engineering

Here's the abstract of the second paper:

I examine a set of stories that are organized on three levels: 1) the entire story trajectory, 2) segments within the trajectory, and 3) sentences within individual segments. I conjecture that the probability distribution from which ChatGPT draws next tokens follows a hierarchy nested according to those three levels and that is encoded in the weights of ChatGPT’s parameters. I arrived at this conjecture to account for the results of experiments in which ChatGPT is given a prompt containing a story along with instructions to create a new story based on that story but changing a key character: the protagonist or the antagonist. That one change then ripples through the rest of the story. The pattern of differences between the old and the new story indicates how ChatGPT maintains story coherence. The nature and extent of the differences between the original story and the new one depends roughly on the degree of difference between the original key character and the one substituted for it. I conclude with a methodological coda: ChatGPT’s behavior must be described and analyzed on three levels: 1) The experiments exhibit surface level behavior. 2) The conjecture is about a middle level that contains the nested hierarchy of probability distributions. 3) The transformer virtual machine is the bottom level.

Add in some complex dynamics, perhaps some conceptual space semantics from Peter Gärdenfors, and who knows what else, and I expect the mystery about how LLMs work to dissipate before AGI arrives.

You are saying hire more poets, though, in fact, and you should be saying it more!

If the Chris Olah chart is true, the natural abstraction hypothesis [LW · GW] is probably false, if the NAH is false, alignment (of superhuman models) would be considerably more difficult.

This is a huge deal!

Question 1 misses the point.

Very rarely does a synonym literally means the same thing.

Your example is a good one... thin and skinny.

The likeness here is only superficial, as in both persons would be on the lower end of weigth for their height.

Skinny does invoke some kind of "unhealthy" in usage:

From, the Oxford dictionary:
adjective

1. (of a person or part of their body) unattractively or unusually thin.

(emphasis mine)

As such, the output is what you would expect if you .. learn about the world, not about transformers.

I agree that as users of a black box app, it makes sense to think this way. In particular, I'm a fan of thinking of what ChatGPT does in literary terms.

But I don't think it results in satisfying explanations of what it's doing. Ideally, we wouldn't settle for fan theories of what it's doing, we'd have some kind of debug access that lets us see how it does it.

Yup, (something like) the human anchor seems surprisingly good as a predictive model when interacting with LLMs. Related, especially for prompting: Large Language Models Are Implicitly Topic Models: Explaining and Finding Good Demonstrations for In-Context Learning; A fine-grained comparison of pragmatic language understanding in humans and language models; Task Ambiguity in Humans and Language Models.