The Plan - 2022 Update

theoretical progress has been considerably faster than expected, while crossing the theory-practice gap has been mildly slower than expected. (Note that “theory progressing faster than expected, practice slower” is a potential red flag for theory coming decoupled from reality

I appreciate you flagging this. I read the former sentence and my immediate next thought was the heuristic in the parenthetical sentence.

by six months ago I was hoping to have efficient algorithms for computing natural abstractions in simulated environments

Can you provide an example of what this would look like?

• Both Jeffery Andrade and myself tried to calculate natural abstractions in the Game of Life, which basically did not work.
• I tried to calculate “local” natural abstractions (in a certain sense) in a generative image net, and that worked quite well.

What are some examples of natural abstractions you were looking for, and how did you calculate or fail to calculate them?

It's great to hear that you have updated away from ambitious value learning towards corrigibility-like targets. It sounds like you now find it plausible that corrigibility will be a natural concept in the AI's ontology, despite it being incompatible with expected utility maximization. Does this mean that you expect we will be able to build advanced AI that doesn't become an expected utility maximizer?

I'm also curious how optimistic you are about the interpretability field being able to solve the empirical side of the abstraction problem in the next 5-10 years. Current interpretability work is focused on low-level abstractions (e.g. identifying how a model represents basic facts about the world) and extending the current approaches to higher-level abstractions seems hard. Do you think the current interpretability approaches will basically get us there or will we need qualitatively different methods? 

Any updates to your model of the socioeconomic path to aligned AI deployment? Namely:

• Any changes to your median timeline until AGI, i. e., do we actually have these 9-14 years?
• Still on the "figure out agency and train up an aligned AGI unilaterally" path?
• Has the FTX fiasco impacted your expectation of us-in-the-future having enough money=compute to do the latter?

I expect there to be no major updates, but seems worthwhile to keep an eye on this.

So my new main position is: which potential alignment targets (human values, corrigibility, Do What I Mean, human mimicry, etc) are naturally expressible in an AI’s internal language (which itself probably includes a lot of mathematics) is an empirical question, and that’s the main question which determines what we should target.

I'd like to make a case that Do What I Mean will potentially turn out to be the better target than corrigibility/value learning.

Primarily, "Do What I Mean" is about translation. Entity 1 compresses some problem specification defined over Entity 1's world-model into a short data structure — an order, a set of values, an objective function, etc. — then Entity 2 uses some algorithm to de-compress that data structure and translate it into a problem specification defined over Entity 2's world-model. The problem of alignment via Do What I Mean, then, is the problem of ensuring that Entity 2 (which we'll assume to be bigger) decompresses a specific type of compressed data structures using the same algorithm that was used to compress them in the first place — i. e., interprets orders the way they were intended/acts on our actual values and not the misspecified proxy/extrapolates our values from the crude objective function/etc.

This potentially has the nice property of collapsing the problem of alignment to the problem of ontology translation, and so unifying the problem of interpreting an NN and the problem of aligning an NN into the same problem.

In addition, it's probably a natural concept, in the sense that "how do I map this high-level description onto a lower-level model" seems like a problem any advanced agent would be running into all the time. There'll almost definitely be concepts and algorithms about that in the AI's world-model, and they may be easily repluggable.

A couple of weeks ago I started blitzing my way through one of your posts on natural abstraction and, wham! it hit me: J.J. Gibson, ecological psychology. Are you familiar with that body of work? Gibson's idea was that the environment has affordances (he's the one who brought that word to prominence) which are natural "points of attachment" [my phrase] for perceptual processes. It seems to me that his affordances are the low-dimensional projections (or whatever) that are the locuses of your natural abstractions. Gibson didn't have the kind of mathematical framework you're interested in, though I have the vague sense that some people who've been influenced by him have worked with complex dynamics.

And then there's the geometry of meaning Peter Gärdenfors has been developing: Conceptual Spaces, MIT 2000 and The Geometry of Meaning, MIT 2014. He argues that natural language semantics is organized into very low dimensional conceptual spaces. Might have some clues of things to look for.

What sort of value do you expect to get out of "crossing the theory-practice gap"?

Do you think that this will result in better insights about which direction to focus in during your research, for example? 

Will we have to wait until Dec 2023 for the next update or will the amount of time until the next one halve for each update, 6 months then 3 months then 6 weeks then 3 weeks?

Someone working full-time on an approach to the alignment problem that they feel optimistic about, and writing annual reflections on their work, is something that has been sorely lacking. +4

Did you not talk to Eliezer (or Stuart or Paul or...) about Corrigibility before the conversation you cited? It seems like they should have been able to change your mind quite easily on the topic, from what you wrote.

Have you done any work on thermodynamic coupling inducing phase transitions? If not, I'd recommend looking using a path integral formulation to frame the issue. David Tong's notes are a good introduction to the topic. Feynman's book on path integrals serves as a great refresher on the topic, with a couple of good chapters on probability theory and thermodynamic coupling. I lost my other reference texts, so I can't recommend anything else off the top of my head. 

Nice update!

On the flip side, I expect we’ll see more discussion about which potential alignment targets (like human values, corrigibility, Do What I Mean, etc) are likely to be naturally expressible in the internal language of neural nets, and how to express them.

While I don't think of these as alignment targets [LW · GW] per se (as I understand the term to be used), I strongly support discussing the internal language of the neural net [LW · GW] and moving away from convoluted inner/outer schemes [LW · GW].

On the theoretical side, Paul Christiano, Scott Garrabrant, and myself had all basically converged to working on roughly the same problem (abstraction, ontology identification, whatever you want to call it) by early 2022.

I haven't been following Scott's work that closely; what part of it are you calling "roughly the same problem?" (The titles of his recent LW posts don't seem to contain relevant words.)

A question about alignment via natural abstractions (if you've addressed it before, please refer me to where): it seems to me plausible that natural abstractions exist but are not useful for alignment, because alignment is a high-dimensional all-or-nothing property. Like, the AI will learn about "trees", but not unintentionally killing everyone depends on whether a palm tree is a tree, or on whether a copse counts as full of trees, or some other questions which depends on unnatural details of the natural abstraction.

• Do you think that edge cases will just naturally be correctly learned?
• Do you think that edge cases just won't end up mattering for alignment?

I’m interested in why the transformer architecture has been so successful and the concept of natural abstraction is useful here. I’m not thinking so much about how transformers work, not in any detail, but about the natural environment in which the architecture was originally designed to function, text. What are the natural abstractions over text?

Let’s look at this, a crucial observation:

That convergence isn’t complete yet - I think a lot of the interpretability crowd hasn’t yet fully internalized the framing of “interpretability is primarily about mapping net-internal structures to corresponding high-level interpretable structures in the environment”. In particular I think a lot of interpretability researchers have not yet internalized that mathematically understanding what kinds of high-level interpretable structures appear in the environment is a core part of the problem of interpretability.

For a worm the external environment is mostly dirt, but other worms as well. Birds, a rather different natural environment. Then we have the great apes, still a different natural environment. 

Our natural environment is very much like that of the great apes, our close biological relatives. Our fellow conspecifics are a big part of that environment. But we have language as well. Language is part of our natural environment, spoken language, but text as well. The advent of writing has made it possible to immerse ourselves in a purely textual world. Arithmetic, too, is a textual world. Think of it as a very specialized form of language, with a relatively small set of primitive elements and a highly constrained syntax. And then there is program code.

You see where this is going?

For the transformer architecture, text is its natural environment. The attention mechanism allows it to get enough context so that it can isolate the natural abstractions in an environment of pure text. What’s in that environment? Language, whether spoken or written is, after all, a physical thing. Alphanumeric characters, spaces between strings of characters, punctuation, capitalization, paragraphing conventions. Stuff like that. It’s the transformer’s superior capacity to abstract over that environment that has made it so successful.

(I note, as an aside, that speech is a physically complex and requires its own natural abstractions for perception. Phoneticians study the physical features of the soundstream itself while phonologists study those aspects of the signal that are linguistically relevant. They're thus looking for the natural abstractions over the physical signal itself. When put online, the physical text is extraordinarily simple – strings of ASCII characters – so so requires no sophisticated perceptual mechanisms.)

Note, of course, that text is text. Ultimately textual meanings have to be grounded in the physical world. No matter how much text a transformer consumes, it’s unlikely to substitute for direct access to the natural world. So the architecture can’t do everything, but it can do a lot.

I’ve recently written two posts examining the output of ChatGPT for higher-level discourse structure, mostly the conventions of dialog. It’s simple stuff, obvious in a way. But it has all but convincement that GPT I is picking up a primitive kind of discourse grammar that is separate from and independent from the word-by-word grammar of individual sentences. I can’t see how it would be able to produce such fluent text if it weren’t doing that.

The posts: 

Of pumpkins, the Falcon Heavy, and Groucho Marx: High-Level discourse structure in ChatGPT [LW · GW]

High level discourse structure in ChatGPT: Part 2 [Quasi-symbolic?] [LW · GW]

Curated! To ramble a bit on why: I love how this post makes me feel like I have a good sense of what John has been up to, been thinking about, and why, the insight of asking "how would an AI ensure a child AI is aligned with it?" feels substantive, and the optimism is nice and doesn't seem entirely foolhardy. Perhaps most significantly, it feels to me like a very big deal if alignment is moving towards something paradigmatic (should models and assumptions and questions and methods). I had thought that something we weren't going to get, but John does point out that many people are converging on similar interpretability/abstraction targets, and now he does point it out, that seems true and hopeful. I'm not an alignment researcher myself, so I don't put too much stock in my assessment, but this update is one of the most hopeful things I've read any time recently.

Forgive me if the answer to this would be obvious given more familiarity with natural abstractions, but is your claim that interpretability research should identify mathematically defined high-level features rather than fuzzily defined features? Supposing that in optimistic versions of interpretability, we're able to say that this neuron corresponds to this one concept and this one circuit in the network is responsible for this one task (and we don't have to worry about polysemanticity). How do we define concepts like "trees" and "summarizing text in a way that labelers like" in mathematical way?

So my new main position is: which potential alignment targets (human values, corrigibility, Do What I Mean, human mimicry, etc) are naturally expressible in an AI’s internal language (which itself probably includes a lot of mathematics) is an empirical question, and that’s the main question which determines what we should target.

Do you expect that the network will have an accurate understanding of its goals? I'd expect that we could train an agentic language model which is still quite messy and isn't able to reliably report information about itself and even if it could, it probably wouldn't know how to express it mathematically. I think a model could be able to write a lot of text about human values and corrigibility and yet fail to have a crisp or mathematical concept for either of them.

Overall, I’ve updated from “just aim for ambitious value learning” to “empirically figure out what potential medium-term alignment targets (e.g. human values, corrigibility, Do What I Mean, human mimicry, etc) are naturally expressible in an AGI’s internal concept-language”.

I like this. In fact, I would argue that some of those medium-term alignment targets are actually necessary stepping stones toward ambitious value learning.

Human mimicry, for one, could serve as a good behavioral prior for IRL agents. AI that can reverse-engineer the policy function of a human (e.g., by minimizing the error between the world-state-trajectory caused by its own actions and that produced by a human's actions) is probably already most of the way there toward reverse-engineering the value function that drives it (e.g., start by looking for common features among the stable fixed points of the learned policy function). I would argue that the intrinsic drive to mimic other humans is a big part of why humans are so adept at aligning to each other.

Do What I Mean (DWIM) would also require modeling humans in a way that would help greatly in modeling human values. A human that gives an AI instructions is mapping some high-dimensional, internally represented goal state into a linear sequence of symbols (or a 2D diagram or whatever). DWIM would require the AI to generate its own high-dimensional, internally represented goal states, optimizing for goals that give a high likelihood to the instructions it received. If achievable, DWIM could also help transform the local incentives for general AI capabilities research into something with a better Nash equilibrium. Systems that are capable of predicting what humans intended for them to do could prove far more valuable to existing stakeholders in AI research than current DL and RL systems, which tend to be rather brittle and prone to overfitting to the heuristics we give them.

Post summary (feel free to suggest edits!):
Last year, the author wrote up an plan [LW · GW] they gave a “better than 50/50 chance” would work before AGI kills us all. This predicted that in 4-5 years, the alignment field would progress from preparadigmatic (unsure of the right questions or tools) to having a general roadmap and toolset. 

They believe this is on track and give 40% likelihood that over the next 1-2 years the field of alignment will converge toward primarily working on decoding the internal language of neural nets - with interpretability on the experimental side, in addition to theoretical work. This could lead to identifying which potential alignment targets (like human values, corrigibility, Do What I Mean, etc) are likely to be naturally expressible in the internal language of neural nets, and how to express them. They think we should then focus on those.

In their personal work, they’ve found theory work faster than expected, and crossing the theory-practice gap mildly slower. In 2022 most of their time went into theory work like the Basic Foundations  [? · GW]sequence, workshops and conferences, training others, and writing up intro-level arguments on alignment strategies.

(If you'd like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries [? · GW] series.)

I like this post! It clarifies a few things I was confused on about your agenda and the progress you describe sounds pretty damn promising, although I only have intuitions here about how everything ties together.

In the interest of making my abstract intuition here more precise, a few weird questions:

Put all that together, extrapolate, and my 40% confidence guess is that over the next 1-2 years the field of alignment will converge toward primarily working on decoding the internal language of neural nets. That will naturally solidify into a paradigm involving interpretability work on the experiment side, plus some kind of theory work figuring out what kinds of meaningful data structures to map the internals of neural networks to.

What does your picture of (realistically) ideal outcomes from theory work look like? Is it more giving interpretability researchers a better frame to reason under (like a more mathematical notion of optimization that we have to figure out how to detect in large nets against adversaries) or something even more ambitious that designs theoretical interpretability processes that Just Work, leaving technical legwork (what ELK seems like to me)?

While they definitely share core ideas of ontology mismatch, it feels like the approaches are pretty different in that you prioritize mathematical definitions a lot and ARC is heuristical. Do you think the mathematical stuff is necessary for sufficient deconfusion, or just a pretty tractable way to arrive at the answers we want?

We can imagine, e.g., the AI imagining itself building a sub-AI while being prone to various sorts of errors, asking how it (the AI) would want the sub-AI to behave in those cases, and learning heuristics that would generalize well to how we would want the AI to behave if it suddenly gained a lot of capability or was considering deceiving its programmers and so on.

I'm not really convinced that even if corrigibility is A Thing (I agree that it's plausible it is, but I think it could also just be trivially part of another Thing given more clarity), it's as good as other medium-term targets. Corrigibility as stated doesn't feel like it covers a large chunk of the likely threat models, and a broader definition seems like it's just rephrasing a bunch of the stuff from Do What I Mean or inner alignment. What am I missing about why it might be as good a target?

I'd like to offer a general observation about ontology. As far as I can tell the concept enter computer science through Old School work in symbolic computation in AI. So you want to build a system that can represent all of human knowledge? OK. What are the primitive elements of such a system? What objects, events, and processes, along with the relations between them, what do you need? That's your ontology. From you generalize to any computing system: what are the primitives and what can you construct from them? 

If you want to take a peak at the Old School literature, John Sowa offers one view. Note that this is not a general survey of the literature. It's one man's distillation of it. Sowa worked at IBM Research (at Armonk I believe) for years.

I was interested in the problem, and still am, and did a variety of work. One of the things I did was write a short article on the "Ontology of Common Sense" for a Handbook of Metaphysics and Ontology which you can find here:

The opening three paragraphs:

The ontology of common sense is the discipline which seeks to establish the categories which are used in everyday life to characterize objects and events. In everyday life steel bars and window panes are solid objects. For the scientist, the glass of the window pane is a liquid, and the solidity of both the window pane and the steel bar is illusory, since the space they occupy consists mostly of empty regions between the sub-atomic particles which constitute these objects. These facts, however, have no bearing on the ontological categories of common sense. Sub-atomic particles and solid liquids do not exist in the domain of common sense. Common sense employs different ontological categories from those used in the various specialized disciplines of science.

Similar examples of differences between common sense and scientific ontologies can be multiplied at will. The common sense world recognizes salt, which is defined in terms of its colour, shape, and, above all, taste. But the chemist deals with sodium chloride, a molecule consisting of sodium and chlorine atoms; taste has no existence in this world. To common sense, human beings are ontologically distinct from animals; we have language and reason, animals do not. To the biologist there is no such distinction; human beings are animals; language and reason evolved because they have survival value. Finally, consider the Morning Star and the Evening Star. Only by moving from the domain of common sense to the domain of astronomy can we assert that these stars are not stars at all, but simply different manifestations of the planet Venus.

In all of these cases the common sense world is organized in terms of one set of object categories, predicates, and events while the scientific accounts of the same phenomena are organized by different concepts. In his seminal discussion of natural kinds, Quine suggested that science evolves by replacing a biologically innate quality space, which gives rise to natural kinds (in our terms, the categories of a common sense ontology), with new quality spaces. However, Quine has little to say about just how scientific ontology evolved from common sense ontology.

I suspect that there's a lot of structure between raw sensory experience and common sense ontology and a lot more between that and the ontologies of various scientific disciplines. But, you know, I wouldn't be surprised if a skilled auto mechanic has their own ontology of cars, a lot of it primarily non-verbal and based on the feels and sounds of working on cars with your hands.

Here's my references, with brief annotations, which indicate something of the range of relevant work that's been done in the past:

Berlin, B., Breedlove, D., Raven, P. 1973. "General Principles of Classification and Nomenclature in Folk Biology," American Anthropologist, 75, 214 - 242. There's been quite a lot of work on folk taxonomy. In some ways it's parallel to the (more) formal taxonomies of modern biology. But there are differences as well.

Hayes, P. J. 1985. "The Second Naive Physics Manifesto," in Formal Theories of the Commonsense World, J. R. Hobbs and R. C. Moore, eds., Ablex Publishing Co., 1 - 36. A lot of work has been done in this area, including work on college students who may have ideas about Newtonian dynamics in their heads but play video games in a more Aristotelian way.

Keil, F. C. 1979. Semantic and Conceptual Development: An Ontological Perspective, Cambridge, Massachusetts and London, England: Harvard University Press. How children develop concepts.

Quine, W. V.1969. "Natural Kinds," in Essays in Honor of Carl G. Hempel, Nicholas Rescher et al., eds., D. Reidel Publishing Co., 5 - 23. That is to say, are there natural kinds or is it culture all the way down.

Rosch, E. et al. 1976. "Basic Objects in Natural Categories," Cognitive Psychology, 8, 382 - 439. A key text introducing something called prototype theory.

Sommers, F. 1963."Types and Ontology," Philosophical Review, 72, 327 - 363. Do you know what philosophers mean by a category mistake? This is about the logic behind them.