I'm going to spend most of this comment responding to your concrete remarks about ELK, but I wanted to start with some meta level discussion because it seems to cut closer to the heart of the issue and might be more generally applicable.

I think a productive way forward (when working on alignment or on other research problems) is to try to identify the hardest concrete difficulties we can understand then try to make progress on them. This involves acknowledging that we can't anticipate all possible problems, but expecting that solving the concrete problems is a useful way to make steps forward and learn general lessons. It involves solving individual challenges, even if none of them will address the whole problem, and even if we have a vague sense that further difficulties will arise. It means not becoming too pessimistic about a direction until we see fairly concretely where it's stuck, partially because we hope that zooming in on a very concrete case where you get stuck is the main way to eventually make progress.

My sense is that you have more faith in a rough intuitive sense you've developed of what the "hard part" of alignment is, and so you'd primarily recommend thinking about that until we feel less confused. I disagree in large part because I feel like your broad intuitive sense has not yet had much opportunity to make contact with either reality or with formal reasoning, and I'd guess it's not precise enough to be a useful guide to research prioritization.

More concretely, you talk about novel mechanisms by which AI systems gain capabilities, but I think you haven't said much concrete about why existing alignment work couldn't address these mechanisms. This looks to me like a pretty unproductive stance; I suspect you are wrong about the shape of the problem, but if you are right then I think your main realistic path to impact involves saying something more concrete about why you think this.

I think you don't see the situation the same way, probably because you feel like you have said plenty concrete. Perhaps this is the most serious disagreement of all. I don't think saying there is a "capabilities well" is helpfully concrete until you say something about what it looks like, why it poses alignment problems different from SGD and why particular approaches don't generalize, etc.

In ARC's day to day work we write down particular models of capabilities that would generalize far outside of training (e.g.: what about a causal model of the world that holds robustly? what about logical deduction from valid premises with longer chains of reasoning? what about continuing to learn by trial and error when deployed in a novel environment?), and ask about whether a given alignment solution would generalize along with them. If we can find any gap, then that it goes on the list of problems. We focus on the gaps that seem least likely to be addressable by using known techniques, and try to develop new techniques or to identify general reasons why the gap is unresolvable.

My guess is that you are playing a roughly similar game much more informally, and that you are just making a mistake because reasoning about this stuff is in fact hard. But I can't really tell, since your thinking is happening in private and we are seeing the vague intuitions that result. (I've been hanging around MIRI for a long time, and I suspect I have a better model of your and Eliezer's position than virtually anyone else outside of MIRI, yet this is still where I'm at.)

Anyway, now turning to your discussion of ELK in particular.

Your first problem is that the recent capabilities gains made by the AGI might not have come from gradient descent (much like how humans’ sudden explosion of technological knowledge accumulated in our culture rather than our genes, once we turned the corner). You might not be able to just "expose the bad behavior" to gradients that you can hit to correct the thing, at least not easily and quickly.

I often think and write about other places where capabilities may come from that could challenge our basic alignment plan. Four particularly salient examples:

1. Your AI might perform search internally, e.g. looking for hypotheses that match the data or for policies that work well.
2. Natural selection may occur internally, e.g. cognitive patterns that acquire power might tend to dominate the behavior of your AI (despite the AI having no explicit prediction that they would work well).
3. Your AI might reason about how to think better, e.g. select cognitive actions based on anticipated consequences of those cognitive actions.
4. Our AI might deploy new algorithms that pose their own alignment risk for different (potentially unanticipated) reasons.

Some of these represent real problems, but none of them seem to fundamentally change the game or be deal-breakers:

1. Aligning the internal search seems very similar to aligning SGD on the outside. We could distinguish two additional difficulties in this case:
   1. Because the search is on the inside, we can't directly apply our alignment insights to align it. Instead we need to ensure that SGD learns to align the search. This itself poses two difficulties: (a) the outer gradient needs to incentivize doing this, (b) we need to argue that it's nearly as easy for SGD to learn the aligned search as to learn the unaligned search (or build scaffolding such that it becomes similarly easily). This is what we're talking about in this appendix, and it's part of why we are skeptical about approaches to ELK based on simple regularizers. But we don't see a reason that either (a) or (b) would be a dealbreaker, and we tentatively think our current approaches to ontology identification would at least solve (a) if they were successful at all. It's pretty hard to talk about (b) without having more clarity about what the alignment scheme actually looks like but we don't see an in principle reason it's hard.
   2. The internal search algorithm may not be SGD, and perhaps our alignment strategy was specific to some detail of SGD. But SGD appears to be amongst the hardest search algorithms, and ARC tries to pursue approaches that work for other algorithms rather than leveraging anything about SGD in particular. We're definitely in the market for other search algorithms that cause trouble but don't yet know of any.
2. Natural selection on the inside is similar but potentially more tricky, because the optimizer has more limited control over how this search works. This is like the analog of memetic selection being smarter than humans and eventually overpowering or hijacking human consequentialism. Another extreme example is that it seems like a large enough neural network may be catastrophically misaligned at initialization simply because of selection amongst activation patterns within a single forward pass. Ultimately we'd like to handle this in exactly the same way that we handle the last point, by some combination of (a) we can just directly apply the same hope from the previous section even to natural selection, (b) we can run explicit searches that are more powerful than implicit search by natural selection within our model, which requires ensuring that our explicit learned search captures whatever is good about natural selection (this seems tough but not at all obvious impossible to me). It's hard to talk about option (a) without seeing if/how we solve the problem from the last point. We could definitely work on option (b) now, and a large enough ARC would be working on it, but it seems like a relatively low priority since it's both very remote from existing systems and seems relatively unlikely-to-me to be the simplest place where we get stuck.
3. If our model is selecting cognitive actions, or designing new algorithms, then our core hope is that an aligned model will try to think in an aligned way. So if we've been succeeding at alignment so far then the model will be trying to stay aligned. By analogy, if humans cared about the amount of human DNA in the universe, then to the extent that cultural evolution was guided by human consequentialism (rather than e.g. being memetic selection), we would be trying to develop cultural machinery that was helpful for maximizing the amount of human DNA in the universe.
4. One way this can go wrong is if our model wants to stay aligned but fails, e.g. because it identifies new techniques for thinking that themselves pose new alignment difficulties (just as we desire human flourishing but may instead implement AI systems that want paperclips). I think this is a real problem, but there are a lot of reasons I don't consider it an existential challenge for our approach:
   1. If you've succeeded at alignment so far, then your AI will also consider this a problem and will be trying to solve it. I think we should relate to our AI, discovering new ways to think that might pose new alignment difficulties, in the same way that we relate to future humans who may encounter alignment difficulties. The AI may solve the problem, or may implement policy solutions, or etc., and our role is to set them up for success just like we are trying to set up future humans for success. AI compresses the timescale both for "new AI algorithms with new alignment problems" but also for all of the solutions to those problems, so I don't think it changes the game from future humans. And so I'd focus on prosaic AI alignment for exactly the same reasons I focus on prosaic AI alignment when trying to help future humans succeed at alignment.
   2. I think that we should be considering the particular algorithms that might pose a new alignment problem, and trying to solve alignment for each of them. If we have some general reason to think that new algorithms will be much harder than old algorithms, or that lessons won't transfer, then we can discuss those and whether they should affect research prioritization. So far I don't think we have such arguments, and so I think we should just be looking for algorithms that might pose problems. (I don't actually think that's the highest priority, because prosaic ML so obviously poses problems, and the other problems we see seem so closely analogous to the ones posed by prosaic ML. But I'm certainly in the market for other problems and think that a large enough research community should already be actively looking for them.)

Possible disagreements between us: (i) you think that at least one of these examples looks really bad for our approach, (ii) you have other examples in mind, (iii) you don't think we can write down a concrete example that looks bad,  but we have reason to expect other kinds of capability gains that will be bad, (iv) nothing looks like a dealbreaker in particular, but it's just contributing to a long list of problems you'd have to solve and that's either a lot of work or something probably won't work out.

For me, the upshot of all of this is that SGD poses some obvious problems, that those problems are the most likely to actually occur, that they seem similar to (and at least subproblems of) the other alignment problems we may face, and that there are neither super compelling alternatives to aligning SGD nor particular arguments that the rest of the problem is harder than this step.

Your second problem is that the AGI's concepts might rapidly get totally uninterpretable to your ELK head. Like, you could imagine doing neuroimaging on your mammals all the way through the evolution process. They've got some hunger instincts in there, but it's not like they’re smart enough yet to represent the concept of "inclusive genetic fitness" correctly, so you figure you'll just fix it when they get capable enough to understand the alternative (of eating because it's instrumentally useful for procreation). And so far you're doing great: you've basically decoded the visual cortex, and have a pretty decent understanding of what it's visualizing. 

Our goal is to learn a reporter that describes the latent knowledge of the model, and to keep this up to date as the model changes under SGD. If thinking about SGD, we usually think concretely about a single step of SGD, and how you could find a good reporter at the end of that gradient descent step assuming you had one at the beginning.

It feels to me like what you are saying here is just "you might not be able to solve ELK." Or else maybe restating the previous point, that the model builds latent knowledge by mechanisms other than SGD and therefore you need to learn a reporter that can also follow along with those other mechanisms.

In either case, I can't speak to whether it's helpful for the audience understanding why ELK is hard, but it is certainly not helping me understand why you think ELK is hard. I think this discussion is just too vague to be helpful.

I think it's not crazy for you to say "ARC's hopes about how to solve ELK are too vague to seem worth engaging with" (this is pretty similar to me saying "Nate's arguments about why alignment is hard are too vague to seem worth engaging with").

Analogously, your ELK head's abilities are liable to fall off a cliff right as the AGI's capabilities start generalizing way outside of its training distribution.

But can you say something concrete about why? What I'd like to do is talk about what the AGI is actually thinking, the particular computation it's running, so that we can talk about why that computation keeps being correlated with reality off distribution and then ask whether the reporter remains correlated with reality. When I go through this exercise I don't see big dealbreakers, and I can't tell if you disagree with that diagnosis, or if you are noticing other things that might be going on inside the AI, or if the difference is that I think "this looks like it might work in all the concrete cases we can see" is a relevant signal and you think "nah the cases we can't see are way worse than those we can see."

And if they don't, then this ELK head is (in this hypothetical) able to decode and understand the workings of an alien mind. Likely a kludgey behemoth of an alien mind. This itself is liable to require quite a lot of capability, quite plausibly of the sort that humanity gets first from the systems that took sharp left-turns, rather than systems that ground along today's scaling curves until they scaled that far. 

Again, this seems too vague to be helpful, or perhaps just mistaken. The reporter is not some other AI looking at your predictor and trying to "decode its workings," or maybe it is but if so it's just because those english words are vague and broad. Can we talk about the particular kinds of cognition that your AI might be performing, such that you don't think this works? (Or which would require the reporter to itself be using magic-mystery-juice-of-intelligence?)

That's really the central theme of my response, so it's worth restating: ARC loves examples of ways an AI might be thinking such that ELK is difficult. But your description of the sharp left turn is too vague to be helpful for this purpose, and so I'd either like to turn this into more concrete discussion of the internals of the algorithm, or else some significantly more precise argument about why we expect the unknown possible internals to be so much less favorable for ELK than any of the concrete examples we can write down.^[1]

1. ^{^}

   I'd like to head off a possible response you might make that I disagree with: "Sure your algorithm works for any example you can write down, but the whole point is that you need it to work for alien cognition, where humans don't understand why it works. So of course it works on concrete examples but not in the unknown real world." . I'm putting this in a footnote because it seems like a digression and I have no idea if this is your view.

   My main response is that we can in fact talk about concrete examples where "why your AI system's cognition works" isn't accessible to humans in the relevant ways:

   • We can consider tricky facts we understand about how to reason, for which our discovery of those facts is empirically contingent (and where discovering those facts is harder than discovering the reasons itself). Then we can consider whether our AI alignment strategies would work even if humans hadn't figured out the relevant facts about reasoning.
   • We can consider AI cognition which is contingent on hypothesized unknown-to-human facts, e.g. about the causal structure of reality, or about key facts about mathematics, or whatever else.
   • Most of our ELK approaches don't make no-holds-barred use of "can a human come up with some story about why this AI cognition may work," and so this just isn't a particularly salient threshold anyway. As a silly example, if you were solving this problem with a speed prior (or indeed with any of the approaches in the regularization section of the ELK document) you wouldn't expect a particular key threshold at the space of strategies that a human understands.

Imaginary John: Well, uh, these days I'm mostly focusing on using my flimsy non-mastered grasp of the common-concept format to try to give a descriptive account of human values, because for some reason that's where I think the hope is. So I'm not actually working too much on this thing that you think takes a swing at the real problem (although I do flirt with it occasionally).

That's not actually what I spend most of my time on, it's just a thing which came up in conversation with Eliezer that one time. I've never actually spent much time on a descriptive account of human values; I generally try to work on things which are bottlenecks to a wide variety of strategies (i.e. convergent hard subproblems), not things which are narrowly targeted to a single strategy.

What I'm actually spending most of my time on right now is figuring out how abstractions end up represented in cognitive systems, and how those representations correspond to structures (presumably natural abstractions) in the environment. In particular, I'd like to say things about convergent representations, such that we can both (a) test the claims on a wide variety of existing systems, and (b) have theorems saying that the claims extend to new kinds of systems.

... which, amusingly, looks like a much more ambitious version of interpretability work.

My guess at part of your views:

1. There's ~one natural structure for capabilities, such that (assuming we don't have deep mastery of intelligence) nearly anything we build that is an AGI will have that structure.
2. Given this, there will be a point where an AI system switches from everything-muddled-in-a-soup to clean capabilities and muddled alignment (the "sharp left turn").

I basically agree that the plans I consider don't engage much with this sort of scenario. This is mostly because I don't expect this scenario and so I'm trying to solve the alignment problem in the worlds I do expect.

(For the reader: I am not saying "we're screwed if the sharp left turn happens so we should ignore it", I am saying that the sharp left turn is unlikely.)

A consequence is that I care a lot about knowing whether the sharp left turn is actually likely. Unfortunately so far I have found it pretty hard to understand why exactly you and Eliezer find it so likely. I think current SOTA on this disagreement is this post [LW · GW] and I'd be keen on more work along those lines.

Some commentary on the conversation with me:

Imaginary Richard/Rohin: You seem awfully confident in this sharp left turn thing. And that the goals it was trained for won't just generalize. This seems characteristically overconfident.

This isn't exactly wrong -- I do think you are overconfident -- but I wouldn't say something like "characteristically overconfident" unless you were advocating for some particular decision right now which depended on others deferring to your high credences in something. It just doesn't seem useful to argue this point most of the time and it doesn't feature much in my reasoning.

For instance, observe that natural selection didn't try to get the inner optimizer to be aligned with inclusive genetic fitness at all. For all we know, a small amount of cleverness in exposing inner-misaligned behavior to the gradients will just be enough to fix the problem.

Good description of why I don't find the evolution analogy compelling for "sharp left turn is very likely".

And even if not that-exact-thing, then there are all sorts of ways that some other thing could come out of left field and just render the problem easy. So I don't see why you're worried.

I'd phrase it as "I don't see why you think [sharp left turn leading to failures of generalization of alignment that we can't notice and fix before we're dead] is very likely to happen". I'm worried too!

Nate: My model says that the hard problem rears its ugly head by default, in a pretty robust way. Clever ideas might suffice to subvert the hard problem (though my guess is that we need something more like understanding and mastery, rather than just a few clever ideas). I have considered an array of clever ideas that look to me like they would predictably-to-me fail to solve the problems, and I admit that my guess is that you're putting most of your hope on small clever ideas that I can already see would fail. But perhaps you have ideas that I do not. Do you yourself have any specific ideas for tackling the hard problem?

Imaginary Richard/Rohin: Train it, while being aware of inner alignment issues, and hope for the best.

I think if you define the hard problem to be the sharp left turn as described at the beginning of my comment then my response is "no, I don't usually focus on that problem" (which I would defend as the correct action to take).

Also if I had to summarize the plan in a sentence it would be "empower your oversight process as much as possible to detect problems in the AI system you're training (both in the outcomes it produces and the reasoning process it employs)".

Nate: That doesn't seem to me to even start to engage with the issue where the capabilities fall into an attractor and the alignment doesn't.

Yup, agreed.

Though if you weaken claim 1, that there is ~one natural structure to capabilities, to instead say that there are many possible structures to capabilities but the default one is deadly EU maximization, then I no longer agree. It seems pretty plausible to me that stronger oversight changes the structure of your capabilities.

Perhaps sometime we can both make a list of ways to train with inner alignment issues in mind, and then share them with each other, so that you can see whether you think I'm lacking awareness of some important tool you expect to be at our disposal, and so that I can go down your list and rattle off the reasons why the proposed training tools don't look to me like they result in alignment that is robust to sharp left turns. (Or find one that surprises me, and update.) But I don't want to delay this post any longer, so, some other time, maybe.

I think the more relevant cruxes are the claims at the top of this comment (particularly claim 1); I think if I've understood the "sharp left turn" correctly I agree with you that the approaches I have in mind don't help much (unless the approaches succeed wildly, to the point of mastering intelligence, e.g. my approaches include mechanistic interpretability which as you agree could in theory get to that point even if they aren't likely to in practice).

I want to highlight a part of your model which I think is making the problem much harder: the belief that at some point we will stop teaching models via gradient descent, and instead do something closer to lifelong in-episode learning.

Nate: This seems to me like it's implicitly assuming that all of the system's cognitive gains come from the training. Like, with every gradient step, we are dragging the system one iota closer to being capable, and also one iota closer to being good, or something like that.

To which I say: I expect many of the cognitive gains to come from elsewhere, much as a huge number of the modern capabilities of humans are encoded in their culture and their textbooks rather than in their genomes. Because there are slopes in capabilities-space that an intelligence can snowball down, picking up lots of cognitive gains, but not alignment, along the way.

I agree that given an unaligned AGI with no defined objective, which then starts learning via an optimization process whose target we cannot understand (like non-GD learning), the fact that “in our outer GD loop our gradients were aligned with human feedback” (Vivek’s scheme) is not very comforting.

But there are two more attributes I think improve our situation. First, during proto-AGI training dominated by GD, the human objective is indeed being pointed to, as follows from the original argument. Second, at some point (either slightly before or after the sharp left turn) the AI becomes strategically aware and starts reasoning about how its actions affect its future values. (This is a big part of Alex Turner and Quintin Pope’s shard theory thing.) This seems like it will tend to “lock in” whatever its values were at that point.

So from my perspective, if strategic awareness pops up before non-GD learning dominates GD learning, we’re in much better shape. I think there are various ways to do training to make this more likely. This occupies a big part of my tractable-alignment-solution space.

Interested to hear where you disagree!

(As a side note, this post made me update to be meaningfully more optimistic, since I do think you’re not necessarily engaging with the strongest responses to your critiques of several of these strategies. But maybe after more back-and-forth I’ll realize my steelman versions of these plans break in some other way.

Also, by your account, several plans fail due to separate predictions in your background model. The probability that each prediction is wrong seems plausibly independent, and so as an ensemble they seem much more promising.)

EDIT: for the sake of being concrete, one major way to make non-GD learning subservient to GD learning for longer is to continue to do gradient updates on the original weights during the in-episode learning phase. Evolution's major problem w.r.t. human alignment is that all it got to do was iterate on the genome and then watch what happened during the in-context learning phase, giving extremely marginal feedback. (This allowed other optimization processes with other objectives, like cultural optimization, to overtake evolution.) However, we don't have that constraint: we can keep training via the same gradient descent scheme even during AI lifetimes. This seems like it would substantially reduce our reliance on improving alignment via the non-GD optimization.

As someone with limited knowledge of AI or alignment, I found this post accessible. There were times when I thought I knew vaguely what Nate meant but would not be able to explain it so I'm recording my confusions here to come back to when I've read up more. (If anyone wants to answer any of these r/NoStupidQuestions questions, that would be very helpful too).

1. "Your first problem is that the recent capabilities gains made by the AGI might not have come from gradient descent". This is something that comes up in response to a few of the plans. Is the idea that during training, for advanced enough AIs capabilities gains come from gradient descent and also through processing input / interacting with the world. Or is the second part only after it has finished training. What does that concretely look like in ML? 
2. Is a lot of the disagreement about these plans just because of others finding the idea of a "sharp left turn" more unlikely than Nate or is there more agreement about that idea but the disagreement is about what proposals might give us a shot at solving it?
3. What might an ambitious interpretability agenda focused on the sharp left turn and the generalization problem look like besides just trying harder at interpretability?
4. Another explanation of the "sharp left turn" would also be really helpful to me.  At the moment, it feels like I can only explain why that happens by using analogies to humans/apes rather than being able to give a clear explanation for why we should expect that by default, using ML/alignment language. 

Thanks for the post, I agree with a lot of it. A few quick comments on your dialogue with imaginary me/Rohin, which highlight the main points of disagreement:

And even if not that-exact-thing, then there are all sorts of ways that some other thing could come out of left field and just render the problem easy. So I don't see why you're worried.

More accurate to say "I don't see why you're so confident". I think I see why you're worried, and I'm worried too for the same reasons. Indeed, I wrote a similar post recently [AF · GW] which lists out research directions and reasons why I don't expect them to solve the problem if it turns out to be hard. So in general you should probably put me down as having a reasonable amount of credence (20%?) on your view, but also considering many other possibilities plausible.

Nate: I have considered an array of clever ideas that look to me like they would predictably-to-me fail to solve the problems, and I admit that my guess is that you're putting most of your hope on small clever ideas that I can already see would fail.

The ideas that come out of left field are generally the ones you haven't considered yet, that's what it means for them to come out of left field. I expect that this is frustrating for you to hear, because it seems my position is therefore unfalsifiable, but I don't think it makes much pragmatic difference - I'm not saying we should relax because ideas will come out of left field. I think we should do a better job of looking for them, which involves people aiming more directly at worlds where the problem is hard, for which posts like this one help. I just also think that there's probably more leeway than you think, because I feel pretty uncertain how far past human level a sharp left turn would happen by default.

Hey, thanks for posting this!

And I apologise - I seem to have again failed to communicate what we're doing here :-(

"Get the AI to ask for labels on ambiguous data"

Having the AI ask is a minor aspect of our current methods, that I've repeatedly tried to de-emphasise (though it does turn it to have an unexpected connection with interpretability). What we're trying to do is:

1. Get the AI to generate candidate extrapolations of its reward data, that include human-survivable candidates.
2. Select among these candidates to get a human-survivable ultimate reward functions.

Possible selection processes include being conservative (see here for how that might work: https://www.lesswrong.com/posts/PADPJ3xac5ogjEGwA/defeating-goodhart-and-the-closest-unblocked-strategy [LW · GW] ), asking humans and then extrapolating the process of what human-answering should idealise to (some initial thoughts on this here: https://www.lesswrong.com/posts/BeeirdrMXCPYZwgfj/the-blue-minimising-robot-and-model-splintering [LW · GW]), removing some of the candidates on syntactic ground (e.g. wireheading, which I've written quite a bit on how it might be syntactically defined). There are some other approaches we've been considering, but they're currently under-developed.

But all those methods will fail if the AI can't generate human-survivable extrapolations of its reward training data. That is what we are currently most focused on. And, given our current results on toy models and a recent literature review, my impression is that there has been almost no decent applicable research done in this area to date. Our current results on HappyFaces are a bit simplistic, but, depressingly, they seem to be the best in the world in reward-function-extrapolation (and not just for image classification) :-(

I really liked this post in that it seems to me to have tried quite seriously to engage with a bunch of other people's research, in a way that I feel like is quite rare in the field, and something I would like to see more of. 

One of the key challenges I see for the rationality/AI-Alignment/EA community is the difficulty of somehow building institutions that are not premised on the quality or tractability of their own work. My current best guess is that the field of AI Alignment has made very little progress in the last few years, which is really not what you might think when you observe the enormous amount of talent, funding and prestige flooding into the space, and the relatively constant refrain of "now that we have cutting edge systems to play around with we are making progress at an unprecedented rate". 

It is quite plausible to me that technical AI Alignment research is not a particularly valuable thing to be doing right now. I don't think I have seen much progress, and the dynamics of the field seem to be enshrining an expert class that seems almost ontologically committed to believing that the things they are working on must be good and tractable, because their salary and social standing relies on believing that. 

This and a few other similar posts last year are the kind of post that helped me come to understand the considerations around this crucial question better, and where I am grateful that Nate, despite having spent a lot of his life on solving the technical AI Alignment problem, is willing to question the tractability of the whole field. This specific post is more oriented around other people's work, though other posts by Nate and Eliezer are also facing the degree to which their past work didn't make the relevant progress they were hoping for. 

As the main author of the "Alignment"-appendix of the truthful AI paper, it seems worth clarifying: I totally don't think that "train your AI to be truthful" in itself is a plan for how to tackle any central alignment problems. Quoting from the alignment appendix:

While we’ve argued that scaleable truthfulness would constitute significant progress on alignment (and might provide a solution outright), we don’t mean to suggest that truthfulness will sidestep all difficulties that have been identified by alignment researchers. On the contrary, we expect work on scaleable truthfulness to encounter many of those same difficulties, and to benefit from many of the same solutions.

In other words: I don't think we had a novel proposal for how to make truthful AI systems, which tackled the hard bits of alignment. I just meant to say that the hard bits of making truthful A(G)I are similar to the hard bits of making aligned A(G)I.

At least from my own perspective, the truthful AI paper was partly about AI truthfulness maybe being a neat thing to aim for governance-wise (quite apart from the alignment problem), and partly about the idea that research on AI truthfulness could be helpful for alignment, and so it's good if people (at least/especially people who wouldn't otherwise work on alignment) work on that problem. (As one example of this: Interpretability seems useful for both truthfulness and alignment, so if people work on interpretability intended to help with truthfulness, then this might also be helpful for alignment.)

I don't think you're into this theory of change, because I suspect that you think that anyone who isn't directly aiming at the alignment problem has negligible chance of contributing any useful progress.

I just wanted to clarify that the truthful AI paper isn't evidence that people who try to hit the hard bits of alignment always miss — it's just a paper doing a different thing.

(And although I can't speak as confidently about others' views, I feel like that last sentence also applies to some of the other sections. E.g. Evan's statement, which seems to be about how you get an alignment solution implemented once you have it, and maybe about trying to find desiderata for alignment solutions, and not at all trying to tackle alignment itself. If you want to critique Evan's proposals for how to build aligned AGI, maybe you should look at this list of proposals [LW · GW] or this positive case for how we might succeed [LW · GW].)

Partisans of the other "hard problem" are also quick to tell people that the things they call research are not in fact targeting the problem at all.  (I wonder if it's something about the name...)

Much like the other hard problem, it's easy to get wrapped up in a particular picture of what properties a solution "must" have, and construct boundaries between your hard problem and all those other non-hard problems.

Turning the universe to diamond is a great example. It's totally reasonable that it could be strictly easier to build an AI to turn the world into diamond than it is to build an AI that is superhuman at doing good things, so that anyone claiming to have ideas about the latter should have even better ideas about the former. But that could also not be the case - the most likely way I see this happening is if if solving the hard left turn problem has details that depend on how you want to load the values, and so genuinely hard-problem-addressing work on value learning could nonetheless not be useful for specifying simple goals. (It may only help you get the diamond-universe AI "the hard way" - by doing the entire value leaning process except with a different target!)

I should acknowledge first that I understand that writing is hard. If the only realistic choice was between this post as it is, and no post at all, then I'm glad we got the post rather than no post.

That said, by the standards I hold my own writing to, I would embarrassed to publish a post like this which criticizes imaginary paraphrases of researchers, rather than citing and quoting the actual text they've actually published. (The post acknowledges this as a flaw, but if it were me, I wouldn't even publish.) The reason I don't think critics necessarily need to be able to pass an author's Ideological Turing Test [LW · GW] is because, as a critic, I can at least be scrupulous in my reasoning about the actual text that the author actually published, even if the stereotype of the author I have in my head is faulty. If I can't produce the quotes to show that I'm not just arguing against a stereotype in my head, then it's not clear why the audience should care.

Flagging that I don't think your description of what ELK is trying to do is that accurate, e.g. we explicitly don't think that you can rely on using ELK to ask your AI if it's being deceptive, because it might just not know. In general, we're currently quite comfortable with not understanding a lot of what our AI is "thinking", as long as we can get answers to a particular set of "narrow" questions we think is sufficient to determine how good the consequences of an action are. More in “Narrow” elicitation and why it might be sufficient.

Separately, I think that ELK isn't intended to address the problem you refer to as a "sharp-left turn" as I understand it. Vaguely, ELK is intended to be an ingredient in an outer-alignment solution, while it seems like the problem you describe falls roughly into the "inner alignment" camp. More specifically, but still at a high-level of gloss, the way I currently see things is:

• If you want to train a powerful AI, currently the set of tasks you can train your AI on will, by default, result in your AI murdering you.
• Because we currently cannot teach our AIs to be powerful by doing anything except rewarding them for doing things that straightforwardly imply that they should disempower humans, you don't need a "sharp left turn" in order for humanity to end up disempowered.
• Given this, it seems like there's still a substantial part of the difficulty of alignment that remains to be solved even if knew how to cope with the "sharp left turn." That is, even if capabilities were continuous in SGD steps, training powerful AIs would still result in catastrophe.
• ELK is intended to be an ingredient in tackling this difficulty, which has been traditionally referred to as "outer alignment."

Even more separately, it currently seems to me like it's very hard to work on the problem you describe while treating other components [like your loss function] like a black box, because my guess is that "outer alignment" solutions need to do non-trivial amounts of "reaching inside the model's head" to be plausible, and a lot of how to ensure capabilities and alignment generalize together is going to depend on details about how would have prevented it from murdering you in [capabilities continuous with SGD] world.

ELK for learned optimizers has some more details.

Many proposals seem doomed to me because they involve one or multiple steps where they assume a representation, then try to point to robust relations in the representation and hope they'll hold in the territory. This wouldn't be so bad on its own but when pointed to it seems like handwaving happens rather than something more like conceptual engineering. I am relatively more hopeful about John's approach as being one that doesn't fail to halt and catch fire at these underspecified steps in other plans. In other areas like math and physics we try to get the representation to fall out of the model by sufficiently constraining the model. I would prefer to try to pin down a doomed model than stay in hand wave land because at least in the process of pinning down the doomed model you might get reusable pieces for an eventual non doomed model. Was happy about eg quantilizers for basically the same reason.

Like, even simpler than the problem of an AGI that puts two identical strawberries on a plate and does nothing else [LW · GW], is the problem of an AGI that turns as much of the universe as possible into diamonds. This is easier because, while it still requires that we have some way to direct the system towards a concept of our choosing, we no longer require corrigibility. (Also, "diamond" is a significantly simpler concept than "strawberry" and "cellularly identical".)

It seems to me that we have basically no idea how to do this. We can train the AGI to be pretty good at building diamond-like things across a lot of training environments, but once it takes that sharp left turn, by default, it will wander off and do some other thing, like how humans wandered off and invented birth control.

Is there a writeup of where you expect this to fail? I recall this MIRI newsletter but I think it also just asserted it was hard/impossible.

Is the difficulty just in "it's gonna hijack it's own reward function?" or is there more to it than that?

What form of plan would address the hard bits of the alignment challenge?

But if I was saying that about a hundred pretty-uncorrelated agendas being pursued by two hundred people, I'd start to think that maybe the odds are in our favor.

Wait a minute − excuse my naïveté, but that doesn't seem that hard!

I assume it is though and thus ask: why is it? Is it that hard to come up with such agendas? What if we had $100B to pay people and/or set bounties with?

Epistemic status: Story. I am just assuming that my current guesses for the answers to outstanding research questions are true here. Which I don't think they are. They're not entangled enough with actual data yet for that to be the case. This is just trying to motivate why I think those are the right kinds of things to research.

Figure out how to measure and plot information flows in ML systems. Develop an understanding of abstractions, natural or otherwise, and how they are embedded in ML systems as information processing modules.

Use these tools to find out how ML systems embed things like subagents, world models, and goals, how they interlink, and how they form during training. I’m still talking about systems like current reinforcement learners/transformer models or things not far removed from them here.

With some better idea of what “goals” in ML systems even look like, formalise these concepts, and find selection theorems that tell you, rigorously, which goals a given loss function used by the outer optimiser will select for. I suspect that in dumb systems, this is (or could be made) pretty predictable and robust to medium sized changes in the loss function, architecture, or outer optimiser. Because that seems to be the case in our brains. E.g. some people are born blind and never use their inborn primitive facial recognition while forming social instincts, yet their values seem to turn out decidedly human-like. Humans need perturbations like sociopath-level breakage of the reward circuitry that is our loss function, or being raised by wolves, to not form human-like values.

Now you know how to make a (dumb) AGI that wants particular things, the definitions of which are linked to concepts in its world model that are free to evolve as it gets smarter. You also know which training situations and setups to avoid to stop the outer optimiser from making new goals you don’t want. 

Use this to train an AGI to have human-baby-like primitive goals/subagents/Shards, like humans being sad is bad, cartoons are fun, etc. As capabilities increase, these primitive goals would seem likely to be generalised by the AGI into human-adult-like preferences even under moderately big perturbations, because that sure seems to happen with humans.

Now train the not-goal parts of the AGI to superhuman capability. Since it wants its own goals to be preserved just as much as you do, it will gladly go along with this. Should a takeoff point be reached, it will use its knowledge of its own structure to self-modify while preserving its extrapolated, human-like values.

I'm a fairly new alignment researcher, although I've been following LessWrong and MIRI's writing, etc for many years. 

My approach so far is trying to understand and use interpretability tools (e.g. learning from Olah's circuits team & Buck at Redwood Research, and others who aren't themselves trying to tackle the 'hard problem' but are coming up with useful stuff like ROME https://rome.baulab.info/ ) to find and control natural abstractions (ala John Wentworth) in a variety of model architectures (starting with transformers).

An important additional aspect is that I think it is plausible to edit a variety of architectures in some generalizable ways to partially separate them into 'modules', based around natural dividing points (from natural abstractions). Something like a hybrid between a singular black box and a mixture of experts. I'd expect this refactoring to result in some 'alignment tax' on capabilities, but not as extreme as going to full silo-ed narrow experts.

One of the goals along the way should be to try to fundamentally understand general intelligence, to be able to better detect and control it.

My main self-criticism: too slow. I think my approach has a pretty good chance given a team of 10 people at least as smart and productive as me, and about 50 years to work on the problem. I super don't think we have anywhere near 50 years, more like 10. Also, I'm not a member of a team of 10 people working on the same idea, I'm new and alone.

Response: My hope is to join a team working on some similar set of goals, and thereby be usefully contributing towards a solution. Also, my hope is to convince people that this problem is big enough and urgent enough that we should even more rapidly scale up the numbers of smart capable people working on it.

But maybe I just don't understand this proposal yet (and I have had some trouble distilling things I recognize as plans out of Evan's writing, so far).

Maybe this [AF · GW] and this [AF · GW] will help.

Curated. I could imagine a world where different people pursue different agendas in a “live and let live” way, with no one waiting to be too critical of anyone else. I think that’s a world where many people could waste a lot of time with nothing prompting them to reconsider. I think posts like this one give us a chance to avoid scenarios like that. And posts like this can spur discussion of the higher-level approaches/intuitions that spawn more object-level research agenda. The top comments here by Paul Christianno, John Wentworth, and others are a great instance of this.

I also kind of like how this just further develops my gears-level understanding of why Nate predicts doom. There’s color here beyond AGI Ruin: List of Lethalities, which I assume captured most of Nate’s pessimism, but in fact I wonder if Nate disagrees with Eliezer and thinks things would be a bunch more hopeful if only people worked on the right stuff (in contrast with the problem is too hard for our civilization).

Lastly I’ll note that I think it’s good that Nate wrote this post even before being confident he could pass other people’s ITT. I’m glad he felt it was okay to be critical (with caveats) even before his criticisms were maximally defensible (e.g. because he thinks he could pass an ITT).

Just noting that given more recent developments than this post, we should be majorly updating on recent progress towards Andrew Critch's strategy. (Still not more likely than not to succeed, but we still need to assign some Bayes points to Critch, and take some away from Nate.)

If it helps, I have a discussion of Concept Extrapolation in the context of aligning a real-deal agent-y AGI in §14.4 here [LW · GW].

So far I can’t quite get the whole story to hang together, as you’ll see from that link. But I definitely see it as a “shot on goal”. (Well, at least, I think the broader project / framework is a “shot on goal”. I don’t find the image classification project to be directly addressing any of my most burning questions.)

Thanks for the post! One narrow point:
You seem to lean at least a bit on the example of 'much like how humans’ sudden explosion of technological knowledge accumulated in our culture rather than our genes, once we turned the corner'.  It seems to me that
a. You don't need to go to humans before you get significant accumulation of important cultural knowledge outside genes (e.g. my understanding is that unaccultured chimps die in the wild)
b.  the genetic bottleneck is a somewhat weird and contingent feature of animal evolution, and I don't think there's a clear analogy in current LLM ML paradigms

I'm not making any claims about takeoff speeds in models, just saying that I don't think arguments that are based on features that are (maybe) contingent on a genetic bottleneck support the same inference for ML. Can you make the same argument without leaning on the genetic bottleneck, or explain to me why the analogy in fact should hold?

I was a bit confused about this quote, so I tried to expand on the ideas a bit. I'm posting it here in case anyone benefits from is or disagrees.

To which I say: I expect many of the cognitive gains to come from elsewhere, much as a huge number of the modern capabilities of humans are encoded in their culture and their textbooks rather than in their genomes. Because there are slopes in capabilities-space that an intelligence can snowball down, picking up lots of cognitive gains, but not alignment, along the way.

I guess saying is saying that an AI will develop a way to learn things without gradient descent, just like humans learned things outside of our genetic update. Some ways to do this would be

• Develop the ability to read things on the internet and learn from them
• Spend cognitive energy on things like doing math or programming
• Do things to actually gain power in the world, like accumulating money or compute

I guess the argument is that, for objectives, only gradient descent is pushing you in the correct direction, whereas for capabilities, the system will develop ways to push itself in the right direction in addition to SGD. Like, it's true that for any objective function its good to be more powerful.;It's not true that for any level of power the system is incentivized to have the more correct objective.

A system wants to be more powerful, but it doesn't want to have a more "correct" objective.

<sociology of AI safety rant>

So, if an Everett-branches traveller told me "well, you know, MIRI folks had the best intentions, but in your branch, made the field pay attention to unproductive directions, and this made your civilization more confused and alignment harder" and I had to guess "how?", one of the top choices would be ongoing strawmanning and misrepresentation of Eric Drexler's ideas.

</rant>

To me, CAIS thinking seems quite different from the description in the op.

Some statements, without much justifications/proofs

- Modularity is a pretty powerful principle/law of intelligent systems. If you look around the world, you see modularity everywhere. Actually in my view you will see more of "modularity" than of "rational agency", suggesting gods of modularity are often stronger than gods of rational agency. Modularity would help as a lot, in contrast to integrated homogenous agents  => one of the key directions of AI safety should be figuring out how to summon Gods of modularity

- Modularity helps with interpretability; once you have "modules" and "interfaces", you have much better shot at understanding what's going on by looking on the interfaces. (For intuitive feel: Imagine you want to make plotting and scheming of three people much more legible, and you can impose this constrain: they need to make all communication between them on Slack, which you can read)

- Any winning strategy needs to solve global coordination at some point, otherwise people will just call a service to destroy the world. Solutions of the type "your aligned superintelligent agent takes over the world and coordinates everyone" are dangerous and won't work; superintelligent agent able takes over the world is something you don't want to bring into existence, and in contrast, you need a security layer to prevent anyone from event attempting that

- There are multiple hard problems; attempting to solve them all at once in one system is not the right tactic. In practice, we want to isolate some problems to separate "modules" or "services" - for example, we want a separate "research and development services", "security service", "ontology mapping service",...

- Many hard problems don't disappear, but there are also technical solutions for them [e.g. distillation]

 

What would make you change your mind about robustness of behavior (or interpretability of internal representations) through the sharp left turn? Or about the existence of such a sharp left turn, as opposed to smooth scaling of ability to learn in-context?

For example, would you change your mind if we found smooth scaling laws for (some good measure of) in-context learning?

By "superbabies", do you mean genetically engineering high intelligence?

Second, it doesn't alleviate enough pressure; the bureaucrats can't tell real solutions from bad ones; the cost to build an unaligned AGI drops each year; etc., etc. 

And making AI researchers answerable to some centralized entity puts a big target on that entity as a thing for sociopaths (broadly construed) to corrupt / capture, and the more that AI researchers are living under the thumb of a corrupt / captured entity, the more they're misaligned with their own values and less likely to be sane.

Like, I see this plan as basically saying "yep, that hard problem is in fact too hard, let's try to dodge it, by having humans + narrow AI services perform the pivotal act". Setting aside how I don't particularly expect this to work, we can at least hopefully agree that it's attempting to route around the problems that seem to me to be central, rather than attempting to solve them.

I think you're being overly critical of this approach. We can build pretty useful [LW · GW]AI without getting anywhere near your "sharp left turn".  For example, the "Strawberry problem" is well within the domain of things narrow AI could solve.

Importantly, if the Alignment Problem turns out to be ill-defined/unsolvable, combining a bunch of narrow AIs into something useful is the only workable approach we've got.

Like, yes, some directions are easier to point an AI in, on account of the direction itself being simpler to conceptualize, but that observation alone doesn't say anything about how to determine which direction an AI is pointing after it falls into the capabilities well.

Identifying which concepts are attractor basins and figuring out how to build a useful AGI out of those concepts sounds like a complete solution to the Alignment Problem.  Later in the article you seem positive about ELK, so maybe we agree on this fact.

On reflection, I can see how what you talk about is a hard part. I haven't focused much on it, because I've seen it as a capabilities question (much of it seems to boil down to an outer optimizer not being capable enough to achieve inner alignment, which depends on the capabilities of the outer optimizer), but on reflection there may be quite worthwhile for alignment researchers to spend time on this.

However, I don't think it's the (only) hard part. Even if we can turn the world into diamondoid or create two identical strawberries, there's still the equally important issue of figuring out human values well enough to direct to that.

I also agree with skepticism about many of the existing research programs.

This doesn't directly fight the hard problem, but it could make it easier: what do you think of putting massive effort into developing BCIs, and then using them to link the brains of people working on the problem together so that they could share understanding or even merge as a single superintelligence able to work on the problem more effectively? Obviously there's a tremendous amount of unknowns about how that would work, but I think it's plausible, and although Neuralink is taking its time, Openwater's planned wearable BCI could get there faster if they are motivated and funded appropriately. I need to make a whole post about this concept, since it's kind of my main interest, but I just wanted to see what you thought about it first.

Very interesting. I'm stuck on the argument about truthfulness being hard because the concept of truth is somehow fraught or too complicated. I'm envisaging an objection based on the T-schema ('<p> is true iff p').

Nate writes:

Now, in real life, building a truthful AGI is much harder than building a diamond optimizer, because 'truth' is a concept that's much more fraught than 'diamond'. (To see this, observe that the definition of "truth" routes through tricky concepts like "ways the AI communicated with the operators" and "the mental state of the operators", and involves grappling with tricky questions like "what ways of translating the AI's foreign concepts into human concepts count as manipulative?" and "what can be honestly elided?", and so on, whereas diamond is just carbon atoms bound covalently in tetrahedral lattices.)

(end of quote)

But this reference to "the definition of 'truth'" seems to presuppose some kind of view, where I'm not sure what that view is, but know it's definitely going to be philosophically controversial.

Some think that 'true' can be defined by taking all the instances of the T-schema, or a (perhaps restricted) universal generalisation of it.

And this seems not totally crazy or irrelevant from an AI design perspective, at least at first blush. I feel I can sort of imagine an AI obeying a rule which says to assert <p> only if p. 

Trying to envisage problems and responses, I hit the idea that the AI would have degrees of belief or credences, and not simply a list of things it thinks are true simpliciter. But perhaps it can have both. And perhaps obeying the T-schema based truthfulness rule would just lead it to confine most of its statements to statements about its own credences or something like that.

I think I see a separate problem about ensuring the AI does not (modify itself in order to) violate the T-schema based truthfulness rule. But that seems different at least from the supposed problem in the OP about the definition of 'true' being fraught or complicated or something.

If it wasn't already clear I'm a philosophy person, not an alignment expert, but I follow alignment with some interest.

Extremely important