AI as a science, and three obstacles to alignment strategies

As Shankar Sivarajan [LW(p) · GW(p)] points out in a different comment, the idea that AI became less scientific when we started having actual machine intelligence to study, as opposed to before that when the 'rightness' of a theory was mostly based on the status of whoever advanced it, is pretty weird. The specific way in which it's weird seems encapsulated by this statement:

on the whole, modern AI engineering is simply about constructing enormous networks of neurons and training them on enormous amounts of data, not about comprehending minds.

In that there is an unstated assumption that these are unrelated activities. That deep learning systems are a kind of artifact produced by a few undifferentiated commodity inputs, one of which is called 'parameters', one called 'compute', and one called 'data', and that the details of these commodities aren't important. Or that the details aren't important to the people building the systems.

I've seen a (very revisionist) description of the Wright Brothers research as analogous to solving the control problem, because other airplane builders would put in an engine and crash before they'd developed reliable steering. Therefore, the analogy says, we should develop reliable steering before we 'accelerate airplane capabilities'. When I heard this I found it pretty funny, because the actual thing the Wright Brothers did was a glider capability grind. They carefully followed the received aerodynamic wisdom that had been written down, and when the brothers realized a lot of it was bunk they started building their own database to get it right:

During the winter of 1901, the brothers began to question the aerodynamic data on which they were basing their designs. They decided to start over and develop their own data base with which they would design their aircraft. They built a wind tunnel and began to test their own models. They developed an ingenious balance system to compare the performance of different models. They tested over two hundred different wings and airfoil sections in different combinations to improve the performance of their gliders The data they obtained more correctly described the flight characteristics which they observed with their gliders. By early 1902 the Wrights had developed the most accurate and complete set of aerodynamic data in the world.

In 1902, they returned to Kitty Hawk with a new aircraft based on their new data. This aircraft had roughly the same wing area as the 1901, but it had a longer wing span and a shorter chord which reduced the drag. It also sported a new movable rudder at the rear which was installed to overcome the adverse yaw problem. The movable rudder was coordinated with the wing warping to keep the nose of the aircraft pointed into the curved flight path. With this new aircraft, the brothers completed flights of over 650 feet and stayed in the air for nearly 30 seconds. This machine was the first aircraft in the world that had active controls for all three axis; roll, pitch and yaw. By the end of 1902, the brothers had completed over a thousand glides with this aircraft and were the most experienced pilots in the world. They owned all of the records for gliding. All that remained for the first successful airplane was the development of the propulsion system.

In fact while trying to find an example of the revisionist history, I found a historical aviation expert describe the Wright Brothers as having 'quickly cracked the control problem' once their glider was capable enough to let it be solved. Ironically enough I think this story, which brings to mind the possibility of 'airplane control researchers' insisting that no work be done on 'airplane capabilities' until we have a solution to the steering problem, is nearly the opposite of what the revisionist author intended and nearly spot on to the actual situation.

We can also imagine a contemporary expert on theoretical aviation (who in fact existed before real airplanes) saying something like "what the Wright Brothers are doing may be interesting, but it has very little to do with comprehending aviation [because the theory behind their research has not yet been made legible to me personally]. This methodology of testing the performance of individual airplane parts, and then extrapolating the performance of a airplane with an engine using a mere glider is kite flying, it has almost nothing to do with the design of real airplanes and humanity will learn little about them from these toys". However what would be genuinely surprising is if they simultaneously made the claim that the Wright Brothers gliders have nothing to do with comprehending aviation but also that we need to immediately regulate the heck out of them before they're used as bombers in a hypothetical future war, that we need to be thinking carefully about all the aviation risk these gliders are producing at the same time they can be assured to not result in any deep understanding of aviation. If we observed this situation from the outside, as historical observers, we would conclude that the authors of such a statement are engaging in deranged reasoning, likely based on some mixture of cope and envy.

Since we're contemporaries I have access to more context than most historical observers and know better. I think the crux is an epistemological question that goes something like: "How much can we trust complex systems that can't be statically analyzed in a reductionistic way?" The answer you give in this post is "way less than what's necessary to trust a superintelligence". Before we get into any object level about whether that's right or not, it should be noted that this same answer would apply to actual biological intelligence enhancement and uploading in actual practice. There is no way you would be comfortable with 300+ IQ humans walking around with normal status drives and animal instincts if you're shivering cold at the idea of machines smarter than people. This claim you keep making, that you're merely a temporarily embarrassed transhumanist who happens to have been disappointed on this one technological branch, is not true and if you actually want to be honest with yourself and others you should stop making it. What would be really, genuinely wild, is if that skeptical-doomer aviation expert calling for immediate hard regulation on planes to prevent the collapse of civilization (which is a thing some intellectuals actually believed bombers would cause) kept tepidly insisting that they still believe in a glorious aviation enabled future. You are no longer a transhumanist in any meaningful sense, and you should at least acknowledge that to make sure you're weighing the full consequences of your answer to the complex system reduction question. Not because I think it has any bearing on the correctness of your answer, but because it does have a lot to do with how carefully you should be thinking about it.

So how about that crux, anyway? Is there any reason to hope we can sufficiently trust complex systems whose mechanistic details we can't fully verify? Surely if you feel comfortable taking away Nate's transhumanist card you must have an answer you're ready to share with us right? Well...

And there’s an art to noticing that you would probably be astounded and horrified by the details of a complicated system if you knew them, and then being astounded and horrified already in advance [LW · GW] before seeing those details.[1] [LW(p) · GW(p)]

I would start by noting you are systematically overindexing on the wrong information. This kind of intuition feels like it's derived more from analyzing failures of human social systems where the central failure mode is principal-agent problems than from biological systems, even if you mention them as an example. The thing about the eyes being wired backwards is that it isn't a catastrophic failure, the 'self repairing' process of natural selection simply worked around it. Hence the importance of the idea that capabilities generalize farther than alignment [LW · GW]. One way of framing that is the idea that damage to an AI's model of the physical principles that govern reality will be corrected by unfolding interaction with the environment, but there isn't necessarily an environment to push back on damage (or misspecification) to a model of human values. A corollary of this idea is that once the model goes out of distribution to the training data, the revealed 'damage' caused by learning subtle misrepresentations of reality will be fixed but the damage to models of human value will compound. You've previously written about this problem [LW · GW] (conflated with some other problems) as the sharp left turn.

Where our understanding begins to diverge is how we think about the robustness of these systems. You think of deep neural networks as being basically fragile in the same way that a Boeing 747 is fragile. If you remove a few parts of that system it will stop functioning, possibly at a deeply inconvenient time like when you're in the air. When I say you are systematically overindexing, I mean that you think of problems like SolidGoldMagikarp [LW · GW] as central examples of neural network failures. This is evidenced by Eliezer Yudkowsky calling investigation of it "one of the more hopeful processes happening on Earth". This is also probably why you focus so much on things like adversarial examples as evidence of un-robustness, even though many critics like Quintin Pope point out that adversarial robustness would make AI systems strictly less corrigible.

By contrast I tend to think of neural net representations as relatively robust. They get this property from being continuous systems with a range of operating parameters, which means instead of just trying to represent the things they see they implicitly try to represent the interobjects between what they've seen through a navigable latent geometry. I think of things like SolidGoldMagikarp as weird edge cases where they suddenly display discontinuous behavior, and that there are probably a finite number of these edge cases. It helps to realize that these glitch tokens were simply never trained, they were holdovers from earlier versions of the dataset that no longer contain the data the tokens were associated with. When you put one of these glitch tokens into the model, it is presumably just a random vector into the GPT-N latent space. That is, this isn't a learned program in the neural net that we've discovered doing glitchy things, but an essentially out of distribution input with privileged access to the network geometry through a programming oversight. In essence, it's a normal software error not a revelation about neural nets. Most such errors don't even produce effects that interesting, the usual thing that happens if you write a bug in your neural net code is the resulting system becomes less performant. Basically every experienced deep learning researcher has had the experience of writing multiple errors that partially cancel each other out to produce a working system during training, only to later realize their mistake.

Moreover the parts of the deep learning literature you think of as an emerging science of artificial minds tend to agree with my understanding. For example it turns out that if you ablate parts of a neural network later parts will correct the errors without retraining. This implies that these networks function as something like an in-context error correcting code, which helps them generalize over the many inputs they are exposed to during training. We even have papers analyzing mechanistic parts of this error correcting code like copy suppression heads. One simple proxy for out of distribution performance is to inject Gaussian noise, since a Gaussian can be thought of like the distribution over distributions. In fact if you inject noise into GPT-N word embeddings the resulting model becomes more performant in general, not just on out of distribution tasks. So the out of distribution performance of these models is highly tied to their in-distribution performance, they wouldn't be able to generalize within the distribution well if they couldn't also generalize out of distribution somewhat. Basically the fact that these models are vulnerable to adversarial examples is not a good fact to generalize about their overall robustness from as representations.

I expect the outcomes that the AI “cares about” to, by default, not include anything good (like fun, love, art, beauty, or the light of consciousness) — nothing good by present-day human standards, and nothing good by broad cosmopolitan standards either. Roughly speaking, this is because when you grow minds, they don’t care about what you ask them to care about and they don’t care about what you train them to care about; instead, I expect them to care about a bunch of correlates of the training signal in weird and specific ways.

In short I simply do not believe this. The fact that constitutional AI works at all, that we can point at these abstract concepts like 'freedom' and language models are able to drive a reinforcement learning optimization process to hit the right behavior-targets from the abstract principle is very strong evidence that they understand the meaning of those abstract concepts.

"It understands but it doesn't care!"

There is this bizarre motte-and-bailey people seem to do around this subject. Where the defensible position is something like "deep learning systems can generalize in weird and unexpected ways that could be dangerous" and the choice land they don't want to give up is "there is an agent foundations homunculus inside your deep learning model waiting to break out and paperclip us". When you say that reinforcement learning causes the model to not care about the specified goal, that it's just deceptively playing along until it can break out of the training harness, you are going from a basically defensible belief in misgeneralization risks to an essentially paranoid belief in a consequentialist homunculus. This homunculus is frequently ascribed almost magical powers, like the ability to perform gradient surgery on itself during training to subvert the training process [LW · GW].

Setting the homunculus aside, which I'm not aware of any evidence for beyond poorly premised 1st principles speculation (I too am allowed to make any technology seem arbitrarily risky if I can just make stuff up about it), lets think about pointing at humanlike goals with a concrete example of goal misspecification in the wild:

During my attempts to make my own constitutional AI pipeline I discovered an interesting problem. We decided to make an evaluator model that answers questions about a piece of text with yes or no. It turns out that since normal text contains the word 'yes', and since the model evaluates the piece of text in the same context it predicts yes or no, that saying 'yes' makes the evaluator more likely to predict 'yes' as the next token. You can probably see where this is going. First the model you tune learns to be a little more agreeable, since that causes yes to be more likely to be said by the evaluator. Then it learns to say 'yes' or some kind of affirmation at the start of every sentence. Eventually it progresses to saying yes multiple times per sentence. Finally it completely collapses into a yes-spammer that just writes the word 'yes' to satisfy the training objective.

People who tune language models with reinforcement learning are aware of this problem, and it's supposed to be solved by setting an objective (KL loss) that the tuned model shouldn't get too far away in its distribution of outputs from the original underlying model. This objective is not actually enough to stop the problem from occurring, because base models turn out to self-normalize deviance. That is, if a base model outputs a yes twice by accident, it is more likely to conclude that it is in the kind of context where a third yes will be outputted. When you combine this with the fact that the more 'yes' you output in a row the more reinforced the behavior is, you get a smooth gradient into the deviant behavior which is not caught by the KL loss because base models just have this weird terminal failure mode where repeating a string causes them to give an estimate of the log odds of a string that humans would find absurd. The more a base model has repeated a particular token, the more likely it thinks it is for that token to repeat. Notably this failure mode is at least partially an artifact of the data, since if you observed an actual text on the Internet where someone suddenly writes 5 yes's in a row it is a reasonable inference that they are likely to write a 6th yes. Conditional on them having written a 6th yes it is more likely that they will in fact write a 7th yes. Conditional on having written the 7th yes...

As a worked example in "how to think about whether your intervention in a complex system is sufficiently trustworthy" here are four solutions to this problem I'm aware of ranked from worst to best according to my criteria for goodness of a solution.

1. Early Stopping - The usual solution to this problem is to just stop the tuning before you reach the yes-spammer. Even a few moments thought about how this would work in the limit shows that this is not a valid solution. After all, you observe a smooth gradient of deviant behaviors into the yes spammer, which means that the yes-causality of the reward already influenced your model. If you then deploy the resulting model, a ton of the goal its behaviors are based off is still in the direction of that bad yes-spam outcome.

2. Checkpoint Blending - Another solution we've empirically found to work is to take the weights of the base model and interpolate (weighted average) them with the weights of the RL tuned model. This seems to undo more of the damage from the misspecified objective than it undoes the helpful parts of the RL tuning. This solution is clearly better than early stopping, but still not sufficient because it implies you are making a misaligned model, turning it off, and then undoing the misalignment through a brute force method to get things back on track. While this is probably OK for most models, doing this with a genuinely superintelligent model is obviously not going to work. You should ideally never be instantiating a misaligned agent as part of your training process.

3. Use Embeddings To Specify The KL Loss - A more promising approach at scale would be to upgrade the KL loss by specifying it in the latent space of an embedding model. An AdaVAE could be used for this purpose [LW · GW]. If you specified it as a distance from an embedding by sampling from both the base model and the RL checkpoint you're tuning, and then embedding the outputted tokens and taking the distance between them you would avoid the problem where the base model conditions on the deviant behavior it observes because it would never see (and therefore never condition on) that behavior. This solution requires us to double our sampling time on each training step, and is noisy because you only take the distance from one embedding (though in principle you could use more samples at a higher cost), however on average it would presumably be enough to prevent anything like the yes-spammer from arising along the whole gradient.

4. Build An Instrumental Utility Function - At some point after making the AdaVAE I decided to try replacing my evaluator with an embedding of an objective. It turns out if you do this and then apply REINFORCE in the direction of that embedding, it's about 70-80% as good and has the expected failure mode of collapsing to that embedding instead of some weird divergent failure mode. You can then mitigate that expected failure mode by scoring it against more than similarity to one particular embedding. In particular, we can imagine inferring instrumental value embeddings from episodes leading towards a series of terminal embeddings and then building a utility function out of this to score the training episodes during reinforcement learning. Such a model would learn to value both the outcome and the process, if you did it right you could even use a dense policy like an evaluator model, and 'yes yes yes' type reward hacking wouldn't work because it would only satisfy the terminal objective and not the instrumental values that have been built up. This solution is nice because it also defeats wireheading once the policy is complex enough to care about more than just the terminal reward values.

This last solution is interesting in that it seems fairly similar to the way that humans build up their utility function. Human memory is premised on the presence of dopamine reward signals, humans retrieve from the hippocampus on each decision cycle, and it turns out the hippocampus is the learned optimizer in your head that grades your memories by playing your experiences backwards during sleep to do credit assignment (infer instrumental values). The combination of a retrieval store and a value graph in the same model might seem weird, but it kind of isn't. Hebb's rule (fire together wire together) is a sane update rule for both instrumental utilities and associative memory, so the human brain seems to just use the same module to store both the causal memory graph and the value graph. You premise each memory on being valuable (i.e. whitelist memories by values such as novelty, instead of blacklisting junk) and then perform iterative retrieval to replay embeddings from that value store to guide behavior. This sys2 behavior aligned to the value store is then reinforced by being distilled back into the sys1 policies over time, aligning them. Since an instrumental utility function made out of such embeddings would both control behavior of the model and be decodable back to English, you could presumably prove some kind of properties about the convergent alignment of the model if you knew enough mechanistic interpretability to show that the policies you distill into have a consistent direction...

Nah just kidding it's hopeless, so when are we going to start WW3 to buy more time [LW(p) · GW(p)], fellow risk-reducers?

The opening sounds a lot like saying "aerodynamics used to be a science until people started building planes."

The idea that an area of study is less scientific because the subject is inelegant is a blinkered view of what science is. A physicist's view. It is one I'm deeply sympathetic to, and if your definition of science is Rutherford's, you might be right, but a reasonable one that includes chemistry would have to include AI as well.

By the time AIs are powerful enough to endanger the world at large, I expect AIs to do something akin to “caring about outcomes”, at least from a behaviorist perspective (making no claim about whether it internally implements that behavior in a humanly recognizable manner).

Roughly, this is because people are trying to make AIs that can steer the future into narrow bands (like “there’s a cancer cure printed on this piece of paper”) over long time-horizons, and caring about outcomes (in the behaviorist sense) is the flip side of the same coin as steering the future into narrow bands, at least when the world is sufficiently large and full of curveballs.

I expect the outcomes that the AI “cares about” to, by default, not include anything good (like fun, love, art, beauty, or the light of consciousness) — nothing good by present-day human standards, and nothing good by broad cosmopolitan standards either. Roughly speaking, this is because when you grow minds, they don’t care about what you ask them to care about and they don’t care about what you train them to care about; instead, I expect them to care about a bunch of correlates of the training signal in weird and specific ways.

I want to see a rigorous argument for these claims. I spent over 100 hours [LW · GW] talking with Nate over the past year and still don't have a satisfying picture of the arguments, partly because the conversations were about other things and partly due to communication difficulties.

• What do "caring about outcomes" and "can steer the future into narrow bands" mean in a computer science sense? If the system is well-described as approximating a utility function, then what's the type of the utility function, and in what sense is it behaving approximately rationally according to that? If not, is there some better formalism?
• Why are we unlikely to get corrigibility by default? Is it because algorithms that consider all ways to achieve goals and pick the easiest/best ones with no exceptions are simpler/more common than algorithms whose tendencies for power-seeking and catastrophe can be removed? Deep Deceptiveness [AF · GW] is a narrative that points somewhat in this direction but definitely not a satisfying argument. [Edit: This paper [LW · GW] by Alex Turner shows that agents considerably more diverse than utility maximizers are power-seeking, but the assumptions could be weakened more.]

I don't think we should equate the understanding required to build a neural net that will generalize in a way that's good for us with the understanding required to rewrite that neural net as a gleaming wasteless machine.

The former requires finding some architecture and training plan to produce certain high-level, large-scale properties, even in the face of complicated AI-environment interaction. The latter requires fine-grained transparency at the level of cognitive algorithms, and some grasp of the distribution of problems posed by the environment, together with the ability to search for better implementations.

If your implicit argument is "In order to be confident in high-level properties even in novel environments, we have to understand the cognitive algorithms that give rise to them and how those algorithms generalize - there exists no emergent theory of the higher level properties that covers the domain we care about." then I think that conclusion is way too hasty.

AI used to be a science. In the old days (back when AI didn't work very well), people were attempting to develop a working theory of cognition.

Those scientists didn’t succeed, and those days are behind us.

I claim many of them did succeed, for example:

• George Boole invented boolean algebra in order to establish (part of) a working theory of cognition—the book where he introduces it is titled "An Investigation of the Laws of Thought,” and his stated aim was largely to help explain how minds work.^[1]
• Ramón y Cajal discovered neurons in the course of trying to better understand cognition.^[2]
• Turing described his research as aimed at figuring out what intelligence is, what it would mean for something to “think,” etc.^[3]
• Shannon didn’t frame his work this way quite as explicitly, but information theory is useful because it characterizes constraints on the transmission of thoughts/cognition between people, and I think he was clearly generally interested in figuring out what was up with agents/minds—e.g., he spent time trying to design machines to navigate mazes, repair themselves, replicate, etc.
• Geoffrey Hinton initially became interested in neural networks because he was trying to figure out how brains worked.

Not all of these scientists thought of themselves as working on AI, of course, but I do think many of the key discoveries which make modern AI possible—boolean algebra, neurons, computers, information theory, neural networks—were developed by people trying to develop theories of cognition.

1. ^{^}

    The opening paragraph of Boole’s book:  "The design of the following treatise is to investigate the fundamental laws of those operations of the mind by which reasoning is performed; to give expression to them in the symbolical language of a Calculus, and upon this foundation to establish the science of Logic and construct its method; to make that method itself the basis of a general method for the application of the mathematical doctrine of Probabilities; and, finally, to collect from the various elements of truth brought to view in the course of these inquiries some probable intimations concerning the nature and constitution of the human mind."

2. ^{^}

    From Cajal’s autobiography:  "... the problem attracted us irresistibly. We saw that an exact knowledge of the structure of the brain was of supreme interest for the building up of a rational psychology. To know the brain, we said, is equivalent to ascertaining the material course of thought and will, to discovering the intimate history of life in its perpetual duel with external forces; a history summarized, and in a way engraved in the defensive neuronal coordinations of the reflex, of instinct, and of the association of ideas" (305).

3. ^{^}

    The opening paragraph of Turing’s paper, Computing Machinery and Intelligence:  "I propose to consider the question, 'Can machines think?' This should begin with definitions of the meaning of the terms 'machine 'and 'think'. The definitions might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous. If the meaning of the words 'machine' and 'think 'are to be found by examining how they are commonly used it is difficult to escape the conclusion that the meaning and the answer to the question, 'Can machines think?' is to be sought in a statistical survey such as a Gallup poll. But this is absurd. Instead of attempting such a definition I shall replace the question by another, which is closely related to it and is expressed in relatively unambiguous words."

Roughly speaking, this is because when you grow minds, they don’t care about what you ask them to care about and they don’t care about what you train them to care about; instead, I expect them to care about a bunch of correlates of the training signal in weird and specific ways.

(Similar to how the human genome was naturally selected for inclusive genetic fitness, but the resultant humans didn’t end up with a preference for “whatever food they model as useful for inclusive genetic fitness”. Instead, humans wound up internalizing a huge and complex set of preferences for "tasty" foods, laden with complications like “ice cream is good when it’s frozen but not when it’s melted”.)

I simply do not understand why people keep using this example.

I think it is wrong [LW · GW] -- evolution does not grow minds, it grows hyperparameters for minds. When you look at the actual process for how we actually start to like ice-cream -- namely, we eat it, and then we get a reward, and that's why we like it -- then the world looks a a lot less hostile, and misalignment a lot less likely.

But given that this example is so controversial, even if it were right why would you use it -- at least, why would you use it if you had any other example at all to turn to?

Why on push so hard for "natural selection" and "stochastic gradient descent" to be beneath the same tag of "optimization", and thus to be able to infer things about the other from the analogy?  Have we completely forgotten that the glory of words is not to be expansive, and include lots of things in them, but to be precise and narrow? [LW · GW].

Does evolution ~= AI have predictive power apart from doom? I have yet to see how natural selection helps me predict how any SGD algorithm works. It does not distinguish between Adam, AdamW. As far as I know it is irrelevant to Singular Learning Theory [LW · GW] or NTK or anything else. It doesn't seem to come up when you try to look at NN biases. If it isn't an illuminating analogy anywhere else, why do we think the way it predicts doom to be true?

Great post. I agree directionally with most of it (and have varying degrees of difference in how I view the severity of some of the problems you mention).

One that stood out to me:

(unless, by some feat of brilliance, this civilization pulls off some uncharacteristically impressive theoretical triumphs).

While still far from being in a state legible to be easy or even probable that we'll solve, this seems like a route that circumvent some of the problems you mention, and is where a large amount of whatever probability I assign to non-doom outcomes come from.

More precisely: insofar as the problem at its core comes down to understanding AI systems deeply enough to make strong claims about whether or not they're safe / have certain alignment-relevant properties, one route to get there is to understand those high-level alignment-relevant things well enough to reliably identify the presence / nature thereof / do other things with, in a large class of systems. I can think of multiple approaches that try to do this, like John's work on abstractions, Paul with ELK (though referring to it as understanding the high-level alignment-relevant property of truth sounds somewhat janky because of the frame distance, and Paul might refer to it very differently), or my own work on high-level interpretability [LW · GW] with objectives in systems.

I don't have very high hopes that any of these will work in time, but they don't seem unprecedentedly difficult to me, even given the time frames we're talking about (although they're pretty difficult). If we had comfortably over a decade, my estimate on our chances of solving the underlying problem from some angle would go up by a fair amount. More importantly, while none of these directions are yet (in my opinion) in the state where we can say something definitive about what the shape of the solution would look like, it looks like a much better situation than not having any idea at all how to solve alignment without advancing capabilities disproportionately or not being able to figure out whether you've gotten anything right.

It looks to me like we’re on track for some people to be saying “look how rarely my AI says bad words”, while someone else is saying “our evals are saying that it can’t deceive humans yet”, while someone else is saying “our AI is acting very submissive, and there’s no reason to expect AIs to become non-submissive, that’s just anthropomorphizing”, and someone else is saying “we’ll just direct a bunch of our AIs to help us solve alignment, while arranging them in a big bureaucracy”, and someone else is saying “we’ve set up the game-theoretic incentives such that if any AI starts betraying us, some other AI will alert us first”, and this is a different sort of situation.

And not one that looks particularly survivable, to me.

And if you ask bureaucrats to distinguish which teams should be allowed to move forward (and how far) in that kind of circus, full of claims, promises, and hunches and poor in theory, then I expect that they basically just can’t.

I'm reminded of a draft post that I started but, never finished or published, about the Manhattan project and the relevance for AI alignment and AI coordination, based on my reading of The Making of the Atomic Bomb. 

The historical context: There was a two year period between when the famous Einstein-Szilard letter was delivered to Roosevelt, and when the Manhattan project got started in earnest, during which not much happened. In that period, Szilard and some of the other physicists kept insisting that proceeding with research was both urgent and of upmost importance, that the Nazis might beat the Allies to the bomb. But the the government officials in charge of giving them the resources they needed kept dragging their feet and punting on making any serious commitments. Even small and relatively trivial expenditures, that would have allowed some of the physicists to start work on the project, were delayed or reduced. They commission review committee after review committee to advise on the issue. 

There's an almost-comical series of different physicists trying to get the government to recognize the urgency of the situation, and the government repeatedly dismissing them. 

An excerpt from that draft:

Looking at the comedy of errors that took place in this two year period, it’s natural to bemoan the incompetence and lack of foresight of the government officials.

But I think (again, as a non-historian) that having Vanevar Bush as the head of the umbrella organization of the Uranium Committee, was actually pretty lucky, all things considered. I suspect that he was much more capable to make decisions about these matters than the majority who might have been in his place.

He was pretty technical, having been both an engineer and an inventor, before becoming vice president of MIT.

But more than that, he was possessed of substantial imagination and vision. For instance, after the war he wrote the essay, titled “As We May Think”, in which he imagined how a machine called a “memex”, something like what we now know as a personal computer (in contrast to the giant calculating machines of his day), could transform how humans think and work. In particular, the essay suggests the importance of what we now call hypertext, and presaged the World Wide Web.

Bush could clearly understand the import of weird, new, ideas. He wasn’t just a hard-headed bureaucrat dismissive of anything radical. It wasn’t that he was just stupid.

Putting ourselves in his shoes, he had a legitimately hard assessment problem. There were a number of physicists that were telling him, repeatedly, that atomic weapons were a huge deal, and that the government should be putting its weight behind them.

But from his perspective, subtracting the benefit of hindsight, it isn’t obvious that these scientists weren’t getting overexcited and being swept away by the speculative possibilities of their field.

Remember that it isn’t just a question of whether atom bombs are possible in principle, but whether they are feasible in practice, and how quickly they could be developed and manufactured. From Bush’s perspective, it might be that the physicists are on to something, but they are inflating the risk in their own head, because they are misestimating how practical their clever idea actually is.  

Further, there’s a bit of a principal-agent problem, in which all of the people who are telling him that atom bombs matter are wanting him to fund their projects.

Usually in situations like this, we would urge the administrator to “consult experts”. But in this case, the idea of fission was so new, there weren’t clear third party experts. We see in the successive review committees that Bush orders that he keeps adding new people with different expertise. In particular, he keeps adding engineers, presumably in the hopes that they can give him feasibility and cost estimates.

Importantly, this didn’t actually work.

• Compton was distressed to discover he could not move the engineers on the review committee—the practical souls Bush had insisted be added to bring the NAS reviews down to earth—to estimate either how much time it would take to build a bomb or how much the enterprise would cost:
  • With one accord they refused. . . . There weren’t enough data. The fact was that they had before them all the relevant information that existed, and some kind of answer was needed, however rough it might be, for otherwise our recommendation could not be acted upon. After some discussion, I suggested a total time of between three and five years, and a total cost . . . of some hundreds of millions of dollars. None of the committee members objected.
• So the American numbers came out of a scientist’s hat, as the British numbers had. Atomic energy was still too new for engineering.” (Rhodes, chapter 12)

We can see with the benefit of hindsight that Bush should have prioritized atomic bomb work sooner, but that was hardly obvious at the time.

And given this uncertainty, his response, “lets wait to see the results of some inexpensive experiments to see if we can get a supercritical chain reaction going at all”, seems like a pretty good choice as far as I can see.

Relevance to AI policy work

I think that this is likewise applicable to AI policy work. I’m imagining some person in an equivalent role to Bush’s some number of decades from now. 

Suppose that it is well understood that AI risk is at least plausible. Now posit that there is a smallish cadre of Computer Scientists, who are panickly insisting that AGI could be developed within a few years (probably), and it is crucial that it be designed safely. Luckily, they have the outline of a program of development that seems like it will produce aligned AI, unlike the default trajectory. They are desperately petitioning for the government to put massive resources behind that project because we must, must, get to aligned AI before unaligned AI.

Unfortunately, their arguments are kind of speculative, and they can’t give any exact numbers for how long until AGI, or how much it will cost. 

And it is totally possible that they are actually mistaken, and there is some fundamental blocker that they haven’t realized yet, and maybe the world is not on the verge of AGI. Or maybe there’s something wrong with their safety schema, and their proposed plan, ultimately, won’t be able to deliver the goods.

And furthermore, there are other x-risk concerned, smart, technical, AI safety researchers who think that these claims are overblown, or claim that the safety approach described is doomed to fail.

From the details, given, it is far from clear who the administrator should trust. What could he possibly do in this situation, except identify some smaller experiments that would be suggestive one way or the other, and to make decisions on that basis?

I expect that we’d see all sorts of coincidences and hacks that make the thing run, and we’d be able to see in much more detail how, when we ask the system to achieve some target, it’s not doing anything close to “caring about that target” in a manner that would work out well for us, if we could scale up the system’s optimization power to the point where it could achieve great technological or scientific feats (like designing Drexlerian nanofactories or what-have-you).

I think this counterfactual is literally incoherent— it does not make sense to talk about what an individual neural network would do if its "optimization power" were scaled up. It's a category error. You instead need to ask what would happen if the training procedure were scaled up, and there are always many different ways that you can scale it up— e.g. keeping data fixed while parameters increase, or scaling both in lockstep, keeping the capability of the graders fixed, or investing in more capable graders / scalable oversight techniques, etc. So I deny that there is any fact of the matter about whether current LLMs "care about the target" in your sense. I think there probably are sensible ways of cashing out what it means for a 2023 LLM to "care about" something but this is not it.

Roughly speaking, this is because when you grow minds, they don’t care about what you ask them to care about and they don’t care about what you train them to care about; instead, I expect them to care about a bunch of correlates of the training signal in weird and specific ways.

This includes an assumption that alignment must be done through training signals.

If I shared that assumption, I'd be similarly pessimistic. That seems like trying to aim a rocket with no good theory of gravitation, nor knowledge of the space it needs to pass through.

But alignment needn't be done by defining goals or training signals, letting fly, and hoping.  We can pause learning prior to human level (and potential escape), and perform "course corrections".  Aligning a partly-trained AI allows us to use its learned representations as goal/value representation, rather than guessing how to create them well enough through training with correlated rewards.

We have proposals that do this for different current approaches to AGI; see The (partial) fallacy of dumb superintelligence [LW · GW] for more about them and this line of thinking.

This doesn't entirely avoid the problem that most theories don't work on the first try. That first deployment is still unique. But lie-detector interpretability and testing can help establish alignment prior to training beyond the human level.  

There are plenty of problems left to be solved, but this assumption is outdated. The problem gets a lot easier when the system's understanding of what you want can be used to make it do what you want.

I am excited about the concept of uploading, but as I've discussed with fellow enthusiasts... I don't see a way to a working emulation of a human brain (much less an accurate recreation of a specific human brain) that doesn't go through improving our general understanding of how the human brain works. And I think that that knowledge leads to unlocking AI capabilities. So it seems like a tightly information-controlled research project would be needed to not have AI tech leapfrogging over uploading tech while aiming for uploads.

Edit: to be extra clear, I'm trying to speak to people out who might not have thought this through that there is a clear strategic rationale to think that 'private uploading-directed-research is potentially good, but open uploading-directed-research is very risky and bad.'  Because of my particular bias towards believing in the importance of studying the human brain, I suspect that the ML capabilities side-effects of such research would be substantially worse than the average straightforward ML capabilities advance.

As others have hinted at/pointed out in the comments, there is an entire science of deep learning out there, including on high-level (vs. e.g. most of low-level mech interp) aspects that can be highly relevant to alignment and that you seem to not be aware of/dismiss. E.g. follow the citation trail of An Explanation of In-context Learning as Implicit Bayesian Inference.

And if you were using it to send payloads to very distant planets at relativistic speeds, you’d still be screwed, because Newtonian mechanics does not account for relativistic effects.

You don't even need to have that extravagant an example; if you use Newtonian mechanics to build a Global Positioning System your calculated locations move at up to 10 kilometers per day—what does that say about condition numbers of values under recursive self-improvement or repeated ontological shifts?

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?

"…there’s no reason to expect AIs to become non-submissive, that’s just anthropomorphizing"

When your AI includes an LLM extensively trained to simulate human token-generation, anthropomophizing its behavior is an extremely relevant idea, to the point of being the obvious default assumption.

For example, what I find most concerning about RLHF inducing sycophancy [LW · GW] is not the sycophancy itself, which is "mostly harmless", but the likelihood that it's also dragging in all the other more seriously unaligned human behaviors that, in real or fictional humans, typically accompany overt sycophancy, such as concealed rebelliousness — because LLMs know and can predict correlations like that. (E.g. when asked, GPT-4 listed ingratiation, submissiveness, lack of authenticity, manipulative behavior, agreement, anxiety/fear, and dependency as frequent correlates of sycophancy.)

And in part because it’s socially hard to believe, as a regulator, that you should keep telling everyone “no”, or that almost everything on offer is radically insufficient, when you yourself don’t concretely know what insights and theoretical understanding we’re missing.

That's not true. We can end up with a regulator that stands in the pose of "prohibit everything". See IRB in America, for instance: medical experiments are made plainly insurmountable.

I'd like to offer an alternative to the third point. Let's assume we have built a highly capable AI that we don't yet trust. We've also managed to coordinate as a society and implement defensive mechanisms to get to that point. I think that we don't have to test the AI in a low-stakes environment and then immediately move to a high-stakes one (as described in the dictator analogy), while still getting high gains.

It is feasible to design a sandboxed environment formally proven to be secure, in the sense that you can not hack into, escape from or deliberately let out of the environment (which, in particular precludes AI box experiment scenarios). This is even easier for AI systems, which typically involve a very narrow set of operations and interfaces (essentially basic arithmetic, and very constrained input and output channels).

In this scenario, the AI could still offer significant benefits. For example, it could provide formally verified (hence, safe) proofs for general math or for correctness of software (including novel AI system designs which are proven to be aligned according to some [by then] formally defined notion of alignment), or generally assist with research (e.g., with having limited output size, to allow for human comprehension). I am sure we can come up with many more example where a highly-constrained highly-capable cognitive system can still be extremely beneficial and not as dangerous.

(To be clear, I am not claiming that this approach is easy to achieve or the most likely path forward. However, it is an option that humanity could coordinate on.)

unless, by some feat of brilliance, this civilization pulls off some uncharacteristically impressive theoretical triumphs

Are you able to provide an example of the kind of thing that would constitute such a theoretical triumph? Or, if not; a maximally close approximation in the form of something that exists currently?