Updating Drexler's CAIS model

I wanted to add several additional elements to the base CAIS model that recent advances and my own experience make obvious:

A.  Stateless, limited time sessions.  Break tasks into the smallest granular subtask, and erase input buffers after each subtask completes, starting over.  This is fundamentally how the stateless microservices of all the large tech giants allow any kind of reliability for hyperscale online services.

B.  Check if the input this frame is in the training distribution.  One way to implement this is to train an autoencoder on the training set, and autoencode each input frame to the model.  So (encoded frame, residual) = autoencoder (input frame).  "frame"** is a discrete set of inputs, it's the sum of all inputs to the model.  

If the input is out of distribution, measured by a large residual, do not send the input to the model.  Transfer control to a trustworthy embedded system.  Autonomous vehicles use a strategy like this: on main system failure/low confidence, control authority is given to a system that uses a single camera and works more like older lane following systems.   If that system fails from hardware faults, the base safety microcontroller takes away control authority from the upper systems and engages the brakes.  

C.  The outer framework must have control authority.  The model isn't in charge, the outer wrapper code (written and carefully validated by humans) ultimately is.

D.  Inputs and outputs to the model must be sparse.  No extra bits.  For an example of this, if you ask GPT-4 to write a piece of code, then move the output text to a different session, the new session of GPT-4 has no context information but the text, which is human readable.  This is fairly sparse, it's difficult for one session of the model to embed "hey fellow AI, keep this bug in the code hidden from the meatsacks" such that the next session will see the secret message.  Even more difficult, close to impossible, if you have the model doing the checking based on a different base model that shares no common context for a secret language.  

Perfectly sparse would be an encoded schema that has nothing but the task instructions, written in a formal schema, and only the information needed to proceed from prior frames.

E.  Different base models must be sufficiently intercompatible you can use them to check each other's work.  For example, there is no reason you couldn't alternate control every frame for a robot using 2 completely different base models, so long as both are rational control systems and there is 1 and only 1 optimal policy that both models can calculate for each frame.

F.  Finally, by using the sparse I/O in D, it must be impossible for models to know if they are operating in the real world and not in the training simulator.  If they get no context but the frame and the prompt, and the frame is in distribution to the training simulator, they cannot know they are not in the sim, which obviously can detect most forms of simulated wrongdoing. 

G.  Train only in the simulator.  The latest generative models are capable of taking real world data and producing many permutations on it that are realistic training examples.  So when the system encounters an unexpected outcome in the real world or a novel situation, you save to an onboard log the frame*, expected outcome, ground truth, etc.  And use a generative model to augment the unexpected situation so that in the simulator the model can practice on many permutations of the situation.  

Since the model's performance in the simulator was measured to be sufficiently safe for the task it is performing, and the simulator was measured to predict the real world sufficiently accurately (you run an instance of the simulator in parallel in the real world on each robot), and you will not give the model control authority when the real world is out of distribution, you can bound your safety probabilities and prove the model is sufficiently* safe for the task it is assigned.  
 

**the "frame" is all inputs, for example it might be a lidar image, 6 cameras, all the currently tracked objects that have been seen on prior frames and their estimated identities and vectors, intermediate internal data that caches some activations from the prior frame, and of course the prompt.  The model(frame, weights, RNG seed) should deterministically produce the same output.  (current sw stacks are not quite deterministic but could be at a small cost of performance)

*depends on the scope of the task.  No system will be perfect, the idea is to stack probabilities to reach your design error rate.  A robot performing heart surgery will need much more reliability than one stacking boxes.  The heart surgery machine may need electromechanical limits because you may have to leave the model running even when it leaves distribution, since stopping an operation is more likely to be fatal than continuing.

Electromechanical limits are things like embedding the hardware for the machine on site, hardwired for one purpose, and not mounting the robotic arms on mobile chassis.  This limits failures to killing the patient instead of the machine able to say escape into the hospital and 'perform surgery' on random people.

Drexler can be forgiven for not talking about foundation models in his report. His report was published at the start of 2019, just months after the idea of "fine-tuning" was popularized in the context of language models, and two months before GPT-2 came out. And many readers can no doubt point out many non-trivial predictions that Drexler got right, such as the idea that we will have millions of AIs, rather than just one huge system that acts as a unified entity. And we're still using deep learning as Drexler foresaw, rather than building general intelligence like a programmer would. Like I said at the beginning, it's not necessarily that the core elements of the CAIS model are wrong; the model just needs an update.

I think this is a good post, and I like the analysis you give in many ways, but I have a small bone to pick with this one. I don't think Drexler was right that we will have millions of AIs rather than just one huge system that acts as a unified entity; I think things are trending in the direction of one huge system that acts as a unified entity and have already trended in that direction substantially since the time Drexler wrote (e.g. ChatGPT-4 is more unified than one would have expected from reading Drexler's writing back in the day).

Who was it Drexler was arguing against, who thought that we wouldn't be using deep learning in 2023?

IMO Yudkowsky's model circa 2017 is looking more prophetic than Drexler's Comprehensive AI Services, take a look at e.g. this random facebook comment from 2017:

Eliezer (6y, via fb):

So what actually happens as near as I can figure (predicting future = hard) is that somebody is trying to teach their research AI to, god knows what, maybe just obey human orders in a safe way, and it seems to be doing that, and a mix of things goes wrong like:

The preferences not being really readable because it's a system of neural nets acting on a world-representation built up by other neural nets, parts of the system are self-modifying and the self-modifiers are being trained by gradient descent in Tensorflow, there's a bunch of people in the company trying to work on a safer version but it's way less powerful than the one that does unrestricted self-modification, they're really excited when the system seems to be substantially improving multiple components, there's a social and cognitive conflict I find hard to empathize with because I personally would be running screaming in the other direction two years earlier, there's a lot of false alarms and suggested or attempted misbehavior that the creators all patch successfully, some instrumental strategies pass this filter because they arose in places that were harder to see and less transparent, the system at some point seems to finally "get it" and lock in to good behavior which is the point at which it has a good enough human model to predict what gets the supervised rewards and what the humans don't want to hear, they scale the system further, it goes past the point of real strategic understanding and having a little agent inside plotting, the programmers shut down six visibly formulated goals to develop cognitive steganography and the seventh one slips through, somebody says "slow down" and somebody else observes that China and Russia both managed to steal a copy of the code from six months ago and while China might proceed cautiously Russia probably won't, the agent starts to conceal some capability gains, it builds an environmental subagent, the environmental agent begins self-improving more freely, undefined things happen as a sensory-supervision ML-based architecture shakes out into the convergent shape of expected utility with a utility function over the environmental model, the main result is driven by whatever the self-modifying decision systems happen to see as locally optimal in their supervised system locally acting on a different domain than the domain of data on which it was trained, the light cone is transformed to the optimum of a utility function that grew out of the stable version of a criterion that originally happened to be about a reward signal counter on a GPU or God knows what.

Perhaps the optimal configuration for utility per unit of matter, under this utility function, happens to be a tiny molecular structure shaped roughly like a paperclip.

That is what a paperclip maximizer is. It does not come from a paperclip factory AI. That would be a silly idea and is a distortion of the original example.
 

I think there's a trilemma with updating CAIS-like systems to the foundational model world, which is: who is doing the business development?

I came up with three broad answers (noting reality will possibly be a mixture):

1. The AGI. This is like Sam Altman's old answer or a sovereign AI; you just ask it to look at the world, figure out what tasks it should do, and then do them.
2. The company that made the AI. I think this was DeepMind's plan; come up with good AI algorithms, find situations where those algorithms can be profitably deployed, and then deploy them, maybe with a partner.
3. The broader economy. This looks to me like OpenAI's current plan; release an API and encourage people to build companies on top of it.

[In pre-foundational-model CAIS, the answer was obviously 3--every business procures its own AI tools to accomplish particular functions, and there's no 'central planning' for computer science.]

I don't think 1 is CAIS, or if it is, then I don't see the daylight between CAIS and good ol' sovereign AI. You gradually morph from the economy as it is now to central planning via AGI, and I don't think you even have much guarantee that it's human overseen or follows the relevant laws.

I think 2 has trouble being comprehensive. There are ten thousand use cases for AI; the AI company has to be massive to have a product for all of them (or be using the AI to do most of the work, in which case we're degenerating into case 1), and then it suffers from internal control problems. (This degenerates into case 3, where individual product teams are like firms and the company that made the AI is like the government.)

I think 3 has trouble being non-agentic and peaceful. Even with GPT-4, people are trying to set it up to act autonomously. I think the Drexlerian response here is something like:

Yes, but why expect them to succeed? When someone tells GPT-4 to make money for it, it'll attempt to deploy some standard strategy, which will fail because a million other people are trying to exact same thing, or only get them an economic rate of return ("put your money in index funds!"). Only in situations where the human operators have a private edge on the rest of the economy (like having a well-built system targeting an existing vertical that the AI can slot into, you have pre-existing tooling able to orient to the frontier of human knowledge, etc.) will you get an AI system with a private edge against the rest of the economy, and it'll be overseen by humans.

My worry here mostly has to do with the balance between offense and defense. If foundational-model-enabled banking systems are able to detect fraud as easily as foundational-model-enabled criminals are able to create fraud, then we get a balance like today's and things are 'normal'. But it's not obvious to me that this will be the case (especially in sectors where crime is better resourced than police are, or sectors where systems are difficult to harden).

That said, I do think I'm more optimistic about the foundational model version of CAIS (where there can be some centralized checks on what the AI is doing for users) than the widespread AI development version. 

However, after looking back on it more than four years later, I think the general picture it gave missed some crucial details about how AI will go.

I feel like this is understating things a bit.

In my view (Drexler probably disagrees?), there are two important parts of CAIS:

1. Most sectors of the economy are under human supervision and iteratively reduce human oversight as trust is gradually built in engineered subsystems (which are understood 'well enough').
2. The existing economy is slightly more sophisticated than what SOTA general AI can produce, and so there's not much free energy for rogue AI systems to eat or to attract investment in creating general AI. (General AI is dangerous, and the main goal of CAIS is to get the benefits without the drawbacks.)

I think a 'foundation model' world probably wrecks both. I think they might be recoverable--and your post goes some of the way to making that visualizable to me--but it still doesn't seem like the default outcome.

[In particular, I like the point [LW(p) · GW(p)] about models with broad world models can still have narrow responsibilities, and think that likely makes them more likely to be safe, at least in the medium term. Having one global moral/law-abiding foundational AI model that many people then slot into their organizations seems way better than everyone training whatever AI model they need for their use case.]

I kind of disagree with your assertion that AI tools will now descend more directly from a common ancestor. GPT is not terribly general, just a language processor with a few more related tricks thrown it. It is very broad, but also very shallow, and it is not clear if it can become a tool of all trades. We will need a lot of domain-specific tools, that are best developed independently, with some synergies where it makes sense, and GPT won't change that. 

I agree that people have gotten vibes from the paper which have been somewhat discredited.

Yet I don't see how that vibe followed from what he wrote. He tried to clarify that having systems with specialized goals does not imply they have only narrow knowledge. See section 21 of the CAIS paper ("Broad world knowledge can support safe task performance").

Are people collapsing "AI with narrow goals" and "AI with only specialized knowledge" into one concept "narrow AI"?

And many readers can no doubt point out many non-trivial predictions that Drexler got right, such as the idea that we will have millions of AIs, rather than just one huge system that acts as a unified entity. And we're still using deep learning as Drexler foresaw, rather than building general intelligence like a programmer would.

One of the simpler and more important lessons one learns from research on forecasting: be wary of evaluating someone’s forecasting skill by drawing up a list of predictions they got right and wrong—their “track record.” One should compare Drexler’s performance against alternative methods/forecasters (especially for a forecast like “we’re still using deep learning”). I’m not saying this is nothing, but I felt compelled to highlight this given how often I’ve seen this potential failure mode.

It now seems clear that AIs will also descend more directly from a common ancestor than you might have naively expected in the CAIS model, since almost every AI will be a modified version of one of only a few base foundation models. That has important safety implications, since problems in the base model might carry over to problems in the downstream models, which will be spread thorought the economy. That said, the fact that foundation model development will be highly centralized, and thus controllable, is perhaps a safety bonus that loosely cancels out this consideration.

The first point here (that problems in a widely-used base model will propagate widely) concerns me as well. From distributed systems we know that

1. Individual components will fail.
2. To withstand failures of components, use redundancy and reduce the correlation of failures.

By point 1, we should expect alignment failures. (It's not so different from bugs and design flaws in software systems, which are inevitable.) By point 2, we can withstand them using redundancy, but only if the failures are sufficiently uncorrelated. Unfortunately, the tendency towards monopolies in base models is increasing the correlation of failures.

As a concrete example, consider AI controlling a military. (As AI improves, there are increasingly strong incentives to do so.) If such a system were to have a bug causing it to enact a military coup, it would (if successful) have seized control of the government from humans. We know from history that successful military coups have happened many times, so this does not require any special properties of AI.

Such a scenario could be prevented by populating the military with multiple AI systems with decorrelated failures. But to do that, we'd need such systems to actually be available.

It seems to me the main problem is the natural tendency to monopoly in technology. The preferable alternative is robust competition of several proprietary and open source options, and that might need government support. (Unfortunately, it seems that many safety-concerned people believe that competition and open source are bad, which I view as misguided for the above reasons.)

This seems to be partially based on (common?) misunderstanding of CAIS as making predictions about concentration of AI development/market power.  As far as I can tell this wasn't Eric's intention: I specifically remember Eric mentioning he can easily imagine the whole "CAIS" ecosystem living in one floor of DeepMind building. 
 

GPT-4 is more general than GPT-3, which was more general than GPT-2, and presumably this trend will continue as we scale up our models indefinitely, rather than shooting up discontinuously past human level at some point.

I don't presume that at all. I think that is in fact something we should not expect. I think that as soon as we cross the threshold where, as you say above "it will begin to meaningfully feed into itself, increasing AI R&D itself, accelerating the rate of technological progress" we should expect that the thing we will see is "shooting up discontinuously past human level". This 'shooting up' seems likely to me to be continuous in the sense that there will be multiple model iterations, each a bit better than the last, rather than a single training run which goes all the way above human level from near current GPT-4 level. It will not be continuous in a historical sense, it will look like a clear departure from a straight line fit to the progress of GPT-1, 2, 3, 4 and the dates at which those progress points occurred. Like, we should expect a period less than 1 year until > human intelligence and generality after the recursive improvement process starts. This is not just my opinion, this has been mentioned in Anthropic's expectations and by Tom Davidson's report. 

Quote from Anthropic: "These models could begin to automate large portions of the economy. ... We believe that companies that train the best 2025/26 models will be too far ahead for anyone to catch up in subsequent cycles."

It seems to me like your model is not necessarily taking into account technical debt sufficiently enough. https://neurobiology.substack.com/p/technical-debt-probably-the-main-roadblack-in-applying-machine-learning-to-medicine

It seems to me like this is the main thing that will slow down the extent to which foundation models can consistently beat newly trained specialized models.

Anecdotally, I know several people who don’t like to use chatgpt because its training cuts off in 2021. This seems like a form of technical debt.

I guess it depends on how easily adaptable foundation models are.