Which possible AI systems are relatively safe?

A note on this part:

If its true thoughts are transparent and expressed in natural language...

Claim: it is impossible for the internal thoughts of a mind to both be expressed mostly in natural language, and to use that natural language in a way at all similar to humans (for instance, no steganography). The reason is that, the way humans use natural language in practice, the words are largely used to "pull mental levers", and then most of the load-bearing reasoning happens in the non-linguistic cognition those levers induce. So if language is used the way humans use it, then most of the cognition is hidden away in non-linguistic channels.

You can see this in practice every time someone has trouble expressing their thoughts in words. Or every time someone is able to express their thoughts in words to someone who already has a bunch of shared mental models (even without necessarily shared jargon to point to those models - e.g. one can say "you know that thing where..." and give a couple examples), but is unable to express their thoughts in words to someone who doesn't already have the relevant mental models.

The closest unblocked requirement would be an AI with lots of parts separated by internal interfaces, where those interfaces use natural language in a human-like way. There's still a lot of non-natural-language cognition going on in between the natural language in and the natural language out, but the interfaces might still provide a lot of useful visibility of intermediates. Basically-all of today's LLMs would be examples of such an architecture: they take natural language in, do some magic, then spit out one more token of natural language.

General comment on this list: the mental image behind most of the items seems to be roughly "break the system into parts, which are individually interpretable and nonmalicious". Intermediates expressed in natural language, decomposition into interpretable subtasks, myopia, minimal hidden state, lack of situational awareness, legibility, process-based, and composition of narrow tools all tie into that general pattern.

These all share similar shortcomings: interpretability/tool-ness/corrigibility/etc are not composable [? · GW], and also (depending on which versions of these ideas one imagines) the not-yet-well-written-up problem of "you don't get to choose the ontology".

That's not really a problem for any of these properties individually - they're each still potentially worthwhile properties which make AI relatively safer. But in aggregate, bear in mind that there's decreasing marginal returns to properties which all pursue roughly-similar upsides and have roughly-similar shortcomings. If this list is your starting point, then there's probably a lot more marginal value in adding properties which which address e.g. the composability problem, rather than additional similar properties to those listed.

I think narrow domain might be one of the most important properties. An AI that is very good at designing drugs and nothing else is much safer than an AGI; it doesn't understand itself or the world at large and can't go on a self improvement rampage. It still has risks but they're ordinary "don't let a radical political terrorist in a BSL4 lab" kinda risks.

Yo Shavit says:

I wish we talked more about which particular AI systems design principles give us confidence in safety.

For example: context is always erased; humans review ~all context and output tokens.

These principles are likely to disappear unless we center them in our analysis & demands.

Less worried rn about issues like "RLHF incentivizes deception"

Much more worried about "a pair of mostly-aligned AI systems talk to each other for 3 hours and then make POST requests, and no human actually reviews the full transcript"

I think it's good this post exists. But I really want to make the distinction between "safe" and "is a solution to the alignment problem," which this post elides. Or maybe "safe" vs. "friendly"?

If we build a superhuman AGI we'd better have solved the alignment problem in the sense of actually making that AGI want to do good things and not bad things. (Past a certain point, "just follow orders" isn't safe unless the order "do good things" works. If it wouldn't work, you've built an unsafe AI, and if it would work, you might as well give it.)

OpenAI's "parallelizable alignment assistant" strategy can work for between 0 and 4 organizations in the world, because it relies on having enough of a lead that you can build something that is safe yet not a solution to the alignment problem, and nobody else will cause an accident in the weeks or months you spend trying to convert this into a solution to the alignment problem.

To look at one example property: taking a random AI and putting a human in the loop makes it more safe. But it does little to nothing for solving alignment. It helps when you're building an AI that's dumber than you, but doesn't really when you're building an AI that's smarter than you.

Or lack of situational awareness. This is actively anti-alignment, because the state of the world is useful information for doing good things. But it's even more anti-capabilities, so it's a fine property to shoot for if you're making an AI that's safe because it has limited capabilities.

I'll come back later with a comment that actually makes suggestions, both ones that trade off for safety and for friendliness.

If its true thoughts are transparent and expressed in natural language(see e.g. Measuring Faithfulness in Chain-of-Thought Reasoning)

This seems technically true but a bit of a trap, since it may be easier to get ‘looks like it expresses its thoughts in natural language’ than ‘reliably actually does’ and specifying the difference may be too subtle for people. 

what lower-level desirable properties determine corrigibility?
 

Has corrigible (or alignment) properties embedded in the attention weights.

All else equal, I think minimizing model entropy is desirable (i.e. the number of weights). In other words, you want to keep the size of the model class small.

Roughly, alignment could be viewed as constructing a list of constraints or criteria that a model must satisfy in order to be considered safe. As the size of the model class grows, more models will satisfy any particular constraint. The complexity of the constraints likely needs to grow along with the complexity of the model class.

If a large number of models satisfy all the constraints, there is a large amount of behavior that is unconstrained and unaccounted for. We've decided that we don't care about any of the behavioral differences between the models that satisfy all the constraints.

This isn't necessarily true. Modern DL models are semi-organically grown rather than engineered, so the set of SGD discoverable models is much smaller than the set of all possible models. And techniques like iterative amplification further shrink the set of learnable models. Or maybe many of the models are behaviorally identical on the subset of inputs we care about.

That said, thinking about model entropy seems helpful.