Systems that cannot be unsafe cannot be safe

My key point: I strongly agree with (my perception of) the intuitions behind this post, but I wouldn't agree with the framing. Or with the title. So I think it would be really good to come up with a framing of these intuitions that wouldn't be controversial.

To illustrate on an example: Suppose I want to use a metal rod for a  particular thingy in a car. And there is some safety standard for this, and the metal rod meets this standard.^[1] And now suppose you instead have that same metal rod, except the safety standard does not exist. I expect most people to argue that your your car will be exactly as safe in both cases.

Now, the fact that the rod is equally safe in the two cases does not mean that using it in the two cases is equally smart. Indeed, my personal view is that without the safety standard, using the rod for your car would be quite dumb. (And using the metaphorical rod for your superintelligent AI would be f.....g insane.) But I think that saying things like "any model without safety standards is unsafe" is misleading, and likely to be unproductive when communicating with people who have a different mindset.^[2]

1. ^{^}

   Then you might perhaps say that "the metal rod is safe for the purpose of being used for a particular thingy in a car"? However, this itself does not guarantee that the metal rod is actually safe for this purpose --- for that to be true, you additionally need the assumption that the safety standard is "reasonable". Where "reasonable" is undefined. And if you try to define it, you will eventually either need to rely on an informal/undefined/common-sense definition somewhere, or you need to figure out how to formalise literally everything. I am not giving up on that latter goal, but we aren't there yet :-).

2. ^{^}

   Personally, I think that being in any sort of decision-making or high-impact role while having such "different mindset" is analogous to driving a bus full of people without having a driver's license. However, what I think about this makes no meaningful impact on how things work in the real world...

I agree, and I made a related claim that alignment of a model [AF · GW] is mostly a type error. I think the same point applies to safety more generally: modulo exotic concerns about inner-optimizers, there is no such thing as an unsafe model, only unsafe applications of that model.

Though I also think the red teaming [AF · GW] that ARC did on GPT-4 actually did successfully demonstrate that GPT-4 is unlikely to fail in certain dangerous ways when integrated into any system which is not already dangerously capable in its own right. This is simultaneously a very impressive safety result for a novel AI model, and not very impressive in any other domain. For example, I'm not too impressed if a car manufacturer demonstrates that their car's entertainment system never takes control of the car and drives it off a cliff under any circumstances. (Though I still hope there is some auto manufacturing safety standard which covers such a case.)

In engineering and design, there is a process that includes, among other stages, specification, creation, verification and validation, and deployment. Verification and validation are where most people focus when thinking about safety - can we make sure the system performs correctly?

Factually, no, I don't think this is where most people's thoughts are. Apart from the stages of the engineering process that you enumerated, there are also manufacturing (i.e., training, in the case of ML systems) and operations (post-deployment phase). AI safety "thought" is more-or-less evenly distributed between design (thinking about how systems should be engineered/architectured/designed in order to be safe), manufacturing (RLHF, scalable oversight, etc.), V&V (evals, red teaming), and operations (monitoring). I discussed this in a little more detail here [LW(p) · GW(p)].

There is no pre-defined standard or set of characteristics that are checked, no real ability to consider application specific requirements, and no ability to specify where the system should not or must not be used.

No standard -- agree, "no ability to consider requirements" -- disagree, you can consider "requirements"^[1]. "No ability to specify where the system should not or must not be used" -- agreed if you specifically use the verb "to specify", but we also can consider "where the system should not or must not be used".

Until we have some safety standard for machine learning models, they aren't "partly safe" or "assumed safe," or "good enough for consumer use." If we lack a standard for safety, ideally one where there is consensus that it is sufficient for a specific application, then exploration or verification of the safety of a machine learning model is meaningless.

You seem to try to import quite an outdated understanding of safety and reliability engineering. Standards are impossible in the case of AI, but they are also unnecessary, as was evidenced by the experience in various domains of engineering, where the presence of standards doesn't "save" one from drifting into failures.

So, I disagree that evals and red teaming in application to AI are "meaningless" because there are no standards.

However, I agree that we must actively invite state-of-the-art thinking about high reliability and resilience in engineering to the conversation about AI safety & resilience, AGI org design for reliability, etc. I'd love to see OpenAI, Anthropic, and Google hire top thinkers in this field as staff or consultants.

1. ^{^}

   Modern systems engineering methodology moves away from "requirements", i.e., deontic specification of what a system should/must do or not do, to more descriptive modality, i.e., use cases. This transition is already complete in modern software engineering, where classic engineering "requirements" are unheard of in the last 10 years. But other engineering fields (electrical, robotic, etc.) move in this direction, too.

I agree with this post. However, I think it's common amongst ML enthusiasts to eschew specification and defer to statistics on everything. (Or datapoints trying to capture an "I know it when I see it" "specification".)

You're making the assumption that the safety methods for cars are appropriate to transfer directly to e.g. LLMs. That's not clearly true to me as there are strong differences in the nature of cars vs the nature of LLMs. For instance the purposes and capacities of cars are known in great detail (driving people from place to place), whereas the purposes of LLMs are not known (we just noticed that they could do a lot of neat things and assumed someone will find a use-case for them) and their capabilities are much broader and less clear.

I would be concerned that your proposed safety method would become very prone to Goodhearting.

It's worse than that. https://arxiv.org/abs/1604.06963

I don't think it's true that the safety of a thing depends on an explicit standard. There's no explicit standard for whether a grizzly bear is safe. There are only guidelines about how best to interact with them, and information about how grizzly bears typically act. I don't think this implies that it's incoherent to talk about the situations in which a grizzly bear is safe.

Similarly, if I make a simple html web site "without a clear indication about what the system can safely be used for... verification that it passed a relevant standard, and clear instruction that it cannot be used elsewhere", I don't think that's sufficient for it to be considered unsafe.

Sometimes a thing will reliably cause serious harm to people who interact with it. It seems to me that this is sufficient for it to be called unsafe. Sometimes a thing will reliably cause no harm, and that seems sufficient for it to be considered safe. Knowledge of whether a thing is safe or not is a different question, and there are edge cases where a thing might occasionally cause minor harm. But I think the requirement you lay out is too stringent.