AI safety without goal-directed behavior

I'm curious if you're more optimistic about non-goal-directed approaches to AI safety than goal-directed approaches, or if you're about equally optimistic (or rather equally pessimistic). The latter would still justify your conclusion that we ought to look into non-goal-directed approaches, but if that's the case I think it would be good to be explicit about it so as to not unintentionally give people false hope (ETA: since so far in this sequence you've mostly talked about the problems associated with goal-directed agents and not so much about problems associated with the alternatives). I think I'm about equally pessimistic, because while goal-directed agents have a bunch of safety problems, they also have a number of advantages that may be pretty hard to replicate in the alternative approaches.

1. We have an existing body of theory about goal-directed agents (which MIRI is working on refining and expanding) which plausibly makes it possible to one day reason rigorously about the kinds of goal-directed agents we might build and determine their safety properties. Paul and others working on his approach are (as I understand it) trying to invent a theory of corrigibility, but I don't know if such a thing even exists in platonic theory space. And if it did, we're starting from scratch so it might take a long time to reach parity with the theory of goal-directed agents.
2. Goal-directed agents give you economic efficiency "for free". Alternative approaches have to simultaneously solve efficiency and safety, and may end up approximating goal-directed agent anyway due to competitive pressures.
3. Goal-directed agents can more easily avoid a bunch of human safety problems that are inherited by alternative approaches which all roughly follow the human-in-the-loop paradigm. These include value drift (including vulnerability to corruption/manipulation), problems with cooperation/coordination, lack of transparency/interpretability, and general untrustworthiness of humans.

Rohin, I really like the distinction you draw between "build[ing] an AI system that could maximize an arbitrary function, and then [trying] to program in the utility function we care about" versus "build[ing] systems in such a way that these properties are inherent in the way that they reason." That was helpful.

However, it seems to me—and please correct me if I'm wrong!—that most or all CIRL papers are framing the problem in terms of understanding a generic goal-seeking system whose goal is "the human gets what they want". Then papers like The Off-Switch Game show that the goal of "the human gets what they want" leads to nice instrumental goals like not disabling off-switches. Do you agree?

So when I was reading CIRL papers, or reading Stuart Russell's new book, I did in fact keep thinking to myself "How do we make sure that the AI really has the goal of "The human gets what they want.", as opposed to a proxy to it that will diverge out-of-distribution?"

IDA / "act-based corrigibility" seems like more of an attempt to break out of the goal-seeking paradigm altogether, although I still haven't convinced myself that it succeeds.

If we're making a list of models for non-goal-directed AI (and we should!!), I would propose two more:

• Non-consequentialist oracle AI: An oracle with the property that the algorithm will not think through the consequences of its own outputs. You ask it a question, it digs through its world-model for a fixed number of computation steps and spits out its best-guess answer, but crucially it does not try to model the causal effects of that output. (Contrast with Eliezer's side-comment about an oracle here [LW · GW], which he of course assumes will be goal-directed, with the goal of "increase the correspondence between the user's belief about relevant consequences and reality".) A non-consequentialist oracle could never be deceptive or manipulative, because deception and manipulation require modeling the causal effects of outputs. I've speculated a bit about how to build something like this [LW · GW] but it's still definitely an open question.

• Interpretable-world-model as AI: Kinda related to the first. Imagine you take an AGI that deeply understands the world, you extract its world-model, and you have a way to browse it—like the world-model is somehow 100% super-easily interpretable. What causes Alzheimers? Well, you would go to the Alzheimers entry of the world-model, and you'll find a beautiful way of thinking about Alzheimers in terms of these other three concepts, which in turn refer to other concepts etc. What would happen if we started a political movement against squirrels? Well, through the world-model interface, we can throw that hypothetical scenario at other entities in the world-model (people, journalists, politicians) and see what the predicted effects are. My intuition here is: (1) It's nice to have a map when you're traveling, (2) It's nice to have wikipedia when you're learning, (3) it would be nice to have a crystal ball when you're planning ... Maybe there's some way to build a system that combines all those things and more, but is still fundamentally tool-ish? My impression is that something like this is at the core of the Kurzweil-ish vision of how brain-computer interfaces are going to solve the problem of AGI safety (see also waitbutwhy on Neuralink). (Needless to say, it's possible to try to implement this vision without brain-computer interfaces, and vice-versa.)

I usually think that logic-based reasoning systems are the canonical example of of an AI without goal-directed behaviour. They just try to prove or disprove a statement, given a database of atoms and relationships. (Usually they're restricted to statements that are decidable by construction so that is always possible).

You can also frame their behaviour as a utility function: U(time, state) = 1 if you have correctly decided the statement at t ≤ time, 0 otherwise. But your statement that

>It seems possible to build systems in such a way that these properties are inherent in the way that they reason, such that it’s not even coherent to ask what happens if we “get the utility function slightly wrong”.

very much applies. I'm fairly sure you can specify the behaviour of _anything_, including "dumb" things like trousers, screwdrivers, rocks and saucepans, as an utility function + perfect optimization, even though for most things this is a very unhelpful way of thinking. Or at least human artifacts. E.g. a screwdriver optimizes "transmit the rotational force that is applied to you", a rock optimizes "keep these molecules bound and respond to forces according to the laws of physics".

May be this article is related to the topic:

"A plurality of values" https://www.academia.edu/173502/A_plurality_of_values

"Abstract: Many maximizing normative theories are monistic in resting upon one core value. But  such theories generate highly counter-intuitive implications. This is especially clear in the case of hedonistic utilitarianism. But an analysis of why we find thoseimplications counter-intuitive implies that we ought to subscribe to a plurality of values. For example, the Repugnant Conclusion implies that we should value a highlevel of average happiness, while the Problem of the Ecstatic Psychopath implies that we should value either a large quantity of total happiness or a large number of worthwhile lives. The problems posed by pleasure-wizards, on the other hand, implythat we should include a non-utilitarian value: namely, equality. And only when suchvalues are kept in play simultaneously can the Repugnant Conclusion, the Problem of the Ecstatic Psychopath and the problems posed by pleasure-wizards all be avoided,thereby demonstrating the superiority of pluralist over monistic normative theories."

Interesting part start on page 10 after the quote: "Brian Barry argues in his early work that we could model trade-offs between principles such as equity and efficiency in a manner that parallels the way in which micro-economists employ indifference-curves to model how we might swap grapes for potatoe".