Two Neglected Problems in Human-AI Safety

I agree with all of this but I don't think it addresses my central point/question. (I'm not sure if you were trying to, or just making a more tangential comment.) To rephrase, it seems to me that ‘ML safety problems in humans’ is a natural/obvious framing that makes clear that alignment to human users/operators is likely far from sufficient to ensure the safety of human-AI systems, that in some ways corrigibility is actually opposed to safety, and that there are likely technical angles of attack on these problems. It seems surprising that someone like me had to point out this framing to people who are intimately familiar with ML safety problems, and also surprising that they largely respond with silence.

Thanks, this is helpful to me.

I’ve also had discussions at CHAI about whether we should expect humans to have adversarial examples (usually vision, not values).

I noticed that in your recent FLI interview, you discussed applying the idea of distributional shifts to human values. Did you write about that or talk to anyone about it before I made my posts?

I can’t speak to all of the cases, but in the example you point out, you’re asking about a paper whose main author is not on this forum (or at least, I’ve never seen a contribution from him, though he could have a username I don’t know). People are busy, it’s hard to read everything, let alone respond to everything.

Oh, I didn't realize that, and thought the paper was more of a team effort. However as far as I can tell there hasn't been a lot of discussion about the paper online, and the comments I wrote under the AF post might be the only substantial public online comments on the paper, so "let alone respond to everything" doesn't seem to make much sense here.

I certainly expect people’s quick, intuitive value judgments to be horribly wrong anywhere outside of current environments; it’s not obvious to me that’s true for probabilities given by explicit reasoning.

It seems plausible that given enough time and opportunities to discuss with other friendly humans, explicit reasoning can eventually converge upon correct judgments, but explicit reasoning can certainly be wrong very often in the short or even medium run, and even eventual convergence might happen only for a small fraction of all humans who are especially good at explicit reasoning. I think there are also likely analogies to adversarial examples for explicit human reasoning, in the form of arguments that are extremely persuasive but wrong.

But I guess humans do have a safety mechanism in that system 1 and system 2 can cross-check each other and make us feel confused when they don't agree. But that doesn't always work since these systems can be wrong in the same direction (motivated cognition is a thing that happens pretty often) or one system can be so confident that it overrides the other. (Also it's not clear what the safe thing to do is (in terms of decision making) when we are confused about our values.)

This safety mechanism may work well enough to prevent exploitation of adversarial examples by other humans a lot of the time, but seems unlikely to hold up under heavier optimization power. (You could perhaps consider things like Nazism, conspiracy theories, and cults to be examples of successful exploitation by other humans.)

(I wonder if people have tried to apply the idea of heterogeneous systems cross-checking each other to adversarial examples in ML. Have you seen any literature on this?)

I think that to the extent you are hoping to scale up safety in parallel with capabilities, which feels a lot more tractable than solving the full problem in one go, this is not a problem you have to deal with yet, and you can outsource it to the future.

I guess that makes sense, but even then there should at least be an acknowledgement that the problem exists and needs to be solved in the future?

Beyond the fact that humans have inputs on which they behave “badly” (from the perspective of our endorsed idealizations), what is the content of the analogy?

The update I made was from "humans probably have the equivalent of software bugs, i.e., bad behavior when dealing with rare edge cases" to "humans probably only behave sensibly in a small, hard to define region in the space of inputs, with a lot of bad behavior all around that region". In other words the analogy seems to call for a much greater level of distrust in the safety of humans, and higher estimate of how difficult it would be to solve or avoid this problem.

I don’t think there is too much disagreement about that basic claim

I haven't seen any explicit disagreement, but have seen AI safety approaches that seem to implicitly assume that humans are safe, and silence when I point out this analogy/claim to the people behind those approaches. (Besides the public example I linked to, I think you saw a private discussion between me and another AI safety researcher where this happened. And to be clear, I'm definitely not including you personally in this group.)

I remain supportive of work in this direction and will probably write about it in more detail some point, but don’t think there is much ambiguity about what I should work on.

I'm happy to see the first part of this statement, but the second part is a bit puzzling. Can you clarify what kind of people you think should work on this class of problems, and why you personally are not in that group? (Without that explanation, it seems to imply that people like you shouldn't work on this class of problems, which might cover almost everyone who is potentially qualified to work on it. I would also be happy if you just stopped at the comma...)

Thanks for your reply!

our value functions probably only "make sense" in a small region of possibility space, and just starts behaving randomly outside of that.

Okay, that helps me understand what you're talking about a bit better. It sounds like the concept of a partial function, and in the ML realm like the notorious brittleness that makes systems incapable of generalizing or extrapolating outside of a limited training set. I understand why you're approaching this from the adversarial angle though, because I suppose you're concerned about the AI just bringing about some state that's outside the domain of definition which just happens to yield a high "random" score.

It doesn't seem right to treat that random behavior as someone's "real values" and try to maximize that.

Upon first reading, I kind of agreed, so I definitely understand this intuition. "Random" behavior certainly doesn't sound great, and "arbitrary" or "undefined" isn't much better. But upon further reflection I'm not so sure.

First of all, what does it mean for a value system to behave randomly/arbitrarily, and is it ever not arbitrary? Arbitrary to me means that there is no reason for something, which sounds a lot like a terminal value to me. If you morally justify having a terminal value X because of reason Y, then X is instrumental to the real terminal value Y.

Secondly, I question whether my value system really is like some kind of partial function that yields random outcomes outside the domain of definition. If you asked me for a (relative) value judgment about two situations that are completely alien to me, then I would imagine being indifferent about their ordering: not ordering them randomly. It's possible that I could be persuaded to order one over the other, but then that seems more about changing my beliefs/knowledge and understanding (the is domain) than it is about changing my values (the ought domain). This may happen in less alien situations too: should we invest in education or healthcare? I don't know, but that's primarily because I can't predict the actual outcomes in terms of things I care about.

Finally, even if a value system was to order two alien situations randomly, how can we say it's wrong? Clearly it wouldn't be wrong according to / compared with that value system, right? And how else are you going to judge whether something is right or wrong, better or worse?

I feel like these questions lead deeply into philosophical territory that I'm not particularly familiar with, but I hope it's useful (rather than nitpicky) to ask these things, because if the intuitive that "random is wrong" is itself wrong, then perhaps there's no actual problem we need to pay extra attention to. I also think that some of my questions here can be answered by pointing out that someone's values may be inconsistent / conflicting. But then that seems to be the problem that needs to be solved.

---

I would like to acknowledge the rest of your comment without responding to it in-depth. I think I have personally spent relatively little time thinking about the complexities of multipolar scenarios (which is likely in part because I haven't stumbled upon as much reading material about it, which may reflect on the AI safety community), so I don't have much to add on this. My previous comment was aimed almost exclusively at your first point (in my mind), because the issue of what value systems are like and what an ASI that's aligned with your might (unintentionally) do wrong seems somewhat separate from the issue of defending against competing ASIs doing bad things to you or others.

I acknowledge that having simpler and constant values may be a competitive advantage, and that it may be difficult to transfer the nuances of when you think it's okay to manipulate/corrupt someone's values into an ASI. I'm less concerned about other people not thinking of the corruption problem (since their ASIs are presumably smarter), and if they simply don't care (and their aligned ASIs don't either), then this seems like a classic case of AI that's misaligned with your values. Unless you want to turn this hypothetical multipolar scenario into a singleton with your ASI at the top, it seems inevitable that some things are going to happen that you don't like.

I also acknowledge that your ASI may in some sense behave suboptimally if it's overly conservative or cautious. If a choice must be made between alien situations, then it may certainly seem prudent to defer judgment until more information can be gathered, but this is again a knowledge issue rather than a values issue. The values system should then help determine a trade-off between the present uncertainty about the alternatives and the utility of spending more time to gather information (presumably getting outcompeted while you do nothing ranks as "bad" according to most value systems). This can certainly go wrong, but again that seems like more of a knowledge issue (although I acknowledge some value systems may have a competitive advantage over others;