Learning human preferences: black-box, white-box, and structured white-box access

post by Stuart_Armstrong · 2020-08-24T11:42:34.734Z · LW · GW · 9 comments


  Knowing an agent
  Levels of access
    and knowledge
    white boxes for a single black box
    structures and tags for one white-box
  Normative assumptions, tags, and structural assumptions
    trivial structural assumptions
    semantics or "thick" concepts

This post is inspired by system identification; however, I'm not an expert in that domain, so any corrections or inspirations on that front are welcome.

I want to thank Rebecca Gorman for her idea on using system identification, and her conversations developing the concept.

Knowing an agent

This is an agent:

Fig. 1

We want to know about its internal mechanisms, its software. But there are several things we could mean by that.


First of all, we might be interested in knowing its input-output behaviour. I've called this its policy in previous posts [LW · GW]; a full map that will allow us to predict its output in any circumstances:

Fig. 2

I'll call this black-box knowledge of the agent's internals.


We might be interested in knowing more about what's actually going on in the agent's algorithm, not just the outputs. I'll call this white-box knowledge; we would be interested in something like this (along with a detailed understanding of the internals of the various modules):

Fig. 3

Structured white-box

And, finally, we might we interested in knowing what the internal modules actually do, or actually mean. This is the semantics of the algorithm, resulting in something like this:

Fig. 4

The "beliefs", "preferences", and "action selectors" are tags that explain what these modules are doing. The tags are part of the structure of the algorithm, which includes the arrows and setup.

If we know those, I'd call it structured white-box knowledge.

Levels of access

We can have different levels of access to the agent. For example, we might be able to run it inside any environment, but not pry it open; hence we know its full input-output behaviour. This would give us (full) black-box access to the agent (partial black box access would be knowing some of its behaviour, but not in all situations).

Or we might be able to follow its internal structure. This gives us white-box access to the agent. Hence we know its algorithm.

Or, finally, we might have a full tagged and structured diagram of the whole agent. This gives us structured white-box access to the agent (the term is my own).

Things can more complicated, of course. We could have only access to parts of the agent/structure/tags. Or we could have a mix of different types of access - grey-box seems to be the term for something between black-box and white-box.

Humans seem to have a mixture of black-box and structured white-box access to each other - we can observe each other's behaviour, and we have our internal theory of mind that provides information like "if someone freezes up on a public speaking stage, they're probably filled with fear".

Access and knowledge

Complete access at one level gives complete knowledge at that level. So, if you have complete black-box access to the agent, you have complete black-box knowledge: you could, at least in principle, compute every input-output map just by running the agent.

So the interesting theoretical challenges are those that involve having access at one level and trying to infer a higher level, or having partial access at one or multiple levels and trying to infer full knowledge.

Multiple white boxes for a single black box

Black-box and white-box identification are have been studied somewhat extensively in system identification. One fact remains true: there are multiple white-box interpretations of the same black-box access.

We can have the "angels pushing particles to resemble general relativity [LW · GW]" situations. We can add useless epicycles, that do nothing, to the model of the white-box; this gives us a more complicated white-box with identical black-box behaviour. Or you could have the matrix mechanics vs wave mechanics situation in quantum mechanics, where two very different formulations were shown to be equivalent.

There are multiple ways of choosing among equivalent white-box models. In system identification, the criteria seems to be "go with what works": the model is to be identified for a specific purpose (for example, to enable control of a system) and that purpose gives criteria that will select the right kind of model. For example, linear regression will work in many rough-and-ready circumstances, while it would be stupid to use it for calibrating sensitive particle detectors when much better models are available. Different problems have different trade-offs.

Another approach is the so called "grey-box" approach, where a class of models is selected in advance, and this class is updated with the black-box data. Here the investigator is making "modelling assumptions" that cut down on the possible space of white-box models to consider.

Finally, in this community and among some philosophers, algorithmic simplicity is seen as good and principled way of deciding between equivalent white-box models.

Multiple structures and tags for one white-box

A similar issue happens again at a higher level: there are multiple ways of assigning tags to the same white-box system. Take the model in figure 4, and erase all the tags (hence giving us figure 3). Now reassign those tags; there are multiple ways we could tag the modules, and still have the same structure as figure 4:

Fig. 5

We might object, at this point, insisting that tags like "beliefs" and "preferences" be assigned to modules for a reason, not just because the structure is correct. But having a good reason to assign those tags is precisely the challenge.

We'll look more into that issue in future sections, but here I should point out that if we consider the tags as purely syntactic, then we can assign any tag to anything:

Fig. 6

What's "Tuna"? Whatever we want it to be.

And since we haven't defined the modules or said anything about their size and roles, we can decompose the interior of the modules and assign tag in completely different ways:

Fig. 7

Normative assumptions, tags, and structural assumptions

We need to do better than that. Paper "Occam’s razor is insufficient to infer the preferences of irrational agents" talked about "normative assumptions", assumptions about the values (or the biases) of the agent.

In this more general setting, I'll refer to them as "structural assumptions", as they can refer to beliefs, or other features of the internal structure and tags of the agent.

Almost trivial structural assumptions

These structural assumptions can be almost trivial; for example, saying "beliefs nad preferences update from knowledge, and update the action selector", is enough to rule out figures 6 and 7. This is equivalent with starting with figure 4, erasing the tags, and wanting to reassign tags to the algorithm while ensuring the graph is isomorphic to figure 4. Hence we have a "desired graph" that we want to fit our algorithm into.

What the Occam's razor paper shows is that we can't get good results from "desired graph + simplicity assumptions". This is unlike the black-box to white-box transition, where simplicity assumptions are very effective on their own.

Figure 5 demonstrated that above: the beliefs and preference modules can be tagged as each other, and we can still get the same desired graph. Even worse, since we still haven't specified anything about the size of these modules, the following tag assignment is also possible. Here, the belief and preference "module" have been reduced to mere conduits, that pass on the information to the action selector, that has expanded to gobble up all of the rest of the agent.

Fig. 8

Note that this decomposition is simpler than a "reasonable" version of figure 4, since the boundaries between the three modules don't need to be specified. Hence algorithmic simplicity will tend to select these degenerate structures more often. Note this is almost exactly the "indifferent planner" of the Occam's razor paper, one of the three simple degenerate structures. The other two - the greedy and anti-greedy planners - are situations where the "Preferences" module has expanded to full size, with the action selector reduced to a small appendage.

Adding semantics or "thick" concepts

To avoid those problems, we need to flesh out the concepts of "beliefs", "preferences[1]", and so on. The more structural assumptions we put on these concepts, the more we can avoid degenerate structured white-box solutions[2].

So we want something closer to our understanding of preferences and beliefs. For example, preferences are supposed to change much more slowly than beliefs. So the impact of observations on the preference module - in an information-theoretic sense, maybe - would be much lower than on the beliefs modules, or at least much slower. Adding that as a structural assumption cuts down on the number of possible structured white-box solutions.

And it we are dealing with humans, trying to figure out their preference - which is my grand project at this time [LW · GW] - then we can add a lot of other structural assumptions. "Situation X is one that updates preferences"; "this behaviour shows a bias"; "sudden updates in preferences are accompanied by large personal crises"; "red faces and shouting denotes anger", etc...

Basically any judgement we can make about human preferences can be used, if added explicitly, to restrict the space of possible structured white-box solutions. But these need to be added in explicitly at some level, not just deduced from observations (ie supervised, not unsupervised learning), since observations can only get you as far as white-box knowledge.

Note the similarity with semantically thick concepts and with my own post on getting semantics empirically [LW · GW]. Basically, we want an understanding of "preferences" that is so rich that only something that is clearly a "preference" can fit the model.

In the optimistic scenario, a few such structural assumptions are enough to enable an algorithm to quickly grasp human theory of mind and quickly sort our brain into plausible modules, and hence isolate our preferences. In the pessimistic scenario, theory of mind, preferences, beliefs, and biases are all so twisted together that even extensive examples are not enough to decompose them. See more in this post [LW · GW].

  1. We might object to the arrow from observations to "preferences": preferences are not supposed to change, at least for ideal agents. But many agents are far from ideal (including humans); we don't want the whole method to fail because there was a stray bit of code or neuron going in one direction, or because two modules reused the same code or the same memory space. ↩︎

  2. Note that I don't give a rigid distinction [LW · GW] between syntax and semantics/meaning/"ground truth". As we accumulate more and more syntactical restrictions, the number of plausible semantic structures plunges. ↩︎


Comments sorted by top scores.

comment by steve2152 · 2020-08-24T12:09:35.330Z · LW(p) · GW(p)

I'm not so sure about the "labeled white box" framing. It presupposes that the thing we care about (e.g. preferences) is part of the model. An alternative possibility is that the model has parameters a,b,c,d,... and there's a function f with

preferences = f(a,b,c,d,...),

but the function f is not part of the algorithm, it's only implemented by us onlookers. Right?

comment by Stuart_Armstrong · 2020-08-25T15:03:30.016Z · LW(p) · GW(p)

but the function f is not part of the algorithm, it's only implemented by us onlookers. Right?

Then isn't that just a model at another level, a (labelled) model in the heads of the onlookers?

comment by G Gordon Worley III (gworley) · 2020-08-25T18:54:12.466Z · LW(p) · GW(p)

Any model is going to be in the head of some onlooker. This is the tough part about the white box approach: it's always an inference about what's "really" going on. Of course, this is true even of the boundaries of black boxes, so it's a fully general problem. And I think that suggests it's not a problem except insofar as we have normal problems setting up correspondence between map and territory.

comment by steve2152 · 2020-08-25T20:31:32.289Z · LW(p) · GW(p)

My understanding of the OP was that there is a robot, and the robot has source code, and "black box" means we don't see the source code but get an impenetrable binary and can do tests of what its input-output behavior is, and "white box" means we get the source code and run it step-by-step in debugging mode but the names of variables, functions, modules, etc. are replaced by random strings. We can still see the structure of the code, like "module A calls module B". And "labeled white box" means we get the source code along with well-chosen names of variables, functions, etc.

Then my question was: what if none of the variables, functions, etc. corresponds to "preferences"? What if "preferences" is a way that we try to interpret the robot, but not a natural subsystem or abstraction or function or anything else that would be useful for the robot's programmer?

But now this conversation is suggesting that I'm not quite understanding it right. "Black box" is what I thought, but "white box" is any source code that produces the same input-output behavior—not necessarily the robot's actual source code—and that includes source code that does extra pointless calculations internally. And then my question doesn't really make sense, because whatever "preferences" is, I can come up a white-box model wherein "preferences" is calculated and then immediately deleted, such that it's not part of the input-output behavior.

Something like that?

comment by Stuart_Armstrong · 2020-08-26T09:35:00.915Z · LW(p) · GW(p)

My understanding of the OP was that there is a robot [...]

That understanding is correct.

Then my question was: what if none of the variables, functions, etc. corresponds to "preferences"? What if "preferences" is a way that we try to interpret the robot, but not a natural subsystem or abstraction or function or anything else that would be useful for the robot's programmer?

I agree that preferences is a way we try to interpret the robot (and how we humans try to interpret each other). The programmer themselves could label the variables; but its also possible that another labelling would be clearer or more useful for our purposes. It might be a "natural" abstraction, once we've put some effort into defining what preferences "naturally" are.

but "white box" is any source code that produces the same input-output behavior

What that section is saying is that there are multiple white boxes that produce the same black box behaviour (hence we cannot read the white box simply from the black box).

comment by John_Maxwell (John_Maxwell_IV) · 2020-08-26T04:06:18.798Z · LW(p) · GW(p)

Note that this decomposition is simpler than a "reasonable" version of figure 4, since the boundaries between the three modules don't need to be specified.

Consider two versions of the same program. One makes use of a bunch of copy/pasted code. The other makes use of a nice set of re-usable abstractions. The second program will be shorter/simpler.

Boundaries between modules don't cost very much, and modularization is super helpful for simplifying things.

comment by Stuart_Armstrong · 2020-08-26T09:28:11.984Z · LW(p) · GW(p)

modularization is super helpful for simplifying things.

The best modularization for simplification will not likely correspond to the best modularization for distinguishing preferences from other parts of the agent's algorithm (that's the "Occam's razor" result).

comment by John_Maxwell (John_Maxwell_IV) · 2020-08-26T11:02:04.690Z · LW(p) · GW(p)

Let's say I'm trying to describe a hockey game. Modularizing the preferences from other aspects of the team algorithm makes it much easier to describe what happens at the start of the second period, when the two teams switch sides.

The fact that humans find an abstraction useful is evidence that an AI will as well. The notion that agents have preferences helps us predict how people will change their plans for achieving their goals when they receive new information. Same for an AI.

comment by Stuart_Armstrong · 2020-08-26T11:54:44.529Z · LW(p) · GW(p)

Humans have a theory of mind, that makes certain types of modularizations easier. That doesn't mean that the same modularization is simple for an agent that doesn't share that theory of mind.

Then again, it might be. This is worth digging into empirically. See my post on the optimistic and pessimistic scenarios [LW · GW]; in the optimistic scenario, preferences, human theory of mind, and all the other elements, are easy to deduce (there's an informal equivalence result; if one of those is easy to deduce, all the others are).

So we need to figure out if we're in the optimistic or the pessimistic scenario.