nostalgebraist: Recursive Goodhart's Law

There are two related, but distinct problems under the heading of Goodhart, both caused by the hard-to-deny fact that any practical metric is only a proxy/estimate/correlate of the actual true goal.

1) anti-inductive behavior. When a metric gets well-known as a control input, mis-aligned agents can spoof their behavior to take advantage.

2) divergence of legible metric from true desire. Typically this increases over time, or over the experienced range of the metric.

I think you're right that there is an underlying assumption that such a true goal exists. I don't think there's any reason to believe that it's an understandable/legible function for human brains or technology. It could be polynomial or not, and it could have millions or billions of terms. In any actual human (and possibly in any embodied agent), it's only partly specified, and even that partial specification isn't fully accessible to introspection.

“it’s better to behave in ways that aren’t as subject to Goodhart’s Law,”

Specifics matter. It's better to behave in ways that give better outcomes, but it's not obvious at all what those ways are. Even ways that are known to be affected by goodhart's law have SOME reason to believe they're beneficial - goodhart isn't (necessarily) a reversal of sign, only a loss of correlation with actual desires.

it can be very hard to explain why these things are true

Again, specifics matter. "Goodhart exists, and here's how it might apply to your proposed metrics" has been an _easy_ discussion every time I've had it. I literally have never had a serious objection to the concept. What's hard is figuring out checksum metrics or triggers to re-evaluate the control inputs. Defining the range and context inside of which a given measurement is "good enough" takes work, but is generally achievable (in the real world; perhaps not for theoretical AI work).

I really like this.

And I want to advise this person to sort of “chill out” in a way closely analogous to the way the hypothetical AI should “chill out” – a little more formally, I want to advise them to use a strategy that works better under model mis-specification, a strategy that doesn’t try to “jump to the end” and do the best thing without a continuous traversal of the intervening territory.

I like the analogy of "traversal of the intervening territory", even though, like the author, I don't know what it formally means.

Unlike the author, I do have some models [LW · GW] of what it means to lack a utility function, but optimize anyway. Within such a model, I would say: it's perfectly fine and possible to come up with a function which approximately represents your preferences, but holding on to such a function even after your preferences have updated away from it leads to a Goodhart-like risk.

More generally, it's not the case that literally everything is well-understood as optimization of some function. There are lots of broadly intelligent processes that aren't best-understood as optimization.

I am fond of describing Buddhism as un-goodharting yourself all the way down.

In brief, you are a Ship of Theseus at sea (or a Ship of Neurath as Quine coined after Otto von Neurath who first made the analogy), navigating by lighthouses that you should not steer directly towards (that's not how lighthouses work!) and avoiding whirlpools (attractors in the space of goal architectures that lock you in to a specific destination).

I agree and glad this is getting upvotes, but for what it's worth I made exactly the same point a year ago [LW · GW] and several people were resistant to the core idea, so this is probably not an easily won insight.

In this situation Goodhart is basically open-loop optimization. An EE analogy would be a high gain op amp with no feedback circuit. The result is predictable: you end up optimized out of the linear mode and into saturation.

You can't explicitly optimize for something you don't know. And you don't know what you really want. You might think you do, but, as usual, beware what you wish for. I don't know if an AI can form a reasonable terminal goal to optimize, but humans surely cannot. Given that some 90% of our brain/mind is not available to introspection, all we have to go by is the vague feeling of "this feels right" or "this is fishy but I cannot put my finger on why". That's why cautiously iterating with periodic feedback is so essential, and open-loop optimization is bound to get you to all the wrong places.

This post reminds me of an insight from one of my uni professors.

Early on at university, I was very frustrated with that the skills that were taught to us did not seem to be immediately applicable to the real world. That frustration was strong enough to snuff out most of the interest I had for studying genuinely (that is, to truly understand and internalize the concepts taught to us). Still, studying was expensive, dropping out was not an option, and I had to pass exams, which is why very early on I started, in what seemed to me to be a classic instance of Goodhart, to game the system - test banks were videly circulated among students, and for the classes with no test banks, there were past exams, which you could go through, trace out some kind of pattern for which topics and kinds of problems the prof puts on exams, and focus only on stuying those. I didn't know it was called "Goodhart" back then, but the significance of this was not lost on me - I felt that by pivoting away from learning subjects and towards learning to pass exams in subjects, I was intellectually cheating. Sure, I was not hiding crib sheets in my sleeves or going to the restroom to look something up on my phone, but it was still gaming the system.

Later on, when I got rather friendly with one of my profs, and extremely worn down by pressures from my probability calculus course, I admitted to him that this was what I was doing, that I felt guilty, and didn't feel able to pass any other way and felt like a fake. He said something to the effect of "Do you think we don't know this? Most students study this way, and that's fine. The characteristic of a well-structured exam isn't that it does not allow cheating, it's that it only allows cheating that is intelligent enough that a successful cheater would have been able to pass fairly."

What he said was essentially a refutation of Goodhart's Law by a sufficiently high-quality proxy. I think this might be relevant to the case you're dealing with here as well. Your "true" global optimum probably is a proxy, but if it's a well-chosen one, it need not be vulnerable to Goodhart.

The author seems to be skipping a step in their argument. I thought Goodhart's Law was about how it's hard to specify a measurable target which exactly matches your true goal, not that true goals don't exist.

For example, if I wanted to donate to a COVID-19 charity, I might pick one with the measurable goal of reducing the official case numbers.. and they could spend all of their money bribing people to not report cases or make testing harder. Or if they're an AI, they could hit this goal perfectly by killing all humans. But just because this goal (and probably all easily measurable goals) are Goodhartable doesn't mean all possible goals are. The thing I actually want is still well defined (I want actual COVID-19 cases to decrease and I want the method to pass a filter defined by my brain), it's just that the real fundamental thing I want is impossible to measure.

Yup. Humans have a sort of useful insanity, where they can expect things to be bad not based on explicitly evaluating the consequences, but off of a model or heurstic about what to expect from different strategies. And then we somehow only apply this reasoning selectively, where it seems appropriate according to even more heuristics.