'X is not about Y' is not about psychology

post by abramdemski · 2017-11-11T02:22:26.887Z · LW · GW · 11 comments

Contents

11 comments

As Robin Hanson says:

Food isn’t about Nutrition
Clothes aren’t about Comfort
Bedrooms aren’t about Sleep
Marriage isn’t about Romance
Talk isn’t about Info
Laughter isn’t about Jokes
Charity isn’t about Helping
Church isn’t about God
Art isn’t about Insight
Medicine isn’t about Health
Consulting isn’t about Advice
School isn’t about Learning
Research isn’t about Progress
Politics isn’t about Policy

As I understand it, the prototypical form of Hansonian skepticism is to look at an X which is overtly about Y, and argue that the actual behavior of humans engaging in X is not consistent with caring about Y. Instead, the Hansonian cynic concludes, some unspoken motives must be at work; most often, status motives. He has a new book with Kevin Silmer which (by the sound of it) covers a lot of examples.

For him, the story of what's going on in cases like this has a lot to do with evolutionary psychology -- humans are adapted to play these games about what we want, even if we don't know it. Our brains keep the conscious justifications seperate from the hidden motives, so that we can explain ourselves in public while still going after private goals.

I'm not denying that this plays a role, but I think there's a more general phenomenon here which has nothing to do with humans or evolutionary psychology.

In economics, models based on an assumption of rational agents are fairly successful at making predictions, even though we know individual humans are far from rational. As long as the market contains inefficiencies which can be exploited, someone can jump in and exploit them. Not everyone needs to be rational, and no one person needs to be rational about everything, for the market to appear rational.

Similarly, the tails come apart whether or not X and Y have to do with humans. Someone might score high on calibration, but that doesn't make their strongest beliefs the most trustworthy -- indeed, they're the ones we ought to downgrade the most, even if we know nothing about human psychology and reason only from statistics. This alone is enough for us not to be too surprised at some of the "X is not about Y" examples -- we might expect explicit justification to be mostly good explanations for behavior, but by default, this shouldn't make us think that the correltion will be perfect. And if we optimize for justifiability, we expect the correlation to decrease.

Almost any problem has to be solved by coming up with proxies for progress. As Goodhart's Law notes, that these proxies will have errors which naturally come to be exploited, simply by virtue of individuals in the system following incentive gradients. "Food is not about nutrition" because evolution found an imperfect proxy for nutritive value, and the modern economy optimized food to cheat the biological system. Organizations fall into lipservice to lost purposes because they set up incentive structures which are good enough initially, but eventually get exploited. You can't usually call out such failures in a way the system will recognize, because the exploitation will happen in a way that's crafted to the blind spots -- and there's no need for the exploiters to be aware, consciously or subconsciously, of what they're doing. Any system will tend to optimize for seeming over being, because seeming is what's directly available to optimize. It's the principal-agent problem all the way down, from national elections to a person's internal motivation system. Call it wireheading. Call it Goodhart. It's all the same cluster of dysfunction.

Machine learning researchers understand this phenomen in their domain -- they call it overfitting, and fight it with regularization. In machine learning, you want predictive accuracy. But, you can't directly optimize for accuracy on the data you have available -- you'll get something which is very bad at predicting future data. The solution to this is regularization: you optimize a combination of predictive accuracy and a simplicity measure. This creates some "drag" on the optimization, so that it doesn't over-optimize predictive accuracy on the available data at the expense of future accuracy. This works extremely well, considering that it doesn't necessarily introduce any new information to correlate things with reality better (though, in the Bayesian case, the regulariser is a prior probability which may contain real information about which things are more likely). If we had a general theory of anti-Goodharting which worked as well as this, it would seem to have broad implications for group coordination and societal problems.

This pessimism shouldn't be pushed too hard; usually, you're better off coming up with some kind of proxy measurement for your goals, so that you can shift your strategies in the face of measured success and failure. Your taste in food isn't that bad a proxy for what's good for you. Profit isn't that bad a proxy for providing value. Scientific consensus isn't that bad a proxy for truth. And so on. But keep in mind: when an optimization process is based on a proxy, there may be systematic failures. X is not about Y.

11 comments

Comments sorted by top scores.