Oh wow, I had never before thought of modern people over-consuming sugar, as being an application of Goodhart's Law. But it is. That's brilliant.

I very much agree with the ideas presented in this post; for people who are interested in finding out more, I very much recommend the book Don't Shoot the Dog, and maybe also The Power of Habit. That said, those books are pretty much written from a behaviorist perspective, so they don't go very much into the way that mental and abstract concepts become associated with value, as in your doctor example.

A couple of minor suggestions on how to improve on your post further: 1) I think that the ∃ in your Claim 2 is meant to be interpreted as "for a given effect and size, there exists a sufficiently small delay such that the desired result is produced", but I wouldn't have understood that notation if I hadn't had math as a minor in my degree, and probably not all readers have 2) It might be good to quickly explain clicker training in a couple of sentences to people who haven't heard about it before.

I claim that what's going on is that the monkey's brain, separate from the monkey/the monkey's S2/any sapient or strategic awareness that the monkey has, is conditioning the monkey.

I think this claim is confusing at best and false at worst. The shifting dopamine response is well-recognized in the neuroscience literature, and explained by Sutton and Barto's Temporal-Difference model.

First, it should be emphasized that midbrain dopamine does not signal reward. The monkey can experience a ton of pleasure without any dopamine reaction. Midbrain dopamine signals reward prediction error, the difference between actual and expected reward. It signals a kind of surprise.

Now the TD model is quite Bayesian. Whereas the Rescorla-Wagner model -- the previously dominant theory of reinforcement -- viewed the prediction error as the difference between actual and expected current reward; the TD model instead views it as the difference between all actual and expected future rewards (properly discounted).

So when the dopamine signal shifts, the monkey is just conserving expected evidence. Initially, it is positively surprised to receive juice. But eventually, it learns that the screen perfectly predicts the juice, and so it is the appearence of the screen itself that becomes the positive surprise. On a classical model of reinforcement, these events are different, as OP seems to recognize. But on the TD model, these are just instances of the very same kind of conditioning event.

For futher reference, see the section "Two Dopamine Responses and One Theory" of Glimcher PW (2011) Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis.

OP seems to recognize all this, but these observations seems to be complemented with somewhat unfounded interpretations and elaborations.

[Epistemic status: confident OP will be confusing to those without RL background knowledge, but still non-negligible credence that OP is explaning exactly the above but from a different perspective]

Nitpick: as far as I can tell, you're describing discounting in general, not hyperbolic discounting. Hyperbolic discounting refers specifically to the surprising result that human discounting isn't exponential, like we expected it to be based on economics research. http://www.behaviorlab.org/Papers/Hyperbolic.pdf

I agree with the overall claim you're making, though I think you're making it out to be a stronger force than I expect it is. In general, I think a good prior is "what your brain is doing by default is pretty damn close to right, and you have to understand it pretty well before you can recognize the actual flaws in its behavior." I suspect you've mistaken where in the chain something is going wrong - I think it's something related to social permission to make mistakes, social permission to succeed/fail at goals, etc. In other words, you get a punishment from your internalized model of other people when you fail to have lost weight.

This seems largely correct to me, although I think hyperbolic discounting of rewards/punishments over time may be less pronounced in human conditioning as compared to animals being conditioned by humans. Humans can think "I'm now rewarding myself for Action A I took earlier" or "I'm being punished for Action B" which can seems, at least in my experience, to decrease the effect of the temporal distance whereas animals seem less able to conceptualize the connection over time. Because of this difference, I think the temporal difference of reward/punishment is less important in people for conditioning as long as the individual is mentally associating the stimulus with the action, although it is still significant.

Also what's the name of the paper for the monkeys and juice study? I'd like to look at it because the result did surprise me.

Apparently "~30min" is a floor, not a ceiling.

For the specific case of weighing yourself, could you create a scale that only gives the positive reward, not the negative one? Like, it only tells you your weight if it's lower than yesterday, or better yet if the trend in your weight is downward over the past week? Maybe it displays a cheerful message and plays a soothing sound when you weigh yourself, and it emails you later at random if you've been losing weight.

An important thing to notice about Goodhart's law is that we roll three different phenomena together. This isn't entirely bad because the three phenomena are very similar, but sometimes it helps to think about them differently.

Goodhart Level 1: Following the sugar/fruit example, imagine there are a bunch of different fruits with different levels of nutrients and sugar, and the nutrient and sugar levels are very correlated. It is still the case that if you optimize for the most sugar, you will get reliably less nutrients than if you optimize for the most nutrients. (This is just the cost of using a proxy. I barely even want to call this Goodhart's law.)

Goodhart Level 2: It is possible that foods with lots of sugar are usually very correlated with foods with lots of nutrients, but there is one type of fruit that is pure sugar with no nutrients. If this fruit occurred in nature, but only very rarely, it would not mess with the statistical correlation between sugar and nutrients very much. Thus when an agent optimizes very hard for sugar, they end up with no nutrients, but if they optimized only slightly, they would have found a normal fruit with lots of sugar and nutrients.

Note that this is more nuanced than just saying we didn't have pure sugar in the ancestral environment and we do now, so the actions that were good in the ancestral environment are a bad proxy for the actions that are good now. (Maybe just using a bad proxy should be called Goodhart Level 0) The point is that the reason that the environment is different now is that we optimized for sugar. We pushed to the section of possible worlds with lots of sugar using our sugar optimization, and the correlation mostly only existed in the worlds with a moderate amount of sugar.

Goodhart Level 3: Say we live in a world with only a fixed amount of nutrients, and someone else wants a larger share of the nutrients. If you are using the proxy of sugar, and other people know this, and adversary might invent candy, and then trade you their candy for some of your fruit. Another agent had a goal that was in conflict with your true goal, but not in conflict with your proxy, so they exploited your proxy and created options that satisfied your proxy but not your true goal.

I say more about this (in math) here: https://agentfoundations.org/item?id=1621

Many instances of people Goodharting themselves falls in Level 2 (If I don't step on the scale, I optimize out of the worlds where the scale number is correlated with my weight). However, I claim that some instances might be at Level 3. In particular rationalization. Maybe part of me wants to save the world and uses the ability to produce justifications for why an action saves the world as a proxy for what saves the world. Then, a different part of me wants to goof off and produces a justification for why goofing off is actually the most world-saving action.

"So my brain is sitting there with mirror-twin goals of maximize exposure to low scale numbers and minimize exposure to high scale numbers, "

I think my answer would be don't give it those goals. Give it rule following goals and model updating goals. Create rules from the models. If following the rules don't actually get you slimmer, update the model and update the rules.