The Loudest Alarm Is Probably False

The LW team is encouraging authors to review their own posts, so:

In retrospect, I think this post set out to do a small thing, and did it well. This isn't a grand concept or a vast inferential distance, it's just a reframe that I think is valuable for many people to try for themselves.

I still bring up this concept quite a lot when I'm trying to help people through their psychological troubles, and when I'm reflecting on my own.

I don't know whether the post belongs in the Best of 2018, but I'm proud of it.

For a data point, after having read the first paragraph I looked for examples and counterexamples. I found 1 example where a person strongly worries they are X while actually they’re anti-X (just like orthonormal describes), 2 examples where a person strongly worries they’re X and they’re indeed X, and 3 examples where they worry they’re X but actually they’re normal, i.e. between X and anti-X.

This was simple and probably helpful to think about, thanks!

(There's now some metaworry about the "What if I'm responding inappropriately to alarms!" alarm, but well, what can you do)

This behavior is also what you would expect to happen if different people had different preferences. The quiet friend might worry about being too loud, because they would actually prefer to be even quieter. You could say that they're projecting their own preferences on you, which is an error, but a slightly different one. And I admit that this doesn't fit all of your examples.

Another issue. Why is there "loudest" in the title? Why would this only happen with loud alarms? Surely, minor alarms can be warped through the same mechanisms, right?

I'd be quite curious about more concrete examples of systems where there is lots of pressure in *the wrong direction*, due to broken alarms. (Be they minds, organisations, or something else.) The OP hints at it with the consulting example, as does habryka in his nomination [LW(p) · GW(p)].

I strongly expect there to be interesting ones, but I have neither observed any nor spent much time looking.

This is anecdotally true in my practice, as well. Though the explanation seems simple enough, I would not frame it as alarms: if you are overly sensitive ("high-gain" in the control theory terms) to a specific stimulus, you are bound to overreact to that stimulus to try to match the others in perceived intensity. This is a simple negative feedback loop. Works almost universally.

However, humans are a very stuck control system by the time they are adults (unlearning is orders of magnitude is more difficult than learning), and deliberately changing the forward and feedback gain is very difficult.

I think this post is a clear explanation of an important point in system dynamics, connected to personal psychology (and thus individual rationality). It connects to other important concepts, like the tails coming apart [LW · GW] and Goodhart, while being distinct (in a way that I think helps clarify them all).

I promoted this to featured for a high usefulness-to-text ratio and generally clear writing.

I like this post's brevity, its usefulness, and the nice call-to-action at the end.

Moved to frontpage.

I mostly second Vaniver's nomination. I've also found this post really useful when thinking about LessWrong as an organization, and how my own preferences might often be actively pushing things in the wrong direction. 

I've been familiar with this pattern for a while, but I didn't explicitly connect it with the statistical explanation. My analysis was, rather, that at some point you learn "I need to work hard to get X right" -- at this point X is actually an important problem for you. So, you get into the habit of worrying about X, finding ways to be better at X, noticing details relevant to X... At some point, you cross the threshhold where X is not so important to work on, and maybe even negative to push any further in that direction. But, you don't automatically notice this point. By default, you keep on thinking X is a problem for you. Moreover, you've developed habits of thinking associated with pushing further in the direction of X; probably the feedback cues you're looking at are making you goodhart on X, and you haven't learned to pick up on the cues that would help you notice you've gone too far, because those would not have been helpful earlier.

So, you keep pushing further and further on X.

Therefore, if your life feels like a constant struggle to be nice, or clean, or punctual, or truthful, etc, it is possible that you're actually already one of the most nice/clean/punctual/truthful/etc people there is and you ought to stop worrying about it and maybe even adjust your habits a bit in the other direction.

Now the next big question is, what to count as reliable reassurance...

Hmm. My biggest alarm is that I don't have enough friends and am not social enough. But this is accurate; I have not one single intellectual friend of the sort I want, and people ignore my writing. I think the reason I don't act to rectify the situation is that I don't know how.

Okay, but what about other alarms, ones I don't notice so much? Well, I have an intense desire to be right (and not just appear to be right in the eyes of anyone else), and be knowledgeable... and so I frequent LW. I spend an unusual amount of time fretting over the accuracy and fairness of things I've posted earlier and often edit them days, weeks, or years afterward to provide more nuance and accuracy. I do this even at the risk of sounding boring and producing less writing, even though I think one needs a lot of writing to gain a following; might this interfere with my social life? Could be. Another alarm says I took a wrong turn to end up working on oil & gas software. I donate to clean energy efforts as a form of "offsetting", but this has the disturbing opportunity cost of reducing my donations to other efforts. That alarm seems reasonable and I'm open to different opportunities.

But, maybe this post has exactly the problem you linked to: offering good advice to those who least need it.