Hedge drift and advanced motte-and-bailey

post by Stefan_Schubert · 2016-05-01T14:45:08.023Z · score: 29 (27 votes) · LW · GW · Legacy · 8 comments

Motte and bailey is a technique by which one protects an interesting but hard-to-defend view by making it similar to a less interesting but more defensible position. Whenever the more interesting position - the bailey - is attacked - one retreats to the more defensible one - the motte -, but when the attackers are gone, one expands again to the bailey. 

In that case, one and the same person switches between two interpretations of the original claim. Here, I rather want to focus on situations where different people make different interpretations of the original claim. The originator of the claim adds a number of caveats and hedges to their claim, which makes it more defensible, but less striking and sometimes also less interesting.* When others refer to the same claim, the caveats and hedges gradually disappear, however, making it more and more motte-like.

A salient example of this is that scientific claims (particularly in messy fields like psychology and economics) often come with a number of caveats and hedges, which tend to get lost when re-told. This is especially so when media writes about these claims, but even other scientists often fail to properly transmit all the hedges and caveats that come with them.

Since this happens over and over again, people probably do expect their hedges to drift to some extent. Indeed, it would not surprise me if some people actually want hedge drift to occur. Such a strategy effectively amounts to a more effective, because less observable, version of the motte-and-bailey-strategy. Rather than switching back and forth between the motte and the bailey - something which is at least moderately observable, and also usually relies on some amount of vagueness, which is undesirable - you let others spread the bailey version of your claim, whilst you sit safe in the motte. This way, you get what you want - the spread of the bailey version - in a much safer way.

Even when people don't use this strategy intentionally, you could argue that they should expect hedge drift, and that omitting to take action against it is, if not ouright intellectually dishonest, then at least approaching that. This argument would rest on the consequentialist notion that if you have strong reasons to believe that some negative event will occur, and you could prevent it from happening by fairly simple means, then you have an obligation to do so. I certainly do think that scientists should do more to prevent their views from being garbled via hedge drift. 

Another way of expressing all this is by saying that when including hedging or caveats, scientists often seem to seek plausible deniability ("I included these hedges; it's not my fault if they were misinterpreted"). They don't actually try to prevent their claims from being misunderstood. 

What concrete steps could one then take to prevent hedge-drift? Here are some suggestions. I am sure there are many more.

  1. Many authors use eye-catching, hedge-free titles and/or abstracts, and then only include hedges in the paper itself. This is a recipe for hedge-drift and should be avoided.
  2. Make abundantly clear, preferably in the abstract, just how dependent the conclusions are on keys and assumptions. Say this not in a way that enables you to claim plausible deniability in case someone misinterprets you, but in a way that actually reduces the risk of hedge-drift as much as possible. 
  3. Explicitly caution against hedge drift, using that term or a similar one, in the abstract of the paper.

* Edited 2/5 2016. By hedges and caveats I mean terms like "somewhat" ("x reduces y somewhat"), "slightly", etc, as well as modelling assumptions without which the conclusions don't follow and qualifications regarding domains in which the thesis don't hold.

8 comments

Comments sorted by top scores.

comment by gjm · 2016-05-01T19:32:07.944Z · score: 12 (12 votes) · LW(p) · GW(p)

Asking scientists to keep their paper titles hedge-drift-resistant means (1) asking each individual scientist to do something that will reduce the visibility of their work relative to others', for the sake of a global benefit -- a class of policy that for obvious reasons doesn't have a great track record -- and (2) asking them to give their papers titles that are boring and wordy.

I agree that the world might be a better place if scientists consistently did this. But it doesn't seem very likely to happen.

(Also, here's what might happen if they almost consistently did this: the better, more conscientious scientists all write carefully hedged articles with carefully hedged titles, and journalists ignore all of them because they all sound like "Correlational analysis of OCEAN traits weakly suggest slight association between conscientiousness and Y-chromosome haplogroup O3". A few less careful scientists write lower-quality papers that, among other things, have titles like "The Chinese work harder: correlational analysis of OCEAN traits and genotype", and those are the ones that the journalists pick up. These are also the ones without the careful hedging in the actual analysis, without serious attempts to correct for multiple correlations, etc. So we end up with worse stuff in the press.)

comment by Stefan_Schubert · 2016-05-02T09:55:06.803Z · score: 0 (0 votes) · LW(p) · GW(p)

Good points. I agree that what you write within parentheses is a potential problem. Indeed, it is a problem for many kinds of far-reaching norms on altruistic behaviour compliance with which is hard to observe: they might handicap conscientious people relative to less conscientious people to such an extent that the norms do more harm than good.

I also agree that individualistic solutions to collective problems have a chequered record. The point of 1)-3) was rather to indicate how you potentially could reduce hedge drift, given that you want to do that. To get scientists and others to want to reduce hedge drift is probably a harder problem.

In conversation, Ben Levinstein suggested that it is partly the editors' role to frame articles in a way such that hedge drift doesn't occur. There is something to that, though it is of course also true that editors often have incentives to encourage hedge drift as well.

comment by TezlaKoil · 2016-05-02T11:07:56.399Z · score: 4 (3 votes) · LW(p) · GW(p)

Thanks for giving a name to this phenomenon.

Indeed, it would not surprise me if some people actually want hedge drift to occur. They don't actually try to prevent their claims from being misunderstood.

It's much worse. In my experience as an academic, most departments simply pre-hedge-drift their press releases. Science journalists don't - and are often not qualified to - read and comment on the actual papers, all they have to work with is the press release.

comment by Stefan_Schubert · 2019-06-14T22:57:52.639Z · score: 2 (1 votes) · LW(p) · GW(p)

Yes, a new paper confirms this.

The association between quality measures of medical university press releases and their corresponding news stories—Important information missing
comment by bbleeker · 2016-05-02T12:07:46.644Z · score: 1 (1 votes) · LW(p) · GW(p)

This made me think of this cartoon: http://www.phdcomics.com/comics.php?n=1174.

comment by Elo · 2016-05-02T03:04:55.445Z · score: 1 (1 votes) · LW(p) · GW(p)

This is funny because I was going to write a piece titled, "delicate use of hedging". Saying similar things. I will try to get this post written/published ASAP to match your post. My post was more about encouraging people to use hedging clearly when having discussions and communications; especially to represent being unsure about something you wish to communicate about

Also a link to describe a hedge would help this post.

comment by NancyLebovitz · 2016-05-01T17:36:43.111Z · score: 1 (1 votes) · LW(p) · GW(p)

On the other hand, people can only be obligated to achieve what is possible, and hedge loss might be inevitable.

There might be a long haul project of lowering the status of sloppy science journalism, but this is very long term.

I assume authors control the titles for journal articles. They don't for journalism.

comment by Val · 2016-05-02T07:31:17.678Z · score: 0 (0 votes) · LW(p) · GW(p)

It's not only in social sciences where this phenomena is common. The most striking examples I've seen were in medicine. An article is published, for example "supplement xyz slightly reduces a few of the side effects encountered during radiotherapy used in cancer treatment", which is then published in the media and on social networks as "What the medical industry doesn't want you to know: supplement xyz instantly cures all forms of cancer!". And often there is a link to the original publication, but people still believe it and forward it. And what's even more sad, probably many people then buy that supplement and don't seek medical help, believing that it alone will help.