An Anchoring Experiment: Results

While this is interesting this seems to have enough confounding variables that I'd really like to see this duplicated in a more controlled setting.

I don't know of any research that has found a wrong direction effect from anchoring, so this may very well be original research on Less Wrong. Nice job.

I think that it could be consistent with existing models of anchoring. There are two theories for why anchoring effects occur, insufficient adjustment and selective accessibility.

Insufficient adjustment: People get their estimate by anchoring and adjustment, as you described. They start at the anchor, realize it's too high (for example), and adjust downward until they reach a plausible value. But they tend to stop too soon, since there are a range of plausible values and they don't go all the way to the most plausible value.

Selective accessibility: Answering the higher/lower question makes you bring to mind information that is relevant to that question. This is a biased set of information, since you bring to mind (for example) information related to the population being over 20 million when considering that possibility. Then when you go to make an estimate in response to the second question, that biased set information remains accessible in your mind and contaminates your judgment.

The general consensus among researchers is that both theories are correct - both processes occur. But there is ongoing disagreement among researchers about precisely when each of the two processes takes place. A few years ago the thought was that there are some cases where anchoring & adjustment takes place and others where selective accessibility happens - with questions like the ones you'd asked, anchoring effects would be due to selective accessibility rather than anchoring & adjustment (as discussed here and here on LW). Since then, a 2010 paper by Simmons, LeBoeuf & Nelson has argued that the typical case is for both selective accessibility and anchoring & adjustment to be operating, so either or both could be involved.

I think that your effect here is consistent with the selective accessibility account. If you're considering whether the population is more or less than 20 million, you might bring to mind reasons why the population could be more than 20 million and reasons why the population could be less than 20 million. That's a biased set of all the information that you have, even for the initial question, which will contain more information consistent with the wrong direction than you'd get in a representative sample of your information. That explains why you get closer to a 50/50 split in people's answers compared with what you'd get if you just looked at people's estimates without the anchoring question. Part of the inspiration for the selective accessibility model was previous research on confirmation bias showing that considering a hypothesis makes it seem more likely, and this wrong direction effect seems consistent with that research.

With anchoring & adjustment, I think you could say that this is a case where the anchor value is already within the range of plausible values. It's not clear what the model says about those cases, but perhaps it could account for the wrong direction effect. If you see the anchor value and think that it's a plausible answer, but you're forced to say "higher" or "lower", then maybe the choice seems kind of arbitrary and ends up closer to a coin flip.

Simmons, J.P., LeBoeuf, R.A., & Nelson, L.D. (2010). The effect of accuracy motivation on anchoring and adjustment: Do people adjust from provided anchors? Journal of Personality and Social Psychology, 99 (6), 917-932. pdf

For me, my answers would have been incredibly varied simply because I had no idea, whatsoever.

The number of respondents in both control groups don't add up. It looks like you had 17 answers for the first question's control group, and 18 for the second.

As someone who participated, I must have misread the directions because I voted higher or lower than anchor on both questions.

Also, a link to the original experiment post might be nice.

Sorry if I'm just being stupid, but why does this go against the hypothesis of subjects "start[ing] from the anchor and continu[ing] in the direction of his/her unbiased estimate, but [not going] far enough." While the percentages are vastly different, the favored option is the same between biased and unbiased groups in both cases.
Also, and sorry if this seems like nitpicking, but in the example we talked about in the Mental Metadata I imagine the subject would have a much stronger feeling about their expected values than in these cases. That aside, interesting experiment.