Sociosexual Orientation Inventory, or failing to perform basic sanity checks

would have to be explained by items 4 to 7, which have relatively low weights!

Item 4 is the killer. It is actually heavily weighted in the paper you link, though not in all versions of the SOI. You can score a 50 without ever having had sex or ever expecting to have sex, and up to 32 of those points can come from a response to 4. 4 is obviously not zero sum and need not average out.

There is very good reason to believe that men fantasize about sex more often than women (who buys most pornography?), and there is also reason to believe men are more likely to fantasize about persons other than their current partner. It's also possible that "once per day" is actually relatively infrequent given how often men think about sex. Put these together and it can explain a lot of the variance. The paper does not show breakdown by answers, at least not that I saw.

It's also worth noting they had some significant response-rate issues (RR < .5 for some demographics). If women who like casual sex, are promiscuous, or are prostitutes are less likely to respond than all other groups, that could explain the variance.

Also, there are other versions of the ROI that don't need to average out, as they lump large numbers together (i.e. "More than 8").

suffers from problems like that?

You never say what the problems are. So what if the subjects lie? Maybe "orientation" as measured by what people want to project is more useful than their actual behavior.

This reminds me of Shalizi's complaints about IQ (not his complaints about intergroup differences that Johnicholas linked). One difference is that for SOI there's a definite number (actual turnover of sex partners) that could be confused with it. But the name seems designed to discourage that confusion.

These are real problems, but it's not obvious that the "basic sanity checks" you suggest would lead to better measures.

I am reminded of a great paper, which isn't online:

Biddlecom, Ann E., and Laurie F. DeRose. “A Cross-National Study of Virgin Births.” Journal of Irreproducible Results 38, no. 3 (May/June 1993): 9-14.

I agree with the others who have said that there doesn't actually seem to be a problem here.

OK, so people lie. That's only a problem if the people who designed the test intended their measure to be based on actual number of partners, but I don't see what reason there is to think that. Someone who is more willing to admit to having many partners, or who thinks having many partners is something to aspire to, is most likely more "open" to the idea of sex-without-commitment than someone who thinks having many partners is something to be ashamed of.

Psychologists have noticed that there are sex differences on questions like 1-3, even though there cannot be sex differences in those behaviors, and have tried to explain that finding. See, for instance, this paper (pdf) by Baumeister, Catanese, & Vohs (2001) which argues that men have a stronger sex drive than women. They write (p. 250):

... Men actually report significantly more sex partners than women, across all studies (e.g., Janus & Janus, 1993; Laumann et al., 1994). Unfortunately this difference suffers from being logically impossible, insofar as heterosexual intercourse involves one man and one woman (so the mean tallies of partners should be equal). Several studies have sought to explain this recurrent finding, and the answers converge on motivated cognition: Some men, but fewer women, tend to rely on estimating the number of sex partners and hence round up, whereas women are more likely to rely on trying to enumerate all prior partners, which tends to lead to occasionally forgetting some partners and hence to producing an undercount (N. R. Brown & Sinclair, 1999; Wiederman, 1997).

...

In our view, the difference in the way people count sex partners is itself an indication that men want more than women.

They also note a study showing that men were more likely than women to classify oral or manual stimulation as sex.

Still, I cannot read papers about it without asking myself - why didn't they bother to perform this basic sanity check - which would detect huge number of outright lies in answers.

I expect they did. But there is little benefit to them is there in emphasising that particular aspect of their findings.

If people have biased expectations of the future amount of sex they'll have, male/female answers for question 2 could differ.

There's another problem with this scale -- which I think is an actual problem, unlike the one taw complained about. The name of the scale suggests it's meant to measure attitudes but 1 and 2 also measure opportunity. Someone might be very open in principle to sex without love or commitment, but find that no one (or no one they find sufficiently appealing) wants to have sex with them.

(Of course it is possible to buy sex for money, but there are reasons why someone might not want to do that that have nothing to do with "sociosexual openness".)

This is called 'face validity'. It's why people get upset at things like this, but not at neologisms like 'philanthrosexual'.

There is rather good debate on the topic of counting sexual partners and the reliability of survey responses here: http://www.nybooks.com/articles/1889

I'm afraid some of the material may be gated.

Volume 42, Number 9 · May 25, 1995 'Sex, Lies, and Social Science': An Exchange By John H. Gagnon, Edward O. Laumann, Robert T. Michael, Stuart Michaels, Richard Sennett, Reply by Richard C. Lewontin

In response to Sex, Lies, and Social Science* (April 20, 1995)

How many different partners do you foresee yourself having sex with during the next five years?

I thought you were going to say that the problem was that men would be more optimistic.