A content analysis of the SQ-R questionnaire and a proposal for testing EQ-SQ theory

post by tailcalled · 2023-08-09T13:51:02.036Z · LW · GW · 2 comments

Contents

  Potential SQ-R Factors
    Technical Interests
    DIY Interests
    Nature Interests
    Orderliness
    Political/Business Interests
    Creative Interests
    Social Attention
    Not Immediately Classifiable
  Potential SQ-R Short Form
    Technical Interests
    DIY Interests
    Nature Interests
    Orderliness
    Political/Business Interests
    Creative Interests
    Social attention
    Not Immediately Classifiable
  Scoring and interpretation
  A qualification on the General Factor point
  Predictions and Theory Testing
  An extension to the theory testing
    EQ-Cognitive Empathy
    EQ-Emotional Empathy
    EQ-Social Skills
    ASQ-Social Skills
    ASQ-Attention Switching
    ASQ-Communication
    ASQ-Imagination
    ASQ-Attention to detail
  Basic Plans for Testing
None
2 comments

If you are more interested in this topic, I have created a Discord server titled Rationalist Psychometrics to discuss these sorts of things. Thank you to Justis Mills for proofreading and feedback.

I've recently been complaining about the EQ-SQ theory of autism [LW · GW] which asserts that autism is caused by having an extremely male brain, and I've felt like it could probably be disproven with a bit of work. Briefly, my opinion is that the measures conflate multiple different things (e.g. technical interests vs nature interests), and I propose that one can test this by seeing whether the items that correlate with autism are the same as the items that correlate with sex. But in order for it to be tested, someone has to actually perform that work.

And part of the trouble here is, the Systemizing Quotient-Revised (SQ-R) is very long, so it would be very expensive to collect comprehensive data on it. I tried contacting some rationalists with reasonably far reach (Scott Alexander and Aella) to see if they would be interested in sharing a comprehensive autism measure to their audience to get me data for free, but so far I have not received any responses yet.

So I need to find some way to make it quicker and cheaper. One way to make it cheaper would be to construct a "short form", which measures the same traits in a shorter way by only using a subset of the items.

There have already been constructed short forms of the SQ-R, but the only ones I have seen have been constructed with a very basic empirical approach of finding items that are highly correlated with the total scores of the scale. This is a problem for investigating measurement biases, as it can obscure the measurement bias and hide invalidity of the construct.[1]

The principled way to solve this would be to perform a factor analysis, searching for groups of correlated items in an SQ-R dataset, and then collecting the few top items on each factor. Unfortunately, I do not have access to an SQ-R dataset; if I did, this problem would be gone.

But, usually the results of a factor analysis can be predicted fairly well by looking at similarity in the content of the items. Furthermore, I have a bunch of experience with personality and sex difference psychometrics, so I can make educated guesses about what sorts of content does or does not matter. To avoid bias, I would like to present my content analysis before I collect the data, so people can comment on it and say what they think.

Potential SQ-R Factors

Based on reading the SQ-R, I felt like something like the following a likely to be factors in the SQ-R.

Technical Interests

1. I find it very easy to use train timetables, even if this involves several connections.
6. I find it difficult to read and understand maps.
9. If I were buying a car, I would want to obtain specific information about its engine capacity.
10. I find it difficult to learn how to programme video recorders.
16. When I look at a building, I am curious about the precise way it was constructed.
17. I am not interested in understanding how wireless communication works (e.g. mobile phones).
18. When travelling by train, I often wonder exactly how the rail networks are coordinated.
32. I am fascinated by how machines work.
45. I rarely read articles or webpages about new technology.
46. I can easily visualise how the motorways in my region link up.
52. If I were buying a camera, I would not look carefully into the quality of the lens.
53. If I were buying a computer, I would want to know exact details about its hard drive capacity and processor speed.
60. If I were buying a stereo, I would want to know about its precise technical features.
66. In maths, I am intrigued by the rules and patterns governing numbers.
70. When I'm in a plane, I do not think about the aerodynamics.

DIY Interests

15. I find it difficult to understand instruction manuals for putting appliances together
19. I enjoy looking through catalogues of products to see the details of each product and how it compares to others.
33. When I look at a piece of furniture, I do not notice the details of how it was constructed.
43. If there was a problem with the electrical wiring in my home, I'd be able to fix it myself.
58. I am not very meticulous when I carry out D.I.Y or home improvements.

Nature Interests

7. When I look at a mountain, I think about how precisely it was formed.
29. When I look at an animal, I like to know the precise species it belongs to.
35. I do not tend to watch science documentaries on television or read articles about science and nature.
41. I am interested in knowing the path a river takes from its source to the sea.
50. When I am walking in the country, I am curious about how the various kinds of trees differ.
63. I do not care to know the names of the plants I see.
64. When I hear the weather forecast, I am not very interested in the meteorological patterns.

Orderliness

2. I like music or book shops because they are clearly organised.
4. When I read something, I always notice whether it is grammatically correct.
14. If I had a collection (e.g. CDs, coins, stamps), it would be highly organised.
20. Whenever I run out of something at home, I always add it to a shopping list.
28. I do not find it distressing if people who live with me upset my routines.
44. My clothes are not carefully organised into different types in my wardrobe.
55. When I get to the checkout at a supermarket, I pack different categories of goods into separate bags.
56. I do not follow any particular system when I'm cleaning at home.
61. I tend to keep things that other people might throw away, in case they might be useful for something in the future.
62. I avoid situations which I cannot control.
65. It does not bother me if things in the house are not in their proper place.
72. When I have a lot of shopping to do, I like to plan which shops I am going to visit and in what order.

Political/Business Interests

3. I would not enjoy organising events e.g. fundraising evenings, fetes, conferences.
8. I am not interested in the details of exchange rates, interest rates, stocks and shares.
13. I like to know how committees are structured in terms of who the different committee members represent or what their functions are.
24. When I learn about historical events, I do not focus on exact dates.
30. I can remember large amounts of information about a topic that interests me e.g. flags of the world, airline logos.
34. I know very little about the different stages of the legislation process in my country.
40. I am not interested in how the government is organised into different ministries and departments.
47. When an election is being held, I am not interested in the results for each constituency.
48. I do not particularly enjoy learning about facts and figures in history.
57. I do not enjoy in-depth political discussions.
69. When I read the newspaper, I am drawn to tables of information, such as football league scores or stock market indices.
25. I find it easy to grasp exactly how odds work in betting.

Creative Interests

37. When I look at a painting, I do not usually think about the technique involved in making it.
74. When I listen to a piece of music, I always notice the way it's structured.
75. I could generate a list of my favourite 10 songs from memory, including the title and the artist's name who performed each song.
42. I have a large collection e.g. of books, CDs, videos etc.
68. I could list my favourite 10 books, recalling titles and authors' names from memory.
73. When I cook, I do not think about exactly how different methods and ingredients contribute to the final product.

Social Attention

5. I find myself categorising people into types (in my own mind).
23. I am interested in my family tree and in understanding how everyone is related to each other in the family.
36. If someone stops to ask me the way, I'd be able to give directions to any part of my home town.
49. I do not tend to remember people's birthdays (in terms of which day and month this falls).
38. I prefer social interactions that are structured around a clear activity, e.g. a hobby.
67. I find it difficult to learn my way around a new city.

Not Immediately Classifiable

11. When I like something I like to collect a lot of different examples of that type of object, so I can see how they differ from each other.
12. When I learn a language, I become intrigued by its grammatical rules.
22. When I was young, I did not enjoy collecting sets of things e.g. stickers, football cards, etc.
26. I do not enjoy games that involve a high degree of strategy (e.g. chess, Risk, Games Workshop).
27. When I learn about a new category, I like to go into detail to understand the small differences between different members of that category.

Potential SQ-R Short Form

To create a version of the scale which captures as much of the same information as possible that the SQ-R does, one should select items from all of the factors listed above. Furthermore, factor analysis works best when one has 3 (or ideally more) items per factor[2], so one should ideally pick multiple items per factor.

Technical Interests

53. If I were buying a computer, I would want to know exact details about its hard drive capacity and processor speed.
9. If I were buying a car, I would want to obtain specific information about its engine capacity.
32. I am fascinated by how machines work.
17. I am not interested in understanding how wireless communication works (e.g. mobile phones).

DIY Interests

15. I find it difficult to understand instruction manuals for putting appliances together
33. When I look at a piece of furniture, I do not notice the details of how it was constructed.
58. I am not very meticulous when I carry out D.I.Y or home improvements.

Nature Interests

7. When I look at a mountain, I think about how precisely it was formed.
35. I do not tend to watch science documentaries on television or read articles about science and nature.
50. When I am walking in the country, I am curious about how the various kinds of trees differ.

Orderliness

4. When I read something, I always notice whether it is grammatically correct.
14. If I had a collection (e.g. CDs, coins, stamps), it would be highly organised.
20. Whenever I run out of something at home, I always add it to a shopping list.
44. My clothes are not carefully organised into different types in my wardrobe.

Political/Business Interests

3. I would not enjoy organising events e.g. fundraising evenings, fetes, conferences.
8. I am not interested in the details of exchange rates, interest rates, stocks and shares.
13. I like to know how committees are structured in terms of who the different committee members represent or what their functions are.
24. When I learn about historical events, I do not focus on exact dates.
47. When an election is being held, I am not interested in the results for each constituency.

Creative Interests

37. When I look at a painting, I do not usually think about the technique involved in making it.
74. When I listen to a piece of music, I always notice the way it's structured.
42. I have a large collection e.g. of books, CDs, videos etc.

Social attention

5. I find myself categorising people into types (in my own mind).
23. I am interested in my family tree and in understanding how everyone is related to each other in the family.
38. I prefer social interactions that are structured around a clear activity, e.g. a hobby.
67. I find it difficult to learn my way around a new city.

Not Immediately Classifiable

One argument one could make is that items which are not immediately classifiable should be included so that we can collect data on them and perform a factor analysis, and thereby classify them. Alternatively one could drop them because they are inconvenient. I have no strong opinion on this except I want to get the project done as cheaply as possible, but if you have an opinion then please share it.

Scoring and interpretation

The original paper on the SQ-R provides details about how to score the test. This is somewhat complicated by the fact that my short form doesn't contain all of the original items, so the scores will not be commensurate with the original scale.

SBC has also constructed a short form of the SQ-R. It contains some items that my Short Form is missing, namely:

32. When I learn about a new category I like to go into detail to understand the small differences between different members of that category.
16. When I'm in a plane, I do not think about the aerodynamics.
27. I am interested in knowing the path a river takes from its source to the sea.
9. When travelling by train, I often wonder exactly how the rail networks are coordinated.
30. When I hear the weather forecast, I am not very interested in the meteorological patterns.
33. I enjoy looking through catalogues of products to see the details of each product and how it compares to others.
12. When I learn a language, I become intrigued by its grammatical rules.

If these items are added to my short form, then I can use them to make my total scores commensurate with that of the SQ-R.

Overall, that would yield a scale with 33 items, a significant shortening over the original 75 items, while still capturing much of the nuances and being suitable for factor analysis in ways that the 10-item version might miss.

In the original SQ paper, SBC frames Systemizing as a General Factor [LW · GW] which drives you to understand systems across a wide variety of contexts. Thus from the perspective of the Systemizing-Empathizing model, the main thing that should matter is your overall score, which should reflect your general tendency to systemize. Deviations on specific items or domains would constitute noise/measurement error, which it is hoped we can average away by having a varied set of items.

A qualification on the General Factor point

In the original SQ paper, SBC describes Systemizing as a general factor, but he also adds important nuances/qualifications to the idea of a general factor of Systemizing:

... Systemizing is the drive to analyse the variables in a system, to derive the underlying rules that govern the behaviour of a system. Systemizing also refers to the drive to construct systems. Systemizing allows you to predict the behaviour of a system, and to control it. A growing body of evidence suggests that, on average, males spontaneously systemize to a greater degree than do females.

...

Initially, we had planned to devise the SQ so that it would tap into each of the domain-specific systems described above. However, this proved to be problematical because individuals who were well rounded but not necessarily good systemizers would end up scoring highly, whereas those who were highly systematic but only interested in one domain would receive a low score. Thus, we decided, instead, to use examples from everyday life in which systemizing could be used to varying degrees. The assumption is that a strong systemizer would be drawn to use their systemizing skills across the range of examples more often than a poor systemizer, and would consequently score higher on the SQ.

As I read it, he is saying that you can't measure Systemizing as a general factor across tons of obscure subjects like mixology and glassblowing and so on, because most people have zero activity in those subjects, and therefore would score 0 on Systemizing in corresponding subject-specific questions. Instead, this would just end up measuring whether you had broad, exploratory interests. However, he is saying that the specific subjects he included in his scales are sufficiently present in everyday life for most people that they should be highly reflective of Systemizing.

Predictions and Theory Testing

The key claim of the Systemizing theory of autism and sex differences is that autistic people and men are higher in the General Factor of Systemizing than allistic people and women. In the past, this has repeatedly been superficially tested by showing how the Systemizing Quotient correlates with autism and maleness. However, the General Factor model has implications that permit stronger tests, and also makes tests much easier.

Because the General Factor of Systemizing is hypothesized to influence Systemizing behavior across main everyday domains, claims about male-female differences or autistic-allistic differences would by default imply that the group differences are evenly spread out over all of the items, rather than being limited to specific domains.

To give an example, imagine if the sex differences on Technical Interests were much bigger than the sex differences on Orderliness, Nature Interests and Creative Interests. This could not be compatible with a single general factor explaining the sex differences across all of these domains; instead it would be compatible with a multicausal model, with the simplest one being that there is some other factor that causes sex differences in Technical Interests, and that there isn't much sex difference in General Systemizing. If true, this would mean that the SQ-R is biased, making men score more Systemizing than they truly are.

To give another example, imagine if the sex differences lie on Technical Interests and Political/Business Interests, while the autistic-allistic differences lie on Nature Interests, Orderliness and Creative Interests. In that case, even though the SQ-R correlates with both autism and sex, this doesn't actually reflect any special relationship between autism and sex, because autism and sex would each have unique reasons for being connected to SQ-R scores.

Thus, detailed item-level or factor-level data provides relatively strong ways of testing the validity of the EQ-SQ theory, simply by looking at how broad or localized the group differences are.

An extension to the theory testing

In addition to looking at whether the scales' group differences are broad or localized to specific items, one can also look at whether the scales' correlations with other scales are broad or localized to specific items. For instance the EQ has three factors which can be measured reasonably well with the following items:

EQ-Cognitive Empathy

EQ-Emotional Empathy

EQ-Social Skills

Also, due to my experience with personality psychology, I imagine it would be beneficial to add items such as "I care about other's feelings", "I take charge", and "I worry about things", to better pin down their relationships to classical personality traits (compassion, assertiveness/charisma, anxiety).

Similarly, the 50-item Autism Spectrum Quotient has been said to have various factor structures. I've created the following 23-item version, which combines items that are highly indicative of autism according to this study with items that are highly informative according to the factor structure found in this study and items that are present in the 10-item version SBC created.

ASQ-Social Skills

ASQ-Attention Switching

ASQ-Communication

ASQ-Imagination

ASQ-Attention to detail

Paying careful attention to how these correlate can also test the EQ-SQ theory harder. For instance, if the ASQ-Attention to detail factor has a special correlation with the SQ-R's Orderliness factor, above and beyond its correlation to General Systemizing, then that complicates models where the relationship between the ASQ and the SQ-R are supposed to be solely due to General Systemizing causing autism, and not due to correlated measurement error. One can use a technique called structural equation modelling to sort of untangle these biases in various ways.

Basic Plans for Testing

In total, I've now discussed around 69 items. I tend to assume that each item takes 7 seconds to answer, and that respondents require 9 GBP/hour to participate. If we assume that we get 100 allistic men, 100 allistic women, 100 autistic men, and 100 autistic women, then we could estimate correlations to an error of around , and group differences to an error of around , which seems adequate for this type of analysis. This would add up to a cost of 483 GBP or 615 USD, which of course is not a completely trivial amount of money, but is also fairly affordable for engineers working for Big Tech.[4]

I'm probably gonna collect this data at some point, but first I would like to hear if anyone has any feedback or concerns about the approach.

  1. ^

    To explain: Imagine you had a "maleness test" which had three subtests: do math, lift heavy things, and build bridges. Plausibly, "do math" and "lift heavy things" are independent factors, and "build bridges" involves both "do math" (to work out the engineering of how the bridge should be designed) and "lift heavy things" (to actually assemble the bridge). In this case, if you constructed a short form of the test by looking for the subtest that correlated the most with overall test scores, then that subtest could be "build bridges" because it loads on both the factors that the test conflates. But if we hypothesize that the sex difference is mostly on "lift heavy things", then this obscures the fact that the test conflates things by favoring a subtest that has the same pattern of conflation as the overall test.

  2. ^

    Because the model is underidentified with 2 items, just-identified with 3 items, and overidentified with 4+ items. A factor model with  indicators has  parameters and  constraints, so you can count your degrees of freedom here to see how that goes.

  3. ^

    There's a case to be made that one could improve these scales by using more varied questions. I should probably do a more in-depth investigation into the ASQ before proceeding.

  4. ^

    That said, for various reasons I am somewhat money-constrained. If I wasn't, I would probably up this to 220+ items by adding the entire SQ-R, ASQ, and EQ to the survey, as well as adding the SPQ and maybe also various other scales to it. (People-things? Political orientation? Utilitarianism? Woo? etc.) Also I would be tempted to upgrade the sample sizes to 200 for each group.

2 comments

Comments sorted by top scores.

comment by tailcalled · 2023-08-15T08:20:25.867Z · LW(p) · GW(p)

I have decided I should explicitly contact Simon Baron-Cohen for comment on this approach before proceeding. We will see what happens.

comment by tailcalled · 2023-08-09T15:55:43.820Z · LW(p) · GW(p)

Accidentally broke the link to the rationalist psychometrics server, but it should be fixed now.