Strong men are socialist - how to use a study's own data to disprove it

post by Jacob Falkovich (Jacobian) · 2017-05-31T04:18:45.142Z · LW · GW · Legacy · 5 comments

This is a link post for https://putanumonit.com/2017/05/27/strong-men-are-socialist-reports-a-study-that-previously-reported-the-opposite/

Contents

5 comments

5 comments

Comments sorted by top scores.

comment by drethelin · 2017-05-31T06:20:33.655Z · LW(p) · GW(p)

Sarcastic and rambling and not in a fun way to read. Either get to the point faster or make your asides less about mocking your target and more interesting.

comment by [deleted] · 2017-06-01T16:36:26.198Z · LW(p) · GW(p)

Enjoyable read.

comment by Pfft · 2017-06-04T00:27:25.689Z · LW(p) · GW(p)

This is amazingly great (I laughed out loud at the "Biceps-controlled socialism" graph), but I feel it only works because the original study authors made the rookie mistake of publishing their data set. The only time I have wanted to try something similar (for the brain mosaic paper), I hoped it would be possible to extract the data from the diagram, but no, the jpg in the pdf is sufficiently low-resolution that it doesn't work.

Replies from: btrettel
↑ comment by btrettel · 2017-06-04T03:57:13.400Z · LW(p) · GW(p)

I hoped it would be possible to extract the data from the diagram, but no, the jpg in the pdf is sufficiently low-resolution that it doesn't work.

I have been compiling a lot of data for part of my PhD and this is a lot more common than I am comfortable with. Personally, as a reviewer I've decided to outright reject papers that don't allow one to extract the data. My preference would be requiring publishing the data straight up, but I can see an editor viewing this as unreasonable, or an author not knowing how to publish data.

With this being said, it's worth asking authors for raw data. My prior on receiving raw data from an author is low, particularly if the study is older. You have nothing to lose, however, and you will sometimes gain respect for helpful researchers. One professor I emailed for about 25 year old data searched his office thoroughly, found the data I wanted on some old floppy disks, got a floppy drive working, and emailed me the files. That was a not insignificant amount of their time, and I'm very appreciative for it.

Also, frequently theses and dissertations have tabulated data. Beware of typos in the tables. Always perform some sort of sanity checks on the data. Checking the tables against the figures is one approach.

Replies from: Pfft
↑ comment by Pfft · 2017-06-04T16:56:44.485Z · LW(p) · GW(p)

So in the case of this particular paper, some other researchers did ask for the raw data, and they got it and carried out exactly the analysis I was interested in knowing about. So I guess it's a happy ending, except I didn't get to write a tumblr post back when there was a lot of buzz in the media about it. :)