[QUESTION]: Academic social science and machine learning

(6) Another possibility (related to (2) and (4)) is that academic social scientists are primarily in the business of sharing information with other academics in their field, so they tend to rely on tools that are already standard within their field. A paper that uses complicated statistics which their colleagues don't understand is going to have less impact than a paper that makes the same point using the field's standard tools. It may also have trouble making it through peer review, if the peers don't have the technical knowledge to evaluate it.

So the incentives aren't there for an academic to stay on the cutting edge in adopting new complicated techniques. People who are in the business of building things to get results have a stronger incentive to adopt complicated new methods which offer any technical advantage.

You should be asking why "statistics" and "machine learning" are different fields. It is a good question!

edit: To clarify, stats is a "service field" for a lot of empirical fields, so lots of them use stats methods and not ML methods. More comp. sci. aligned areas also use more ML e.g. computational bio. There's been a lot of cross fertilization lately, and stats and ML are converging, but the fact that there is "department level division" is supremely weird.

Neural networks are non-linear regression.

Arguably we often can't usefully interpret statistical models unless they correspond to causal ones! One of the historical differences between ML and stats is that the latter was always concerned about experiments and interpretability, and thus about causal matters, whereas the former more about prediction and fancy algorithms.

It is very weird to me ML is so little interested in causality.

In my understanding of academia, people can be very resistant to integrating ideas from "unrelated" fields. The statistical tools that any one person uses are probably more determined by the status quo than anything else. I vote for options (2) and (4) as the most likely.

Every point you made (0)-(5) is correct!

(0) There are some social scientists, especially in political science, who are focused on applying machine learning and text mining methods to political texts. This is a big movement and it's under the heading "text as data". Most publications use fairly simple methods, basically calibrated regressions, but a lot of thought went into choosing those and some of the people publishing are mathematically sophisticated.

Example: http://www.justingrimmer.org/

Another prominent example comes in Social Networks, where people from the CS and physics world work on the social side, and some social scientists use the methodology too.

Example: http://cs.stanford.edu/people/jure/

At the Santa Fe institute people from all kinds of disciplines do all kind of stuff, but an overall theme is methods drawing from math and physics applied to social sciences. This include networks, statistical physics, and game theory.

Not exactly social science, but Jennifer Dunne applies network analysis to food webs: http://www.santafe.edu/about/people/profile/Jennifer%20A.%20Dunne

I am certain that cutting edge mathematics and ML are applied in pockets of econometrics too. Finance is often in economics departments and ML has thoroughly invaded that, but I admit that's a stretch.

(1) Social science academics have only recently gained access to large datasets. Especially in survey-based fields like sociology and experimental psychology, small-data-oriented methods are definitely the focus. Large datasets include medical datasets, to the extent that they have access; various massive text repositories including academic paper databases and online datasets; and a very few surveys that have the size and depth to support fancier analyses.

This applies less to probit and more to clustering, bayes nets, decision trees, etc.

(2) The culture is definitely conservative. I've talked to many people interested in the more advanced methods and they have to fight harder to get published; but the tide is changing.

(3) Absolutely. It's very hard to figure out what coefficients represent when data is ambiguous and many factors are highly correlated (as they are in social science) and when the model is very possibly misspecified. Clusterings with "high score" from most methods can be completely spurious and it take advanced statistical knowledge to identify this. ML is good for prediction and classification, but this is very rarely the goal of social scientists (though one can imagine how it could be). SVMs and decision trees do a poor job of extracting causal relationships with any certainty.

(4) Again, the culture is conservative and many don't have these training. A good number know their way around R though, and newer ones often come in with quite a bit of stats/CS knowledge. The amount of statistical knowledge in the social sciences is growing fast.

(5) Yes; this is especially true of something like neural networks.

Giving that we are on Lesswrong you missed a core one: Academic social science uses a bunch of frequentist statistics that perform well if your goal is to prove that your thesis has "statistical significance" but that aren't useful for learning what's true. Machine learning algorithms don't give you p values.

You really should have mentioned here one of your Facebook responses that maybe the data generating processes seen in social science problems don't look like (the output of generative versions of) ML algorithms. What's the point of using a ML method that scales well computationally if looking at more data doesn't bring you to the truth (consistency guarantees can go away if the truth is outside the support of your model class) or has terrible bang for the buck (even if you keep consistency, you may take an efficiency hit)?

Also, think about how well these methods work over the entire research process. Looking at probit modeling, the first thing that pops out about it is how those light normal tails suggest that it is sensitive to outliers. If you gave a statistician a big, messy-looking data set on an unfamiliar subject, this would probably push them to use something like logistic regression with its reliance on a heavier-tailed distribution and better expected robustness. But if you're the social scientist who assembled the data set, you may be sure that you've dealt with any data collection, data entry, measurement, etc. errors and you may be deeply familiar with each observation. At this stage, outliers are not unknowable random noise that gets in the way of the signal but may themselves be the signal, as they have an disproportionate effect on the learned model. At the least, they are where additional scrutiny should be focused, as long as the entity doing the analysis can provide that scrutiny.