The robust beauty of improper linear models

So that's the real role of the expert here

I work in the data science industry - as a programmer, not a data scientist or statistician. From my general understanding of the field what you're describing is a broadly accepted assumption. But I might be misled by the fact that the company I work for bases its product on this assumption, so I'm not sure if you're just describing this thing from another angle or if there's a different point that I'm missing here or if, in fact, many people spend too much effort trying to hand-tune models.

The data scientists I work with make predictive models in two stages. The first one is to invent (or choose) "features", which include natural variables from the original dataset, functions acting on one or more variables, or supplementary datasets that they think are relevant. The data scientist applies their understanding of statistics as well as domain knowledge to tell the computer which things to look for and which are clearly false positives to be ignored. And the second stage is to build the actual models using mostly standard algorithms like Random Forest or XGBoost or whatnot, where the data scientist might tweak arguments but the underlying algorithm is generally given and doesn't allow for as much user choice.

A common toy example is the Titanic dataset. This is a list of passengers on the Titanic, with variables like age, name, ticket class, etc.. The task is to build a model that predicts which ones survived when the ship sank. A data scientist would mostly work on feature engineering, e.g. introducing a variable that deduces passenger's sex from their name, and focus less on model tuning, e.g. determining the exact weight that should be given to the feature in the model (women and children had much higher rates of survival).

In a more serious example, a data scientist might work on figuring out which generic datasets are relevant at all. Suppose you're trying to predict where to best open a new Starbucks branch. Should you look at the locations of competing coffee shops? Noise from nearby construction? Public transit stops or parking lots? Nearby tourist attractions or campuses or who knows what else? You can't really afford to look at everything, it would both take too long (and maybe cost too much) and risk false positives. A good domain expert is the one who generates the best hypotheses. But to actually test those hypotheses, you use standard algorithms to build predictive models, and if a simple linear model works, that's a good thing - it shows your chosen features were really powerful predictors.

Relevant term is judgmental bootstrapping in the forecasting literature if anyone wants to dive deeper. It is extremely practically relevant for many circumstances such as hiring, where adhoc linear models outperformed veteran hiring managers.

I don't think this is why improper linear models work. If you have a large number of variables, most of which are irrelevant in the sense of being uncorrelated with the outcome, then the irrelevant variables will be randomly assigned to +1 or -1 weights and will on average cancel out, leaving the signal from the relevant variables who do not cancel each other out.

So even without an implicit prior from an expert relevance selection effect or any explicit prior enforcing sparsity, you would still get good performance from improper linear models. (And IIRC, when you use something like ridge regression or Laplacian priors, the typical result, especially in high-dimensional settings like genomics or biology, most of the variables do drop out or get set to zero, so even in these 'enriched' datasets, most of the variables are irrelevant. What's sauce for the goose is sauce for the gander.)

Adding in more irrelevant variables does change things quantitatively by lowering power due to increased variance and requiring more data, but I don't see how this leads to any qualitative transition from working to not working such that it might explain why they work. That seems to have more to do with the human subjects overweighting noise and the 'bet on sparsity' principle.

I am not sure of the point you are making. In particular, I don't see why would anyone use those improper linear models. It's not 1979 and we can easily fit a variety of linear models including robust ones. Under which circumstances would you prefer an improper linear model to other alternatives available?