Mathematical simplicity bias and exponential functions

I have this vague idea that sometime in our past, people thought that knowledge was like an almanac; a repository of zillions of tiny true facts that summed up to being able to predict stuff about stuff, but without a general understanding of how things work. There was no general understanding because any heuristic that would begin to explain how things work would immediately be discounted by the single tiny fact, easily found, that contradicted it. Details and concern with minutia and complexity is actually anti-science for this reason. It’s not that details and complexity aren’t important, but you make no progress if you consider them from the beginning.

And then I wondered: is this knee-jerk reaction to dismiss any challenge of the keep-it-simple conventional wisdom the reason why we’re not making more progress in complex fields like biology?

For classical physics it has been the case that the simpler the hypothetical model you verify, the more you cash out in terms of understanding physics. The simpler the hypothesis you test, the easier it is to establish if the hypothesis is true and the more you learn about physics if it is true. However, what considering and verifying simpler and simpler hypotheses actually does is transfer the difficulty of understanding the real-world problem to the experimental set-up. To verify your super-simple hypothesis, you need to eliminate confounding factors from the experiment. Success in classical physics has occurred because when experiments were done, confounding factors could be eliminated through a well-designed set-up or were small enough to neglect. (Consider Galileo’s purported experiment of dropping two objects from a height – in real life that particular experiment doesn’t work because the lighter object may fall more slowly.)

In complex fields this type of modeling via simplification doesn’t seem to cash out as well, because it’s more difficult to control the experimental set-up and the confounding effects aren't negligible. So while I've always believed that models need to be simple, I would consider a different paradigm if it could work. How could understanding the world work any other way than through simple models?

Some method trends in biology: high through-put, random searches, brute force, etc.

I recall a discussion I had with a fellow econ student on the effects of higher taxes. He said something to the effect of, "Higher taxes are inefficient, and all you need to do to prove that is to draw the graph." (Unfortunately the topic changed before I could point out the problems with this statement.)

This (rather common) view reflects two major problems with modeling (particularly in economics): an amoral value (economic efficiency) becomes a normative value because it's relatively easy to understand and (in theory) measure, and, more relevant as an example for this post, the model is seen as demonstrating reality, rather than vice versa. The model thus becomes a complete way of looking at the world, as it is both normative and the world is supposed to conform to it.

I think a lot of scientists see theory as the highest good: reality is defective insofar as it fails to conform to an elegant theory, rather than the other way around. When expressed this way, it's obviously a foolish idea, but it's an insidious one nonetheless. "I'd be right if it weren't for all those confounding variables!" may be true, but you're still wrong.

One of biases that are extremely prevalent in science, but are rarely talked about anywhere, is bias towards models that are mathematically simple and easier to operate on.

I think that this is a heuristic rather than a bias, because favoring simple models over complex ones is generally a good thing. In particular, the complexity prior is claimed by some to be a fundemental principle of intelligence.

To me, the problem is not "Mathematical Simplicity Bias," but rather, failing to check the model with empirical data. It seems totally reasonable to start with a simple model and add complexity necessary to explain the phenomenon. (Of course it is best to test the model on new data.)

Also, if you're going to claim Mathematical Simplicity Bias is, "One of biases that are extremely prevalent in science," it would help to provide real examples of scientists failing because of it.

In general, rules of thumb have two dimensions - applicability (that is the size of the domain where it applies) and efficacy (the amount or degree of guidance that the rule provides).

Simplicity, a.k.a Occam's Razor, is mentioned frequently as a guide in these (philosophy of science/atheist/AI aficionado) circles. However, it is notable more for its broad applicability than for its efficacy compared to other, less-broadly-applicable guidelines.

Try formulating a rule for listing natural numbers (positive integers) without repeats that does not generally trend upwards. For example, you could alternate between powers of ten and powers of two: 1, 10, 2, 100, 4, 1000, ... Regardless of the rule, you cannot list the natural numbers from largest to smallest; there is no largest. Whichever you pick as your first, you will eventually be forced past by the "no repeats" clause.

A general learner can be viewed as outputting a list of numbers (coding for hypotheses). Occam's razor is roughly the observation "the numbers will generally trend upwards". There's still a lot up in the air after that observation.

A discharging capacitor is a pretty good fit for exponential decay. (At least, until it's very very close to being completely discharged.)

I don't see how they can even try to apply -any- curve to something that has feedbacks over time, like population or gdp. Technology is an obvious confounding variable there, with natural events coming into play as well.

"All models are wrong, some are useful" - George Box

Comparing a made-up exponential to a process that no scientist who knew anything about radioactivity would expect to model with anything but a sum of coupled exponentials is a bit of a straw man. There's a bias to simplicity, certainly, but there's not that much bias!

Obligatory topical XKCD link. Though it's linear, not exponential.

I do not think your claim is what you think it is.

I think your claim is that some people mistake the model for the reality, the map for the territory. Of course models are simpler than reality! That's why they're called "models."

Physics seems to have gotten wiser about this. The Newtonians, and later the Copenhagenites, did fall quite hard for this trap (though the Newtonians can be forgiven to some degree!). More recently, however, the undisputed champion physical model, whose predictions hold to 987 digits of accuracy (not really), has the humble name "The Standard Model," and it's clear that no one thinks it's the ultimate true nature of reality.

Can you give specific examples of people making big mistakes from map/territory confusion? The closest thing I can think of offhand is the Stern Report, which tries to make economic calculations a century from now based on our current best climate+social+political+economic models.

Any continuous function is approximately linear over a small enough scale. ;)

My computer is biased toward not running at 100 petahertz and having 70 petabytes of RAM. My brain is biased toward not using so many complicated models that it needs 1 trillion neurons each with 1 million connections and firing up to 10,000 times per second.

And now for something perhaps more useful than sarcasm, it seems to me that people tend to simply come up with the consistent model that is either the easiest one to compute or the simplest one to describe. Are heuristics for inconsistency, such as "exponential growth/decay rarely occurs in nature", quickly spread and often used? How about better approximations such as logistical growth?

I recall a discussion I had with a fellow econ student on the effects of higher taxes. He said, "Higher taxes are inefficient; should I draw the graph." (Unfortunately the topic changed before I could dissect this for him.)

This (rather common) view reflects two major problems with modeling (particularly in economics): an amoral value (economic efficiency) becomes a normative value because it's relatively easy to understand and (in theory) measure, and, more relevant as an example for this post, the model is seen as demonstrating reality, and not vice versa. The model thus becomes a complete way of looking at the world, as it is both normative and the world is supposed to conform to it.

I think a lot of scientists see theory as the highest good: reality is defective insofar as it fails to conform to an elegant theory, rather than the other way around. When expressed this way, it's obviously a foolish idea, but it's an insidious one nonetheless. "I'd be right if it weren't for all those confounding variables!" may be true, but you're still wrong.

Generally (and therefore somewhat inaccurately) speaking, one way that our brains seem to handle the sheer complexity computing in the real world us is a tendency to simplify the information we gather.

In many cases these sorts of extremely simple models didn't start that way. They may have started with more parameters and complexity. But as they were repeated, explained and applied the model becomes, in effect, simpler. The example begins to represent the entire model, rather than serving to show only a piece of it.

Technically the exponential radioactive decay model for radioactivity of a mixture has most of the pieces you describe fairly directly. But this hardly means they will be appropriately applied, that they will be available when we are thinking of how to use the model. We need to fight the simplification effect to be able to make our models more nuanced and detailed - even though they are still almost certainly lossy compression of the facts, observations, and phenomena they were built from.

On the other hand, the simplification serves its purpose too, if we could devote unlimited cognitive resources to a model, then we risk not being unable to actually reach a decision from the model.