I'd blame the MIT press release organ for being clickbait, but the paper isn't much better. It's almost entirely flash with very little substance. This is not to say there's no math - the math just doesn't much apply to the real world. For example, the idea that deep neural networks work well because they recreate the hierarchical generative process for the data is a common misconception.

And then from this starting point you want to start speculating?

This might make some sense if DNNs were being used to further our understanding of theoretical physics, but afaik they're not. They're being used to classify cat pics. SInce when do you use polynomial Hamiltonians to recognise cats?

These properties mean that neural networks do not need to approximate an infinitude of possible mathematical functions but only a tiny subset of the simplest ones

No finite DNN can approximate sin(x) over the entire real numbers, unless you cheat by having a sin(x) activation function.

I have another theory on how Deep Learning works: http://lesswrong.com/lw/m9p/approximating_solomonoff_induction/

The idea is that neural networks are a (somewhat crude) approximation of solomonoff induction.

How can neural networks approximate functions well in practice, when the set of possible functions is exponentially larger than the set of practically possible networks?

This question answers itself. If neural networks could really approximate every possible function, they could never generalize. That is the whole point of statistical learning theory: you get a Probably Approximately Correct (PAC) generalization bound when 1) your learning machine gets good empirical accuracy and 2) the number of possible functions expressible by the machine is small in some sense compared to the volume of training data.

This also reminds me of the scale of the universe claims that everything from subatomics to galaxy clusters scale at 10x9.

animations: http://scaleofuniverse.com/

http://apod.nasa.gov/apod/ap120312.html

cosmological https://archive.org/details/arxiv-astro-ph9404054

table http://physicsoftheuniverse.com/numbers.html

it looks like this is a big boost to the simulation argument.

It could be that you only get civilizations "in universes is governed by a tiny subset of all possible functions" because else wise either evolution can't "discover" how to create intelligent life, or evolved intelligent life can't figure out science.

More patterns, and set limiters found.

https://www.quantamagazine.org/20161115-strange-numbers-found-in-particle-collisions/

“It seems so that the periods which nature wants are a smaller set than the periods mathematics can define, but we cannot define very cleanly what this subset really is.”

Brown is looking to prove that there’s a kind of mathematical group — a Galois group — acting on the set of periods that come from Feynman diagrams. “The answer seems to be yes in every single case that’s ever been computed,” he said, but proof that the relationship holds categorically is still in the distance. “If it were true that there were a group acting on the numbers coming from physics, that means you’re finding a huge class of symmetries,” Brown said. “If that’s true, then the next step is to ask why there’s this big symmetry group and what possible physics meaning could it have.”

I am interested in this line of research, I feel it needs a lot more work than one paper, though.

A key question is whether we can dig down into the relationship between environments and learning agents. Are there low complexity environments that neural networks do badly in?

What is really essential about our laws of physics to create a world that neural networks do relatively well in?

and since you can't "look inside a NN, you cant even see problems developing

"If there hadn’t been an interpretable model, Malioutov cautions, “you could accidentally kill people.”

This is why so many are reluctant to gamble on the mysteries of neural networks."

http://nautil.us/issue/40/learning/is-artificial-intelligence-permanently-inscrutable

and since you can't "look inside a NN, you cant even see problems developing

"If there hadn’t been an interpretable model, Malioutov cautions, “you could accidentally kill people.”

This is why so many are reluctant to gamble on the mysteries of neural networks."

http://nautil.us/issue/40/learning/is-artificial-intelligence-permanently-inscrutable

and since you can't "look inside a NN, you cant even see problems developing

"If there hadn’t been an interpretable model, Malioutov cautions, “you could accidentally kill people.”

This is why so many are reluctant to gamble on the mysteries of neural networks."

http://nautil.us/issue/40/learning/is-artificial-intelligence-permanently-inscrutable