The Inefficiency of Theoretical Discovery

I vastly disagree. I will just state it for now, and hopefully this will be a commitment to explain it further when I have the time. Here are my unjustified assertions about the nature of philosophy regarding OP's topics:

1. Philosophy has the most huge search space known to man, it encompasses everything (a) without a good clear-cut solution and (b) which has any hope to be solved (this rules out two extremes: science and religion).

2. Philosophy, by its very nature, has few systematized methods for efficient search. Seems like we discovered logical and clear thinking recently, but that's almost about it.

3. Because it is so difficult, philosophy is wrong 99,9% of the time.

4. When philosophy is right, major breakthroughs are made, sciences are created, new reasoning tools, higher moral standards and so on.

5. There's a massive and astronomical hindsight bias. Once solved a problem is no longer on the realm of philosophy and the solution tend to seem extremely obvious after 1 or 2 generations.

Thus, low hanging fruits in philosophy are nowhere to be found. Most of your examples were already found, they just need to be worked on. I chalenge you to present a yet unknown low hanging fruit, one all your peers don't know it already, one which would knock Nick's socks off.

Luke,

I think you are mistaken about the relative efficiency / inefficiency of scientific research. I believe that research is comparably efficient to much of industry, and that many of the things that look like inefficiencies are actually trading off small local gains for large global gains. I've come to this conclusion as the result of years of doing scientific research, where almost every promising idea I've come up with (including some that I thought quite clever) had already been explored by someone else. In fact, the typical case for when I was able to make progress was when solving the problem required a combination of tools, each of which individually was relatively rare in the field.

For instance, my paper on stochastic verification required: (i) familiarity of sum-of-squares programming; (ii) the application of supermartingale techniques from statistics; and (iii) the ability to produce relatively non-trivial convex relaxations of a difficult optimization problem. In robotics, most people are familiar with convex optimization, and at least some are familiar with sum-of-squares programming and supermartingales. In fact, at least one other person had already published a fairly similar paper but had not formulated the problem in a way that was useful for systems of practical levels of complexity, probably because they had (i) and (ii) but lacked (iii).

Of course, it is true that your point #3 ("new tools open up promising new angles of attack") often does lead to low-hanging fruit. In machine learning, often when a new tool is discovered there will be a sizable contingent of the community who devotes resources to exploring possible uses of that tool. However, I think the time-scale for this is shorter than you might imagine. I would guess that, in machine learning at least, most of the ``easy'' applications of a new tool have been exhausted within five years of its introduction. This is not to say that new applications don't trickle in, but they tend to either be esoteric or else require adapting the tool in some non-trivial way.

My impression is that MIRI believes that most of what drives what they perceive to be low-hanging fruit comes from #4 ("progress is only valuable to those with unusual views"). I think this is probably true, but not, as you probably believe, due to differing views about the intelligent explosion; I believe instead that MIRI's differing views come from a misunderstanding of the context of their research.

For instance, MIRI repeatedly brings up Paul's probabilistic metamathematics as an important piece of research progress produced by MIRI. I've mostly hedged when people ask me about this, because I do think it is an interesting result, but I feel by now that it has been oversold by MIRI. One could argue that the result is relevant to one or more of logic, philosophy, or AI, but I'll focus on the last one because it is presumably what MIRI cares about most. The standard argument in favor of this line of research is that Lob's theorem implies that purely logical systems will have difficulty reasoning about their future behavior, and that therefore it is worth asking whether a probabilistic language of thought can overcome this obstacle. But I believe this is due to a failure to think sufficiently concretely about the problem. To give two examples: first, humans seem perfectly capable of making predictions about their future behavior despite this issue, and I have yet to see an argument for why AIs should be different. Secondly, we manage to routinely prove facts about the behavior of programs (this is the field of program analysis) despite the fact that in theory this should be "undecidable". This is because the undecidability issues don't really crop up in practice. If MIRI wanted to make the case that they will crop up for a human-level AI in important ways, they should at least mention and respond to the large literature pointing in the opposite direction, and explain why those tools or their successors would be insufficient.

A full response to the misguidedness of the Lob obstacle as a problem of interest probably necessitates its own post, but hopefully this serves to at least partially illustrate my point. Another example would be decision theory of modal agents. I also won't take the time to treat this in detail, but will simply note that this work studies a form of decision theory that MIRI itself invented, and that no one else uses or studies. It should therefore perhaps be unsurprising that it is relatively easy to prove novel results about it. It would be the equivalent of an activity I did to amuse myself in high school, which was to invent new mathematical objects and then study their properties. But I think it would be mistaken to point to this as an example of inefficiency in research. (And yes, I do think the idea of formalizing decision theory in terms of code is an interesting one. I just don't think you get to point to this as an example of why research is inefficient.)

I'm not sure this is the best description of my objections to this post, but I need to start work now and it seems best to avoid keeping this around as a draft for too long, so I'm going to go ahead and post it and wait for everyone to tear it to shreds.

Regardless of why the opportunity has presented itself, can we hope that the MIRI research team and associated researchers will use (or are using) the fact that "visible progress in decision theory is one way to “make a name” for oneself" and proceed to do so? Seems like pretty low-hanging status-fruit given the team's progress so far.

Just wanted to mention that Physics is not immune to this. Bell's theorem requires only a first-year college math skill, yet it took 30 odd years after EPR to formulate it. Not even Einstein himself was able do it. Event horizons and inescapability of singularity required virtually no new math beyond 1917, yet it took some 50 years for the physicists to understand the picture. There are clearly some mental blocks people have which take decades and new generations to overcome.

Charles Fort, Lo!: "If human thought is a growth, like all other growths, its logic is without foundation of its own, and is only the adjusting constructiveness of all other growing things. A tree can not find out, as it were, how to blossom, until comes blossom-time. A social growth cannot find out the use of steam engines, until comes steam-engine-time. For whatever is supposed to be meant by progress, there is no need in human minds for standards of their own: this is in the sense that no part of a growing plant needs guidance of its own devising, nor special knowledge of its own as to how to become a leaf or a root. It needs no base of its own, because the relative wholeness of the plant is relative baseness to its parts."

The means to make steam engines existed centuries before the age of steam. Ground glass lenses existed long before anyone put one in front of the other to make a telescope. The first people of the Americas had wheels - on toys, not carts. Great music could happen sooner (it usually doesn't rely on new inventions).

I am convinced we are drowning in miraculous solutions to weighty problems and we are simply overlooking them. If I knew how to prepend 'steam engine time,' I assure you I would do so.

Some examples of a different kind of inefficiency, from AntiFragile:

It struck me how lacking in imagination we are: we had been putting our suitcases on top of a cart with wheels, but nobody thought of putting tiny wheels directly under the suitcase. Can you imagine that it took close to six thousand years between the invention of the wheel (by, we assume, the Mesopotamians) and this brilliant implementation (by some luggage maker in a drab industrial suburb)? And billions of hours spent by travelers like myself schlepping luggage through corridors full of rude customs officers. Worse, this took place three decades or so after we put a man on the moon.

...The story of the wheel itself is even more humbling than that of the suitcase: we keep being reminded that the Mesoamericans did not invent the wheel. They did. They had wheels. But the wheels were on small toys for children. It was just like the story of the suitcase: the Mayans and Zapotecs did not make the leap to the application. They used vast quantities of human labor, corn maize, and lactic acid to move gigantic slabs of stone in the flat spaces ideal for pushcarts and chariots where they built their pyramids. They even rolled them on logs of wood. Meanwhile, their small children were rolling their toys on the stucco floors (or perhaps not even doing that, as the toys might have been solely used for mortuary purposes).

The same story holds for the steam engine: the Greeks had an operating version of it, for amusement, of course: the aeolipyle, a turbine that spins when heated, as described by Hero of Alexandria. But it took the Industrial Revolution for us to discover this earlier discovery.

I think conformity effects play a huge role in this area. The large majority of modern philosophers all have similar educational and cultural backgrounds. They go to elite universities. They read the standard Western philosophical canon. They work very hard to publish a lot of papers in prestigious journals. They are friends with other academics and with other high achievers in the "standard" fields like finance, law, and medicine. Their parents were probably academics or from the upper middle class. They have spent most of their lives in a university setting.

If I had to take an honest guess? Theoretical discovery will behave "inefficiently" when it requires a breadth-first (or at least, breadth-focused) search through the idea space before you can find things that "fit together". Only once you have a bunch of things which "fit together" can you look at the shape of the "hole in idea-space" they all border, dive to the bottom of that lake, and bring up an entirely new idea which links them or unifies them.

So:

1) Mostly agreed, as described above.

2) As described above. My reasoning is sociological: our current reward system for researchers optimizes our research process for depth rather than breadth. Looking where others are paid not to look would usually be a decent way to find things others haven't seen.

3) I don't know more than the bare minimum about decision theory, so I can't say.

Now, as to analysis of an intelligence explosion taking such a long time, I have an Opinion (beware the capital letter): there may not be an intelligence explosion. Current research into AI indicates that coming up with formal models of utility-optimal agents in unknown active environments is the easy part, making them conscious and (stably) self-modifying is the "next part" currently under research, and scaling them down to fit inside the real universe is the hard part.

Schmidhuber (and when it comes to UAI I definitely root for the Schmidhubristic team ;-)) has claimed that Goedel Machine self-rewrites would dramatically speed up a mere AIXI paired with a mere HSearch until they became effective within the real world, but that's putting his faith in the Goedel Machine's proof searcher rather than in his (and his student's) own proofs that their algorithms genuinely are optimal and the problems they're tackling genuinely do have these nasty time-and-space bounds. If the first team to build an AI has to either keep it in an exponentially-small environment (existing AIXI models) or wait for astronomical periods of time for even the first self-rewrite, then the human race will die of an asteroid strike long before Friend Clippy can take us out.

This is the same nasty problem that AI has faced since the GOFAI days: easy to find an algorithm that locates the optimal move by searching the entire solution space, hard as hell to prove it will take any usefully small period of time to do so.

It strikes me that there's a second set of reasons that read something like:

1. There is significant motivation to get it wrong, or wrong answers are very obvious.

See: the free will problem.

Normative uncertainty does not seem particularly interesting. You can just use set multiplication over probable futures and evaluations of them, and wind up with a regular problem with certain evaluations. It even agrees with normal methods if you assume a single, certain evaluation.

Now, as a concept it's important - realizing that you might not know what you want is a tremendous source of uncertainty that is easy to overlook. I just don't think that any new concepts or mathematical tools are needed to tackle it.

Do you agree about the relative inefficiency of theoretical discovery?

In presence, yes. In degree, no. Even the efficient market hypothesis presumes (indeed, requires) some delay inefficiencies, and even restricting to fields where economic incentives are very, very high we see some pretty significant delay inefficiencies.

The modern containerization revolution is dependent on certain types of math and metallurgy being available, but the underlying tools were probably available before the first world war (and, indeed, would have been even more useful if developed it then, due to varying railway gauges), but didn't take off until the 1950s, and were controversial for a near decade after that. Supply chain management innovations since, including as recently as WalMart's (and to a lesser extent, Amazon's) shipping models, have trailed their underlying technologies what I'd estimate to at least a decade. Even purely digital financial markets have remarkable innovations that seem obvious in retrospect yet weren't around for the longest time.

Philosophy seems to have further delays on top of that, but that they're present in other fields suggests that we can overestimate the impact of some individual causes.

What are some other signs of likely low-hanging fruit for theoretical progress? What’s up with decision theory having so much low-hanging fruit?

I've been looking at things from a more basic model. Innovation seems to be the ability to non-randomly(1) put together component concepts(2) into an implementable (3) result, some of the meaningful attributes will be where :

• Where the typical connections of thoughts are non-complete, where they are based on untested or (especially) false assumptions.
• Where we have unusual component concepts available, or (more commonly) where the average researcher is discouraged from outside component concepts. Interdisciplinary discoveries are major soft spots, here.
• Where the costs of testing hypothesis are very small, have recently become smaller, or are unusually small for our skillset but not the average researcher.

These seems to cover decision theory, although it should be noted not all of the side effects of these attributes are beneficial.