What's next for instrumental rationality?

FWIW, I'm trying to create something of a bridge between "the ancient wisdom of people who thought deeply about this sort of thing a long time ago" and "modern social science which with all its limitations at least attempts to test hypotheses with some rigor sometimes" in my sequence on virtues [? · GW]. That might serve as a useful platform from which to launch this new rigorous instrumental rationality guide.

I love the "E+M" name. It reminds of electricity and magnetism, and IMO embedded agency and multi-agent rationality will eventually be seen as two sides of the same coin about as much as electricity and magnetism.

I think our current best theories of both don't look much like each other, and predict that as we progress on each, they will slowly look more and more like one field.

A minor point, but I think decision theory and game theory are actually more tried and true than causal inference. At least they are older than the theory of causal graphs. In particular Richard Jeffrey's decision theory can be seen as an extension of probability theory.

No Free Lunch means that optimization requires taking advantage of underlying structure in the set of possible environments. In the case epistemics, we all share close-to-the-same-environment (including having similar minds), so there are a lot of universally-useful optimizations for learning about the environment. 

Optimizations over the space of "how-to-behave instructions" requires some similar underlying structure. Such structure can emerge for two reasons: (1) because of the shared environment, or (2) because of shared goals. (Yeah, I'm thinking about agents as cartesian, in the sense of separating the goals and the environment, but to be fair so do L+P+S+C.)

On the environment side, this leads to convergent behaviours (which can also be thought of as behaviours resulting from selection theorems), like good epistemics, or gaining power over resources.

When it comes to goals, on the other hand, it is both possible (by the orthogonality thesis) and the case that different peole have vastly different goals (e.g. some people want to live forever, some want to commit suicide, and these two groups probably require mostly different strategies). Less in common between different people's goals means less universally-useful how-to-behave instructions. Nonetheless, optimizing behaviours that are commonly prioritized is close enough to universally useful, e.g. doing relationships well.

Perhaps an "Instrumental Sequences" would include the above categories as major chapters. In such a case, as indicated in the post, current reseaerch being posted on Lesswrong gives an approximate idea of what such sequences could look like.

I would be enormously interested in any sort of "universal" multi-agent rationality model, but it seems to me that there is an enormous amount of work that would need to be done to get there. All the existing work that I've read has been either or both of extremely narrow, or not at all well founded.

Barring superhuman advances in math, I suspect that any such thing is more than a hundred years of widespread concerted research away. I would love to be proven wrong on this!