Definition Practice: Applied Rationality

Rationality is the application of reason --systematized ways of thinking that have been found to be truth-preserving -- to empirical evidence, in order to distinguish truth from untruth and thus usually arrive at (more) accurate beliefs. Applied rationality is doing this in the pursuit of practical ends such as saving lives, getting rich, having fun, etc. -- by arriving at more accurate beliefs either about matters relevant to those ends, or about matters relevant to pursuing them. (The latter may apply even when the actual pursuit doesn't directly involve reasoning about evidence. If you try to play tennis by explicitly estimating probability distributions over what your opponent will do and where the ball will land as a result, you will lose, but you might do well while not in mid-game to think carefully about what sort of practice regime works best for you, or how to prepare for your next opponent's style of play, or when to retire from your successful tennis career for the overall-happiest life.)

Applied rationality is the attempt to pursue one's goals more effectively by thinking more clearly.

The single sentence is, of course, punchier and clearer. The paragraph goes into a little more depth about what "thinking more clearly" means here (it means things that have been found not to lead from truth to falsehood, so that you can do it at length without your thinking itself leading you into error, in ways that have themselves been thought about and understood); it gives a few somewhat-varied examples of high-level goals because I think without that some people would think that "pursue one's goals" implies pursuing selfish goals, and maybe some others would think it implies the reverse; and it makes explicit the fact that you're doing "applied rationality" if you use reason and evidence to decide how to proceed, even if the way it tells you to proceed is something other than using reason and evidence. The single sentence on its own might suggest a straw-Vulcan sort of rationality; I hope the paragraph makes it clearer that that would be a mistake.

It seems that James Lucassen and I have quite different notions of "applied rationality" in mind; I am concerned with what the activity is that the term denotes and he is concerned with how to assess how much of it an agent possesses. I don't commonly actually use the term "applied rationality" nor do I often see it used, but I would expect "my" usage to be more common.

When a bounded agent attempts a task, we observe some degree of success. But the degree of success depends on many factors that are not "part of" the agent - outside the Cartesian boundary that we (the observers) choose to draw for modeling purposes. These factors include things like power, luck, task difficulty, assistance, etc. If we are concerned with the agent as a learner and don't consider knowledge as part of the agent, factors like knowledge, skills, beliefs, etc. are also externalized. Applied rationality is the result of attempting to distill this big complicated mapping from (agent, power, luck, task, knowledge, skills, beliefs, etc) -> success down to just agent -> success. This lets us assign each agent a one-dimensional score: "how well do you achieve goals overall?" Note that for no-free-lunch reasons, this already-fuzzy thing is further fuzzified by considering tasks according to the stuff the observer cares about somehow.

Applied rationality is a property of a bounded agent, which attempts to describe how successful that agent tends to be when you throw tasks at it, while controlling for both "environmental" factors such as luck and "epistemic" factors such as beliefs.

In this framing, it's pretty easy to define epistemic rationality analogously compressing from everything -> prediction loss to just agent -> prediction loss.

However, in retrospect I think the definition I gave here is pretty identical to how I would have defined "intelligence", just without reference to the "mapping broad start distribution to narrow outcome distribution" idea (optimization power) that I usually associate with that term. If anyone could clarify specifically the difference between applied rationality and intelligence, I would be interested. 

Maybe you also have to control for "computational factors" like raw processing power, or something? But then what's left inside the Cartesian boundary? Just the algorithm? That seems like it has potential, but still feels messy.