What are the open problems in Human Rationality?

I went through all my LW posts and gathered the ones that either presented or reminded me of some problem in human rationality.

1. [LW · GW] As we become more rational, how do we translate/transfer our old values embodied in the less rational subsystems?

2. [LW · GW] How to figure out one's comparative advantage?

3. [LW · GW] Meta-ethics. It's hard to be rational if you don't know where your values are supposed to come from.

4. [LW · GW] Normative ethics. How much weight to put on altruism? Population ethics. Hedonic vs preference utilitarianism. Moral circle. Etc. It's hard to be rational if you don't know what your values are.

5. [LW · GW] Which mental subsystem has one's real values, or how to weigh them.

6. [LW · GW] How to handle moral uncertainty? For example should we discount total utilitarianism because we would have made a deal to for total utilitarianism to give up control in this universe?

7. [LW · GW] If we apply UDT to humans, what does it actually say in various real-life situations like voting or contributing to x-risk reduction?

8. [LW · GW] Does Aumann Agreement apply to humans, and if so how?

9. [LW · GW] Meta-philosophy. It's hard to be rational if one doesn't know how to solve philosophical problems related to rationality.

10. [LW · GW] It's not clear how selfishness works in UDT, which might be a problem if that's the right decision theory for humans.

11. [LW · GW] Bargaining, politics, building alliances, fair division, we still don't know how to apply game theory to a lot of messy real-world problems, especially those involving more than a few people.

12. [LW · GW] Reality fluid vs. caring measure. Subjective anticipation. Anthropics in general.

13. [LW · GW] What is the nature of rationality, and more generally normativity?

14. [LW · GW] What is the right way to handle logical uncertainty, and how does that interact with decision theory, bargaining, and other problems?

Comparing the rate of problems opened vs problems closed, we have so far to go....

I've actually been thinking about this for a while, here's a very rough draft outline of what I've got:

1. Which questions are important?
a. How should we practice cause prioritization in effective altruism?
b. How should we think about long shots at very large effects? (Pascal's Mugging)
c. How much should we be focusing on the global level, vs. our own happiness and ability to lead a normal life?
d. How do we identify gaps in our knowledge that might be wrong and need further evaluation?
e. How do we identify unexamined areas of our lives or decisions we make automatically? Should we examine those areas and make those decisions less automatically?

2. How do we determine whether we are operating in the right paradigm?
a. What are paradigms? Are they useful to think about?
b. If we were using the wrong paradigm, how would we know? How could we change it?
c. How do we learn new paradigms well enough to judge them at all?

3. How do we determine what the possible hypotheses are?
a. Are we unreasonably bad at generating new hypotheses once we have one, due to confirmation bias? How do we solve this?
b. Are there surprising techniques that can help us with this problem?

4. Which of the possible hypotheses is true?
a. How do we make accurate predictions?
b. How do we calibrate our probabilities?

5. How do we balance our explicit reasoning vs. that of other people and society?
a. Inside vs. outside view?
b. How do we identify experts? How much should we trust them?
c. Does cultural evolution produce accurate beliefs? How willing should we be to break tradition?
d. How much should the replication crisis affect our trust in science?
e. How well does good judgment travel across domains?

6. How do we go from accurate beliefs to accurate aliefs and effective action?
a. Akrasia and procrastination
b. Do different parts of the brain have different agendas? How can they all get on the same page?

7. How do we create an internal environment conducive to getting these questions right?
a. Do strong emotions help or hinder rationality?
b. Do meditation and related practices help or hinder rationality?
c. Do psychedelic drugs help or hinder rationality?

8. How do we create a community conducive to getting these questions right?
a. Is having "a rationalist community" useful?
b. How do strong communities arise and maintain themselves?
c. Should a community be organically grown or carefully structured?
d. How do we balance conflicting desires for an accepting where everyone can bring their friends and have fun, vs. high-standards devotion to a serious mission?
e. How do we prevent a rationalist community from becoming insular / echo chambery / cultish?
f. ...without also admitting every homeopath who wants to convince us that "homeopathy is rational"?
g. How do we balance the need for a strong community hub with the need for strong communities on the rim?
h. Can these problems be solved by having many overlapping communities with slightly different standards?

9. How does this community maintain its existence in the face of outside pressure?

I feel daunted by the question, "what are the big open questions the field of Human Rationality needs to answer, in order to help people have more accurate beliefs and/or make better decisions?", but I also think that it's the question at the heart of my research interests. So rather than trying to answer the original question directly, I'm going to share a sampling of my current research interests.

Over in the AMA, I wrote [LW(p) · GW(p)], "My way of investigating always pushes into what I can’t yet see or grasp or articulate. Thus, it has the unfortunate property of being quite difficult to communicate about directly until the research program is mostly complete. So I can say a lot about my earlier work on noticing, but talking coherently about what exactly CFAR’s been paying me for lately is much harder." This will not be a clean bulleted list that doubles as a map of rationality, sorry. It'll be more like sampling of snapshots from the parts of my mind that are trying to build rationality. Here's the collage, in no particular order:

There are things you’re subject to, and things you can take as object. For example, I used to do things like cry when an ambulance went by with its siren on, or say “ouch!” when I put a plate away and it went “clink”, yet I wasn’t aware that I was sensitive to sounds. If asked, “Are you sensitive to sounds?” I’d have said “No.” I did avoid certain sounds in local hill-climby ways, like making music playlists with lots of low strings but no trumpets, or not hanging out with people who speak loudly. But I didn’t “know” I was doing these things; I was *subject* to my sound sensitivity. I could not take it as *object*, so I couldn’t deliberately design my daily life to account for it. Now that I can take my sound sensitivity (and many related things) as object, I’m in a much more powerful position. And it *terrifies* me that I went a quarter of a century without recognizing these basic facts of my experience. It terrifies me even more when I imagine an AI researcher being subject to some similarly crucial thing about how agents work. I would very much like to know what other basic facts of my experience I remain unaware of. I would like to know how to find out what I am currently unable to take as object.

On a related note, you know how an awful lot of people in our community are autistic? It seem to me that our community is subject to this fact. (It also seems to me that many individual people in our community remain subject to most of their autistic patterns, and that this is more like the rule than the exception.) I would like to know what’s going on here, and whether some other state of affairs would be preferable, and how to instantiate that state of affairs.

Why do so many people seem to wait around for other people to teach them things, even when they seem to be trying very hard to learn? Do they think they need permission? Do they think they need authority? What are they protecting? Am I inadvertently destroying it when I try to figure things out for myself? What stops people from interrogating the world on their own terms?

I get an awful lot of use out of asking myself questions. I think I’m unusually good at doing this, and that I know a few other people with this property. I suspect that the really useful thing isn’t so much the questions, as whatever I’m doing with my mind most of the time that allows me to ask good questions. I’d like to know what other people are doing with their minds that prevents this, and whether there’s a different thing to do that’s better.

What is “quality”?

Suppose religion is symbiotic, and not just parasitic. What exactly is it doing for people? How is it doing those things? Are there specific problems it’s solving? What are the problems? How can we solve those problems without tolerating the damage religion causes?

[Some spoilers for bits of the premise of A Fire Upon The Deep and other stories in that sequence.] There’s this alien race in Verner Vinge books called the Tines. A “person” of the Tines species looks at first like a pack of several animals. The singleton members that make up a pack use high-frequency sound, rather than chemical neurotransmitters, to think as one mind. The singleton members of a pack age, so when one of your singletons dies, you adopt a new singleton. Since singletons are all slightly different and sort of have their own personalities, part of personal health and hygiene for Tines involves managing these transitions wisely. If you do a good job — never letting several members die in quick succession, never adopting a singleton that can’t harmonize with the rest of you, taking on new singletons before the oldest ones loose the ability to communicate — then you’re effectively immortal. You just keep amassing new skills and perspectives and thought styles, without drifting too far from your original intentions. If you manage the transitions poorly, though — choosing recklessly, not understanding the patterns an old member has been contributing, participating in a war where several of your singletons may die at once — then your mind could easily become suddenly very different, or disorganized and chaotic, or outright insane, in a way you’ve lost the ability to recover from. I think about the Tines a lot when I experiment with new ways of thinking and feeling. I think much of rationality poses a similar danger to the one faced by the Tines. So I’d like to know what practices constitute personal health and hygiene for cognitive growth and development in humans.

What is original seeing? How does it work? When is it most important? When is it the wrong move? How can I become better at it? How can people who are worse at it than I am become better at it?

In another thread, Adam made a comment that I thought was fantastic. I typed to him, “That comment is fantastic!” As I did so, I noticed that I had an option about how to relate to the comment, and to Adam, when I felt a bid from somewhere in my mind to re-phrase as, “I really like that comment,” or, “I enjoyed reading your comment,” or “I’m excited and impressed by your comment.” That bid came from a place that shares a lot of values with Lesswrong-style rationalists, and 20th century science, and really with liberalism in general. It values objectivity, respect, independence, autonomy, and consent, among other things. It holds map-territory distinctions and keeps its distance from the world, in an attempt to see all things clearly. But I decided to stand behind my claim that the “the comment is fantastic”. I did not “own my experience”, in this case, or highlight that my values are part of me rather than part of the world. I have a feeling that something really important is lost in the careful distance we keep all the time from the world and from each other. Something about the power to act, to affect each other in ways that create small-to-mid-sized superorganisms like teams and communities, something about tending our relationship to the world so that we don’t float off in bubbles of abstraction. Whatever that important thing is, I want to understand it. And I want to protect it, and to incorporate it into my patterns of thought, without loosing all I gain from cold clarity and distance.

I would like to think more clearly, especially when it seems important to do so. There are a lot of things that might affect how clearly you think, some of which are discussed in the Sequences. For example, one common pattern of muddy thought is rationalization, so one way to increase your cognitive clarity is to stop completely ignoring the existence of rationalization. I’ve lately been interested in a category of clarity-increasing thingies that might be sensibly described as “the relationship between a cognitive process and its environment”. By “environment”, I meant to include several things:

• The internal mental environment: the cognitive and emotional situation in which a thought pattern finds itself. Example: When part of my mind is trying to tally up how much money I spent in the past month, and local mental processes desperately want the answer to be “very little” for some reason, my clarity of thought while tallying might not be so great. I expect that well maintained internal mental environments — ones that promote clear thinking — tend to have properties like abundance, spaciousness, and groundedness.
• The internal physical environment: the physiological state of a body. For example, hydration seems to play a shockingly important role in how well I maintain my internal mental environment while I think. If I’m trying to solve a math problem and have had nothing to drink for two hours, it’s likely I’m trying to work in a state of frustration and impatience. Similar things are true of sleep and exercise.
• The external physical environment: the sensory info coming in from the outside world, and the feedback patterns created by external objects and perceptual processes. When I’ve been having a conversation in one room, and then I move to another room, it often feels as though I’ve left half my thoughts behind. I think this is because I’m making extensive use of the walls and couches and such in my computations. I claim that one’s relationship to the external environment can make more or less use of the environment’s supportive potential, and that environments can be arranged in ways that promote clarity of thought.
• The social environment: people, especially frequently encountered ones. The social environment is basically just part of the external physical environment, but it’s such an unusual part that I think it ought to be singled out. First of all, it has powerful effects on the internal mental environment. The phrase “politics is the mind killer” means something like “if you want to design the social environment to maximize muddiness of thought, have I got a deal for you”. Secondly, other minds have the remarkable property of containing complex cognitive processes, which are themselves situated in every level of environment. If you’ve ever confided in a close, reasonable friend who had some distance from your own internal turmoil, you know what I’m getting at here. I’ve thought a lot lately about how to build a “healthy community” in which to situate my thoughts. A good way to think about what I’m trying to do is that I want to cultivate the properties of interpersonal interaction that lead to the highest quality, best maintained internal mental environments for all involved.

What is "groundedness"?

I built a loft bed recently. Not from scratch, just Ikea-style. When I was about halfway through the process, I realized that I’d put one of the panels on backward. I’d made the mistake toward the beginning, so there were already many pieces screwed into that panel, and no way to flip it around without taking the whole bed apart again. At that point, I had a few thoughts in quick succession:

• I really don’t want to take the whole bed apart and put it back together again.
• Maybe I could unscrew the pieces connected to that panel, then carefully balance all of them while I flip the panel around? (Something would probably break if I did that.)
• You know what, maybe I don’t want a dumb loft bed anyway.

It so happens that in this particular case, I sighed, took the bed apart, carefully noted where each bit was supposed to go, flipped the panel around, and put it all back together again perfectly. But I’ve certainly been in similar situations where for some reason, I let one mistake lead to more mistakes. I rushed, broke things, lost pieces, hurt other people, or gave up. I’d like to know what circumstances obtain when I get this right, and what circumstances obtain when I don’t. Where can I get patience, groundedness, clarity, gumption, and care?

I’ve developed a taste for reading books that I hate. I like to try on the perspective of one author after another, authors with whom I think I have really fundamental disagreements about how the world works, how one ought to think, and whether yellow is really such a bad color after all. There’s a generalized version of “reading books you hate” that I might call “perceptual dexterity”, or I might call “the ground of creativity”, which is something like having a thousand prehensile eye-stalks in your mind, and I think prehensile eye-stalks are pretty cool. But I also think it’s generally a good idea to avoid reading books you hate, because your hatred of them is often trying to protect you from “your self and worldview falling apart”, or something. I’d like to know whether my self and worldview are falling apart, or whatever. And if not, I’d like to know whether I’m doing something to prevent it that other people could learn to do, and whether they’d thereby gain access to a whole lot more perspectives from which they could triangulate reality.

There seem some foundational questions to the 'Rationality project', and (reprising my role as querulous critic) are oddly neglected in the 5-10 year history of the rationalist community: conspicuously, I find the best insight into these questions comes from psychology academia.

Is rationality best thought of as a single construct?

It roughly makes sense to talk of 'intelligence' or 'physical fitness' because performance in sub-components positively correlate: although it is hard to say which of an elite ultramarathoner, Judoka, or shotputter is fittest, I can confidently say all of them are fitter than I, and I am fitter than someone who is bedbound.

Is the same true of rationality? If it were the case that performance on tests of (say) callibration, sunk cost fallacy, and anchoring were all independent, then this would suggest 'rationality' is a circle our natural language draws around a grab-bag of skills or practices. The term could therefore mislead us into thinking it is a unified skill which we can 'generally' improve, and our efforts are better addressed at a finer level of granularity.

I think this is plausibly the case (or at least closer to the truth). The main evidence I have in mind is Stanovich's CART, whereby tests on individual sub-components we'd mark as fairly 'pure rationality' (e.g. base-rate neglect, framing, overconfidence - other parts of the CART look very IQ-testy like syllogistic reasoning, on which more later) have only weak correlations with one another (e.g. 0.2 ish).

Is rationality a skill, or a trait?

Perhaps key is that rationality (general sense) is something you can get stronger at or 'level up' in. Yet there is a facially plausible story that rationality (especially so-called 'epistemic' rationality) is something more like IQ: essentially a trait where training can at best enhance performance on sub-components yet not transfer back to the broader construct. Briefly:

• Overall measures of rationality (principally Stanovich's CART) correlate about 0.7 with IQ - not much worse than IQ test subtests correlate with one another or g.
• Infamous challenges in transfer. People whose job relies on a particular 'rationality skill' (e.g. gamblers and calibration) show greater performance in this area but not, as I recall, transfer improvements to others. This improved performance is often not only isolated but also context dependent: people may learn to avoid a particular cognitive bias in their professional lives, but remain generally susceptible to it otherwise.
• The general dearth of well-evidenced successes from training. (cf. the old TAM panel on this topic, where most were autumnal).
• For superforecasters, the GJP sees it can get some boost from training, but (as I understand it) the majority of their performance is attributed to selection, grouping, and aggregation.

It wouldn't necessarily be 'game over' for the 'Rationality project' even if this turns out to be the true story. Even if it is the case that 'drilling vocab' doesn't really improve my g, I might value a larger vocabulary for its own sake. In a similar way, even if there's no transfer, some rationality skills might prove generally useful (and 'improvable') such that drilling them to be useful on their own terms.

The superforecasting point can be argued the other way: that training can still get modest increases in performance in a composite test of epistemic rationality from people already exhibiting elite performance. But it does seem crucial to get a general sense of how well (and how broadly) can training be expected to work: else embarking on a program to 'improve rationality' may end up as ill-starred as the 'brain-training' games/apps fad a few years ago.

One more, because one of my posts presented two open problems, and I only listed one of them above:

15. [LW · GW] Our current theoretical foundations for rationality all assume a fully specified utility function (or the equivalent), or at least a probability distribution on utility functions (to express moral/value uncertainty). But to the extent that humans can be considered to have a utility function at all, it's may best be viewed as a partial function that returns "unknown" for most of the input domain. Our current decision theories can't handle this because they would end up trying to add "unknown" to a numerical value during expected utility computation. Forcing humans to come up with an utility function or even a probability distribution on utility functions in order to use decision theory seems highly unsafe so we need an alternative.

In my opinion, the Hamming problem of group rationality, and possibly the Hamming problem of rationality generally, is how to preserve epistemic rationality under the inherent political pressures existing in a group produces.

It is the Hamming problem because if it isn't solved, everything else, including all the progress made on individual rationality, is doomed to become utterly worthless. We are not designed to be rational, and this is most harmful in group contexts, where the elephants in our brains take the most control from the riders and we have the least idea of what goals we are actually working towards.

I do not currently have any good models on how to attack it. The one person I thought might be making some progress on it was Brent, but he's now been justly exiled, and I have the sense that his pre-exile intellectual output is now subject to a high degree of scrutiny. This is understandable, but since I think his explicit models were superior to any anyone else has publicly shared, it's a significant setback.

Since that exile happened, I've attempted to find prior art elsewhere to build on, but the best prospect so far (C. Fred Alford, Group Psychology and Political Theory) turned out to be Freudian garbage.

Group rationality is a big one. It wouldn't surprise me if rationalists are less good on average at co-ordinating than other group because rationalists tend to be more individualistic and have their own opinions of what needs to be done. As an example, how long did it take for us to produce a new LW forum despite half of the people here being programmers? And rationality still doesn't have its own version of CEA.

Did you ever see that early (some might say, premature) trailer for the anthropic horror game SOMA where Jarret was wandering around, woefully confused, trying desperately to figure out where his brain was located?

That's how humans are about their values.

I can't find my utility function.

It's supposed to be inside me, but I see other people whose utility functions are definitely outside of their body, subjected to hellish machinations of capricious tribal egregores and cultural traumas, and they're suffering a lot.

I think my utility function might be an epiphenomenon of my tribe (where is my tribe???), but I'm not sure. There are things you can do to a tribe that change its whole values, so this doesn't seem to draw a firm enough boundary.

My values seem to change from hour to hour. Sometimes the idea of a homogeneous superhappy hedonium blob seems condemnably ugly, other times, they seem fine and good and worthy of living. Sometimes I am filled with compassion for all things, and sometimes I'm just a normal human who draws lines between ingroup and outgroup and only cares about what happens on the inner side.

The only people I know who claim to have fixed utility functions appear to be mutilating themselves to get that way, and I pale at the thought of such scarification, but what is the alternative? Is endless mutation really a value more intrinsic than any other? Have we made some kind of ultimate submission to evolution that will eventually depose us completely in favour of whatever disloyal offspring fight up from us?

Where is my utility function?

The problem of interfaces between cultures.

Humans live in different cultures. A simple version of this is in how cultures greet each other. The Italian double kiss, the ultra orthodox Jewish non touch, the hippie hug, the handshake of various cultures, the Japanese bow/nod, and many more. It's possible to gravely offend a different culture with the way you do introductions.

Now think about the same potential offence but for all conversation culture.

I have the open question of how to successfully interface with other cultures.

For applied rationality, my 10% improvement problem: https://www.lesswrong.com/posts/Aq8QSD3wb2epxuzEC/the-10-improvement-problem

Basically, how do you notice small (10% or less) improvements in areas that are hard to quantify. This is important, because after reaping the low-hanging fruits, stacking those small improvements is how you get ahead.

To me the biggest open problem is how to make existing wisdom more palatable to people who are drawn to the rationalist community. What I have in mind as an expression of this problem is the tension between the post/metarationalists and the, I don't know, hard core of rationalists: I don't think the two are in conflict; the former are trying to bring in things from outside the traditional sources historically liked by rationalists; the latter see themselves as defending rationality from being polluted by antirationalist stuff; and both are trying to make rationality better (the former via adding; the latter via protecting and refining). The result is conflict even if I think the missions are not in conflict, though, so it seems an open problem is figuring out how to address that conflict.

Is there a way to integrate probability based forecasting into the daily life of the average person that's clearly beneficial for them?

I don't think we are yet at that point where I can clearly say, that we are there. I think we would need new software to do this well.

What sort of standards for intellectual honesty make sense, given that:

• There's a large number of free variables in what information you present to people. You can be quite misleading while saying purely true information. "Not lying" doesn't seem sufficient a norm. [LW · GW]
• It's hard to build norms around complex behavior. Humans have an easier time following (and flagging violations of) bright lines, compared to more nuanced guidelines.

How about: "What is rationality?" and "Will rationality actually help you if you're not trying to design an AI?"

Don't get me wrong. I really like LessWrong. I've been fairly involved in the Seattle rationality community. Yet, all the same, I can't help but think that actual rationality hasn't really helped me all that much in my everyday life. I can point to very few things where I've used a Rationality Technique to make a decision, and none of those decisions were especially high-impact.

In my life, rationality has been a hobby. If I weren't reading the sequences, I'd be arguing about geopolitics, or playing board games. So, to me, the most open question in rationality is, "Why should one bother? What special claim does rationality have over my time and attention that, say, Starcraft does not?"

Dealing with uncertainty about our own logic. It's a circular sort of problem: any logic I use to deal with my potentially flawed reasoning is itself potentially flawed. It gets worse when you deal with potential rationalization.

I think a big open question is how to think about rationality across paradigms or incompatible ontological schemas. In focusing only on belief evaluation, we miss there is generally tacit framework of background understanding which is required for the beliefs to be understood and evaluated. 

What happens when people have vastly different background understandings? How does rationality operate in such contexts?


 

One open problem:

The problem of communication across agents, and generally what I call "miscommunication".

I'm a natural systems scientist, who noticed some time ago that it takes organization in nature to use energy, and then takes development to create organization. I was a physicist, but physics seems limited to studying things that can be represented with numbers. Organization is really not so open to that. So I developed various pattern languages, starting with the seemingly universal pattern of "natural growth," as a process of organization development. The pattern begins with an accumulation of small things or changes expanding into big ones, and then to produce stable organization in the end the progression has to reverse, with accumulating smaller and smaller things or changes leading toward a state of completion. If you look close, really everything we do follows that pattern, of diverging then converging accumulations. That seems to mark "time" as an organizational, not a numeric process.

So... for science to apparently not to notice that seems to be a very strong hint as to what in the world is wrong with human thinking. It's not just Descartes. Somewhere in the evolution of human thought, seemingly well before science chose to define nature with numbers, we seem to have decided that it was the job of reason to define reality. Of course we then get it wrong every time, as reality is already defined and so defining it is just not our job. Nature defines reality, and we can only do our best to study it. So, I don't know is my questions will help anyone else yet, but they might. It does seem "less wrong" as a point of view though, rather than to err by assuming our best rules of prediction define the world we are trying to predict. That's contradictory.

There's one overwhelmingly big problem, which is solving epistemology.