Branches of rationality

I think there are many important unsolved problems in the theoretical/philosophical parts of rationality, and this post seems to under-emphasize them.

What would a picture of rationality’s goal that correctly emphasized them look like?

The point is, we don't just want to turn humans into coherent agents, we want to turn them into coherent agents who can be said to have the same preferences as the original humans. But given that we don't have a theory of preferences for incoherent agents, how do we know that any given trick intended to improve coherence is preference-preserving? Right now we have little to guide us except intuition.

I absolutely agree. The actual question I had written on my sheet, as I tried to figure out what a more powerful “rationality” might include, was “... into coherent agents, with something like the goals ‘we’ wish to have?” Branch #8 above is exactly the art of not having the goals-one-acts-on be at odds with the goals-one-actually-cares-about (and includes much mention of the usefulness of theory).

My impression, though, is that some of the other branches of rationality in the post are very helpful for self-modifying in a manner you’re less likely to regret. Philosophy always holds dangers, but a person approaching the question of “What goals shall I choose?”, and encountering confusing information that may affect what he wants (e.g., encountering arguments in meta-ethics, or realizing his religion is false, or realizing he might be able to positively or negatively affect a disorienting number of lives) will be much better off if he already has good self-knowledge and has accepted that his current state is his current state (vs. if he wants desperately to maintain that, say, he doesn’t care about status and that only utilitarian expected-global-happiness-impacts affect his behavior -- a surprisingly common nerd failure mode).

I don’t know how to extrapolate the preferences of myself or other people either, but my guess is, while further theoretical work is critical, it’ll be easier to do this work in a non-insane fashion in the context of a larger, or more whole-personed, rationality. What are your thoughts here?

I, the entity that is typing these words, do not approve of unconscious preferences when they conflict with conscious ones.

I think there are many important unsolved problems in the theoretical/philosophical parts of rationality, and this post seems to under-emphasize them.

Agreed to an extent, but most folk aren't out to become Friendliness philosophers. One branch that went unmentioned in the post and would be useful for both philosophers and pragmatists includes the ability to construct (largely cross-domain / overarching) ontologies out of experience and abstract knowledge, the ability to maintain such ontologies (propagating beliefs across domains, noticing implications of belief structures and patterns, noticing incoherence), and the disposition of staying non-attached to familiar ontologies (e.g. naturalism/reductionism) and non-averse to unfamiliar/enemy ontologies (e.g. spiritualism/phenomenology). This is largely what distinguishes exemplary rationalists from merely good rationalists, and it's barely talked about at all on Less Wrong.

Yeah. I've been thinking a bit about this. For some people, realising that they can think about thinking is revelatory, and the notion that they can change how they think is filled with fearful trepidation.

What other extremely basic things like this are there?

We are probably inadvertently selecting for people who have some self taught/innate ability to self modify, since people who aren't good at this don't get as much out of the sequences. I'm currently working with a couple people that are bad at this to see if its something simple and easily fixed, and will report back with lessons learned.

I'm extremely glad to hear that. Do please report back, whether or not you succeed in teaching the skills.

Your overall point also seems very good.

I actually had "Forming effective teams" on my original list, but then erased it and several others to avoid muddying the discussion with borderline cases of “rationality”. But collecting best practices for teamwork would be extremely cool.

Some other items from my original brainstorm list:

1. Good brain and body health. Exercise, sufficient sleep, social connectedness, regular concrete accomplishments, and anything else that helps keep the brain in its zone of intended functioning. Maintaining high energy levels and a habit of rapidly implementing new ideas and gathering data, so as to avoid building up excuse mechanisms around tasks or inferences that require effort.

2. Analogy and pattern-recognition. Much of your inferential power comes from automatic processes of pattern recognition. One could learn to train this process on good examples (that will help it correctly predict the problems you’re actually facing), and to notice what sort of an impression you have in a given instance, and, from track records and/or priors, how likely that impression is to be correct.

3. Skills learning. Become skilled at learning non-verbal or implicit “doing” competencies, and at trading information back and forth between verbal and non-verbal systems. Example competencies include emotional self-regulation, posture and movement, driving, social perception and interaction, drawing, and martial arts.

Related is another branch that has received relatively little attention (due to a sort of taboo, possibly not unjustified): how to spread one's beliefs (and/or goals) to other people.

A good point. I wonder who does have that knowledge?

Somewhat related to this is the relatively recent field of social epistemology. Just a heads-up.

Especially with vastly abstract topics -- economics, philosophy, etc. -- I find nothing substitutes for working through concrete examples. My brain's ability to just gloss over abstractions is hard to overestimate. So I've sort of trained myself to sound an alarm whenever my feet "don't touch bottom"... that is, when I can't think of concrete examples of the thing I'm talking about.

For example: I remember a few years ago suddenly realizing, in the course of a conversation about currency exchange rates, that I had no idea how such rates are set. I had been working with them, but had never asked myself where they come from... I hadn't really been thinking about them as information that moves from one place to another, but just vaguely taking them for granted as aspects of my environment. That was embarrassing.

Another useful approach in particular domains is to come up with a checklist of questions to ask about every new feature of that domain, and then ask those questions every time.

This is something I started doing a while ago as part of requirements analysis, and it works pretty well in stable domains, though I sheepishly admit that I dropped the discipline once I had internalized the questions. (This is bad practice and I don't endorse it; checklists are useful.)

It's not quite so useful as a general-analysis technique, admittedly, because the scale differences start to kill you. Still it's better than nothing.

Also, at the risk of repeating myself, I find that restating the thing-about-which-the-question-is-or-might-be is a good way to make myself notice gaps.

I sometimes find myself noticing unasked questions when I try to rigorously prove (or just make a convincing argument for) something obvious. And I find myself looking for proofs and/or convincing arguments when I try to generalize something obvious to something not quite so obvious.

Good examples of this can be found in the famous thought experiments of theoretical physics - Maxwell's demons, Carnot's engines, and Einstein's moving observers seeking to synchronize clocks. Examples in the analysis of rationality can be found in the thinking leading to the famous axiomatic characterizations of rational decision making - von Neumann's rational utility maximizers, Nash's rational bargainers, Rubinstein's impatient variant bargainers, and Harsanyi's empathetic utilitarians.

But moving now to the meta-level: You mention two questions - how to ask the right question and how to notice there is a question to be asked. You suggest that the second of these is the true 'right question'. May I suggest that perhaps the best question is "how do you know that you are asking the right question once you have discovered the question?"

Which thinking just draws you in deeper. The 'right' question, to my mind, is the fruitful one - the one that allows you to make progress. But then we need to ask: what do we mean by progress? How do we recognize that we are making progress?

What habits or tricks are helping you apply things in real life, that Less Wrong isn't explicitly teaching?

I phrased them badly.

The following things, I think, are not mutually contradictory:

• Noticing as much as possible about what your brain is doing. Taking special pains to notice defensiveness, wire-heading patterns, and other things like that that you might flinch away from seeing.
• Accepting that your current state is in fact your current state. Getting used to the idea thoroughly, and to its details, so that you won't find yourself painfully flinching away to avoid seeing your current state.
• Helping information get from one part of your brain to another. For example, I find I'm often defensive and mean when I don't want to notice something (e.g., when I don't want to notice that I'm stressed out about a task I haven't done). So, if I instead notice my discomfort about the unfinished task (which may have nothing to do with the poor person who just reminded me of the task), I can short-circuit my brain's defensive reaction.

I guess I did mean more than the above by "accepting as large a part of yourself as possible", but I'm not sure how to unpack it or whether I was right about the other parts.

Anna seems to suggest that a person who goes through life with no goals more significant than a day-by-day maximization of utility is somehow 'wasting their potential' (my choice of words).

That's not what I was trying to say at all. I was trying to note that we pursue even most of our day-to-day goals ineffectively, such that the result looks like a lack of coherent action rather than like choosing a satisfying smallscale life.

For example, I'll often find myself automatically blurting out a correction to something someone said in conversation, even in cases where such argument will decrease my enjoyment, the other person's enjoyment, and the relationship I'm trying to build. Or I'll find myself replaying worries as I walk home, instead of either enjoying the walk, thinking about something fun, or thinking about something useful. Or, as Eliezer notes, procrastinating with a process that is less fun as well as less productive. These automatic behaviors serve neither my day-to-day nor my larger scale goals; and automatic action-patterns like these seem to be more common than is coherent action toward (local or any other) purpose.

One possible line of argument:

If, as you imply, you find it compelling that being a danger to the rest of humankind is something to avoid, then presumably you should also find it compelling that reducing other such dangers is something to seek out.

And it's pretty clear that projects that require more than a day's effort to show results will never be undertaken by a system with no goals larger than day-to-day optimization.

It seems to follow from there that if there exist dangers to humankind that require more than a day's effort to measurably reduce, it's a good idea to have goals larger than day-to-day optimization.

Maybe?

While it's true that epistemic rationality, and accurate ideas are useful for everything, it's also true that really wanting something (having Something to protect, if you like) can give a huge boost to one's willingness to face unpleasant facts, and to one's epistemic rationality more generally.

While of course teaching others may be exactly the greatest utilitarian impact available, whenever someone's first and only move after coming to a new belief is to preach it, I suspect they're wireheading status or knowing-powerful-secrets (on top of my suspicion if they're seeking payment). If they were to actually test their ideas in practice, those ideas become more trustworthy - at the very least, bugs will be fixed, and their promotion will be more realistic and persuasive (if their ideas actually held up).

For example, some guru who sells a way of being happy and successful, may be happy and successful only because he gets the thrill of speaking in front of thousands of rapt listeners. If you take away his lecturing gig, what do his "highly effective habits" really earn him?