Desire is the direction, rationality is the magnitude

I like your write-up, very clear and accessible. You certainly have a gift for popularization, not just research. A rare combination.

I would just note upfront that

Reasoning well has little to do with what you're reasoning towards.

and

Rationality of this kind is not about changing where you're going, it's about changing how far you can go.

are white lies, as you well know. It's not unusual in the process of reasoning of how to best achieve your goal to find that the goal itself shifts or evaporates.

"How to best serve God" may result in deconversion.

"How to make my relationship with partner a happy one" may result in discovering that they are a narcissistic little shit I should run away from. Or that both of us should find other partners.

"How to help my neighborhood out of poverty" might become "How to make the most money" in order to donate as much as possible.

This goal-evaporation danger is rarely as extreme, but it is ubiquitous. Every goalpost shifts when you optimize your shot hard enough. Your analogy

Your deepest desires are not a burden, but a compass

is very apt: following it strictly helps you reach your destination, but does not mean your destination is there or has what you expected. Or won't get you killed in the process.

In this essay you talk about Instrumental Rationality as if it were separate from Epistemic. It is not. The dangers of good reasoning ought to be noted upfront, the way MoR!Harry did to Draco, only more so. Hopefully you already plan to talk about it in one of your remaining three essays.

(And, trust me, brains have found a whole lot of the bad ones. What do you expect, when you run programs that screwed themselves into existence on computers made of meat?)

Fairly flippant for what's supposed to be a primer.

The rationality of Rationality: AI to Zombies isn't about using cold logic to choose what to care about. Reasoning well has little to do with what you're reasoning towards. If your goal is to enjoy life to the fullest and love without restraint, then better reasoning (while hot or cold, while rushed or relaxed) will help you do so. But if your goal is to annihilate as many puppies as possible, then this-kind-of-rationality will also help you annihilate more puppies.

This seems like bad salesmanship. There's no need to say things like "rationality is compatible with puppy annihilation" in a paragraph that's supposed to be reassuring people rationality is not immoral or hostile to human emotions. Instead, I would say you should to the opposite, and tell people that if they really love someone, that means that it is actually a rational thing to want to make sacrifices for them, because if you weigh everything up their happiness matters more to you than their own.

So, whereas the message of this post can be summarized as "rationality is neutral", I think the post's message should be closer to "rationality is good - it lets you do things like love EVEN BETTER".

(Unfortunately, this usage of the word "rationality" does not match the colloquial usage. I wish we had a better word for the study of how to improve one's reasoning in all its forms that didn't also evoke images of people sacrificing their emotions on the altar of cold logic. But alas, that ship has sailed.)

In academia that study is called "decision science".

Over time I'm becoming more skeptical about this blanket way of thinking where we consider a single mode of thinking the ultimate way of thinking.

I can be in a mode where I think about what I'm going to say in a reflective way before I say it. I can also be in a mode where I speak without such filtering.

Sometimes, when the stakes are incredibly high and we have time available, we deploy the machinery of logic

In academia there debate about unconscious vs. conscious thought and what's better. While the results are mixed it doesn't seem to be clear that having time available means that the time is usually well spent with using the machinery of logic.

Sometimes it feels to me like using that machinery is about not trusting your brain to do it's work unconsciously in the background. Like the brain doesn't work when it's not micromanaged.

Many think that these aspiring rationalists are attempting some sort of dark ritual to sacrifice emotion once and for all, while failing to notice that the emotions they wish to sacrifice are the very things which give them their humanity.

It's not as if no member of this community goes down that paths. Many PUA's use rejection therapy in a way to numb certain emotions. Similar things pop up from time to time.

Nicely put for an introduction, but of course things are in reality not as clear-cut, "rationality" changing the direction and "desire" the magnitude.

1. Rationality can make you realize some contradictions between your desires, and force you to change them. It can also make you realize that what you truly desire isn't what you thought you desired. Or it can make you desire whole new things, that you didn't believe to be possible initially.

2. Desire will affect the magnitude because it'll affect how much effort you put in your endeavor. With things like akrasia and procastination around, if you don't have strong desire to do something, you are much likely to do it, especially if there is an initial cost. That's what Eliezer calls "something to protect".

Of course those two are mostly positive feedback between rationality and desire, but there also be can negative feedbacks between the two, usually due to human imperfections.

I object to the concept that there's no such thing as hot reasoning. It's basically everything I do. I reason because I care.

Still thinking in vectors :)