Remaining human

The overwhelming majority of humans do, in fact, want to be human, much to the annoyance of the transhumanist minority.

Putting that aside, though, I see what I think is a different problem, though perhaps I'm overgeneralizing from my own motivations. Human endeavors tend to feel worthwhile because they are a challenge. Assuming that we do develop the ability to self-modify, recursively improving our physical and mental abilities, I worry that things will seem better and better -- until suddenly they seem worse. When anyone can be as strong or as fast as they want, there will be no such thing as athletics or martial arts. When anyone can be as smart as they want, there will be no such thing as puzzles or games. Etc. When all the hard questions have been answered, what will be left, except wireheading?

Humans suck.

Testing confirms this. Though sometimes they break your heart and kill you.

I agree! That's at least part of why my concern is pedagogical. Unless your plan is more of just run for the stars and kill everyone who didn't come along.

Agree 100%. Upvoted.

A loss of empathy with "regular people". My friend, for instance, loves the opera Tosca where the ultimate plight and trial comes down to the lead soprano, Tosca, committing suicide despite certain damnation.

The rational mind (of the temperature often suggested here) might have a difficult time mirroring that sort of conundrum, however it's been used to talk about and explore the topics of depression and sacrifice for just over a century now.

So if you take part of your job to be an educator of those still under the compulsion of strange mythology, you probably will have a hard time communicating with them if you absolve all connection to that mythology.

if you embrace rationality as your raison d'etre, you almost inevitably conclude that human beings must be exterminated.

Let me try to guess your reasoning. If you have "I want to be rational" as one of your terminal values, you will decide that your human brain is a mere hindrance, and so you will turn yourself into a rational robot. But since we are talking about human values, it should be noted that smelling flowers, love, and having family are also among your terminal values. So this robot would still enjoy smelling flowers, love, and having family - after all, if you value doing something, you wouldn't want to stop liking it, because if you didn't like it you would stop doing it.

But then, because rational agents always get stuck in genocidal cul-de-sacs, this robot who still feels love is overwhelmed by the need to kill all humans, leading to the extermination of the human race.

Since I probably wasn't close at all, maybe you could explain?

Civilizations based on the irrational revelations of prophets have proven themselves to be more successful and appealing over a longer period of time than any rationalist society to date.

Depends what you mean by "based on" (and to a lesser extent "prophet" if you want to argue about North Korea, China and the old USSR). People seem to prefer, for example, America over Iran as a place to live.

As we speak, the vast majority of humans being born are not adopting, and never will adopt, a rational belief system in place of religion. Rationalists are quite literally a dying breed.

Hang on, that's a bit of a non-sequiter. Just because rationalists won't become a majority within the current generational cohort doesn't mean we're shrinking in number, or even in proportion. I haven't seen the statistics for other countries (where coercion and violence likely play some role in religious matters) but in Western nations non religious people have been increasing in number. In my own nation we're seeing the priesthood age and shrink (implying that the proportion of "religious" people committed enough to make it their career is falling) and in my city Adelaide, the "City of Churches", quite a few have been converted into shops and nightclubs.

It's fun to fantasize about transcending the human condition via [contextual proxy for rationality], but I'm skeptical in the extreme that such a thing will happen -- at least in a way that is not repugnant to most current value systems.

What would the least repugnant possible future look like? -- keeping in mind that all the details of such a future would have to actually hold together? (Since "least repugnant possible", taken literally, would mean the details would hold together by coincidence, consider instead a future that were, say, one-in-a-billion for its non-repugnance.) If bringing about the least repugnant future you could were your only goal, what would you do -- what actions would you take?

When I imagine those actions, they resemble rationality, including trying to develop formal methods to understand as best you can which parts of the world are value systems which deserve to be taken into account for purposes of defining repugnance, how to avoid missing or persistently disregarding any value systems that deserved to be taken into account, how to take those value systems into account even where they seem to contradict each other, and how to avoid missing or persistently disregarding major implications of those value systems; as well as being very careful not to gloss over flaws in your formal methods or overall approach -- especially foundational problems like Gödelian undecidability, unsystematic use of reflection, bounded rationality, and definition of slippery concepts like "repugnant" --, in case the flaws point to a better alternative.

What do the actions of someone whose only goal was to bring about the least repugnant future they could resemble when you imagine them?

(How much repugnantness is there in the "default"/"normal"/"if only it could be normal" future you imagine? Is that amount of repugnantness the amount you take for granted -- do you assume that no substantially less repugnant future is achievable, and do you assume that to safely achieve a future at least roughly that non-repugnant would not generally require doing anything unprecedented? How repugnant would a typical future be in which humanity had preventably gone extinct because of irrationality, how repugnant would a future be in which humanity had gone extinct because of a preventable choice for repugnance-insensitive rationality, and how relatively likely would these extinctions be under the two conditions of global irrationality and global repugnance-insensitive rationality? Would a person who cared about the repugnance of the future, when choosing between advocacy of reason and of unreason, try to think over effects like this and try to take them into account, given that the repugnance of the future was at stake?)

I feel like this is close to the heart of a lot of concerns here: really it's a restatement of the Friendly AI problem, no?

The back door seems to always be that rationality is "winning" and therefore if you find yourself getting caught up in an unpleasant loop, you stop and reexamine. So we should just be on the lookout for what's happy and joyful and right—

But I fear there's a Catch 22 there in that the more on the lookout you are, the further you wander from a place where you can really experience these things.

I want to disagree that "post-Enlightenment civilization [is] a historical bubble" because I think civilization today is at least partially stable (maybe less so in the US than elsewhere). I, of course, can't be to certain without some wildly dictatorial world policy experiments, but curing diseases and supporting general human rights seem like positive "superhuman" steps that could stably exist.