The Sword of Good

I don't advise you to, and anyway who'd translate a Russian fanfic into English and put it online?

"Did you learn this from an unbiased source?"

I'm pretty sure it was in Tolkien's notes.

"Suppose you're the prime minister of a parliamentary republic, and the neighboring country is ruled by hereditary nobility that mostly hate each other, and wars between the barons ruin a lot of the land and kill a lot of the peasants. You, being a genius engineer, have figured out a way to control people, but it requires they wear the device for an extended period of time, the effects are obvious, and they can take it off before the process is complete if they feel like it."

Except that's exactly what Sauron DIDN'T do. Mordor was not a parliamentary republic; more like a military dictatorship with semi-mindless orc drones enforcing Sauron's commands over his human subjects. The monarchs who were given the rings - however just or unjust their rule might have been, and however flawed the notion of monarchy as a political system - were lied to about what the rings did, and the rings' effects were very subtle at first.

Its also worth noting that the human kings didn't become any kinder or more democratic in their sensibilities once they fell under Sauron's influence. The Witch King was still a king, and a much more murderous one than he was in life. Unleashing barrow-wrights on a partially civilian population, torturing Gollum for information, and stabbing an innocent (if possibly misguided) hobbit when he didn't have to are all things that the Witch King did in person.

"This hereditary nobility situation is obviously not going to fix itself- and you figure that the easiest way to fix it is to corrupt all the nobility, playing on their hatred of each other to get them to wear the devices long enough for them to work, and then have them give you power in a bloodless coup. As a bonus, you now have fanatically loyal assassins / spec ops forces, and an eternity of servitude seems like a fitting punishment for their misconduct as rulers."

In other words, the only way to improve the world is to become just as bad as the people currently running it? The best solution to dictatorships is to make slaves of your own, and for all eternity no less?

I think you're going out of your way to defend Brin's essay rather than actually using your own moral judgement. You can easily say that the "good guys" in Lord of the Rings weren't all that good, but Sauron was very obviously worse.

The way children get so few civil rights they're used as excuses for removing rights from adults.

I am aware of the poor state of affairs re: children's rights, but I'm not sure what you're getting at by citing consequences for adults. Can you elaborate?

As far as schools, do you mean something about the specific way that we have schooling set up currently (and do you include universities in that?) or do you mean more generally?

I had in mind the education of children in school, as done in, I think, all of the developed world and a lot of the rest, and critiques like this one.

Universities may also have their faults, but not on the scale of misdeeds being considered, and, anyway, the people in them chose to go there.

User:betterthanwell, I presume.

You missed the step where you assert that most people assert it is wrong to eat sentient animals, which is what would create the inconsistency, were most people to assert that.

Specifically, most people assert that animals are sentient; yet most people are not vegetarians, even though eating meat is no longer necessary for survival. There is an inconsistency between these positions.

No there isn't. It implies that they violate another norm that you value but it is not inconsistent.

I don't see why. For clarity, since we probably agree it's wrong, imagine you're making the same argument for cannibalism instead. One person says, "I'm fine with eating and farming humans but if I get to know one first, doing it would make me feel bad." Another says, "Screw that, I'll eat anyone, even if I know them and their children!"

The second person is more morally consistent and also more callous. Even if there's no difference in the way they live their lives, trying to end the holocaust of humans for food would be easier in a world full of the first type than the second.

Just as I would prefer the opposite of rule of law when the law is uniformly terrible, I prefer the opposite of moral consistency when a morality is terrible.

I've never understood this argument. I have a visceral reaction against surgery (even the sight of blood can set me off); I certainly couldn't stand to be in the same room in which surgery was being performed. Does this mean that for consistency I'm required to morally oppose surgery?

Though it wasn't explicitly said, it was heavily implied that either choice would be for a potentially infinite duration. This is a world of fantasy and prophecy, after all: I got the impression that the current social order was stable, and given that there was magic (not psychic ability but magic) it's also fair to assume that the scientific method doesn't work (not that this makes any sense, but you have to suspend that disbelief for magic to work [gnomes are still allowed to build complex machines, they're just not allowed to build useful machines]).

The way I interpreted it was that he had a choice between the status quo for 1000 years, or and unknown change, guided by good intentions, for 1000 years.

Besides, the Big Bad was Marty Stu. How could I not side with him?

(Another great work, Yudkowski - you really should send one of these to Asimov's SciFi)

As the sword killed 90% of those who touched it, Vhazhar could have, upon reading the records, discovered that the sword only allowed to survive those who help increase the CEV for sentient life (and thus slaughtering a ridiculous number of Cohen-esque "heroes").

The modified people can be quite a bit smarter than you are too, so long as you can see their minds and modify them. Groves et al managed to mostly control the Manhattan project despite dozens of its scientists being smarter than any of their supervisors and many having communist sympathies. If he actually shared their earlier memories and could look inside their heads... There's a limit to control, you still won't control an adversarial super intelligence this way, but a friendly human who appreciates your need for power over them? I bet they can have a >50 IQ point advantage, maybe even >70. Schoolteachers control children who have 70 IQ points on them with the help of institutions.

Eliezer, I'm with you that a properly designed mind will be great, but mere uploads will still be much more awesome than normal humans on fast forward.

Without hacking on how your mind fundamentally works, it seems pretty likely that being software would allow a better interface with other software than mouse, keyboard and display does now. Hacking on just the interface would (it seems to me) lead to improvements in mental capability beyond mere speed. This sounds like mind hacking to me (software enhancing a software mind will likely lead to blurry edges around which part we call "the mind"), and seems pretty safe.

Some (pretty safe*) cognitive enhancements:

• Unmodified humans using larger displays are better at many tasks than humans using small displays (somewhat fluffy pdf research). It'll be pretty surprising if being software doesn't allow a better visual interface than a 30" screen.
• Unmodified humans who can touch-type spend less time and attention on the mechanics of human machine interface and can be more productive (no research close to hand). Who thinks that uploaded humans are not going to be able to figure better interfaces than virtual keyboards?
• Argument maps improve critical thinking, but the interfaces are currently clumsy enough to discourage use (lots of clicking and dragging). Who thinks that being software won't provide a better way to quickly generate argument maps?
• In front of a computer loaded up with my keyboard shortcuts and browser plugins I have easy access to very fast lookup on various web reference sites. At the moment the lookup delay is still long enough that short term memory management (stack overflow after a mere 7±2 pushes) is a problem (when I need a reference I push my current task onto a mental stack; it takes time and attention to pop that task when the reference has been found). Who thinks I couldn't be smarter with a reference interface better than a keyboard?

All of which is just to say that I don't think you've tried very hard to think of safe self-modifications. I'm pretty confident that you could come up with more, and better, and safer than I have.

* Where "pretty safe" means "safe enough to propose to the LW community, but not safe enough to try before submitting for public ridicule"

You can make volunteers out of your own copies. As long as the modified people aren't too smart, it's safe keep them in a sandbox and look through the theoretical work they produce on overdrive.

*blinks* I understand your "oh hell no" reaction to self modification and "use the speedup to buy extra time to solve FAI" suggestion.

However, I don't quite understand why you think "attempted upgrading of other" is all that much better. If you get that one wrong in a "result is super smart but insane (or, more precisely, very sane but with the goal architecture all screwed up) doesn't one end up with the same potential paths to disaster? At that point, if nothing else, what would stop the target from then going down the self modification path?

What, you mean try to self-modify? Oh hell no. Human brain not designed for that

Perhaps you mean to say that we're not particularly trustworthy in our choices of what we modify ourselves to do or prefer?

Human brains, after all, are most exquisitely designed for modifying themselves, and can do it quite autonomously. They're just not very good at predicting the broader implications of those modifications, or at finding the right things to modify.

Perhaps an "anti arms race" would be a more accurate notion. ie, in once sense, waiting for the mathematics of FAI to be solved would be preferable. Would be safer to get to a point that we can mathematically ensure that the thing will be well behaved.

On the other hand, while waiting, how many will suffer and die irretrievably? If the cost for waiting was much smaller, then the answer of "wait for the math and construct the FAI rather than trying to patchwork update a spaghetti coded human mind" would be, to me, the clearly preferable choice.

Even given avoiding self modification, massive speedup would still correspond to significant amount of power. We already know how easily humans... change... with power. And when sped up, obviously people not sped up would seem different, "lesser"... helping to reinforce the "I am above them" sense. One might try to solve this by figuring out how to self modify enough to, well, not to that. But self modification itself being a starting point for, if one does not do it absolutely perfectly, potential disaster, well...

Anyways, so your suggestion would basically be "only use the power to, well, defend against the power" rather than use it to actually try to fix some of the annoying little problems in the world (like... death and and and and and... ?)

He's going to be the emperor. He could implement Parliament, he could create jury trials. He could even put Dolf and Selena on trial for their crimes.

It's interesting that Hirou holds the world accountable to his own moral code, which assumes power corrupts. Then, at the last moment, he grants absolute power to Vhazhar. So in the middle of choosing to use our world's morality, which is built upon centuries of learning to doubt human nature, in the middle of that - Vhazhar's good intentions are so good that they justify granting him absolute power. Lesson not learned.

If Vhazhar has the option of editing the nasty bits out of reality and then stepping down from power, I'd help him. If he must personally become a ruler for all eternity, I'd kill him, then smash the goddamn device, then try to somehow ensure that future aspiring Dark Lords also get killed in time.

I assume that the sword tests the correspondence of person's intentions (plan) to their preference. If the sword uses a static concept of preference that comes with the sword instead, why would Vhazhar be interested in sword's standard of preference? Thus, given that the Vhazhar's plan involves control over the fabric of the World, the plan must be sound and result in correct installation of Vhazhar's preference in the rules of the world. This excludes the technical worries about the failure modes of human mind in wielding too much power (which is how I initially interpreted "personal control" -- as a recipe for failure modes).

I'm not sure what it means for the other people's preferences (and specifically mine). I can't exclude the possibility that it's worse than the do-nothing option, but it doesn't seem obviously so either, given psychological unity of humans. From what I know, on the spot I'd favor Vhazhar's personal preference, if the better alternative is unlikely, given that this choice instantly wards off existential risk and lack of progress.

Characters who FAIL ETHICS FOREVER can still be entertaining. For example, plenty of villains clearly have little regard for ethics. Authors who FAIL ETHICS FOREVER are usually less desirable. For example, Terry Goodkind. I've rarely felt personally insulted by a work of fiction, but, well, Naked Empire somehow managed to contain the purest, unadulterated essence of Ethics Fail I've ever encountered - it even managed to contradict the explicit moral lessons of the earlier books in the series!

I agree that screaming at characters that they FAIL ETHICS FOREVER can interrupt enjoyment of a story, but it is far worse to never realize that their actions are, in fact, contemptible.

I didn't get that from the story. All those fantasy books he's read, and he only now ponders whether something is good just because the author labeled it "Good"?

I think you're being a little optimistic here in thinking your skepticism is at all general.

Why was Norman Spinrad's _The Iron Dream_ so critically well-received and still read? (If you haven't read it, it's much like Eliezer's story except without the sane hero.) Because it demonstrated that most readers weren't critical, that they'd been reading fantasy stories for literally decades without cottoning onto how well the same stories justified genocide and fascism!

also he seems to be quite an excellent moral philosopher - someone who actually perceives morality.

I didn't get that from the story. All those fantasy books he's read

Not Hirou, Vhazhar. For some reason, even as a very young child facing religious indoctrination, I couldn't quite accept that Abraham had made the right choice in trying to sacrifice Isaac upon God's command. That was one of my first moral breaks with Judaism. The Lord of Dark is - almost necessarily - actually visualizing situations and reacting to them as if seen, rather than processing words however the people around him expect to process them; there's no other way he could reject the values of his society to that extent, and even then, the amount of convergence he exhibits with our own civilization is implausible barring extremely optimistic assumptions about (a) the amount of absolute coherence (b) our own society's intelligence and (c) the Lord of Dark's intelligence; but of course the story wouldn't have worked otherwise.

I guess I'm just surprised to see an allegory from you in which someone solves Friendliness by applying thirty seconds of his at-best-slightly-above-average moral intuition.

Vhazhar's been working on it for some unknown number of years, having successfully realized that sucking the life from worms may be icky but doesn't actually hurt any sentient beings. (Though I wasn't assuming Vhazhar was ancient, he very well could be, and that would make a number of things more plausible, really.) Hirou has a whole civilization behind him and just needed to wake up and actually think.

I've had similar experiences myself, and try not to have them again. Evidence builds up behind a wall of denial, and when the dam breaks the flood is loosed.

Value is associated with states of knowledge (about consequences), not with precise outcomes. What you are saying is that uncertainty confers low value, and so is generally less preferable than (well-known) status quo. This is not generally correct.

Agree with the morals -> enjoy reading crude stereotypes of your moral opponents. Get enough enjoyment from that and the story's a net positive even if it has no other redeeming qualities.

So Chuunin Exam Day, then? I've never read it, but I've heard of it.

Considering that I was able to identify the author and possibly the exact fic from the information that the morality was being heavily lambasted, may I suggest that readers noticing nonlampshaded evil doesn't actually happen all that often? TV Tropes is good at noticing Moral Dissonance, but literally nowhere else that I've ever heard of. It took a critic on the order of David Brin to point out that Aragorn wasn't democratically elected.

update: haven't done it nine times yet. It's still brilliant though.

Let me amplify: By "double spaces" I mean two spaces after a period, not double spaces between lines.

I think I disagree. The arbitrary and unfeeling processes of the universe can probably be outperformed by anything with a shred of empathy and intellect. You'd just want to be really, really careful, and try to create something better than yourself to hand off control to.

Yeah, of course. Not to mention the actual direct perpetrators of the evils themselves.

It's a convention of fantasy and science fiction that there can exist sentient races which are, by their very nature, inimical to mankind, and can therefore be justifiably killed on sight

And by the way, I'm willing to buy that. Hirou's sin is that he didn't actually buy it, as in, pay for the conclusion, if you see what I mean.

I thought "end of the world" meant "end of the world as it was"... ie, "by the time he wakes back up, the spell will have completed and our friendly neighborhood dark lord will have already started fixing the place up"