r/HPMOR on heroic responsibility

Well yes, but what exactly are you going to substitute for individual judgement of what actions to take? If you decide that following code X, and enforcing that others do so as well, even when it seems like breaking the code this one time would give better results, is the best course of action (perhaps for reasons of precommitment or average-case utility or what-have-you)... then oops, you seem to have made a decision in accordance with your best judgement, there.

There is an obvious parallel between HJPEV and AGI: he can do (and does) stuff no (other) human can even conceive of doing.

How do you know if your values and goals are constructive or destructive? It all comes down to the same hard question of FAI (well, FHHI, friendly human hero intelligence, in this case): are your values CEV-aligned? So, the first thing Harry should do is stop running around saving people and derive the CEV :)

One can, of course, argue that, as a human hero, HJPEV has the right CEV built-in and should just run around implementing it. However, given that even his friends and allies disagree (Dumbledore is a deathist, McGonagall is a disciplinarian, Hermione thinks they are too young, Weasley twins just want to have fun), and he gets the most help from the anti-hero Quirrell, this point of view is hard to defend.

The situation is much worse outside of fiction, where buggy or limited wetware constantly leads would-be heroes (Lenin, Castro, Lincoln or even Hitler) to commit or cause suffering or destruction of the same group of people they aspired to help.

So the first question a would-be hero should ask herself is whether she is prepared to live with the consequences of her actions if they backfire. (And if the answer is yes, she is clearly a villain.)

That's the Übermensch morality of dictators and totalitarian regimes. The problem is that every dictator thinks of themselves as a benevolent dictator, but it turns out that they are often mistaken.

Even when there are multiple aspiring dictators, all with essentially benevolent values, conflict on who should rule can be very much destructive.

Of course this presupposes consequentialism. In deontologism morality is typically viewed as a social contract where each party has well defined responsabilities.

I don't think it's an accident that violent totalitarian ideologies tend to be consequentialist: these innocent heretics/Jews/bourgeois stand between us and our utopia. Kill'em all!

I would say that he was making at least the argument that "this level of responsibility is something you should adopt if you want to be a hero", and probably the more general argument that "people should adopt this attitude toward responsibility.

Thinking back, I'm almost certain that the way I was introduced to Less Wrong was through HPMOR. The long gaps between updates encouraged me to start reading the sequences, even.

It's probably the most effective recruiting tool for rationality in the world as of now.

I think many of them are secretly (not secretly) LW readers :P

At least for myself , I first heard of Eliezer via the HPMOR TV Tropes page. There's a good chance I would have read the sequences sooner or later even if I hadn't (my brother found them independently and recommended them), but it definitely helped.

And I wouldn't say I was an idiot before, but twenty minutes of conversation with myself from a couple years ago might change my mind. And of course it's hard to tell how much of the difference is LW's influence and how much is just a matter of being older and wiser.

He is maximizing his utility part-function: people should not be harmed. If I know of a situation such that: Someone will be harmed if I do nothing; They will be harmed equally if I tell the authorities; If I intercede with my allies, they will be harmed less.

The first two actions are equal. The third one is better. The ability to determine the actual consequences of future actions is magical. Fortunately for Harry, he has access to magic. I expect Harry to develop a procedure where he makes several plans, checks for a note indicating which plan he should use and any changes he should make, executes the plan, and then time-turns a note back to himself indicating which plan worked and any retroactive changes to it, then assists himself if needed.

It sounds like you're assuming that all Morality-speak is a ploy to get others to follow our will, and a way of taking stock of others' comparable attempts to influence us. I'd like to suggest that there are other ways to construe Morality-speak like 'hero' and 'virtue:'

1. One could simply treat one's values and Morality-speak as interchangeable. Why use one language for the second- and third-person, and a totally different language for the first-person? It's confusing, makes generalizations of your desires difficult, and greatly weakens your rhetorical power (since you seem to be subscribing to a double standard, if only terminologically). If your values are such that you don't actually give a privileged status to your own welfare as opposed to others, then this linguistic shift conceals an important symmetry.

2. Morality-speak might be an idealization of one's optimized values, one's values once they've been brought into optimal reflective equilibrium. What you currently care about might not be what you think you ought care about, even if you ultimately define 'oughtness' in terms of the aforementioned preferences. Otherwise it would be incoherent to lament how much one presently likes something, to wish for a reform to one's secondary concerns that would bring them into greater internal harmony.

Actually, he seems a little worse than this, in that any failure makes him a bad boy, which is an impossible standard.

Two issues: First, he's probably exaggerating at least a little. Second, it's clear that he's adopting the standard in question for utilitarian reasons; he happens to be especially motivated by falling short of some lofty ideal. For most people an at least somewhat less exacting standard would probably be desirable; but since standards are just heuristics for winning, the question of how demanding to be is one for empirical psychology.

Decius is right that there aren't really spoilers, but I would argue that your time would be better spent reading HP:MOR than the discussion.

You won't gain knowledge about a twist in the plot by reading the article; the comments contain descriptions of events in the story, but I didn't see anything that I would call a spoiler.