Savulescu: "Genetically enhance humanity or face extinction"

Christian, FAI is hard because it doesn't necessarily provide any feedback. There lots of are scenarios where the first failed FAI just kills us all.

That's why I am advocating IA as a way to up the odds of the human race producing FAI before uFAI.

But really, the more I think about it, the more I think that we would do better to avoid AGI all together, and build brain emulations. Editing the mental states of ems and watching the results will provide feedback, and will allow us to "look before we jump".

I find it interesting that you make a distinction between people making choices that are not in their own best interests and choices not in line with their own stated goals.

Generally what I had in mind there is selecting concrete goals without regard for likely consequences, or with incorrect weighting due to, e.g. extreme hyperbolic discounting, or being cognitively impaired. In other words, when someone's expectations about a stated goal are wrong and the actual outcome will be something they personally consider undesirable.

If they really do know what they're getting into and are okay with it, then fine, not my problem.

If it helps, I also have no problem with someone valuing self-determination so highly that they'd rather suffer severe negative consequences than be deprived of choice, since in that case interfering would lead to an outcome they'd like even less, which misses the entire point. I strongly doubt that applies to more than a tiny minority of people, though.

There's a world of difference between informing someone of a perceived danger that you suspect they are unaware of (a cliff they're about to walk off) and forcibly preventing them from taking some action once they have been made aware of your concerns.

Actually making someone aware of a danger they're approaching is often easier said than done. People have a habit of disregarding things they don't want to listen to. What's that Douglas Adams quote? Something like, "Humans are remarkable among species both for having the ability to learn from others' mistakes, and for their consistent disinclination to do so."

Incidentally I don't believe there is a general moral obligation to warn someone away from taking an action that you believe may harm them. It may be morally praiseworthy to go out of your way to warn them but it is not 'evil' to refrain from doing so in my opinion.

I strenuously disagree that inaction is ever morally neutral. Given an opportunity to intervene, choosing to do nothing is still a choice to allow the situation to continue. Passivity is no excuse to dodge moral responsibility for one's choices.

I begin to suspect that may be the root of our actual disagreement here.

In general this is in a different category from the kinds of issues we've been talking about (forcing 'help' on someone who doesn't want it).

It's a completely different issue, actually.

...but there's a huge amount of overlap. Simply by virtue of living in society, almost any choice an individual makes imposes some sort of externality on others, positive or negative. The externalities may be tiny, or diffuse, but still there.

Tying back to the "helping people against their will" issue, for instance: Consider an otherwise successful individual, who one day has an emotional collapse after a romantic relationship fails, goes out and gets extremely drunk. Upon returning home, in a fit of rage, he destroys and throws out a variety of items that were gifts from the ex-lover. Badly hung over, he doesn't show up to work the next day and is fired from his job. He eventually finds a new, lower-paid and less skilled, job, but is now unable to make mortgage payments and loses his house.

On the surface, his actions have harmed only himself. However, consider what the society as a whole has lost: 1) The economic value of his work for the period where he was unemployed 2) The greater economic value of a skilled, better-paid worker 3) The wealth represented by the destroyed gifts 4) The transaction costs and economic inefficiency resulting from the foreclosure, job search, &c. 5) The value of any other economic activity he would have participated in, had these events not occurred. [0]

A very serious loss? Not really. Certainly, it would be extremely dubious to say the least for some authority to intervene. But the loss remains, and imposes a very real, if small, negative impact on every other individual.

Now, multiply the essence of that scenario by countless individuals; the cumulative foolishness of the masses, reckless and irrational, the costs of their mistakes borne by everyone alike. Justification for micromanaging everyone's lives? No--if only because that doesn't generally work out very well. Yet, lacking a solution doesn't make the problem any less real.

So, to return to the original discussion, with a hypothetical medical procedure to make people smarter and more sensible, or whatever; if it would reduce the losses from minor foolishness, then not forcing people to accept it is equivalent to forcing people to continue paying the costs incurred by those mistakes.

Not to say I wouldn't also be suspicious of such a proposition, but don't pretend that opposing the idea is free. It's not, so long as we're all sharing this society.

Maybe you're happy to pay the costs of allowing other people to make mistakes, but I'm not. It may very well be that the alternatives are worse, but that doesn't make the situation any more pleasant.

Where this issue does get a little complicated is when the negative externalities you are trying to prevent cannot be eliminated without forcing something upon others. The current vaccination debate is an example - there should be no problem allowing people to refuse vaccines if they only harmed themselves but they may pose risks to the very old and the very young (who cannot be vaccinated for medical reasons) through their choices. In theory you could resolve this dilemma by denying access to public spaces for people who refused to be vaccinated but there are obvious practical implementation difficulties with that approach.

Complicated? That's clear as day. People can either accept the vaccine or find another society to live in. Freeloading off of everyone else and objectively endangering those who are truly unable to participate is irresponsible, intolerable, reckless idiocy of staggering proportion.

[0] One might be tempted to argue that many of these aren't really a loss, because someone else will derive value from selling the house, the destroyed items will increase demand for items of that type, &c. This is the mistake of treating wealth as zero-sum, isomorphic to the Broken Window Fallacy, wherein the whole economy takes a net loss even though some individuals may profit.

Just out of curiosity, are you for or against the Friendly AI project? I tend to think that it might go against the expressed beforehand will of a lot of people, who would rather watch Simpsons and have sex than have their lives radically transformed by some oversized toaster.

There no abolishion of torture anyone in the US.

That's not the case. The United States signed and ratified the United Nations Convention against Torture and Other Cruel, Inhuman or Degrading Treatment or Punishment.

Right - but it has been banned elsewhere:

http://en.wikipedia.org/wiki/European_Convention_on_Human_Rights#Article_3_-_torture

According to:

http://en.wikipedia.org/wiki/Moral_realism

...this involves attributing truth and falsity to moral statements - whereas it seems more realistic to say that moral truth has a subjective component.

However, the idea of moral progress does not mean there is "one true morality".

It just means that some moralities are better than others. The moral landscape could have many peaks - not just one.

I see no problem with the concept of moral progress. The idea that all moralities are of equal merit seems like totally inexcusable cultural relativism to me. Politically correct, perhaps - but also silly.

Morality is about how best to behave. We have a whole bunch of theory from evolutionary biology that relates to that issue - saying what goals organisms have - which actions are most likely to attain them - how individual goals conflict with goals that are seen acceptable to society - and so on. Some of it will be a reflection of historical accidents - while other parts of it will be shared with most human cultures - and most alien races.

Um, what? Why did we evolve to fear death? I suspect I'm missing something here.

That the way we fear death has the effect of maximizing our measure, but not the number of branches we are in, is perhaps a puzzle. See also http://lesswrong.com/lw/19d/the_anthropic_trilemma/14r8 starting at "But a problem with that".

You're converting an "is" to an "ought" there with no explanation, or else I don't know in what sense you're using "should".

I'm pointing out a possible position one might take, not one that I agree with myself. See http://lesswrong.com/lw/196/boredom_vs_scope_insensitivity/14jn

Yes, and I had a discussion with him last year at http://old.nabble.com/language%2C-cloning-and-thought-experiments-tt22185985.html#a22189232 (Thanks for the reminder.)

If you follow the above link, you'll see that I actually took a position that's opposite of my position here: I said that people mostly don't care about measure. I think the lesson here is that A) I have a very bad memory :-) and B) I don't know how to formalize human preferences.

But, suppose that what you really care about is what you're about to experience next, rather than what the absolute value of the complex number that premultiplies that experience is?

I think this is a more reasonable alternative to "caring about measure" (as opposed to "caring about the number of branches" which is mainly what I was arguing against in my first reply to you in this thread). I'm not sure what I can say about this that might be new to you. I guess I can point out that this is not something that "evolution would do" if mind copying technology were available, but that's another "clue" that I'm not sure what to make of.