Not for the Sake of Pleasure Alone

Most people say they wouldn't choose the pleasure machine

Possibly because the word "machine" is sneaking in connotations that lead to the observed conclusion: we picture something like a morphine pump, or something perhaps only slightly less primitive.

What if we interpret "machine" to mean "a very large computer running a polis under a fun-theoretically optimal set of rules" and "hook up your brain" to mean "upload"?

Both you and Eliezer seem to be replying to this argument:

• People only intrinsically desire pleasure.

• An FAI should maximize whatever people intrinsically desire.

• Therefore, an FAI should maximize pleasure.

I am convinced that this argument fails for the reasons you cite. But who is making that argument? Is this supposed to be the best argument for hedonistic utilitarianism?

I wonder how much taking these facts into account helps. The error that gets people round up to simplistic goals such as "maximize pleasure" could just be replayed at a more sophisticated level, where they'd say "maximize neural correlates of wanting" or something like that, and move to the next simplest thing that their current understanding of neuroscience doesn't authoritatively forbid.

[Most people] begin to realize that even though they usually experience pleasure when they get what they desired, they want more than just pleasure. They also want to visit Costa Rica and have good sex and help their loved ones succeed.

Actually, they claim to also want those other things. Your post strongly implies that this claim is justified, when it seems much more plausible to me to just assume they instead also want a dopamine hit. In other words, to properly wirehead them, Nozick's machine would not just stimulate one part (liking), but two (+wanting) or three (+learning).

So I don't really see the point of this article. If I take it at face-value (FAI won't just optimize the liking response), then it's true, but obvious. However, the subtext seems to be that FAI will have to care about a whole bunch of values and desires (like travels, sex, companionship, status and so on), but that doesn't follow from your arguments.

It seems to me that you are forcing your conclusions in a certain direction that you so far haven't justified. Maybe that's just because this is an early post and you'll still get to the real meat (I assume that is the case and will wait), but I'm uncomfortable with the way it looks right now.

Hey Lukeprog, thanks for your article.

I take it you have read the new book "Pleasures of the Brain" by Kringelbach & Berridge? I've got the book here but haven't yet had the time/urge to read it bookend to bookend. From what I've glimpsed while superficially thumbing through it however, it's basically your article in book-format. Although I believe I remember that they also give "learning" the same attention as they give to liking and wanting, as one of your last quotations hints at.

It's quite a fascinating thought, that the "virtue" of curiosity which some people display is simply because they get a major kick out of learning new things - as I suspect most people here do.

Anyway, I've never quite bought the experience machine argument for two reasons:

1) As we probably all know: what people say they WOULD DO is next to worthless. Look at the hypnotic pull of World of Warcraft. It's easy to resist for me, but there may very well be possible virtual realities, that strike my tastes in such an irresistible manner that my willpower will be powerless and I'd prefer the virtual reality over this one. I may feel quite guilty about it, but it may be so heroin-like and/or pleasurable, that there is simply no escape for me. It's phenomenally easy to bullshit ourselves into thinking we would do what other people and what we ourselves would expect from us, but realistically speaking we're all suckers for temptation instead of kings of willpower. I'd say that if the experience machine existed and was easily obtainable, then our streets would be deserted.

2) For the sake of argument let's assume purely on speculation, that the human ability of imagination has brought with it the rise of a kind of psychological fail-safe against losing oneself in pleasurable daydreams - since the ability to vividly imagine things and lose oneself in daydreams may obviously be an evolutionary counter-adaptive feature of the human brain. If we accept this speculative premise, it may be reasonable to expect, that our brain always quickly runs a query about whether the origin of our pleasure has real/tangible or "imaginative" reasons, and makes us value the former more/differently.

(I won't be nit-picky about the words "real" and "imaginary" here, of cause all feelings are caused by neurological and real phenomena. I trust you catch my drift and understand in which way I use these two words here).

The human brain can tell the difference between fiction and non-fiction very well - it would be rather bothersome to mistake an image of a tiger or even our pure imagination for the real deal. And for similar adaptive reasons it seems, we are very aware of whether our hedonic pleasures have tangible or imaginary origins. So when I ask "do you want to experience "imaginary" pleasures for the rest of your existence even though nothing real or tangible will ever be the source of your pleasure" then of cause people will reject it.

We humans can obviously indulge in imaginary pleasures a lot, but we usually all crave at least some "reality-bound" pleasures as well. So asking someone to hook up to the experience machine is like depriving them of a "distinct form" of pleasure that may absolutely depend on being real.

So I think the experience machine thought experiment isn't really telling us, whether or not pleasure is the only thing that people desire. Instead, it may simple be the case, that it is indeed an important precondition for a wide range of pleasure to be based on reality - and being entirely deprived of this reality-correlating pleasure may in a way feel like being deprived of water - horrible. People may be right to reject the experience machine, but that tells us nothing about whether or not pleasure is the only thing they desire. So I think the experience machine is a rather poor argument to make in order to illustrate, that there are other things besides pleasure that people care about - the research you cite however is much clearer and quite unmistakable in this respect.

On another note, I think we're missing the most interesting question here: Indeed, there seem to be other things besides pleasure that people care about... but SHOULD they? Should I ever want something that doesn't make me happy, just because my genes wired me up to value it? Why should I want something if it doesn't make me directly or indirectly happy or satisfied?

Wanting something without necessarily liking it may be an integral part of how my human psychology operates, and one that cannot simply be tinkered with... but in the light of advances in neuroscience and nanobots, maybe I should opt to rewire/redesign myself as a hedonist from the ground up?! Valuing things without liking them is no fun... so can someone here tell me why I should want to care about something that doesn't make me happy in some shape or form?

PS: Sorry for the long comment, I didn't have time for a short one.

I don't understand what everything after the Nozick's experience machine scenario is for. That is, the rest of the post doesn't seem to support the idea that we intrinsically value things other than pleasure. It tells us why sometimes what we think will give us pleasure is wrong (wanting more of the drug), and why some things give us pleasure even though we didn't know they would (salt when salt-deprived)... but none of this means that pleasure isn't our objective.

Once we know about salt, we'll want it for the pleasure. We would also normally stop wanting something that turns out not to give us pleasure, as I stopped wanting to eat several chocolate bars at a time. It might not work like this in the drug example, though, that's true; but does this mean that there's something about the drug experience we desire besides the pleasure, or are our brains being "fooled" by the dopamine?

I think the latter. There is no aspect wanted, just the wanting itself.... possibly because dopamine is usually associated with - supposed to signal - pleasure?

(I think once anyone got in your Nozick's machine variation, they would change their minds and not want to get out. We think we'd experience loneliness etc. because we currently would: we can "feel" it when we imagine it now. But we wouldn't afterward; we'd never stop being content with the machine once inside. Unless the machine only gives us a certain type of pleasure and we need several. But this is a different subject, not really meant as a refutation... since even knowing this, I'm not sure I'd get in.)

I'd get in Nozick's machine for the wireheading. I figure it's likely enough that I'm in a simulation anyway, and his simulation can be better than my current one. I figure I'm atypical though.

Also related: Are Wireheads Happy?, You cannot be mistaken about (not) wanting to wirehead.

This sounds to me like a word game. It depends on what the initial intention for 'pleasure' is. If you say the device gives 'maximal pleasure' meaning to point to a cloud of good-stuffs and then you later use a more precise meaning for pleasure that is an incomplete model of the good-stuffs, you are then talking of different things.

The meaningful thought experiment for me is whether I would use a box that maximized pleasure\wanting\desire\happiness\whatever-is-going-on-at-the-best-moments-of-life while completely separating me as an actor or participant from the rest of the universe as I currently know it. In that sense of the experiment, you aren't allowed to say 'no' because of how you might feel after the machine is turned on, because then the machine is by definition failing. The argument has to be made that the current pre-machine-you does not want to become the post-machine-you, even while the post-machine-you thinks the choice was obvious.

We have an experience machine at our disposal if we could figure out the API. Ever have a lucid dream?

Indeed, it appears that mammals can come to want something that they have never before experienced pleasure when getting.

Duh - otherwise sexual reproduction in mammals would be a non-starter.

While the article shows with neat scientific references that it is possible to want something that we don't end up liking, this is irrelevant to the problem of value in ethics, or in AI. You could as well say without any scientific studies that a child may want to put their hand in the fire and end up not liking the experience. It is well possible to want something by mistake. But it is not possible to like something by mistake, as far as I know. Differently from wanting, "liking" is valuable in itself.

Wanting is a bad thing according to Epicurus, for example. Consider the Greek concepts of akrasia and ataraxia. Wanting has instrumental value for motivating us, though, but it may feel bad.

Examine Yudkowsky, with his theory of Coherent Extrapolated Volition. He saw only the variance and not what is common to it (his stated goal doesn't have an abstract constancy such as feeling good, but instead it is "fulfilling every persons' extrapolated volition", or their wish in case they had unlimited intelligence). This is a smarter version of preference utilitarianism. However, since people's basic condition is essentially the same, there needn't be this variance. It doesn't matter if people like different flavors of ice cream, they all want it to taste good.

On the other hand, standard utilitarianism seems to see only the constancy and not take account of the variance, and for this reason it is criticized. It is like giving strawberry ice cream to everyone because Bentham thought that it was the best. Some people may hate strawberry ice cream and want it chocolate instead, and they criticize standard utilitarianism and may go to an ethical nihilism because of flavor disputes. What does this translate into in terms of feelings? One could prefer love, rough sex, solitude, company, insight, meaningfulness, flow, pleasure, etc. to different extents, and value different sensory inputs differently, especially if one is an alien or of another species.

Ethics is real and objective in abstraction and subjective in mental interpretation of content. In other words, it's like an equation or algorithm with a free variable, which is the subject's interpretation of feelings (which is just noise in the data), and and objective evaluation of it in an axis of good or bad, which corresponds to real moral value.

The free variable doesn't mean that ethics is not objective. It is actually a noise in the data caused by a chain of events that is longer than should be considered. If we looked only at the hardware algorithm (or "molecular signature", as David Pearce calls it) of good and bad feelings, we might see it as completely objective, but in humans there is a complex labyrinth between a given sensory stimulus and the output to this hardware algorithm of good and bad, such that a same stimulus may produce a different result for different organisms, because first they need to pass through a different labyrinth.

This is the reason for some variance in preference of feelings (affective preference? experiential preference?), or as also could be said, preference in tastes. Some people like strawberry and some prefer chocolate, but the end result in terms of good feelings is similarly valuable.

Since sentient experience seems to be all that matters, instead of, say, rocks, and in sentience the quality of the experience seems to be what matters, then to achieve value (quality of experience) there's still a variable which is the variation in people's tastes. This variation is not in the value itself (that is, feeling better) but on the particular tastes that are linked to it for each person. The value is still constant despite this variance (they may have different taste buds, but presented with the right stimuli they all lead to feeling good or feeling bad).

When people say "pleasure" in this context, they usually just mean to refer to whatever it is that the human brain likes internally. To then say that people don't just like pleasure - since they also like happiness/bliss - or whatever - seems to be rather missing the point.

As for the rather distinct claim that people want external success, not just internal hedonistic pleasure signals - that seems to depend on the person under consideration. Few want their pleasure to end with them being fired, and running out of drug money (though we do still have our addicts, even today) - but apart from that, some people really do seem to want the pleasure - or whatever you want to call it. For example, consider David Pearce.

Disclaimer: I don't think that maximizing pleasure is an FAI solution; however, I didn't find your arguments against it convincing.

With regards to the experience machine, further study has found that people's responses are generally due to status quo bias; a more recent study found that a slight majority of people would prefer to remain in the simulation.

With regards to the distinction between desire and pleasure: well, yes, but you seem to be just assuming that our desires are what ought to be satisfied/maximized instead of pleasure; I would assume that many of the people you're talking with take pleasure to be a terminal value (or, at least, I used to, and I'm generalizing from one example and all that).

Intuitively, this feels true. I rarely do things based on how much pleasure they bring me. Some of my decisions are indirectly linked to future pleasure, or other people's pleasure, i.e. choosing to work 6 am shifts instead of sleeping in because then I won't be poor, or doing things I don't really want to but said I would do because other people are relying on me and their plans will be messed up if I don't do it, and I wouldn't want them to do that to me... Actually, when I think about it, an awful lot of my actions have more to do with other people's pleasure than with my own, something which the pleasure machine doesn't fulfill. In fact, I would worry that a pleasure machine would distract me from helping others.

I feel like I am missing something. You separated pleasure from wanting.

I don't see how this backs up your point though. Unless the machine offered is a desire-fulfilling machine and not a pleasure machine.

If it is a pleasure machine, giving pleasure regardless of the state of wanting, why would we turn it down? You said we usually want more than just pleasure, because getting what we want doesn't always give us pleasure. If wanting and pleasure are different, then of course this makes sense.

But saying we want more than pleasure? That doesn't make sense. You seem to be confusing the two terms your article sets out to separate. We want pleasure, we are just not always sure how to get it. We know we have desires, so why try to fill them, and that doesn't always work. But remember, pleasure and wanting are separate.

If a machine knew what would give us pleasure, and gave us pleasure instead of what we "wanted," then we would always be getting pleasure. Even when we don't get what we want.

Unless your machine doesn't work as advertised, of course.

Oddly, those two arguments end up cancelling out for me.

You explained how pleasure from our natural environment "caps out" past a certain threshold - I can't eat infinity sugar and derive infinity pleasure. So, obviously, my instinctive evaluation is that if I get wire-headed, I'll eventually get sick of it and want something else!

Equally, your research shows that we're not always perfect at evaluating what I want. Therefore, I'd have an instinctive aversion to wire-heading because I might have guessed wrong, and it's obviously very difficult to escape once committed.

Therefore, I conclude, the reason I am adverse to wire-heading is because it violates the structures of my ancestral environment. My initial rejection of it based on the Nozick's experience machine was irrational, and I might actually really enjoy wire-heading.

Given I've always been confused why I reject wire-heading, this is actually rather a nice enlightening post, albeit doing the opposite of what you intended :)

BBC article: "The science of addiction: Do you always like the things you want?", Edmonds.

We act not for the sake of pleasure alone. We cannot solve the Friendly AI problem just by programming an AI to maximize pleasure.

IAWYC, but would like to hear more about why you think the last sentence is supported by the previous sentence. I don't see an easy argument from "X is a terminal value for many people" to "X should be promoted by the FAI." Are you supposing a sort of idealized desire fulfilment view about value? That's fine--it's a sensible enough view. I just wouldn't have thought it so obvious that it would be a good idea to go around invisibly assuming it.

Is their meaningful data on thalamic stimulators with erodic side-effects? (See entry #1 here: http://www.cracked.com/blog/5-disturbing-ways-the-human-body-will-evolve-in-the-future/ ). Cracked gives the addictive potential of an accidental orgasm switch plenty of attention while citing just two examples (it's a comedy site after all), but have other cases been studied? I'm not convinced this couldn't be done intentionally with current models of the brain.

Most people say they wouldn't choose the pleasure machine.

Well that was easy. In my (limited) experience most people making such claims do not really anticipate being pleasure-maximized, and thus can claim to want this without problems. It's only "real" ways of maximizing pleasure that they care about, so you need to find a "real" counterexample.

That said, I have less experience with such people than you, I guess, so I may be atypical in this regard.

Going for what you "want" is merely going for what you like the thought of. To like the thought of something is to like something (in this case the "something" that you like is the thought of something; a thought is also something). This means that wanting cannot happen unless there is liking that creates the wanting. So, of wanting and liking, liking is the only thing that can ever independently make us make any choice we make. Wanting which is not entirely contingent on liking never makes us make any decisions, because there is no such thing as wanting which is not entirely contingent on liking.

Suppose you can save mankind, but only by taking a drug that makes you forget that you have saved mankind, and also makes you suffer horribly for two minutes and then kills you. The fact that you can reasonably choose to take such a drug may seem to suggest that you can make a choice which you know will lead to a situation that you know you will not like being in. But there too, you actually just go for what you like: you like the thought of saving mankind, so you do whatever action seems associated with that thought. You may intellectually understand that you will suffer and feel no pleasure from the very moment after your decision is made, but this is hard for your subconscious to fully believe if you at the thought of that future actually feel pleasure (or at least less pain than you feel at the thought of the alternative), so your subconscious continues assuming that what you like thinking about is what will create situations that you will like. And the subconscious may be the one making the decision for you, even if it feels like you are making a conscious decision. So your decision may be a function exclusively of what you like, not by what you "want but don't like".

To merely like the thought of doing something can be motivating enough, and this is what makes so many people overeat, smoke, drink, take drugs, skip doing physical excercise, et cetera. After the point when you know you have already eaten enough, you couldn't want to eat more unless you in some sense liked the thought of eating more. Our wanting something always implies an expectation of a future which we at least like thinking of. Wanting may sometimes appear to point in a different direction than liking does, but wanting is always merely liking the thought of something (more than one likes the thought of the alternatives).

Going for what you "want" (that is, going for what you merely like the thought of having) may be a very dumb and extremely short-sighted way of going for what you like, but it's still a way of going for what you like.

I notice that I'm a bit confused, especially when reading, "programming a machine superintelligence to maximize pleasure." What would this mean?

It also seems like some arguments are going on in the comments about the definition of "like", "pleasure", "desire" etc. I'm tempted ask everyone to pull out the taboo game on these words here.

A helpful direction I see this article pointing toward, though, is how we personally evaluate an AI's behavior. Of course, by no means does an AI have to mimic human internal workings 100%, so taking the way we DO work, how can we use that knowledge to construct an AI that interacts with with us in good way?

I don't know what "good way" means here though. That's an excellent question/point I got from the article though.

I have a different argument for why people don't want to wirehead themselves. It's easiest to think about if you imagine a self-modifying AI. Let's suppose that this particular AI is a paperclip maximizer. Its utility function is equal to the number of paperclips in the world.

Now, this AI is self-modifying, so of course it can wirehead itself. In other words it can replace its utility function of "U = number of paperclips" with a different utility function "U = VERYLARGENUMBER". Suppose the AI is considering whether to make this change.

How does the AI decide whether to wirehead itself? It asks: what will be my future utility, according to my current utility function, of me replacing my utility function with VERYLARGENUMBER? In other words, how many paperclips will there be in the future, if I wirehead myself?

And of course it sees that the utility from this course of action is terrible, and it doesn't wirehead itself. (And it goes on to convert the world into paperclips.)

You might imagine an AI that was written differently, so that it evaluated its utility of wireheading itself according to its hypothetical future utility function. I would argue that this would be a bug, and that if you write an AI with this bug, you have not really written an AI at all; you've just written a very expensive while(true) loop.

I've thought of a bit of intuition here, maybe someone will benefit by it or be kind enough to critique it;

Say you took two (sufficiently identical) copies of that person C1 and C2, and exposed C1 to the wirehead situation (by plugging them in) and showed C2 what was happening to C1.

It seems likely that C1 would want to remain in the situation and C2 would want to remove C1 from the wirehead device. This seems to be the case even if the wirehead machine doesn't raise dopamine levels very much and thus the user does not become dependent on it.

However, even in the pleasure maximizing scenario you have a range of possible futures; any future where abolitionism is carried out is viable. Of course, a pure pleasure maximizer would probably force the most resource efficient way to induce pleasure for the largest number of people for the longest amount of time, so wireheading is a likely outcome.

That said, it seems like some careful balancing of the desires of C1 and C2 would allow for a mutually acceptable outcome that they would both move to, an abolitionist future with plenty of novelty (a fun place to live, in other words).

This is sort of what I currently envision the CEV would do, run a simulated partial-individual, C1, on a broad range of possible futures and detect whether distance is occurring by testing against a second copy of the initial state of C1, C2. Distance would be present if C2 is put off by or does not comprehend the desires of C1 even though C1 prefers its current state to most other possible states. To resolve the distance issue, CEV would then select the set of highest utility futures that C1 is run on and chooses the one that appears most preferable to C2 (which represents C1's initial state). Assuming you have a metric for distance, CEV could then reject or accept this contingent upon whether distance falls under a certain threshold.

Does this seem sensible?

How about a machine that maximizes your own concept of pleasure and makes you believe that it is probably not a machine simulation (or thinks that machine simulation is an irrelevant argument)?

Then they will blast you and the pleasure machine into deep space at near light-speed so that you could never be interfered with. Would you let them do this for you?

Most people say they wouldn't choose the pleasure machine.

Well, no wonder. The way the hypothetical scenario is presented evokes a giant array of ways it could go wrong.

What if the pleasure machine doesn't work? What if it fails after a month? What if it works for 100 years, followed by 1,000 years of loneliness and suffering?

Staying on Earth sounds a lot safer.

Suppose the person you are asking is religious. Will they be forfeiting an eternity in heaven by opting for passive pleasure in this life? They would say no, yet they are ultimately after eternal pleasure (heaven).

If you want to be fair to the person to whom you're talking, propose a pleasure machine they can activate at their convenience, and deactivate at any time. In addition, phrase it so that the person will remain healthy as long as they're in the machine, and they'll receive a minimum-wage income for spent time.

I suspect that, with these much safer sounding provisions, most people would opt to have access to the machine rather than not, and would eventually use it for 100% of the time.

Religious people may still not, if they fear losing access to heaven.

Personal example: The greatest feeling of bliss I have experienced is dozing off in a naturally doped up state after highly satisfying sex. This state is utterly passive, but so thoroughly pleasant that I could see myself opting for an eternity in it. I would still, however, only opt for this if I knew it could not end in suffering, e.g. by becoming immune to pleasant states of mind in the end

The pleasure machine argument is flawed for a number of reasons:

1) It assumes that, despite having never been inside the pleasure machine, but having lots of experience of the world outside of it, you could make an unbiased decision about whether to enter the pleasure machine or not. It's like asking someone if he would move all his money from a bank he knows a lot about to a bank he knows basically nothing about and that is merely claimed to make him richer than his current bank. I'm sure that if someone would build a machine that, after I stepped into it, actually made me continually very, very much happier than I've ever been, it would have the same effect on me as very heavy paradise drugs have on people: I would absolutely want to stay inside the machine for as long as I could. For eternity, if possible. I'm not saying it would be a wise decision to step into the pleasure machine, (see point 2, 3 and 4 below), but after having stepped into it, I would probably want to stay there for as long as I could. Just as this choice might be considered biased because my experience of the pleasure machine can be said to have made me "unhealthily addicted" to the machine, you are just as biased in the other direction if you have never been inside of it. It seems most people have only a very vague idea about how wonderful it would actually feel to be continually super happy, and this makes them draw unfair conclusions when faced with the pleasure machine argument.

2) We know that "pleasure machines" either don't yet exist at all, or, if they exist, have so far always seemed to come with too high a prize in the long run (for example, we are told that drugs tend to create more pain than pleasure in the long run). This makes us spontaneously tend to feel sceptical about the whole idea that the pleasure machine suggested in the thought experiment would actually give its user a net pleasure increase in the long run. This skepticism may never reach our conscious mind, it may stay in our subconscious, but nevertheless it affects our attitude toward the concept of a "pleasure machine". The concept of a pleasure machine that actually increases pleasure in the long run is a concept that never gets a fair chance to convince us on its own merits before we subconsciously dismiss it because we know that if someone claimed to have built such a machine in the real world, it would most likely be a false claim.

3) Extreme happiness tends to make us lose control of our actions. Giving up control of our actions usually decreases our chances to maximize our pleasure in the long run, so this further contributes to make the pleasure machine argument unfair.

4) If all human beings stepped into pleasure machines and never got out of them, there would be no more development (by humans): If instead some or all humans continue to further the tech development and further expand in universe, it will be possible to build even better pleasure machines later on, than the pleasure machine in the thought experiment. There will always be a trade-off between "cashing in" (by using some time and other resources to build and stay in "pleasure machines") and postponing pleasure for the sake of tech development and expansion in order to make possible even greater future pleasure. The most pleasure-maximizing such trade-off may very well be one that doesn't include any long stays in pleasure machines for the nearest 100 years or so. (At some point, we should "cash in" and enjoy huge amounts of pleasure at the expense of further tech development and expansion in universum, but that point may be in a very distant future.)

-

In the last decade, neuroscience has confirmed what intuition could only suggest: that we desire more than pleasure. We act not for the sake of pleasure alone. We cannot solve the Friendly AI problem just by programming an AI to maximize pleasure.

Either this conclusion contradicts the whole point of the article, or I don't understand what is meant by the various terms "desire", "want", "pleasure", etc. If pleasure is "that which we like", then yes we can solve FAI by programming an AI to maximize pleasure.

The mistake you (lukeprog, but also eliezer) are apparently making worries me very much. It is irrelevant what we desire or want, as is what we act for. The only thing that is relevant is that which we like. Tell me, if the experience machine gave you that which you like ("pleasure" or "complex fun" or whatchamacallit), would you hook up to it? Surely you would have to be superstitious to refuse!

While humans may not be maximizing pleasure they are certainly maximizing some utility function which can be characterized. Human concerns can then be programmed to optimize this function in your FAI.

this is my very first comment. So this will result of immorality of a person. I'm a Christian and I want to point out only sexual pleasure. It promotes to indulgence to worldly things - machines and lust. Since in RC, masturbation is prohibited then this things will occur together.

Is it just me or is odd that Eliezer calls himself a math prodigy?

EY has been in the AI world claiming to be an AI research for more then 10 years. He also claims that he needs to spend a year studying math. Now most people even non math prodigies can get a B.S., M.S., and most of a PhD in the same amount of time.

EY also states his knowledge of math is tiny compared to a mathematician. How can this be 10 years studying math and a "prodigy" so math should be easy since he is as he claims so good at explaining it and he has a good sense for right answers half the time.

Now let me get this straight very little math knowledge compared to a mathematician. A math prodigy so its easy. Great taste in right answers. Yet after 10 years in a math field (AI) he doesn't know that much math.

Most math people I know right math papers especially when they are good at it. Not EY in fact he doesn't nor has he ever published a high level math paper. Anyone else confused?

EY please explain this puzzling issue to me.