Thou Art Godshatter

Godshatter? What I may or may not have shat out of my divine anus is of no concern of yours.

Signed, God (big bearded guy in the sky)

Eliezer, you wrote:

"Or else what would we do with the future? What would we do with the billion galaxies in the night sky? Fill them with maximally efficient replicators? Should our descendants deliberately obsess about maximizing their inclusive genetic fitness, regarding all else only as a means to that end?"

Won't our descendants who do have genes or code that causes them to maximize their genetic fitness come to dominate the billions of galaxies. How can there be any other stable long term equilibrium in a universe in which many lifeforms have the ability to choose their own utility functions?

For everyone who hasn't read A Fire Upon The Deep (Vinge): Godshatter is the term he uses for a superintelligence ramming data and thought patterns into a human brain.

James, that is a problem: http://www.nickbostrom.com/fut/evolution.html

And "Thou art God" comes from Stranger in a Strange Land.

The human brain also has the complicating factor of memes and what might be called "inclusive memetic fitness." If you hypothesize that human behavior is influenced by two different sets of selfish replicators, we could certainly have an equilibrium in which natural selection doesn't produce behavior that maximizes the fitness of only one of them. (Incidentally, does it seem to anyone else that humans are "designed" to have unwanted pregnancies?)

(Also, "The Mote In God's Eye" isn't necessarily the best example. The Moties aren't specifically motivated by the desire to maximize inclusive genetic fitness, but their biology requires them to reproduce as much as possible whether their brains think they should or not. "Protector" might be a better choice, as it describes highly intelligent individuals that have maximizing the reproductive success of their offspring as their primary conscious motivation.)

Why is Eliezer so obsessed with the "high-falutin'" expression?

"Obsess all day about inclusive genetic fitness? Where's the fun in that?" Might not our descendants evolve to consider it fun?

I agree with James Miller on the unstable equilibrium. I figure I'll be dead by then though.

I don't find it particular comforting that we are made of many small shards of desire, rather than a single unified desire. We like the fact that the universe is at least locally dominated by creatures who like what we like. This will always be true, that the majority will see a world of creatures mostly like themselves.

I don't find it particular comforting that we are made of many small shards of desire, rather than a single unified desire.

It seems like this is a large part of what makes us eudaemonic agents, in Bostrom's terminology. However, the most admired people are frequently those who display more of a deep commitment than usual to one or a few passions.

"Thou Art Godshatter"! Finally, a name for my Christian/Prog/Electronica combo!

We have no instinctive revulsion of condoms or oral sex. Our brains, those supreme reproductive organs, don't perform a check for reproductive efficacy before granting us sexual pleasure.

If we could barely arrange to have enough sex to cause 2 pregnancies per lifetime, then we would have a revulsion of condoms, oral sex, etc.

If for example we spent almost all of each year alone, and once a year men and women would meet on a sandy beach (or just offshore) and have sex, and that would be the only chance for women to get +regnant until the next year, men would compete intensely for the women who seem to be fertile, and the losers and the apparently-infertile women could console each other. When you only get one chance you don't waste it.

But humans use sex for social signalling. A man who publicly explained that he intended never to have sex except when he was trying to cause a pregnancy, might find himself at a disadvantage among some women.

But human brains clearly can imagine these links in protein. So when the Evolution Fairy made humans, why did It bother with any motivation except inclusive genetic fitness?

How are we supposed to tell? Like, we need food for energy and for building materials. When you don't get enough energy you feel that. When you don't get enough of some building material you feel that too, and you might learn to recognise that particular feeling. I've read about africans who specificly get hungry for meat, who identify the specific feeling of protein deficiency. How many others are there? Each individual amino acid? Each individual vitamin? I think instead you get rewarded by the feeling you get when there's enough of everything and no glut of something that causes problems. And it's up to your behavioral reinforcement system to notice foods that give you that feeling.

So humans love the taste of sugar and fat, and we love our sons and daughters. We seek social status, and sex. We sing and dance and play. We learn for the love of learning.

There maybe wasn't much chance to overdose on sugar or far in the old days. Or salt, if you were inland. Give it a few more generations and we might do that sort of thing less. Are Pima amerindians more susceptible to the amount of sugar we eat, or do they eat more because they can? I expect that's been tested but I don't know the answer myself. Maybe they haven't been exposed to sugar for as long, so they don't have the defenses against it we do.

You can learn strategies to play chess. Wouldn't it be nice if evolution had provided us with a chess-fitness optimiser? Instead of thinking about strategies, you just make the right move. But that requires the problem of winning at chess to be already solved.

It might very well turn out that in our future people who successfully reproduce will tend to be people who want to, and who figure out how to. There's some of that now, asnd maybe those people wind up with fitter children than the ones who become parents through carelessness. That might be hard to measure just now. Maybe the big majority of the children come from parents who give little thought to how they'll take care of their children, and the ones who're cautious wind up under-reproducing as a result. In the long run the evolutionary process will note what's worked whether we manage to measure it or not.

If we could barely arrange to have enough sex to cause 2 pregnancies per lifetime, then we would have a revulsion of condoms, oral sex, etc.

If that were true in the ancestral environment, and we had access to contraception in the AE, yes. I doubt a person who now found themselves in this situation would develop this revulsion.

I've read about africans who specificly get hungry for meat, who identify the specific feeling of protein deficiency.

[anecdote] Is this surprising? I've always been able to tell whether or not I need proteins/carbohydrates/fat (usually acting accordingly), and kind of assumed other people had the same sense. Of course, this is a fairly weak sense; calorie-rich food still tastes good when I've had too much, and it still takes willpower to stop eating it. [/anecdote]

Eliezer: poetic and informative. I like it.

@Elizier, you are slowly changing your point of view and are on a path to rethink old thoughts. Save yourself some time and go read the Principia Cybernetica Web. Only after that you will be able to tread on new ground.

@Nick Tarleton, yes - avoiding a dystopia of non-eudomatic agents is a challenge.

As a chicken is a way for an egg to create another egg, I would like to 'tell my genes to jump in a lake', as Steven Pinker puts it, but considering so many of my preferences are in sync with my genes, I have the feeling they are very good at getting me to rationalize their preferences. I don't think there's intrinsic meaning in anything, but when I see connections, or patterns, in music or jokes or anything, that I haven't noticed before, I find that meaningful, pleasurable in a way my genes can't understand. But my love for my kids, and the meaning it gives me, clearly the gene gremlins are at work

You know, you're getting repetitive. What does this post add to all the other related posts? "Evolutions" are stupid and slow. Okay. But I would guess that many people here want to know your thoughts about AI. I do.

@Tiiba, my paper on friendly AI theory should provide an answer to your question.

"I doubt a person who now found themselves in this situation would develop this revulsion."

By about the second generation a lot would. They would mostly be descended from people who hadn't used them. There is a minority that has a revulsion for condoms now. The idea of giving up practically your only change to have children, deliberately, would start seeming strange when everybody in the world had parents who hadn't done it. Cultures change faster when that happens.

"@Tiiba, my paper on friendly AI theory should provide an answer to your question."

I don't see any connection between my question and your answer. At least one of us is confused.

I base friendliness (universally) on the mechanism of natural selection and claim in short "that is good what increases fitness". You can find more on my blog at http://jame5.com

You don't understand. I'm asking ELIEZER what he is thinking. His homepage says that he has some fresh ideas about AI that are not yet published, yet he continues to write about evolution, rehashing the same idea every day. That is what I said. I don't even know what question you're answering.

"Being a thousand shards of desire isn't always fun, but at least it's not boring."

I like that. I have a feeling Lord Gautama would have liked it too.

I will venture to say that Eliezer's habit (this isn't the first instance) of teasing out the same subject again and again from slightly different angles is highly illuminating for me, at least. (And, I suspect, for him as well... though that's conjecture).

I'm a bit slower than your average Overcoming Bias lurker, it would seem from the level of discourse here. Sometimes I think I barely grasp what everyone is even talking about, though I try to read the background links people provide. But I'm an intelligent person in general, and I have an interest in the concepts and methods Robin, Eliezer and the rest hash out in this space. You could argue that all humans do, whether they realize it or not. Either Eliezer added something new to this post, or reading post after post on this has finally hammered the point through my brain. But this evening I feel like I finally get it, and by "it" I mean merely the most basic concepts.... I grasped them abstractly right away, but more important for overcoming one's own operating biases is really getting it in a way that will allow one to spot one's faulty reasoning in the past and the future.

@Tiiba, trust me - I am quite certain that I do, but this is not the right forum - PM me if you want to continue off this blog.

@Elizier, you are slowly changing your point of view and are on a path to rethink old thoughts. I didn't notice any evidence of that. He said that he had greatly changed his view in the past, but that was before he started blogging here. What have you seen since then that makes you think that?

@TGGP: This forum really is not the right place to get into details. It would not be fair towards Eliezer and that I posted something at all is an embarrassing revelation in regards to my intellectual vanity. Mea culpa.

Consider the Laestadians (look them up in Wikipedia if you haven't heard of them). They tend to have lots of children; one TV program some years ago mentioned that families with 10 children are common among them.

Unless a lot of those children abandon (or at least modify) their parents' faith, the future belongs to them and similar groups.

Religion can be a powerful fitness maximizer.

@ Tiiba # 1: Without wishing to second-guess Eliezer, I'd suggest that his prolonged examination of the buggy, ad-hoc character of human intelligence may be intended to preface a discussion AI, its goals and methods. After all, the contrast with human intelliegence could be illuminating.

That missing word: "of".

The blind idiot god's single monomaniacal goal splintered into a thousand shards of desire.

This would explain why our formalised moral systems are either hideously complicated, or fail to capture important parts of our morality... We just have far more urges wants and needs than we realise.

As many comments have suggested, now that evolution has produced creatures that can consciously seek goals, and also has instilled in some of these creatures, to some extent, the goal of bearing and raising children, all that evolution needs to do is to reinforce this desire, and in time it will manage to produce a conscious fitness maximizer.

Abandoning biology is not a way to avoid this result, since biology is not the problem, but reproduction and its historical consequences. Leaving behind biology could even speed up the process dramatically.

Maybe the alien god, despite being blind, slow, and stupid, will get its way in the end. In the distant future, intelligent fitness maximizers might laugh at the ridiculous idea, now long extinct, that it is better to have a random collection of unrelated desires for no reason except historical accident, than to seek the unified goal of fitness. After all, they'll say, obviously nothing is worth seeking except fitness. And besides, seeking anything else is self-destructive.

If this comes to pass, what would be wrong with it? If the wrongness is only from our point of view, why should our point of view have more validity than theirs? Since we have reason to think they would be find their goal boring or miserable, we have no obvious reason to be horrified at the idea of this society.

Just as an aside, fitness maximizers usually have to accept a finite population size in a finite biome with a finite carrying capacity. There's the possible goal of expanding into the galaxy and neighboring galaxies, but in the short run we have a finite carrying capacity.

And a fitness maximizer that is too successful has to accept it needs to preserve a lot of diversity in its gene pool or else face problems that would essentially reduce carrying capacity.

A conscious fitness maximizer at some point must realise that it survives by maintaining its numbers in a diverse population, rather than maximizing the frequency of its genes.

@ Unknown: Well, one reason why our point of view is more valid than their's is that we exist and they don't.

In addition, it is probably worth stressing that inclusive fitness is not, strictly speaking, the goal of anything at all. Goals only make sense relative to intentions, values and so forth - the usual accoutrements of mentality. These are all things that we humans (and perhaps some other creatures) possess, but which evolution, and our genes, do not. No minds, you see. Despite appearances.

This said, there might be something to be said for engineering or breeding descendants whose drives are more harmonious than our own. For instance, they might be happier. Still, there's no particular reason why we should choose to make inclusive fitness the goal of all their striving, as opposed to something else.

@J Thomas, the trick lies in ensuring continued co-existence.

@Stefan: I enjoyed your book and was fascinated by your FAI perspective, but your comments here could be read as overly self-promoting, which would be counterproductive. An evil, paranoid maniac might even imagine you write comments to maximize how many links to your blog you can cram onto a page! Maybe limiting the links to yourself might curb such insanity in your audience.

@Recovering irrationalist, good points, thank you - I just wanted to save time and space by linking to relevant stuff on my blog without repeating myself over and over. My apologies for overdoing it. I guess I feel like talking to a wall or being deliberately ignored due to the lack of feedback. I shall curb my enthusiasm and let things take its course. You know where to find me.

[anecdote] Is this surprising? I've always been able to tell whether or not I need proteins/carbohydrates/fat (usually acting accordingly)....

sorry, guys, this wisdom-of-the-body stuff hasn't held up that well. i've given the link below for a lengthy but thorough account of studies that were done on rats, for the two or fewer people here who might be interested. while there is some evidence for behavioral changes based on mineral deficiency, it's extremely complicated and the changes in the animal's behavior are not that "accurate" (in the sense that the animal truly seeks out the depleted nutrient). bottom line, "In many experimental situations, animals do not choose an optimal diet. This is especially the case for omnivores." i hope this wasn't entirely off-topic; just wanted to clean out this little rafter....

http://ajpregu.physiology.org/cgi/content/full/279/2/R357

This would explain why our formalised moral systems are either hideously complicated, or fail to capture important parts of our morality... We just have far more urges wants and needs than we realise.

Congratulations to Stuart Armstrong on nailing my hidden subtext.

(Albeit even the hideously complicated moral systems still don't capture a fraction of our morality.)

@Tiiba: You seem to think I can just blurt out my AI ideas. I've tried that. It doesn't work.

Having watched other AIfolk "explaining" their ideas, I know very well how to convince someone that you've just conveyed an AI theory - just pick a word like "complexity", "emergence", or "Bayesian" and call it the secret of the universe; or draw a big diagram full of connected boxes with suggestive names drawn from cognitive science. Unlike these other AIfolk, I've actually learned a little about how intelligence works, and so I know this would be unhelpful and dishonest.

Bayes is the secret of the universe, but believing this statement will not help you.

If you seek enlightenment upon this matter of AI, then I must ask whether you've read existing textbooks such as Machine Learning by Mitchell, Probabilistic Reasoning in Intelligent Systems, Artificial Intelligence: A Modern Approach (2nd Ed), and Elements of Statistical Learning. Recommended in that order.

@Unknown: I am horrified by the thought of humanity evolving into beings who have no art, have no fun, and don't love one another. There is nothing in the universe that would likewise be horrified, but I am. Morality is subjectively objective: It feels like an unalterable objective fact that love is more important than maximizing inclusive fitness, and the one who feels this way is me. And since I know that goals, no matter how important, need minds to be goals in, I know that morality will never be anything other than subjectively objective.

With all that said, I hope you won't mind if I use objective language to say:

"Evolving into obsessive replicators would be a waste of humanity's potential. They might not mind, just as sociopaths don't mind killing, but I mind. I will avoid such a future with every power of my intelligence."

Brendon, you can't expect a learning system to quickly get an exact solution to a problem in N simultaneous equations. But when improvements result in a sense of well-being, they might tend to gradually zero in on solutions. So for nutrition you need sufficient energy and your body might have pre-programmed goals for repair and growth, and whatever helps meet those targets could provide that sense of well-being that announces something worked.

Simpler than having thousands of individual goals programmed in.

"Being a thousand shards of desire isn't always fun, but at least it's not boring."

I like that. I have a feeling Lord Gautama would have liked it too.

I alway thought the exact opposite, that Lord Gautama had a profound experience that made him relatively indifferent to the thousand shards. Specifically, a full-blown ecstatic or mystical experience is a million of times more pleasurable than any other experience the mystic has had or will have, which I always thought would make one less attached to ordinary pleasures and ordinary reinforcers. Once a religion becomes a popular movement or part of the ruling class's justification, its leaders are tempted to modify it to broaden its appeal, which is how I always thought Buddhism acquired the habit of promising an end to suffering. One of my friends had a profound mystical experience, personally attests to the "a million of times more pleasurable", is very scrupulous and truthful and derives no reputational benefit from having had the experience. (He says that I am practically the only one with whom he has ever discussed his experience in any detail.)

Moreover, it is my hypothesis that being indifferent to the thousand shards is a powerful enhancer of mental and moral clarity in the right conditions. One of these conditions is that the indifference be not so total or so early in onset that it extinguishes curiosity during the person's youth, which of course is just totally pernicious in an environment as rich in true scientific information as our environment is. Another adherent of this hypothesis is academician John Stewart. Mystical experience is quite risky and dangerous; competent supervision is recommended and I would suppose is available at low or no cost to students who show high potential. Another common adverse outcome of mystical experience seem to be to make the person more confident in his beliefs, especially about the moral and political environment, and of course people tend to be too confident about their beliefs already.

if a moderator has time, could he replace my http://users.tpg.com.au/users/jes999/ with http://users.tpg.com.au/users/jes999/EvSpirit.htm

J Thomas : "So for nutrition you need sufficient energy and your body might have pre-programmed goals for repair and growth, and whatever helps meet those targets could provide that sense of well-being that announces something worked."

this sort of system might work for thirst or even carbs and protein but would be pretty bad at things like getting you to eat balanced amounts of vitamins and minerals. for instance, your diet could be vitamin b12 poor for months or maybe longer before you would feel the pinch (your body stores the vitamin pretty well, i'm told), and i doubt you would then start 'craving' vitamin b12 rich foods - particularly because there had never been an appetite-satiety relationship you could have picked up on, even unconsciously.

Some new info re: evolution you might want to consider before taking the gene view of evolution to its logical conclusions:

http://www.springerlink.com/content/qh67113u60887314/ "Although we agree that evolutionary theory is not undergoing a Kuhnian revolution, the incorporation of new data and ideas about hereditary variation, and about the role of development in generating it, is leading to a version of Darwinism that is very different from the gene-centred one that dominated evolutionary thinking in the second half of the twentieth century."

http://www.sciencedaily.com/releases/2003/09/030929054959.htm how new thinking applies to societies

That still doesn't explain why Eliezer has been using the expression "high-falutin'" so much. Is it from some recently read book, perhaps?

Brendon, I find your reasoning plausible. I don't know how true it is. I don't want to give myself pernicious anemia to test it, so I'll settle for saying it looks plausible.

If you have a vitamin deficiency, and you get a dose of the vitamin that makes you somewhat less deficient, will you feel better within a few hours? If so then it might be reinforced. On the other hand, one single experience of nerve poisoning a few hours after eating a particular new food can be enough to establish a lifelong distaste for that food.

Specifically, a full-blown ecstatic or mystical experience is a million of times more pleasurable than any other experience the mystic has had or will have

This seems unlikely - it's far more probable that mystical experiences are highly satisfying rather than so intensely pleasurable.

Richard:

I suppose this counts as threadjacking, but this thread seems about played out, so I'll respond to your response to my off-topic aside.

I'm interested in what you say. I don't think it's necessarily off base. But my little cheeky comment was in reference to the Buddhist concept of anatta, or non-self. That is, Eliezer's insistence that there is no purposeful unifying force behind what we experience as "our" desires reminded me of an analogous teaching of the Buddha. Evolution can be seen as a unifying force, I suppose, since it is the common wellspring of our desires, but as Eliezer is rightly at pains to point out, it is decidedly not purposeful. "A thousand shards of desire" is what we are left with.

One of the key concepts of Buddhist meditation and scholarship is that desires are ultimately independent of the desirer. [Note: I differentiate serious, classical Buddhism, which has a ridiculously large set of founding texts and canonical commentaries, from pop Buddhism. or the selective Western brand of Buddhism which takes the concepts that have appeal for people brought up in a society where the dominant religious traditions are monotheistic and authoritarian (the West, that is) while leaving behind the less sexy teachings which are in fact the core of the practice.] In the first stages of serious meditation, before you achieve any mystical bliss or whatnot, it becomes quite clear that the thoughts and desires that we take for granted as "our own" are in fact caused by specific conditions and fall away when those conditions cease. That's the practice-based observation. The theoretical concept that springs from that is that, in fact, we build our mistaken sense of a unified "I" out of these falsely-apprehended experiences. (I say theoretical because my personal inquiries have not yet fully borne this out... perhaps they will, perhaps not... there are Buddhist scholars and monks who claim to know this to be ontologically true... I have reasons to doubt them, but I also have reasons to believe them... further inquiry is required).

Of course this teaching comes from a time before any understanding of evolutionary theory, and is practiced today by people who, broadly speaking, still don't have any real understanding of such (yours truly included!). I don't want to throw around too much sloppy thinking here, but I will suggest that there may be more than one angle at which to come to an understanding. Both disciplined scientific inquiry and disciplined meditational inquiry are (properly) undertaken with a desire to get at an understanding of reality while systematically eliminating misapprehensions and biases as they arise.

Anyway, all that is not to refute what you said, but to explain my comment.

I will take issue with your positing that the teachings on the end of suffering were added by later theocrats or rulers who wanted to broaden its appeal for the masses. In the oldest texts we have (written down around 2200 BC, after 300 or so years surviving in an oral tradition the fidelity of which has been shown in other contexts to be remarkable), the Buddha teaches again and again about suffering. Several places in the sutras he is quoted as saying, "I teach one thing: suffering and its end." The teaching on the Four Noble Truths (said to be the first teaching he ever gave though admittedly that's pretty hard to ascertain for sure) is the central teaching of the Buddhist canon. Many, many, many of the Buddha's teachings came in for debate, abandonment and wholesale tortion as they spread to various different societies with their own cultural norms and mores and institutions and languages. But the teachings on suffering and its end are the same in Tibet as they are in Sri Lanka as they are in Japan. You might argue that the original teaching was somehow a cynical appeal to the masses (I am very much inclined to say it was not), but it's clearly not a later corruption.

I'm very interested in parallels between the kind of ruthlessly rational inquiry displayed by the thinkers on this blog and that displayed by the early Buddhist, including the Buddha himself. I find myself looking for ways to reconcile the two. Of course, in even admitting that, I'm busting myself! If I have my desired conclusion in mind as I sift through the evidence, I have already forgotten the central teachings of Overcoming Bias! ... I'll press on though, catching myself where I can! ;)

Nit: surely you mean "220 BC," not "2200 BC".

I will take issue with your positing that the teachings on the end of suffering were added by later theocrats or rulers who wanted to broaden its appeal for the masses."

I stand corrected. Thank you for your thoughtful reply.

I find myself looking for ways to reconcile the two. Of course, in even admitting that, I'm busting myself! If I have my desired conclusion in mind as I sift through the evidence, I have already forgotten the central teachings of Overcoming Bias!

Hmm. I wonder whether in ordinary cases it is okay to construct tentative models of reality at a profligate pace provided one remains sufficiently eager to revise and discard. I'm pretty sure that I derive pleasure when one of the tentative models I have constructed is destroyed by a counterexample or counter-evidence (and that this pleasure is caused by the same mechanism that causes the pleasure I get when I learn a new fact) and that that pleasure outweighs the pleasure I derive from feeling certain that I am right. In particular, I hypothesize that my early experience desensitized me to doubt including doubt about my own morality -- feelings that most people who did not have my experience seem to find quite aversive.

I believe that our environment is "awash in evidence" in that most hypotheses we need to entertain to lead a very effective and very ethical life have the property that if a person ignores evidence for the hypothesis, the only thing he sacrifices is time because the mere passage of time will bring more evidence for the hypothesis. Now of course I recognize exceptions to this general observation. I am willing to believe for example that in competitive situations like military combat or wheeling and dealing in business or simply in buying and selling, the person who pays closer attention to scarce evidence can have a decisive advantage. (Hmm: these situations also seem to share the property that denying the opponent information about one's situation is often decisive.) But in the main it remains true IMO.

In summary, the worst cognitive biases seem to me to be those in which the person is actively motivated by the human reward circuitry to ignore certain classes of evidence in a consistent manner. I propose that in comparison, merely ignoring most but not all evidence on some point and profligately building causal models on scant evidence are minor sins. Consequently, I advise paying close attention to one's emotional responses around belief formation and belief rejection.

Since that proposition seems to contradict a point Eliezer has made several times, I will counter the possibility that I will be misunderstood by saying that I agree with him at least 98% of the time and have personally learned far more from his writings than I have from any other author since 2001, when I discovered his writings. How much to trust or to give our loyalty to our emotions might be the biggest place he and I disagree, with my maintaining that it is critical for a person who aspires to be a culture leader to ignore as much as practical species-typical emotional associations when choosing one's beliefs and terminal values.

I advise a young person who wishes to become a mature adult who is not an arrant slave to species-typical cognitive biases to pay copious attention to what thoughts and beliefs cause him pleasure and which cause discomfort. I suggest that over the long term, if a person begins the project while still a teenager, a person has quite a bit of control over his emotional responses -- can for example probably cause himself to become an adult who takes great pleasure in learning new scientific information.

Two hints on that one. First, being rewarded (with e.g. money or grades) for learning will tend to extinguish the "intrinsic" motivation to learn which is so valuable. So if you must undergo the formal educational system, be as indifferent to grades as practical. Second, the pleasure to be derived from learning or from exercising scientific or technical creativity is minor compared to the pleasure a teenager can derive from success in the popularity game that high school is famous for, sex and perhaps dominating opponents on the athletic field. If you can manage to derive most of your pleasure from learning during the critical age from about 14 to 17 -- by making a point not to develop the habit of getting your pleasure from the three more powerful reinforcers I just mentioned, then you will have gone a long way to setting yourself up for "good emotional responses" throughout your adulthood. (Before the age of 14, most people will not have sufficient executive skills to engage in such a program of "emotional shaping", but if you think you do have the skills or if you have adults you trust helping you, I say go for it.)

Buddhist pursuits of the type Humphries engages in seems to be a fine aid to becoming a relatively-unbiased adult, particular what the Buddhists have to say about cultivating an observing self.

Let me counter the possibility I will be misunderstood by saying that I have no practical experience educating young people except what I have learned from observing myself and listening to the recollections of a handful of friends. Still so much of what I read about pedagogy strikes me as misguided that I chose to speak out.

I am threadjacking of course, but I consider it not worth the costs to try to keep the conversation in neat little boxes especially once a thread has aged for a few days. I'll of course defer to the judgement of original poster and the owner of the blog.

Humphries and Hollerith, your comments would be too long even if they were on-topic. However you can resubmit the comments to an Open Thread, after which they will be deleted here. Thank you.

If they're too long for this page, I suggest that they're too long for an Open Thread, too. I have copied Humphries' latest and my two comments to my web site and emailed Humphries with a notification of what I did (followed by an offer to delete his words from my site if that is his preference).

Deep Blue has many desires too. It knows that a knight is three times as desirable as a pawn - unless the pawn is well advanced. It knows about the value of the centre, and the importance of quiescence - and so on.

The important point to realise is that these desires all represent imperfections. They are not useful features - to be retained and deliberately implemented in future designs, but rather simple heuristics intended to deal with hardware and software limitations - and that in the future their preservation may well lead to mistakes, errors - and losses.

Also: nature gave us brains. Brains help organisms deal with a varying environment. Part of the purpose of the brain is, I claim, is to reassemble your desires into evolution's intended "target" - a.k.a making grandchildren. The target itself cannot be built in directly - because of the limited space in the genome, and because of the varying nature of the environment.

The brain does this reassembly successfully in some individuals. They realise that high calorie foods are not good for them - because their environment is not the one their ancestors evolved in. They wake up to the idea that advertisers are trying to play the imperfections of their mind like an organ, for their own ends, and make efforts to compensate. And they understand the consequnces of the use of contraceptive devices. The in-built desires are subjugated in favour of higher level goals. If your brain has not managed such a reconstruction, you may want to consider the hypothesis that it is broken or malfunctioning - and thus to wonder if there is anything you can do to fix it.

The Mote in God's Eye by Niven and Pournelle depicts an intelligent species that stayed biological a little too long, slowly becoming truly enslaved by evolution, gradually turning into true fitness maximizers obsessed with outreproducing each other. But thankfully that's not what happened. Not here on Earth. At least not yet.

This is an interesting and important paragraph; and it explains some things about Eliezer's views. It's important enough to justify. But I don't see evidence for the idea that evolution gets more oppressive as time passes. Is this a trend in historical data? No; organisms acquire more degrees of freedom as they become more complex.

The unspoken assumption is that organisms continue to evolve, yet don't increase in complexity - imagining humans to continue to evolve, yet without passing beyond the human stage. Perhaps moties would be the result. We see here in the US that evolution very rapidly rewards cultures and religions that forbid birth control and/or encourage large families.

On the other hand, these cultures' and religions' growth in number of humans does not result in an equal growth in money and power and control of resources.

I don't have an answer; but this idea, almost skipped over, that evolution will inevitably lead to bad things, is a powerful motivator of FAI, CEV, and all such take-over-the-universe schemes. So it needs much more explication than a one-sentence reference to a Niven novel.

<3

I'm mostly in agreement with this, but feel I must point out that from the perspective of social primate evolution the "sex only when it will result in offspring" paradigm is a perversion invented (or at least reinvented) by modern humans. Sex is primarily a bonding mechanism, as evidenced by the fact that sexual desire is mediated as much by social circumstances as by other considerations. Of course, social standing is ultimately directed at improving genetic fitness, but sex has been repurposed by the primate social system so that, essentially, it improves fitness in two ways rather than just the one you seem to be seeing. Given this, and the fact that the important number is (as some evolutionary biologists have pointed out) not the number of children born but the number of one's children who themselves reproduce, and you have a perfectly good reason why humans in every place and time have been trying like he'll to invent reliable birth control, for those numerous times when the "social bonding" part is desired but not the "potentially getting pregnant" part.

I agree with the thrust here, but it does seem that you're conflating two different distinctions.

More specifically: you contrast explicit cognitive representations with implicit genetic representations (1), and it's not always clear when you are talking about the distinction between implicit and explicit representations, and when you are talking about the difference between cognitive and genetic ones.

And it seems to matter: if I ask why my genetic representations aren't recapitulated as cognitive ones, the kind of answer you give here is a fine one, but if I ask why my implicit representations aren't recapitulated as explicit ones, that answer is insufficient. I am ignorant not only of what "my genes want," but also of much of what "my brain wants," and the stubborn implicitness of that second kind of information is not proximally due to evolution's inability to quickly refactor code.

I don't think any of that actually alters your main point, which is primarily about genetic vs. cognitive representations. Still, it's worth emphasizing that not all cognitive representations are explicit ones, and there are good reasons for that over and above the genetic "godshatter" effect.

(1) I'm using "representation" here in a very loose way, admittedly.

This is possibly the best creation myth I've ever read. Possibly because unlike other creation myths, this one is actually true.

You've found amazing poetry in this grand cycle of gene warfare. But now I must wonder: How self-contained are all these desires? Will we evolve some of them to extinction? It is very hard, and somewhat disconcerting, to think of what today is human as only a passing phase on an endless continuum. Yet to assume humanity would always remain as it is seems both unrealistic, and unsatisfying - we want to see growth and novelty. So I guess I hope we will become more complex, more interesting... Rather than get narrowed toward a less fragmented sense of purpose.

It just seems the evolution has failed to build a Friendly (to the evolution) AI.

Why not become a pure reproductive consequentialist?

Reading these posts I notice a preference for altruism, utilitarianism and rejecting some of the intuitions that natural selection gave us. Moreover, almost everyone working on evolutionary psychology takes a lot of effort to avoid the naturalistic fallacy: Not confusing what is with what ought to be (see Richard Dawkins - "The Selfish Gene" or Steven Pinker - "The Blank Slate").

Still I am wondering what is so "good" about altruism? Knowing that our preference for altruism also developed by natural selection, because it either benefits our genes in other humans (W.D. Hamilton - Kin Altruism ) or leads to reciprocal benefits for ourselfs (Robert Trivers - Reciprocal Altruism ), or it least did in small hunter-gatherer tribes in the ancestral environment. Utilitarianism is now projecting this altruism that we naturally feel towards friends and family (which was good for our genes) onto humanity as a whole (which probably isn't). Usually there is the assumption that every human life is worth the same.

I agree that you can't take your values from evolution, but why assume that there are any (objective) values at all? Why not embrace nihilism? Why not become a pure reproductive consequentialist?

Some practical consequences of this value system (some of them pretty weird):

• Valuing your family, esp. your children more than strangers (anybody does that intuitively anyway)

• Valuing your friends more than strangers (because they reciprocate; people also naturally do this)

• Sacrificing your life for your children if that improves their survival more than it reduces your chance for future surviving children (you also see this very often in the real world, fathers drowning to save their children, etc.; Of course you could do the math much better than our adaptation which basically says "save drowning children")

• Switching to full altruism after your chance (or plan) to have future children has fallen to zero. Of course, still caring more about your relatives more than others (grandparents do this a lot, also Bill Gates would be an example)

• Ignoring your will to have sex unless you plan to have children or it becomes distracting and reduces your ability to achieve your other goals.

• Going to the sperm bank (spreading your genes and getting paid for it. That's what I call a win-win situation.)

• Avoiding fatty and sugary food, following the paleo diet. (to improve your direct fitness and sexual attractiveness)

• Not having any higher moral values whatsoever. Following your moral intuitions only when they are useful to other goals.

• Basically acting like Gordon Gekko from Wall Street. Only that you would try to turn your money and power into a lot of children, likely from different women. (Like the Aztec or Inca emperors which had thousands of women. Unfortunately for the inclusive fitness of today's powerful men this has become nearly impossible. It's better for the average man I guess.)

I am not planning to act out this slightly silly idea in my life. Still I am astonished how well it approximates what people actually do considering the change from our ancestral environment. I would like to hear your thoughts.

14 years later, I notice that Eliezer missed the other reason why evolution didn't design organisms that have fitness maximization as an explicit motivation. It's not just that it can't plan well enough to get there, it's also that such a motivation would have a disadvantage compared to a set of heuristics: higher computational cost. A hypothetical mind only concerned with fitness maximization would probably have to rediscover a bunch of heuristics like "excessive pain is bad" to survive practice. (At that point, it would indeed have an advantage in that it could avoid many of the failure modes of heuristics.)

I would submit that most other species on the planet, were they to rise to our level of intelligence, would not bother inventing condoms.  In most other species, the females generally have no particular interest in sex unless they want babies.

Humans though, are weird.  Because of our long phase of immaturity, and the massive amount of work involved in raising a child, we need really strong social bonds.  Evolution, being a big fan of "The first thing I stumble across that gets the job done is the solution" repurposed sex into a pair-bonding trigger, and then, as our ancestors' offspring required longer and longer care, divorced it from any specific attempt to make a baby at that particular moment.

Now fast forward to the point where infant mortality drops and churning out babies as fast as possible is no longer the best strategy.  But we still need the pair bonding because the length of childhood hasn't gotten any shorter, and it still goes way better with two sets of hands to look after the little one.  Evolution would probably come up with another quick hack for this...  (One might suggest that it already has in the form of oral sex.)  But it will take a while.  Our brains are faster.

Evolution now will simply need to favor genetics that introduce an explicit desire for children, rather than the other behaviours which used to inevitably lead to them.  Which...  There are a lot of people out there for whom not wanting children is a dealbreaker when looking for a potential spouse.  So it seems like it's already on top of that one too.