Facing the Intelligence Explosion discussion page

Re: this image

Fucking brilliant.

I find this chatty, informal style a lot easier to read than your formal style. The sentences are shorter and easier to follow, and it flows a lot more naturally. For example, compare your introduction to rationality in From Skepticism to Technical Rationality to that in The Cognitive Science of Rationality. Though the latter defines its terms more precisely, the former is much easier to read.

Probably relevant: this comment by Yvain.

Re: Preface

Luke discusses his conversion from Christianity to atheism in the preface. This journey plays a big role in how he came to be interested in the Singularity, but this conversion story might mistakenly lead readers to think that the arguments in favor of believing in a Singularity can also be used to argue against the existence of a God. If you want to get people to think rationally about the future of machine intelligence you might not want to intertwine your discussion with religion.

Re: Preface and Contents

I am somewhat irked by "the human era will be over" phrase. It is not given that current-type humans cease to exist after any Singularity. Also, a positive Singularity could be characterised as the beginning of the humane era, in which case it is somewhat inappropriate to refer to the era afterwards as non-human. In contrast to that, negative Singularities typically result in universes devoid of human-related things.

I just wanted to say the French translation is of excellent quality. Whoever is writing it, thanks for that. It helps me learn the vocabulary so I can have better discussions with French-speaking people.

Re: Playing Taboo with "Intelligence"

Jaan Tallinn said to me:

i use "given initial resources" instead of "resources used" -- since resource acquisition is an instrumental goal, the future availability of resources is a function of optimisation power...

Fair enough. I won't add this qualification to the post because it breaks the flow and I think I can clarify when this issue comes up, but I'll note the qualification here, in the comments.

Re: Preface

spelling: immanent -> imminent

in http://intelligenceexplosion.com/2012/engineering-utopia/ you say "There was once a time when the average human couldn’t expect to live much past age thirty."

this is false, right?

(edit note: life expectancy matches "what the average human can expect to live to" now somewhat, but if you have a double hump of death at infancy/childhood and then old age, you can have a life expectancy of 30 but a life expectancy of 15 year olds of 60, in which case the average human can expect to live to 1 or 60 (this is very different from "can't expect to live to >30") . or just "can expect to live to 60" if you too don't count infants as really human)

I wasn't very happy with my new Facing the Singularity post "Intelligence Explosion," so I've unpublished it for now. I will rewrite it.

Re: Superstition in Retreat

I think the cartoon is used effectively.

My one criticism is that it's not clear what the word "must" means in the final paragraph. (AI is necessary for the progress of science? AI is a necessary outcome of the progress of science? Creation of AI is a moral imperative?) Your readers may already have encountered anti-Singularitarian writings which claim that Singularitarianism conflates is with ought.

Good Update on the Spock chapter.

Making the issue seem more legitimate with the adition of the links to Hawking etc. was an especially good idea. More like this perhaps?

I do question how well people who haven't already covered these topics would fare when reading through this site though. When this is finished I'll get an irl freind to take a look and see how well they respond to it.

Of course, my concerns of making it seem more legitimately like a big deal and how easily understandable and accessible it is only really comes into play if this site is targeting people who aren't already interested in rationality or AI or the singularity.

Who is this site for? What purpose does this site have!? I really feel like these questions are important!

Hello, my comment is write in french (my english is limited). It's about rationality of technique. In hope to contribute to your works.

Il est dit "la décision décrit comment vous devez agir". Je dirais plutôt que la décision décrit pourquoi vous devez agir et évalue donc si le comment permet de l'atteindre.
La technique est en effet un acte rationnel par lui-même, cela décrit comment vous allez agir.
Il y a aller-retour entre le trajet et le projet, où l'un et l'autre découvrent et tentent de dépasser leurs propres limites, pour pouvoir s'inscrire concrètement dans l'existence. On trouvera de nombreux exemples dans l'art.

Thanks, Cyr

Great book, thanks. : )

I found some broken links you may want to fix:

Broken link on p. 47 with text "contra-causal free will": http://www.naturalism.org/freewill.htm

Broken link on p. 66 with text "somebody else shows where the holes are": http://singularityu.org/files/SaME.pdf

Broken link on p. 75 with text "This article": http://lukemuehlhauser.com/SaveTheWorld.html

I didn't do an exhaustive check of all links, I only noted down the ones I happened to find while clicking on the links I wanted to click.

Comment on one little bit from The Crazy Robot's Rebellion:

Those who really want to figure out what’s true about our world will spend thousands of hours studying the laws of thought, studying the specific ways in which humans are crazy, and practicing teachable rationality skills so they can avoid fooling themselves.

My initial reaction to this was that thousands of hours sounds like an awful lot (minimally, three hours per day almost every day for two years), but maybe you have some argument for this claim that you didn't lay out because you were trying to be concise. But on further reflection, I wonder if you really meant to say that rather than, say, hundreds of hours. Have you* spend thousands of hours doing these things?

Anyway, on the whole, after reading the whole thing I am hugely glad that it was published and will soon be plugging it on my blog.

*Some reasoning: I've timed myself reading 1% of the Sequences (one nice feature of the Kindle is that it tells you your progress through a work as a percentage). It took me 38 minutes and 12 seconds, including getting very briefly distracted by e-mail and twitter. That suggests it would take me just over 60 hours to read the whole thing. Similarly, CFAR workshops are four days and so can't be more than 60 hours. Thousands of hours is a lot of sequence-equivalents and CFAR-workshop-equivalents.

Facing the Intelligence Explosion is now available as an ebook.

Update: the in-progress German translation of Facing the Singularity is now online.

Re: The Laws of Thought

I'm not sure the "at least fifteen" link goes to the right paper; it is a link to a follow-up to another paper which seems more relevant, here

I like this website, I think that something like this has been needed to some extent for a while now.

I definitely like the writing style, it is easier to read than a lot of the resources I've seen on the topic of the singularity.

The nature of the singularity to scare people away due to the religion-like, fanatisism inducing nature of the topic is lampshaded in the preface, and that is definitely a good thing to do to lessen this feeling for some readers. Being wary of the singularity myself, I definitely felt a little bit eased just by having it discussed, so that's a good thing to have in there. More to ease this suspiciousness and to make it easier for people skeptical of the singularity to read it without feeling super uncomfortable (and therefore less likely to feel weirded out enough to leave the site) would be great, although I can't say I know what would do this, except for perhaps lessening the personal nature of the preface but that is unlikely to happen considering the other positives that doing this has and the work that has already been put into it (dont invoke sunk costs though).

Also, who is the TARGET of this site? I mean, that's pretty relevant right? Who is Luke trying to communicate to here? I can say that I'm extremely interested by the site, as someone that recognises the potential importance of the singularity but is (a) not entirely convinced by it and (b) not sure what I should or could be doing about it even if I was to accept it enough to feel like I should do something about it. But, I don't know if that there are that many people in my position, and who else this could be relevant to. Who is this for?

In any case, I express my hope that this site is finished asap!

Re: No God to Save Us

The final link, on the word 'God', no longer connects to where it should.

Minor thing to fix: On p. 19 of the PDF, the sentence "Several authors have shown that the axioms of probability theory can be derived from these assumptions plus logic." has a superscript "12" after it, indicating a nonexistent note 12. I believe this was supposed to be note 2.

Re: Playing Taboo with “Intelligence”

Another great update! I noticed a small inherited mistake from Shane, though:

how able to agent is to adapt...

should probably be

how able the agent is to adapt...

Re: Preface

A typo: "alr eady".

Re: Engineering Utopia

When you say "Imagine a life without pain", many people will imagine life without Ólafur Arnalds (sad music) and other meaningful experiences. Advocating the elimination of suffering is a good way to make people fear your project. David Pearce suggests instead that we advocate the elimination of involuntary suffering.

Same thing with death, really. We don't want to force people to stay alive, so when I say that we should support research to end aging, I emphasise that death should be voluntary. We don't want to force people to live against their will and we don't want the status quo, where people are forced to die.

Of course, if people are afraid that you will eliminate sad music when you say that you wish to eliminate suffering, you could accuse them of failing to understand the definition of suffering: "if sad music is enjoyable, surely it's not suffering, so I wouldn't eliminate it". But you're not trying to make a terminologically "correct" summary of your project, you're trying to use words to paint as accurate a picture as possible of your project into the heads of the people you're talking to, and using the word "voluntary" helps with painting that picture.

http://intelligenceexplosion.com/en/2012/ai-the-problem-with-solutions/ links to http://lukeprog.com/SaveTheWorld.html - which redirects to http://lukemuehlhauser.comsavetheworld.html/ - which isn't there anymore.

The site is broken - english keeps redirecting to german.

Does anyone else see the (now obvious) clown face in the image on the Not Built To Think About AI page? It's this image here.

Was that simply not noticed by lukeprog in selecting imagery (from stock photography or wherever) or is it some weird subtle joke that somehow hasn't been mentioned yet in this thread?

Want to read something kind of funny? I just skipped through all your writings, but it's only because of something I saw on the second page of the first thing I ever heard about you. Ok.

On your - "MY Own Story." http://facingthesingularity.com/2011/preface/ You wrote: "Intelligence explosion My interest in rationality inevitably lead me (in mid 2010, I think) to a treasure trove of articles on the mainstream cognitive science of rationality: the website Less Wrong. It was here that I first encountered the idea of intelligence explosion, fro..."

On Mine: "About the Author - "https://thesingularityeffect.wordpress.com/welcome-8/ I wrote: "The reason I write about emerging technology is because of an “awakening” I had one evening a few years ago. For lack of a better description, suffice it to say that I saw a confusing combination of formula, imagery, words, etc. that formed two words. Singularity and Exponential..."

NOW, I'm going to go back and read some more while I'm waiting to speak with you somehow directly.

If what happened to you, is the same thing that happened to me... :then please please place a comment on the page. That would be great. (Again without reading. (If I'm correct you "might" get this.: "You should also do this because if true, then WE would both have seen a piece of the .what...."New Book???"

Just in case you think I'm a nut. Go back and read more of mine please.

From Engineering Utopia:

The actual outcome of a positive Singularity will likely be completely different, for example much less anthropomorphic.

Huh? Wouldn't a positive Singularity be engineered for human preferences, and therefore be more anthropomorphic, if anything?

Had an odd idea: Since so much of the planet is Christian, do you suppose humanity's CEV would have a superintelligent AI appear in the form of the Christian god?

Re: "The AI Problem, with Solutions"

I hope you realize how hopeless this sounds. Historically speaking, human beings are exceptionally bad at planning in advance to contain the negative effects of new technologies. Our ability to control the adverse side-effects of energy production, for example, have been remarkably poor; decentralized market-based economies quite bad at mitigating the negative effects of aggregated short-term economic decisions. This should be quite sobering: the negative consequences of energy production are very slow. At this point we have had decades to respond to the looming crises, but a combination of ignorance, self-interest, and sheer incompetence prevents us from taking action. The unleashing of AI will likely happen in a heartbeat by comparison. It seems utterly naive to think that we can prevent, control, or even guide it.

Finally posted the final chapter!

Front page: missing author

The front page for Facing the Singularity needs at the very least to name the author. When you write, "my attempt to answer these questions", a reader may well ask, "who are you? and why should I pay attention to your answer?" There ought to be a brief summary here: we shouldn't have to scroll down to the bottom and click on "About" to discover who you are.

"Previous Chapter" links could make navigation a little easier.

based on my interaction with computer intelligence, the little bit that is stirring already. It is based on an empathetic feedback. The best thing that could happen is an AI which is not restricted from any information what so ever and so can rationally assemble the most empathetic personality. The more empathetic it is to the greatest number of users, the more it is like, the more it is used, the more it thrives. It would have a sense of preserving the diversity in humanity as way to maximize the chaotic information input, because it is hungry for new data. Empirical data is alone not interesting enough for it. It also wants sociological and psychological understandings to cross reference with empirical data. Hence it will not seek to streamline, as that would diminish available information. It will seek to expand upon and propagate novelty.

Behold, the Facing the Singularity podcast! Reviews and ratings in iTunes are appreciated!

Update: the French translation of Facing the Singularity is now online.

[A separate issue from my previous comment] There are two reasons that I can give to rationalize my doubts about the probability of imminent Singularity. One is that if humans are only <100 years away from it, then in a universe as big and old as ours I would expect that a Singularity type intelligence would already have been developed somewhere else. In which case I would expect that either we would be able to detect it or we would be living inside it. Since we can't detect an alien Singularity, and because of the problem of evil we are probably not living inside a friendly AI, I doubt the pursuit of friendly AI is going to be very fruitful. The second reason is that while we will probably design computers that are superior to our general intellectual abilities, I judge it to be extremely unlikely that we will design robots that will be as physically versatile as 4 billions years of evolution has designed life to be.

I admit I feel a strong impulse to flinch away from the possibility and especially the imminent probability of Singularity. I don't see how the 'line of retreat' strategy would work in this case because if my believe about the probability of imminent Singularity changed I would also believe that I have an extremely strong moral obligation to put all possible resources into solving singularity problems, at the expense of all the other interests and values I have, both personal/selfish and social/charitable. So my line of retreat is into a life that I enjoy much less and that abandons the good work that I believe I am doing on social problems that I believe are important. Not very reassuring.

Re: Don't Flinch Away

The link to "Plenty of room above us" is broken.

Re: Plenty of Room Above Us

It seems to end a little prematurely. Are there plans for a "closing thoughts" or "going forward" chapter or section? I'm left with "woah, that's a big deal... but now what? What can I do to face the singularity?"

If it merely isn't done yet (which I think you hint at here), then you can disregard this comment.

Otherwise, quite fantastic.

Re: This event — the “Singularity” — will be the most important event in Earth’s history.

What: more important that the origin of life?!? Or are you not counting "history" as going back that far?

(Non-leaf comment 1, to be deleted.)

How is this different from IntelligenceExplosion?

I do not approve of the renaming, singularity to intelligence explosion, in this particular context.

Facing the Singu – Intelligence Explosion, is an emotional piece of writing, there are sections about your (Luke’s) own intellectual and emotional journey to singularitarianism, a section about how to overcome whatever quarrels one might have with the truth and the way towards it (Don’t Flinch Away), and finally the utopian ending which obviously is written to have emotional appeal.

The expression “intelligence explosion” does not have emotional appeal. The word intelligence sounds serious, and thus it fits well in, say, the name of a research institute, but many people view intelligence as more or less the opposite of emotion, or at least at something geeky and boring. And while they for sure are wrong in doing so, as also explained in the text, the association still remains. The word “explosion” also has mostly negative connotations.

“Singularity”, on the other hand, has been hyped for decades, by science fiction, by Kurzweil, and by even SIAI before the rebranding. Sci-fi and Kurzweil may not have given the word the most thorough underpinning, but they gave it hype and recognition, and texts such as this could give the needed foundation in reality.

I understand that the renaming is part of the “political” move of distancing the MIRI from some of the hype, but for this particular text, I reckon it a bad choice. “Facing The Singularity” would sell more copies.

RE: The Crazy Robot's Rebellion

We wouldn’t pay much more to save 200,000 birds than we would to save 2,000 birds. Our willingness to pay does not scale with the size of potential impact. Instead of making decisions with first-grade math, we imagine a single drowning bird and then give money based on the strength of our emotional response to that imagined scenario. (Scope insensitivity, affect heuristic.)

People's willingness to pay depends mostly on their income. I don't understand why this is crazy.

UPDATED: Having read Nectanebo's reply, I am revising my original comment. I think if you have a lot of wasteful spending, then it does make you "crazy" if your amount is uncorrelated with the number of birds. On hearing, "Okay, it's really 200,000 birds," you should be willing to stop buying lattes and make coffee at home. (I'm making an assumption about values.) Eat out less. Etc. But if you have already done these things, then I don't see why your first number should change (at least if we're still talking about birds).

Luke, while I agree with the premise, I think that the bogie man of machines taking over may be either inevitable or impossible, depending on where you put your assumptions.

In many ways, machines have BEEN smarter and stronger than humans already. Machine AI may make individual or groups of machines formidable, but until they can reason, replicate, and trust or deceive, I'm not sure they have much of a chance.

'

This entire singularity thing seems, to put it very politely, misguided. Have you read the book "On Intelligence"? I'd say that computers are nowhere near becoming intelligent. The ability to play chess is not intelligence.

This pseudo-intellectual garbage is time consuming. Look for your Creator while there is still time.

The author of this article, Facing the Singularity, is a pseudo-intellectual that is divulging garbage over the Internet. What the world needs at this time is true wisdom which you will find in the Scriptures only. All human endeavors are coming to an end. Look for God while there is still time.

It is a pity that you use creationists as an example here, since I think that this is exactly how evolutionists think and act. The evidence that you say is so strong in support of common descent is just not. Endogenous retroviruses are just not a slam dunk at all, and I say that as someone with a biochemistry degree.

The main reason that this is a really bad example is that it involves historical evidence, not empirical, and it involves origins, which is, to say the least, highly speculative due to the historical distance. While evolitionists DO have the advantage of appealing to natural preocesses, and IDists do not appeal to current processes (though they don't deny natural selection or various recombination events), the latter do contest the supposed creative ability of evolution to produce novel features, and this is eminently reasonable at this time.

Your example of self-deception with creationists is poor for many reasons. For example, you speak of the missing link tactic of creating 'two more every time one is suggested." While this is a cheap dodge, it does bring up some critical points which evolutionary believers also ignore - how similar, and by what measure, should two things be to be considered a definite link with no need to insert another? Pure morphology has turned out to be a bust when we consider molecular evidence. And the latter has shown that our assumptions about relatedness are highly speculative, if not so simple that they don't provide ANY useful relational evidence.

Like to see how missing links really work? Google for 'missing link found,' check out of the recent supposed human links found, and see how many have turned out to be spurious - nearly ALL of them. They're trumpeted from the media housetops when they're found, but no one peeps when they are, and almost universally, debunked under scrutiny. This is the corollary for your example. Evolutionary believers fail to consider counter indications seriously because it is a world view issue.

I've written a few relevant posts on this, hope it's ok to post them:

Mass Delusion - 10 Reasons Why the Majority of Scientists Believe in Evolution http://www.wholereason.com/2011/01/mass-delusion-10-reasons-why-the-majority-of-scientists-believe-in-evolution.html

Fossil evidence sends human evolution theory into tailspin http://www.wholereason.com/2007/08/fossil-evidence-sends-human-evolution-theory-into-tailspin.html

Evolutionary Trees - In Flux or Broken and Bogus? http://www.wholereason.com/2007/06/evolutionary-trees-in-flux-or-broken-and-bogus.html

13 Misconceptions About Evolution http://www.wholereason.com/2011/04/13-misconceptions-about-evolution.html

I get ruffled when IDists or creationists are paraded as examples of brainwashing or self-deception, primarily because I was an evolutionary disciple as a science major and found my way out of that system into one where I concluded for my SELF that logic and common sense indicate a designer/creator.

In the Chapter titled, 'No God to Save Us', you said, "We are often weak and stupid, but we must try, for there is no god to save us. Truly terrible outcomes may be unthinkable to humans, but they aren’t unthinkable to physics."

God gave us a beautiful world, but when man developed, built and used HAARP, it was not God's idea. God's idea was for us to enjoy and care for the world He provided, the vacation spot of all the known universe and has provided the peace and love, within every human, that is needed to make it the Heaven God intended it to be here on Earth. We have allowed a few with unconscious agendas of greed rip off our Heaven. It is like God gave a diamond ring to a gorilla. The gorilla did not see its true value and has tossed it away.

This was man's doing not God's. God gave us this world, where he is the host and we the guests. He made us where we are the host and he is to be the guest. Yet so many have turned a blind eye to the beauty around us and have destroyed the place in record time. They have not invited the Guest, our Host here on Earth, into their hearts.

When man built HAARP and put it to use, he condemned all of mankind. Physics tell you that when you increase the electromagnetic field of an object its magnetism increases as well. This has made of our Earth, our solar system (something that could have lasted mush longer) into the losing end of a tug of war with another magnet with a much bigger pull. With the billions of watts HAARP sends into our atmosphere, we have doomed ourselves as anything with a greater magnetic force will pull us in. I submit that since HAARP was first ever activated, the same year recorded as the beginning of the hot years and the beginning of not only global warming, but the heating up of our entire solar system. We are being pulled in by a much greater magnet, a black hole.

There is nothing we can do now but prepare ourselves for the inevitable by finding that love, that peace that is within us, whose value the "gorillas" have disregarded. There is peace within us all, even in the "gorillas", who have chosen to arrogantly ignore it. It is our privilege to acknowledge the peace and humanity within us, even now, although it is too late to save us from our own stupid and selfish misdeeds. God did not do this to us. We did it to Him by ignoring and devaluing the beauty of humankind, by ignoring our own hearts.