A Less Wrong singularity article?

I hope such a document addresses the Mañana response: yeah, sure, figuring out how to control the AIs and make sure they're friendly is important, but there's no time pressure. It's not like powerful AI is going to be here anytime soon. In fact it's probably impossible to figure out how to control the AI, since we still have no idea how it will work.

I expect this kind of response is common among AI researchers, who do believe in the possibility of AI, but, having an up-close view of the sorry state of the field, have trouble getting excited about prophecies of doom.

It's as though no one here has ever heard of the bystander effect. The deadline is January 15th. Setting up a wiki page and saying "Anyone's free to edit." is the equivalent to killing this thing.

Also this is a philosophy, psychology, and technology journal, which means that despite the list of references for Singularity research you will also need to link this with the philosophical and/or public policy issues that the journal wants you to address (take a look at the two guest editors).

Another worry to me is that in all the back issues of this journal I looked over, the papers were almost always monographs (and baring that 2). I suspect that having many authors might kill the chances for this paper.

Devote the second half to discussing the question of FAI, with references to e.g. Joshua Greene's thesis and other relevant sources for establishing this argument.

(Now that this is struck out it might not matter, but) I wonder if, in addition to possibly overrating Greene's significance as an exponent of moral irrealism, we don't overrate the significance of moral realism as an obstacle to understanding FAI ("shiny pitchforks"). I would expect the academic target audience of this paper, especially the more technical subset, to be metaethically confused but not moral realists. Much more tentatively, I suspect that for the technical audience, more important than resolving metaethical confusion would be communicating the complexity of values: getting across "no, you really don't just value intelligence or knowledge or 'complexity', you really don't want humans replaced by arbitrary greater intelligences (or possibly replaced by anything; values can be path-dependent), and, while that would be true even if most superintelligences had complex values and built complex worlds, there's reason to think most would produce dull monocultures." ("Which Consequentialism? Machine Ethics and Moral Divergence" is closely related.)

But maybe I overstate the difference between that and metaethical confusion; both fall under "reasons to think we get a good outcome by default", both are supported by intuitions against "arbitrary" values, and probably have other psychology in common.

The arguments that I found most compelling for a hard takeoff are found in LOGI part 3 and the Wiki interview with Eliezer from 2003 or so, for anyone who needs help on references or argument ideas from outside of the sequences.

This is a great idea Kaj, thanks for taking the initiative.

As noted by others, one issue with the AI Risks chapter is that it attempts to cover so much ground. I would suggest starting with just hard take-off, or local take-off, and presenting a focused case for that, without also getting into the FAI questions. This could also cut back on some duplication of effort, as SIAI folk were already planning to submit a paper (refined from some work done for a recent conference) for that issue on "machine ethics for superintelligence", which will be discussing the FAI problem.

P.S. If people would like to volunteer to review drafts of those papers in a few weeks to give feedback before they are submitted, that would be much appreciated. (You can email me at myfirstname AT mylastname at gmail)

I would be surprised if Eliezer would cite Joshua Greene's moral anti-realist view with approval.

This is a problem for both those who'd want to critique the concept, and for those who are more open-minded and would want to learn more about it.

Nit: This implies that people who disagree are closed minded.

I think this is very needed. When reviewing singularity models for a paper I wrote I could not find many readily citable references to certain areas that I know exist as "folklore". I don't like mentioning such ideas because it makes it look (to outsiders) as I have come up with them, and the insiders would likely think I was trying to steal credit.

There are whole fields like friendly AI theory that need a big review. Both to actually gather what has been understood, and in order to make it accessible to outsiders so that the community thinking about it can grow and deepen.

Whether this is a crowdsourcable project is another matter, but at the very least crowdsourcing raw input for later expert paper construction sounds like a good idea. I would expect that eventually it would need to boil down to one or two main authors doing most of the job, and a set of co-authors for speciality skills and prestige. But since this community is less driven by publish-or-perish and more by rationality concerns I expect ordering of co-authors may be less important.

A problem with this proposal is whether this paper can be seen as authorative. A critic might worry that if they study and respond to this paper they will be told it does not represent the best pro-Singularity arguments. So the paper would need to be endorsed enough to gain enough status to become worth criticizing.

One argument against a hard takeoff is Greg Egan's idea about there not being a strong hierarchy of power in general intelligence, which he goes into here, for example. Would this be worth covering?

This is a problem for both those who'd want to critique the concept, and for those who are more open-minded and would want to learn more about it.

Anyone who is sufficiently technically minded undoubtedly finds frustration in reading books which give broad brush stroked counterfactuals to decision making and explanation without delving into the details of their processes. I am thinking of books like Freakonomics, Paradox of Choice, Outliers, Nudge etc..

These books are very accessible but lack the in depth analysis which are expected to be thoroughly critiqued and understood in depth. Writings like Global catastrophic risks and any of the other written deconstructions of the necessary steps of technological singularity lack those spell-it-out-for-us-all sections that Gladwell et al. make their living from. Reasonably so. The issue of singularity is so much more complex and involved that it does not do the field justice to give slogans and banner phrases. Indeed it is arguably detrimental and has the ability to backfire by simplifying too much.

I think however what is needed is a clear, short and easily understood consensus on why this crazy AI thing is the inevitable result of reason, why it is necessary to think about, how it will help humanity, how it could reasonably hurt humanity.

The SIAI tried to do this:

http://www.singinst.org/overview/whatisthesingularity

http://www.singinst.org/overview/whyworktowardthesingularity

Neither of these is compelling in my view. They both go into some detail and leave the un-knowledgeable reader behind. Most importantly neither has what people want: a clear vision of exactly what we are working for. The problem is there isn't a clear vision; there is no consensus on how to start. Which is why in my view the SIAI is more focused on "Global risks" rather than just stating "We want to build an AI"; frankly, people get scared by the latter.

So is this paper going to resolve the dichotomy between the simplified and complex approach, or will we simply be replicating what the SIAI has already done?

Robin criticizes Eliezer for not having written up his arguments about the Singularity in a standard style and submitted them for publication. Others, too, make the same complaint: the arguments involved are covered over such a huge mountain of posts that it's impossible for most outsiders to seriously evaluate them.

Did everyone forget about "Artificial Intelligence as a Positive and Negative Factor in Global Risk"?

As a Foom skeptic, what would convince me of taking the concept seriously, is an argument that intelligence/power is a quantity that we reason with in the same way as we reason about the number of neutrons in a nuclear reactor/bomb. Power seems like a slippery ephemeral concept, optimisation power appears to be able evaporate at the drop of a hat (if someone comes to know an opponents source code and can emulate them entirely).

Any thoughts on what the impact of the http://www.research.ibm.com/deepqa/ IBM Watson Deepqa project would be on a Foom timescale, if it is successful (in the sense of approximate parity with human competitors)? My impression was that classical AI failed primarily because of brittle closed-world approximations, and this project looks like it (if successful) would largely overcome those obstacles. For instance, it seems like one could integrate a deepqa engine with planning and optimization engines in a fairly straightforward way. To put it another way, in the form of an idea futures proposition: conditional on deepqa being fully successful at a human-equivalent level, what are the odds of a human equivalent UFAI in all areas needed for recursive self-improvement plus replication without human intervention in say 5 years after that point? Best wishes, -Jeff

Hmmm... Maybe you could base some of it off of Eliezer's Future Salon talk (http://singinst.org/upload/futuresalon.pdf)? That's only about 11K words (sans references), while his book chapters are ~40K words and his OB/LW posts are hundreds of thousands of words.

Relevant article by E.Y.:

http://lesswrong.com/lw/we/recursive_selfimprovement/

Any thoughts on what the impact of the IBM Watson Deepqa project would be on a Foom timescale, if it is successful (in the sense of approximate parity with human competitors. My impression was that classical AI failed primarily because of brittle closed-world approximations, and this project looks like it (if successful) would largely overcome those obstacles. For instance, it seems like one could integrate a deepqa engine with planning and optimization engines in a fairly straightforward way. To put it another way, in the form of an idea futures proposition: conditional on deepqa being fully successful at a human-equivalent level, what are the odds of a human equivalent UFAI in all areas needed for recursive self-improvement plus replication without human intervention in say 5 years after that point? Best wishes, -Jeff

Kaj, great idea

in a standard style and submitted them for publication

This will be one of the greater challenges; we know the argument and how to write well, but each academic discipline has rigid rules for style in publications. This particular journal, with its wide scope, may be a bit more tolerant, but in general learning the style is important if one wants to influence academia.

I imagine that one will have to go beyond the crowdsourcing approach in achieving this.

If you are coordinating, let me know if and how I can help.

More recent criticism comes from Mike Treder - managing director of the Institute for Ethics and Emerging Technologies in his article "Fearing the Wrong Monsters" => http://ieet.org/index.php/IEET/more/treder20091031/

Aubrey argues for "the singularity" here:

"The singularity and the Methuselarity: similarities and differences" - by Aubrey de Grey

http://www.sens.org/files/sens/FHTI07-deGrey.pdf

He uses the argument from personal incredulity though - one of the weakest forms of argument known.

He says:

"But wait – who’s to say that progress will remain “only” exponential? Might not progress exceed this rate, following an inverse polynomial curve (like gravity) or even an inverse exponential curve? I, for one, don’t see why it shouldn’t. If we consider specifically the means whereby the Singularity is most widely expected to occur, namely the development of computers with the capacity for recursive improvement of their own workings, I can see no argument why the rate at which such a computer would improve itself should not follow an inverse exponential curve, i.e. one in which the time taken to achieve a given degree of improvement takes time X, the time taken to repeat that degree of improvement is X/2, then X/4 and so on."

My reply: to build a fast computer you don't just need to perform computations quickly. You also need to be able to design and perform real-world experiments and tests. While sensor and motor capabilities are improving, they are not doing so with the same doubling time as exists for computers. Consequently, progress in these fields (and overall progress) is correspondingly slower.

Then there's the influence of limits. Clock speed has already run into diminishing returns. By the time the slower-doubling systems have time to double very many times, the faster-doubling ones will have maxed-out - and will be hindering overall progress.

IMO, nobody seems to have thought the "singularity" idea through :-(

Very constructive proposal Kaj. But...

Since it appears (do correct me if I'm wrong!) that Eliezer doesn't currently consider it worth the time and effort to do this, why not enlist the LW community in summarizing his arguments the best we can and submit them somewhere once we're done?

If Eliezer does not find it a worthwhile investment of his time - why should we?

==Re comments on "Singularity Paper"== Re comments, I had been given to understand that the point of the page was to summarize and cite Eliezer's arguments for the audience of ''Minds and Machines''. Do you think this was just a bad idea from the start? (That's a serious question; it might very well be.) Or do you think the endeavor is a good one, but the writing on the page is just lame? --User:Zack M. Davis 20:19, 21 November 2009 (UTC)

(this is about my opinion on the writing in the wiki page)

No, just use his writing as much as possible- directly in the text of the paper. Whole articles/posts in sequence for the whole paper would be best, or try to copy-paste together some of the key points of a series of articles/posts (but do you really want to do that and leave out the rich, coherent, consistent explanation that these points are surrounded in?)

My comments may seem to imply that we would essentially be putting together a book. That would be an AWESOME book... we could call it "Intelligence Explosion".

If someone ended up doing a book like that, they might as well include a section on FAI. If SIAI produces a relevant FAI paper, that could be included (or merged) into the FAI section

SEE THIS:

http://www.imminst.org/forum/index.php?showtopic=35318&hl=