A Prodigy of Refutation

"And I wonder if that advice will turn out not to help most people, until they've personally blown off their own foot, saying to themselves all the while, correctly, "Clearly I'm winning this argument.""

I fell into this pattern for quite a while. My basic conception was that, if everyone presented their ideas and argued about them, the best ideas would win. Hence, arguing was beneficial for both me and the people on transhumanist forums- we both threw out mistaken ideas and accepted correct ones. Eliezer_2006 even seemed to support my position, with Virtue #5. It never really occurred to me that the best of everyone's ideas might not be good enough.

"It is Nature that I am facing off against, who does not match Her problems to your skill, who is not obliged to offer you a fair chance to win in return for a diligent effort, who does not care if you are the best who ever lived, if you are not good enough."

Perhaps we should create an online database of open problems, if one doesn't exist already. There are several precedents (http://en.wikipedia.org/wiki/Hilbert%27s_problems). So far as I know, if one wishes to attack open problems in physics/chemistry/biology/comp. sci./FAI, the main courses of action are to attack famous problems (where you're expected to fail and don't feel bad if you do), or to read the educational literature (where the level of problems is pre-matched to the level of the material).

You posted your raw email address needlessly. Yum.

So, what if it becomes clear that human intelligence is not enough to implement FAI with the desirable degree of confidence, and transhuman intelligence is necessary? After all, the universe has no special obligation to set the problem up to be humanly achievable.

If so, then instead of coming up with some elaborate weighting scheme like CEV, it'd be easier to pursue IA or have the AI suck the utility function directly out of some human -- the latter being "at least as good" as an IA Singularity.

If programmer X can never be confident that the FAI will actually work, with the threat of a Hell Outcome or Near Miss constantly looming, they might decide that the easiest way out is just to blow everything up.

How can you tell if someone is an idiot not worth refuting, or if they're a genius who's so far ahead of you to sound crazy to you? Could we think an AI had gone mad, and reboot it, when it is really genius.

I've never been on a transhumanist mailing list, but I would have said, "Being able to figure out what's right isn't the same as actually doing it. You can't just increase the one and assume it takes care of the other. Many people do things they know (or could figure out) are wrong."

It's the type of objection you'd have seen in the op-ed pages if you announced your project on CNN. I guess that makes me another stupid man saying the sun is shining. At first, I was surprised that it wasn't on the list of objections you encountered. But I guess it makes sense that transhumanists wouldn't hold up humans as a bad example.

What if it's not too hard? you then risk extremely bad things like mature molecular nanotechnology in the hands of humans such as the US government (for example, perhaps at this rate more like, the Japanese government) simply because you didn't try.

In the case that current human intelligence we cannot prove beyond doubt the friendliness of some theory of err friendliness is actually friendly, then no harm. At minimum, it would result in a great deal preliminary work on FAI being completed.

When its obvious that you need a bit more of this 'intelligence' thing to move on, you could either switch to working on IA until a sufficient amount of intelligence enhancement is done then go back to FAI. Or you could keep on slogging on with FAI while getting the benefits of IA which far more people are working towards compared to FAI.

On this note, as a side note, I see usefulness in slowing down research in potentially dangerous technologies such as molecular nanotechnology. Perhaps then if you cannot do more work on FAI (your not smart enough) you could switch careers to work with Foresight nanotech institute (or something else, be imaginative!) to either bring the date of FAI closer, or give more 'breathing space' so as FAI can be completed sooner etc.

"Surely no supermind would be stupid enough to turn the galaxy into paperclips; surely, being so intelligent, it will also know what's right far better than a human being could."

Sounds like Bill Hibbard, doesn't it?

By George! You all need to make a hollywood blockbuster about the singularity and get all these national-security soccor moms screaming hellfire about regulating nanotechnology... "THE END IS NEAR!" I mean, with 'Left Behind' being so popular and all, your cause should fit right into the current milieu of paranoia in America.

I can see the preview now, children are quietly singing "My Country 'tis of Thee" in an old-fashioned classroom, a shot zooms from out the window to show suburban homes, a man taking out the trash with a dog, a woman gardening, a newscast can be overheard intermingling with the singing, "Ha Ha Mark, well, today's been a big day for science! Japanese physicist Uki Murakazi has unveiled his new, very tiny, and CUTE I might add, hydrogen-fuel creating nanobots..." Woman looks up as sky starts to darken. Silence 'What if all that ever mattered to you...' Lone voice, "Mommy?" Screaming chaos, school busses get sucked into some pit in the earth, upclose shots of hot half-naked woman running away in a towel with a bruise crying, firemen running pel-mell, buildings collapsing, the works... "What if all of it..." Dramatic "EUNK!" sound upon a black screen... Voices fade in, "God, where are you?" "I don't think we can stop it..." "Mommy? Where are we?" "Be prepared, because this September," violins making that very high pitched mournful noise, the words "The Singularity is Near" appear on the screen.

It practically writes itself... Then at the high point of the movie's popularity, you begin making press releases, interviews, etc. that declare you find such doomsdays scenarios (though not exactly as depicted) possible and of important security risk. Could backfire and make you look insane, I suppose... But even so, there's a lot of money in Hollywood- think about the Scientologists.

It's easier to say where someone else's argument is wrong, then to get the fact of the matter right;

You posted your raw email address needlessly. Yum.

How can you tell if someone is an idiot not worth refuting, or if they're a genius who's so far ahead of you to sound crazy to you? Could we think an AI had gone mad, and reboot it, when it is really genius.

In the case of you considering taking action against the entity (as in your example of deleting the AI), this is partly self-regulating: A sufficiently intelligent entity should see such an attack coming and have effective countermeasures in place (for instance, by communicating better to you so you don't conclude it has gone mad). If you attack it and succeed, that by itself places limits on how intelligent the target really was. Note that this part doesn't work if both sides are unmodified humans, because the relative differences in intelligence aren't large enough.

I think your post can be boiled down to simply, "If you always win arguments, you are collecting errors."

Or you're always choosing the right side. Like I said, it's not that simple.

Define 'winning the argument'. Taboo it, if you will.

In many cases, what's meant by that phrase is that the other side is convinced or is considered to have been defeated by observers. What isn't meant is that a corrent and coherent argument was presented, much less by the 'winner'.

How does the argument that a superintelligence wouldn't be able to determine what was correct by being superintelligent coexist with the argument that certain forwarded positions have had a great deal of thought put into them and must therefore be right?

"Or you're always choosing the right side."

The problem is that if you ever win an argument when you are wrong, subsequent arguments with anyone who has accepted your false conclusion leads to further errors. Furthermore, to avoid this, it is not enough to always choose the right side. You must be right about everything you convince your opponent of. Even the right conclusion can be supported by false evidence. Lastly, you will probably engage in arguments that have no right side or conclusion. Such arguments should not be won or lost - rather, both sides should admit when there is insufficient evidence to support either case.

Of course, you could always choose the right side, one hundred percent of the time, every time. How likely is that compared to the likelihood of being argumentatively superior, though? Having a compelling enough argument to convince someone else of a false conclusion means having a compelling enough argument to convince yourself of the same thing.

I could be wrong about this, though :)

Re: You all need to make a hollywood blockbuster about the singularity and get all these national-security soccor moms screaming hellfire about regulating nanotechnology.

If you relinquish powerful and important technologies, what happens is that someone else invents them and sends your economy up the tubes. Probably best to let the eco-people blather on about global warming instead. They seem quite happy doing that - and at least there they can't do much damage.

"You cannot rely on anyone else to argue you out of your mistakes; you cannot rely on anyone else to save you; you and only you are obligated to find the flaws in your positions; if you put that burden down, don't expect anyone else to pick it up."

Really? Over the last few years I've changed my outlook and positions on lots of issues based on reading sites like this, among other material I didn't generate myself.

I guess the willingness to change based on new information must have been in place already to some extent. Yet I feel like I was pushed quite distinctly by exposing myself to papers and books about cognitive bias and such.

"...but the younger me, let's call him Eliezer1997, set out to argue inescapably that creating superintelligence is the right thing to do."

Of course it's the right thing to do: How else could we ever achieve a truly Cultural lifestyle? (meaning, as in Iain M. Banks' Culture)

Ryan--your obervation is true and I agree your resolution...if you don't want to improve, you probably won't. But seeking out related literature for application often speeds up one's rate of progress.

Ian-- Genius demonstrates some convergence...ask the AI a hard math problem, for example, and if it solves it, you know it's smart. On the other hand, if it's smart and doesn't want you to know that, you'll have a hard time finding out anyway. In general, if you know an agent's utility function, you can infer its intelligence based on how well it drives the world towards its target space of preferred outcomes. The uncertainty of knowing the utility function makes this hard. Eli posted on this in more detail very recently.

Tom--This seems useful, though you won't know what's really unsolved versus what's out there on the internet but just not found by you yet. This sounds like a wiki to organize knowledge...further, it's not clear that you should remove problems once you solve them, since you have added some structure to help classify and locate the problem. In general, the internet itself is already functioning as your database, but you could make a subset which prunes itself more efficiently over the problems we care about.

Michael--an AI that "sucks out one's utility function" and doesn't lead to a failure mode itself requires extrapolation of at least one human. Hopefully, many different humans extrapolate similarly...the more this is the case, the less one needs a complicated CEV weighting system. In the extreme case, it could be that 1 human leads to the same outcome as some CEV of all humanity. But this seems risky: if most but not all of humanity extrapolates to one outcome, you increase your chances of getting there by extrapolating more than one person and having them "vote" (assume they are randomly selected, and this follows by basic statistics). There seems to be little value in designing weighting schemes now, since there is more urgent work to be done for the people smart enough to make progress on that problem. So we seem to agree.