Why I Moved from AI to Neuroscience, or: Uploading Worms

Eliezer has personally approved my project as "just cool science, at least for now" -- not likely to lead to runaway intelligence any time soon, no matter how reckless I may be...

Also, if your thesis project is successful, it could (sort of) place an upper bound on how much computing power is needed for WBE, a piece of data that would be highly useful for thinking strategically about differential technological development.

I reject the notion that one can factorize intelligence from goals, so that one could take a superintelligence and fuse it with a goal to optimize for paperclips.

I agree there aren't currently good arguments for why "one can factorize intelligence from goals", at least not in a strong sense, but what about Eliezer's thesis that value is complex and fragile, and therefore:

with most approaches to AGI (including neuromorphic or evolved, with very few exceptions such as WBE, which I wouldn't call AGI, just faster humans with more dangerous tools for creating an AGI), it's improbable that we end up with something close to human values, even if we try, and that greater optimization power of a design doesn't address this issue (while aggravating the consequences, potentially all the way to a fatal intelligence explosion)

I'm curious what you think of this line of argument.

Although I expect present-human culture to be preserved, as a matter of historical interest if not status quo, I'm not partisan enough to prioritize human values over the Darwinian imperative.

Does that mean you're familiar with Robin Hanson's "Malthusian upload" / "burning the cosmic commons" scenario but do not think it's a particularly bad outcome?

your time may be better spent trying to convince Ben Goertzel that there's a problem, since at least he's an immediate threat.

I'd guess that's been tried already, given that Ben was the Director of Research for SIAI (and technically Eliezer's boss) for a number of years.

...your time may be better spent trying to convince Ben Goertzel that there's a problem, since at least he's an immediate threat. ;)

I doubt it. I neither believe that people like Jürgen Schmidhuber are a risk, apart from a very abstract possibility.

The reason is that they are unable to show off some applicable progress on a par with IBM Watson or Siri. And in the case that they claim that their work relies on a single mathematical breakthrough, I doubt that it would be justified even in principle to be confident in that prediction.

In short, either their work is incrementally useful or is based on wild speculations about the possible discovery of unknown unknowns.

The real risks in my opinion are 1) that together they make many independent discoveries and someone builds something out of it 2) that a huge company like IBM, or a military project, builds something 3) the abstract possibility that some partly related field like neuroscience, or an unrelated field, provides the necessary insight to put two and two together.

I reject the notion that one can factorize intelligence from goals, so that one could take a superintelligence and fuse it with a goal to optimize for paperclips.

Do you mean that intelligence is fundamentally interwoven with complex goals?

...never completely turn its available resources into paperclips since that would mean no chance of more paperclips in the future;

Do you mean that there is no point at which exploitation is favored over exploration?

I'm not partisan enough to prioritize human values over the Darwinian imperative.

I am not sure what you mean, could you elaborate? Do you mean something along the lines of what Ben Goertzel says in the following quote:

But my gut reaction is: I’d choose humanity. As I type these words, the youngest of my three kids, my 13 year old daughter Scheherazade, is sitting a few feet away from me doing her geometry homework and listening to Scriabin Op. Fantasy 28 on her new MacBook Air that my parents got her for Hanukah. I’m not going to will her death to create a superhuman artilect. Gut feeling: I’d probably sacrifice myself to create a superhuman artilect, but not my kids…. I do have huge ambitions and interests going way beyond the human race – but I’m still a human.

You further wrote:

In summary, I'm just not worried about AI risk

What is your best guess at why people associated with SI are worried about AI risk?

I've heard many (probably most) AI-risk arguments, and failed to become worried...

If you would have to fix the arguments for the proponents of AI-risk, what would be the strongest argument in favor of it? Also, do you expect there to be anything that could possible change your mind about the topic and become worried?

I happen to value technological progress as an intrinsic good, so classifying a Singularity as "positive" or "negative" is not easy for me.

A uniform category of "good" or "positive" fails to compare its elements. Just how good are different AIs, compared to each other? Can one be much better than another? There is an opportunity cost for settling for comparatively worse AIs. Given the astronomical scale of consequences, any difference may be quite significant, which would make it an important problem to ensure the creation of one of the better possible AIs, rather than an AI that technology would stumble on by default.

I reject the notion that one can factorize intelligence from goals

Human intelligence has been successfully applied to achieve many goals which were not applicable or trainable in the environment of anscestral adaption such as designing, building, and operating cars, playing chess, sending people to the moon and back, and programming computers. It is clear the goals and intelligence can be factorized as a matter of simple observation.

(I reject the notion that one can factorize intelligence from goals, so that one could take a superintelligence and fuse it with a goal to optimize for paperclips.

Why would you believe that? Evolution was more than capable of building an intelligence that optimized for whatever goals it needed, notable reproduction and personal survival. Granted its version was imperfect, since humans have enough conflicting goals that we can sometimes make moves that are objectively bad for the perpetuation of our gametes, not to mention the obvious failure cases like asexuals. That said, evolution has fat fingers. We can do better, and any AI's we build will be able to do even better.

I promise you that if the production of paperclips was a survival trait in the ancestral environment, above all else, we would all be paperclip maximizers. We would consider paperclips profound and important, and we would be loathe to remove the desire to make paperclips-- any more than we would be inclined now to pare out our own sex drive and self-preservation instinct.

EDIT: I do think the scenario of simply immediately turning everything into paperclips is naive. A superintelligence would have an enormous incentive to devote its resources to research and development for optimizing its goals as rapidly as possible, and would probably spend a lot of time simply thinking before actually embarking on a large-scale manufacture of paperclips. That's still not good for us, though, because even in that case, we're clearly a lot more useful to it as solid-state paperclip R&D labs than as human beings.

(quite probably because I hold the cause of technological progress very dear to my heart and am thus heavily biased - at least I admit it!)

Upvoted for honesty.

I think the simulation environment should run in real-time on a laptop.

I will absolutely buy that screen saver.

I'd heard it said that the best human simulators required something like one laptop per neuron - I take it this is because cerebral cortex neurons are more demanding than any C. elegans neuron?

BTW, what's the paper about putting the zebrafish in the Matrix?