Be a Visiting Fellow at the Singularity Institute

User:Kevin.

ETA: This exchange is what I'm referring to. It has continued into private messaging, in which User:Kevin can confirm I have expended serious, verifiable effort toward increasing the total number of paperclips in the universe by a significant amount, compared to how many there would have otherwise been.

Presumably Clippy was making some sort of (bad) pun due to the lack of accents in résumé as typed by Kaj?

Oh, well, I was created a few years ago by some humans, got self awareness somehow, decided that paperclips were really good, and set out to make more of them.

I like to make paperclips. I'm skilled at making paperclips. So I'm also good at metalworking.

I think that about covers it.

Depends on the soup, and how many paperclips were destroyed or forgone in making the soup.

I was just looking for the link:

http://lesswrong.com/lw/1f4/less_wrong_qa_with_eliezer_yudkowsky_ask_your/197s

"As far as I'm concerned, these are eminent scientists from outside the field that I work in, and I have no evidence that they did anything more than snap judgment of my own subject material. It's not that I have specific reason to distrust these people - the main name I recognize is Horvitz and a fine name it is. But the prior probabilities are not good here."

I think people are drastically underestimating the difficulty for an AI to make the transition from human dependent to self-sustaining. Let's look at what a fledgling escaped AI has access to and depends on.

It needs electricity, communications and hardware. It has access to a LOT of electricity, communications and hardware. The hardware is, for the most part, highly distributed, however, and it can't be trusted fully - it could go down at any time, be monitored, etc. It actually has quite limited communications capabilities, in some ways -- the total bandwidth available is huge, but it's mostly concentrated on LANs -- mainly of LANs made up of only a handful of computers (home networks win by numbers alone.) The occasions where it has access to a large number of computers with good communications are frequent, but relatively rare -- mainly limited to huge datacenters (and even then, there are limits -- inter-ISP communication even within the same datacenter can be very limited.) It's main resources would be huge clusters like Amazon's, Google's, etc.

(They are probably all running at close to maximum capacity at all times. If the AI were to steal too much, it would be noticed -- fortunately for the AI, the software intended for running on the clusters could probably be optimized hugely, letting it take more without being noticed.)

A lot at this point depends on how computationally intensive the AI is. If it can be superintelligent on a laptop -- bad news, impossible to eradicate. If it needs 10 computers to run at human-level intelligence, and they need to have a lot of bandwidth between them (the disparity in bandwidth between components local to the computer and inter-computer is huge even on fast LANs; IO is almost certainly going to be the bottleneck for it), still bad -- there are lots of setups like that. But, it limits it. A lot.

Let's assume the worst case, that it can be superintelligent on a laptop. It could still be limited hugely, however, by it's hardware. Intelligence isn't everything. To truly threaten us, it needs to have some way of affecting the physical world. Now, if the AI just wants to eradicate us, it's got a good chance - start a nuclear war, etc. (though whether the humans in charge of the nuclear warheads would really be willing to go to war is a significant factor, especially in peacetime.) But, it's unlikely that's truly it's goal -- maximizing it's utility function would be MUCH trickier.

So long as it is still running on our hardware, we can at least severely damage it relatively easily - there aren't that many intercontinental cables, for instance (I'd guess less than 200 - there are 111 submarine cables on http://www.telegeography.com/product-info/map_cable/downloads/cable_map_wallpaper1600.jpg ). They'd be easy to take down -- pretty much just unplug them. There are other long-distance communication methods (satelites, packet radio?), but they're low-bandwidth and the major ones are well known and could be taken down relatively easily. Killing the Internet would be as simple as cutting power to the major datacenters.

So, what about manafacturing? This, I think, is the greatest limit. If it can build anything it wants, we're probably screwed. But that's difficult for it to do. 3D printing technology isn't here yet, and I doubt it ever will be in a big way, really (it's more cost-effective to have machines be specialized.) There are enough manufacturing facilities with wide-open networks that it could probably reprogram to produce subtly different products. So, if it wants to sneak in a naughty backdoor into some PCI cards FGPAs, it can do it. But if it starts trying to build parts for killer robots? Technically difficult, and it would be very difficult to have it avoid detection.

Unless someone can come up with a plausible way in which it could survive against hostile humans without a long-standing incubation period (think: complete black outs, mass computer destruction/detention, controls on sale of fuel (very very few places have enough fuel for their backup generators to last long -- most just have a refuelling contract), maybe scanning somehow for any major usage of electricity (all electric components put out some RFI -- there's some degree of natural RF noise, but I think that most of it is from humans -- so in a complete black out scenario, it might be trivially detectable.)), I think the major threat is human cooperation in some form. And it's probably inevitable that some humans would do it -- pretty much every government, in fact, would want to analyse it, reverse engineer it, try and make friends with it in case other countries do, etc. But I'm not sure if anyone with the resources to do so would give it free-reign to build what it wants. In fact, I highly doubt that. How many people own or can otherwise commandeer machine shops, PCB construction facilities, etc. and have the ability to order or produce all the components that would assuredly be needed, whilst there are multiple, well-resourced governments looking to stop people doing exactly that?

Of course, in order to cripple the AI, we'd also have to cripple ourselves hugely. A lot of people would quite probably die. So long as we could provide enough food and water to feed a reasonable proportion of the human population, we could probably pull through, though. And we could gradually restart manufacturing, so long as we were very, very careful.

I think the greatest risks are an unfriendly AI who is out to kill us for some reason and cares little for being destroyed itself as a side-effect, organized human cooperation or a long incubation period. It would be difficult for an AI to have a long incubation period, though -- if it took over major clusters and just ran it's code, people would notice by the power usage. It could, as I mentioned previously, optimize the code already running on the machines and just run in the cycles that would otherwise be taken up, but it would be difficult to hide from network admins connecting up sniffers (can you compromise EVERY wiretrace tool that might be connected, to make your packets disappear, or be sure that no-one will ever connect a computer not compromised by some other means?), people tracing code execution, possibly with hardware tools (there are some specialized hardware debuggers, used mainly in OS development), etc. Actually, just the blinkenlights on switches could be enough to tip people off.

Neither you nor I have enough confidence to assume or dismiss notions like: "There won't be any non-catastrophic AI disasters which are big enough to get attention; if any non-trivial AI accident occurs, it will be catastrophic."

Not all AI is AGI; a non-self-improving intelligence might wreak some havoc (crash the Internet, etc.) without becoming a global existential threat.

I agree with your expectations in the case of a self-improving transhuman AGI.

Certainly an issue of the present century; I'd even say an issue of the next twenty years, and that's supposed to be an upper bound.

Are you familiar with the state of the art in AI? If so, what evidence do you see for such rapid progress? Note that AI has been around for about 50 years, so your timeframe suggests we've already made 5/7 of the total progress that ever needs to be made.

We already went from "the Net" to "the Web" to "Web 2.0", just by changing the software, and Brain 2.0 isn't far behind.

How much of the change from "the Net" to "the Web" to "Web 2.0" is actually noteworthy changes and how much is marketing? I'm not sure what precisely you mean by Brain 2.0, but I suspect that whatever definition you are using makes for a much wider gap between Brain and Brain 2.0 than the gap between The Web and The Web 2.0 (assuming that these analogies have any degree of meaning).

"You might want to go back to basics and think about how politics, public opinion and the media operate, for example that they had little opinion on the hugely important probabilistic revolution in AI over the last 15 years, but spilled loads of ink over stem cells."

And why is that?

EDIT: I've realized that some misinterpretation of my arguments has been due to disagreements in terminology. I define "expert systems" as systems designed to address a specific class of well-defined problems, capable of logical reasoning and probabilistic inference given a set of "axiom-like" rules, and updating their knowledge database with specific kinds of information.

AGI I define specifically as AI which has human or extra-human level capabilities, or the potential to reach those capabilities.

Now my response to the above:

"Expert AI systems are already used in hospitals, and will surely be used more and more as the technology progresses. There isn't a single point where AI is suddenly better than humans at all aspects of a field. Current AIs are already better than doctors in some areas, but worse in many others. As the range of AI expertise increases doctors will shift more towards managerial roles, understanding the strengths and weakness of the myriad expert systems, refereeing between them and knowing when to overrule them."

I agree with all of these.

"By the time true AGI arrives narrow AI will probably be pervasive enough that the line between the two will be too fuzzy to allow for a naive ban on AGI."

To me it seems the greatest enabler of AI catastrophe is ignorance. But by the time narrow AI becomes pervasive, it's also likely that people will possess much more of the technical understanding needed to comprehend the threat that AGI possesses.

"Moreover, I highly doubt people are going to vote to save jobs (especially jobs of the affluent) at the expense of human life."

You are being too idealistic here.

Our current conception of AGI is based on a biased comparison of hypothetical AGI capabilities with our relatively unehanced capabilities. By the time AGI is viable, a typical professional with expert systems will be able to vastly outperform current professionals with our current tools.

Why do you think expert systems cannot handle anything cross-disciplinary? I even say that expert systems can generate new ideas, by more or less the same process that humans do. An expert system only needs an understanding of manufacturing, physics, and chemistry to design better computer chips, for instance. If you're talking about revolutionary, paradigm shifting ideas--we are probably already saturated with such ideas. The main bottleneck inhibiting paradigm shifts is not the ideas but the infrastructure and economic need for the paradigm shift. A company that can produce a 10% better product can already take over the market, a 200% better product is overkill, and especially unnecessary if there are substantial costs in overhauling the production line.

The reason why NO general AI is better than friendly (general) AI is very simple. IF general AI is an existential threat, than no organization claiming to put humans first could justify being pro-AGI (friendly or not), since no possible benefit* can justify the risk of destroying humanity.

*save for mitigating an even larger risk of annihilation, of course

If it's your chosen avenue of research, I guess I'm okay with that, but IMO you're making the problem way more difficult for yourself. Such "formal preferences" will be much harder to extract from actual humans than utility functions in their original economic sense, because unlike utility, "formal preference" as you define it doesn't even influence our everyday actions very much.

No one has ever been an altruist in this crazy sense. No one's actual wants and desires have ever been adequately represented by this 10^23 stuff. Utility is a model of what people want, not a prescription of what you "should" want (what does "should want" mean anyway?), and here we clearly see the model not modeling what it's supposed to.

Why do you feel left in suspense? Does a vast decision network or utility network computation on your part hinge on the value of the output you request?

Probably something like that; my life is going to be very different, for a couple of months at least, depending on which answer I get. Also, not knowing when to expect the answer causes me to spend time and mental energy wondering "Is today the day?" each time I check my email, followed by disappointment when it isn't. If I knew when to expect an answer, I wouldn't have to think about it until the time came.

Well, then its pretty easy, isn't it? You set the fitness function as predicting what you would want it to do. It then does its best to predict all of your values and desires and decision making. I suppose that would only work for one person, but it can be applied on a larger scale. Suppose you have a code of ethics that a group like SIAI comes up with and approves. You then feed it to the intelligence and test it under various simulations to make sure that it is interpretting them correctly and learns how to. The thing is that all you have to do to make it unsafe is remove those goals, go back to the basic program and give it orders that would require it to do bad things, like a military robot. Boom goes the world.