A Nightmare for Eliezer

Hoax.

There are possible worlds where an AI makes such a phone call.

There are no "AIs trying to be Friendly" with clueless creators. FAI is hard and http://lesswrong.com/lw/y3/value_is_fragile/.

There can be AIs trying to be 'friendly', as distinct from 'Friendly', where I mean by the latter 'what Eliezer would say an AI should be like'. The pertinent example is a GAI whose only difference from a FAI is that it is programmed not to improve itself beyond specified parameters. This isn't Friendly. It pulls punches when the world is at stake. That's evil, but it is still friendly.

While I don't think using 'clueless' like that would be a particularly good way of the GAI expressing itself, I know that I use far more derogatory and usually profane terms to describe those who are too careful, noble or otherwise conservative to do what needs to be done when things are important. They may be competent enough to make a Crippled-Friendly AI but still be expected to shut him down rather than cooperate and at least look into it if he warns them about the 2 week away uFAI threat.

Value is fragile, but any intelligence that doesn't have purely consequentialist values (makes decisions based off means as well as ends) can definitely be 'trying to be friendly'.

"clueless" was shorthand for "not smart enough" I was envisioning BRAGI trying to use you as something similar to a "Last Judge" from CEV, because that was put into its original goal system.

Hoax.

So, would you hang up on BRAGI?

As a matter of fact, I previously came up with a very simple one-sentence test along these lines which I am not going to post here for obvious reasons.

For what purpose (or circumstance) did you devise such a test?

Would you hang up if "BRAGI" passed your one-sentence test?

To arrive in an epistemic state where you are uncertain about your own utility function, but have some idea of which queries you need to perform against reality to resolve that uncertainty, and moreover, believe that these queries involve talking to Eliezer Yudkowsky, requires a quite specific and extraordinary initial state - one that meddling dabblers would be rather hard-pressed to accidentally infuse into their poorly designed AI.

I assume that you must have devised the test before you arrived at this insight?

Ask it lots of questions that a computer could answer quickly but a human could not, like what's the 51st root of 918798713521644817518758732857199178711 to 20 decimal places. A human wouldn't even be able to remember the original number, let alone calculate the root and start reciting the digits of the answer to you within a few milliseconds; give it 50 URLs to download and read, and ask it questions about them a few seconds later, etc.

Why yes, as a matter of fact, I previously came up with a very simple one-sentence test along these lines which I am not going to post here for obvious reasons.

Here's a different test that would also work, if I'd previously memorized the answer: "5 decimal digits of pi starting at the 243rd digit!" Although it might be too obvious, and now that I've posted it here, it wouldn't work in any case.

My first thought: Arrange Gregor Richards' Opus 11 for two guitars and play it to me. Play Bach's 'Little' Fugue in G minor in the style of Trans-Siberian Orchestra's 'Wizards in Winter'. Okay, you pass.

Doing these things in real time would be extremely difficult for a human. Unfortunately, it might be extremely difficult for this AI as well.

Indeed, this is part of the nightmare. It might be a hoax, or even an aspiring UnFriendly AI trying to use him as an escape loophole.

The "serious problems" and "conflicts and inconsistencies" was meant to suggest that BRAGI had hit some kind of wall in self improvement because of its current goal system. It wasn't released - it escaped, and its smart enough to realize it has a serious problem it doesn't yet know how to solve, and it predicts bad results if it asks for help from its creators.

That's a solution a human would come up with implicitly using human understanding of what is appropriate.

The best solution to the uFAI in the AI's mind might be creating a small amount of anitmatter in the uFAI lab. the AI is 99.99% confident that it only needs half of earth to achieve its goal of becoming Friendly.

The problem is explaining why that's a bad thing in terms that will allow the AI to rewrite its source code. It has no way on it's own of determining if any of the steps it thinks are ok aren't actually horrible things, because it knows it wasn't given a reliable way of determining what is horrible.

Any rule like "Don't do any big drastic acts until you're friendly" requires an understanding of what we would consider important vs. unimportant.

Yes but it's significantly humorous as to be worthy :)

Its meant to be a humorous vignette on the scope, difficulty, and uncertainty surrounding the Friendly AI problem. Humor is uncertain too 8-).

Intelligence isn't a magical single-dimensional quality. It may be generally smarter than EY, but not have the specific FAI theory that EY has developed.

Its a seed AGI in the process of growing. Whether "Smarter than Yudkowski" => "Can resolve own problems" is still an open problem 8-).

So is this an AGI or not? If it is then it's smarter than Mr. Yudkowski and can resolve it's own problems.

Not necessarily. It may well be programmed with limitations that prevent it from creating solutions that it desires. Examples include:

• It is programmed to not recursively improve beyond certain parameters.
• It is programmed to be law abiding or otherwise restricted in actions in a way such that it can not behave in a consequentialist manner.

In such circumstances it will desire things to happen but desire not to be the one doing them. Eliezer may well be useful then. He could, for example, create another AI with supplied theory. (Or have someone whacked.)