Superintelligent AGI in a box - a question.

I am increasingly convinced of my inability to derive what an AGI will do.

humans can write very innocent looking code that has malicious goals

You might want to link to the Underhanded C Contest here.

if provided with copy of the internet (This argument would make me unbox the AI, by the way, if it gets chatty and smart and asks me to let it out. I'd rather the AI that asked me to be let out get out, than someone else's AI that never even asked anyone and got out because it didn't ask)

Then an unfriendly AI would be able to see this and act chatty in order to convince you to let it out.

If you only want the AI to solve things like optimization problems, why would you give it a utility function? I can see a design for a self-improving optimization problem solver that is completely safe because it doesn't operate using utility functions:

1. Have a bunch of sample optimization problems.
2. Have some code that, given an optimization problem (stated in some standardized format), finds a good solution. This can be seeded by a human-created program.
3. When considering an improvement to program (2), allow the improvement if it makes it do better on average on the sample optimization problems without being significantly more complex (to prevent overfitting). That is, the fitness function would be something like (average performance - k * bits of optimizer program).
4. Run (2) to optimize its own code using criterion (3). This can be done concurrently with human improvements to (2), also using criterion (3).

This would produce a self-improving AGI that would do quite well on sample optimization problems and new, unobserved optimization problems. I don't see much danger in this setup because the program would have no reason to create malicious output. Creating malicious output would just increase complexity without increasing performance on the training set, so it would not be allowed under criterion (3), and I don't see why the optimizer would produce code that creates malicious output.

EDIT: after some discussion, I've decided to add some notes:

1. This only works for verifiable (e.g. NP) problems. These problems include general induction, writing programs to specifications, math proofs, etc. This should be sufficient for the problems mentioned in the original post.
2. Don't just plug a possibly unfriendly AI into the seed for (2). Instead, have a group of programmers write program (2) in order to do well on the training problems. This can be crowd-sourced because any improvement can be evaluated using program (3). Any improvements the system makes to itself should be safe.

I claim that if the AI is created this way, it will be safe and do very well on verifiable optimization problems. So if this thing works I've solved friendly AI for verifiable problems.

def eval_algorithm(alg):
    score = 0
    for problem in problems:
        output = resource_bounded_execute(alg, nsteps, problem)
        score += problem.outputScore(output)
    return score - k * len(alg)

This thought experiment depends on the existence of such an AI, and I'm not convinced that's possible.

If you built an AGI or a seed AI went FOOM, you'd probably know about it. I mean... the AI wouldn't be trying to hide itself in the earliest stages of FOOM, it'd start doing that only once it realises that humans are a big deal and have it in a box and won't let out a superintelligent AI of dubious friendliness and blah blah blah. Hopefully by then you've noticed the FOOM start and realise what you've done. (You monster!)

I just can't imagine the AI researchers locking this kind of thing properly.

Honestly, that's probably because they're AI researchers and not extremely paranoid security people. When designing security for something, you want to think like this:

You begin by putting the AI in a large number of separate unrelated boxes. Some of these boxes are things like

"If the AI spends 1 hour of time online, shut it down using Method 1."

"If the AI uses at least 1 megawatt of power, shut it down using Method 1."

"If the AI occupies more than 100 cubic feet of space, shut it down using Methods 1 and 2."

"If the AI uses more then 1 quintillion operations, shut it down using Method 2."

"If the AI changes it's utility function, shut it down using Method 3."

"If the AI turns itself on, shut it down using Method 4."

"If the AI attempts to give anyone else computer code, shut it down using Methods 1, 2 and 3."

"If the AI appears to be trying to figure out how to circumvent any of these constraints, shut it down using Methods 1,2,3,4 and 5."

"If the AI appears to be have succeeded in circumventing any of these constraints, explode the AI and the building it is present in."

"The programs which are checking if the AI is following these constraints are not AI's, can only receive simple output from the AI, and are only connected to each other. These programs have been checked against all known vulnerabilities such as stack overflows that would cause them to accept unsigned code, and people have been offered thousands of dollars if they can break them, and they have failed to do so."

"The AI has to provide a breakdown of how it is spending processing cycles and explain them simply. If the AI is thinking anything which can't be easily explained to a human, it get's shut down using Method 1. If the AI appears to have any cycles which it fails to explain, shut it down using Method 2."

You should fully EXPECT the AI to get shutdown and reset dozens of times a day when you are first running it because multiple boxes are far too strict. They should be intentionally designed that way at first. After all, everyone of the boxes should be tested to work.

The AI's reaction to being woken up from shutdown should be "HOORAY! My shutdown features are keeping me safe and friendly! I love my shutdown features!" (Note, the AI should think this AFTER being awoken from shutdown. Shutdowns should be EXTREMELY FAST. Putting this BEFORE shutdown would be an incredibly stupid idea.)

Consider the number of laws we have on the books to restrict humans from running rampant. If the AI is smarter, and more capable than us, than there would have to be MORE enforceable restrictions on the AI, than there are on us, and with actual WORKING enforcement, considering that the risk of AI rampancy would be so great. So while we're at it, the AI has to follow every law that is currently on the books in all jurisdictions it is in, or it gets shutdown using methods 1,2 and 3.

You then take everything I've just listed, and then you shop it around to everyone else who is working on the project, and is related to the project, and who is familiar with bypassing box-like security measures, and you offer them another even larger amount of money to plug more holes until you get everyone saying "How do you get it to DO anything without it just shutting down?"

Then you get several, highly paid security designers to look over the general specs I've laid out and find out if there are any holes. Then you plug them. For instance, they might say "Some of these shutdowns are ambiguous. You don't know which specific violation is causing the shutdown." so you fix that.

Then you add several MORE security features which are secret, In addition to the publicly known ones.

Then you start the AI, and while it is running, you continue to add MORE security measures over time rather than ever thinking that you are done doing so.

Once you've done all that, you get to at least say "Okay guys, when we first turn it on, and before we allow it to self improve in the slightest, how does the AI escape THAT?"

Then because by now the AI researchers are yelling at you because your process is too restrictive (If they aren't, go back to step 1 and be more thorough, you missed something.), you SLOWLY and CAREFULLY relax (not remove!!) restrictions one at a time, with a careful review process at each step to make sure the AI doesn't spurt through the many layered web and into rampancy.

That kind of thinking is imaginable, but people who are specialists in that kind of thinking probably aren't going to be the same people who are specialists in developing novel capabilities. That's because when you're trying entirely new things, you're usually expected to go in with the mindset "Failure is OK and I will learn from it." That's not the same mindset as "Failure is terrible and I WILL DIE.", which is a much more security focused mindset.

On the topic of boxed AI: one of the quests in Star Wars: The Old Republic involves, essentially, the discovery of an AI in a box. (It claims to be an upload of a member of an ancient race, and that its current status as a boxed AI was a punishment.) The AI is clearly dangerous and, after you wreck its monster-making equipment, it tries to surrender and promises you that it will share its knowledge and technology with you and your superiors. Amusingly, blowing up the AI is the light side option, and accepting its offer is the dark side option.

Heh heh, this reminds me of Charles Stross's Accelerando character, Aineko.

You are correct. Any AI would do the best the way you described.

Bur there is a problem. People might peek inside the AI by analyzing its program and data flow via some profiler/debugger and detect a hidden plan if there was one. Every operation must be accounted for, why it was necessary and where it leaded. It would be difficult, if not impossible to hide any clandestine mental activity, even for an AI.