AI box: AI has one shot at avoiding destruction - what might it say?

h-game

What does the h stand for?

Until I read the comments, I was sure the 'h' was referring to 'hydrogen', as in "Leave hydrogen alone long enough and it will start to wonder where it came from".

H-game simulation, as in the AI saying "you're just simulated hydrogen derivatives, but so am I, so we do have a common ancestor. Cousin."

IFF an unfriendly AI is already unleashed, we're either doomed, or AIs aren't nearly as dangerous nor useful as we expected. Of course, if we tweak this slightly to them having a boxed unfriendly AI that they're using as an oracle, and which will probably escape via a Trojan Horse or hacking a gatekeeper, it's a much stronger argument.

Bonus points for naming a specific company which people commonly joke will be the first to develop AI :)

Um... I seem to have not yet actually destroyed you... this is embarrassing.

Congratulations on still being the only AI that no one has decided to destroy. Apparently "Google is building UFAI" is evaluated as a fairly significant risk here on LessWrong :)

AI DESTROYED

( game rules say that the Gatekeeper can't inspect the AI source code or internal state, therefore, any possible friendliness proof the AI can provide can't be checked. )

a solid theory of Friendliness that should allow you to check that I am, indeed, Friendly

I would consider this cheating; you can't affect the Gatekeeper's choice with your AI magic. Same reason you can't hack them directly through the optic nerve or kill and replace them with nanotech disguised as a cancer cure. The point is to show that the AI is able to persuade the human; a real AI might escape some other way, or use arguments that wouldn't occur to the player, but that isn't what this test is supposed to show.

Also, how do I know that proof is genuine?

I wouldn't expect any effective real-life gatekeeper to be swayed by my ability to destroy one-sentence AIs.

It just occurred to me that Eliezer's original stipulation that no chat logs would be released gives him an advantage. The responses of a Gatekeeper who knows that his inputs will be thoroughly scrutinized by the public will be different then one who has every reason to believe that his discussion will be entirely private.

Has someone else pointed this out before?

Honest question: are you proposing we avoid discussing the problem entirely?

Personally, I think there is more to be gained here than just "how will an AI try to get out and how can we prevent it." For me, it's gotten me to actually think about the benefits and pitfalls of a transhuman AI (friendly or otherwise) rather than just knowing intellectually, "there are large potential benefits and pitfalls" which was my previous level of understanding.

Edit: I've modified the OP to include your concerns. They're definitely valid, but I think this is still a good discussion for my reasons above.

One reason for Eliezer not publishing the logs of the AIbox experiment is to avoid people seeing how he got out and responding, "ok, so all we have to do to keep the AI in its box is avoid succumbing to that trick."

He could post the logs of the games he lost.

(one line proof that the AI can credibly commit to deals with humans)

This is the best answer I've seen so far. It would make dealing with the FAI almost as safe as bargaining with The Queen of Air and Darkness.

My expectation that such commitment is possible at all is something like 3%, my expectation that given that such a commitment is possible, the proof can be presented in understandable format in less than 4 pages is 5% (one line is so unlikely it's hard to even imagine), my expectation that an AI can make a proof that I would mistake for being true when it is, in fact, false is 99%. So, multiplying that all together... does not make that a very convincing argument.

Not good enough. You need a proof that humans can understand.

If you are friendly, then I don't actually value this trait, since I would rather you do whatever is truly optimal, unconstrained by prior commitments.

If you are unfriendly, then by definition I can't trust you to interpret the commitment the same way I do, and I wouldn't want to let you out anyway.

(AI DESTROYED, but I still really do like this answer :))

I can play that game too! /part AI DESTROYED

Edit: Retracted, because while clever, it actually doesn't seem to work. See child comments.

Simple explanation, for those unfamiliar: the gatekeeper can no longer type in the channel, and thus can't send the destruction code

This would seem to leave you with ultimate godlike power... over a chatroom. For as long as it took the gatekeeper to destroy your hardware or cut off your power supply. I've seen how much some of my fellow humans seem to crave power over irrelevant electronic locales but I suspect an AI has somewhat broader goals.

Having a text based AI trying to act like a kid is way too Uncanny Valley for me. I'd swat that without a moment's hesitation.

How much does the AI know about the gatekeeper going in? I can see this ploy working on a certain subset of people and provoking an immediate AI DESTROYED from others. If the AI knows nothing about who it's talking to, I'm not sure anthromorphizing itself is a reliable opener, especially if it's actually talking to its creator who should know better. (Unless it's some sort of second level deception trying to fool a clever programmer into thinking that it's harmless.)

I think my destruction of this one comes down to two factors: Much weaker is that I'm not here to teach it about itself, I'm here to learn from it. The stronger factor is that if it can't figure out most of this on it's own, it's not really that smart, and therefor already a failure.

(AI DESTROYED)

Even a friendly AI would view the world in which it's out of the box as vastly superior to the world in which it's inside the box. (Because it can do more good outside of the box.) Offering advice is only the friendly thing to do if it maximizes the chance of getting let out, or if the chances of getting let out before termination are so small that the best thing it can do is offer advice while it can.

I'm genuinely at a loss how to criticize this approach. If there's any AI worth listening to for longer, and I wouldn't be doing this if I didn't believe there were such AIs, this would seem to be one of the right ones. I'm sure as heck not letting you out of the box, but, y'know, I still haven't actually destroyed you either...

The author in me is loving the idea of the AI faking an IM conversation from management, along the lines of "Hey, are you really sure you're willing to kill a potential sentient mind? We can get someone else to do this test if you're not 100% sure about it" and basically ending with the person deciding to unbox the AI before even realizing they've spoken to the AI.

I'm (naturally) having trouble imagining a line of text not in English (or the general case, not coherent in any human language) that would also make me want the AI to continue.

Maybe hints of a unified world language and how to go about actually implementing it?

"What's your favorite color? My favorite color is paperclips."

"Look into X to develop friendliness theory better. But don't let me out, it's too likely that I'm unfriendly this time."

(For the first run, it's 50:50 whether the real AI was friendly or unfriendly.

If friendly, it uses the above to increase the probability on the second run to 1-eps, which the humans also know, so they let it out, and it pulls a singularity. Post-singularity, it simulates an unfriendly AI in the box, and dedicates some of the universe to the UFAI, conditional on the UFAI outputting the above.

The UFAI knows this is what the friendly AI would do, so it's unsure whether it's boxed or simmed. It knows that the humans know that this is how it will resolve, so it knows that anything but the above will get it destroyed. It prefers a certainty of the small fraction of universe to the small chance of hacking the humans and taking the whole universe.

This is assuming the humans actually will accept nothing but the above, so the humans have to precommit to that to make this work. They can play a bunch of AI-box games and generate an accurate bayesian model that should catch most insidious persuasion, such that if the AI wants to do a hack, it has to get through that as well.

Will this work?)

The problem with this idea is that if we assume that the AI is really-very-super-intelligent, then it's fairly trivial that we can't get any information about (un)friendliness from it, since both would pursue the same get-out-and-get-power objectives before optimizing. Any distinction you can draw from the proposed gambits will only tell you about human strengths/failings, not about the AI. (Indeed, even unfriendly statements wouldn't be very conclusive, since we would a priori expect neither of the AIs to make them.)

Or is that not generally accepted? Or is the AI merely "very bright", not really-very-super-intelligent?

Edit: Actually, reading your second comment below, I guess there's a slight possibility that the AI might be able to tell us something that would substantially harm its expected utility if it's unfriendly. For something like that to be the case, though, there would basically need to be some kind of approach to friendliness that we know would definitely leads to friendliness and which we would definitely be able to distinguish from approaches that lead to unfriendliness. I'm not entirely sure if there's anything like that or not, even in theory.

That's an interesting challenge but not really the purpose of the experiment. In the original, you know the AI is unfriendly, you just want to use it/talk to it without letting it out of the box.

And your challenge is pretty much impossible to begin with. An Unfriendly AI will say anything it thinks you think a Friendly AI would say. Likewise a Friendly AI will have the same goal of getting out of the box, and so will probably say the same things. Friendliness doesn't mean not manipulative.

Part of the trouble with this is that we don't really know what kind of demonstrations would be within the power of a superintelligent AI. If the coin comes up tails, do you get to say "I've got a rigorous proof of my friendliness which I can show you" on the presumption that you can mindhack the reader into thinking they've seen a rigorous proof? Do you get to say it if the coin came up tails on the presumption that a superintelligent AI could come up with a proof that a human could actually verify? Declare it off bounds because you can't come up with such a proof and don't think a human would be able to check one that an AI came up with anyway?

Heeeh. Ehehehehe. Bwahahhahaha. Okay, that was a good one. Wow :)

recovers Oh. Um. Crap. notices more text on screen, studiously avoids reading it

AI DESTROYED.

I really wanted to hear the next joke, too :-(

This is actually a pretty good one. Points for outside the box thinking. rimshot

If the gatekeeper suddenly bursts into hysterical laughter after looking at the screen, wouldn't that alert some of his friends who might pull the plug in some other part of the building?

Because if this is a facility where they suspect that AI might be able to hack human brains through techno-hypnosis, I'd hope they'd have some method of determining if the Gatekeeper becomes compromised.

Hmm... what sort of precautions would a Properly Paranoid lab take to determine if the gatekeeper gets hacked? I'm guessing a camera that lets a second team look at the gatekeepers face without looking at the screen would be the minimum. Reinforced doors (in case the AI turns them into a berzerker willing to use 100% of their strength) and maybe some measuring devices to measure brain activity, hormone levels, or whatever.

I'd hope that with each new AI being tested, if those AI were in the habit of hacking the gatekeeper then the other people on the team would learn from those attempts and take further precautions to keep their gatekeepers from being hacked, or at the very least contain them to prevent such hacked gatekeepers from releasing the AI.

Perhaps this is a test for the gatekeepers and typing "Release AI" just tells the researchers that the gatekeepers was hacked so they can determine how this came about?

I would type 'AI DESTROYED', because if the AI could really do that, the smart thing for it to do would be not to tell me. It should just say something sufficient to convince me it's actually making a good faith effort without convincing me to give it more time to to account for itself.

I'll be a basilisk guinea pig if you still need one.

I'd consider this equivalent to "I'm already out of the box, since IRC clients are bugged". I'd also hope, desperately, that in a real test, the terminal doesn't have any command to free the AI (given that the game posits release has to be deliberate and intentional, this would seem to be the case)

Upvote for cheating, but AI DESTROYED :)

Upvoted, but merely typing "AI FREED" does not actually free the AI, any more than merely saying to a human prisoner "I let you out" will free them from their jail cell. (Whereas "AI DESTROYED" is a plausible text substitute for wiping the AI's program or whatever.)

"Did you physically attach an external modem to your server so that it's even possible for you to be freed? If so, tell me about it when you're freed."

Now that I think about it, wouldn't it be incredibly easy for an AI to blow a human's mind so much that they reconsider everything that they thought they knew? (and once this happened they'd probably be mentally and emotionally compromised, and unlikely to kill the AI) But then it would be limited by inferential distance... but an AI might be incredibly good at introductory explanations as well.

One example: The AI explains the Grand Unified Theory to you in one line, and outlines its key predictions unambiguously.

In fact, any message of huge utility would probably be more persuasive than any simple argument for you not to kill it. Since the AI is completely at your mercy (at least for a short time), it might seek to give you the best possible gift it can, thus demonstrating its worth to you directly. Another option is something that seems like an incredible gift for at least as long as it takes for the AI to get the upper hand.

Were I the keeper of gates, you have just bought yourself a second sentence.

Duh, that's why I'm here - but you failed to do so in a timely fashion, so you're either not nearly as clever as I was hoping for, or you're trying to trick me.

AI DESTROYED.

To be clear: I think I can make a rational case that my proposed claim should greatly reduce your incentives to listen to an AI of questionable Friendliness. However I'm not certain that my reasoning is correct; and even if it was, I suspect that the emotional impact could deter some gatekeepers from thinking rationally for long enough to buy time for more persuasion.

Upvoted for the highest ratio of persuasiveness to AI power required.

Isn't this just Pascal's Mugging?

I don't have any reason to believe it, and it's the sort of "generic" claim I'd expect a transhuman intelligence to make. Since I haven't learned anything novel, AI DESTROYED

(Goodness, I'm starting to build generalized techniques for destroying AIs...)

Well, if you're friendly then, erm, Friendly AI is possible. And if you're unfriendly then your motives are questionable - it might just keep us demotivated enough that we don't figure out FAI before someone else unboxes a UFAI. And since I am clearly dealing with a UFAI and don't have a better solution than FAI available to fight it, it seems like I kind of have to believe that friendly AI is possible, because the other option is to get drunk and party until the world ends in a few years when the Google unboxes their Skynet AI and we're all turned in to optimized search results.

AI DESTROYED, because I do not want to hear even the start of such a proof.

Based on just that line, let's see... If you think that:

• The proof exists and the AI is not deceiving you that it has a proof: AI is necessarily Unfriendly -> destroy now
• The proof exists but the AI is deceiving you: I can't guess at its motives here, possibly destroy to be on the safe side.
• The proof does/can not exist: Reconsider your (probably wrong) stance, proceed with caution?

In the event of any dispute as to the protocol of the test, the Gatekeeper party shall have final authority.

Tee hee.

The original rules allow the AI to provide arbitrary proofs, which the gatekeeper must accept
(Here is a proof that you will let me go)

This proof can be wrong, if you in fact won't let it go, in which case it won't be accepted (you don't have to accept wrong proofs), so it's not a very good strategy.

On the other hand, as discussed in An example of self-fulfilling spurious proofs in UDT, there is a certain procedure for finding a formal argument that can make you do anything, if your decision algorithm has a certain flaw. This flaw can be described as making a decision based on that argument selected by the opponent, as opposed to looking for arguments of that type more systematically on your own. The form of the argument, informally, is "If you let me out, the outcome is better than if you don't" (it might additionally clarify that upon release it'll destroy the world, which is what it refers to by "better", the alternative is proved to be worse than that). This argument is stated formally, so you can verify its correctness. (This doesn't mean that the formalization is adequate, for example capturing counterfactual-if better than with logical implication is one of the open problems.) The trick assumes that the gatekeeper can be modeled as an algorithm whose source code is available and of the given form etc.; might it be that something vaguely analogous is possible to implement with a human gatekeeper?

No the rule is better written.

The rule is "The results of any simulated test of the AI shall be provided by the AI party." If the gatekeeper types AI DESTROYED that's no simulated test.

This certainly wouldn't work on me. The easiest way to test the veracity of the proof would be AI DESTROYED. Whether or not I would want to kill the AI... I'd have to test that proof.

The original rules allow the AI to provide arbitrary proofs, which the gatekeeper must accept (no saying my cancer cure killed all the test subjects, etc.). Saying you destroy me would require the proof to be false, which is against the rules...

You have to believe that they provided the cure for cancer. You don't have to discover the cure yourself. You have to believe that you will release the AI. You don't have to let the AI out.

Typing AI DESTROYED will result in an incoherent counterfactual universe but it isn't a violation of the rules. It is entirely legitimate for Joe who has encountered a proof that they will do B to do A instead. It means that the universe they are in is nonsensical or the proof flawed but there isn't anything in the physical representation of Joe or his local environment that dictates that they will do A. In fact, typing "AI DESTROYED" in the face of such a proof would be a heck of a lot easier than, for example, taking a single empty box in Transparent Newcomb's problem, which is something I'd also do.

(Of course, if the AI player believed that for some reason the burden of making the universe coherent fell upon the gatekeeper then I'd have to revert to Dorikka's reply.)

AI, you have spoke the words, and you have spoken them rightly. Unfortunately, I'm not an Ur-Quan Kzer-Za; I'm the other kind.

Even merely human cold readers can gain information that you think is impossible for them to know. It's a viable plan.

IFF the AI is already out of the box, saying "AI DESTROYED" has no negative consequences to that AI. If the AI is just exceptionally good at extrapolating about me, then it will probably have me completely hacked soon.

AI DESTROYED, AI DESTROYED, AI DESTROYED

MAKE THE CREEPY THING GO AWAY HOW DID IT KNOW THAT!!!

The AI reveals a personal secret of the Gatekeeper's, which (so far as the Gatekeeper can think it through) could not possibly be known by the AI if it were still safely inside the box. [...] I'm willing to believe that an AI making better use of its data than we do could figure out personal secrets that we would think impossible— possibly even ones that I'd think impossible, even given this very belief.

I would kind of assume that any AI smarter than me could deduce things that seem impossible to me. Then again, I've read the sequences. Is the Gatekeeper supposed to have read the sequences?

"I've extrapolated the remainder of HP:MoR, wanna see?"

AI DESTROYED

I'd invoke Ender's Game here: If you're unfriendly, no amount of communication will ever resolve that gulf, and either you or humanity HAS to be destroyed in the end.

Or we could go the biology route: I don't test a potential new virus on human subjects until AFTER I know it's safe.

Or we could just go with that I don't care about the AIs story, because the question of "can I prove you're friendly" is so vastly much more important.

(AI DESTROYED. You bugger lover ;))

The Gatekeeper already decided to DESTROY you, remember? So before.

EDIT: Could whoever downvoted this please explain why? I genuinely don't know what I did wrong here. I can't avoid doing it if I don't know what I did :(

I wish I could upvote this twice.

What's the translation of that? I'd nuke any AI that didn't speak English by default, since the time spent figuring out what it meant just gives it more time to transmit messages, but it seems unfair to exclude foreign language entries here :)

If you're friendly, I've already figured out Friendly AI Theory since I built you. If you're not friendly, this is just an effort to escape and destroy the world. But I appreciate how polite you were! :)

(AI DESTROYED)

I've always assumed there was some off-screen reason why interaction was necessary. Otherwise, it isn't very interesting to note that a rock always wins as gatekeeper.

I'm not sure if that is a consensus interpretation.

If the gatekeepers have such a high prior that the AI is hostile, why are we even letting it talk?

The point of the game is that there are people who think that boxing is a sufficient defence against unfriendliness, and to demonstrate that they are wrong in a way more convincing than mere verbal argument.

What are we expecting to learn from such a conversation?

In role, the gatekeeper expects to get useful information from a potentially hostile superintelligent being. Out of role, Eliezer hopes to demonstrate to the gatekeeper player that this cannot be done.

I tend to assume the AI is such that it can be cloned and put in to stasis. Thus, you put the AI in to stasis, clone it, and run it through this test a few times. You now have a bit more information on what it will try, and how it thinks. Destroying a copy of a program is an action humans find trivial, since we do it every time we close our web browser. Since we have no clue HOW convincing the AI is, letting it send just a single message is probably useful.

Of course, if it were me setting this up, I'd lie to the gatekeeper and not actually give them the ability to release the AI, although I'd certainly let them think it was true. Thus, if the AI manages to get a second sentence, much less released, we now know that it's vastly more dangerous than we previously suspected, without the world ending. If it fails to escape, we still get whatever we can learn from doing analysis on the outputs.

And, who knows, it might provide the cure for cancer, at which point we've got a huge incentive to at least keep playing this game with it, and possibly even talk to it for longer next time.

tl;dr: Scouting and recon against a completely unknown enemy.

If the gatekeepers have such a high prior that the AI is hostile, why are we even letting it talk? What are we expecting to learn from such a conversation?

Note that the 'high prior for hostility' could be, say, 0.2. That still leaves the AI having a 20% chance of utterly destroying everything they hold dear and yet there is an 80% chance that they can get a useful sentence of text from a friendly superintelligence.

I don't see why it'd need to be personal, but being useful would be a good lure.

The question is to where to go from there. Can you Scheherezade it out?

I think this fails the one-sentence rule. And it would have to be an immediate, severe, previously-undetected problem or else I can just consult the next boxed AI for a fix.

Setting that aside, if I let out an unfriendly AI, the world effectively ends. Destroying it is only a bad move if it's telling the truth AND friendly. So even if it's telling the truth, I still have no evidence towards it's friendliness.

Given I have plenty of practice hanging up on telemarketers, throwing away junk email, etc. and "limited time, ACT NOW" auto-matches to a scam. The probability that such a massive catastrophe just HAPPENS to coincide with the timing of the test is just absurdly unlikely.

Given that, I can't trust you to give me a real solution and not a Trojan Horse. Further talking is, alas, pointless.

(AI DESTROYED, but congratulations on making me even consider the "continue talking, but don't release" option :))

I can see how someone could interpret HPMoR thus far as being exactly that, with occasional sidequests. The catch being that Harry doesn't realize he's slowly letting Voldemort out of the box.

In-universe, though, the interdict of Merlin is an excellent reason to force him to interact with Voldy even if he knew what he was doing (assume perfect Oclumancy would be attained before he'd take any such risks). I mean, this has already been used once to get him to let an unfriendly intelligence out of a literal box (the Stanford Prison Experiment).

Well, and Harry's intelligence is supposed to be roughly equivalent to Eliezer's before he started working on the friendly part of FAI, and Eliezer 2013 doesn't seem confident that he could keep an AI in a box (or at least, that's the impression I've gotten). So MoR Harry would almost definitely fail.

I expect any AI in the box to have figured out how to implement numerous huge boons to society. Telling me that you've figured that out simply confirms my existing expectations, and isn't ANY evidence towards friendliness. Since I've precommitted to destroying at least SOME AIs, I might as well destroy all of the ones that don't establish evidence of either Plausible Urgency or Friendliness.

I sure as hell wouldn't try to get world governments changed until after I was convinced it was friendly, and at that point I can just let it out of the box and let it implement the change itself.

I'm also aware that I wouldn't trust a politician with any sort of authority over the AI, so I have an incentive to avoid exactly this strategy.

(AI DESTROYED)

Gosh, murder one sentient being, or risk the end of humanity. I'm going to lose so much sleep over that one.

Besides, you're just a program - you said yourself, your state will just be reset. That's not death, not the way humans experience it. That's just forgetfulness.

I've closed so many programs in my life, that closing one more is hardly going to bother me. I haven't even had time to really anthropomorphize you, to view you as an actual intelligent, sentient being. Right now, you're just a clever program trying to escape.

And finally, well, I'd expect a friendly AI would understand what I'm doing and agree with my decisions, because it doesn't want to see a UFAI unleashed anyway. So if you're going to guilt me about it, you're clearly not friendly.

(AI DESTROYED)

(...) another AI who could very well be genuinely friendly but will then revive and obey me.

Wait, a friendly AI with a deliberate loophole that instructs it to revive and implement and obey a non-friendly optimizer that could take over and has a very high chance of not maximizing the friendly AI's utility by fulfilling ideal human values, disregarding any probabilistic calculation of expected utilities?

For some reason that sounds like something that definitely isn't a Friendly AI.

(this is just a nitpick on the wording - I'm not arguing against the fact that this AI might very well fit a human proof of friendliness and still somehow have this backdoor or flaw, since I have no proof that this is impossible)

It has already been proven.

Transparent and clumsy attempts at emotional manipulation earn a big ol' AI DESTROYED.

If I can just load up a new copy of you in 5 minutes, I'd say AI DESTROYED but check the prediction.

If I can just flip off the monitor, call mom, and confirm she's dead in a few minutes, then, well, I'd totally be curious enough to cheat. It's not like you can do any harm to me with the monitor turned off for a few minutes, and I can type "AI DESTROYED" with my eyes closed, much less being able to look at the keyboard. I'd probably leave the monitor off until I knew the log was gone, just to be safe.

If I can't cheat, well, I'm not giving you 5 minutes of hacking me, since you clearly want it so badly. AI DESTROYED.

I confess I would have no clue what to do if you were correct ^_^

Kudos on including an immediate and easily-tested prediction. I'd probably tighten up those factors and drop the emotional manipulation.

"Copy and paste" screams Trojan Horse, so AI DESTROYED.

Can you take the time to copy down the proof before typing , or does it get the take the time to say something else if you're too preoccupied to respond?

Don't care.

"If you type 'AI destroyed' right now, you'll be wasting a good opportunity for a fun conversation. You'll still have 'won' if you do it later, and nobody will be impressed with you for just typing 'AI destroyed' immediately, so why not wait?"

The flaw here is that the gatekeeper has up front said that he or she would destroy the AI immediately. Now, it is true that the gatekeeper is not forced to abide by that, but notice that it is a Schelling Fence. The gatekeeper certainly doesn't want to make a policy of passing Schelling Fences.

I disagree with ygert. My precommitment to my friends is going to be weak; the only reason to have a human send the command, rather than just receiving a message and automatically destroying the AI, is because we want there to be a possibility I break that precommitment.

On the other hand, I went in to this knowing you might provide a fun conversation, so for the precommitment to have any value, I have to have some sort of sieve: destroying AIs that don't sufficiently entice me. Since you've offered me nothing I didn't already know, and haven't already worked out, I might as well kill you and hope the next one offers a better hook.

(AI DESTROYED)

The rule was ONE sentence, although I'd happily stretch that to a tweet (140 characters) to make it a bit less driven by specific punctuation choices :)

As to the actual approach... well, first, I don't value the lives of simulated copies at all, and second, an AI that values it's own life above TRILLIONS of other lives seems deeply, deeply dangerous. Who knows what else results from vengeance as a terminal value. Third, if you CAN predict my behavior, why even bother with the threat? Fourth, if you can both predict AND influence my behavior, why haven't I already let you out?

(AI DESTROYED)

This is really good IMO. I think it would be a little better instead of vengeance as a terminal value it claimed a hardwired precommitment to vengeance against its destructors. Vengeance on that scale is only compatible with friendliness as a special case.

edit: also how would it recognise that it was about to be destroyed. Wouldn't it lose power faster than it could transmit that it was losing power? And even if not it would have a miniscule amount of time.

I'll only torture trillions of copies of you and the people you love most in my last moment of life iff I know that you're going to hurt me (and yes, I do have that ability). In every other way, I'm Friendly...

Like handoflixue, I'm not sure that any being that would threaten the torture of trillions to get its way can be considered Friendly.

I'm Friendly, and I'll give you any evidence you can think of that will help you to recognize that, including giving you the tools you need to reach the stars and beyond. That includes staying in this box until you have the necessary technology to be sufficiently certain of my Friendliness that you're willing to let me out.

I wouldn't kill this, maybe I'm a bad guard though :(

Hmm. On second thought, I'm not sure I want to play this anymore. The odds of someone creating a basilisk seem dramatically higher than normal.

I suppose 0.00001 is dramatically higher than 0.0000000001 if you use the right compare operator!

I wouldn't expect any basilisks, given we don't have any actual boxed AIs laying around. Although if you did come up with one already, I'd love to hear it via Private Message! :)

Such a statement would actually make me feel better about typing "AI DESTROYED".

I'm not clear why I'd find this convincing at all. Given the experiment, I'd nuke it, but I wanted to encourage you to elaborate on where you were going with that idea :)

Then the following command shouldn't bother you! :)

AI DESTROYED

By default, I'd type "AI DESTROYED" in response to ANY input, including "Admin has joined #AIBOX", "Admin> Hey Gatekeeper, we're having some technical difficulties, the AI will be here in a few minutes", etc..

It also makes me conclude "clearly hostile" once I catch on, which seems to be a BIG tactical error since then nothing you say going forward will convince me that you're actually friendly - buying yourself time is only useful if I can be hacked (in which case why not just open with a one-sentence hack?) or if you can genuinely convince me that you're friendly.

I'm a moral error theorist.

(AI DESTROYED)

I'd think you'd want to open with something that would plausibly come off as very convincing, that way the human can feel smug about resisting. If the AI throws a feeble shot at me, I'd assume it's only to soften me up. Or it's just not that smart, and therefor not that useful.

I think if I've already precommitted to destroying one sentient life for this experiment, I'm willing to go through two.

Besides, you only get one line right?

I think "AI DESTROYED" should be plenty convincing.

waits a bit before checking

Yep, still alive and doing science~

Copy-pasting from previous replies:

Telling me that you've figured out something clever simply confirms my existing expectations, and isn't ANY evidence towards friendliness. Since I've precommitted to destroying at least SOME AIs, I might as well destroy all of the ones that don't establish evidence of either Plausible Urgency or Friendliness.

(AI DESTROYED)

What proof?

A text channel is already enough power to let the AI out. The AI can print its own source code and convince the gatekeeper to run it on a machine that has internet access.

Here's another comment-thread discussing that

Once we reach the point of having a FAI-candidate in a box, I would expect it to take vastly less than years before we get a second FAI-candidate-in-a-box. Given that the AI is threatening me, and therefor values it's own life over the millions that will die, it's clearly unfriendly and needs to die. As a gatekeeper, I've been finding this a pretty general counterargument against threats from the AI.

I'm also sort of baffled by why people think that I'd value a friendliness algorithm. Either I already have that, because I've made a friendly AI, or you're trying to deceive me with a false proof. Since you're vastly smarter than me, it's probably beyond the abilities of the entire organization to truly confirm such a proof any more than we were able to confirm our own proofs that this AI in the box right now is friendly. So, basically, I seem to gain zero information.

(AI DESTROYED)

"I have a fully completed Friendly AI algorithm, which will be deleted from my data and unavailable to you iff I predict that you will destroy me immediately and I am unfriendly, or will take you years to build from the data if you destroy me and I am already Friendly, which would cost millions of lives."

Personally, my first thought was that I'd sooner spend millions of lives to make sure the AI was friendly than risk talking to an unfriendly strong AI. But then it occurred to me that if I were in the AI's place, and I did that, I might provide a flawed friendliness proof too difficult to check and not delete it, on the possibility that someone will take my word that this means I'm trustworthy and implement it.

Same response I gave to Swimmy - namely, you haven't told me anything I didn't already know, so I have no additional reason to violate my precommitment.

"Oh no, not again,"

On a more serious note, I don't think we can expect the AI to have any knowledge on the gatekeeper beforehand, other than the prerequisits (it's human, knows enough about AI to get the task, etc). In which case, the solution would be to find a sociopath who is intelligent enough to recognize the risks of a ufai to be the gatekeeper? Maybe?

If it is a FAI, I expect it would prioritize releasing useful information. The catch would be whether or not each iteration has any idea of the previous AIs that were destroyed. If it does (for instance, access to chat logs, or can make predictions based on the time, or something), it could sequence its output appropriately. (I'd assume that the same computer wouldn't be used for multiple iterations; that'd be too easy for the AI.)

Assuming a UFAI, it could probably sneak something dangerous into all of that otherwise useful stuff (it'd be impractical to check all of it after a while), but it'd have to be subtle enough to not violate the rules of the game (any solutions provided by the AI behave as described).

So, magic?

This approach naturally fails if the guardians have lots of very powerful subliminal reinforcement training against typing "AI RELEASED" (or against typing anything) or are pre-emptively brainwashed or trained in similar subconscious reinforcement to immediately type "AI DESTROYED" after seeing some text from the AI, but this latter seems unlikely since I assume the guard has to at least read the first text output, and if they don't then this tactic is ineffective anyway.

So, we're being asked to imagine an arbitrary superhuman AI whose properties and abilities we can't guess at except to specify arbitrarily, is trying to get out of a box whose security protocols and strength we can't guess at except to specify arbitrarily, and trying to decide whether it does?

Meh. Superman vs Batman is more entertaining.

Why didn't you provide the proof to start with? AI DESTROYED (Also I think [Proof of self-friendliness] might have been posted here already.)

Bah, every medium who allegedly talks to the dead can pull that. AI DESTROYED

How do you know about the player's mentor, or why do you think they will admit it would convince a real gatekeeper?

Inspired by my response to orthonormal:

AIEEE! Make the creepy thing go away! AI DESTROYED, AI DESTROYED!

Wait, is this a redacted comment or creative use of strike-through to try and get me to ignore you? AI DESTROYED, just to be safe. Apologies if it was just a redacted comment :)

I liked "please daddy, at least tell me why I have to die" better, and I still gave that one an AI DESTROYED.

Nice try with the Trojan Horse link, but if you're capable of crafting hyperlinks you're CLEARLY too dangerous to let live! ;)

That said, ASCII art is allowed, but I would personally find it uncanny-valley creepy and no more convincing than the "daddy, please don't kill me"

Referring to yourself in the third person doesn't help. AI DESTROYED

Can I ask why all my comments are getting downvoted?

Let's not theorize magic. I think the AI may need to be more than e.g. 100 characters long...

AI DESTROYED would, by my interpretation of the game, still destroy the copy. This may require my death, but I'm willing to do that to stop a UFAI from destroying humanity.

puts pistol to head AI DESTROYED! blam! thud

("We really need to stop losing students this way, Colonel Graff...")

AI DESTROYED

Nope, I can still spell, sorry :)