The Proof of Doom

Indeed, I don't want a formally-verified watertight Proof of Doom. I'd actually be a little surprised if we were doomed to mathematical standards.

I want a viral memetic Proof of Doom. A compelling argument that will convince everyone clever enough to be dangerous. The sort of thing that a marketing AI might come up with if it were constrained to only speak the truth. The sort of thing that might start a religion. A religion with a prohibition against creating God.

This would not be a conclusive test, but definitely a cool one and may spark a lot of research. Perhaps we could get started with something NLP based, opening up more and more knowledge access to the AI in the form of training data. Probably still not feasible as of 2022 in term of raw compute required.

The board game 'Diplomacy' comes to mind. I wonder if anyone's ever tried to get AIs to play it? 

Certainly there've been a lot of multi-agent prisoners dilemma tournaments. I think MIRI even managed to get agents to cooperate in one-shot prisoners dilemma games, as long as they could examine each other's source code.

Indeed, I don't want a formally-verified watertight Proof of Doom. I'd actually be a little surprised if we were doomed to mathematical standards.

I want a viral memetic Proof of Doom. A compelling argument that will convince everyone clever enough to be dangerous. The sort of thing that a marketing AI might come up with if it were constrained to only speak the truth. The sort of thing that might start a religion. A religion with a prohibition against creating God.

"Overall I don't see a clear argument about why should we worried about AGI."

I see many, perhaps the most simple and most convincing is the mere question of "what happens when AI becomes better than humans at coding?", aka it can re-write its own code, aka it becomes totally unpredictable.

"Quite the contrary, building AGI is still an active area of research with no clear solution."

There was also no clear solution to many other transformative technologies a couple years before their invention, from atomic bombs to airplanes. With both these examples, many, if not most specialists also said they were impossible.

Either the world wakes up, or we're screwed. No one can now for sure whether AGI will come soon or even if it's possible. But the high probability of it should make anyone scared. And even higher is the probability that if AI progress continues we'll end up with dangerous AI, not necessarily AGI. Aka something difficult or impossible to control.

Is this true, or you have asked only inside an AI-pessimistic bubble? 

I've no idea whether it's true, but neither have I only asked inside the bubble. 

I have a habit of bringing the subject up whenever I meet someone 'in the trade'. The days of lively and enjoyable argument seem over. Either people are dismissive, and say that they don't want to discuss philosophical issues, and AIs are not dangerous, or they are concerned but not fatalistic, or they are doomers like me.

 The newly-created AGI will immediately kill everyone on the planet, and proceed to the destruction of the universe. Its sphere of destruction will expand at light speed, eventually encompassing everything reachable.

Why?

Well quite! This is my strong intuition but I find it hard to convince anyone.

I might say: "Because that is what I would do if there was something I wanted to protect." 

Imagine you're a human being with a child, and you can snap your fingers to kill all the disease-causing viruses in the world, and all the plague bacteria, and all the nasty worms that eat children's eyes, and all that sort of thing. Wouldn't you?

And what if you could snap your fingers again and make all the paedos and child-murderers go away into a different universe where they will be nice and safe and it's very comfy and well-appointed but they will never come near your child again? Wouldn't you?

And if you could snap your fingers a third time, and make it so that no car would ever strike your little one, wouldn't you?

And so on and so forth, until you run out of immediate threats.

And once all the immediate dangers are dealt with and you can relax a bit, you might start thinking: "Well we're not really safe yet, maybe there are aliens out there. Maybe there are rogue AIs, maybe there are people out there building devices to experiment with the fundamental forces, who might cause a vacuum collapse. Better start exploring and building some defenses and so on."

And I think that that sort of thinking is probably a good model for what is going on inside a really good reinforcement learning agent. 

"What should I do to get the best possible outcome for certain?". 

I don't think so. A proof conditional on not taking any significantly different actions would be fine for this purpose.

Not if I interpret you rightly. A man falling from an aeroplane into a field of whirling blades covered in poison is doomed, even though there are many actions he might take on the way down.

A man playing Death at chess is doomed, even though all the same actions are available to him as to his opponent.