All AGI safety questions welcome (especially basic ones) [Sept 2022]

Technically, you can never be sure - it's possible an AI has developed and hidden capabilities from us, unlikely as it seems. That said, to the best of my knowledge we have not developed any sort of AI system capable of long-term planning over the course of days or weeks in the real world, which would be a prerequisite for plans of this nature.

So, that would be my threshold - when an AI is capable of making long-term real-world plans, it is at least theoretically capable of making long-term real-world plans that would lead to bad outcomes for us.

This twitter thread by Jack Clark contains a great deal of information about AI Policy which I was previously unaware of.

GovAI is probably one of the densest places to find that. You could also check out FHI's AI Governance group.

Not yet. There will soon be a $200k prize on Superlinear for people to try and define agency in a formal way, then write a program to detect it.

Not yet one that fits in the universe but we've finally at least got one that doesn't, which is a big improvement over "uh no idea lol": https://arxiv.org/abs/2208.08345

There is no consensus about what constitutes a moral patient and I have seen nothing convincing to rule out that an AGI could be a moral patient.

However, when it comes to AGI some extreme measures are needed.

I'll try with an analogy. Suppose that you traveled back in time to Berlin 1933. Hitler has yet to do anything significantly bad but you still expect his action to have some really bad consequences.

Now I guess that most wouldn't feel terribly conflicted about removing Hitler's right of privacy or even life to prevent Holocaust.

For a longtermist the risks we expect from AGI are order of magnitudes worse than the Holocaust.

Have these issues been discussed somewhere in the canon?

The closest thing of this being discussed that I can think of is when it comes to Suffering Risks from AGI. The most clear cut example (not necessarily probable) is if an AGI would spin up sub-processes that simulate humans that experience immense suffering. Might be that you find something if you search for that.

This is massive amounts of overthink, and could be actively dangerous. Where are we getting the idea that AIs amount to the equivalent of people? They're programmed machines that do what their developers give them the ability to do. I'd like to think we haven't crossed the event horizon of confusing "passes the Turing test" with "being alive", because that's a horror scenario for me. We have to remember that we're talking about something that differs only in degree from my PC, and I, for one, would just as soon turn it off. Any reluctance to do so when faced with a power we have no other recourse against could, yeah, lead to some very undesirable outcomes.

my take - almost certainly stopping a program that is an agi is only equivalent to putting a human under theoretical perfect anesthesia that we don't have methods to do right now. your brain, or the ai's brain, are still there - on the hard drive, or in your inline neural weights. on a computer, you can safely move the soul between types of memory, as long as you don't delete it. forgetting information that defines agency or structure which is valued by agency is the moral catastrophe, not pausing contextual updating of the structure.

They can if researchers (intentionally or accidentally) turn the SL/UL system into a goal based agent. For example, imagine a SayCan-like system which uses a language model to create plans, and then a robotic system to execute those plans. I'm personally not sure how likely this is to happen by accident, but I think this is very likely to happen intentionally anyway.

Interpolate Ray Kurtzweil's vision with what the world looks like today, with alpha being how skeptical of Ray you are.

With a grain of salt: for 2 million years there were various species of homo dotting africa, and eventually the world. Then humans became generally intelligent, and immediately wiped all of them out. Even hiding on an island in the middle of an ocean and specializing biologically for living on small islands was not enough to survive.

humans have fairly significant alignment issues and have developed a number of fields of research to deal with them. those fields include game theory, psychology, moral philosophy, law, economics, some religions, defense analysis... there are probably a few other key ones that didn't come to mind.

humans were fairly well aligned by our species' long history of strong self cooperation, at least compared to many other species, but being able to coordinate in groups well enough that we can reliably establish shared language is already very impressive, and the fact that we still have misalignments between each other isn't shocking. The concern is that AI could potentially be as unaligned as an arbitrary animal, but even more alien than the most alien species depending on the AI architecture.

Firstly - humans are by far the strongest natural general intelligences around, and most humans are aligned with humans. We wouldn't have much problem building an AI that was aligned with itself.

Secondly - almost no humans have the ability to actually cause major damage to the world even if they set about trying to. Nick Bostrom's Vulnerable World Hypothesis explores the idea that this may not always be true, as technology improves.

The overall consensus is such that:

1. Pulling the plug on an intelligent agent would be difficult, because that agent would try to stop you from doing so - for most goals the agent might have, it would correctly conclude that being turned off is detrimental towards that goal.

2. It's not enough that we don't make AI that won't let itself be turned off, we have to ensure nobody else does that either.

3. There are strong economic incentives to build AGI that make 2 difficult.

4. is the big point here - your point seems like reasonable common sense to most in the AI safety field, but the people in AI safety and the people building towards AGI aren't the same people.

How do we know that the AI has a correct and reasonable instrumentation of the risk of harming a human? What if the AI has an incorrect definition of human, or deliberately corrupts its definition of human?

More likely, the AI just finds a website with a non-compliant GET request, or a GET request with a SQL injection vulnerability.

I don't think so. It would probably make AI alignment easier, if we were able to define morality in a relatively simple way that allowed the AGI to derive the rest logically. That still doesn't counter the Orthogonality Thesis, in that an AGI doesn't necessarily have to have morality. We would still have to program it in - it would just be (probably) easier to do that than to find a robust definition of human values.