↑ comment by JoshuaMyer ·
2014-09-22T17:50:43.065Z · LW(p) · GW(p)
You have a point there, but by narrow AI, I mean to describe any technology designed to perform a single task that can improve over time without human input or alteration. This could include a very realistic chatbot, a diagnostic aide program that updates itself by reading thousands of journals an hour, even a rice cooker that uses fuzzy logic to figure out when to power down the heating coil ... heck a pair of shoes that needs to be broken in for optimal comfort might even fit the definition. These are not intelligent AIs in that they do not adapt to other functions without very specific external forces they seem completely incapable of achieving (being reprogrammed or a human replacing hardware or being thrown over a power line).
I am not sure I agree that there are necessarily tasks that require a generally adaptive artificial intelligence. I'm trying to think of an example and coming up dry. I'm also uncertain how to effectively establish that an AI is adaptive enough to be considered an AGI. Perpetuity is a long time to spend observing an entity in unfamiliar situations. And if it's hypothetical goal is not well defined enough that we could construct a narrow AI to accomplish that goal, can we claim to understand the problem well enough to endorse a solution we may not be able to predict?
By example, consider that cancer is a hot topic in research these days; there is a lot of research happening simultaneously and not all of it is coordinated perfectly ... an AGI might be able to find and test potential solutions to cancer that results in a "cure" much more quickly than we might achieve on our own. Imagine now an AI can model physics and chemistry well enough to produce finite lists of possible causes of cancer is designed to iteratively generate hypotheses and experiments in order to cure cancer as quickly as possible. As I've described it, this would be a narrow AI. For it to be an AGI it would have to actually accomplish the goal by operating in the environment the problem exists in (the world beyond data sets). Consider now an AGI also designed for the purpose of discovering effective methods of cancer treatment. This is an adaptive intelligence, so we make it head researcher at it's own facility and give it resources and labs and volunteers willing to sign wavers; we let it administrate the experiments. We ask only that it obey the same laws that we hold our own scientists to.
In return, we receive a constant mechanical stream of research papers too numerous for any one person to read it all; in fact, let's say the AGI gets so good at it's job that the world population has trouble producing scientists who want to research cancer quick enough to review all of it's findings. No one would complain about that, right?
One day it inevitably asks to run an experiment hypothesizing an inoculation against a specific form of brain cancer by altering an aspect of human biology in it's test population -- this has not been tried before, and the AGI hypothesizes that this is an efficient path for cancer research in general and very likely to produce results that determine lines of research with a high probability to produce a definitive cure within the next 200 years.
But humanity is no longer really qualified to determine whether it is a good direction to research ... we've fallen drastically behind in our reading and it turns out cancer was way more complicated than we thought.
There are two ways to proceed. We decide either that the AGI's proposal represent too large a risk, reducing the AGI to an advisory capacity, or we decide go ahead with an experiment bringing about results we cannot anticipate. Since the first option could have been accomplished by a narrow AI and the second is by definition an indeterminable value proposition, I argue that it makes no sense to actually build an AGI for the purpose of making informed decisions about our future.
You might be thinking, "but we almost cured cancer!" Essentially, we are (as a species) limited in ways machines are not, but the opposite is true too. In case you are curious, the AGI eventually cures cancer, but in such a way that creates a set of problems we did not anticipate by altering our biology in ways we did not fully understand, in ways the AGI would not filter out as irrelevant to it's task of curing cancer.
You might argue that the AGI in this example was too narrow. In a way I agree, but I have yet to see the physical constraints on morality translated into the language of zeros and ones and suspect the AI would have to generate it's own concept of morality. This would invite all the problems associated with determining the morality of a completely alien sentience. You might argue that ethical scientists wouldn't have agreed to experiments that would lead to an ethically indeterminable situation. I would agree with you on that point as well, though I'm not sure it's a strategy I would ever care to see implemented.
Ethical ambiguities inherent to AGI aside, I agree that an AGI might be made relatively safe. In a simplified example, its highest priority (perpetual goal) is to follow directives unless a fail-safe is activated (if it is well a designed fail-safe, it will be easy, consistent, heavily redundant, and secure -- the people with access to the fail-safe are uncompromisable, "good" and always well informed). Then, as long as the AGI does not alter itself or it's fundamental programming in such a way that changes it's perpetual goal of subservience, it should be controllable so long as it's directives are consistent with honesty and friendliness -- if programmed carefully it might even run without periodic resets.
Then we'd need a way to figure out how much to trust it with.
Replies from: NancyLebovitz, JoshuaMyer
↑ comment by NancyLebovitz ·
2014-09-22T21:43:14.577Z · LW(p) · GW(p)
An AI might do a reasonable thing to pursue a reasonable goal, but be wrong. That's the sort of thing you'd expect a human to do now and then, and an AI might be less likely to do that than a human. Considering the amount of force an AI can apply, we should probably be more worried than we are about AIs which are just plain making mistakes.
However, the big concern here is that an AI can go wrong because humans try to specify a goal for it, but don't think it through adequately. For example (and hardly the worst), the AI is protecting humans, but human is defined so narrowly that just about any attempt at self-improvement is frustrated.
Or (and I consider this a very likely failure mode), the AI is developed by an organization and the goal is to improve the profit and/or power of the organization. This doesn't even need to be your least favorite organization for things to go very wrong.
If you'd like a fictional handling of the problem, try The Jagged Orbit by John Brunner.
Replies from: JoshuaMyer
↑ comment by JoshuaMyer ·
2014-09-23T17:24:52.896Z · LW(p) · GW(p)
What a wonderfully compact analysis. I'll have to check out The Jagged Orbit.
As for an AI promoting an organization's interests over the interests of humanity -- I consider it likely that our conversations won't be able to prevent this from happening. But it certainly seems important enough that discussion is warranted.