AI alignment with humans... but with which humans?

In the near term, when we are still talking about things like "the person who bought the AI to help run the traffic lights" rather than "the person who unleashed AI to write its values upon the stars," I think it is actually totally fine to try to build AIs that are "aligned" (in their own not-too-bright way) with the person who bought them.

It is not the AI the army buys to control tanks that I'm worried about aligning to the broad swath of human values. It is the AI that gets built with no need for a buyer, by research labs who recognize that it's going to have a huge impact on the future.

Okay, with that out of the way - is such a notion of "alignment" feasible, given that humans oppose each other about stuff?

Yes.

The world could be better than it is today, in ways that would please almost everyone. This is all I really want from aligned AI. I'm reminded of Transhumansism is Simplified Humanism [LW · GW]. There is someone dying of cancer. Should they be saved? Yes! No trick question!

Sure, certain human values for dominance, or killing, or even just using resources unsustainably might forever be impossible to fulfill all the time. So don't try to do impossible things, just build an AI that does the good things that are possible!

How to do this in practice, I think, looks like starting out with a notion of "the broad swath of human values" that defines that term the way the designers (aided, realistically, by a random sample of Mechanical Turkers) would define "human values," and then updating that picture based on observing and interacting with humans out in the real world.

While it's true that AI alignment raises difficult ethical questions, there's still a lot of low-hanging fruit to keep us busy. Nobody wants an AI that tortures everyone to death.

All 7.8 billion humans that the AI system should be aligned with, so it doesn't impose global catastrophic risks

That's the main one - once there is a super intelligent AI aligned with humanity as a whole, then /it/ can solve the lower-scale instances.

That said, there are a lot of caveats and contradictions to that too:

• People have values about the values they ought to have that are distinct from actual values they have - and we might want the AI to pay more attention to former?
• People's values are often contradictory (both a single person may do thing they will regret, some people would explicitly value suffering of others, and all kinds of other biases and inconsistencies)
• It's very unclear how the values should be generalized beyond the routine scenarios people encounter in life.
• Should we weight everybody's values equally (including little kids)? Or should we assume that some people are better informed that others, have spent more time thinking about moral and ethical issues, and should be trusted more to represent the desired values?

and many more.

There is alignment in the sense of an AI doing what is requested, and there is alignment in the sense of, an AI whose values could be an acceptable basis for a transhuman extension of human civilization.

For the latter, there was originally CEV, the coherent extrapolated volition of humanity. There was some hope that humans have a common cognitive kernel from which a metaethics true to human nature could be inferred, and this would provide both the thing to be extrapolated and the way in which to extrapolate it, including an approach to social choice (aggregating the values of multiple agents).

Lately MIRI's thinking switched to the idea that there isn't time to resolve all the issues here, before lethally unfriendly AGI is created by accident, so plan B is that the first AGI should be used to stop the world and freeze all AI research everywhere, until the meta meta issues of method and goal are truly sorted out.

However, June Ku has continued with a version of the original CEV concept, which can be seen at Metaethical.AI.

When In Rome

Thank you for posting this Geoffrey. I myself have recently been considering posting the question, “Aligned with which values exactly?”

TL;DR - Could an AI be trained to deduce a default set and system of human values by reviewing all human constitutions, laws, policies and regulations in the manner of AlphaGo?

I come at this from a very different angle than you do. I am not an academic but rather am retired after a thirty year career in IT systems management at the national and provincial (Canada) levels.

Aside from my career my lifelong personal interest has been, well let’s call it “Human Nature”. So long before I had any interest in AI I was reading about anthropology, archeology, philosophy, psychology, history and so on but during the last decade mostly focused on human values. Schwartz and all that. In a very unacademic way, I came to the conclusion that human values seem to explain everything with regards to what individual people feel, think, say and do and the same goes for groups.

Now that I’m retired I write hard science fiction novellas and short stories about social robots. I don’t write hoping for publication but rather to explore issues of human nature both social (e.g. justice) and personal (e.g. purpose). Writing about how and why social robots might function, and with the theory of convergent evolution in mind, I came to the conclusion that social robots would have to have an operating system based on values.

From my reading up to this point I had the gained impression that the study of human values was largely considered a pseudoscience (my apologies if you feel otherwise). Given my view of the foundational importance of values I found this attitude and the accompanying lack of hard scientific research into values frustrating.

However as I did the research into artificial intelligence that was necessary to write my stories I realized that my sense of the importance of values was about to be vindicated. The opening paragraph of one of my chapters is as follows… 

During the great expansionist period of the Republic, it was not the fashion to pursue an interest in philosophy. There was much practical work to be done. Science, administration, law and engineering were well regarded careers. The questions of philosophy popular with young people were understandable and tolerated but where expected to be put aside upon entering adulthood.

All that changed with the advent of artificial intelligence.

As I continued to explore the issues of an AI values based operating system the enormity of the problem became clear and is expressed as follows in another chapter…

Until the advent of artificial intelligence the study of human values had not been taken seriously. Values had been spoken of for millennia however scientifically no one actually knew what they were, whether they had any physical basis or how they worked as a system. Yet it seemed that humans based most if not all of their decisions on values and a great deal of the brain’s development between the ages of five and twenty five had to do with values. When AI researchers began to investigate the process by which humans made decisions based on values they found some values seemed to be genetically based but they could not determine in what way, some were learned yet could be inherited and the entire genetic, epigenetic and extra-genetic system of values interacted in a manner that was a complete mystery.

They slowly realized they faced one of the greatest challenges in scientific history.

I’ve come to the conclusion that values are too complex a system to be understood by our current sciences. I believe in this regard that we are about where the ancient Greeks were regarding the structure of matter or where genetics was around the time of Gregor Mendel.

Expert systems or even our most advanced mathematics are not going to be enough nor even suitable approaches towards solving the problem. Something new will be required. I reviewed Stuart Russell’s approach which I interpret as "learning by example" and felt it glossed over some significant issues, for example children learn many things from their parents, not all of them good.

So in answer to your question, “AI alignment with humans... but with which humans?” might I suggest another approach? Could an AI be trained to deduce a default set and system of human values by reviewing all human constitutions, laws, policies and regulations in the manner of AlphaGo? In every culture and region, constitutions, law, policies and regulations represent our best attempts to formalize and institutionalize human values based on our ideas of ethics and justice.

I do appreciate the issue of values conflict that you raise. The Nazis passed some laws. But that’s where the AI and the system it develops comes in. Perhaps we don’t currently have an AI that is up to the task but it appears we are getting there.

This approach it seems would solve three problems; 1) the problem of "which humans" (because it includes source material from all cultures etc.), 2) the problem of "which values" for the same reason and 3) your examples of the contextual problem of "which values apply in which situations" with the approach of “When in Rome, do as the Romans do”.

if the AI makes a habit of killing people for disagreeing, then we failed. If the AI enforces "western" anything in particular, then we failed. of course, you're asserting that success is impossible, and that it's not possible to find an intersection of human values across the globe that produces a significant increase in empirical cooperation; I don't think I'm ready to give up yet. I do agree that conflict is currently fundamental to life, but I also don't think we need to keep conflicting at whole-organism scale - most likely disagreements can be resolved at sub-organism scale, fighting over who gets to override whose preferences how much, and viciously minimize this kind of conflict. eg, if someone is a sadist, and desires the creation and preservation of conflicting situations for their own sake - then that person is someone who I personally would claim should be stopped by the global cooperation group.

Your phrasing sounds to me like you'd endorse a claim that might makes right, or that you feel others do so tit-for-tat requires that you endorse might makes right.

But based on results in various forms of game theory, especially evolutionary game theory, I expect generous-tit-for-tat-with-forgiveness is in fact able to win long term, and that we can end up with a very highly cooperative society that still ensures every being maintains tit-for-tat behavior. Ultimately the difficulty of reducing conflict boils down to reducing scarcity relative to current number of organisms, and I think if used well, AI can get us out of the energy-availability mess that society has gotten ourselves into.

You'll have to look up the actual fields of research I'm referencing to get details; I make no claim to be an expert on the references I'm making, and of course as I'm hypothesizing out loud to your bitter commentary, I wouldn't expect you to be impressed. But it's the response that I have to give.

I think one of the main worries is that the AI will fail to serve anyone, even if attempted to arbitrarily unfairly serve a single individual.

It is like arguing where our rocketship should go when we don't even know whether we can reach orbit or not explode on launch pad.

Quick note to say I don't think this comment is great (or your other comments on this thread) – I don't think it's quite substantive enough to justify the tone used. I might be more sympathetic if there was more elaboration on:

handwaving away the problem. Conflict is fundamental to human behavior and human morality is very fungible.

It's midnight local time for me, so I won't take further action right now, but wanted to say something quick about this (and the others) not being the kind of comment I'm enthusiastic to have on LessWrong.

[comment quality: informed speculation dictated to phone in excitement; lit search needed]

this is a great point and I'm glad you bring it up. I would argue that the core of strongly superintelligent ai safety requires finding the set of actions that constraint-satisfy the [preferences/values/needs/etc] of as many [humans/mammals/life forms/agents/etc] as possible; and that the question is how to ensure that we're aiming towards that with minimal regret. in other words, I would argue that fully solving ai safety cannot reduce to anything less than fully and completely solving conflict between all beings - effectively-perfect defense analysis and game theory. a key step on that path seems to me to be drastically increasing usable-resource availability per agent and as such I wouldn't be surprised to find out that the bulk of the action-space solution to AI safety ends up being building a really big power plant or something surprisingly simple like that. I expect that a perfect solution to the hard version of ai safety would look like a series of game theory style proofs that show every military in the world unilaterally stronger paths towards least conflict that the humans in charge of them can actually understand.

on the lead up, though, projects focusing on collective intelligence and cooperative learning are promising, imo. the ipam collective intelligence workshop had some great talks about problem solving with groups of computational nodes, that's on YouTube. the Simons institute has had many talks this year on cooperative game theory and social networks. I've got a bunch of links on my shortform of academic talk channels on YouTube that I feel weigh in on this sort of stuff besides those two.

I suspect a significant part of the project of cooperative ai will be to encourage ai to become good at mapping and communicating the trade-off landscape and mediating discussions between people with conflicting preferences.

When AI alignment researchers talk about 'alignment', they often seem to have a mental model where either (1) there's a single relevant human user whose latent preferences the AI system should become aligned with (e.g. a self-driving car with a single passenger); or (2) there's all 7.8 billion humans that the AI system should be aligned with, so it doesn't impose global catastrophic risks.

[...]

So, I'm left wondering what AI safety researchers are really talking about when they talk about 'alignment'.

The simple answer here is that many technical AI safety researchers on this forum talk exclusively about (1) and (2) so that they can avoid confronting all of the difficult socio-political issues you mention. Many of them avoid it specifically because they believe they would not be very good at politics anyway.

This is of course a shame, because the cases between (1) and (2) have a level of complexity that also needs to be investigated. I am a technical AI safety researcher who is increasingly moving into the space between (1) and (2), in part also because I consider (1) and (2) to be more solved [? · GW] than many other AI safety researchers on this forum like to believe.

This then has me talking about alignment with locally applicable social contracts, and about the technology of how such social contracts can be encoded into an AI. See for example the intro post and paper here [LW · GW].

Yes

It's supposed to mean alignment with EY, who will then be able to perform the pivotal act of ensuring nobody else can create AGI