The UFAI among us

It suggests to me that human values don't depend on the hardware, and are not a matter of historical accident. They are a predictable, repeatable response to a competitive environment and a particular level of intelligence.

Another possibility is that individual humans occasionally influence corporations' behavior in ways that cause that behavior to occasionally reflect human values.

How similar are their values actually?

One obvious difference seems to be their position on the exploration/exploitation scale, most corporations do not get bored (the rare cases where they do seem to get bored can probably be explained by an individual executive getting bored, or by customers getting bored and the corporation managing to adapt).

Corporations also do not seem to have very much compassion for other corporations, while they do sometimes co-operate I have yet to see an example one corporation giving money to another, without anticipating some sort of gain from this action (any altruism they display towards humans is more likely caused by the individuals running things or done for signalling purposes, if they were really altruistic you would expect it to be towards each-other).

Do they really value independence and individuality? If so then why do they sometimes merge? I suppose you could say that the difference between they and humans is that they can merge while we can't, but I'm not convinced we would do so even if we could.

There may be superficial similarities between their values and ours, but it seems to me like we're quite different where it matters most. A hypothetical future which lacks creativity, altruism or individuality can be safely considered to have lost almost all of its potential value.

Should other large human organizations like governments and some religions also count as UFAIs?

Unlike programmed AIs, corporations cannot FOOM. This leaves them with limited intelligence and power, heavily constrained by other corporations, government, and consumers.

The corporations that have come the closest to FOOMing are known as monopolies, and they tend to be among the least friendly.

The SIAI is a "501(c)(3) nonprofit organization." Such organizations are sometimes called nonprofit corporations. Is SIAI also an unfriendly AI? If not, why not?

P.S. I think corporations exist mostly for the purpose of streamlining governmental functions that could otherwise be strucured in law, although with less efficiency. Like taxation, and financial liability, and who should be able to sue and be sued. Corporations, even big hierarchical organizations like multinationals, are simply not structured with the complexity of Searles' Chinese Room.

It suggests to me that human values don't depend on the hardware, and are not a matter of historical accident. They are a predictable, repeatable response to a competitive environment and a particular level of intelligence.

I don't understand why you think that the rest of your post seems to suggest this. It appears to me that you're proposing that human (terminal?) values are universal to all intelligences at our level of intelligence on the basis that humans and corporations share values, but this doesn't hold up because corporations are composed of humans, so I think that the natural state would be for them to value human values.

I don't think it is useful to call Ancient Egypt a UFAI, even though they ended up tiling the desert in giant useless mausoleums at an extraordinary cost in wealth and human lives. Similarly, the Aztecs fought costly wars to capture human slaves, most of whom were then wasted as blood sacrifices to the gods.

If any human group can be UFAI, then does the term UFAI have any meaning?

It suggests to me that human values don't depend on the hardware, and are not a matter of historical accident. They are a predictable, repeatable response to a competitive environment and a particular level of intelligence.

By human values we mean how we treat things that are not part of the competitive environment.

The greatness of a nation and its moral progress can be judged by the way its animals are treated.

-- Mahatma Gandhi

Obviously a paperclip maximizer wouldn't punch you in the face if you could destroy it. But if it is stronger than all other agents and doesn't expect to ever having to prove its benevolence towards lesser agents, then there'll be no reason to care about them? The only reason I could imagine for a psychopathic agent to care about agents that are less powerful is if there is some benefit in being friendly towards them. For example if there are a lot of superhuman agents out there and general friendliness enables cooperation and makes you less likely to be perceived as a threat and subsequently allows you to use less resources to fight.

Another point to consider would be my Imperfect levers article and this one. I believe that the organizations that show the first ability to foom would foom effectively and spread their values around. This is not in any way, new. I, of indian origin, am writing in english and share more values with some californian transhumanists than with my neighbours. If not for the previous fooms of the british empire, the computer revolution and the internet, this would not have been possible.

The question is how close to sociopathic rationality are any of these organizations. Almost all of them exhibit omohundro's basic drives. I would disagree with the premise that alliances, status, power, resources are basic human values. They are instrumental values, subsets of the basic drives.

In organizations where a lot of decisions are being made on a mechanical basis, it is possible that some mechanism just takes over as long as it continues satisfying the incentive/hitting the button.

Common instrumental values are in the air today.

The more values are found to be instrumental, the more the complexity of value thesis is eroded.

Charlie Stross seems to share this line of thought

We are now living in a global state that has been structured for the benefit of non-human entities with non-human goals. They have enormous media reach, which they use to distract attention from threats to their own survival. They also have an enormous ability to support litigation against public participation, except in the very limited circumstances where such action is forbidden.

I was expecting a post questioning who/what is really behind this project to make paperclips invisible.

Michael Vassar raised some of the same points in his talk at H+, 2 weeks before I posted this.

Corporations (and governments) are not usually regarded as sharing human values by those who consider the question. This brief blog post is a good example. I would certainly argue that the 'U' is appropriate; but then I tend to regard 'UFAI' as meaning 'the complement of FAI in mind space'.

Corporations develop values similar to human values. They value loyalty, alliances, status, resources, independence, and power. They compete with other corporations, and face the same problems people do in establishing trust, making and breaking alliances, weighing the present against the future, and game-theoretic strategies. They even went through stages of social development similar to those of people, starting out as cutthroat competitors, and developing different social structures for cooperation (oligarchy/guild, feudalism/keiretsu, voters/stockholders, criminal law/contract law). This despite having different physicality and different needs.

It suggests to me that human values don't depend on the hardware, and are not a matter of historical accident. They are a predictable, repeatable response to a competitive environment and a particular level of intelligence.

It seems more likely that corporations act like humans because corporations are ran by humans. I've yet to meet an alien CEO or board member!

edit: and then I realized Dorikka said it first.

edit: and TheOtherDave.

Is my grandma an Unfriendly AI?

Corporations today are not very good AI architectures - they're good at passing information down a hierarchy, but poor at passing it up, and even worse at adding up small correlations in the evaluations of their agents.

I'd be cautious about the use of "good" here - the thing you describe mostly seem "good" from the point of view who cares about the humans being used by the corporations; it's not nearly as clear that they are "good" (bringing more benefits than downsides) for the final goals of the corporation.

If you were talking about say a computer system that balances water circulation in a network of pipes, and has a bunch of "local" subsystems with more-or-less reliable measures for flow, damage to the installation, leaks, power-efficiency of pumps, you might care less about things like which way the information flows as long as the overal system works well. You couldn't worry about whether a particular node had it's feeling hurt by the central node ignoring it's information (which may be because the central node has limited bandwidth, processing power, and has to deal with high undertainty about which nodes provide accurate information).

The differences between: a 90% human 10% machine company...

...and a 10% human 90% machine company...

...may be instructive if viewed from this perspective.

And yet it seems likely that your typical corporation, even your typical monopoly, does much more good for the world than a typical human would do with a similar amount of optimization power. Even billionaires who donate to charity probably don't generate as much utility for mankind as your typical aggressively run Chinese manufacturing company.