Friendlier AI through politics

I found Brin's post flowery and worthless. Michael Anissimov correctly points out the problems with it in the comments. (Is he registered on LW, I wonder?)

People seem to have serious problems grasping the idea that AIs are machines that will stupidly want anything we program them to want, not elf-children that will magically absorb our "kindness" and "understanding". When I first came across Eliezer's writings about FAI, that point seemed absolutely obvious... I guess David Brin hasn't read it, that's about the only possible explanation.

Here is a better post than Brin's on the subject, by Anissimov and Smigrodzki, from 2006, on Accelerating Future.

David Brin suggests that some kind of political system populated with humans and diverse but imperfectly rational and friendly AIs would evolve in a satisfactory direction for humans.

Brin:

a great diversity of mega minds, contending with each other, politely, and under civil rules, but vigorously nonetheless, holding each other to account and ensuring everything is above-board.

Source

These AIs are all friendly, because they have the constraint of not using force upon each other, i.e. you would have to solve the friendliness structure problem and put in some content about not destroying the other AIs.

Brin:

But the real reason to do this is far more pragmatic. If the new AIs feel vested in a civilization that considers them "human" then they may engage in our give and take process of shining light upon delusion. Each others delusions, above all. Reciprocal accountability -- extrapolated to a higher level -- may thus maintain the core innovation of our civilization. It's central and vital insight. And thus, we may find that our new leaders -- our godlike grandchildren -- will still care about us... and keep trying to explain.

Brin wants to create AI that is conditionally altruistic, and then be nice to it, and then hope that it will be nice back, whilst also having more than one such AI around, each with distinct goals, but also impose the constraint upon them that they have to not use force upon each other.

Eliezer, this must be painful for you to read.

I think AIs will always be imperfectly rational as they cannot optimize everything and still make decisions, and besides, optimal outcomes are not the same as perfect outcomes. I do think self-improving AI will tend to self-improve toward perfect rationality to the point where there is no significant difference between optimal and perfect.

As for a system in which AIs that disagree on friendliness are not dangerous: I can't imagine one. If friendliness includes protecting humans from unfriendly AI, a perfectly friendly AI would destroy imperfectly friendly AIs because imperfect friendliness is unfriendly compared to perfect friendliness. Therefore, an imperfectly friendly AI that thought it was perfectly friendly would destroy other AIs it disagreed with. To put it crudely, a "women's lib" AI could not coexist with a "Sharia" AI. And an imperfectly friendly AI that knew it was imperfectly friendly could not trust itself to make friendly decisions, and so would be acting in an unfriendly way if it tried to make decisions for people. Of course, this is all pretty primitive word vomit that I haven't spent much time thinking about, and ignores the reality that if imperfectly rational and friendly humans can get along, AIs should be able to as well.

Interestingly, Brin argues from the question Shall we maintain momentum and fealty to the underlying concepts of the Western Enlightenment? What he fails to consider here is the very genuine possibility that an AGI may be created by a non-Enlightenment culture, i.e. Japan or China. Perhaps a likelier possibility yet is an AGI created by a coalition of cultures, researchers from different countries working together, which could further extend to a multinational corporation which may not reflect any particular culture other than a basically capitalist one. While capitalism is a pretty fundamental Enlightenment ideal, it is only a framework in which other cultures may function very differently. The way a Japanese businessman or CEO considers competition is not the same as how a Russian or an American or a Brazilian views it. How different would a Japanese AGI be from an American one, or an AGI built by the Chinese government vs. one built by a private lab in the EU?

David Brin suggests that some kind of political system populated with humans and diverse but imperfectly rational and friendly AIs would evolve in a satisfactory direction for humans.

If I believed David Brin had the ability to significantly encourage the creation of such a system I would do whatever was in my power to thwart him. The only political moves that could make the outcome one favourable to humanity are those that prevent it.

About disputation arenas, I wish more people were at least thinking like Brin.

There are many projects for hypertext debate and analysis, but I don't know of any which would let analysts break down a hard ambiguous question using the Bayesian formulations of evidence, explaining-away, or transfer learning from related cases through hierarchical priors. And no project lets analysts reduce disagreements about instrumental values to disagreements about parameters or approximations in decision theory. Also, I know of no game-theoretic analysis of participants' incentives to enter evidence selectively or falsely, even though those incentives affect the conditional probabilities of observing that evidence.

What counts as an AGI? A system that can solve resource-allocation problems that satisfy its inbuilt values in a dynamic environment?

If one were to take Hayek's view of human marketplaces, we could consider our currently existing markets as greater-than-human intelligences by that standard.

What legal principles enable markets to work well? Theory and practice suggest that the necessary and sufficient conditions are: 1) property ownership and enforcement and 2) a sound currency of value-measurement.

So: how well have we imbued our emergently-superintelligent markets with values consistent with human well-being? I would suggest that we have succeed to the extent that the conditions above are met. Further, I'd suggest that FAI development should also hinge on those principles.

What kind of competitive or political system would make fragmented squabbling AIs safer than an attempt to get the monolithic approach right?

This is a tremendously important question! (David Brin isn't the first person to raise the idea, BTW. I raised it at the first AGI workshop in 2006, and probably before that on OB. I would be surprised if no-one else had also raised it before that.)

Brin's essay doesn't really touch on any of the important problems with doing so, though.

One of the dangers of trying to implement this is our own horrendously inaccurate understanding of how checks-and-balances work in our own system. Brin's essay, and the ideas of just about every American who speaks on this topic, are fundamentally unsound because they start from the presumption that democracy works, for everything, all the time, everywhere. We've made democracy such a concept of reverence that we have never critiqued it. We haven't even collected the data we would need to do so. Even in Iraq, where we urgently need to do so, we still haven't asked the question "Why does democracy not seem to work here? Would something else work better?"

For starters, we can't hope to create an ecology of AIs until we can figure out how to create a government that doesn't immediately decay into a 2-party system. We want more than 2 AIs.

EDIT: Folks, this is a very important point, for your own survival. I strongly encourage you to explain why you down-voted this comment.

BTW, a bit off-topic, but does anybody else wonder if being able to create a friendly AI requires being able to be friendly? Because we don't have a lot of that here.