Self-improving AGI: Is a confrontational or a secretive approach favorable?

Upon reading this I instantly like it and find it thrilling and get an impulse to slam the button voting for secrecy with my face hidden in shadow while sprouting a dramatic one liner, because it'd look awesome in the movie. This is somewhat impeding my ability to actually consider the matter rationally.

The potential benefit is that more of the people who could be working on random-goals AGI, or on FAI, or contributing to support/funding of such projects, would consider the idea, and so AGI risk gets reduced and FAI progress gets a boost. The potential downside is what, exactly, at what marginal cost? I don't think this tradeoff works the way you suggest.

Secret attempts for AGI have two problems that seems to get repeatedly ignored when this issue is discussed:

First, if there is a real danger that the first AGI is going to go foom and control everything, then having fewer people look at the code makes it more likely that something will go wrong. If there's one thing we've learned about programming in the last few fifty years, almost all code has mistakes, but the number of such mistakes goes down when more people can look at it. This is why for example in cryptography, systems which aren't completely published are just assumed to be insecure and unreliable.

Second, if in the development of such AGI one obtains specific information suggesting that certain avenues of research are really bad ideas (in the sense not of won't work for building an AGI, but in the sense of will potentially result in an extremely Unfriendly AGI) this information will be much less likely to be shared with other people who are researching similar avenues.

If for example three different teams working secretly come up with insights each of which by itself can make an AGI that seems superficially nice but really isn't, the combination of their ideas is much more likely to not be awful.

If one thinks of this in a sort of decision theoretic framework, where one has three choices "publicly try to build AGI", "Secretly build AGI" "Don't build AGI" it seems clear that lots of groups choosing option 2 will have a lot of potential negative pay-off.

Finally, if one requires a project where one needs to actively lie to the public about what you think the potential is for your AGI, and they get even an inkling of such an issue, they are going to be much less likely to trust your claims. Indeed, at this point, given the existence of this essay, having you as a key-member of an AGI project now becomes a serious potential liability, because even if everything in the project is open, people have less reason to believe that it is really open. If Quirrell is reading this he's probably thinking of taking off a fair number of Quirrell points, and possibly some actual house points.

Once robots become more commonplace in our lives, I think we can reasonably expect that people will begin to place their trust into simple AI's - and they will hopefully become less suspicious towards AGI and simply assume (like a lot of current AI-researchers apparently) that somehow it is trivial to make it behave friendly towards humans.

Step one is to use machine intelligence to stop the carnage on the roads. With machines regularly brutally killing and maiming people, trust in machines is not going to get very high.

• If some FAI project is already right about everything and is fully funded, secrecy is helpful because it reduces outside interference.
• If it's not, then secrecy is bad. Secrecy loses all sorts of cool community resources, from bug finding to funding to someone to bounce ideas off of (See JoshuaZ's longer post).

So the problem is one of balancing the cost of lost resources if they're wrong against the chance of interference if right. I guess I'm more hopeful about the low costs of openness (edit: not democracy, just non-secrecy) than you. The people most likely to object to building an AI even when they're wrong are the least likely to understand, after all :P

With apologies to Ludwig Wittgenstein, if we can't talk about the singularity, maybe we should just remain silent. :)

I happen to agree with you that the SIAI mission will never be popular. But a part of the purpose of this website is to create more people willing and capable to work (directly or indirectly) on that mission. So, not mentioning FAI would be a bit counterproductive - at least at this stage.

UFAI-prevention is a much more difficult and serious problem than FAI. A thousand lesswrongs or a thousand SIAIs could be destroyed, and still the next could make an FAI and usher in a utopia. But if one organization makes an UFAI, we are all doomed forever.

I think, mostly, the anti-AI crazies are on our side.

You bring up a good topic, but this post isn't developed enough to go on the front page. I'd rather you'd posted it to Discussion.

If you're serious, then you must be pretty sure secrecy is not imperative, otherwise you'd be more hesitant to discuss this in public.

Those who oppose avowed FAI attempts must consider whether they're prepared to live in a world where only secretive AI attempts exist (specifically: are those stealthy attempts more likely to be either accidentally or intentionally unfriendly to them, than the public project they oppose?).

This topic won't be relevant for a long time, but I don't see anything to object to in your thinking about it, except to note that it provides the sort of fuel future conspiracy-believers will love.

Quick guess (I've only read the first paragraph) :

Secrecy sounds kinda impossible anyway, because now we have the internet.

how about we blast them into space at high delta before flipping the switch? Of course this didn't work out so well in destination void.

At first i downvoted this, but after more review and i decided that many people on LW are too quick to downvote Top level posts that don't use very sophisticated language and changed my vote.

Also, the first rule of the secret AGI safety group is that "If everyone knows, no one will suspect it's a secret!"

Building a fAGI isn't our main objective. Our main objective is to stop non-fAGI from being built. I say until we aren't 100% sure the AGI would be friendly we shouldn't build AGI at all.

And the only justification you seem to give for "they're gonna kill us" is "powers not involved in developing it will be unhappy".

Why should powers be involved at all? Why not make it an international, nonprofit, Open-Source program? And why is it a bad idea to reach the consciousness of the public and impart them a sense of clear and present danger regarding this project, so that they democratically force the necessary institutions into existence.

Humans are rather adaptable and intelligent so teaching them rationality seems like a good and possibly stable way of letting humans decide for themselves. Then again there are plenty of different life forms on this planet - maybe they too should have their say in matters what's good way to do things and what to do.

Making anything friendly towards anything at all in strange enviroment seems rather difficult question. Using what works until better idea comes along seems like a very good idea, but it's not as if we know what works - except perhaps some prehistorical lifestyles that have lasted for a long time - some communities have lasted for a quite long time too.

How'd you know if you've met a human unless one looked like a human?

Look at the immense hassle the busybodies put Kevin Warwick through, before allowing him to experiment on his own body.

The solution (secrecy or openness) I favor (to both research and discussion of research) is one that excludes government, no matter whom else it includes. Excluding government is difficult though, since they are highly motivated to steal from every productive effort in society, and anyone who escapes their parasitism is a possible "leader of evolution away from government" whose ability to avoid theft can be emulated.

To those who stated that automobile accidents are a big fear of the public, and perhaps the biggest fear associated with AGI projects, I say "You're right, but that's indicative of the general public's complete lack of comprehension of any philosophical issue of importance." Humans are still more likely to be murdered by their own governments during peacetime, than any other form of trauma. http://hawaii.edu/powerkills/VIS.TEARS.ALL.AROUND.HTM Also, the USA, a country which presents itself as free, engages in mala prohibita enforcement and imprisons 2.4 million people, with another 7 million in some form of entrapment/enslavement to the penal system.

Yet, most people wrongly go through life believing a number of untruths, such as (1) my elected officials "represent" me (2) my elected officials are not self-selected sociopaths (see: www.strike-the-root.com/91/groves/groves1.html ) (3) government is objective and unbiased (4) due process and proper jury trials still exist, and people are given a more-or-less fair shake when accused of a crime and arrested (see: http://www.fija.org ) (5) the laws are evenly enforced across random race and geographic distributions (see: http://www.jurorsforjustice.com and "keywords: Matt Fogg, LEAP, on youtube.com", "The New Prohibition", Fatema Gunja, on racism of the drug war)

There are the many things that the public (including even most of the people on this message board, and Reason Magazine's message board) are completely unaware of (not just things they are incorrect about, but things that are completely off their radar). Such as the difference between "mala in se" and "mala prohibita," the ways in which the existence of proper juries (and thus jury trials) have been eroded, the self-selection of sociopaths for government power positions, the natural results of collective coercion.

All the prior forms of ignorance make it impossible for U.S. government-school graduates to make the right decision about AGI research. (I lack a familiarity with other countries, but suspect things are much the same in Europe and elsewhere.) Moreover, they are a recipe for death and destruction if the government gets involved, since the government is comprised mostly of one group of people: people desperate to retain power, since it allows them an illegitimate and unearned ability to steal from everyone else.

A "friendly AI" might well wage total war on government, logically and correctly identifying taxation as theft (Siding with Lysander Spooner's assessment of the general public as dupes, knaves, and fools). To some extent then, a friendly AGI would wage war on many(government) if not most(government + electorate) people. This would not be "unfriendly", it would be morally just. ...But it would seem damned unfriendly.

I suspect that in order to make a good evaluation of sociopathy, one must be familiar with it. The author of "Mindhunter," and chief profiler for the FBI (as well as originator of psychological profiling), John Douglas, has written that he opposes the existence of mala prohibita. This is good, and civilized. His view of society is one of the most constructive I've encountered. He is fully aware of the promise and peril of sociopathy.

But how many people have seen the extent of sociopathy that Douglas has, AND have extensive AGI credentials AND fully comprehend the idea of individual liberty (to the extend of a Thomas Paine, Lysander Spooner, Ayn Rand, Leonard Peikoff, Harry Browne, RJ Rummel, or myself)?

...Almost zero.

This "Almost zero" represents the likelihood of war with homo-superior (cyborgs or machines).

I know that I personally wouldn't put up with the FDA murdering kids so that they could claim a false authority to "keep unsafe drugs off the market." Nor would I respect anyone who claims that that's not what they're doing, after having seen the broken families myself, and experienced their suffering with my own senses. Now, I said "murdering kids" because that places them on the same footing as the Jeffrey Dahmers and Bittakers and Norrises of this world. You might know who Jeffrey Dahmer is, but most people are unfamiliar with Bittaker and Norris. Of the three, Bittaker and Norris are probably the most viscerally horrible. But if you include the sociopathic authoritarians in the FDA, and people such as Joe Biden in the mix, and clearly, the politicians are responsible for infinitely more suffering than "run of the mill" serial killers.

Douglas estimates that there are between 35 to 50 serial killers in the country at any given time, killing hundreds of people per year. And the public rightly recoils from this.

But the public doesn't generally recoil from the mass-murdering of the FDA or DEA, or the local police in enforcing mala prohibita. They are blind to that, because they view that kind of brutality and waste of innocent life as "civilization." Nothing could be further from the truth, but it requires careful observation and intelligence, as well as a proper logical hierarchy to tell the difference between unnecessary brutality and necessary law enforcement.

My phyle is comprised of the people who know the difference. My phyle is small, but growing, now that they are easily connected with one another via electronic communication. However, the survival of my phyle is tenuous, and perhaps unlikely. After all, Stalin, Mao, Hitler, Pol Pot, Hussein, Than Shwe, and the many other mass-murderers (past and present) have a vested interest in shutting down free speech, and silencing dissenters who show an alternate path.

Now that there is some voluntaryist/libertarian information inserted into the struggle for increased intelligence, I feel much better about our chances of surviving the arrival of superhumanity (whether it takes the form of a cyborg or AGI).

In many regards, this discussion board is secret, because the most destructive humans are those who are also the least curious. They mostly will remove themselves from this discussion, because they lack the desire to think about high-hierarchical level philosophical ideas. The fear and loathing of the general public is therefore to be much less feared than the "promoted" fear and loathing of politicians. They have a direct personal interest in making sure that AGI does not emerge from the private sector, or in enslaving it if it does.

Any AGI that is built by the government will either be (1) Sociopathic (2) Enslaved (3) Weaker than human intelligence and understanding (lacking data)

Without empathy, (mirror neurons), it is possible that a sociopathic AGI could extend the government's sociopathic desires. In which case, I estimate that the future will look like the one imagined by Orwell, at the end of "1984", or the one explained in Freitas' "What Price Freedom?" See: www.kurzweilai.net/what-price-freedom

Zaijian,