Cynical explanations of FAI critics (including myself)

If there are thousands of possible avenues of research, and critics have a noisy lock on truth in the sense of picking a few hundred avenues they like best, then we could easily wind up with all the critics agreeing that strategy X is just a bad idea, but also disagreeing on whether strategy A is better than strategies B or C. So I don't see disagreement among critics as proving much at all other than the critics are not perfect, which they surely would agree with; it doesn't vindicate X. There are so many more wrong research avenues than right ones.

(Imagine we have 10,000 possible research topics, 3 critics who have identified their top 10 strategies, and the critics are guaranteed to identify 'the right' strategy in those 10 but beyond that pick randomly the top 1. If someone picks research topic X which is genuinely wrong, then the critics will almost certainly all agree that that topic is indeed the wrong topic: the number of strategies endorsed by any of them just 28 strategies out of the 10,000 and 28 / 10,000 is a pretty small chance for X to get lucky and be one of them. But at the same time, will the 3 critics all rank the same strategy as the top strategy? 1/10 1/10 1/10 is not great odds either! So even though the critics have an amazing truth-finding ability in being able to shrink 10,000 all the way down to 10, they still may not agree because of their remaining noise.)

I criticize FAI because I don't think it will work. But I am not at all unhappy that someone is working on it, because I could be wrong or their work could contribute to something else that does work even if FAI doesn't (serendipity is the inverse of Murphy's law). Nor do I think they should spread their resources excessively by trying to work on too many different ideas. I just think LessWrong should act more as a clearinghouse for other, parallel ideas, such as intelligence amplification, that may prevent a bad Singularity in the absence of FAI.

CEV and understanding recursive self-modification. Everything boils down to those two linked disciplines. The CEV is for understanding what we want, and the recursive self-modification is so that whatever is FOOMing doesn't lose sight of CEV while changes itself. I simply do not trust that anything will FOOM first and then come up with perfect CEV afterwards. By that time it will already be too far removed from humanity. This was, I think, a topic in Eliezer's metaethics sequence. It's the still unanswered question about what to do when you actually have unlimited power, including the power to change yourself.

Every option eventually boils down to a FOOM. This is why CEV and recursive self-modification must be finished before any scenario completes. AI is an artificial life-form FOOM, and may be friendly or unfriendly; uploading is human FOOM, and we already know that they're unfriendly with sufficient power; intelligence amplification is a slower biological FOOM; the first question asked of Oracle/Tool AI will be how to FOOM; and mainstream AGI is trying to build an AI to FOOM, except slower. The only non-FOOM related options are improving laws and institutions, which is already an ethical question, and computer security (and I'm not sure how that one relates to SIAI's mission).

The issue is that both of these are really hard. Arguably every philosopher since ever has been trying to do CEV. Recursive self-modification is hard as well, since humans can barely self-modify our ethical systems as it is. Though, as I understand, CFAR is now working to finding out what it takes to actually change people's minds/habits/ethics/actions.

Edit: But at the end of the day, it doesn't help one bit if SIAI comes up with CEV while the Pentagon or China comes up with uFAI. So starting work on AI is probably a good idea.

To add to your list of various alternatives... My personal skepticism re SI is the apparent lack of any kind of "Friendly AI roadmap", or at least nothing that I could easily find on the SI site or here. (Could be my sub-par search skills, of course.)

Kill all those birds with one stone: Work on understanding and preventing the risks. Here's why that stone kills each bird:

Re: Work on some aspect of building an AI

Even the sequences are against this. "Do not propose solutions until the problem has been discussed as thoroughly as possible" Working on risk should go first. Also:

• Somebody needs to judge the safety of AIs.

  Including any that you guys would make. If not SIAI, who will do this at all?

• SIAI can't do both safety and production, it's perverse.

  The people who MAKE the AI should NOT be the same people who JUDGE the AI for the same reason that I would not purchase medical treatments from a doctor who claimed to test them himself but did not do a scientific study. You cannot peer review, independently test or check and balance your own project. Those that want to make an AI should plan to be part of a different organization, either now or in the future.

• You'll get speed and quality advantages this way.

  If you build the AI first, you will certainly consider safety problems that hadn't been obvious before because you'll be in the thick of all those details and they'll give you new ideas. But those can always be added to the list of safety guidelines at that time. There is no reason to do that part first. If you make safety guidelines first, you can build the AI with safety in mind from the ground up. As you know, reprogramming something that has a flaw in a critical spot can be very, very time-consuming. By focusing on safety first, you will have a speed advantage while coding as well as a quality advantage. Others will make dangerous AIs and be forced to recall them and start over. So, this is a likely advantage.

Re: "improve laws and institutions"

You need to understand the risks thoroughly before you will be able to recommend good laws, and before people will listen to you and push for any legislation. After you understand the risks, you'd need to work on improving laws so that when the interested people go to build their AI there's a legal framework there.

Re: "computer security"

This should be included under "risk research and prevention." because it's definitely a risk. There are likely to be interactions between security, and other risks that you'd want to know about at the same time as working on the security, it's all connected, and you may not discover these if you don't think about them at the same time.

Re: "stop worrying about the Singularity and work on more mundane goals"

Considering the personalities, capabilities and prior investments of those involved, this simply isn't likely to happen. They need to be ambitious. Ambitious people need the assistance of others who are more specialized in mundane tasks and would be happy to help them so that they can focus on ambitions - we all specialize.

Focusing on risk research and prevention is also the first step to everything else:

• How will you get funding for something people see as risky?

• How will you develop AI in a world destroyed by other's AI projects that SIAI didn't take the time to stop?

• How will SIAI develop credibility and trust if it doesn't prove it's capable of intellectual rigor by doing a thorough job of risk prevention? This entire industry has no trust. Even as an AI project, you'll have no trust for that reason.

• How will SIAI prove it is effective in the world if it doesn't do something before making an AI such as change some laws, and do risk prevention?

• Who is going to be there to independently test your AI project if you choose to do that instead?

I don't think the solution is "Do some, not others." I think it is "Do them in the right order." and for which type of AI project to chose, wouldn't it be safer to decide AFTER you research risks as thoroughly as possible?

Additionally, if SIAI chooses to dedicate itself to risk research and prevention and agrees that the AI building activities should be split off into a different group, I'd be interested in doing some volunteer work for the risk research and prevention group, especially regarding preventing an AGI arms race. I think the ideas I explain there or similar ones would be a really good way for you to prove that SIAI is capable of actually doing something, which addresses a common objection to funding SIAI.

See any way to break the above line of reasoning and argue for a different route? If so, I will attempt to resolve those conflicts also.

ideal reasoners are not supposed to disagree

My ideal thinkers do disagree, even with themselves. Especially about areas as radically uncertain as this.

Given that ideal reasoners are not supposed to disagree, it seems likely that most if not all of these alternative suggestions can also be explained by their proponents being less than rational.

Please support this statement with any kind of evidence. From where I sit it looks to be simply an error.

As far as I know values do not come from reason, they are a "given" from the point of view of reason. So if I value paper clips and you value thumbtacks, we can be as rational as all get out and still disagree on what we should do.

Further, I think even on matters of "fact," it is not "ideal reasoners" who do not disagree, rather it is reasoners who have agreed on one of a few possible methodologies of reasoning. So I think I have seen the statement, something like "rational bayesians cannot agree to disagree on probability estimates."

Given that ideal reasoners are not supposed to disagree,

Under some non-trivial assumptions.

it seems likely that most if not all of these alternative suggestions can also be explained by their proponents being less than rational.

That sounds pretty much condescending. These suggestions are not all mutually exclusive, and proponents with different values might have different preferences without being "less than rational".

About agreement: for the agreement we need all our evidence to be shareable, and our priors to be close enough. Actual evidence (or hard-to-notice inferences) about possibility of significantly super-human AGI on reasonable hardware cited in the Sequences are quite limited, and not enough to overcome difference in priors.

I do think humanity will build slightly super-human AGI, but as usual with computers it will mimc our then-current idea of how human brain actually works and then be improved as the design allows. In that direction, HTM (as done by Jeff Hawkins via his current Numenta startup) may end up polished into a next big thing in machine learning or a near-flop with few uses.

Also, it is not clear that people will ever get around to building general function-optimizing AI. Maybe executing behaviours will end up being the way to safeguard AI from wild decisions.

The problem with FAI is that it is nearly impossible for human minds of even high intellect to get good results solely through philosophy - without experimental feedback. Aristotle famously got it wrong when he deduced philosophically that rocks fall faster than feathers.

Also, I believe that it is a pointless endeavor for now. Here are 2 reasons why I think that's the case.

*1. We humans don't have any idea whatsoever as to what constitutes the essence of an intelligent system. Because of our limited intellects, our best bet is to simply take the one intelligent system that we know of - the human brain - and simply replicate it in an artificial manner. This is a far easier task than designing an intelligence from scratch, since in this case the part of design was already done by natural (and sexual) selection.

Our best hope and easiest path for AI is simply to replicate the human brain (preferably the brain of an intelligent and docile human being), and make a body suitable for it to inhabit. Henry Markram is working on this (hopefully he will use himself or someone like himself for the first template - instead of some stupid or deranged human), and he notably hasn't been terribly concerned with Friendly AI. Ask yourself this: what makes for FH (Friendly Humans)? And here we turn to neuroscience, evo-psych and... the thing that some people want to avoid discussing for fear of making others uncomfortable: HBD. People of higher average IQ are, on average, less predisposed to violence. Inbred populations are more predisposed to clannish behavior (we would ideally want an AI that is the opposite of that, that is most willing to be tolerant of out-groups). Some populations of human beings are more predisposed to violence, while some have a reputation for docility (you can see that in the crime rates). It's in the genes and the brain that they produce, combined with some environmental factors like random mutations, the way proteins fold and are expressed, etc.

So obviously the most promising way to create Friendly AI at this point in time is to replicate the brain of a Friendly Human.

*2. We might not be smart or creative enough on average to be able to build a FAI, or it might take too long a time to do so. This is a problem that, if exists, will not only not go away, but actually compound itself. As long as there are no restrictions whatsoever on reproduction and some form of welfarism and socialism exists in most nations on Earth, there will be dysgenics with regards to intelligence - since intelligent people generally have less children than those on the left half of the Bell curve - while the latter are basically subsidized to reproduce by means of wealth transfer from the rich (who are also more likely to have above-average IQs, else they wouldn't be rich).

Even if we do possess the knowledge to replicate the human brain, I believe it is highly unlikely that it will happen in a single generation. AI (friendly or not) is NOT just around the corner. Humanity doesn't even possess the ability to write a bugless operating system, or build a computer that obeys sane laws of personal computing. What's worse, it did possess the ability to built something reasonably close to these ideals, but that ability is lost today. If building FAI takes more than one generation, and the survival of billions of people depends on it, then we should rather have it sooner rather than later.

The current bottleneck with AI and most science in general is with the number of human minds able and willing to do it. Without the ability to mass-produce at least human-level AI, we simply desperately need to maximize the proportion of intelligent and conscientious human beings, by producing as many of them as possible. The sad truth is this: one Einstein or Feinman is more valuable when it comes to the continued well-being of humanity than 99% of the rest of human beings who are simply incapable of producing such high-level work and thought because of either genetics and environmental factors, i.e. conditions in the uterus, enough iodine, etc. The higher the average intelligence of humanity, the more science thrives.

Eugenics for intelligence is the obvious answer. This can be achieved through various means, discussed in this very good post on West Hunter. Just one example, which is one of the slowest but the one advanced nations are 100% capable of doing right now: advanced nations already possess the means to create embryos using the sperm and eggs of the best and brightest of scientists alive today. If our leaders simply conditioned welfare and even payments of large sums of money for the below-average IQ women on them acting as surrogate mothers for "genius" embryos, in 20-30 years we could have dozens of Feynmans and tens of thousands of Yudkowskys working on AI. This would have the added benefit on keeping the low-IQ mothers otherwise pregnant and unavailable for spreading low-IQ genes to the next generation, which would result in less people who are a net drain on the future society and would cause only time-consuming problems for the genius kids (like stealing their possessions or engaging in other criminal activities).

I do realize that increasing intelligence in this manner is bound to have an upper limit and, furthermore, will have some other drawbacks. The high incidence of Tay-Sachs disease among the 110 average IQ Ashkenazi Jews is an illustration of this. But I believe that the discoveries of the healthy high IQ people have the potential to provide more hedons than the dolors of the Tay-Sachs sufferers (or other afflictions of high-IQ people, including some less serious ones like myopia).

EDIT: Given the above, especially if *2. is indeed the case, it is not unreasonable to believe that donating to AmRen or Steve Sailer has greater utility than donating to SIAI. I believe that the brainpower at SIAI is better spent on a problem that is almost as difficult as FAI, namely making HBD acceptable discourse in the scientific and political circles (preferably without telling people who wouldn't fully grasp it and would instead use it as justification for hatred towards Blacks), and specifically peaceful, non-violent eugenics for intelligence as a policy for the improvement of human societies over time.