Building toward a Friendly AI team

Some comments on the recruiting plan:

1. I think a highly rational person would have high moral uncertainty at this point and not necessarily be described as "altruistic". For example I consider Eliezer's apparent high certainty in utilitarianism (assuming it's not just a front for PR purposes) as evidence against his rationality. Given a choice between a more altruistic candidate and a more rational candidate, I think SI ought to choose the latter.
2. Similarly for "deeply committed to AI risk reduction". I think a highly rational person would think that working on AI risk reduction is probably the best thing to do at this point but would be pretty uncertain about this and be ready to change their mind if new evidence or theories come along.
3. What does "trustworthy" mean, apart from "rationality"? Something like psychological stability?
4. It seems like the plan is to have one Eliezer-type (philosophy oriented) person in the team with the rest being math focused. I don't understand why it isn't more like half and half, or aiming for a balance of skills in all recruits. If there is only one philosophy oriented person in the team, how will the others catch his mistakes? If the reason is that you don't expect to be able to recruit more than one Eliezer-type (of sufficient skill), then I think that's enough reason to not build an FAI team.
5. I think a highly desirable trait in an FAI team member is having a strong suspicion that flaws lurk in every idea. This seems to work better in motivating one to try to find flaws than just "having something to protect".

especially skilled in maths, probably at the IMO medal-winning level

What is with you guys and the math olympiad?

Hi, I'm new here, so I'm not quite familiar with all the ideas here. However, I am a young mathematician who has some familiarity with how mathematical theories are developed.

Highly intelligent, and especially skilled in maths, probably at the IMO medal-winning level. (FAI team members will need to create lots of new math during the course of the FAI research initiative.)

It might be much cheaper to accept more average mathematicians who meet the other criteria. Generally, to build a new theory, you'll need a few people who can come up with lots of creative ideas, and lots of people who are capable of understanding the ideas, and then taking those ideas and building them into a fleshed out theory. Many mathematicians accept that they are of the second type, and work towards developing a theory to the point where a new creative type can clearly see what new ideas are needed.

Trustworthy. (Most FAI work is not "Friendliness theory" but instead AI architectures work that could be made more dangerous if released to a wider community that is less concerned with AI safety.)

Shouldn't this just be a subset of number 5? I'm sure you would rather have someone who would lie to keep AI risk low than someone who would tell the truth no matter what the cost.

The trouble is that the number of people on Earth who qualify may be very close to 0.

This sounds terribly arrogant until you realize that the requirement of "Deeply committed to AI risk reduction" is on the list. Should probably be emphasized around this statement in the post.

especially skilled in maths, probably at the IMO medal-winning level

(Should distinguish raw intelligence, contest training and research math training. Raw intelligence is crucial for good performance in both contests and math research, but getting good at math takes many years of training that IMO winners won't automatically have.)

Deeply committed to AI risk reduction. (It would be risky to have people who could be pulled off the team—with all their potentially dangerous knowledge—by offers from hedge funds or Google.)

To me this seems naive. Having someone with actually worked in SI on FAI going to Google might be a good thing. It creates connection between Google and SI. If he sees major issues inside Google that invalidate your work on FAI he might be able to alert you. If Google does something that dangerous according to the SI consensus then he's around to tell them about the danger.

Being open is a good thing.

I'm going to open my clueless mouth again: Many of the problems associated with FAI haven't been defined to that well yet. Maybe solving them will require new math, but it seems possible that existing math already provides the necessary tools. Perhaps it would be a good idea to have a generalist who has limited familiarity with a large variety of mathematical tools and can direct the team towards existing tools that might solve their problem. See the section called "The Right Way To Learn Math" in this post for more:

http://steve-yegge.blogspot.com/2006/03/math-for-programmers.html

And a metalevel comment: Presumably folks at SI are discussing these issues independently of the discussion on Less Wrong; they don't seem to be posting here much. I'm curious why this is considered optimal. It seems to me that posting your arguments on Internet is a good way to get falsifying evidence for them. If the box does not contain a diamond, I wish to believe the box does not contain a diamond and whatnot.

Either way, I think that building toward an FAI team is good for AI risk reduction, even if we decide (later) that an SI-hosted FAI team is not the best thing to do.

I question this assumption. I think that building an FAI team may damage your overall goal of AI risk reduction for several reasons:

1. By setting yourself up as a competitor to other AGI research efforts, you strongly decrease the chance that they will listen to you. It will be far easier for them to write off your calls for consideration of friendliness issues as self-serving.

2. You risk undermining your credibility on risk reduction by tarring yourselves as crackpots. In particular, looking for good mathematicians to work out your theories comes off as "we already know the truth, now we just need people to prove it."

3. You're a small organization. Splitting your focus is not a recipe for greater effectiveness.

The idea of getting FAI contributors who are unlikely to ever switch jobs seems like it might be the most stringent hiring requirement. It might be worthwhile to look into people who gain government clearances and then move to a nongovernment job, to see if they abuse the top-secret information they had access to.

Shift the Singularity Summit toward being more directly useful for AI risk reduction, and also toward greater profitability—so that we have at least one funding source that is not donations. (Currently underway.)

This in particular seems to be a good subgoal, and I would be interested in the details of what a more directly useful Singularity Summit looks like, and how you get there. (I attended in 2010, and found it to be fun, somewhat educational, but unfocused. (And somewhat useful in that it attracted attention to SIAI.))

The kind of people we'd need for an FAI team are:

1. Highly intelligent, and especially skilled in maths, probably at the IMO medal-winning level. (FAI team members will need to create lots of new math during the course of the FAI research initiative.)
2. Trustworthy. (Most FAI work is not "Friendliness theory" but instead AI architectures work that could be made more dangerous if released to a wider community that is less concerned with AI safety.)

If FAI is or can be made tractable, it will be a technological system: some combination of hardware and software, an actual practical invention. The parenthetical comment in your second point indicates you seem to acknowledge that FAI work mainly consists of safe AI architecture work.

If you look back on the mountain of historical evidence concerning invention, there is a rough general pattern or archetype for inventors. They may have more names or synonyms today: entrepreneur, hacker, programmer, engineer, etc, but the historical pattern remains.

The inventor mentality is characterized by relentless curiosity, creativity, dedication, knowledge, and intelligence. Formal education appears to be of little importance, and may in fact slightly negatively correlate with invention capability. Bill Gates dropped out of college, but Orville Wright dropped out of high school, and this trend appears across the technological landscape. Early mathematical ability may correlate with later inventorhood, but studying formal mathematics (being a mathematician) has a strong negative correlation with later invention. One immediate explanation is that any time spent studying formal mathematics is a waste of extremely precious higher cortical capacity which rivals are entirely devoting to pure technological study.

The Wright brothers didn't need much or any mathematics to innovate in flight. They needed to understand flight, understand the landscape of prior art, and then quickly iterate and innovate within that space. Their most powerful tool was not mathematics or 'rationality', but rather the wind tunnel.

We can attract some of the people meeting these criteria by . ..

You don't need accomplished mathematicians, if anything they would actually reduce your chances of success.

You need to attract the specific people who are going to or would develop AGI before you. They will almost certainly not be mathematicians (think Wright, Farnsworth, Edison, Tesla, Bill Gates, and not Terrance Tao). The historical evidence says they are unlikely to have any existing record of high status work.

How could you identify a future AGI-inventor? That should be the question, and the answer is clearly not "recruit math contest winners".

Hopefully you have also considered extracting specific limited-scope math problems and farming them out with grants, like you do for papers. This would increase the pool of available talent and not require training them in AI or rationality.

Shouldn't the very first goal be to fully define an ethical theory of friendliness before even starting on a goal system to implement the theory? I have some doubts that an acceptable theory can be formalized. Our ethical systems are so human-centric that formalizing them in rational terms will likely lead to either a very weak human-centric theory with potential loopholes or a general theory of ethics that places no particular importance on the concerns of humans, even large groups of them. For instance, I find it much more likely that a general theory of ethics would have little ability to differentiate between the desires of rocks and dirt, bacteria, and humans. What measurable properties of humans makes our actions worthy of more attention than the actions of bacteria or of a solar system or galaxy? Is it the ability for self-referential thought, a consciousness, or conscience? Even assuming a suitable definition for a class of beings whose utilities are valued can be found, the weighting of utility for each being will almost certainly not be in our favor. There are potentially infinitely many more potential future beings whose values and existence depend on actions in the present. There are potentially existing extraterrestrial beings whose values must be considered in case we ever affect them.

Maybe I have missed important discussions where the details of friendliness theory have already been hashed out. If so, I apologize and would appreciate links to them.

I think the other strategy you mentioned, about promoting FAI research by paying known academics to write papers on the topic, is a better idea. It is more plausible, more direct, less cultish, etc.

Trustworthy. (Most FAI work is not "Friendliness theory" but instead AI architectures work that could be made more dangerous if released to a wider community that is less concerned with AI safety.)

Ah, so this explains why there is no source code visible. To be honest, I was a little worried that all the effort was purely going to (very useful) essays and not actually building things, but it is clear that it is kept under wraps. That is such a shame, but I suppose necessary.

One caution worth noting here is that "trustworthiness" and "altruism" may not be traits that are stable across different situations. As I noted in this post, there's good reason to think human behavior evolved to follow conditional rules, so observed trustworthiness and altruism under some conditions may be very poor evidence of Friendliness for superintelligence-coding purposes.

Professional cultivation of big donors

I've seen a couple sources argue that intelligence enhancement will ideally come before AGI. This could deal with the math ability constraint, which seems to be your strongest. Maybe you feel that sponsoring an intelligence enhancement effort would be beyond SI's organizational scope?

Some important aspects of the future AI 'friendliness' would probably link up with the greater economy surrounding us; and more importantly, it would depend upon the nature of AI interaction with people as well as their behaviour. So, besides the obvious component of mathematics, I feel that some members of the FAI team should also have some background in subjects such as psychology; and also a generic perspective on global issues such as resource management.

Somehow absent from the objectives is "finding out if SI's existence is at all warranted". You also want people "deeply committed to AI risk reduction", not people deeply committed to e.g. actually finding the truth (which could be that all properties of AI you take for true were a wrong wild guess, which should be considered highly likely due to shoot in the dark nature of those guesses). This puts nail in your coffin completely. A former theologist building a religious organization. Jesus Christ, man.

Also, something else: in the game development there's the recurring pattern of a newbie or newbies with MMORPG ideas who only needs to get super tech guys to implement it. Thing is, tech guys do not need those newbies at all, they with their ideas are a net negative (because the ideas are misinformed) and represent the category of people to be kept out of development entirely, and the worst that can be done is channelling the money through such newbies. I don't see why you guys (SI, MMORPG development newbies, all sorts of startup founders whom only have ideas) even think you (instead of others) having the money to hire people to develop your ideas is not a net negative. You think you can have incredibly useful ideas without having powers to develop anything, while the fact is that the power to have useful ideas stems directly from the power to build those ideas bottom up. I think the best recommendation for those with money would be to hold on and not give them to SI, so that SI would not put incompetent ideas ahead and waste people time. All chances are AI risk reduction doesn't even need your money, that kind of work won't be someone's first innovative work, and innovation pays off pretty damn well. You want to decrease AI risk, or you want awesome MMORPG made, or something - what ever you want, please do not funnel your money through this kind of idea guys. It really is net negative.