Intelligence Amplification and Friendly AI

I think this is an overly simplistic and binary way of looking at it:

Maybe FAI is so hard that we can only get FAI with a large team of IQ 200+ humans...I doubt (1) is true. I think IQ 130–170 humans could figure out FAI in 50–150 ...

If it's possible or not, IMHO, isn't the only question. A better question may be "What are the odds that a team of IQ 150 humans thinks they have developed a FAI and are correct, vs. the odds that they think they have developed a FAI and are wrong? Are those odds better or worse if it's a team of IQ 200+ individuals? "

I think that a group of normal people could probably develop a FAI. But I also think that a group of IA people are more likely to do so correctly on the first try without missing any vital details, given that in practice you may only have one shot at it.

I would also say that if something does go badly wrong, a group of IA people (or people with other augmentations, like brain-computer interfaces) probably have a better shot at figuring it out in time and responding properly (not necessarily that they have good odds at it, but probably at least better odds). They're also probably less likely to fail at other related forms of AI danger (trying to create a GAI that is designed to not self-improve, for example, at least not until the team is convinced that it is friendly; or maintaining control over some kind of oracle AI, or losing control over a narrow AI with significant destructive capability).

Note that I'm not necessarily saying that those are good ideas, but either way, AI risk is probably lowered if IA comes first. Very smart people still may intentionally make uFAI for whatever reason, but at least they're less likely to try to make FAI but mess it up.

Where in the linked article does it say that atheism correlates with IQ past 140? I cannot find this.

Eugenics might work for (2) if people interested in FAI become early users of the technology and raise their children to care about FAI. Perhaps MIRI and CFAR should offer to pay for intelligence-enhancing eugenics technologies as an employee benefit when these technologies become available.

I have been mulling around a rough and mostly unformed idea in my head regarding AI-first vs IA-first strategies, but I was loathe to try and put it into words until I saw this post, and noticed that one of the scenarios that I consider highly probable was completely absent.

On the basis that subhuman AGI poses minimal risk to humanity, and that IA increases the level of optimization ability required of an AI to be considered human level or above, it seems that there is a substantial probability that an IA-first strategy could lead to a scenario in which no superhuman AGI can be developed because it is economically infeasible to research that field as opposed to optimizing accelerating returns from IA creation and implementation. Development of AI whether friendly or not would certainly occur at a faster pace, but if IA proves to simply be easier than AI, which given our poor ability estimate the difficulty of both approaches may be true, development in that field would continue to outpace it. It could certainly instigate either a fast or slow takeoff event from our current perspective, but from the perspective of enhanced humans it would be simply an extension of existing trends.

A similar argument could be made in regard to Hanson's WBEM based scenarios, through the implication that given the ability to store a mind to some hardware system, it would be more economically efficient to emulate that mind at a faster pace than to parallel process multiple copies of that mind in the same hardware space, and likewise hardware design would trend toward rapid emulation of single workers rather than multiple instances in order to reduce costs accrued by redundancy and increase gains in efficiency accrued by experience. This would imply that mind enhancement of a few high efficiency minds would occur much earlier and that exceptional numbers of emulated workers would be unlikely to be created, but rather that a few high value workers would occupy a large majority of relevant hardware very soon after the creation of such technology.

An IA field with greater pace than AI does of course present its own problems, and I'm not trying to endorse moving towards an IA-first approach with my ramblings. I suppose I'm simply trying to express the belief that discussion of IA as an alternative to AI rather than an instrument toward AI is rather lacking in this forum and I find myself confused as to why.

Increased intelligence does not mean [EDIT: does not automatically translate to] increased rationality... but still, with proper education, more intelligent people could become rational more quickly.

If we invent a pill for increasing everyone's IQ by 50, distributing the pill to the world will not make it more rational. But if CFAR develops a curriculum for making people more rational, those who already took the pill will be more successful students.

(In the opposite direction: If a mad scientist releases a virus that will lower everyone's IQ to 80, some specific forms of irrationality may disappear... but CFAR's or MIRI's missions will become completely hopeless.)

At the moment, I (sadly) lean toward the view that slower Earths have a better chance at FAI. (Much of my brain doesn’t know this, though: I remember reading the Summers news with glee, and then remembering that on my current model this was actually bad news for FAI.)

Feeling the vertigo, huh? I went through a stage around the 2012 election where I thought growth was bad for somewhat similar reasons. It strongly affected how I voted. But with another year's consideration I admit to simply having no clue whether GDP growth overall is good for the far future. So I'd now say growth is good but that its importance shrinks with how much you care about the far future over the present.

That's uncertain, but I probably won't think much more about it given the likelihood of better future-affecting interventions than influencing growth.

(1) I've butted heads with you on timelines before. We're about a single decade away from AGI, if reasonable and appropriate resources are allocated to such a project. FAI in the sense that MIRI defines the term - provably friendly - may or may not be possible, and just finding that out is likely to take more time than we have left. I'm glad you made this post because if your estimate for a FAI theory timeline is correct, then MIRI is on the entirely wrong track, and alternatives or hybrid alternatives involving IA need to be considered. This is a discussion which needs to happen, and in public. (Aside: this is why I have not, and continue to refuse to donate to MIRI. You're solving the wrong problem, albeit with the best of intentions, and my money and time is better spent elsewhere.)

(2) Secrecy rarely has the intended outcome, is too easily undone, and is itself an self-destructive battle that would introduce severe risks. Achieving and maintaining operational security is a major entropy-fighting effort which distracts from the project goals, often drives away participants, and amplifies power dynamics among project leaders. That's a potent, and very bad mix.

(3-5) Seem mostly right, or wrong without negative consequences. I don't think there's any specific reasons in there not to take an IA route.

Take for example an UFAI (not actively unfriendly, just MIRI's definition of not-provably-FAI) tasked with the singular goal of augmenting the intelligence of humanity. This would be much safer than the Scary Idea strawman that MIRI usually paints, as it would in practice be engineering its own demise through an explicit goal to create runaway intelligence in humans.

If you were to actually implement this, the goal may need a little clarity:

• Instead of “humanity” you may need to explicitly specify a group of humans (pre-chosen by the community or a committee for their own history of moral action and rational decision making), as well as constraints that they all advance / are augmented at approximately the same rate.

• The AI should be penalized for any route it takes which results in the AI being even temporarily smarter than the humans. Presumably there is a tradeoff and an approximation here as the AI needs to be at least some level of superhuman smart in order to start the augmentation process, and needs to continue to improve itself in order to come up with even better augmentations. But it should favor plans which require smaller AI/human intelligence differentials.

• To prevent weird outcomes, utility of future states should be weighted by an exponential decay. The AI should be focused on just getting existing humans augmented in the near term, and not worry itself over what it thinks future outcome would be millennia from now - that's for the augmented humans to worry about.

And I'm sure there are literally hundreds of other potential problems and small protective tweaks required. I would rather that MIRI spent it's time and money working on scenarios like this and formulating the various risks and counter measures, rather than obsessing over Löb obstacles (an near complete waste of time).

This is similar to how things are done in computer security. We have a well understood repertoire of attacks and general counter measures. Cryptographers then design specific protocols which through their unique construction are not vulnerable to the known attacks, and auditors make sure that implementations are free of side-channel vulnerabilities and such. How many security systems are provably secure? Very few, and none if you consider that those which have proofs have underlying assumptions which are not universally true. Nevertheless the process works and with each iterative design we move the ball forward towards end goal of a fully secure system in practice.

I'm not interested in an airtight mathematical proof of an AGI design which by your own estimate take an order of magnitude longer to develop than an unfriendly AGI. Money and time spent towards that is better directed towards other projects. I'd much rather see effort towards the evaluation of existing designs for self-modifying AGI, such as the GOLEM architecture[1], and accompanying “safe” goal systems implementing hybrid IA like I outlined above, or an AGI nanny architecture, etc.

EDIT: See Wai Dei's post[2] for a similar argument.

EDIT2: If you want to down-vote, that's fine. But please explain why in a reply.

1. https://www.youtube.com/watch?v=XDf4uT70W-U

2. http://lesswrong.com/lw/7n5/wanted_backup_plans_for_seed_ai_turns_out_to_be/

...there could be a global nuclear war that kills all but about 100,000 people...

Did you mean to say 100,000,000?

Edit: D'oh, missed the "but."

Interesting analysis, not so much because it is particularly insightful on itself as it stands, but more because it gives a hard step back in order to have a wider view. I have been intending to investigate another alternative: a soft take off through moral enhancement initially solving the transference of value problem. This is not the only reason I decided to study this, but it does seem like a worthy idea to explore. Hopefully, I will have some interesting stuff to post here later. I am working on a doctoral thesis proposal about this, I use some material from LessWrong - but, for evil-academic reasons, not as often and directly as I would like. It would be nice to have some feedback from LW.

(2) looks awfully hard, unless we can find a powerful IA technique that also, say, gives you a 10% chance of cancer.

Possible candidate.

Edit: if the utility function of EAs working on FAI is really such that they would take a 10% chance of cancer for powerful IA, is it safe to assume that they are taking advantage of the moderate IA that's reportedly possible with existing nootropics, apparently without significant side effects? (Though, on that thread in particular, there's probably a bit of a selection effect where folks who use nootropics long-term are the ones who respond well to them.)

(2) looks awfully hard, unless we can find a powerful IA technique that also, say, gives you a 10% chance of cancer. Then some EAs devoted to building FAI might just use the technique, and maybe the AI community in general doesn’t.

Using early IA techniques is probably risky in most cases. Commited altruists might have a general advantage here.

You might repeat Intelligence Amplification here, as I was confused until I read the previous post links.