Could you tell me what's wrong with this?

I just realised, you're the guy from my first post. You're first sentence now makes a lot more sense. I think that the problem is not so much that I'm massively overconfident though that may also be the case) its just that when I'm writing here I appear too bold. I'll definitely try and reduce that, though I thought I had done fairly well on this post. Looking back, I guess I could've been clearer. I was thinking of putting a disclaimer at the beginning saying 'Warning! This post is not to be seen as representing the posters views. It is meant to be dismantled and thoroughly destroyed so the poster can learn about his AI misconceptions.' But it was late, and I couldn't put it into words properly.

Anyway, thanks for being patient with me. I must have sounded like a bit of a twat, and you've been pleasantly polite. It really is appreciated.

Not to put too fine a point on it but many of the posters here have never sat through a single class on AI of any kind nor read any books on actually programming AI's nor ever touched any of the common tools in the current state of the art nor learned about any existing or historical AI designs yet as long as they go along with the flow and stick to the right Applause Lights they don't get anyone demanding they go off and read papers on it.

As a result conversations on here can occasionally be frustratingly similar to theological discussions with posters who've read pop-comp-sci material like Superintelligence who simply assume that an AI will instantly gain any capability up to and including things which would require more energy than there exists in the universe or more computational power than would be available from turning every atom in the universe into computronium.

Beware of isolated demands for rigor.

http://slatestarcodex.com/2014/08/14/beware-isolated-demands-for-rigor/

Does one of these papers answer this? http://lesswrong.com/r/discussion/lw/m26/could_you_tell_me_whats_wrong_with_this/c9em I suspect these papers would be difficult and time-consuming to understand, so I would prefer to not read all of them just to figure this one out.

I think you are being a little too exacting here. True, most advances in well-studied fields are likely to be made by experts. That doesn't mean that non-experts should be barred from discussing the issue, for educational and entertainment purposes if nothing else.

That is not to say that there isn't a minimum level of subject-matter literacy required for an acceptable post, especially when the poster in question posts frequently. I imagine your point may be that Algon has not cleared that threshold (or is close to the line) - but your post seems to imply a MUCH higher threshold for posting.

The main purpose of this post is not to actually propose a solution; I do not think I have some golden idea here. And if I gave that impression, I really am sorry (not sarcastic). What this was meant to be was a learning opportunity, because I really couldn't see many things wrong with this avenue. So I posted it here to see what was wrong with it, and so far I've had a few people reply and give me decent reasons as to why it is wrong. I've quibbled with a few of them, but that'll probably be cleared up in time.

Though I guess my post was probably too confident in tone. I'll try to correct that in the future and make my intentions better known. I hope I've cleared up any misunderstandings, and thanks for taking the time to reply. I'll certainly check those recommendation out.

Also, go do a couple years of study of theology before you can talk about whether God exists or not.

Why do people down vote this? "Seed AI" is basically the correct search term to find this idea discussed in detail.

You may have a point there. But I think that the problem's you've outlined are ones that we could circumvent.

With 1) We don't know exactly how to describe what an FAI should do, or be like, so we might present an AI with the challenge of 'what would an FAI be like for humanity?' and then use that as a goal for FAI research.

2) I should think that its technically possible to construct it in such a way so that it can't just become a super-intellect, whilst still allowing it to grow in pursuit of its goal. I would have to think for a while to present a decent starting point to this task, but I think it is more reasonable than solving the FAI problem.

3) I should think that this is partly circumvented by 1) If we know what it should look like, we can examine it to see if its going in the right direction. Since it will be constructed by a human level intellect, we should notice any errors. And if anything does slip through, then the next AI would be able to pick it up. I mean, that's part of the point of the couple of years (or less) time limit; we can stop an AI before it becomes too powerful, or malevolent, and the next AI would not be predisposed in that way, so we can make sure it doesn't happen again.

Thanks for replying, though. You made some good points. I hope I have adjusted the plan so that it is more to your liking (not sarcastic, I just didn't know the best way to phrase that)

EDIT: By the way, this is strictly a backup. I am not saying that we shouldn't pursue FAI. I'm just saying that this might be a reasonable avenue to pursue if it becomes clear that FAI is just too damn tough.

An AGI also has the advantage of being able to duplicate instances. If you would create 1000000 copies of the 10 best computer security researchers, that army could likely hack itself into every computer system available.

Just a little nitpick here, but your last statement is unlikely. There are actual computer systems that are completely unhackable even by any team of skilled security researchers (non-networked devices, etc).

Of course, the generalization of your point is much more likely, but the likely forms of AI takeover probably involve a diverse set of skills, including subtle and complex strategic social/cultural manipulations. Human brains are more hackable than many computer systems in this regard.

Well, that's another limitation you'd have to put on it. I get that the AI would work differently to a human. And now that I think of it, every year or so when you reset it, you could adjust it so its better suited for the task at hand. Thanks for the reply!

Yeah, the first point started to make itself clear as I read through the comments. I think that the 'how do you verify...' part of your post is the most compelling. When we look at it, we don't know whether or not there will be any problems with this. But since it was produced by human level intellect, we should be able to understand it.

I had the idea of the second in my mind; I think that this scenario has a fair chance of happening, but a large group of humans would be able to stop the AIs attempts. Of course, they could still let it slip past their fingers, which would be Bad. I would say that that is preferable to having FAI turn out to be beyond us, and waiting until some tit decides to go ahead with an AGI anyway. But that does not make it a pleasant choice.

All in all, I think that this avenue wasn't so terrible, but it is still fairly risky. It seems that this is the sort of thing you'd use as maybe a Plan 3, or Plan 2 if someone came along and made it much more robust.

Hmm. So you too think that this fails as a backup? I guess I overestimated the difficulty of doing such a thing; some others have commented that this path would be even more difficult than just creating FAI.

Though I am not suggesting we kill the AIs, as that would be highly unethical.I don't think that the prospect of being put to sleep temporarily would be such an issue. You might also put that as part of their terminal value, alongside the 'solve FAI' or describe FAI. I responded to another commentator who made a similar comment to yours. Do you mind checking it out and saying what you thinks wrong with my reply? I'd be quite grateful.

P.S. some people have misunderstood my intentions here. I do not in any way expect this to be the NEXT GREAT IDEA. I just couldn't see anything wrong with this, which almost certainly meant there were gaps in my knowledge. And I thought the fastest way to see where I went wrong would be to post my idea here and see what people say. I apologise for any confusion I caused. I'll try to be more clear next time.

I'm not sure how useful or relevant a point this is, but I was just thinking about this when I saw the comment: IQ is defined within the range of human ability, where an arbitrarily large IQ just means being the smartest human in an arbitrarily large human population. "IQ 1000" and "IQ 180" might both be close enough to the asymptote of the upper limit of natural human ability that the difference is indiscernible. Quantum probabilities of humans being born with really weird "superhuman" brain architectures notwithstanding, a truly superintelligent being might have a true IQ of "infinity" or "N/A", which sounds much less likely to stick to the expectations we have of human savants.

Seems to me the "killer app" is not superintelligence per se, but superintelligence plus self-modification. With highly intelligent people that problem is often that they can't modify themselves, so in addition to their high intelligence they also have some problems they can't rid of.

Maybe Kasparov plays chess because he genuinely believes that playing chess is the most useful thing he could ever do. But more likely there are other reasons; for example he probably enjoys playing chess, and the more useful things are boring to him. Or maybe he is emotionally insecure and wants to stay in an area he is already good at, because he couldn't emotionally bear giving up the glory and starting something as a total newbie, even if a few years later it would pay off. (This is just a random idea; I have no idea what Kasparov really is like. Also, maybe he is doing other things too, we just don't know about them.)

Imagine a person who would be able to edit their own mind. For example, if they would realize that their plans would advance further if they had better social skills, they would (a) temporarily self-modify to enjoy scientifically studying social skills, then (b) do a research about social skills and find out what is evidence-based, and finally (c) self-modify to have those traits that reliably work. And then they would approach every obstacle in the same way. Procrastinating too much? Find the parts of your mind that make you do so, and edit them out. Want to stay fit, but you hate exercising? Modify yourself to enjoy exercising, but only for a limited time every day. Want to study finance, but have a childhood emotional trauma related to money? Remove the trauma.

You would get a person who, upon seeing an optimal way, would start following that way. They would probably soon realize that their time is a limited resource, and start employing other people to do some work for them. They would use various possible computer tools, maybe even employ people to improve those tools for them. They would research possibilities of self-improvement, use them for themselves, and also trade the knowledge with other people for resources or loyalty. After a while, they would have an "army" of loyal improved followers, and would also interact a lot with the rest of the world. And that would be just a beginning.

Maybe for a human there would be some obstacle, for example that the average human life is too short to research and implement immortality. Or maybe they would reach the escape velocity and become something transhuman.

Most savants simply don't care about changing the world

I would guess they still find some things frustrating (e.g. when someone or something is interrupting them from their hobby), they are just no strategic enough to remove all sources of frustration. Either they do not bother to plan long-term, or they don't believe such long-term planning could work.

the 190 IQ son of a waiter is probably a cook

Let's imagine a country with a strict caste system, where the IQ 190 person born in the lower caste must remain there. If the society is really protected against any attempts to break the system, for example if the people from the lower caste are forbidden all education and internet, and are under constant surveillance, there is probably not much he could do. But if it's a society more or less like ours, only it is illegal and socially frowned upon hiring people to do other caste's work, a strategic person could start exploring possibilites to cheat -- for example, could they fake a different, higher-caste identity? Or perhaps move to a country without the caste system. Or if there are exceptions where the change of caste is allowed, they would try one, and could try to cheat to get the exception easier. (For example, if a higher-caste person can "adopt" you into their caste, you could try to make a deal with one, or blackmail one, or maybe somehow fake the whole process of being adopted.) They could also try to somehow undermine the caste system; create a community that ignores the caste rules; etc.

Well, I suspect that this is a known possibility to AI researchers. I mean, when I've heard people talk about the problems in AGI they're not so much saying that the intelligence can only be supremely good or supremely bad, but they're just showing the range of possibilities. Some have mentioned how AI might just advance society enough so that they can get off world and explore the universe, or it might just turn out to be completely uninterested in our goals and just laze about playing GO.

When people talk about FAI, I think they're not just saying 'We need to make an AI that is not going to kill us all' because although that sort of AI may not end all humanity, it may not help us either. Part of the goal of FAI is that you get a specific sort of AI that will want to help us, and not be like what you've described.

But I suspect someone vastly more knowledgeable than me on this subject will come and address your problem.