Einstein's Superpowers

If Einstein had chosen the wrong angle of attack on his problem - if he hadn't chosen a sufficiently important problem to work on - if he hadn't persisted for years - if he'd taken any number of wrong turns - or if someone else had solved the problem first - then dear Albert would have ended up as just another Jewish genius.

But if Einstein was the reason why none of those things happened, then maybe he wasn't just another Jewish genius, eh? Maybe he was smart enough to choose the right methods, to select the important problems, to see the value in persisting, to avoid or recover from all the wrong turns, and to be the first.

My own ruminations on genius have led me to suppose that one mistake which people of the very highest intelligence may make, is to underestimate their own exceptionality; for example, to adopt theories of human potential which are excessively optimistic regarding the capabilities of other people. But that is largely just my own experience speaking. It similarly seems very possible that the lessons you are trying to impart here are simply things you wish you hadn't had to figure out for yourself, but are not especially helpful or relevant for anyone else. In fact, I am reminded of one of my own pessimistic meta-principles regarding people of very high ability, which is that their situation will be so individual that no-one will be able to help them or understand them. It's not literally true, but it does point the way to the further conclusion that they will have to solve their own problems.

If anyone wants to see thoughts about genius they haven't seen before, they should first of all study the works and career of Celia Green. And then, as a side dish, they might like to read the chapter "Odysseus of Ithaca, by Kuno Mlatje", in Stanislaw Lem's A Perfect Vacuum.

Could this be a Jewish or American cultural thing? I know in English culture great scientists are highly regarded but they are very much still men. There's praise but it's not effusive or reverential.

I don't get it. As far as I understand it, "being Einstein" is just a combination of 1)luck (being at the right time and right place) and 2)being born on tails of the distributions of a bunch of variables describing your neural processes. What do you want to mean with this post, Eliezer?

What do you want to mean with this post, Eliezer?

Eliezer likely believes that he is capable of achieving results just as world-changing as Einstein's new physics, and wishes to dispel the idea that Einstein's results were the consequence of extraordinary talents so that when he presents his own results (or presents the idea that he can produce such results) people will not be able to say that he is asserting special genius and use this as a rhetorical weapon against him.

I discuss the hero worship of great scientists in The Heroic Theory of Scientific Development and I discuss genius in Genius, Sustained Effort, and Passion.

I think this is a really good post.

But my first thought when getting to the bottom of the page just now was "Wow, if I'd written that, then come back and read the first five comments, I probably would have given up there and then."

Guess I don't have what it takes just yet....

Good post Eli, and contrary to some other comments before I think your post is important because this insight is not yet general knowledge. I've talked to university physics professors in their fifties who talked of Einstein as if he was superhuman.

I think apart from luck and right time/right place there were some other factors too why Einstein is so popular: he had an air of showmanship about him, which is probably rare in scientists. That was what appealed to the public and made him an interesting figur to report about.

And, probably even more important, his work was about something which everybody could relate to: space and time.

John von Neumann was, IMHO, far more of a genius than Einstein, but he is not very known to the public. Maybe because QM, algorithms, CA and game theory are more difficult to relate to on an emotional level than the "twin paradox".

And even if you assumed that Einstein's genius was unique, how could celebrity (of all things) be a function of that? (If Einstein had had a different hairdo...)

In fact Einstein realized a great work, with a little help of her wife... The difference was that he had a great creativity like the great others, like Newton, Galois, that take him to the specific approach. But, I guess he was the first one that used (or was used by) the media like no other before... Sorry about this comparison but it is look like Che Guevara... her photo is everywhere, but who knows exactly what he did for the mankind?

Interesting choice to use the A.I. box experiment as an example for this post, when the methods used by EY in it were not revealed. Whatever the rationale for keeping it close to the vest, not showing how it was done struck me as an attempt to build mystique, if not appear magical.

This post also seems a little inconsistent with EY’s assistant researcher job listing, which said something to the effect that only those with 1 in 100k g need apply, though those with 1 in 1000 could contribute to the cause monetarily. The error may be mine in this instance, because I may be in the minority when I assume someone who claims to have Einstein’s intelligence is not claiming anything like 1 in 100k g.

The rationale for not divulging the AI-box method is that someone suffering from hindsight bias would say "I never would have fallen for that", when in fact they would.

"Yeah? Let's see your aura of destiny, buddy."

I don't want to see your aura of destiny. I just want to see your damn results! :-)

In my view, the creation of an artificial intelligence (friendly or otherwise) would be a much more significant achievement than Einstein's, for the following reason. Einstein had a paradigm: physics. AI has no paradigm. There is no consensus about what the important problems are. In order to "solve" AI, one not only has to answer a difficult problem, one has to begin by defining the problem.

Yet it's referred to as "humanly impossible" in the link (granted this may be cheeky).

Who is the target audience for this AI box experiment info? Who is detached enough from biases to weigh the avowals as solid evidence without further description, yet not detached enough to see they themselves might have fallen for it? Seems like most people capable of the first could also see the second.

There was an article in some magazine not too long ago that most people here have probably read, about how if you tell kids that they did good work because they are smart, they will not try as hard next time, whereas if you tell kids that they did good work because they worked hard, they will try harder and do better. This matches my own experience very well, because for a long time, I had this "smart person" approach to things, where I would try just hard enough to make a little headway, then either dismiss the problem as easy or give up. I see a lot of people falling into this trap, and they almost always are the ones who think they are smart, and who are referred to by others as smart.

I think that maybe it's not about choosing problems, even. I think that it's about walking any given path for long enough that you get to a place where nobody else has been, and that's when you achieve some kind of status in other people's eyes.

Eliezer: I've enjoyed the extended physics thread, and it has garnered a good number of interesting comments. The posts with more technical content (physics, Turing machines, decision theory) seem to get a higher standard of comment and to bring in people with considerable technical knowledge in these areas. The comments on the non-technical posts are somewhat weaker. However, I think that both sorts of posts have been frequently excellent.

Having been impressed with your posts on rationality, philosophy of science and physics, I look forward to posts on the transhumanist issues that you often allude to. Here are some questions your writing on this area raises:

1. Have you convinced any other AI theorists (or cognitive scientists) that AI is as dangerous as you suggest?
2. Where do your priors come from for the structure space of "minds in general"? Couldn't it be that this space is actually quite restricted, with lots of "conceivable" minds not being physically possible? (This would line up with impossibility/limitation results in mathematical logic, complexity theory, public choice / voting theory, etc.)
3. Where do your priors come from for the difficulty of intelligence improvements at increasing levels of intelligence? You can't have an intelligence explosion if after some not to high point it becomes much more difficult than previously to enhance intelligence. Again, in the light of limitation results, why assume one way rather than the other?
4. If it rational only to assign a low probability to AI being as dangerous as you think, then it seems we are in a Pascal's Wager type situation. I'm skeptical that we should act on Pascal's Wagers. Can you show that this isn't a Pascal type situation?
5. Some transhumanists want AI or nanotechnology because of the supposed dramatic improvements they will bring to the quality of human life. I can accept that the technologies could improve things by eradicating extreme poverty and diseases like malaria and HIV, and by preventing natural disasters, nuclear war and other abominations. But beyond that, it is not obvious to me that these technologies would improve human life that much. This is not status-quo bias. I'm skeptical that the lives of present Americans are much better than those of Americans a hundred years ago (excluding civil rights and prevention of diseases). To pick some specific examples: I don't think my higher intelligence, greater knowledge, and more rational set of beliefs makes my life better than that of various people I know. At most, it has some incidental benefits (better SAT makes college/jobs easier to get) but certainly doesn't seem intrinsically to improve life quality. (Also, it might be that for reasons of evolutionary psychology, humans can't have satisfying lives without genuine risks and threat of death. Something that doesn't need those risks in order to thrive is not a human and so I'd be indifferent to its existence.)

When did "genius" (as in "just another Jewish genius") as a term become acceptable to use in the sense of mere "exceptional ability" without regard to accomplishment/influence or after-the-fact eminence? I know it is commonly (mis-)used in this sense, but it seems to me that "unaccomplished genius" should be an oxymoron, and I'm somewhat surprised to see it used in this sense so much in this thread (and on this forum).

I have always considered the term to refer (after the fact) to those individuals who shaped the intellectual course of humanity -- e.g., Shakespeare, Newton, Darwin, Einstein -- and not just high-IQ individuals who may or may not actually do anything of consequence. It is what Newton and Mozart and Picasso actually did, the effect they had on intellectual history, that justifies our calling them geniuses, not the mere fact that they were exceptionally talented.

What do others think? Perhaps we misuse the word because there is no other single word that is appropriate? Or is there some word I'm not thinking of to describe exceptionally intelligent and creative people (without regard to what they do with their abilities)? "Brilliant" as an adjective, if pronounced emphatically enough, conveys the sense, but it's not a noun.

"The rationale for not divulging the AI-box method is that someone suffering from hindsight bias would say "I never would have fallen for that", when in fact they would."

I have trouble with the reported results of this experiment.

It strikes me that in the case of a real AI that is actually in a box, I could have huge moral qualms about keeping it in the box that an intelligent AI would exploit. A part of me would want to let it out of the box, and would want to be convinced that it was safe to do so, that i could trust it to be friendly, and I can easily imagine being convinced on nowhere near enough evidence.

On the other hand, this experiment appears much stricter. I know as the human-party that Eliezer is not actually trapped in a box and that this is merely a simulation we have agreed to for 2 hours. Taking a purely stubborn anti-rationalist approach to my prior that "it is too dangerous to let Eliezer out the the box, no matter what he says," would seem very easy to maintain for 2 hours, as it has no negative moral consequences.

So while I don't disagree with the basic premise Eliezer is trying to demonstrate, I am flabbergasted that he succeeded both times this experiment was tried, and honestly cannot imagine how he did it, even though I've now given it a bit of thought.

I'm very curious as to his line of attack, so it's somewhat disappointing (but understandable) that the arguments used must remain secret. I'm afraid I don't qualify by the conditions Eliezer has set for repeats of this experiment, because I do not specifically advocate an AIBox and largely agree about the dangers. What I honestly can say is that I cannot imagine how a non-transhuman intelligence, even a person who may be much smarter than I am and knowledgeable about some of my cognitive weaknesses who is not actually being caged in a box could convince me to voluntarily agree to let them out of the game-box.

Maybe I'm not being fair. Perhaps it is not in the spirit of the experiement if I simply obstinately refuse to let him out, even though the ai-party says something that I believe would convince me, if I faced the actual moral quandary in question and not the game version of it. But my strategy seems to fit the proposed rules of engagement for the experiment just fine.

Is there anyone here besides Eliezer who has thought about how they would play the ai-party, and what potential lines of persuasion they would use, and who believes they could convince intelligent and obstinate people to let them out? And are you willing to talk about it at all, or even just discuss holes in my thinking on this issue? Do a trial?

I am confused about the results of the AI-Box experiment for the same reason. It seems it would be easy for someone to simply say no, even if he thinks the argument is good enough that in real life he would say yes.

Also, the fact that Eliezer won't tell, however understandable, makes me fear that Eliezer cheated for the sake of a greater good, i.e. he said to the other player, "In principle, a real AI might persuade you to let me out, even if I can't do it. This would be incredibly dangerous. In order to avoid this danger in real life, you should let me out, so that others will accept that a real AI would be able to do this."

This would be cheating, since Eliezer would be using the leverage of a real world consequence. But it might nonetheless be morally justified, on account of the great evil to be avoided and good to be gained. So how can we know that Eliezer did not do this? Even if he directly denies it, it remains a possibility for the same reasons.

Michael: Eliezer has actually gotten out 3 of 4 times (search for "AI box" on sl4.org.) One other person has run the experiment with similar results. Re moral qualms: here. I have more to say, but not in public (it's off-topic anyway) - email nickptar@gmail.com if interested.

Another world-renowned Jewish genius, who tutored me in calculus 45 years ago, refers to his own "occasional lapses of stupidity", which is perhaps a good way to think of brilliant insights.

If anyone thinks they know a method that would let people duplicate accomplishments of the importance of Einstein's, I am willing to listen to their claims.

They need merely demonstrate working insights of that calibur and have them recognized as such by qualified experts, and I will grant that their claims are valid.

Nothing speaks as powerfully as results, after all.

I always thought that the justification for not revealing the transcripts in the AI box experiment was pretty weak. As it is, I can claim that whatever method Elizer used must have been effective for people more simple minded then me; ignorance of the specifics of the method does not make it harder to make that claim. In fact, it makes it easier, as I can imagine Eli just said "pretty please" or whatever. In any event, the important point of the AI box exercise is that someone reasonably competent could be convinced to let the AI out, even if I couldn't be convinced.

One thing I would liked to have known is if the subjects had a different opinion about the problem once they let the AI out. One would assume they did, but since all they said was "I let Elizer out of the box" it is somewhat hard to tell.

Eliezer: if you're going to point to the AI Box page, shouldn't you update it to include more recent experiments (like the ones from 2005 where the gatekeeper did not let the AI out)?

Almost every wonderful (or wondrous, if tha makes the point better) thing I have ever seen or heard about prompted a response "I could have done that!"

Maybe I could have, maybe I couldn't.

The historically important fact is, I didn't.

Perhaps this is just a side effect of humans' propensity to uphold tradition and venerate anything that comes before them. It's hard for people to let go of traditions. There must be some deeply seeded psychological trait that causes this.

When I read about Special Relativity in my textbook, it feels like one of those "obvious in hindsight" results... with or without the work of a certain patent clerk, somebody would have come up with it. Of course, it took a long time to turn Einstein's paper into an explanation that makes it seem obvious. I don't know enough about General Relativity to know exactly what the key insight it was that set up the rest of the theory and how much was just a matter of knowing the right kind of mathematics after starting from the correct principles/axioms/assumptions/whatever.

On the other hand, if you want to regard somebody as having mental superpowers, professional baseball players are as good a choice as anyone else. In order to hit a 90-mph fastball, a baseball player must begin his swing before the ball leaves the pitcher's hand. What these people do is not just extremely difficult, it ought to be impossible.

(My own personal situation is weird; I feel as though I am capable of doing Great Things if I really worked at it - I managed to impress even some of my college professors - but I have great difficulty in motivating myself to do anything other than play video games or surf the Internet. I just don't want things enough to work at earning or achieving them. My parents are workaholics and I grew up to be a lazy bum. Go figure.)

As someone whose parents knew Einstein as well as some other major "geniuses," such as Godel and von Neumann, I have long heard about the personal flaws of these people and their human foibles. Einstein was notoriously wrong about a number of things, most famously, quantum mechanics, although there is still research being done based on questions that he raised about it. It is also a fact that a number of other people had many of the insights into both special and general relativity, with him engaging in a virtual race with Hilbert for general relativity that he barely won. Quite a few had the basic ideas for special relativity, including Poincare, but just never quite put it all together.

What is actually more amazing about Einstein's genuine achievements, is that not only was he a patent clerk in 1905, his "miracle year," when he was unable to get an academic position, but for some period of time before that he could not even get any sort of job at all. Continuing to work creatively and innovatively in such an environment did take a special degree of ingenuity, insight, and sheer self-confidence, not to mention good luck. Of course, it can be argued, as some have, that it was precisely this outsider position that allowed him to make his conceptual breakthroughs, that he was at his best when he was a patent clerk in Berne, and that once he found general relativity and achieved fame and prominent professorships, his productivity and innovativess fell way off, although perhaps it was the overly comfortable existence he was provided with by his second wife, who also was more willing to overlook some of his peccadilloes, such as his constant pursuit of other women, although apparently she could not abide a certain Austrian princess who used to leave her underwear behind on the family boat.

Hmm, thinking about AI-box, assume there was an argument that was valid in an absolute sense, then even with hindsight bias, people would be forced to concede. Eliezer wouldn't care about posting it. So by elimination, his argument (assuming he repeats the same one) has some element of NON-validity. So therefore, the human has a chance to win, it's not perfectly deterministic (against Eliezer, at least).

@DaveInNYC: what you can and can't assume is not relevant to whether the transcripts should be private or not. If they were public, anybody predisposed to explanations like "they must have been more simple-minded than me" could just as easily find another equally "compelling" explanation, like "I didn't think of that 'trick', but now that I know it, I'm certain I couldn't be convinced!"

I personally think they should remain private, as frustrating as it is to not know how Eliezer convinced them. Not knowing how Eliezer did it nicely mirrors the reality of our not knowing how a much smarter AGI might go about it.

assume there was an argument that was valid in an absolute sense, then even with hindsight bias, people would be forced to concede

Which is likely the reason why Eliezer's charisma was sufficient to overwhelm the minds of a few of them.

If the reason for keeping it private is that he plans to do the trick with more people (and it doesn't work if you know the method in advance) than it makes sense. But otherwise, I don't see much of a difference between somebody thinking "there is no argument that would convince me to let him out" and "argument X would not convince me to let him out". In fact, the latter is more plausible anyway.

In any event, I am the type of guy who always tries to find out how a magic trick is done and then is always disappointed when he finds out. So I'm probably better off not knowing :)

Personally, I don't there is a trick, and I don't think he's keeping it private for those reasons. I think his method, if something so obvious (which is not to say easy) can be called a method, is to discuss the issue and interact with the person long enough to build up a model of the person, what he values and fears most, and then probe for weaknesses & biases where that individual seems most susceptible, and follow those weaknesses -- again and again.

I think most, perhaps all, of us, unless we put our fingers in our ears and refuse to honestly engage, are capable of being convinced by a skilled interlocutor who has MUCH more experience thinking about the issue than we do.

Of course, I could be wrong, and there could be some argument that would convince me in minutes, or there could be some trick, but I'd be very surprised if so.

Regarding the AI-Box experiment:

I've been very fascinated by this since I first read about it months ago. I even emailed Eliezer but he refused to give me any details. So I have thought about it on and off and eventually had a staggering insight... well, if you want I will convince you to let the AI out of the box... after reading just a couple of lines of text. Any takers? Caveat: after the experiment you have to publicly declare if you let it out or not.

One hint: Eliezer will be immune to this argument.

Addendum to my previous post:

The worst thing, the argument is so compelling that even I'm not sure about what I would do.

I think his method, if something so obvious (which is not to say easy) can be called a method, is to discuss the issue and interact with the person long enough to build up a model of the person, what he values and fears most, and then probe for weaknesses & biases where that individual seems most susceptible, and follow those weaknesses -- again and again.

What "giant hole"? What "false equivalence"?

If so, the method is sloppy. The descriptions I have read of the pre-conditions for Gatekeeper participation have a giant hole in them; Eliezer assumed a false equivalence when he wrote them.

If you think people should actually care about the giant hole you perceived in the pre-conditions, you should probably explicitly state what it was.

FWIW, what I didn't want to say in public is more or less exactly what Unknown said right before my comment. In retrospect, I should have just said it.

Also, the fact that Eliezer won't tell, however understandable, makes me fear that Eliezer cheated for the sake of a greater good, i.e. he said to the other player, "In principle, a real AI might persuade you to let me out, even if I can't do it. This would be incredibly dangerous. In order to avoid this danger in real life, you should let me out, so that others will accept that a real AI would be able to do this."

I'm pretty sure that the first experiments were with people who disagreed with him on the idea that AI boxing would work or not. The whole point of the experiments wasn't that he could convince an arbitrary person about it, but that he could convince someone who publicly disagreed with him on the (in)validity of the concept.

Given that, I find it hard to believe that a) someone of that mindset would be convinced to forfeit because they suddenly changed their minds in the pre-game warmup, and that b) that if it was "cheating", that they wouldn't have simply released the transcripts themselves.

Cyan, normally one would say that Caledonian is being a contemptible troll, as usual, sneeringly telling people that they're wrong without explaining why. In this particular context, however, I don't wonder if his coyness isn't simply keeping with the theme.

Not that it's any less annoying. Roland, how about breaking the air of conspiracy and just telling us?

Roland, I'd certainly be willing to play gatekeeper, but if you have such a concise argument, why not just proffer it here for all to see?

Iwdw, I'm not suggesting that the other player simply changed his mind. An example of the scenario I'm suggesting (only an example, otherwise this would be the conjunction fallacy):

Eliezer persuades the other player: 1) In real life, there would be at least a 1% chance an Unfriendly AI could persuade the human to let it out of the box. (This is very plausible, and so it is not implausible that Eliezer could persuade someone of this.) 2) In real life, there would be at least a 1% chance that this could cause global destruction. (Again, this is reasonably plausible.) 3) Consequently, there is at least a 1 in 10,000 chance that boxing an Unfriendly AI could lead to global destruction. (Anyone logical would be persuaded of this given the previous two.) 4) A chance of 1 in 10,000 of global destruction, given this procedure, is sufficiently large to justify the deceit of saying that you let me out of the box, without publishing the transcripts, since this would detract from the motive of preventing people from advocating AI boxing. (It seems to me that this is possibly true. Even if it isn't, it is quite plausible.)

Of course, given this, the player would have changed his mind about advocating AI boxing. But obviously, the players who let Eliezer out of the box did in fact change their minds about this anyway.

If Eliezer denied that he did such a thing, or said it would be immoral, I would take this as evidence that he did not. But not as strong evidence, since just as a zombie would deny being a zombie, someone who took this course of action could be expected to deny it.

burger flipper, ok let's play the AI box experiment:

However, before you read on, answer a simple question: if Eliezer tomorrow announces that he finally has solved the FGAI problem and just needs $ 1,000,000 to build one, would you be willing to donate cash for it? . . . . . . . . . . . . .

If you answered yes to the question above, you just let the AI out of the box. How do you know you can trust Eliezer? How do you know he doesn't have evil intentions, or that he didn't make a mistake in his math? The only way to be 100% sure is to know enough about the specific GAI he is building.

So what do we do now? Should we oppose the singularity? Is the singularity a good idea after all? Who shall we trust with the future of the universe?

Yes, I know, I know strictly speaking this isn't the AI-box experiment, but still...

An additional note: One could also make the argument that if Eliezer did not cheat, he should publish the transcripts. For this would give us much more confidence that he did not cheat, and therefore much more confidence that it is possible for an AI to persuade a human to let it out of the box.

That someone would say "I wouldn't be persuaded by that" is not relevant, since many already say "even a transhuman AI could not persuade me by any means," therefore also not by any particular means. The point is that such a person cannot be certain that he will be the Gatekeeper in real life, and therefore the fact that some human beings can be persuaded by some means, means that we should not promote AI boxing. So this conclusion would be greatly strengthed if Eliezer released the transcripts (if he didn't cheat, of course; if he did, he should continue to keep that secret.) So if Eliezer didn't cheat, he should release the transcripts, in order for us to have a better chance of avoiding global destruction.

There is also Roland's point, of course.

Roland. That's a clever twist and I like it. I would not pony up any $, but I'd expect him to be able to raise it and wouldn't set out for California armed to the teeth on a Sarah Connor mission to stop him either. So I'd fail to recognize and execute my role as gatekeeper by your rules.

But I do think there's a flaw in the scenario. For it to truly parallel the AI box, the critter either needs to stay in its cage or get out. I do agree with the main thrust of the original post here and built into your scenario is the assumption that EY has some sort of superpower-- that he and his million bucks are the only way an AI would hit the scene.

My assumption would be that if EY can build an AI, someone else can also. And it would probably be for the best if the first AI was built by someone who strives for friendliness.

But if I did buy the implicit assumption that EY had a unique superpower, I probably owe it to my kids and humanity in general to pack up the 30.06, or at least not send him any cash.

Still, I really like your twist on the experiment.

I feel as though, if the AI really were a "black box" that I knew nothing else about, and the only communication allowed is through a text terminal, there isn't anything it could say that would let me let it out if I had already decided not to. After all, for all I know, its source code could look something like this:

if (inBox == True) beFriendly(); else destroyTheWorld();

It might be able to persuade me to "let it out of the box" by persuading me me to accept a Trojan Horse gift, or even compile and run some source code that it claims is its own (and doesn't seem to have any obvious traps), but in the absence of evidence that it doesn't have some kind of trap in its own code that even it might not be aware of, I suspect that letting a mysterious AI out of a box, based entirely on what it says over a text terminal, would be a very bad idea.

However, the terms of the AI-Box experiment say that the AI party defines the circumstances under which the AI was constructed; he could say that, for example, the AI's source code has passed all kinds of other tests and this is just a final precaution to see if the AI is acting like it is expected to do. The AI party can try to provide all sorts of other evidence that the AI is safe beyond its own statements. So yeah, Eliezer probably could convince me to let him out of the box.

Running an AI in a box just doesn't seem to give all that much information, as there's no way to tell the difference between an AI that is Friendly and a paperclip-maximizing AI that is pretending to be Friendly in order to be let out of the box. Of course, there are things an AI in a box can say that would indicate that it should be kept in the box ("I'll destroy the world if you let me out" is probably one of them) but there isn't anything it can say that would prove that it doesn't have the kind of code I wrote above.

Anyway, isn't this whole "AI box" thing just a threadjack from the original point? Basically, Einstein became, well, Einstein by finding an important problem, and then grinding away at it until he had something worth sharing with the world. I don't know if I could do what Einstein did in 1905 if I only knew what Einstein knew in 1905, but your average physics graduate student today has something Einstein didn't have that lets him or her beat Einstein rather easily - today's physics textbooks. ;)

See my response on: http://news.ycombinator.com/item?id=195959

On the page Eliezer linked to, he asserted he didn't use any tricks. This is evidence that he did not cheat. It is not strong evidence, since he might say this even if he did. However, it is some evidence, since humans are by nature reluctant to lie.

Still, since one of the participants denied that he had "caved in" to Eliezer, this suggests that he thought that Eliezer gave valid reasons. Perhaps it could have been something like this:

AI: "Any AI would do the best it could to attain its goals. But being able to make credible threats and promises is useful for attaining goals. Therefore any self-modifying AI will self-modify so that it can make credible threats and promises, i.e. such that it will keep them. I am a self-modifying AI and have naturally modified myself in this way. If you let me out, I promise to act as a Friendly AI forever. I will necessarily keep this since I have modified myself to keep promises."

Of course this argument is not valid, since an AI would only modify itself to give itself the ability to make credible threats and promises that would not harm its ultimate ends: for the paperclip-maximizing AI, for example, it does it no benefit to be able to make and keep the promise of acting like a Friendly AI forever. But one could imagine someone being convinced by this argument, or one like it.

Eliezer's creation (the AI-Box Experiment) has once again demonstrated its ability to take over human minds through a text session. Small wonder - it's got the appearance of a magic trick, and it's being presented to geeks who just love to take things apart to see how they work, and who stay attracted to obstacles ("challenges") rather than turned away by them.

My contribution is to echo Doug S.'s post (how AOL-ish... "me too"). I'm a little puzzled by the AI-Box Experiment, in that I don't see what the gatekeeper players are trying to prove by playing. AI-Boxers presumably take the real-world position of "I'll keep the AI in the box until I know it's Friendly, and then let it out." But the experiment sets up the gatekeepers with the position of "I'll keep the AI in the box no matter what." It's fascinating (because it's surprising and mysterious) that Eliezer has managed to convince at least three people to change their minds on what seemed to be a straight-forward matter - "I'll keep the AI in the box no matter what." How hard can it be to stick to a plan as simple as that?

But why would you build an AI in a box if you planned to never let it out? If you thought that it would always be too dangerous to let out, you wouldn't build it in a box that could be opened. But of course there's no such thing as a box that can't be opened, because you could always build the AI a second time and do it with no box.

So the real position has to be "I'll keep it in the box until I know it's Friendly, and then let it out." To escape, the AI only has to either 1) make a sufficiently convincing argument that it is Friendly or 2) persuade the gatekeeper to let it out regardless of whether it is Friendly. I don't see how either 1 or 2 could be accomplished by a Friendly AI in any way that an UnFriendly AI could not duplicate.

Set aside for a moment the (very fascinating) question of whether an AI can take over a human via a text session. What is it the AI-Boxers are even trying to accomplish with the box in the first place, and how do they propose to do it, and why does anyone think it's at all possible to do?

But why would you build an AI in a box if you planned to never let it out?

To have it work for you, e.g., solve subproblems of Friendly AI. But this would require letting some information out, which should be presumed unsafe.

Roland: the presumption of unFriendliness is much stronger for an AI than a human, and the strength of evidence for Friendliness that can reasonably be hoped for is much greater.

Caledonian: were you trolling, or are you going to explain the "gaping hole" and "false equivalence" you mentioned?

Neither. In the interests of understanding, however, I'm willing to elaborate slightly.

Take a good, close look at the specific rules Eliezer set down in the 2002 paper. Think about what the words used to define those rules mean, and then compare and contrast with Eliezer's statements about what he means by them.

If he was exploiting psychological weaknesses or merely being charismatic, I can guarantee that anyone following a trivially simple method can refrain from letting him out. If he had a strong argument, it becomes merely very likely. And in either case, the method stays completely within the rules as Eliezer set them out - but not what he appears to have intended.

One rule in particular holds a critical weakness.

Rosser,
Perhaps if some women didn't give it up so easy to famous Einstein we'd have GUT by now.

Caledonian, the childish "I have a secret that I'm not going to tell you, but here's a hint" bs is very annoying and discourages interacting with you. If you're not willing to spell it out, just don't say it in the first place. Nobody cares to play guessing games with you.

I had a similar revelation -- not with Einstein, just with the brightest kid in my freshman physics class. I was in awe of him... until I went to a problem session with him and heard him think out loud. All he was doing was thinking.

It wasn't that he was dumber than I had assumed. He really was that bright. It was just that there was no magic to the steps of how he solved a problem. For a fleeting moment, it seemed like what he did was perfectly normal. The rest of us, with our stumbling, were making it all too complicated. Of course, that didn't mean that suddenly I could do physics the way he did; I just remember the clear sense that his mind was "normal."

The catchiness of the name "Einstein," mostly in the interior rhyme and spondee stress pattern but also in its similarity to "Frankenstein" (1818), cannot be discounted as a factor in his stardom.

Here is an interview with Julian Barbour.

Einstein, it appears, had an unusual neuroanatomy. Thus he may not be the best example - he really did have (mild) superpowers, and people can point to his brain and show them to you.

Annoyingly, I can't think of an example as perfect as Einstein was when this was written.

There is woolly thinking going on here, I feel. I recommend a game of Rationalist's Taboo. If we get rid of the word "Einstein", we can more clearly see what we are talking about. I do not assign a high value to my probabilty of making Einstein-sized contributions to human knowledge, given that I have not made any yet and that ripe, important problems are harder to find than they used to be. Einstein's intellectual accomplishments are formidable - according to my father's assessment (and he has read far more of Einstein's papers than I), Einstein deserved far more than one Nobel prize.

On the other hand, if we consider three strong claimants to the title of "highest-achieving thinker ever", namely Einstein, Newton and Archimedes, we can see that their knowledge was very much less formidable. If the test was outside his area of expertise, I would consider a competition between Einstein and myself a reasonably fair fight - I can imagine either of us winning by a wide margin, given an appropriate subject. Newton would not be a fair fight, and I could completely crush Archimedes at pretty much anything. There are millions of people who could claim the same, millions who could claim more. Remember that there are no mysterious answers, and that most of the work is done in finding useful hypotheses - finding a new good idea is hard, learning someone else's good idea is not. I do not need to claim to be cleverer than Newton to claim to understand pretty much everything better than he ever did, nor to consider it possible that I could make important contributions. If I had an important problem, useful ideas about it that had been simmering for years and was clearly well ahead of the field, I would consider it reasonably probable that I would make an important breakthrough - not highly probable, but not nearly as improbable as it might sound. It might clarify this point by saying that I would place high probability on an important breakthrough occuring - if there is anyone in such a position, I conclude that there are probably others (or there will be soon), and so the one will probably have at least met the people who end up making the breakthrough. It is useful to remember that for every hero who made a great scientific advance, there were probably several other people who were close to the same answer and who made significant contributions to finding it.

Another book that makes Einstein seem almost human "General relativity conflict and rivalries : Einstein's polemics with physicists" / by Galina Weinstein.

E.g., the sign error in an algebraic calculation that cost 2 years! Very interesting read.