[News] Turing Test passed

Scott Aaronson has posted a transcript of his "conversation" with Eugene Goostman.

I am a bit sceptical about whether or not it actually passed the Turing test. To me it looks more like a publicity stunts for the following reasons:

1) 5 minutes is a short period of time.

2) I don't believe Turing mentioned anything about 30% . I might be wrong on this one.

3) I don't know if the judges were properly trained. What questions did they ask? I feel like there must be plenty of questions related to IQ and creativity that a thirteen year old could answer with ease but that Eugene Goostman would struggle with. Examples: "Cow is to bull like, bitch is to ....?", or "Once upon a time there lived a pink unicorn in a big mushroom house with three invisible potatoes. Could you finish the story for me in a creative way and explain why the unicorn ended up painting the potatoes pink?" . The idea with the Turing test is that the computer should be indistinguishable from a human (in this case a 13 year old non native english speaker). I don't believe this criteria has been met until I see a chat transcript with reasonably hard questions.

4) Having the bot pose as a none native English speaking 13 year old might not be a violation of the rules, but I very much feel like it goes against the spirit of the Turing test. It reminds me a bit of this comic (http://existentialcomics.com/comic/15). But this is beside the point, I don't even think the bot would pass the Ukrainian-13-year-old-boy-turing-test if it was asked reasonably hard questions.

Until I learn more about the proceedings I remain utterly unconvinced that this is the milestone in AI media portrait it to be. It is nonetheless pretty cool!

This is courtesy Kevin Warwick at the University of Reading, who is good at making media claims along these lines, and has done for several years. I advise disregarding anything containing his name.

I don't know why they're calling this the "first time"

In 1972 bots were able to convince trained professionals that they were human schizophrenics:

Kenneth Colby created PARRY in 1972, a program described as "ELIZA with attitude".[28] It attempted to model the behaviour of a paranoid schizophrenic, using a similar (if more advanced) approach to that employed by Weizenbaum. In order to validate the work, PARRY was tested in the early 1970s using a variation of the Turing Test. A group of experienced psychiatrists analysed a combination of real patients and computers running PARRY through teleprinters. Another group of 33 psychiatrists were shown transcripts of the conversations. The two groups were then asked to identify which of the "patients" were human and which were computer programs.[29] The psychiatrists were able to make the correct identification only 48 percent of the time — a figure consistent with random guessing.[30]

foreign 13 year old who isn't being challenged is a low bar to pass.

a bot which posts bellow youtube videos and does nothing but spew racial abuse and “lol” would be indistinguishable from the 13 year old humans doing the same thing so would technically pass the turing test.

I’ll be much more interested when it can convince a group of professionals that it’s another professional in their field, much more useful.

Let's discuss a new type of Reverse Turing Test.

This simply consists of coming up with a general class of question that you think would reliably distinguish between a chatbot and a human within about 5 minutes of conversation, and explaining which feature of "intelligence" this class of question probes.

If you're not able to formulate the broad requirements for such a class of question, you have no business being the judge in a Turing Test. You're only playing the chatbot as you would play a video game.

One of my candidates for questions of this kind: ask the interviewee to explain a common error of reasoning that people make, or can make. For instance: "If you look at the numbers, there's quite a correlation between sales of ice cream in coastal locations and number of drownings. Some people might be tempted to conclude that ice cream causes people to drown. Do you think that's right, and if not, why not?"

For another example, Dennett discusses having the chatbot explain a joke.

ETA: Scott Aaronson passes with flying colors. Chatbots are likely to lack basic encyclopedic knowledge about the world which every human possesses. (To some extent things like the Wolfram platform could overcome this for precise questions such as Scott's first - but that still leaves variants like "what's more dangerous, a tiger or an edible plant" that are vague enough that quantitative answers probably won't be accessible to a chatbot.)

Blatant bullshit. Nothing even close to the Turing Test was passed. Too much charity toward a bullshit publicity stunt.

This is bunk: see here. Working in an AI lab, everyone I know who has heard about this release is either extremely annoyed or actively scornful of it (or both). I would like to humbly propose that we talk about anything other than this.

And in other news, a 13-year-old boy has convinced a panel of judges that he is a human being!

I've read one transcript of a judge conversation. What I find striking is that the judge seems to be doing their best to be fooled! Of course, no one wants to get a 13 year old upset.

In a Turing Test situation I'd start by trying a bunch of Winograd Schemas.

As others noted, the bot only succeeded in passing the test because the judges themselves failed it.

On that note, why don't we have a thread where people try to unmask Eugene's true nature with a single question (replies indicating server downtime do not count).

Try your best here: http://default-environment-sdqm3mrmp4.elasticbeanstalk.com/

Just proof that the Turing Test is not what Turing imagined it would be. It's more an application of exploiting vulnerabilities in judges than in genuinely advancing AI.

The question then becomes: how can a harder variant of the Turing Test be created that would stay true to the spirit of the original, yet motivate high-quality, generally-applicable research?

Absolute bullshit; it's shameful that FHI went anywhere near this. It's not even techically true since the Turing Test as originally specified includes three participants: judge, human, and machine.

Of course the "news" is bunk, but I don't see that this LW post deserves all the downvotes it's evidently received. Halo/horns effect in action?

(I would be more certain that the post doesn't deserve the downvotes if Suart had put quotation marks around "passed" in its title.)

Somehow this emotion-sensing mood-improving robot seems like more of an achievement: http://edition.cnn.com/2014/06/06/tech/innovation/pepper-robot-emotions/

Side note for those who might not have come across him before: Kevin Warwick is a Professor of Cybernetics who, among other things, has communicated with his wife through electrical cables attached directly to nerves in their forearms.

Eugene has actually been around for many years, since 2008 (http://en.wikipedia.org/wiki/Loebner_Prize), and at that time he convinced one of 12 judges in a parallel test. One of the judges found him more human than an actual human conversation partner.

People keep saying the test is bad, but I feel the standards are very high already. You have to fool a human judge who is on the lookout for a bot. Based on the news articles, it's not clear if Eugene competed against a human partner in side-by-side conversations, but since they're so insistent about the "true" Turing test I'd guess he did. The ability to fool an unsuspecting judge has been around since AIM bots.

Now, I don't want to say I told you so...