Come up with better Turing Tests

This is not that much better. The article that The Most Human Human is based on talks about the difficulty of communication in a 5-minute window, and the lack of knowledge lay judges have about what AI involves. The author consistently got named a human by better-applying the tactics of the some of the most successful bots: controlling conversation flow and using humor.

It's an improvement, but a winner would still win by "gaming" judges' psychology.

The test is a response to the Problem Of Other Minds.

Simply, no other test will be accepted by people that [insert something non human here] is genuinely intelligent.

The reasoning goes: strictly speaking the problem of other minds applies to other humans as well but we politely assume that the humans we're talking to are genuinely intelligent or at least conscious on little more than the basis that we're talking to them and they're talking back like conscious human beings.

the longer and more involved the test the harder it is to use tricks to fake genuine intelligence.

Is the Turing Test really all that useful or important?

It did seem like a useful tool for measuring (some types of) intelligence. Since it doesn't work, it would be useful to have a substitute...

Hmm ... that and a la shminux's xkcd link gives me an idea for a test protocol: instead of having the judges interrogate subjects, the judges give each pair of subjects a discussion topic a la Omegle's "spy" mode:

Spy mode gives you and a stranger a random question to discuss. The question is submitted by a third stranger who can watch the conversation, but can't join in.

...and the subjects have a set period of time they are permitted to talk about it. At the end of that time, the judge rates the interesting-ness of each subject's contribution, and each subject rates their partner. The ratings of confirmed-human subjects would be a basis for evaluating the judges, I presume (although you would probably want a trusted panel of experts to confirm this by inspection of live results), and any subjects who get high ratings out of the unconfirmed pool would be selected for further consideration.

For the same reason that the test shouldn't try to simulate a human with poor English skills, it also shouldn't try to simulate a human who isn't willing to cooperate with a questioner. A random human off the street wouldn't answer the millipede question, but a random human recruited to take part in an experiment and told to answer reasonable questions probably would.

Maybe just do some roleplaying, with the judge as the DM.

Nope; general game-playing is a well-studied area of AI; the AI's aren't great at it, but if you aren't playing them for a long time they can certainly pass as a bad human. Zendo-like "analogy-finding" has also been studied.

By only demanding very structured action types, instead of a more free-flowing, natural-language based interaction, you are handicapping yourself as a judge immensely.

EDIT: 6. is bad. It would exclude equally many actual AI and blind people as well.

So we wouldn't use blind people in the human control group. We'd want to get rid of any disabilities that the AI could use as an excuse (like the whole 13 year-old foreign boy).

As for excluding AIs... the Turing test was conceived as a sufficient, but not necessary, measure of intelligence. If AI passes, then intelligent, not the converse (which is much harder).

Physics problems are an interesting test-- you could check for typical human mistakes.

You could throw in an unsolvable problem and see whether you get plausibly human reactions.

Speaking of original Turing Test, the Wikipedia page has an interesting discussion of the tests proposed in Turing's original paper. One of the possible reads of that paper includes another possible variation on the test: play Turing's male-female imitation game, but with the female player replaced by a computer. (If this were the proposed test, I believe many human players would want a bit of advance notice to research makeup techniques, of course.) (Also, I'd want to have 'all' four conditions represented: male & female human players, male human & computer, computer & female human, and computer & computer.)

We want a test to tell us when an AI is intelligent and powerful. But we'll know a powerful AI even without a test, because we'll see it using its power and achieving things that really matter [...] narrow AIs [can be made that are] useless for anything other than passing the test. A new test might repeat the same story. Or it might turn out to be too hard and only be achieved long after many other AI capabilities that would greatly change the world.

I think we'll see (arguably have already seen) AI changing the world before we see a general AI passing the Turing test. But I don't think that makes the Turing test useless, or a red herring.

Narrow AI is plenty powerful. It drives cars, flies military drones, runs short-term trading systems, and plays chess, and does (or will shortly do) them all better than the best humans in their domains. Right now that hasn't dramatically changed the world, but I don't think it's too much of a stretch to imagine a world that has been transformed by narrow AI applications.

But there are still things the Turing test or a successor would be useful for. For one thing, as AI techniques advance, I expect the line between narrow and general AI to blur. I can't rule out purpose-built AGI before this becomes significant, but if that doesn't make the problem completely irrelevant, then the Turing test serves as a pretty good marker of generalizability: if your trading system (that scrapes Reuters for headlines and does some sophisticated NLP and concept-mapping stuff that you're pretty proud of) starts asking you hilariously bizarre questions about business ethics, you're probably well on your way to dealing with something that can no longer be described as narrow AI. If it starts asking you good questions about business ethics... well, you're probably very lucky.

Less significantly from an AGI perspective, but still interestingly, there's a bunch of semi-narrow AI applications that focus tightly on interaction with humans. Siri, Google Now, and Cortana are probably the most salient examples right now, along with all those godawful customer-service phone systems; we could also imagine things like automated hotel concierges or caretakers for the elderly. The Turing test is an excellent benchmark for their performance; I no longer think we can take a pass as evidence of strong general intelligence, but humanlike responses are so useful in these roles that I still think it's a good thing to shoot for. A successor test in this role gives us a less gameable objective.

The "nonspecific praise" approach fooled me the first time I saw it, but it gets pretty obvious after a while.

Ditto the "extract declarative statements, act incredulous" approach.

Certainly a few commenters there easily pass for computers.

Why not? (1) You are not required to respond to other people's comments. (2) People generally don't suspect you of being a chatbot on a blog, so they will not test you explicitly.

So the chatbot could be designed to play safe, and reply only in situations it believes it understands.

Point 3, "properly motivated".

No. It's just that its something a chatterbot is spectacularly ill-equipped to respond to, unless they've been specifically programmed for these sort of things. It's a meta-instruction, using the properties of the test that are not derived from vocabulary parsing.

The manner in which they fail or succeed is relevant. When I ran Stuart_Armstrong's sentence on this Web version of ELIZA, for example, it failed by immediately replying:

Perhaps you would like to be human, simply do nothing for 4 minutes, then re-type this sentence you've just written here, skipping one word out of 2?

That said, I agree that passing the test is not much of a feat.

Though we may have to consider the hideous but unlikely possibility... that it might... lie.

Suggestion: ask questions which are easy to execute for persons with evolved physical-world intuitions, but hard[er] to calculate otherwise. For example:

Suppose I have a yardstick which was blank on one side and marked in inches on the other. First, I take an unopened 12-oz beverage can and lay it lengthwise on one end of the yardstick so that half the height of the can is touching the yardstick and half is not, and duct-tape it to the yardstick in that position. Second, I take one-liter plastic water bottle, filled with water, and duct-tape it to the other end in a similar sort of position. If I lay a deck of playing cards in the middle of the open floor and place the yardstick so that the 18-inch mark is centered on top of the deck of cards, when I let go, what will happen?

Familiarity with imperial units is hardly something I would call an evolved physical-world intuition...

Your test seems overly complicated; what about simple estimates? Like "how long would it take to fly from Paris, France, to Paris, USA" or similar? Add in some Fermi estimates, get them to show your work, etc...

By the way, as a human being, I'm pretty sure that I would react to your lazy test with eloquent, discursive indignation while you sat back and watched

If the human subject is properly motivated to want to appear human, they'd relax and follow the instructions. Indignation is another arena in which non-comprehending programs can hide their lack of comprehension.