A chatbot probably did not pass the Turing Test in Artificial Intelligence


Recent headlines have been awash with stories of a computer program ‘disguised’ as a 13 year old Ukrainian boy Eugene Goostman has passed the Turing Test. Similar claims have been made in the past (this paper was shared with me slightly tongue in cheek by Professor Joanna Bryson), indeed a couple of years ago it was claimed that Eugene came ‘close to passing the Turing Test’ when ‘he/it’ was the overall winner in a similar tournament. We will likely here of more programs that have passed the Turing Test in the near future – how close these claims are to the real ‘spirit’ of the Turing Test is certainly very debatable.

Author and Roboticist, Professor Alan Winfield told me he was less than convinced last Saturday’s announcement resulted in the Turing Test being passed, especially as Turing “intended higher thresholds, >5mins, and >30%.” The pass rate claimed in the University of Reading announcement was 33%, just a very narrow pass if Turing really was stuck on the greater than 30% of judges threshold, which is highly unlikely.

In fact Turing does specify 5 minutes but I am in agreement with Alan, he does not indicate how many judges or what percentage. What he says is:

“An average interrogator will not have more than 70 per cent chance of making the right identification after five minutes of questioning.”

Is this the same as the University of Reading and the organizers claim?  “If a computer is mistaken for a human more than 30% of the time during a series of five minute keyboard conversations it passes the test.”

So far very little has been shared about the organization of the event, other than the claim that 30 judges took part and ‘in each five minutes a judge was communicating with both a human and a machine,’ and 33% of judges were convinced that Eugene was human.

At no time does Turing specify in his paper 30 per cent of judges, and I would posit that he would expect the 70 per cent to be nearer the threshold.

To help analyze the number of judges, Turing states:

“A number of interrogators could be used, and statistics compiled to show how often the right identification was given.”

Turing originally set out that the Imitation game should also be based on gender.

The new form of the problem can be described in terms of a game which we call the ‘imitation game.” It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart from the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either “X is A and Y is B” or “X is B and Y is A.”

“What will happen when a machine takes the part of A in this game?”  Adding later…  “Could the interrogator distinguish between them?”

It is not clear if the judges in the University of Reading organized event focused on gender, maybe they will clarify this when the ‘peer reviewed’ papers are released which they have referenced.

I am not convinced the Turing Test has been passed although it is fair to say that in recent years we have seen considerable progress towards the goal of a machine that can ‘trick’ judges into thinking it is human or more precisely a woman.

Murray Shanahan, a professor of cognitive robotics at Imperial College London, who is quoted in the Daily Telegraph as saying: “I think the claim is completely misplaced, and it devalues real AI research. It makes it seem like science fiction AI is nearly here, when in fact it’s not and it’s incredibly difficult.” Or as Alan Turing himself concluded his famous paper: “We can only see a short distance ahead, but we can see plenty there that needs to be done.”

One thought on “A chatbot probably did not pass the Turing Test in Artificial Intelligence

  1. Hi,

    While there is a real possibility that the Turing test was not actually cleared by the computer, another important question I would love to have answered is the about the difference in previous similar attempts and this.

    Even if this one did not, the fact is that IBM Watson already answered questions and won Jeopardy in 2011.

    Either ways, I would think the Turing Test milestone may not be too far away.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s