Loebner Prize

From Wikipedia, the free encyclopedia

Jump to: navigation, search

The Loebner Prize is an annual competition in artificial intelligence that awards prizes to the chatterbot considered by the judges to be the most human-like. The format of the competition is that of a standard Turing test. A human judge poses text questions to a computer program and a human being via computer. Based upon the answers, the judge must decide which is which. In 2008 a variety of judges, including experts and non-experts, adults and children, native and non-native English speakers participated in the University of Reading hosted contest.[1]

The contest was begun in 1990 by Hugh Loebner in conjunction with the Cambridge Center for Behavioral Studies, Massachusetts, United States. It has since been associated with Flinders University, Dartmouth College, the Science Museum in London, and most recently the University of Reading. In 2004 and 2005, it was held in Loebner's apartment in New York City.

Within the field of artificial intelligence, the Loebner Prize is somewhat controversial; the most prominent critic, Marvin Minsky, has called it a publicity stunt that does not help the field along.[2]

In addition, the time limit of 5 minutes and the use of untrained/unsophisticated interrogator/judges has resulted in some wins that may be due to trickery rather than to plausible intelligence, as one can judge from transcripts of winning conversations (linked below).[citation needed]


[edit] Prizes

Originally, $2,000 was awarded for the most human-seeming chatterbot in the competition. The prize was $3,000 in 2005 and $2,250 in 2006. In 2008, $3,000 was awarded.

In addition, there are two one-time-only prizes that have never been awarded. $25,000 is offered for the first chatterbot that judges cannot distinguish from a real human and that can convince judges that the human is the computer program. $100,000 is the reward for the first chatterbot that judges cannot distinguish from a real human in a Turing test that includes deciphering and understanding text, visual, and auditory input. Once this is achieved, the annual competition will end.

[edit] Competition rules and restrictions

The rules have varied over the years and early competitions featured restricted conversation Turing tests but since 1995 the discussion has been unrestricted.[3]

For the three entries in 2007, Robert Medeksza, Noah Duncan and Rollo Carpenter,[4] some basic "screening questions" were used by the sponsor to evaluate the state of the technology. These included simple questions about the time, what round of the contest it is, etc.; general knowledge ("What is a hammer for?"); comparisons ("Which is faster, a train or a plane?"); and questions demonstrating memory for preceding parts of the same conversation. "All nouns, adjectives and verbs will come from a dictionary suitable for children or adolescents under the age of 12." Entries did not need to respond "intelligently" to the questions to be accepted.

For the first time in 2008 the sponsor allowed introduction of a preliminary phase to the contest opening up the competition to previously disallowed web-based entries judged by a variety of invited interrogators. The available rules do not state how interrogators are selected or instructed. Interrogators (who judge the systems) have limited time: 5 minutes per entity in the 2003 prize; 20+ per pair in 2004-2007 prizes, and 5 minutes to conduct simultaneous conversations with a human and the program in 2008.

[edit] Contests

[edit] 2006

In 2006, the contest was organised by Tim Child (CEO of Televirtual) and Huma Shah.[5][6] On August 30, the four finalists were announced:

  • Rollo Carpenter
  • Richard Churchill and Marie-Claire Jenkins
  • Noah Duncan
  • Robert Medeksza

The contest was held on 17 September in the VR theatre, Torrington Place campus of University College London. The judges included the University of Reading's cybernetics professor, Kevin Warwick, a professor of artificial intelligence, John Barnden (specialist in metaphor research at the University of Birmingham), a barrister, Victoria Butler-Cole and a journalist, Graham Duncan-Rowe. The latter's experience of the event can be found in an article in Technology Review.[7][8] The winner was 'Joan', based on Jabberwacky, both created by Rollo Carpenter.

[edit] 2007

The 2007 competition was held on 21 October in New York City. The judges were: Computer Science Professor Russ Abbott, Philosophy Professor Hartry Field, Psychology Assistant Professor Clayton Curtis and English lecturer Scott Hutchins.[9]

No bot passed the Turing Test, but the judges ranked the three contestants as follows:

  • 1st: Robert Medeksza from Zabaware, creator of Ultra Hal Assistant
  • 2nd: Noah Duncan, a private entry, creator of Cletus
  • 3rd: Rollo Carpenter from Icogno, creator of Jabberwacky

The winner received $2,250 and the annual medal. The runners-up received $250 each.

[edit] 2008

The 2008 competition was organised by Professor Kevin Warwick, coordinated by Huma Shah and held on 12 October at the University of Reading, UK.[10] After testing by over one hundred judges during the preliminary phase, in June and July 2008, six finalists were selected from thirteen original entrants - artificial conversational entity (ACE). Five of those invited competed in the finals:

  • Brother Jerome, Peter Cole and Benji Adams
  • Elbot, Fred Roberts / Artificial Solutions
  • Eugene Goostman, Vladimir Veselov, Eugene Demchenko and Sergey Ulasen
  • Jabberwacky, Rollo Carpenter
  • Ultra Hal, Robert Medeksza

In the finals, each of the judges was given five minutes to conduct simultaneous, split-screen conversations with two hidden entities. Elbot[11] of Artificial Solutions[12] won the 2008 Loebner Prize bronze award, for most human-like artificial conversational entity, through fooling three of the twelve judges who interrogated it (in the human-parallel comparisons) into believing it was human. This is coming very close to the 30% traditionally required to consider that a program has actually passed the Turing test. Eugene Goostman[13] and Ultra Hal[14] both deceived one judge each that it was the human.

Will Pavia, a journalist for The Times, has written about his experience; a Loebner finals' judge, he was deceived by Elbot and Eugene [15]. Huma Shah, the coordinator of the 2008 event has blogged about various misunderstandings of the 2008 Loebner Prize.[16]

[edit] Winners

Year Winner Program
1991 Joseph Weintraub PC Therapist
1992 Joseph Weintraub PC Therapist
1993 Joseph Weintraub PC Therapist
1994 Thomas Whalen TIPS
1995 Joseph Weintraub PC Therapist
1996 Jason Hutchens HeX
1997 David Levy Converse
1998 Robby Garner Albert One
1999 Robby Garner Albert One
2000 Richard Wallace Artificial Linguistic Internet Computer Entity (A.L.I.C.E.)
2001 Richard Wallace Artificial Linguistic Internet Computer Entity (A.L.I.C.E.)
2002 Kevin Copple Ella
2003 Juergen Pirner Jabberwock
2004 Richard Wallace Artificial Linguistic Internet Computer Entity (A.L.I.C.E.)
2005 Rollo Carpenter George
2006 Rollo Carpenter Joan
2007 Robert Medeksza Ultra Hal
2008 Fred Roberts Elbot

[edit] References

[edit] See also

[edit] External links

Personal tools