CoChat
Sign Up
PreprintCited by 223Artificial Intelligence ApplicationsCybernetics and Technology in SocietyAI-based Problem Solving and Planning

Lessons from a Restricted Turing Test

Open access
Shieber, Stuart M.·1994·arXiv
DOI 10.48550/arxiv.cmp-lg/9404002arXiv cmp-lg/9404002
Open this paper in CoChat — free accountOpen PDF

Abstract

We report on the recent Loebner prize competition inspired by Turing's test of intelligent behavior. The presentation covers the structure of the competition and the outcome of its first instantiation in an actual event, and an analysis of the purpose, design, and appropriateness of such a competition. We argue that the competition has no clear purpose, that its design prevents any useful outcome, and that such a competition is inappropriate given the current level of technology. We then speculate as to suitable alternatives to the Loebner prize.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.cmp-lg/9404002

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Loebner Prize ran from 1990 to 2020 without deeming a single system to have passed.

    “The Loebner Prize [47]-an annual competition in which entrant systems tried to fool a panel of expert judges-ran from 1990 to 2020 without deeming a single system to have passed.”

    From People cannot distinguish GPT-4 from a human in a Turing test (2025) · cites this paper as [47]

    In “T,” the paper dates the first Loebner Prize tournament to November 8, 1991, contrary to the claim that the competition ran from 1990. It also says the judges were selected from the general public for having “no special expertise in computer science,” not that they were expert judges. The paper does not address whether any system had passed by 2020. (LLM, conf=0.85)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • The two-party formulation of the Turing test eliminates the confound of the third player's humanlikeness and is easier to implement.

    “While this differs from Turing's original threeparty formulation, it has become a standard operationalisation of the test because it eliminates the confound of the third player's humanlikeness and is easier to implement [13,45,47].”

    From People cannot distinguish GPT-4 from a human in a Turing test (2025) · cites this paper as [47]

    The available passage in “Lessons from a Restricted Turing Test” (section T; page unavailable) describes a judge distinguishing a human from a computer and discusses the confederates needed for a tournament. Searches for a two-party formulation, the third player’s humanlikeness, and ease of implementation found no passage addressing the claimed reasons for adopting that formulation. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 21 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Shieber, S. M.
Affiliations
Harvard University (US)
Corresponding author
Shieber, Stuart M.
ORCID
Shieber, Stuart M.
Venue
arXiv
Publisher
arXiv
Published
1994
Volume
37
Issue
6
Pages
70-78
ISSN
0001-0782 (ISSN-L)
DOI
10.48550/arxiv.cmp-lg/9404002
arXiv
cmp-lg/9404002
OpenAlex
W2003453784
Cited by
222
References
21
Citations by year
2026: 2 · 2025: 21 · 2024: 7 · 2023: 5 · 2022: 5 · 2021: 20 · 2020: 5 · 2019: 7 · 2018: 9 · 2017: 1 · 2016: 13 · 2015: 3 · 2014: 6 · 2012: 5
FWCI
4.63
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/assumed-1991-2003/
Locations
  • Communications of the ACM (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Cybernetics and Technology in Society
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
Turing test, machine intelligence, AI evaluation, competition design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review