The DOI matches its DataCite record; 19 of the 21 references it cites were found in a registry or an index.
Lessons from a Restricted Turing Test
Open accessAbstract
We report on the recent Loebner prize competition inspired by Turing's test of intelligent behavior. The presentation covers the structure of the competition and the outcome of its first instantiation in an actual event, and an analysis of the purpose, design, and appropriateness of such a competition. We argue that the competition has no clear purpose, that its design prevents any useful outcome, and that such a competition is inappropriate given the current level of technology. We then speculate as to suitable alternatives to the Loebner prize.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The Loebner Prize [47]-an annual competition in which entrant systems tried to fool a panel of expert judges-ran from 1990 to 2020 without deeming a single system to have passed.”
From People cannot distinguish GPT-4 from a human in a Turing test (2025) · cites this paper as [47]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“While this differs from Turing's original threeparty formulation, it has become a standard operationalisation of the test because it eliminates the confound of the third player's humanlikeness and is easier to implement [13,45,47].”
From People cannot distinguish GPT-4 from a human in a Turing test (2025) · cites this paper as [47]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Searle, J. R. (1980) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00005756 · Cited by 6,954
Computing Machinery and Intelligence (1950)
Turing, A. M. (2004) · doi:10.1093/oso/9780198250791.003.0017 · Cited by 6,120
What computers still can't do: a critique of artificial reason
Penny, S., & Dreyfus, H. (1994) · Leonardo · doi:10.2307/1575958 · Cited by 1,637
Computer power and human reason
Kuipers, B. J., McCarthy, J., & Weizenbaum, J. (1976) · ACM SIGART Bulletin · doi:10.1145/1045264.1045265 · Cited by 1,056
Colby, K. M. (1981) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00000030 · Cited by 966
Computer Power and Human Reason: From Judgement to Calculation.
Crowther, J. F., & Weizenbaum, J. (1977) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2062802 · Cited by 726
Block, N. (1981) · The Philosophical Review · doi:10.2307/2184371 · Cited by 562
Computing Machinery and Intelligence
Turing, A. M. (2004) · The MIT Press eBooks · doi:10.7551/mitpress/6928.003.0012 · Cited by 273
Alchemy and Artificial Intelligence
Dreyfus, H. L. (1965) · Cited by 215
Subcognition and the Limits of the TuringTest
French, R. M. (1990) · Mind · doi:10.1093/mind/xcix.393.53 · Cited by 198
Colby, K. M., Hilf, F. D., Weber, S., & KRAEMER, H. C. (1972) · Artificial Intelligence · doi:10.1016/0004-3702(72)90049-5 · Cited by 155
(1986) · Telematics and Informatics · doi:10.1016/s0736-5853(86)80080-6 · Cited by 139
Meltzer, B. D. (1977) · Cited by 122
An analysis of the turing test
Moor, J. H. (1976) · Philosophical Studies · doi:10.1007/bf00372497 · Cited by 115
Gustafson, D. (1986) · Teaching Philosophy · doi:10.5840/teachphil19869468 · Cited by 99
The Computer Model of the Mind
Block, N. J. (1993) · The MIT Press eBooks · doi:10.7551/mitpress/5782.003.0046 · Cited by 67
The Quest for the Thinking Computer
Epstein, R. (2007) · Springer eBooks · doi:10.1007/978-1-4020-6710-5_1 · Cited by 63
Why machines can't think: A reply to James Moor
Stalker, D. F. (1978) · Philosophical Studies · doi:10.1007/bf00372897 · Cited by 18
Moor, J. H. (1978) · Philosophical Studies · doi:10.1007/bf00372899 · Cited by 11
Cited by
Sorted by citations
Levesque, H. J., Davis, E., & Morgenstern, L. (2011) · Cited by 879
Universal intelligence: a definition of machine intelligence
Legg, S., & Hutter, M. (2007) · Minds and Machines · doi:10.1007/s11023-007-9079-x · Cited by 803
From Eliza to XiaoIce: Challenges and opportunities with social chatbots
Shum, H.-Y., He, X.-D., & Li, D. (2018) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1700826 · Cited by 749
“In the past, the Turing test and its extensions have been used to evaluate chitchat performance (Shieber, 1994).”
BackgroundMethodInfluentialHaidt, J., & Joseph, C. (2008) · The Innate Mind, Volume 3 · doi:10.1093/acprof:oso/9780195332834.003.0019 · Cited by 650
Pinar Saygin, A., Cicekli, I., & Akman, V. (2000) · Minds and Machines · doi:10.1023/a:1011288000451 · Cited by 383
Metadata
- Authors
- Shieber, S. M.
- Affiliations
- Harvard University (US)
- Corresponding author
- Shieber, Stuart M.
- ORCID
- Shieber, Stuart M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 1994
- Volume
- 37
- Issue
- 6
- Pages
- 70-78
- ISSN
- 0001-0782 (ISSN-L)
- arXiv
- cmp-lg/9404002
- OpenAlex
- W2003453784
- Cited by
- 222
- References
- 21
- Citations by year
- 2026: 2 · 2025: 21 · 2024: 7 · 2023: 5 · 2022: 5 · 2021: 20 · 2020: 5 · 2019: 7 · 2018: 9 · 2017: 1 · 2016: 13 · 2015: 3 · 2014: 6 · 2012: 5
- FWCI
- 4.63
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Communications of the ACM (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Cybernetics and Technology in Society
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- Turing test, machine intelligence, AI evaluation, competition design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
