The DOI matches its DataCite record; 19 of the 21 references it cites were found in a registry or an index.
Lessons from a Restricted Turing Test
Open accessAbstract
We report on the recent Loebner prize competition inspired by Turing's test of intelligent behavior. The presentation covers the structure of the competition and the outcome of its first instantiation in an actual event, and an analysis of the purpose, design, and appropriateness of such a competition. We argue that the competition has no clear purpose, that its design prevents any useful outcome, and that such a competition is inappropriate given the current level of technology. We then speculate as to suitable alternatives to the Loebner prize.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The Loebner Prize [47]-an annual competition in which entrant systems tried to fool a panel of expert judges-ran from 1990 to 2020 without deeming a single system to have passed.”
From People cannot distinguish GPT-4 from a human in a Turing test (2025) · cites this paper as [47]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“While this differs from Turing's original threeparty formulation, it has become a standard operationalisation of the test because it eliminates the confound of the third player's humanlikeness and is easier to implement [13,45,47].”
From People cannot distinguish GPT-4 from a human in a Turing test (2025) · cites this paper as [47]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Searle, J. R. (1980) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00005756 · Cited by 6,954
Computing Machinery and Intelligence (1950)
Turing, A. M. (2004) · doi:10.1093/oso/9780198250791.003.0017 · Cited by 6,120
What computers still can't do: a critique of artificial reason
Penny, S., & Dreyfus, H. (1994) · Leonardo · doi:10.2307/1575958 · Cited by 1,637
Computer power and human reason
Kuipers, B. J., McCarthy, J., & Weizenbaum, J. (1976) · ACM SIGART Bulletin · doi:10.1145/1045264.1045265 · Cited by 1,056
Colby, K. M. (1981) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00000030 · Cited by 966
Computer Power and Human Reason: From Judgement to Calculation.
Crowther, J. F., & Weizenbaum, J. (1977) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2062802 · Cited by 726
Block, N. (1981) · The Philosophical Review · doi:10.2307/2184371 · Cited by 562
Computing Machinery and Intelligence
Turing, A. M. (2004) · The MIT Press eBooks · doi:10.7551/mitpress/6928.003.0012 · Cited by 273
Alchemy and Artificial Intelligence
Dreyfus, H. L. (1965) · Cited by 215
Subcognition and the Limits of the TuringTest
French, R. M. (1990) · Mind · doi:10.1093/mind/xcix.393.53 · Cited by 198
Cited by
Sorted by citations
Items 101–200 of 223
16 Adaptationism, Culture, and the Malleability of Human Nature
Sripada, C. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0016 · Cited by 7
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0005 · Cited by 6
Turing Revisited: A Cognitively-Inspired Decomposition
Besold, T. R. (2012) · Studies in applied philosophy, epistemology and rational ethics · doi:10.1007/978-3-642-31674-6_9 · Cited by 6
Teckchandani, N., Santokhee, A., & Bekaroo, G. (2019) · Lecture notes in electrical engineering · doi:10.1007/978-3-030-18240-3_30 · Cited by 6
Harnad, S. R. (2003) · Studies in cognitive systems · doi:10.1007/978-94-010-0105-2_14 · Cited by 6
That's AI? : a history and critique of the field
Sweeney, L. (1989) · doi:10.1184/r1/6610337 · Cited by 6
Turing's Imitation Game: Conversations with the Unknown
Warwick, K., & Shah, H. (2016) · Cited by 6
Copeland, B. J. (2003) · Studies in cognitive systems · doi:10.1007/978-94-010-0105-2_1 · Cited by 6
A non-programming introduction to computer science via NLP, IR, and AI
Lee, L. (2002) · doi:10.3115/1118108.1118113 · Cited by 6
Look Who's Moving the Goal Posts Now
Hauser, L. (2001) · Minds and Machines · doi:10.1023/a:1011262707720 · Cited by 6
Networks of Deep Impression: Shakespeare and the History of Information
Galey, A. (2010) · Shakespeare Quarterly · doi:10.1353/shq.2010.0007 · Cited by 6
Rapaport, W. J. (2005) · Cited by 6
Universal Intelligence: A Definition of Machine IntelligencePreprint
Legg, S., & Hütter, M. (2007) · arXiv (Cornell University) · doi:10.48550/arxiv.0712.3329 · Cited by 6
Proceedings of the First Workshop on NLP for Conversational AI
(2019) · doi:10.18653/v1/w19-41 · Cited by 7
Generative AI Decision-Making Attributes in Complex Health Services: A Rapid Review
Doreswamy, N., & Horstmanshof, L. (2025) · Cureus · doi:10.7759/cureus.78257 · Cited by 11
12 The Creative Aspect of Language Use and Nonbiological Nativism
Baker, M. C. (2008) · Oxford University Press eBooks · doi:10.1093/oso/9780195332827.003.0012 · Cited by 6
Modeling Text Retrieval in Biomedicine
Wilbur, W. J. (2006) · Kluwer Academic Publishers eBooks · doi:10.1007/0-387-25739-x_10 · Cited by 5
Rapaport, W. J. (2006) · Elsevier eBooks · doi:10.1016/b0-08-044854-2/00933-0 · Cited by 5
Examining Artificial Intelligence Anxiety in the Context of Anthropocentrism and the Art Turing Test
Xu, R., Hsu, Y., & Wang, X. (2025) · Empirical Studies of the Arts · doi:10.1177/02762374251353671 · Cited by 5
The Computational Evolution of Cognitive Architectures
Kotseruba, I., & Tsotsos, J. K. (2025) · doi:10.1093/oso/9780192844835.001.0001 · Cited by 4
Kalaivani, S., Banu, D. N., Begum, I. P., & Sivasamy, J. (2023) · Journal of Artificial Intelligence and Capsule Networks · doi:10.36548/jaicn.2023.4.002 · Cited by 4
Nebel, B. (2019) · J.B. Metzler eBooks · doi:10.1007/978-3-476-05604-7_57 · Cited by 4
15 Innate Constraints on Judgment and Decision‐Making?
Santos, L. R., & Lakshminarayanan, V. R. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0015 · Cited by 4
The Science and Art of Voice Interfaces
Krahmer, E. (2001) · Data Archiving and Networked Services (DANS) · Cited by 4
Tests of Machine IntelligencePreprint
Legg, S., & Hütter, M. (2007) · arXiv (Cornell University) · doi:10.48550/arxiv.0712.3825 · Cited by 4
Kyllonen, P. C. (2021) · Educational research and innovation · doi:10.1787/feecd512-en · Cited by 2
Braggaar, A., He, L., & de Wit, J. (2024) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3640794.3665889 · Cited by 3
Challenges for Social Embodiment
André, E. (2014) · doi:10.1145/2666253.2666265 · Cited by 3
Rapaport, W. J. (2003) · Studies in cognitive systems · doi:10.1007/978-94-010-0105-2_9 · Cited by 3
A Definition and a Test for Human-Level Artificial Intelligence
Park, D., Mondol, M. A. R., Pothula, A., & Islam, M. (2023) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci52147.2023.10372029 · Cited by 4
Moor, J. H. (2003) · Encyclopedia of Computer Science · Cited by 3
Natale, S. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780190080365.003.0006 · Cited by 2
Natale, S. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780190080365.003.0001 · Cited by 2
Christopher Winch on the Representational Theory of Language and its Pedagogic Relevance
Mackenzie, J. (2001) · Educational Philosophy and Theory · doi:10.1080/00131850124794 · Cited by 2
Rajaraman, V. (1997) · Resonance · doi:10.1007/bf02835001 · Cited by 2
Negobot: Detecting paedophile activity with a conversational agent based on game theory
Laorden, C., Galán-García, P., Santos, I., Sanz, B., Nieves, J., Bringas, P. G., & Hidalgo, J. M. G. (2014) · Logic Journal of IGPL · doi:10.1093/jigpal/jzu034 · Cited by 2
The design of interactive conversation agents
Goh, O. S., & Fung, C. C. (2008) · Murdoch University Research Portal · doi:10.5555/1467049.1467054 · Cited by 2
SEEKER: A Conversational Agent as a Natural Language Interface to a relational Database
Smith, E. V., Crockett, K. A., Latham, A., & Buckingham, F. J. (2014) · e-space (Manchester Metropolitan University) · Cited by 2
11 Of Pigeons, Humans, Language, and the Mind
Bonatti, L. L. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0011 · Cited by 2
Covert Implementations of the Turing Test: A More Level Playing Field?
Burden, D. J. H., Savin‐Baden, M., & Bhakta, R. (2016) · Springer eBooks · doi:10.1007/978-3-319-47175-4_13 · Cited by 2
Smith, K. T., Coventry, L., & Greensmith, R. (2018) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-05297-3_13 · Cited by 2
Seeam, P., Teckchandani, N., Booneyad, H., Torul, V. P., & Seeam, A. K. (2018) · International Conference on Intelligent and Innovative Computing Applications · doi:10.1109/iconic.2018.8601271 · Cited by 2
The Cartesian Test for Automatism
Erion, G. J. (2003) · Studies in cognitive systems · doi:10.1007/978-94-010-0105-2_13 · Cited by 2
A ladder to human-comparable intelligence: an empirical metric
Šprogar, M. (2018) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2018.1509897 · Cited by 2
Carruthers, P., Laurence, S., & Stich, S. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0001 · Cited by 2
A framework and evaluation of conversation agents
Goh, O. S. (2008) · Murdoch University Research Portal · Cited by 2
Improvisation in interactive music systems
Gifford, T. (2011) · Cited by 2
Toward a Human-Level Video Understanding IntelligencePreprint
Heo, Y., Lee, M. S., Choi, S.-H., Choi, W. S., Shin, M., Jung, M., Ryu, J.-K., & Zhang, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04203 · Cited by 2
Benn, C. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00865-w · Cited by 2
Research Programs Based on Machine Intelligence Games
Tamburrini, G., & Altiero, F. (2020) · Philosophy of engineering and technology · doi:10.1007/978-3-030-54522-2_11 · Cited by 1
Measuring Chatbot Effectiveness
Mohelská, H., & Sokolová, M. (2021) · Hradec Economic Days ... · doi:10.36689/uhk/hed/2021-01-058 · Cited by 1
New approaches to understanding the impact of computers on work and education
Révai, N., Staneva, M., & Baret, A. (2021) · Educational research and innovation · doi:10.1787/65774a32-en · Cited by 1
Research and Application Status of Text Generation Tasks Based on Generative Adversarial Network
Wang, W., Jiang, D., & Cao, S. (2023) · Lecture notes in operations research · doi:10.1007/978-981-99-3618-2_11 · Cited by 1
Video Turing Test: A first step towards human‐level AI
Lee, M., Heo, Y., Choi, S.-H., Choi, W. S., & Zhang, B. (2023) · AI Magazine · doi:10.1002/aaai.12128 · Cited by 3
Shujie, Z. (2025) · Sino-US English Teaching · doi:10.17265/1539-8072/2025.02.001 · Cited by 1
Hiraga, R., Bresin, R., Hirata, K., & Katayose, H. (2003) · Cited by 1
Current computational transportation science
Winter, S., Chen, X., & Xu, B. (2016) · GeoInformatica · doi:10.1007/s10707-016-0249-y · Cited by 1
Natale, S. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780190080365.003.0002 · Cited by 1
A Computational Behaviorist Takes Turing’s Test
Whalen, T. E. (2007) · Springer eBooks · doi:10.1007/978-1-4020-6710-5_21 · Cited by 1
Maintaining credible dialogs in a VideoBot system with special audio techniques
DeGroot, D. (2002) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2001.974483 · Cited by 1
Noah, the Ark and the Turing Test
Epstein, R. G. (1996) · ACM SIGCAS Computers and Society · doi:10.1145/236394.606021 · Cited by 1
Understanding speech understanding
Moore, R. K. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1998.675448 · Cited by 1
An interactive multimedia continuously learning helpdesk system : (when Hal met Sally)
Groh, M. L. (1999) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 1
On the Winograd Schema: Situating Language Understanding in the Data-Information-Knowledge ContinuumPreprint
Saba, W. S. (2018) · PhilSci-Archive (University of Pittsburgh) · doi:10.48550/arxiv.1810.00324 · Cited by 1
Passing Loebner’s Turing Test: A Case of Conflicting Discourse Functions
Zdenek, S. (2003) · Studies in cognitive systems · doi:10.1007/978-94-010-0105-2_6 · Cited by 1
The Turing Test in the Classroom
Torrey, L. A., Johnson, K., Sondergard, S., Ponce, P., & Desmond, L. (2016) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v30i1.9853
Ejercitacitación introductoria a la inteligencia artificial con el test de Turing
Cecchi, L. (2000) · El Servicio de Difusión de la Creación Intelectual (National University of La Plata)
Avaliação de Agentes de Conversação: A influência de elementos Multimédia
Brandão, C. F. M. (2012) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · doi:10.34626/3hwm-rh92
How to Pass a Turing Test: Syntax Suffices for Understanding Natural Language
Rapaport, W. J. (1999)
Imitation Games and Political Discourse
Sanín, F. A. G. (2008) · Forum qualitative Sozialforschung · doi:10.17169/fqs-2.1.971
(1997) · OCLC Systems & Services · doi:10.1108/10650759710189434
Trust Based Knowledge Acquisition for Conversation Agents
Goh, O. S., & Fung, C. C. (2008) · Murdoch University Research Portal
Turing Test: An Approach to Deflning Machine Intelligence
Sayeed, A., & Saifullah, M. (2005)
Agenttien uskottavuus ja tekotyperyys digitaalisissa peleissä
Domander, R. (2012) · Jyväskylä University Digital Archive (University of Jyväskylä)
3 Genes, Environments, and Concepts of Biological Inheritance
Mameli, M. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0003
Gutachter, E., Wallmannsberger, J., & Gutachter, Z. (2006)
Geoinformation zur Navigationsunterstützung
Winter, S. (2019) · Springer Reference Naturwissenschaften · doi:10.1007/978-3-662-47096-1_65
Koren, O., & Perel, N. (2016) · AI Matters · doi:10.1145/2911172.2911175
History of Conversation Systems: FromElizatoEugene Goostman
Warwick, K., & Shah, H. (2016) · Cambridge University Press eBooks · doi:10.1017/cbo9781107297234.006
A disembodied developmental robotic agent
Norbert, B. (2015)
Designing Conversational Agents to Encourage User Narrative and Self-Reflection in Mental Wellbeing
Park, S. H. (2020) · Seoul National University Open Repository (Seoul National University)
Natale, S. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780190080365.003.0003
Natale, S. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780190080365.003.0005
Natale, S. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780190080365.003.0008
Self-recognition in conversational agentsPreprint
Oktar, Y., Okur, E., & Türkan, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02334 · Cited by 4
Mind Reading at Work: Cooperation without common groundPreprint
Wallis, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.01949
Common sense skills: Artificial intelligence and the workplace
Cheke, L. G., Halina, M., & Crosby, M. (2021) · Educational research and innovation · doi:10.1787/5fe70a0c-en
Assessing Natural Language Processing
Graham, Y. (2021) · Educational research and innovation · doi:10.1787/fcd5e244-en
Theory of Mind in Man-Machine Interactions
Hamada, F., & Miyamoto, E. T. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-27387-2_11
Look Who’s Moving the Goal Posts Now
Hauser, L. (2003) · Studies in cognitive systems · doi:10.1007/978-94-010-0105-2_10
IBOT: AN AGENT-BASED SOFTWARE FRAMEWORK FOR CREATING DOMAIN CONVERSATIONAL AGENTS
Velmovitsky, P. E. (2018) · doi:10.17771/pucrio.acad.35430
Geoinformation zur Navigationsunterstützung
Winter, S. (2015) · Springer eBooks · doi:10.1007/978-3-662-46900-2_65-1
(2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.002.0007
(2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.002.0005
(2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.002.0004
OECD (2022) · Educational research and innovation · doi:10.1787/39c4481a-fr
OECD (2022) · Educational research and innovation · doi:10.1787/cf9db84c-de
Metadata
- Authors
- Shieber, S. M.
- Affiliations
- Harvard University (US)
- Corresponding author
- Shieber, Stuart M.
- ORCID
- Shieber, Stuart M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 1994
- Volume
- 37
- Issue
- 6
- Pages
- 70-78
- ISSN
- 0001-0782 (ISSN-L)
- arXiv
- cmp-lg/9404002
- OpenAlex
- W2003453784
- Cited by
- 222
- References
- 21
- Citations by year
- 2026: 2 · 2025: 21 · 2024: 7 · 2023: 5 · 2022: 5 · 2021: 20 · 2020: 5 · 2019: 7 · 2018: 9 · 2017: 1 · 2016: 13 · 2015: 3 · 2014: 6 · 2012: 5
- FWCI
- 4.63
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Communications of the ACM (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Cybernetics and Technology in Society
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- Turing test, machine intelligence, AI evaluation, competition design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
