The DOI matches its DataCite record; 19 of the 21 references it cites were found in a registry or an index.
Lessons from a Restricted Turing Test
Open accessAbstract
We report on the recent Loebner prize competition inspired by Turing's test of intelligent behavior. The presentation covers the structure of the competition and the outcome of its first instantiation in an actual event, and an analysis of the purpose, design, and appropriateness of such a competition. We argue that the competition has no clear purpose, that its design prevents any useful outcome, and that such a competition is inappropriate given the current level of technology. We then speculate as to suitable alternatives to the Loebner prize.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The Loebner Prize [47]-an annual competition in which entrant systems tried to fool a panel of expert judges-ran from 1990 to 2020 without deeming a single system to have passed.”
From People cannot distinguish GPT-4 from a human in a Turing test (2025) · cites this paper as [47]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“While this differs from Turing's original threeparty formulation, it has become a standard operationalisation of the test because it eliminates the confound of the third player's humanlikeness and is easier to implement [13,45,47].”
From People cannot distinguish GPT-4 from a human in a Turing test (2025) · cites this paper as [47]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Searle, J. R. (1980) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00005756 · Cited by 6,954
Computing Machinery and Intelligence (1950)
Turing, A. M. (2004) · doi:10.1093/oso/9780198250791.003.0017 · Cited by 6,120
What computers still can't do: a critique of artificial reason
Penny, S., & Dreyfus, H. (1994) · Leonardo · doi:10.2307/1575958 · Cited by 1,637
Computer power and human reason
Kuipers, B. J., McCarthy, J., & Weizenbaum, J. (1976) · ACM SIGART Bulletin · doi:10.1145/1045264.1045265 · Cited by 1,056
Colby, K. M. (1981) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00000030 · Cited by 966
Computer Power and Human Reason: From Judgement to Calculation.
Crowther, J. F., & Weizenbaum, J. (1977) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2062802 · Cited by 726
Block, N. (1981) · The Philosophical Review · doi:10.2307/2184371 · Cited by 562
Computing Machinery and Intelligence
Turing, A. M. (2004) · The MIT Press eBooks · doi:10.7551/mitpress/6928.003.0012 · Cited by 273
Alchemy and Artificial Intelligence
Dreyfus, H. L. (1965) · Cited by 215
Subcognition and the Limits of the TuringTest
French, R. M. (1990) · Mind · doi:10.1093/mind/xcix.393.53 · Cited by 198
Cited by
Sorted by citations
Levesque, H. J., Davis, E., & Morgenstern, L. (2011) · Cited by 879
Universal intelligence: a definition of machine intelligence
Legg, S., & Hutter, M. (2007) · Minds and Machines · doi:10.1007/s11023-007-9079-x · Cited by 803
From Eliza to XiaoIce: Challenges and opportunities with social chatbots
Shum, H.-Y., He, X.-D., & Li, D. (2018) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1700826 · Cited by 749
“In the past, the Turing test and its extensions have been used to evaluate chitchat performance (Shieber, 1994).”
BackgroundMethodInfluentialHaidt, J., & Joseph, C. (2008) · The Innate Mind, Volume 3 · doi:10.1093/acprof:oso/9780195332834.003.0019 · Cited by 650
Pinar Saygin, A., Cicekli, I., & Akman, V. (2000) · Minds and Machines · doi:10.1023/a:1011288000451 · Cited by 383
ChatterBots, TinyMuds, and the Turing test: entering the Loebner Prize competition
Mauldin, M. L. (1994) · National Conference on Artificial Intelligence · Cited by 304
The Turing Test: the first 50 years
French, R. M. (2000) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01453-4 · Cited by 250
Deceitful Media: Artificial Intelligence and Social Life after the Turing Test
Natale, S. (2021) · doi:10.1093/oso/9780190080365.001.0001 · Cited by 209
A Turing Test for Computer Game Bots
Hingston, P. (2009) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2009.2032534 · Cited by 167
Stich, S. G. W. (2008) · doi:10.1093/acprof:oso/9780195332834.001.0001 · Cited by 160
Different measurements metrics to evaluate a chatbot system
Shawar, B. A., & Atwell, E. (2007) · doi:10.3115/1556328.1556341 · Cited by 151
AI and the Future of Skills, Volume 1
OECD (2021) · Educational research and innovation · doi:10.1787/5ee71f34-en · Cited by 233
Computer Science Unplugged: off-line activities and games for all ages
Bell, T., Witten, I. H., & Fellows, M. R. (1998) · Cited by 141
Copeland, B. J. (2000) · Minds and Machines · doi:10.1023/a:1011285919106 · Cited by 134
Hernández‐Orallo, J. (2000) · Journal of Logic Language and Information · doi:10.1023/a:1008367325700 · Cited by 117
The Interrogator as Critic: The Turing Test and the Evaluation of Generative Music Systems
Ariza, C. (2009) · Computer Music Journal · doi:10.1162/comj.2009.33.2.48 · Cited by 109
Youkui, W., Zhang, N., & Zhao, X. (2020) · Social Science Computer Review · doi:10.1177/0894439320980132 · Cited by 116
Gooding, M. J., Smith, A. J., Tariq, M., Aljabar, P., Peressutti, D., van der Stoep, J., Reymen, B., Emans, D., Hattu, D., van Loon, J., et al. (2018) · Medical Physics · doi:10.1002/mp.13200 · Cited by 89
Legg, S. (2008) · reroDoc Digital Library · Cited by 86
Does GPT-4 pass the Turing test?
Jones, C., & Bergen, B. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.290 · Cited by 86
17 Some Innate Foundations of Social and Moral Cognition
Wynn, K. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0017 · Cited by 69
Metrics and benchmarks in human-robot interaction: Recent advances in cognitive robotics
Aly, A., Griffiths, S., & Stramandinoli, F. (2016) · Cognitive Systems Research · doi:10.1016/j.cogsys.2016.06.002 · Cited by 67
People cannot distinguish GPT-4 from a human in a Turing test
Jones, C. R., Rathi, I., Taylor, S., & Bergen, B. K. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732108 · Cited by 76
(2001) · The MIT Press eBooks · doi:10.7551/mitpress/4617.003.0004 · Cited by 54
Rapaport, W. J. (2005) · Computational Linguistics · doi:10.1162/089120105774321127 · Cited by 53
Turing's Two Tests for Intelligence*
Sterrett, S. G. (2000) · Minds and Machines · doi:10.1023/a:1011242120015 · Cited by 52
The Study of the Application of a Keywords-based Chatbot System on the Teaching of Foreign LanguagesPreprint
Jia, J. (2003) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0310018 · Cited by 52
How to Pass the Turing Test by Cheating
Hutchens, J. L. (1997) · Cited by 51
Saygın, A. P., Cicekli, I., & Akman, V. (2003) · Studies in cognitive systems · doi:10.1007/978-94-010-0105-2_2 · Cited by 50
Harnad, S. R. (2000) · Journal of Logic Language and Information · doi:10.1023/a:1008315308862 · Cited by 49
(2009) · Springer eBooks · doi:10.1007/978-3-211-75540-2_10 · Cited by 47
A Neurally Controlled Computer Game Avatar With Humanlike Behavior
Gamez, D., Fountas, Z., & Fidjeland, A. K. (2012) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2012.2228483 · Cited by 45
Rapaport, W. J. (2000) · Journal of Logic Language and Information · doi:10.1023/a:1008319409770 · Cited by 43
Dialogue management systems: a survey and overview
Churcher, G. E., Atwell, E., & Souter, C. (1997) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 41
Oppy, G., & Dowe, D. L. (2003) · Stanford encyclopedia of philosophy · Cited by 40
The Turing Test is a Thought Experiment
Gonçalves, B. (2022) · Minds and Machines · doi:10.1007/s11023-022-09616-8 · Cited by 47
Text compression as a test for artificial intelligence
Mahoney, M. V. (1999) · Cited by 36
Minds, Machines and Turing: The Indistinguishability of IndistinguishablesPreprint
Harnad, S. R. (2000) · CogPrints (University of Southampton) · Cited by 36
In cyberspace all they see is your words
Whitley, E. A. (1997) · Information Technology and People · doi:10.1108/09593849710174995 · Cited by 34
Smart Hotel Using Intelligent Chatbot : A Review
Parmar, S., Meshram, M., Parmar, P., Patel, M., & Desai, P. (2019) · International Journal of Scientific Research in Computer Science Engineering and Information Technology · doi:10.32628/cseit11952246 · Cited by 34
Turing’s Two Tests for Intelligence
Sterrett, S. G. (2003) · Studies in cognitive systems · doi:10.1007/978-94-010-0105-2_3 · Cited by 32
Hidalgo, J. M. G., & Álvarez, G. (2011) · Advances in computers · doi:10.1016/b978-0-12-385510-7.00003-5 · Cited by 31
10 Rational Statistical Inference and Cognitive Development
Xu, F. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0010 · Cited by 34
Hutchens, J. L., & Alder, M. D. (1998) · Conference on Computational Natural Language Learning · doi:10.3115/1603899.1603945 · Cited by 31
Legg, S., & Hütter, M. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-77296-5_22 · Cited by 30
Making the Right Identification in the Turing Test1
Traiger, S. (2000) · Minds and Machines · doi:10.1023/a:1011254505902 · Cited by 29
Eisenmann, C., Mlynář, J., Turowetz, J. J., & Rawls, A. W. (2023) · AI & Society · doi:10.1007/s00146-023-01793-z · Cited by 28
Rising Up from the MUD: Inscribing Gender in Software Design
Zdenek, S. (1999) · Discourse & Society · doi:10.1177/0957926599010003005 · Cited by 27
13 The Creative Action Theory of Creativity
Carruthers, P. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0013 · Cited by 27
The total Turing test and the Loebner prize
Powers, D. M. W. (1998) · Conference on Computational Natural Language Learning · doi:10.3115/1603899.1603947 · Cited by 25
The Cartesian Test for Automatism1
Erion, G. J. (2001) · Minds and Machines · doi:10.1023/a:1011258623649 · Cited by 24
Beyond Chatbot-K: On Large Language Models, “Generative AI,” and Rise of Chatbots—An Introduction
Goodlad, L. M. E., & Stone, M. (2024) · Critical AI · doi:10.1215/2834703x-11205147 · Cited by 33
2 Is Innateness a Confused Concept?
Samuels, R. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0002 · Cited by 25
Methods for chatbot detection in distributed text-based communications
McIntire, J. P., McIntire, L. K., & Havig, P. R. (2010) · International Conference on Collaboration Technologies and Systems (CTS) · doi:10.1109/cts.2010.5478478 · Cited by 24
Toward a Comprehension Challenge, Using Crowdsourcing as a Tool
Paritosh, P. K., & Marcus, G. (2016) · AI Magazine · doi:10.1609/aimag.v37i1.2649 · Cited by 23
Making the Right Identification in the Turing Test
Traiger, S. (2003) · Studies in cognitive systems · doi:10.1007/978-94-010-0105-2_4 · Cited by 22
Simulating conversations: The communion game
Cowley, S. J., & MacDorman, K. F. (1995) · AI & Society · doi:10.1007/bf01210600 · Cited by 22
Prospects for in-depth story understanding by computerPreprint
Mueller, E. T. (2000) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0003003 · Cited by 22
State-of-the-Art in Open-Domain Conversational AI: A Survey
Adewumi, T., Liwicki, F., & Liwicki, M. (2022) · Information · doi:10.3390/info13060298 · Cited by 24
Rencon 2004: Turing Test for Musical Expression
Hiraga, R., Bresin, R., Hirata, K., Technology, R. I. O., & Kristinas, D., V (2004) · Cited by 21
Have we witnessed a real-life Turing Test?
Krol, M. (1999) · Computer · doi:10.1109/2.751325 · Cited by 20
Imagine: Thought Experiments in Information Systems Research
Introna, L. D., & Whitley, E. A. (1997) · Springer eBooks · doi:10.1007/978-0-387-35309-8_24 · Cited by 20
Towards a more natural and intelligent interface with embodied conversation agent
Goh, O. S., Wong, K. W., Fung, C. C., & Depickere, A. A. (2006) · doi:10.5555/1234341.1234370 · Cited by 20
Copeland, J., & Proudfoot, D. (2007) · Springer eBooks · doi:10.1007/978-1-4020-6710-5_9 · Cited by 20
University Entrance Exam as a Guiding Test for Artificial Intelligence
Silveira, I. C., & Mauá, D. D. (2017) · Brazilian Conference on Intelligent Systems · doi:10.1109/bracis.2017.44 · Cited by 19
4 Innateness and Genetic Information
Godfrey‐Smith, P. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0004 · Cited by 19
The Origins of Generative AI in Transcription and Machine Translation, and Why That Matters
Stone, M., Goodlad, L. M. E., & Sammons, M. (2024) · Critical AI · doi:10.1215/2834703x-11256853 · Cited by 21
A corpus-based approach to generalising a chatbot system
Shawar, B. A., & Atwell, E. (2003) · RUA, Repositorio Institucional de la Universidad de Alicante (Universidad de Alicante) · Cited by 18
Leslie, A. M., Gallistel, C. R., & Gelman, R. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0007 · Cited by 17
A fuzzy logic approach to experience-based reasoning
Sun, Z., & Finnie, G. R. (2007) · International Journal of Intelligent Systems · doi:10.1002/int.20220 · Cited by 17
Health State EstimationPreprint
Nag, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09312 · Cited by 19
Warwick, K., & Shah, H. (2016) · Ai & Society · doi:10.1007/s00146-015-0588-5 · Cited by 15
Gams, M., & Kramar, S. (2024) · Journal of Computer and Communications · doi:10.4236/jcc.2024.123014 · Cited by 15
6 Poverty of Stimulus Arguments Concerning Language and Folk Psychology
Segal, G. (2008) · Oxford University Press eBooks · doi:10.1093/oso/9780195332827.003.0006 · Cited by 16
Passing Loebner's Turing Test: A Case of Conflicting Discourse Functions1
Zdenek, S. (2001) · Minds and Machines · doi:10.1023/a:1011214808628 · Cited by 15
Building Safer AGI by introducing Artificial Stupidity
Trazzi, M., & Yampolskiy, R. V. (2018) · arXiv · doi:10.48550/arxiv.1808.03644 · Cited by 15
Principles for Designing an AI Competition, or Why the Turing Test Fails as an Inducement Prize
Shieber, S. M. (2016) · AI Magazine · doi:10.1609/aimag.v37i1.2646 · Cited by 15
Self-citationRapaport, W. J. (2016) · Cited by 14
The Computer, Artificial Intelligence, and the Turing Test
Copeland, B. J., & Proudfoot, D. (2004) · Springer eBooks · doi:10.1007/978-3-662-05642-4_13 · Cited by 13
McCoy, J. P., & Ullman, T. D. (2018) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2018.05.007 · Cited by 13
My AI students: Evaluating the proficiency of three AI chatbots in completeness and accuracy
Govender, R. G. (2024) · Contemporary Educational Technology · doi:10.30935/cedtech/14564 · Cited by 15
Mueller, S. T., & Minnery, B. S. (2008) · Cited by 12
8 Linguistic Determinism and the Innate Basis of Number
Laurence, S., & Margolis, E. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0008 · Cited by 15
Intelligent Conversational Bot for Massive Online Open Courses (MOOCs)Preprint
Lim, S. L., & Goh, O. S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.07065 · Cited by 13
A Web-Based Real Time Chat Application for User Friendly Interaction
Rawat, V., Jain, P., Tomar, A., Guzar, P., Negi, V., & Gupta, V. K. (2024) · International Conference on Advanced Computing and Emerging Technologies (ACET) · doi:10.1109/acet61898.2024.10730503 · Cited by 11
How Would You Know if You Synthesized a Thinking Thing?
Kary, M., & Mahner, M. (2002) · Minds and Machines · doi:10.1023/a:1013738831686 · Cited by 11
Is it enough to get the behaviour right
Levesque, H. J. (2009) · Cited by 10
5 Genes and Human Psychological Traits
Bouchard, T. J. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0005 · Cited by 10
18 Two Theories About the Cognitive Architecture Underlying Morality
Kelly, D., & Stich, S. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0018 · Cited by 30
A Chatbot Dialogue Manager - Chatbots and Dialogue Systems: A Hybrid Approach
van Woudenberg, A. (2014) · DSpace (Open University in the Netherlands) · Cited by 10
Blaha, L. M., Abrams, M., Bibyk, S., Bonial, C., Hartzler, B. M., Hsu, C. D., Khemlani, S., King, J. M., Amant, R. S., Trafton, J. G., et al. (2022) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2022.800280 · Cited by 17
Imitation Versus Communication: Testing for Human-Like Intelligence
Cullen, J. S. (2009) · Minds and Machines · doi:10.1007/s11023-009-9149-3 · Cited by 9
14 Space and the Language-Cognition Interface
Anna Papafragou, A. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0014 · Cited by 11
Open Domain Chatbot Based on Attentive End-to-End Seq2Seq Mechanism
Salahudeen, S. A., Sun, Y., Abdullahi, A., & Umar, A. (2019) · International Conference on Algorithms, Computing and Artificial Intelligence (ACAI) · doi:10.1145/3377713.3377773 · Cited by 9
Turing’s Test, a Beautiful Thought Experiment
Gonçalves, B. (2024) · IEEE Annals of the History of Computing · doi:10.1109/mahc.2024.3432278 · Cited by 9
Insights from Building an Open-Ended Conversational Agent
Gupta, K., Joshi, M., Chatterjee, A., Damani, S., Narahari, K. N., & Agrawal, P. (2019) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/w19-4112 · Cited by 8
9 Learning “About” Versus Learning “From” Other Minds
Gergely, G. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195332834.003.0009 · Cited by 8
Natale, S. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780190080365.003.0007 · Cited by 9
Aucouturier, J., & Ikegami, T. (2009) · Adaptive Behavior · doi:10.1177/1059712309344420 · Cited by 7
Natale, S. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780190080365.003.0004 · Cited by 13
Metadata
- Authors
- Shieber, S. M.
- Affiliations
- Harvard University (US)
- Corresponding author
- Shieber, Stuart M.
- ORCID
- Shieber, Stuart M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 1994
- Volume
- 37
- Issue
- 6
- Pages
- 70-78
- ISSN
- 0001-0782 (ISSN-L)
- arXiv
- cmp-lg/9404002
- OpenAlex
- W2003453784
- Cited by
- 222
- References
- 21
- Citations by year
- 2026: 2 · 2025: 21 · 2024: 7 · 2023: 5 · 2022: 5 · 2021: 20 · 2020: 5 · 2019: 7 · 2018: 9 · 2017: 1 · 2016: 13 · 2015: 3 · 2014: 6 · 2012: 5
- FWCI
- 4.63
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Communications of the ACM (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Cybernetics and Technology in Society
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- Turing test, machine intelligence, AI evaluation, competition design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
