TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation
Open accessAbstract
Recent progress in generative language models has enabled machines to generate astonishingly realistic texts.While there are many legitimate applications of such models, there is also a rising need to distinguish machine-generated texts from human-written ones (e.g., fake news detection).However, to our best knowledge, there is currently no benchmark environment with datasets and tasks to systematically study the so-called "Turing Test" problem for neural text generation methods.In this work, we present the TURINGBENCH benchmark environment, which is comprised of (1) a dataset with 200K human-or machine-generated samples across 20 labels {Human, GPT-1, GPT-2_small, GPT-2_medium, GPT-2_large, GPT-2_xl, GPT-2_PyTorch, GPT-3, GROVER_base, GROVER_large, GROVER_mega, CTRL, XLM, XLNET_base, XLNET_large, FAIR_wmt19, FAIR_wmt20, TRANS-FORMER_XL, PPLM_distil, PPLM_gpt2}, (2) two benchmark tasks-i.e., Turing Test (TT) and Authorship Attribution (AA), and (3) a website with leaderboards.Our preliminary experimental results using TURINGBENCH show that FAIR_wmt20 and GPT-3 are the current winners, among all language models tested, in generating the most human-like indistinguishable texts with the lowest F1 score by five state-of-the-art TT detection models.The TURINGBENCH is available at: https: //turingbench.ist.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Variations in training data, model architecture, and fine-tuning processes contribute to the distinctive outputs of different LLMs, though these differences are often subtle and difficult to detect.
“Variations in training data, model architecture, and fine-tuning processes contribute to the distinctive outputs of different LLMs, though these differences are often subtle and difficult to detect [Uchendu et al., 2021].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Uchendu et al., 2021
It is common practice to group different human writers into a single category because humans exhibit a broader spectrum of writing styles and proficiency levels compared to machines.
“To simplify the classification process, it is common practice to group different human writers into a single category because humans exhibit a broader spectrum of writing styles and proficiency levels compared to machines [Uchendu et al., 2021].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Uchendu et al., 2021
Previous studies have often oversimplified the problem by categorizing all human-written text into a single category.
“Previous studies have often oversimplified the problem by categorizing all human-written text into a single category [Uchendu et al., 2021;He et al., 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Uchendu et al., 2021
TuringBench comprises human-written content from 10k news articles, predominantly from reputable sources such as CNN, with only articles ranging between 200 and 400 words selected.
“It comprises human-written content derived from 10k news articles, predominantly from reputable sources such as CNN. For the purpose of this dataset, only articles ranging between 200 to 400 words were selected.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Uchendu et al. 2021
TuringBench emerged as one of the pioneering benchmark environments for the detection of LLM-generated text.
“Notably, TuringBench emerged as one of the pioneering benchmark environments for the detection of LLM-generated text.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Uchendu et al. 2021
Each model contributed 8k samples to TuringBench, categorized by label type.
“Each model contributed 8k samples, categorized by label type.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Uchendu et al. 2021
TuringBench's LLM-generated text is produced by 19 distinct text generation models, including GPT-1, GPT-2 variants, GPT-3, GROVER, CTRL, XLM, XLNET, FAIR, Transformer-XL, and PLM variants.
“LLM-generated text within this dataset is produced by 19 distinct text generation models, including GPT-1, GPT-2 variants (small, medium, large, xl, and pytorch), GPT-3, different versions of GROVER (base, large, and mega), CTRL, XLM, XLNET variants (base and large), FAIR for both WMT19”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Uchendu et al. 2021
The citation lists haven’t been fetched yet
References 69 · Cited by 82
- Authors
- Uchendu, A., Ma, Z., Le, T., Zhang, R., & Lee, D.
- Affiliations
- Pennsylvania State University (US)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2021
- Published
- 2021
- Pages
- 2001-2016
- OpenAlex
- W3204318298
- Cited by
- 82
- References
- 69
- Citations by year
- 2026: 18 · 2025: 19 · 2024: 17 · 2023: 22 · 2022: 5 · 2021: 1
- FWCI
- 5.20
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1820609, 2114824, 1742702, 1915801, 1909702)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- Turing test, AI-generated text detection, neural text generation, authorship attribution, benchmark dataset, generative language models, GPT-3, leaderboards
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
