GLTR: Statistical Detection and Visualization of Generated Text
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
The rapid improvement of language models has raised the specter of abuse of text generation systems.This progress motivates the development of simple methods for detecting generated text that can be used by and explained to non-experts.We develop GLTR, a tool to support humans in detecting whether a text was generated by a model.GLTR applies a suite of baseline statistical methods that can detect generation artifacts across common sampling schemes.In a human-subjects study, we show that the annotation scheme provided by GLTR improves the human detection-rate of fake text from 54% to 72% without any prior training.GLTR is open-source and publicly deployed, and has already been widely used to detect generated outputs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The rest of the zero-shot detectors (Solaiman et al., 2019; Gehrmann et al., 2019; Ippolito et al., 2019) also perform poorly after our attack.”
From Can AI-Generated Text be Reliably Detected? (2023) · cites this paper as Gehrmann et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“To effectively analyze and classify human-LLM co-authored texts versus those solely authored by humans or LLMs, feature-based methods from the problem of LLM-generated text detection, such as Log-likelihood [Solaiman et al., 2019], GLTR [Gehrmann et al., 2019a], and log-rank [Mitchell et al., 2023], are adapted to this task.”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Gehrmann et al., 2019a
The cited source says what the claim attributes to it.
In the citing paper
“Human capability to discern AI-generated text from human-written content barely exceeds that of a random classifier (Gehrmann et al., 2019;Else, 2023;Clark et al., 2021), heightening the risk that unsubstantiated generated text can masquerade as authoritative, evidence-based writing.”
From Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (2024) · cites this paper as Gehrmann et al., 2019
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We consider only the case of human-readable prompts to large models, which contrasts with methods such as prompt-tuning [16,20], which learns a continuous prompt embedding, and auto-prompting [33], which attempts to search for prompts from scratch, both of which require a training step.”
From Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models (2022) · cites this paper as [33]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The evaluation of the quality of machine-generated excerpts has traditionally relied on human judgement, which is considered the gold standard for open-domain generation systems [van der Lee et al., 2019;Gehrmann et al., 2019a].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Gehrmann et al., 2019a
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 6×
“The success of pretrained language models for natural language understanding (McCann et al., 2017; Devlin et al., 2018; Peters et al., 2018) has led to a race to train unprecedentedly large language models (Radford et al.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 2×
“The success of pretrained language models for natural language understanding (McCann et al., 2017; Devlin et al., 2018; Peters et al., 2018) has led to a race to train unprecedentedly large language models (Radford et al.”
BackgroundFake News Detection on Social Media: A Data Mining Perspective
Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, H. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3137597.3137600 · Cited by 3,304 · cited 1×
“Finally, we distinguish this task from detecting misinformation in text (e.g. Shu et al., 2017).”
MethodCentering: A Framework for Modelling the Local Coherence of Discourse,
Grosz, B. J., Joshi, A. K., & Weinstein, S. (1995) · doi:10.21236/ada324949 · Cited by 1,773 · cited 1×
“While humans would vary expressions in real texts, models rarely generate synonyms or referring expressions for entities, which does not follow the theory of centering in discourse analysis (Grosz et al., 1995).”
Background"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection
Wang, W. Y. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p17-2067 · Cited by 1,516 · cited 1×
“Other scenarios include the possibility of generating false articles (Wang, 2017) or misleading reviews (Fornaciari and Poe-sio, 2014).”
BackgroundHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266 · cited 2×
“, 2017), k-max sampling (Fan et al., 2018), beam search (Chorowski and Jaitly, 2016; Shao et al.”
BackgroundHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
Hill, F., Bordes, A., Chopra, S., & Weston, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02301 · Cited by 467 · cited 2×
“graphs from the bAbI task children book corpus (CBT) (Hill et al., 2015), New York Times articles (NYT), and scientific abstracts from the journals nature and science (SA).”
BackgroundMethodTowards Better Decoding and Language Model Integration in Sequence to Sequence Models
Chorowski, J., & Jaitly, N. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-343 · Cited by 358 · cited 1×
“, 2018), beam search (Chorowski and Jaitly, 2016; Shao et al., 2017), temperature-modulated sampling (Dagan and Engelson, 1995), or even implicitly with rule-based templated approaches.”
Background
Research works printed but not linked: 4
- EA Grechnikov, GG Gusev, AA Kustarev, and A Raigorodsky. 2009. Detection of artificial texts. RCDL2009 Proceedings. Petrozavodsk, pages 306- 308.
- James V. Grimaldi. 2018. U.s. investigating fake com- ments on net neutrality. The Wall Street Journal.
- Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Con- textualized word vectors. In Advances in Neural In- formation Processing Systems, pages 6294-6305.
- Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. arXiv preprint arXiv:1802.05365. · arXiv
Cited by
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Perkins, M. (2023) · Journal of University Teaching and Learning Practice · doi:10.53761/1.20.02.07 · Cited by 839
“Technological methods to identify text generated by LLMs have been proposed by Gehrmann et al. (2019) who present the results of a study using a tool named GLTR.”
Human heuristics for AI-generated language are flawed
Jakesch, M., Hancock, J. T., & Naaman, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208839120 · Cited by 342
Release Strategies and the Social Impacts of Language Models
Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J. W., Kreps, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1908.09203 · Cited by 292
“on GLTR [26], which shows these probabilities to humans in a friendly interface.”
BackgroundMethodInfluentialAll That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290
Metadata
- Authors
- Gehrmann, S., Strobelt, H., & Rush, A.
- Affiliations
- Harvard University (US)
- IBM (United States) (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations
- Published
- 2019
- Pages
- 111-116
- OpenAlex
- W2949028947
- Cited by
- 470
- References
- 19
- Citations by year
- 2026: 108 · 2025: 123 · 2024: 75 · 2023: 76 · 2022: 19 · 2021: 34 · 2020: 31 · 2019: 4
- FWCI
- 8.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1845664)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- statistical detection methods, sampling artifacts, manual annotation, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: GLTR: Statistical Detection and Visualization of Generated Text. https://app.cochat.ai/papers/10.18653/v1/p19-3019
- The paper
- Gehrmann, S., Strobelt, H., & Rush, A. (2019). GLTR: Statistical Detection and Visualization of Generated Text. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. https://doi.org/10.18653/v1/p19-3019
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.