Technical Report on the Pangram AI-Generated Text Classifier
Open accessAbstract
We present Pangram Text, a transformer-based neural network trained to distinguish text written by large language models from text written by humans. Pangram Text outperforms zero-shot methods such as DetectGPT as well as leading commercial AI detection tools with over 38 times lower error rates on a comprehensive benchmark comprised of 10 text domains (student writing, creative writing, scientific writing, books, encyclopedias, news, email, scientific papers, short-form Q&A) and 8 open- and closed-source large language models. We propose a training algorithm, hard negative mining with synthetic mirrors, that enables our classifier to achieve orders of magnitude lower false positive rates on high-data domains such as reviews. Finally, we show that Pangram Text is not biased against nonnative English speakers and generalizes to domains and models unseen during training.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Pangram operates on overlapping windows of text rather than whole documents, allowing it to flag localized AI-written segments embedded within otherwise human-authored manuscripts.
“Crucially, Pangram operates on overlapping windows of text rather than whole documents, allowing it to flag localized AI-written segments embedded within otherwise human-authored manuscripts.”
From Fine-Grained Detection of AI-Generated Writing in the Biomedical Literature (2026) · cites this paper as [8]
The Abstract says Pangram distinguishes AI-written from human-written text, and §3.1 evaluates it on documents. Our copy does not say it operates on overlapping windows or can flag localized AI-written segments within otherwise human-authored manuscripts (searched: overlapping windows, sliding windows, and partially AI-written text). The passages have no page numbers. (LLM, conf=0.82)
Experimental — accuracy not yet measuredPangram reaches approximately 99-99.8% accuracy with false-positive rates on the order of 0.01-0.1% across domains, including scientific writing and essays by non-native English speakers, and remains highly effective against adversarial humanizer paraphrases.
“In head-to-head benchmarks it reaches approximately 99-99.8% accuracy with false-positive rates on the order of 0.01-0.1% across domains, including in scientific writing and essays by non-native English speakers, and remains highly effective even against adversarial 'humanizer' paraphrases that are explicitly optimized to evade detection 8 .”
From Fine-Grained Detection of AI-Generated Writing in the Biomedical Literature (2026) · cites this paper as [8]
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“Our model is a slightly modified transformer-style architecture (Vaswani et al., 2017).”
MethodObject Detection with Discriminatively Trained Part-Based Models
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.167 · Cited by 9,853 · cited 1× · see citing works
“(Felzenszwalb et al., 2010) (Shrivastava et al., 2016).”
Training Region-Based Object Detectors with Online Hard Example Mining
Shrivastava, A., Gupta, A., & Girshick, R. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.89 · Cited by 2,686 · cited 1× · see citing works
“(Felzenszwalb et al., 2010) (Shrivastava et al., 2016).”
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266 · cited 1× · see citing works
“Creative writing is taken mostly from the Reddit WritingPrompts subreddit (Fan et al., 2018) as well as the Ghostbusters subset (Verma et al., 2023).”
MethodA Survey of Deep Active Learning
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Gupta, B. B., Chen, X., & Wang, X. (2022) · ACM Computing Surveys · doi:10.1145/3472291 · Cited by 1,169 · cited 1× · see citing works
“Deep active learning is also a well-researched field of study and an excellent survey can be found here (Ren et al., 2020).”
BackgroundTesting of detection tools for AI-generated text
Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023) · International Journal for Educational Integrity · doi:10.1007/s40979-023-00146-z · Cited by 582 · see citing works
GLTR: Statistical Detection and Visualization of Generated Text
Gehrmann, S., Strobelt, H., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3019 · Cited by 470 · cited 1× · see citing works
“GLTR (Gehrmann et al., 2019) uses entropy, probability, and probability rank to determine whether or not the statistical patterns in the text are AI-generated.”
MethodDetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated TextPreprint
Su, J., Zhuo, T. Y., Wang, D., & Nakov, P. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2306.05540 · Cited by 347 · cited 2× · see citing works
“DetectLLM (Su et al., 2023) builds upon DetectGPT by also introducing similar methods based on log-rank information.”
MethodCasal, J. E., & Kessler, M. (2023) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2023.100068 · Cited by 261 · cited 1× · see citing works
“5, and ChatGPT series (OpenAI, 2023a), Gemini (Google, 2023), and several open source models are now capable of producing fluent text that even trained experts are not able to distinguish perfectly (Casal and Kessler, 2023).”
BackgroundXL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages
Hasan, T., Bhattacharjee, A., Islam, M. S., Mubasshir, K. S., Li, Y.-F., Kang, Y., Rahman, M. S., & Shahriyar, R. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.413 · Cited by 240 · cited 1× · see citing works
“To benchmark our performance, we use the Amazon Multilingual Reviews dataset (Keung et al., 2020), the XLSum dataset (Hasan et al., 2021), and the multilingual Wikipedia datasets, and generate synthetic mirrors for each.”
MethodA question-entailment approach to question answering
Abacha, A. B., & Demner‐Fushman, D. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3119-4 · Cited by 221 · cited 1× · see citing works
“QA is taken from a mix of sources, including Reddit ELI5, AskHistorians, AskScience (Fan et al., 2019), as well as the open source MedQUAD (Asma and Demner-Fushman, 2019) and FIQA datasets (fiq, 2018).”
MethodThe Multilingual Amazon Reviews Corpus
Keung, P., Lu, Y., Szarvas, G., & Smith, N. A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.369 · Cited by 162 · cited 1× · see citing works
“To benchmark our performance, we use the Amazon Multilingual Reviews dataset (Keung et al., 2020), the XLSum dataset (Hasan et al., 2021), and the multilingual Wikipedia datasets, and generate synthetic mirrors for each.”
MethodDetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2301.11305 · Cited by 151 · cited 12× · see citing works
“Perplexity based methods - such as DetectGPT, Sniffer and DetectLLM - fail on human-written documents that happen to be in the training set of LLMs, such as the Declaration of Independence.”
MethodResultInfluentialParaphrasing evades detectors of AI-generated text, but retrieval is an effective defense
Krishna, K., Song, Y., Karpinska, M., Wieting, J., & Iyyer, M. (2023) · arXiv · doi:10.48550/arxiv.2303.13408 · Cited by 150 · cited 1× · see citing works
“Furthermore, recent studies (Sadasivan et al., 2023) (Krishna et al., 2023) have shown that watermarking schemes can easily be bypassed using another model that can paraphrase the output of an LLM.”
BackgroundActive Learning for Deep Detection Neural Networks
Aghdam, H. H., Gonzalez-Garcia, A., Lopez, A., & Weijer, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00377 · Cited by 148 · cited 1× · see citing works
“A salient example of active learning being used in a detection task is in (Aghdam et al., 2019) where the neural network’s mispredictions of bounding boxes in an object detection task is used to query human labelers for frame-level annotations.”
BackgroundA large-scale corpus for assessing written argumentation: PERSUADE 2.0
Crossley, S. A., Tian, Y., Baffour, P., et al. (2024) · Assessing Writing: An International Journal · doi:10.1016/j.asw.2024.100865 · Cited by 62 · cited 2×
“Commercial AI detectors such as TurnItIn have demonstrated inadequate accuracy for the context of academic plagarism - primarily due to high false positive rates (Vanderbilt, 2023).”
BackgroundMethodSmaller Language Models are Better Black-box Machine-Generated Text DetectorsPreprint
Mireshghallah, F., Mattern, J., Gao, S.-C., Shokri, R., & Berg-Kirkpatrick, T. (2023) · arXiv.org · doi:10.48550/arxiv.2305.09859 · Cited by 57 · cited 2× · see citing works
“For DetectGPT, we evaluate it in the most favorable black box setting for ChatGPT, using the best performing model from the large-scale study of cross detection (Mireshghallah et al., 2023), with 100 perturbations from T5-3B and using OPT-125M as the cross-detector.”
BackgroundMethodA Survey of Deep Active LearningPreprint
Ren, P., Yun, X., Chang, X., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.00236 · Cited by 38 · see citing works
Chakraborty, M., Tonmoy, S., Zaman, S. M. M., Sharma, K., Barman, N. R., Gupta, C., Gautam, S., Kumar, T., Jain, V., Chadha, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2310.05030 · Cited by 35 · cited 1× · see citing works
“According to (Chakraborty et al., 2023), the GPT-4 family of models is the most difficult family to classify based on probability-based features, showing the limitations of the probability-based approaches to detect increasingly capable LLM output.”
BackgroundXL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 LanguagesPreprint
Hasan, T., Bhattacharjee, A., Islam, M. S., Samin, K., Li, Y.-F., Kang, Y., Rahman, M. S., & Shahriyar, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13822 · Cited by 32 · see citing works
On the Zero-Shot Generalization of Machine-Generated Text DetectorsPreprint
Pu, X., Zhang, J. (., Han, X.-C., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2310.05165 · Cited by 24 · cited 1× · see citing works
“A recent study (Pu et al., 2023) has shown that deep learning based AI-generated text classifiers exhibit some generalization ability to detect text from larger models when trained on outputs of smaller models from the same family.”
Background
Flagged first, then by citations
Research works printed but not linked: 1
- S. Ishikawa. The ICNALE Guide: An Introduction to a Learner Corpus Study on Asian Learners' L2 English. Routledge, 2023.
Cited by
Sorted by citations
AI tool detects LLM-generated text in research papers and peer reviews
Naddaf, M. (2025) · Nature · doi:10.1038/d41586-025-02936-6 · Cited by 7 · see citing works
Fine-Grained Detection of AI-Generated Writing in the Biomedical Literature
She, R. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.01.01.697311 · Cited by 4 · see citing works
“…99–99.8% accuracy with false-positive rates on the order of 0.01–0.1% across domains, including in scientific writing and essays by non-native English speakers, and remains highly effective even against adversarial “humanizer” paraphrases that are explicitly optimized to evade detection 8 .”
InfluentialStoryScope: Investigating idiosyncrasies in AI fiction
Russell, J., Rajendhran, R., Pham, C. M., Iyyer, M., & Wieting, J. (2026) · arXiv · doi:10.48550/arxiv.2604.03136
From Planning to Revision: How AI Writing Support at Different Stages Alters Ownership
Gero, K. I., Long, T., Schnitzler, C., & Dhillon, P. S. (2026) · Designing Interactive Systems Conference · doi:10.1145/3800645.3813003 · Cited by 2 · see citing works
“(2) Coherence and cohesion – how logically the essay flows and how ideas are connected.”
EditLens: Quantifying the Extent of AI Editing in Text
Thai, K., Emi, B., Masrour, E., & Iyyer, M. (2025) · arXiv · doi:10.48550/arxiv.2510.03154 · Cited by 3
“Ghostbusters (Verma et al., 2024) is an open-weight classifier trained on simple features from the text, while closed-source classifiers such as GPTZero (Tian & Cui, 2023) and Pangram (Emi & Spero, 2024) have more recently emerged as accurate AI text classifiers.”
InfluentialSelf-citation
- Authors
- Emi, B., & Spero, M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2402.14873
- OpenAlex
- W4392221720
- Cited by
- 2
- References
- 19
- Citations by year
- 2026: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- AI-generated text detection, large language models, Transformer classifier, hard negative mining, zero-shot detection, false positive reduction, non-native English speakers, domain generalization, student writing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
