The DOI matches its DataCite record; 28 of the 28 references it cites were found in a registry or an index.
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
Open accessAbstract
The increasing fluency and widespread usage of large language models (LLMs) highlight the desirability of corresponding tools aiding detection of LLM-generated text. In this paper, we identify a property of the structure of an LLM's probability function that is useful for such detection. Specifically, we demonstrate that text sampled from an LLM tends to occupy negative curvature regions of the model's log probability function. Leveraging this observation, we then define a new curvature-based criterion for judging if a passage is generated from a given LLM. This approach, which we call DetectGPT, does not require training a separate classifier, collecting a dataset of real or generated passages, or explicitly watermarking generated text. It uses only log probabilities computed by the model of interest and random perturbations of the passage from another generic pre-trained language model (e.g., T5). We find DetectGPT is more discriminative than existing zero-shot methods for model sample detection, notably improving detection of fake news articles generated by 20B parameter GPT-NeoX from 0.81 AUROC for the strongest zero-shot baseline to 0.95 AUROC for DetectGPT. See https://ericmitchell.ai/detectgpt for code, data, and other project information.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“After paraphrasing, the area under the receiver operating characteristic (AUROC) curves of zero-shot detectors (Mitchell et al., 2023) drops from 96 . 5% to 25 . 2%.”
From Can AI-Generated Text be Reliably Detected? (2023) · cites this paper as Mitchell et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In particular, the detector LLM is not given explicit access to information such as perplexity, which in many detection methods is a crucial component (Mitchell et al., 2023;Hans et al., 2024).”
From LLM Evaluators Recognize and Favor Their Own Generations (2024) · cites this paper as Mitchell et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The AUROC scores of DetectGPT (Mitchell et al., 2023) drop from 96 . 5 % (before the attack) to 59 . 8 % (after the attack).”
From Can AI-Generated Text be Reliably Detected? (2023) · cites this paper as Mitchell et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Mireshghallah et al. [110] experimented to test whether other language models can be used to detect machine-generated text from one language model and found out that smaller language models like OPT-125M are better AIGC detectors than large language models like GPTJ-6B.”
From Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity (2024) · cites this paper as [110]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, after altering the content using an Automated Paraphrasing tool (APT), the detection rate decreased to 4.6%.”
From Simple techniques to bypass GenAI text detectors: implications for inclusive education (2024) · cites this paper as Mitchell et al. (2023)
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Mitchell et al. (2023) identified that DetectGPT correctly detected 70.3% of model-generated sequences from GPT2-XL.”
From Simple techniques to bypass GenAI text detectors: implications for inclusive education (2024) · cites this paper as Mitchell et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These detectors require retraining when encountering text generated by new LLMs to ensure reliable detection [Mitchell et al., 2023;Chakraborty et al., 2023b].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Mitchell et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, these methods remain limited in their precision and scope.”
From On the Risk of Misinformation Pollution with Large Language Models (2023) · cites this paper as Mitchell et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Mitchell et al. (2023) observe that AI-generated passages tend to lie in negative curvature of log probability of texts.”
From Can AI-Generated Text be Reliably Detected? (2023) · cites this paper as Mitchell et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“The majority of previous research on automatic detection of machine-generated text has focused on binary classification [Jawahar et al., 2020b;Mitchell et al., 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Mitchell et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“To effectively analyze and classify human-LLM co-authored texts versus those solely authored by humans or LLMs, feature-based methods from the problem of LLM-generated text detection, such as Log-likelihood [Solaiman et al., 2019], GLTR [Gehrmann et al., 2019a], and log-rank [Mitchell et al., 2023], are adapted to this task.”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Mitchell et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Models like DNA-GPT [Yang et al., 2023a] and DetectGPT [Mitchell et al., 2023] utilized the T5 model [Raffel et al., 2020] to simulate scenarios in which humans modify LLMgenerated texts.”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Mitchell et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 9
- Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901.
- Curran Asso- ciates, Inc., 2020. URL https://proceedings. neurips.cc/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper. pdf.
- Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30.
- Curran Associates, Inc., 2017. URL https://proceedings.neurips. cc/paper_files/paper/2017/file/ d5e2c0adad503c91f91df240d0cd4e49-Paper. pdf.
- Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., and Goldstein, T. A watermark for large language mod- els, 2023. URL https://arxiv.org/abs/2301. 10226.
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Ex- ploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http://jmlr. org/papers/v21/20-074.html.
- Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert- Voss, A., Wu, J., Radford, A., and Wang, J. Release strategies and the social impacts of language models, 2019. URL https://arxiv.org/ftp/arxiv/ papers/1908/1908.09203.pdf.
- Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., and Choi, Y. Defending against neural fake news. In Neural Information Processing Systems, 2019.
- Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mi- haylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. Opt: Open pre-trained transformer language mod- els, 2022. URL https://arxiv.org/abs/2205. 01068.
Cited by
Sorted by citations
Regulating ChatGPT and other Large Generative AI Models
Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466
“This should be coupled with research on AI-content detection to highlight such output where watermarks fail [121, 140].”
BackgroundA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Large AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B. J., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 267
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258
“The randomness in LLMs typically comes from the sampling methods used during text generation, such as top-k sampling or nucleus sampling [31, 49].”
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212
Metadata
- Authors
- Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., & Finn, C.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2301.11305
- OpenAlex
- W4318351452
- Cited by
- 151
- References
- 0
- Citations by year
- 2026: 7 · 2025: 32 · 2024: 51 · 2023: 60
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Canadian Institute for Advanced Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- AI-generated text detection, large language models, zero-shot detection, fake news detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
