AuthentiGPT: Detecting Machine-Generated Text via Black-Box Language Models Denoising
Open accessAbstract
Large language models (LLMs) have opened up enormous opportunities while simultaneously posing ethical dilemmas. One of the major concerns is their ability to create text that closely mimics human writing, which can lead to potential misuse, such as academic misconduct, disinformation, and fraud. To address this problem, we present AuthentiGPT, an efficient classifier that distinguishes between machine-generated and human-written texts. Under the assumption that human-written text resides outside the distribution of machine-generated text, AuthentiGPT leverages a black-box LLM to denoise input text with artificially added noise, and then semantically compares the denoised text with the original to determine if the content is machine-generated. With only one trainable parameter, AuthentiGPT eliminates the need for a large training dataset, watermarking the LLM's output, or computing the log-likelihood. Importantly, the detection capability of AuthentiGPT can be easily adapted to any generative language model. With a 0.918 AUROC score on a domain-specific dataset, AuthentiGPT demonstrates its effectiveness over other commercial algorithms, highlighting its potential for detecting machine-generated text in academic settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Guo and Yu (2023) introduced a denoising-based black-box zero-shot statistics method that employs a black-box LLM to denoise artificially added noise to input texts, resulting in an AUROC score of 91.8%.
“Guo and Yu (2023) introduced a denoising-based black-box zero-shot statistics method that employs a black-box LLM to denoise artificially added noise to input texts. The denoised texts are then semantically compared to the original texts, resulting in an AUROC score of 91.8%.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Guo and Yu 2023
The AUROC Scores passage reports the black-box denoising and semantic-comparison method and gives AuthentiGPT an AUROC of 0.918 on the aggregated PubMedQA and GPT-generated QA datasets. However, it says the classifier is trained on a small set of examples; the methods section specifies training samples. Calling the method zero-shot is therefore not supported. (LLM, conf=0.89)
Experimental — accuracy not yet measuredDetectors might struggle to identify text generated with complex prompts.
“For instance, the study by Guo et al. (2023) demonstrates that detectors might struggle to identify text generated with complex prompts.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Guo et al. (2023)
The Results section (page unavailable) reports that GPTZero and AuthentiGPT had more difficulty identifying GPT-rewritten text than newly generated question-answer pairs. It suggests that retained human writing style and meaning may explain the difference; it does not test prompt complexity or show that complex prompts cause the difficulty. (LLM, conf=0.84)
Experimental — accuracy not yet measuredHC3-zh encompasses 22k human answers and 17k ChatGPT responses.
“On the other hand, HC3-zh encompasses a broader spectrum of domains, featuring 22k human answers and 17k ChatGPT responses.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Guo et al. 2023
Section 4.1 (Dataset; page unavailable) describes a PubMedQA-based dataset, not HC3-zh. Searches for HC3-zh and the stated human-answer and ChatGPT-response counts found no passage in our copy that addresses those figures. (LLM, conf=0.76)
Experimental — accuracy not yet measuredHC3 has been utilized in numerous subsequent studies as a valuable resource.
“Due to its pioneering contributions in this field, the HC3 corpus has been utilized in numerous subsequent studies as a valuable resource.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Guo et al. 2023
The Dataset section (4.1; page unavailable) describes use of PubMedQA, not HC3. Searches for HC3 and for a human–ChatGPT comparison corpus found no passage addressing whether HC3 was used in numerous subsequent studies. (LLM, conf=0.74)
Experimental — accuracy not yet measuredHC3 offers datasets in both English and Chinese.
“The corpus offers datasets in both English and Chinese.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Guo et al. 2023
Section 4.1 (Dataset; page unavailable) describes PubMedQA-based datasets, but does not discuss HC3 or their languages. Searches for “HC3 English Chinese” and “corpus datasets in both English and Chinese” found no passage addressing the claim. (LLM, conf=0.75)
Experimental — accuracy not yet measured
HC3 involves collecting both human and ChatGPT responses to identical questions.
“It involves collecting both human and ChatGPT responses to identical questions.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Guo et al. 2023
Section 4.1 (page unavailable) describes human-written PubMedQA question–answer pairs and GPT-generated pairs, but does not identify this dataset as HC3. Searches for “HC3” and for human and ChatGPT responses to identical questions found no passage describing HC3’s collection process. (LLM, conf=0.75)
Experimental — accuracy not yet measuredHC3 dataset has a limitation: lack of diversity in prompts used for data creation.
“However, it is pertinent to note some limitations of the HC3 dataset, such as the lack of diversity in prompts used for data creation.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Guo et al. 2023
Section 4.1 (Dataset; page unavailable) describes a PubMedQA-based dataset and its two sets of generation instructions. Section 6 (Limitations and Future Work; page unavailable) discusses limitations of AuthentiGPT, but neither section addresses HC3 or a lack of diversity in prompts used to create it. Searches for HC3 and prompt diversity found no passage supporting the claim. (LLM, conf=0.74)
Experimental — accuracy not yet measuredHC3-en questions primarily originate from ELI5, WikiQA, Crawled Wikipedia, Medical Dialog, and FiQA.
“Specifically, HC3-en comprises 58k human responses and 26k ChatGPT responses, derived from 24k questions, which primarily originate from the ELI5 dataset, WikiQA dataset, Crawled Wikipedia, Medical Dialog dataset, and FiQA dataset.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Guo et al. 2023
Section 4.1 (Dataset; page unavailable) describes a PubMedQA-based evaluation dataset, not HC3-en. Searches for HC3-en, HC3, the named source datasets, and the response and question counts found no passage addressing where HC3-en questions originate. (LLM, conf=0.74)
Experimental — accuracy not yet measuredHC3 (Guo et al. 2023) is one of the initial open-source efforts to compare ChatGPT-generated text with human-written text.
“The Human ChatGPT Comparison Corpus (HC3) (Guo et al. 2023) stands as one of the initial open-source efforts to compare ChatGPT-generated text with human-written text.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Guo et al. 2023
The Dataset section describes a PubMedQA-based comparison of human-written and GPT-generated text, not HC3. Searches for “HC3,” “Human ChatGPT Comparison Corpus,” and the claimed early open-source comparison found no passage addressing HC3 or its status among early efforts. (LLM, conf=0.76)
Experimental — accuracy not yet measuredHC3-zh data spans seven sources: WebTextQA, BaikeQA, Crawled BaiduBaike, NLPCC-DBQA, Medical Dialog, Baidu AI Studio, and LegalQA.
“The data within HC3zh spans seven sources: WebTextQA, BaikeQA, Crawled BaiduBaike, NLPCC-DBQA dataset, Medical Dialog dataset, Baidu AI Studio, and the LegalQA dataset.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Guo et al. 2023
Section 4.1 (Dataset; page unavailable) describes PubMedQA and GPT-generated question–answer pairs, not HC3-zh. Searches for HC3-zh, the seven named sources, and Chinese dataset sources found no passage addressing the claim. (LLM, conf=0.75)
Experimental — accuracy not yet measuredHC3-en comprises 58k human responses and 26k ChatGPT responses, derived from 24k questions.
“Specifically, HC3-en comprises 58k human responses and 26k ChatGPT responses, derived from 24k questions, which primarily originate from the ELI5 dataset, WikiQA dataset, Crawled Wikipedia, Medical Dialog dataset, and FiQA dataset.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Guo et al. 2023
Section 4.1 (Dataset; page unavailable) describes PubMedQA-based question–answer pairs, not HC3-en. Searches for HC3-en, the response and question counts, and the named source datasets found no passage in our copy addressing the claim. (LLM, conf=0.76)
Experimental — accuracy not yet measuredThe detection capability of detectors is directly proportional to the size of the fine-tuned LMs.
“From the perspective of the size of the supervised classifiers, the detection capability of the detectors is directly proportional to the size of the fine-tuned LMs (Guo et al. 2023).”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Guo et al. 2023
Section 2.1 discusses language-model-based classifiers but does not say that detection capability increases in proportion to the size of a fine-tuned model; it notes that such classifiers may struggle with larger models. Searches for classifier size and detection performance found no passage establishing the claimed relationship. The indexed passages provide section labels but no page numbers. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
Krishna, K., Song, Y., Karpinska, M., Wieting, J., & Iyyer, M. (2023) · arXiv · doi:10.48550/arxiv.2303.13408 · Cited by 150 · cited 1× · see citing works
“However, this technique needs a balance between imperceptibility and detectability, while also addressing potential attacks on the watermarks to ensure the integrity of the detection process [23].”
Flagged first, then by citations
Research works printed but not linked: 31
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roes- ner, and Yejin Choi. Defending against neural fake news. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Informa- tion Processing Systems, volume 32.
- Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808-1822, Online, July 2020. Association for Computational Linguistics.
- Evan Crothers, Nathalie Japkowicz, and Herna Viktor. Machine generated text: A compre- hensive survey of threat models and detection methods. arXiv preprint arXiv:2210.07321, 2022. · arXiv
- Yogesh K Dwivedi, Nir Kshetri, Laurie Hughes, Emma Louise Slade, Anand Jeyaraj, Arpan Ku- mar Kar, Abdullah M Baabdullah, Alex Koohang, Vishnupriya Raghavan, Manju Ahuja, et al. "so what if chatgpt wrote it?" multidisciplinary perspectives on opportunities, challenges and implications of generative conversational ai for research, practice and policy. International Journal of Information Management, 71:102642, 2023.
- Michael Liebrenz, Roman Schleifer, Anna Buadze, Dinesh Bhugra, and Alexander Smith. Generating scholarly content with chatgpt: ethical challenges for medical publishing. The Lancet Digital Health, 5(3):e105-e106, 2023.
- Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. Mgtbench: Bench- marking machine-generated text detection. arXiv preprint arXiv:2303.14822, 2023. · arXiv
- Sameer Badaskar, Sachin Agarwal, and Shilpa Arora. Identifying real or fake articles: Towards better language modeling. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-II, 2008.
- Daria Beresneva. Computer-generated text detection using machine learning: A systematic re- view. In Natural Language Processing and Information Systems: 21st International Conference on Applications of Natural Language to Information Systems, NLDB 2016, Salford, UK, June 22-24, 2016, Proceedings 21, pages 421-426. Springer, 2016.
- Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. Automatic detection of machine generated text: A critical survey. arXiv preprint arXiv:2011.01314, 2020. · arXiv
- Sandra Mitrović, Davide Andreoletti, and Omran Ayoub. Chatgpt or human? detect and explain. explaining decisions of machine learning model for detecting short chatgpt-generated text. arXiv preprint arXiv:2301.13852, 2023. · arXiv
- Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.11156, 2023. · arXiv
- Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. On the possibilities of ai-generated text detection. arXiv preprint arXiv:2304.04736, 2023. · arXiv
- Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. The science of detecting llm-generated texts. arXiv preprint arXiv:2303.07205, 2023. · arXiv
- Fabrizio Sebastiani. Machine learning in automated text categorization. ACM computing surveys (CSUR), 34(1):1-47, 2002.
- Niful Islam, Debopom Sutradhar, Humaira Noor, Jarin Tasnim Raya, Monowara Tabassum Maisha, and Dewan Md Farid. Distinguishing human generated text from chatgpt generated text using machine learning. arXiv preprint arXiv:2306.01761, 2023. · arXiv
- Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305, 2023. · arXiv
- Zhijie Deng, Hongcheng Gao, Yibo Miao, and Hao Zhang. Efficient detection of llm-generated texts with a bayesian surrogate model. arXiv preprint arXiv:2305.16617, 2023. · arXiv
- John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint arXiv:2301.10226, 2023. · arXiv
- Travis Munyer and Xin Zhong. Deeptextmark: Deep learning based text watermarking for detection of large language model generated text. arXiv preprint arXiv:2305.05773, 2023. · arXiv
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
- Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. Gpt detectors are biased against non-native english writers. arXiv preprint arXiv:2304.02819, 2023. · arXiv
- Robert C Moore and William Lewis. Intelligent selection of language model training data. In Proceedings of the ACL 2010 conference short papers, pages 220-224, 2010.
- Christoforos Vasilatos, Manaar Alam, Talal Rahwan, Yasir Zaki, and Michail Maniatakos. Howkgpt: Investigating the detection of chatgpt-generated university student homework through context-aware perplexity analysis. arXiv preprint arXiv:2305.18226, 2023. · arXiv
- Xianjun Yang, Wei Cheng, Linda Petzold, William Yang Wang, and Haifeng Chen. Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text. arXiv preprint arXiv:2305.17359, 2023. · arXiv
- Mark S. Graham, Walter H.L. Pinaya, Petru-Daniel Tudosiu, Parashkev Nachev, Sebastien Ourselin, and Jorge Cardoso. Denoising diffusion models for out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 2947-2956, June 2023.
- Remi M Sakia. The box-cox transformation technique: a review. Journal of the Royal Statistical Society: Series D (The Statistician), 41(2):169-178, 1992.
- Carl Rasmussen. The infinite gaussian mixture model. Advances in neural information processing systems, 12, 1999.
- Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741(659-663), 2009.
- Elizabeth R DeLong, David M DeLong, and Daniel L Clarke-Pearson. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, pages 837-845, 1988.
- Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146, 2019. · arXiv
- Zhen Guo, Peiqi Wang, Yanwei Wang, and Shangdi Yu. Dr. llama: Improving small language models in domain-specific qa via generative data augmentation. arXiv preprint arXiv:2305.07804, 2023. · arXiv
Cited by
Sorted by citations
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212 · see citing works
- Authors
- Guo, Z., & Yu, S.
- Affiliations
- Massachusetts Institute of Technology (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2311.07700
- OpenAlex
- W4388717809
- Cited by
- 4
- References
- 1
- Citations by year
- 2025: 1 · 2024: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- AI-generated text detection, semantic comparison, LLM watermarking, academic misconduct detection, disinformation, generative AI misuse
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
