The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?
Open accessAbstract
Large language models (LLMs) such as ChatGPT are increasingly being used for various use cases, including text content generation at scale. Although detection methods for such AI-generated text exist already, we investigate ChatGPT's performance as a detector on such AI-generated text, inspired by works that use ChatGPT as a data labeler or annotator. We evaluate the zeroshot performance of ChatGPT in the task of human-written vs. AI-generated text detection, and perform experiments on publicly available datasets. We empirically investigate if ChatGPT is symmetrically effective in detecting AI-generated or human-written text. Our findings provide insight on how ChatGPT and similar LLMs may be leveraged in automated detection pipelines by simply focusing on solving a specific aspect of the problem and deriving the rest from that solution. All code and data is available at https://github.com/AmritaBh/ChatGPT-as-Detector.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This suggests that exposure to news during training misleads the judgment of zero-shot detectors, which could also explain the poor performance reported in the literature [8] when using LLMs to distinguish LLM-generated news from human-authored news.”
From Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement (2025) · cites this paper as [8]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · arXiv · doi:10.48550/arxiv.2303.15056 · Cited by 1,207
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · cited 1×
“LLMs have also been evaluated as annotators [12; 17] with recent work showing some LLMs perform at par or even out-perform human crowd workers for text annotation and question-answering [16; 15; 35].”
MethodBang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682
Defending Against Neural Fake News
Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1905.12616 · Cited by 569
How is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507
Research works printed but not linked: 20
- S. Allardice. The confident wrongness of chatgpt, 2023. Accessed on June 29, 2023.
- N. Bian, X. Han, L. Sun, H. Lin, Y. Lu, and B. He. Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models. arXiv preprint arXiv:2303.16421, 2023. · arXiv
- M. Bohanno. Lawyer used chatgpt in court-and cited fake cases. a judge is considering sanctions, 2023. Accessed on June 29, 2023.
- E. Clark, T. August, S. Serrano, N. Haduong, S. Gururangan, and N. A. Smith. All that's' human'is not gold: Evaluating human evaluation of generated text. arXiv preprint arXiv:2107.00061, 2021. · arXiv
- A. Efrat and O. Levy. The turking test: Can language models understand instructions? arXiv preprint arXiv:2010.11982, 2020. · arXiv
- B. Guo, X. Zhang, Z. Wang, M. Jiang, J. Nie, Y. Ding, J. Yue, and Y. Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597, 2023. · arXiv
- X. He, Z. Lin, Y. Gong, A. Jin, H. Zhang, C. Lin, J. Jiao, S. M. Yiu, N. Duan,W. Chen, et al. Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303.16854, 2023. · arXiv
- M. Heikkilä. Why detecting ai-generated text is so difficult (and what to do about it), 2023. Accessed on June 29, 2023.
- A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. · arXiv
- J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein. A watermark for large language models. arXiv preprint arXiv:2301.10226, 2023. · arXiv
- J. Koco´n, I. Cichecki, O. Kaszyca, M. Kochanek, J. Baran, J. Bielaniewicz, M. Gruza, A. Janz, K. Kanclerz, et al. Chatgpt: Jack of all trades, master of none. arXiv preprint arXiv:2302.10724, 2023. · arXiv
- Y. Liao, X. Jiang, and Q. Liu. Probabilistically masked language model capable of autoregressive generation in arbitrary word order. arXiv preprint arXiv:2004.11579, 2020. · arXiv
- R. OpenAI. Gpt-4 technical report. arXiv, pages 2303--08774, 2023.
- C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476, 2023. · arXiv
- A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.
- T. Ryan-Mosley. Junk websites filled with ai-generated text are pulling in money from programmatic ads, 2023. Accessed on July 1, 2023.
- V. S. Sadasivan, A. Kumar, S. Balasubramanian, W. Wang, and S. Feizi. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.11156, 2023. · arXiv
- M. Sadeghi and L. Arvanitis. Rise of the newsbots: Aigenerated news websites proliferating online, 2023. Accessed on July 1, 2023.
- R. Tan, B. A. Plummer, and K. Saenko. Detecting crossmodal inconsistency to defend against neural fake news. arXiv preprint arXiv:2009.07698, 2020. · arXiv
- X. Zhao, Y.-X. Wang, and L. Li. Protecting language generation models via invisible watermarking. arXiv preprint arXiv:2302.03162, 2023. · arXiv
Cited by
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Wong, D. F., & Chao, L. S. (2023) · arXiv · doi:10.48550/arxiv.2310.14724 · Cited by 212
“A recent reevaluation conducted by Bhattacharjee and Liu [116] on more recent LLMs like ChatGPT and GPT-4 yielded that neither could reliably identify text generated by various LLMs.”
User Modeling in the Era of Large Language Models: Current Research and Future DirectionsPreprint
Tan, Z., & Jiang, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.11518 · Cited by 80
Human vs. Machine: A Comparative Study on the Detection of AI-Generated Content
Boutadjine, A., Harrag, F., & Shaalan, K. F. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3708889 · Cited by 56
Larooij, M., & Törnberg, P. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11412-6 · Cited by 87
Metadata
- Authors
- Bhattacharjee, A., & Liu, H.
- Affiliations
- Arizona State University (US)
- Venue
- ACM SIGKDD Explorations Newsletter
- Published
- 2024-03-26
- Volume
- 25
- Issue
- 2
- Pages
- 14-21
- ISSN
- 1931-0145 (ISSN-L)
- 1931-0153
- OpenAlex
- W4393262489
- Cited by
- 65
- References
- 5
- Citations by year
- 2026: 7 · 2025: 35 · 2024: 22
- FWCI
- 9.48
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM SIGKDD Explorations Newsletter (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- U.S. Department of Defense
- Defense Advanced Research Projects Agency
- Office of Naval Research (N00014-21-1-4002, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- AI-generated text detection, ChatGPT, large language models, zero-shot performance, text generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text? https://app.cochat.ai/papers/10.1145/3655103.3655106
- The paper
- Bhattacharjee, A., & Liu, H. (2024). Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text? ACM SIGKDD Explorations Newsletter. https://doi.org/10.1145/3655103.3655106
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.