CoChat
Sign Up
PreprintCited by 5Authorship Attribution and ProfilingTopic ModelingNatural Language Processing Techniques

DEMASQ: unmasking the ChatGPT wordsmith

Open access
Kumari, Kavita, Pegoraro, Alessandro, Fereidooni, Hossein, Sadeghi, Ahmad-Reza·2023·arXiv
DOI 10.48550/arxiv.2311.05019arXiv 2311.05019
Open this paper in CoChat — free account

Abstract

The potential misuse of ChatGPT and other Large Language Models (LLMs) has raised concerns regarding the dissemination of false information, plagiarism, academic dishonesty, and fraudulent activities. Consequently, distinguishing between AI-generated and human-generated content has emerged as an intriguing research topic. However, current text detection methods lack precision and are often restricted to specific tasks or domains, making them inadequate for identifying content generated by ChatGPT. In this paper, we propose an effective ChatGPT detector named DEMASQ, which accurately identifies ChatGPT-generated content. Our method addresses two critical factors: (i) the distinct biases in text composition observed in human- and machine-generated content and (ii) the alterations made by humans to evade previous detection methods. DEMASQ is an energy-based detection model that incorporates novel aspects, such as (i) optimization inspired by the Doppler effect to capture the interdependence between input text embeddings and output labels, and (ii) the use of explainable AI techniques to generate diverse perturbations. To evaluate our detector, we create a benchmark dataset comprising a mixture of prompts from both ChatGPT and humans, encompassing domains such as medical, open Q&A, finance, wiki, and Reddit. Our evaluation demonstrates that DEMASQ achieves high accuracy in identifying content generated by ChatGPT.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2311.05019

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There is a trade-off: detectors robust against genre changes (e.g., ConDA) are poor at detecting MGT created with a model different from the one they were trained on, while detectors robust against changes in LLM (e.g., DEMASQ) fail when there is a change in genre.

    “There is a trade-off where detectors that are robust against genre changes like ConDA (Bhattacharjee et al., 2023) (F1-score when trained with news dropped from 0.99 to 0.67 when testing essays) are poor at detecting MGT created with a model different than the one it was trained on, when trained with StableLM (StableLMTuned-Alpha, n.d.) the F1-score testing Claude drops to 0.00. On the other hand,”

    From AI for scientific integrity: detecting ethical breaches, errors, and misconduct in manuscripts (2025) · cites this paper as Kumari et al., 2023

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 35

  1. Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In International Conference on Learning Representations, 2018.
  2. Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. What is relevant in a text document?: An interpretable machine learning approach. PloS one, 12(8):e0181142, 2017.
  3. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023, 2023. · arXiv
  4. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human prefer- ences. Advances in neural information processing systems, 30, 2017.
  5. Debby RE Cotton, Peter A Cotton, and J Reuben Shipway. Chatting and cheating: Ensuring academic integrity in the era of chatgpt. Innovations in Education and Teaching International, pages 1-12, 2023.
  6. Luigi De Angelis, Francesco Baglivo, Guglielmo Arzilli, Gaetano Pier- paolo Privitera, Paolo Ferragina, Alberto Eugenio Tozzi, and Caterina Rizzo. Chatgpt and the rise of large language models: The new ai-driven infodemic threat in public health. Available at SSRN 4352931, 2023.
  7. Michael Dowling and Brian Lucey. Chatgpt for (finance) research: The bananarama conjecture. Finance Research Letters, page 103662, 2023.
  8. Leon Fröhling and Arkaitz Zubiaga. Feature-based detection of au- tomated language models: tackling gpt-2, gpt-3 and grover. PeerJ Computer Science, 7:e443, April 2021. https://peerj.com/articles/cs-443/ \#supplementary-material.
  9. Catherine A Gao, Frederick M Howard, Nikolay S Markov, Emma C Dyer, Siddhi Ramesh, Yuan Luo, and Alexander T Pearson. Comparing scientific abstracts generated by chatgpt to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. bioRxiv, pages 2022-12, 2022.
  10. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043, 2019. http://gltr.io/. · arXiv
  11. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597, 2023. https://github.com/Hello-SimpleAI/ chatgpt-comparison-detection. · arXiv
  12. Philipp Hacker, Andreas Engel, and Marco Mauer. Regulating chatgpt and other large generative ai models. arXiv preprint arXiv:2302.02337, 2023. · arXiv
  13. Tim Ingold. Culture and the perception of the environment. In Bush base, forest farm, pages 38-56. Routledge, 2002.
  14. Alon Jacovi, Oren Sar Shalom, and Yoav Goldberg. Understanding convolutional neural networks for text classification. arXiv preprint arXiv:1809.08037, 2018. · arXiv
  15. Christopher HM Jenkins and Umesh A Korde. Membrane vibration experiments: An historical review and recent results. Journal of Sound and Vibration, 295(3-5):602-613, 2006.
  16. Mohammad Khalil and Erkan Er. Will chatgpt get you caught? rethinking of plagiarism detection. arXiv preprint arXiv:2302.04335, 2023. · arXiv
  17. Tharindu Kumarage, Joshua Garland, Amrita Bhattacharjee, Kirill Trapeznikov, Scott Ruston, and Huan Liu. Stylometric detection of ai-generated text in twitter timelines. arXiv preprint arXiv:2303.03697, 2023. https://github.com/TSKumarage/ Stylo-Det-AI-Gen-Twitter-Timelines. · arXiv
  18. Laida Kushnareva, Daniil Cherniavskii, Vladislav Mikhailov, Ekate- rina Artemova, Serguei Barannikov, Alexander Bernstein, Irina Pio- ntkovskaya, Dmitri Piontkovski, and Evgeny Burnaev. Artificial text detection via examining the topology of attention maps. arXiv preprint arXiv:2109.04825, 2021. https://github.com/danchern97/tda4atd. · arXiv
  19. Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
  20. Zeyan Liu, Zijun Yao, Fengjun Li, and Bo Luo. Check me if you can: Detecting chatgpt-generated academic writing using checkgpt. arXiv preprint arXiv:2306.05524, 2023. · arXiv
  21. Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st international conference on neural information processing systems, pages 4768-4777, 2017.
  22. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Man- ning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305, 2023. https://github.com/eric-mitchell/detect-gpt https://detectgpt. ericmitchell.ai/. · arXiv
  23. Sandra Mitrović, Davide Andreoletti, and Omran Ayoub. Chatgpt or human? detect and explain. explaining decisions of machine learn- ing model for detecting short chatgpt-generated text. arXiv preprint arXiv:2301.13852, 2023. · arXiv
  24. Reham Omar, Omij Mangukiya, Panos Kalnis, and Essam Mansour. Chatgpt versus traditional question answering for knowledge graphs: Current status and future directions towards knowledge graph chatbots. arXiv preprint arXiv:2302.06466, 2023. · arXiv
  25. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016.
  26. Jürgen Rudolph, Samson Tan, and Shannon Tan. Chatgpt: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1), 2023.
  27. Bernard Sapoval, Th Gobron, and A Margolina. Vibrations of fractal drums. Physical review letters, 67(21):2974, 1991.
  28. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Inter- national Conference on Machine Learning, pages 3145-3153. PMLR, 2017.
  29. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203, 2019. https://openai-openai-detector.hf.space/ https://github.com/openai/ gpt-2-output-dataset/tree/master/detector. · arXiv
  30. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319-3328. PMLR, 2017.
  31. Teo Susnjak. Applying bert and chatgpt for sentiment analysis of lyme disease in scientific literature. arXiv preprint arXiv:2302.06474, 2023. · arXiv
  32. Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. The science of detecting llm-generated texts. arXiv preprint arXiv:2303.07205, 2023. · arXiv
  33. Agnieszka Wykowska, Jasmin Kajopoulos, Miguel Obando-Leiton, Sushil Singh Chauhan, John-John Cabibihan, and Gordon Cheng. Hu- mans are well tuned to detecting agents among non-agents: examining the sensitivity of human perception to behavioral characteristics of intentional systems. International Journal of Social Robotics, 7:767- 781, 2015.
  34. Yee Hui Yeo, Jamil S Samaan, Wee Han Ng, Peng-Sheng Ting, Hirsh Trivedi, Aarshi Vipani, Walid Ayoub, Ju Dong Yang, Omer Liran, Brennan Spiegel, et al. Assessing the performance of chatgpt in answering questions regarding cirrhosis and hepatocellular carcinoma. medRxiv, pages 2023-02, 2023.
  35. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019. https://rowanzellers.com/grover/.

Cited by

Sorted by citations

Authors
Kumari, K., Pegoraro, A., Fereidooni, H., & Sadeghi, A.-R.
Affiliations
Technische Universität Darmstadt (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2311.05019
arXiv
2311.05019
OpenAlex
W4391724813
Cited by
5
References
0
Citations by year
2026: 1 · 2025: 2 · 2024: 2
FWCI
1.48
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Network and Distributed System Security Symposium (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • European Commission (101093126, 101070537)
  • Deutsche Forschungsgemeinschaft (SFB 1119, 236615297)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
ChatGPT, large language models, text generation, text forgery detection, AI-generated text

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review