CoChat
Sign Up
PreprintCited by 73Authorship Attribution and ProfilingDigital Media Forensic DetectionArtificial Intelligence in Education

Classification of human- and AI-generated texts: Investigating features for ChatGPT

Open access
Mindner, Lorenz, Schlippe, Tim, Schaaff, Kristina·2023·arXiv
DOI 10.48550/arxiv.2308.05341arXiv 2308.05341
Open this paper in CoChat — free accountOpen PDF

Abstract

Recently, generative AIs like ChatGPT have become available to the wide public. These tools can for instance be used by students to generate essays or whole theses. But how does a teacher know whether a text is written by a student or an AI? In our work, we explore traditional and new features to (1) detect text generated by AI from scratch and (2) text rephrased by AI. Since we found that classification is more difficult when the AI has been instructed to create the text in a way that a human would not recognize that it was generated by an AI, we also investigate this more advanced case. For our experiments, we produced a new text corpus covering 10 school topics. Our best systems to classify basic and advanced human-generated/AI-generated texts have F1-scores of over 96%. Our best systems for classifying basic and advanced human-generated/AI-rephrased texts have F1-scores of more than 78%. The systems use a combination of perplexity, semantic, list lookup, error-based, readability, AI feedback, and text vector features. Our results show that the new features substantially help to improve the performance of many classifiers. Our best basic text rephrasing detection system even outperforms GPTZero by 183.8% relative in F1-score.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2308.05341

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Mindner, Schlippe, and Schaaff (2023) employed a multidimensional approach to enhance classifier's discriminative power, and the optimized detector's performance exceeded that of GPTZero by 183.8% in F1 Score.

    “Mindner, Schlippe, and Schaaff (2023) further employed a multidimensional approach to enhance the classifier's discriminative power, which included complexity measures, semantic analysis, list searches, error-based features, readability assessments, artificial intelligence feedback, and text vector features. Ultimately, the optimized detector's performance exceeded that of GPTZero by 183.8% in F1”

    From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Mindner, Schlippe, and Schaaff 2023

    The Experiments and Results section reports that the best system outperformed GPTZero by 183.8% relative in F1, but specifically on the basic text rephrasing task. The paper also evaluates the listed feature categories; the citing sentence omits the task condition and makes the performance figure sound general. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 24 references couldn’t be matched to a known paper.

Research works printed but not linked: 7

  1. Adiwardana, D., Luong, M.-T., So, D.R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., Le, Q.V.: Towards a Human-Like Open-Domain Chatbot. ArXiv Preprint ArXiv:2001.09977 (2020) · arXiv
  2. Falala-Séchet, C., Antoine, L., Thiriez, I., Bungener, C.: OWLIE: A Chatbot that Provides Emotional Support for Coping With Psychological Difficulties. In: Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, pp. 236-237 (2019)
  3. Jiao, W., Wang, W., Huang, J.-t., Wang, X., Tu, Z.: Is ChatGPT a Good Translator? A Preliminary Study. ArXiv Preprint ArXiv:2301.08745 (2023) · arXiv
  4. Kenton, J.D.M.-W.C., Toutanova, L.K.: BERT: Pre-training of Deep Bidirec- tional Transformers for Language Understanding. In: Proceedings of NAACL- HLT, pp. 4171-4186 (2019)
  5. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs/1907.11692 (2019) 1907.11692 · arXiv
  6. Mitrović, S., Andreoletti, D., Ayoub, O.: ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-Generated Text. arXiv preprint arXiv:2301.13852 (2023) · arXiv
  7. Bird, S., Loper, E.: NLTK: The Natural Language Toolkit. In: Proceedings of the ACL Interactive Poster and Demonstration Sessions, pp. 214-217. Association for Computational Linguistics, Barcelona, Spain (2004). https://aclanthology.org/ P04-3031

Cited by

Sorted by citations

Related

Authors
Mindner, L., Schlippe, T., & Schaaff, K.
Affiliations
IU International University of Applied Sciences (DE)
Corresponding author
Schlippe, Tim
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
152-170
ISSN
2367-4512 (ISSN-L)
DOI
10.48550/arxiv.2308.05341
arXiv
2308.05341
OpenAlex
W4388491879
Cited by
73
References
24
Citations by year
2026: 21 · 2025: 35 · 2024: 17
FWCI
12.96
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes on data engineering and communications technologies (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
AI-generated text detection, ChatGPT, readability features, generative AI in education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review