CoChat
Sign Up
PreprintCited by 22Authorship Attribution and ProfilingTopic ModelingDigital Media Forensic Detection

LLM-Detector: Improving AI-Generated Chinese Text Detection with Open-Source LLM Instruction Tuning

Open access
Wang, Rongsheng, Chen, Haoming, Zhou, Ruizhe, Ma, Han, Duan, Yaofei, Kang, Yanlan, Yang, Songhua, Fan, Baoyu +1 more·2024·arXiv
DOI 10.48550/arxiv.2402.01158arXiv 2402.01158
Open this paper in CoChat — free accountOpen PDF

Abstract

ChatGPT and other general large language models (LLMs) have achieved remarkable success, but they have also raised concerns about the misuse of AI-generated texts. Existing AI-generated text detection models, such as based on BERT and RoBERTa, are prone to in-domain over-fitting, leading to poor out-of-domain (OOD) detection performance. In this paper, we first collected Chinese text responses generated by human experts and 9 types of LLMs, for which to multiple domains questions, and further created a dataset that mixed human-written sentences and sentences polished by LLMs. We then proposed LLM-Detector, a novel method for both document-level and sentence-level text detection through Instruction Tuning of LLMs. Our method leverages the wealth of knowledge LLMs acquire during pre-training, enabling them to detect the text they generate. Instruction tuning aligns the model's responses with the user's expected text detection tasks. Experimental results show that previous methods struggle with sentence-level AI-generated text detection and OOD detection. In contrast, our proposed method not only significantly outperforms baseline methods in both sentence-level and document-level text detection but also demonstrates strong generalization capabilities. Furthermore, since LLM-Detector is trained based on open-source LLMs, it is easy to customize for deployment.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.01158

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Wissam Antoun et al. implemented transformer-based algorithms such as Roberta, ELECTRA, CamemBERTa, and XLM-R on a modified Human ChatGPT Comparision Corpus (HC3) dataset containing English and French text.

    “Wissam Antoun et al. [119] implemented different transformer-based algorithms such as Roberta, ELECTRA, CamemBERTa, and XLM-R on a modified Human ChatGPT Comparision Corpus (HC3) dataset containing English and French text on which a 99.86 F-Score was achieved.”

    From Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity (2024) · cites this paper as [119]

  • Wissam Antoun et al. achieved a 99.86 F-Score on a modified HC3 dataset containing English and French text.

    “Wissam Antoun et al. [119] implemented different transformer-based algorithms such as Roberta, ELECTRA, CamemBERTa, and XLM-R on a modified Human ChatGPT Comparision Corpus (HC3) dataset containing English and French text on which a 99.86 F-Score was achieved.”

    From Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity (2024) · cites this paper as [119]

Cites

Still being checked.

Cited by

Sorted by citations

Related

Authors
Wang, R., Chen, H., Zhou, R., Ma, H., Duan, Y., Kang, Y., Yang, S., Fan, B., & Tan, T.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.01158
arXiv
2402.01158
OpenAlex
W4391555855
Cited by
5
References
0
Citations by year
2025: 2 · 2024: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
Keywords
AI-generated text detection, Chinese text, instruction tuning, large language models, out-of-domain detection, overfitting, open-source LLMs

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review