CoChat
Sign Up
Cited by 49Authorship Attribution and ProfilingNatural Language Processing TechniquesTopic Modeling

RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors

Open access
Dugan, Liam, Hwang, Alyssa, Trhlík, Filip, Zhu, Andrew, Ludan, Josh Magnus, Xu, Hainiu, Ippolito, Daphne, Callison-Burch, Chris·2024·Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/2024.acl-long.674
Open this paper in CoChat — free accountOpen PDF

Abstract

Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, Chris Callison-Burch. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.acl-long.674

Cites

0 citation errors · 27 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing detection methods are mostly binary classifiers and cannot distinguish complex LLM-generated content such as mixed human-LLM input.

    “current methods often focus on binary classification, failing to address the complexities of realworld scenarios like human-LLM collaboration.”

    From Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement (2025) · cites this paper as [18]

    Section 5.2, Detector Evaluation (page unavailable), evaluates detectors by converting their scores into binary human-written versus machine-generated predictions. The Conclusion reports that detectors lack robustness in several settings, but the paper does not establish that they fail specifically on mixed human–LLM writing or collaboration. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 10

  1. Aaditya Bhat. 2023. Gpt-wiki-intro. · DOI
  2. Rinaldo Gagiano, Maria Myung-Hee Kim, Xiuzhen Zhang, and Jennifer Biggs. 2021. Robustness analy- sis of grover for machine-generated news detection. In Proceedings of the The 19th Annual Workshop of the Australasian Language Technology Associa- tion, pages 119-127, Online. Australasian Language Technology Association.
  3. Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2023. Radar: Robust ai-text detection via adversarial learn- ing. Advances in Neural Information Processing Systems.
  4. Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. 2022. The stack: 3 tb of permissively licensed source code. Preprint.
  5. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
  6. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. Detectgpt: zero-shot machine-generated text detec- tion using probability curvature. In Proceedings of the 40th International Conference on Machine Learn- ing, ICML'23. JMLR.org.
  7. Artidoro Pagnoni, Martin Graciarena, and Yulia Tsvetkov. 2022. Threat scenarios and best practices to detect neural fake news. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 1233-1249, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  8. Sayak Paul and Soumik Rakshit. 2021. arxiv paper abstracts. https://www.kaggle.com/datasets/ spsayakpaul/arxiv-paper-abstracts.
  9. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-5551.
  10. Dietmar Schabus, Marcin Skowron, and Martin Trapp. 2017. One million posts: A data set of german on- line discussions. In Proceedings of the 40th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, page 1241-1244, New York, NY, USA. Association for Computing Machinery. · DOI

Cited by

Sorted by citations

Related

Authors
Dugan, L., Hwang, A., Trhlík, F., Zhu, A., Ludan, J. M., Xu, H., Ippolito, D., & Callison-Burch, C.
Affiliations
  • University College London (GB)
  • King's College London (GB)
  • Carnegie Mellon University (US)
ORCID
Venue
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2024
Pages
12463-12492
ISSN
0736-587X (ISSN-L)
DOI
10.18653/v1/2024.acl-long.674
OpenAlex
W4402672003
Cited by
48
References
0
Citations by year
2026: 32 · 2025: 15 · 2024: 1
FWCI
14.16
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
Funders
  • Office of the Director of National Intelligence
  • Intelligence Advanced Research Projects Activity
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
AI-generated text detection, benchmark, robustness analysis, synthetic data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review