Cited by 53Natural Language Processing TechniquesAuthorship Attribution and ProfilingTopic Modeling

Multitude: Large-scale multilingual machine-generated text detection benchmark

Open access
Macko, Dominik, Moro, Robert, Uchendu, Adaku, Lucas, Jason, Yamashita, Michiharu, Pikuliak, Matúš, Srba, Ivan, Le, Thai +3 more·2023·Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

Abstract

Dominik Macko, Robert Moro, Adaku Uchendu, Jason Lucas, Michiharu Yamashita, Matúš Pikuliak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, Maria Bielikova. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2023.emnlp-main.616

Cites

23 of 26 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • MULTITuDE consists of 74k machine-generated texts and 7k human-written texts across 11 languages.

    “This dataset consists of 74k machine-generated texts and 7k human-written texts across 11 languages (Macko et al. 2023), including Arabic, Catalan, Chinese, Czech, Dutch, English, German, Portuguese, Russian, Spanish, and Ukrainian.”

    From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Macko et al. 2023

    The Abstract (page unavailable) confirms that MULTITuDE covers the 11 listed languages, but says 74,081 is the total number of human-written and machine-generated texts—not the number of machine-generated texts alone. The human-text count of 7k is not established by the cited passages; Appendix B.1 (page unavailable) gives per-language sampling figures rather than that total. (LLM, conf=0.90)

    Experimental — accuracy not yet measured
  • MULTITuDE's machine-generated texts are produced by eight generative models, including Alpaca-Lora, GPT-3.5-turbo, GPT-4, LLaMA, OPT, OPT-IML-Max, Text-Davinci-003, and Vicuna.

    “The machinegenerated texts are produced by eight generative models, including Alpaca-Lora, GPT3.5-turbo, GPT-4, LLaMA, OPT, OPT-IML-Max, Text-Davinci-003, and Vicuna.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 20

  1. Evan Crothers, Nathalie Japkowicz, Herna Viktor, and Paula Branco. 2022b. Adversarial robustness of neural-statistical features in detection of generative transformers. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE.
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR, abs/1810.04805. · arXiv
  3. Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Callison-Burch. 2022. Real or fake text?: Investigating human ability to de- tect boundaries between human-written and machine- generated text. arXiv preprint arXiv:2212.12672. · arXiv
  4. Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, An- tonio Martella, and Maurizio Tesconi. 2021. Tweep- fake: About detecting deepfake tweets. Plos one, 16(5):e0251415.
  5. Rinaldo Gagiano, Maria Myung-Hee Kim, Xi- uzhen Jenny Zhang, and Jennifer Biggs. 2021. Ro- bustness analysis of grover for machine-generated news detection. In Proceedings of the The 19th An- nual Workshop of the Australasian Language Tech- nology Association, pages 119-127.
  6. Matthias Gallé, Jos Rozen, Germán Kruszewski, and Hady Elsahar. 2021. Unsupervised and distributional detection of machine-generated text. arXiv preprint arXiv:2111.02878. · arXiv
  7. Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.00027. · arXiv
  8. Sebastian Gehrmann, Hendrik Strobelt, and Alexan- der M Rush. 2019. Gltr: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics: System Demonstrations, pages 111-116.
  9. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021b. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
  10. Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023. Mgtbench: Benchmarking machine-generated text detection. arXiv preprint arXiv:2303.14822. · arXiv
  11. Tharindu Kumarage, Joshua Garland, Amrita Bhat- tacharjee, Kirill Trapeznikov, Scott Ruston, and Huan Liu. 2023. Stylometric detection of ai- generated text in twitter timelines. arXiv preprint arXiv:2303.03697. · arXiv
  12. Laida Kushnareva, Daniil Cherniavskii, Vladislav Mikhailov, Ekaterina Artemova, Serguei Barannikov, Alexander Bernstein, Irina Piontkovskaya, Dmitri Piontkovski, and Evgeny Burnaev. 2021. Artificial text detection via examining the topology of atten- tion maps. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 635-649.
  13. Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. 2023. Deepfake text detection in the wild. arXiv preprint arXiv:2305.13242. · arXiv
  14. Xiaoming Liu, Zhaohan Zhang, Yichen Wang, Yu Lan, and Chao Shen. 2022. Coco: Coherence-enhanced machine-generated text detection under data lim- itation with contrastive learning. arXiv preprint arXiv:2212.10341. · arXiv
  15. Yikang Liu, Ziyin Zhang, Wanyang Zhang, Shisen Yue, Xiaojing Zhao, Xinyuan Cheng, Yiwen Zhang, and Hai Hu. 2023. Argugpt: evaluating, understanding and identifying argumentative essays generated by gpt models. arXiv preprint arXiv:2304.07666. · arXiv
  16. Vijini Liyanage, Davide Buscaldi, and Adeline Nazarenko. 2022. A benchmark corpus for the de- tection of automatically generated text in academic publications. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 4692-4700, Marseille, France. European Language Resources Association.
  17. Artidoro Pagnoni, Martin Graciarena, and Yulia Tsvetkov. 2022. Threat scenarios and best practices to detect neural fake news. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 1233-1249.
  18. Jiashu Pu, Ziyi Huang, Yadong Xi, Guandan Chen, Wei- jie Chen, and Rongsheng Zhang. 2022b. Unraveling the mystery of artifacts in machine generated text. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6889-6898.
  19. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  20. Adaku Uchendu, Thai Le, and Dongwon Lee. 2023b. TopRoBERTa: Topology-aware author- ship attribution of deepfake texts. arXiv preprint arXiv:2309.12934. · arXiv

Cited by

Sorted by citations

Authors
Macko, D., Moro, R., Uchendu, A., Lucas, J., Yamashita, M., Pikuliak, M., Srba, I., Le, T., Lee, D., Simko, J., & Bielikova, M.
Affiliations
  • MIT Lincoln Laboratory (US)
  • Pennsylvania State University (US)
  • University of Mississippi (US)
Venue
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Published
2023
Pages
9960-9987
OpenAlex
W4389520338
Cited by
38
References
22
Citations by year
2026: 13 · 2025: 17 · 2024: 8
FWCI
6.73
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (1820609, 1934782, 2114824, 2131144)
  • European Commission (101070093, 101073921)
  • HORIZON EUROPE Framework Programme
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
AI-generated text detection, multilingual text detection, benchmark, large-scale benchmark, multilingual NLP

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review