CoChat

Prevalence of nonsensical algorithmically generated papers in the scientific literature

Open access
Cabanac, Guillaume, Labbé, Cyril · 2021 · Journal of the Association for Information Science and Technology
Cited by 83

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Abstract

In 2014 leading publishers withdrew more than 120 nonsensical publications automatically generated with the SCIgen program. Casual observations suggested that similar problematic papers are still published and sold, without follow‐up retractions. No systematic screening has been performed and the prevalence of such nonsensical publications in the scientific literature is unknown. Our contribution is 2‐fold. First, we designed a detector that combs the scientific literature for grammar‐based computer‐generated papers. Applied to SCIgen, it has a 83.6% precision. Second, we performed a scientometric study of the 243 detected SCIgen‐papers from 19 publishers. We estimate the prevalence of SCIgen‐papers to be 75 per million papers in Information and Computing Sciences. Only 19% of the 243 problematic papers were dealt with: formal retraction (12) or silent removal (34). Publishers still serve and sometimes sell the remaining 197 papers without any caveat. We found evidence of citation manipulation via edited SCIgen bibliographies. This work reveals metric gaming up to the point of absurdity: fraudsters publish nonsensical algorithmically generated papers featuring genuine references. It stresses the need to screen papers for nonsense before peer‐review and chase citation manipulation in published papers. Overall, this is yet another illustration of the harmful effects of the pressure to publish or perish.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/asi.24495

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 30

  1. 10.1007/s11192-015-1637-z · DOI · Paper page
  2. 10.7551/mitpress/11087.003.0018 · DOI · Paper page
  3. Avros R. &Volkovich Z.(2018).Detection of computer‐generated papers using oneclass SVM and cluster approaches. InMLDM'18: Proceedings of the International Conference on Machine Learning and Data Mining in Pattern Recognition(Vol.LNCS 10935 pp.42–55).https://doi.org/10.1007/978-3-319-96133-0_4 · doi:10.1007/978-3-319-96133-0_4
  4. 10.1038/nature03653 · DOI
  5. 10.1126/science.348.6230.18 · DOI · Paper page
  6. 10.1002/asi.23445 · DOI · Paper page
  7. 10.1126/science.aal0625 · doi:10.1126/science.aal0625
  8. 10.12688/f1000research.23847.2 · DOI · Paper page
  9. Dalkilic M. M. Clark W. T. Costello J. C. &Radivojac P.(2006).Using compression to identify classes of inauthentic texts. InProceedings of the 2006 SIAM International Conference on Data Mining.https://doi.org/10.1137/1.9781611972764.69 · DOI
  10. 10.1007/s11948-014-9521-4 · DOI
  11. 10.1038/d41586-018-04190-5 · DOI
  12. 10.1038/508044a · DOI
  13. 10.3233/isu-160799 · doi:10.3233/isu-160799
  14. 10.1089/blr.2009.9918 · doi:10.1089/blr.2009.9918
  15. Labbé C., 2010, Ike Antkare, one of the great stars in the scientific firmament, ISSI Newsletter, 6, 48
  16. 10.1007/s11192-012-0781-y · DOI · Paper page
  17. 10.1007/978-3-319-24403-7_8 · DOI
  18. Lavoie A. &Krishnamoorthy M.(2010 August 4).Algorithmic detection of computer generated text. arXiv. Retrieved fromhttps://arxiv.org/abs/1008.0706
  19. 10.1109/tip.1996.8100841 · doi:10.1109/tip.1996.8100841
  20. 10.1038/s42256-020-0223-0 · DOI
  21. Nguyen M. T.(2018).Detection of automatically generated texts(doctoral dissertation Université Grenoble Alpes). Retrieved fromhttps://tel.archives-ouvertes.fr/tel-01919207
  22. 10.1007/s11192-018-2789-4 · DOI
  23. 10.1038/d41586-019-02142-1 · DOI
  24. Schneegans S., 2015, UNESCO science report: Towards 2030 (Tech. Rep.)
  25. 10.1038/nature.2014.16269 · DOI
  26. 10.1038/nature.2014.14763 · DOI · Paper page
  27. 10.1007/978-3-319-25087-8_32 · DOI
  28. 10.1109/KESE.2009.62 · DOI
  29. 10.1145/1044956.1044958 · DOI
  30. 10.1145/1255175.1255220 · DOI

Cited by

See all 83

Metadata

Authors
Cabanac, G., & Labbé, C.
Venue
Journal of the Association for Information Science and Technology
Publisher
Wiley
Published
2021-05-25
Volume
72
Issue
12
Pages
1461-1476
ISSN
  • 2330-1635 (ISSN-L)
  • 2330-1643
OpenAlex
W3165883805
Cited by
83
References
7
FWCI
17.61
Type
journal article
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.