Citation Contamination by Paper Mill Articles in Systematic Reviews of the Life Sciences
Open accessAbstract
Importance: Systematic reviews are the criterion standard for evidence synthesis in the life sciences, yet their reliability and integrity are threatened by citation contamination from fabricated publications produced by paper mills. Despite growing awareness, the extent and implications of this issue remain unclear. Objectives: To analyze the prevalence, characteristics, affected subject areas, and citation patterns of retracted paper mill articles cited in systematic reviews. Design, Setting, and Participants: This cross-sectional study analyzed systematic reviews published between 2013 and 2024, indexed in Web of Science (WoS). References were matched against the Retraction Watch dataset, and full texts were reviewed to identify retracted paper mill articles incorporated into the evidence synthesis. Main Outcomes and Measures: The study assessed (1) contamination prevalence, defined as the proportion of systematic reviews incorporating retracted paper mill articles into the evidence synthesis; (2) geographic distribution of citing authors according to institutional affiliations; (3) citation timing and trends, including the time lag between incorporation and article retraction; (4) affected research areas, categorized by WoS subject classifications; and (5) citation patterns, including highly contaminated reviews (≥3 incorporations of retracted articles). Results: Of the total of 200 000 systematic reviews, 299 incorporated at least 1 retracted paper mill article into the evidence synthesis (contamination rate, 0.15%). Among them, 256 (85.6%) included a single retracted article, and 43 (14.4%) included multiple such articles. Of 1802 author affiliations associated with the contaminated reviews, 660 (36.6%) were from institutions in China. Of 385 total citations, 124 (32.2%) occurred after retraction, including 13 occurring more than 500 days after the retraction date. Oncology was the most affected field (48 of 299 [16.1%]). Five reviews each included 5 or more retracted articles, all published in journals under questionable publishers. Conclusions and Relevance: In this cross-sectional study of life sciences systematic reviews, contamination remained low but increased over time, posing a risk to research integrity. Continued citation of retracted articles, even after retraction, highlights the need for rigorous screening practices. Correcting contaminated reviews and developing automated detection tools are essential to preserving the credibility of systematic reviews.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Systematic reviews that inform clinical guidelines have included paper mill articles.
“Paper mill articles have been included in systematic reviews informing clinical guidelines; 10”
From Fabricated citations: an audit across 2·5 million biomedical papers (2026) · cites this paper as [10]
The Results (Contamination Prevalence; page unavailable) identify 299 life-sciences systematic reviews that incorporated paper mill articles. The Introduction says systematic reviews inform clinical decision-making, but the paper does not establish that the contaminated reviews informed clinical guidelines specifically. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
3 retracted
Retracted publications and their citation in dental literature: A systematic review
Rapani, A., Lombardi, T., Berton, F., Lupo, V. D., Lenarda, R. D., & Stacchi, C. (2020) · Clinical and Experimental Dental Research · doi:10.1002/cre2.292 · Cited by 64 · see citing works
RetractedRetracted publications in pharmacy systematic reviews
Brown, S. J., Bakker, C., & Theis-Mahon, N. R. (2022) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2022.1280 · Cited by 35 · see citing works
RetractedCandal‐Pedreira, C., Guerra-Tort, C., Ruano‐Raviña, A., Freijedo-Farinas, F., Rey‐Brandariz, J., Ross, J. S., & Pérez‐Ríos, M. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111397 · Cited by 35 · see citing works
RetractedThe PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283 · see citing works
Morrison, A., Polisena, J., Husereau, D., Moulton, K., Clark, M., Fiander, M., Mierzwinski-Urban, M., Clifford, T., Hutton, B., & Rabb, D. (2012) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462312000086 · Cited by 1,257 · see citing works
Searching for grey literature for systematic reviews: challenges and benefits
Mahood, Q., Van Eerd, D., & Irvin, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1106 · Cited by 814 · see citing works
Introduction to systematic review and meta-analysis
Ahn, E. J., & Kang, H. (2018) · Korean Journal of Anesthesiology · doi:10.4097/kjae.2018.71.2.103 · Cited by 670 · see citing works
Snilstveit, B., Oliver, S. R., & Vojtkova, M. (2012) · Journal of Development Effectiveness · doi:10.1080/19439342.2012.710641 · Cited by 597 · see citing works
Pollock, A., & Berge, E. (2017) · International Journal of Stroke · doi:10.1177/1747493017743796 · Cited by 596 · see citing works
Publication bias: What are the challenges and can they be overcome?
Joober, R., Schmitz, N., Annable, L., & Boksa, P. (2012) · Journal of Psychiatry and Neuroscience · doi:10.1503/jpn.120065 · Cited by 382 · see citing works
Research Techniques Made Simple: Assessing Risk of Bias in Systematic Reviews
Drucker, A. M., Fleming, P., & Chan, A. (2016) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2016.08.021 · Cited by 316 · see citing works
Systematic reviews of the effectiveness of quality improvement strategies and programmes
Grimshaw, J. (2003) · BMJ Quality & Safety · doi:10.1136/qhc.12.4.298 · Cited by 290 · see citing works
Needleman, I. G. (2002) · Journal Of Clinical Periodontology · doi:10.1034/j.1600-051x.29.s3.15.x · Cited by 260 · see citing works
The fight against fake-paper factories that churn out sham science
Else, H., & Van Noorden, R. (2021) · Nature · doi:10.1038/d41586-021-00733-5 · Cited by 251 · see citing works
Catalogue of bias: publication bias
DeVito, N., & Goldacre, B. M. (2018) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2018-111107 · Cited by 226 · see citing works
Oviedo‐García, M. Á. (2021) · Research Evaluation · doi:10.1093/reseval/rvab020 · Cited by 172 · see citing works
Systematic review of publication bias in studies on publication bias
Dubben, H.-H., & Beck‐Bornholdt, H. (2005) · BMJ · doi:10.1136/bmj.38478.497164.f7 · Cited by 153 · see citing works
Byrne, J. A., & Christopher, J. (2020) · FEBS Letters · doi:10.1002/1873-3468.13747 · Cited by 133 · see citing works
Abalkina, A. (2023) · Learned Publishing · doi:10.1002/leap.1574 · Cited by 101 · see citing works
The raw truth about paper mills
Christopher, J. (2021) · FEBS Letters · doi:10.1002/1873-3468.14143 · Cited by 99 · see citing works
Paper mill challenges: past, present, and future
Parker, L., Boughton, S., Bero, L., & Byrne, J. A. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111549 · Cited by 81 · see citing works
Retractions, Fake Peer Reviews, and Paper Mills
Rivera, H., & da Silva, J. A. T. (2021) · Journal of Korean Medical Science · doi:10.3346/jkms.2021.36.e165 · Cited by 79 · see citing works
How to avoid bias in systematic reviews of observational studies
de Almeida, C. P. B., & de Goulart, B. N. G. (2017) · Revista CEFAC · doi:10.1590/1982-021620171941117 · Cited by 78 · see citing works
Clinical practice guidelines: Concepts, limitations and challenges
Franco, J. V. A., Marcelo Arancibia, M., Meza, N., Eva Madrid, A., & Kopitowski, K. S. (2020) · Medwave · doi:10.5867/medwave.2020.03.7887 · Cited by 65 · see citing works
How Naunyn-Schmiedeberg’s Archives of Pharmacology deals with fraudulent papers from paper mills
Seifert, R. (2021) · Naunyn-Schmiedeberg's Archives of Pharmacology · doi:10.1007/s00210-021-02056-8 · Cited by 62 · see citing works
Byrne, J. A., & Labbé, C. (2017) · Scientometrics · doi:10.1007/s11192-016-2209-6 · Cited by 50 · see citing works
Team size and retracted citations reveal the patterns of retractions from 1981 to 2020
Sharma, K. (2021) · Scientometrics · doi:10.1007/s11192-021-04125-4 · Cited by 45 · see citing works
A call for research to address the threat of paper mills
Byrne, J. A., Abalkina, A., Akinduro-Aje, O., Christopher, J., Eaton, S. E., Joshi, N., Scheffler, U., Wise, N. H., & Wright, J. (2024) · PLoS Biology · doi:10.1371/journal.pbio.3002931 · Cited by 39 · see citing works
Systematic fabrication of scientific images revealed
Christopher, J. (2018) · FEBS Letters · doi:10.1002/1873-3468.13201 · Cited by 38 · see citing works
Lunny, C., Kanji, S., Thabet, P., Haidich, A., Bougioukas, K. I., & Pieper, D. (2024) · BMJ Medicine · doi:10.1136/bmjmed-2023-000604 · Cited by 34 · see citing works
MicroRNA as Epigenetic Modifiers in Endometrial Cancer: A Systematic Review
Favier, A., Rocher, G., Larsen, A. K., Delangle, R., Uzan, C., Sabbah, M., Castela, M., Duval, A., Méhats, C., & Canlorbe, G. (2021) · Cancers · doi:10.3390/cancers13051137 · Cited by 34 · see citing works
Expression of MicroRNAs in Sepsis-Related Organ Dysfunction: A Systematic Review
Maiese, A., Scatena, A., Costantino, A., Chiti, E., Occhipinti, C., Russa, R. L., Paolo, M. D., Turillazzi, E., Frati, P., & Fineschi, V. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms23169354 · Cited by 33 · see citing works
Zhou, Q. X., Wang, Z., Shi, Q., Zhao, S., Xun, Y., Liu, H., Zhang, H., Xiao, L., Wang, X., Yao, L., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.08.013 · Cited by 33 · see citing works
Identification of retracted publications and completeness of retraction notices in public health
Bakker, C. J., Reardon, E. E., Brown, S. J., Theis-Mahon, N., Schroter, S., Bouter, L., & Zeegers, M. P. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111427 · Cited by 32 · see citing works
Paper mills: a novel form of publishing malpractice affecting psychology
Bishop, D., & Abalkina, A. (2023) · Meta-Psychology · doi:10.15626/mp.2022.3422 · Cited by 30 · see citing works
Systematic reviews do not (yet) represent the ‘gold standard’ of evidence: A position paper
Moore, A., Fisher, E., & Eccleston, C. (2022) · European Journal of Pain · doi:10.1002/ejp.1905 · Cited by 30 · see citing works
MicroRNA in Papillary Thyroid Carcinoma: A Systematic Review from 2018 to June 2020
Hîțu, L., Gábora, K., Bonci, E.-A., Piciu, A., Hitu, A.-C., Ștefan, P.-A., & Piciu, D. (2020) · Cancers · doi:10.3390/cancers12113118 · Cited by 30 · see citing works
The Mechanism of Long Non-coding RNA in Cancer Radioresistance/Radiosensitivity: A Systematic Review
Wu, W., Zhang, S., & He, J. (2022) · Frontiers in Pharmacology · doi:10.3389/fphar.2022.879704 · Cited by 30 · see citing works
Yeo-Teh, N. S. L., & Tang, B. L. (2022) · Publications · doi:10.3390/publications10030029 · Cited by 28 · see citing works
Klicka, K., Grzywa, T. M., Klinke, A., Mielniczuk, A., & Wlodarski, P. K. (2021) · Cancers · doi:10.3390/cancers13143393 · Cited by 27 · see citing works
Systematic reviews supporting practice guideline recommendations lack protection against bias
Brito, J. P., Τσάπας, Α., Griebeler, M. L., Wang, Z., Prutsky, G. J., Domecq, J. P., Murad, M. H., & Montori, V. M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.01.008 · Cited by 26 · see citing works
Yet another problem with systematic reviews: a living review update
Uttley, L., Weng, Y., & Falzon, L. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111608 · Cited by 23 · see citing works
The quality of systematic reviews and other synthesis in the time of COVID-19
Baumeister, A., Corrin, T., Abid, H., Young, K. M., Ayache, D., & Waddell, L. A. (2021) · Epidemiology and Infection · doi:10.1017/s0950268821001758 · Cited by 20 · see citing works
Recent trends: Retractions of articles in the oncology field
Quan, Q., Jiaqun, H., Wu, Y., Pan, Y., Zhuang, J., & Yang, X. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e33007 · Cited by 16 · see citing works
Fake papers compromise research syntheses
Else, H. (2024) · Science · doi:10.1126/science.adu8281 · Cited by 7 · see citing works
Workshop 11-sources of bias in studies of systematic reviews with or without meta-analysis.
Yu, I. T.-S., & Tse, S. L. A. (2013) · PubMed · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Fabricated citations: an audit across 2·5 million biomedical papers
Topaz, M., Roguin, N., Gupta, P., Zhang, Z., & Peltonen, L.-M. (2026) · The Lancet · doi:10.1016/s0140-6736(26)00603-3 · Cited by 85 · see citing works
Spick, M. P., Onoja, A., Harrison, C., Stender, S., Byrne, J. A., & Geifman, N. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112203 · Cited by 39 · see citing works
The Machine Consumes Itself: Artificial Intelligence and the End of Publish-or-Perish
Manabu, T. (2026) · Postdigital Science and Education · doi:10.1007/s42438-026-00666-0 · Cited by 5 · see citing works
Suspected distortion of citations in high-impact cancer journalsPreprint
Scancar, B., Byrne, J. A., Causeur, D., & Barnett, A. G. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.25.727627 · Cited by 4 · see citing works
Zhou, X., Hu, D., Xiong, H., Zhou, H., Xu, H., & Tang, H. (2026) · Digital Health · doi:10.1177/20552076261433666 · Cited by 2 · see citing works
Zil‐E‐Ali, A. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112252 · Cited by 2 · see citing works
Marcelino, R. (2026) · Research Integrity and Peer Review · doi:10.1186/s41073-026-00197-z · Cited by 2 · see citing works
Beyond Retractions: A Topic-Based Analysis of the Impact and Lifespan of Retracted Papers
Lou, Y., Zhou, Z., & Li, M. (2026) · Journal of Data and Information Science · doi:10.1515/jdis-2026-0064 · Cited by 1 · see citing works
Tracking the retracted paper mill articles: a bibliometric study
Cheng, M., Xinhua, Y., & Allen, R. M. (2026) · Scientometrics · doi:10.1007/s11192-026-05751-6 · Cited by 2 · see citing works
Begagić, E., Skenderi, F., & Vranić, S. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112429 · Cited by 1 · see citing works
Krystal, J. H., Barch, D. M., Bugno, R., & Carter, C. S. (2025) · Biological Psychiatry · doi:10.1016/j.biopsych.2025.11.017 · Cited by 2 · see citing works
Navigating the new frontier: growth, integrity, and our vision for 2026
Lalwani, N. (2026) · Abdominal Radiology · doi:10.1007/s00261-025-05350-8 · Cited by 1 · see citing works
Paper mills: la industria del fraude que amenaza la ciencia
Giraldo, J. C. J. (2026) · doi:10.25012/blog.23.07.2026 · Cited by 1 · see citing works
Kamel, R., & Ghassemi-Barghi, N. (2025) · Trends in Scholarly Publishing · doi:10.21124/tsp.2025.81.87
Why does research integrity affect us all?
Funck‐Brentano, C., & Filliatreau, G. (2026) · European Heart Journal - Cardiovascular Pharmacotherapy · doi:10.1093/ehjcvp/pvag015
AI for detecting paper mill papers
Candal‐Pedreira, C., & Ruano‐Raviña, A. (2026) · BMJ · doi:10.1136/bmj.s564
Fabricated references are twice as common in medRxiv preprints as in peer‐reviewed articles
Topaz, M., Zhang, Z., Gupta, P., Roguin, N., & Peltonen, L. (2026) · Journal of Internal Medicine · doi:10.1111/joim.70139
The landscape of problematic papers in the field of non-coding RNA
Lou, Y., Zhou, Z., Wang, G., Shen, Z., & Li, M. (2026) · Journal of Informetrics · doi:10.1016/j.joi.2026.101851
Let's use ChatGPT for research: fictitious citations, who's responsible?
Abdullah, S., Anwar, F., Khalid, Z., & Siddique, A. (2026) · Consortium Psychiatricum · doi:10.17816/cp15879
How publish-or-perish culture drives evidence fragmentation in oncology
Chau, M. T., Ofori-Manteaw, B. B., Ago, J. L., Spuur, K. M., Kerr, H., & White, S. (2026) · Annals of Medicine · doi:10.1080/07853890.2026.2726583
Integrity and trust in anesthesiology research: recognizing and combating scientific misconduct
Schmidt, A. P., Bilotta, F., Carmona, M. J. C., & Azi, L. M. T. D. A. (2026) · Brazilian Journal of Anesthesiology (English Edition) · doi:10.1016/j.bjane.2026.844843
Suspected distortion of citations in high-impact cancer journals
Scancar, B., Byrne, J. A., Causeur, D., & Barnett, A. G. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2618402123 · Cited by 3 · see citing works
A Multi-dimensional Analysis of Paper Mill Related Retractions in the Medical Field
Wang, Y., Yang, Y., Yin, H., & Gao, D. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-6913-6_19
Li, M. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-6913-6_17
Paper Mills: Industrialised Fraud and the Erosion of Trust in Science
Nourmohammadi, F., & Sarwar, M. (2026) · Trends in Scholarly Publishing · doi:10.21124/tsp.2026.138.145
- Authors
- Tang, G., & Cai, H.
- Affiliations
- University of Calgary (CA)
- Chongqing Medical University (CN)
- Corresponding author
- Tang, Gengyan
- ORCID
- Tang, Gengyan
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2025-06-12
- Volume
- 8
- Issue
- 6
- Pages
- e2515160
- ISSN
- 2574-3805 (ISSN-L)
- PMC
- PMC12163679
- OpenAlex
- W4411237045
- Cited by
- 25
- References
- 44
- Citations by year
- 2026: 23 · 2025: 2
- FWCI
- 3.87
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JAMA Network Open (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- paper mills, retracted article, systematic reviews, research integrity, evidence synthesis, citation patterns
- MeSH terms
- Systematic Reviews as Topic, Biological Science Disciplines, Cross-Sectional Studies, Humans, Retraction of Publication as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
