CoChat
Sign Up
Cited by 13Meta-analysis and systematic reviewsArtificial Intelligence in Healthcare and EducationAcademic Publishing and Open Access

Large Language Models and the Analyses of Adherence to Reporting Guidelines in Systematic Reviews and Overviews of Reviews (PRISMA 2020 and PRIOR)

Open access
Forero, Diego A., Abreu, Sandra E., Tovar, Blanca E., Oermann, Marilyn H.·2025·Journal of Medical Systems
DOI 10.1007/s10916-025-02212-0PMC PMC12162794
Open this paper in CoChat — free accountOpen PDF

Abstract

In the context of Evidence-Based Practice (EBP), Systematic Reviews (SRs), Meta-Analyses (MAs) and overview of reviews have become cornerstones for the synthesis of research findings. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 and Preferred Reporting Items for Overviews of Reviews (PRIOR) statements have become major reporting guidelines for SRs/MAs and for overviews of reviews, respectively. In recent years, advances in Generative Artificial Intelligence (genAI) have been proposed as a potential major paradigm shift in scientific research. The main aim of this research was to examine the performance of four LLMs for the analysis of adherence to PRISMA 2020 and PRIOR, in a sample of 20 SRs and 20 overviews of reviews. We tested the free versions of four commonly used LLMs: ChatGPT (GPT-4o), DeepSeek (V3), Gemini (2.0 Flash) and Qwen (2.5 Max). Adherence to PRISMA 2020 and PRIOR was compared with scores defined previously by human experts, using several statistical tests. In our results, all the four LLMs showed a low performance for the analysis of adherence to PRISMA 2020, overestimating the percentage of adherence (from 23 to 30%). For PRIOR, the LLMs presented lower differences in the estimation of adherence (from 6 to 14%) and ChatGPT showed a performance similar to human experts. This is the first report of the performance of four commonly used LLMs for the analysis of adherence to PRISMA 2020 and PRIOR. Future studies of adherence to other reporting guidelines will be helpful in health sciences research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10916-025-02212-0

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. [1]Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet. 2017;390(10092):415–23. 28215660 · DOI · Paper page
  2. [3]Forero DA, Lopez-Leon S, Gonzalez-Giraldo Y, Bagos PG. Ten simple rules for carrying out and writing meta-analyses. PLoS Comput Biol. 2019;15(5):e1006922. 31095553 PMC6521986 · DOI
  3. [4]Chalmers I, Hedges LV, Cooper H. A brief history of research synthesis. Eval Health Prof. 2002;25(1):12–37. 11868442 · DOI · Paper page
  4. [5]Patsopoulos NA, Analatos AA, Ioannidis JP. Relative citation impact of various study designs in the health sciences. JAMA. 2005;293(19):2362–6. 15900006 · DOI
  5. [6]Murad MH, Montori VM, Ioannidis JP, Jaeschke R, Devereaux PJ, Prasad K, et al. How to read a systematic review and meta-analysis and apply the results to patient care: users’ guides to the medical literature. JAMA. 2014;312(2):171–9. 25005654 · DOI · Paper page
  6. [7]Fusar-Poli P, Radua J. Ten simple rules for conducting umbrella reviews. Evid Based Ment Health. 2018;21(3):95–100. 30006442 PMC10270421 · DOI · Paper page
  7. [8]Bougioukas KI, Pamporis K, Vounzoulaki E, Karagiannis T, Haidich AB. Types and associated methodologies of overviews of reviews in health care: a methodological study with published examples. J Clin Epidemiol. 2023;153:13–25. 36351511 · DOI
  8. [9]Altman DG, Simera I. A history of the evolution of guidelines for reporting medical research: the long road to the EQUATOR Network. J R Soc Med. 2016;109(2):67–77. 26880653 PMC4793768 · DOI
  9. [12]Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29(8):1930–40. 37460753 · DOI · Paper page
  10. [13]Woelfle T, Hirt J, Janiaud P, Kappos L, Ioannidis JPA, Hemkens LG. Benchmarking Human-AI collaboration for common evidence appraisal tools. J Clin Epidemiol. 2024;175:111533. 39277058 · DOI
  11. [15]Ioannidis JPA. Meta-research: Why research on research matters. PLoS Biol. 2018;16(3):e2005468. 29534060 PMC5865753 · DOI
  12. [16]Collins GS. Innovative solutions are needed to overcome implementation barriers to using reporting guidelines. BMJ. 2025;389:r718. 40228827 · DOI
  13. [17]Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, Aleman FL, et al. Gpt-4 technical report. arXiv (preprint). 2023.
  14. [18]Liu A, Feng B, Xue B, Wang B, Wu B, Lu C, et al. Deepseek-v3 technical report. arXiv (preprint). 2024.
  15. [19]Gemini Team, Anil R, Borgeaud S, Alayrac J-B, Yu J, Soricut R, et al. Gemini: a family of highly capable multimodal models. arXiv (preprint). 2023.
  16. [20]Yang A, Yang B, Zhang B, Hui B, Zheng B, Yu B, et al. Qwen2. 5 technical report. arXiv (preprint). 2024.
  17. [21]Roberts RH, Ali SR, Hutchings HA, Dobbs TD, Whitaker IS. Comparative study of ChatGPT and human evaluators on the assessment of medical literature according to recognised reporting standards. BMJ Health Care Inform. 2023;30(1). PMC10583079 37827724
  18. [22]Wrightson JG, Blazey P, Moher D, Khan KM, Ardern CL. GPT for RCTs? Using AI to determine adherence to clinical trial reporting guidelines. BMJ Open. 2025;15(3):e088735. 40107689 PMC11927406 · DOI
  19. [23]Srinivasan A, Berkowitz J, Kivelson S, Friedrich N, Tatonetti N. Evaluating the Reporting Quality of 21,041 Randomized Controlled Trial Articles. Preprint, medRxiv. 2025:2025.03. 06.25323528.
  20. [25]Qin C, Ma H, Mandizadza OO, Xu X, Ji C. Reporting quality of meta-analyses in acupuncture: Investigating adherence to the PRISMA statement. Medicine (Baltimore). 2024;103(39):e39933. 39331860 PMC11441946 · DOI
  21. [26]Lu T, Liu B, Lu C, Du Z, Yang K, Ge L. Reporting quality of acupuncture overviews: A methodological investigation based on the PRIOR statement. Complement Ther Med. 2024;82:103034. 38521419 · DOI
  22. [27]Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. 28935701 PMC5833365 · DOI · Paper page
  23. [28]Alharbi F, Asiri S. Automated Assessment of Reporting Completeness in Orthodontic Research Using LLMs: An Observational Study. Applied Sciences. 2024;14(22):10323. · DOI
  24. [29]Innocenti T, Feller D, Giagio S, Salvioli S, Minnucci S, Brindisino F, et al. Adherence to the PRISMA statement and its association with risk of bias in systematic reviews published in rehabilitation journals: A meta-research study. Braz J Phys Ther. 2022;26(5):100450. 36270163 PMC9583447 · DOI
  25. [32]Bewick V, Cheek L, Ball J. Statistics review 9: one-way analysis of variance. Crit Care. 2004;8(2):130–6. 15025774 PMC420045 · DOI
  26. [33]McHugh ML. Multiple comparison analysis testing in ANOVA. Biochem Med (Zagreb). 2011;21(3):203–9. 22420233 · DOI
  27. [34]Bewick V, Cheek L, Ball J. Statistics review 7: Correlation and regression. Crit Care. 2003;7(6):451–9. 14624685 PMC374386 · DOI
  28. [36]Li Z, Luo X, Yang Z, Zhang H, Wang B, Ge L, et al. RAPID: Reliable and efficient Automatic generation of submission rePorting checklists with Large language moDels. Preprint, bioRxiv. 2025:2025.02. 13.638015. PMC12277685 40574698
  29. [37]Gartlehner G, Kahwati L, Nussbaumer-Streit B, Crotty K, Hilscher R, Kugley S, et al. From promise to practice: challenges and pitfalls in the evaluation of large language models for data extraction in evidence synthesis. BMJ Evid Based Med. 2024. PMC12703319 39797673
  30. [38]Love J, Selker R, Marsman M, Jamil T, Dropmann D, Verhagen J, et al. JASP: Graphical statistical software for common statistical designs. Journal of Statistical Software. 2019;88:1–17. · DOI
  31. [39]Park SH, Suh CH, Lee JH, Kahn CE, Moy L. Minimum Reporting Items for Clear Evaluation of Accuracy Reports of Large Language Models in Healthcare (MI-CLEAR-LLM). Korean J Radiol. 2024;25(10):865–8. 39344542 PMC11444851 · DOI
  32. [40]Smith AL, Greaves F, Panch T. Hallucination or Confabulation? Neuroanatomy as metaphor in Large Language Models. PLOS Digit Health. 2023;2(11):e0000388. 37910473 PMC10619792 · DOI
  33. [41]Kim Y, Jeong H, Chen S, Li SS, Lu M, Alhamoud K, et al. Medical Hallucination in Foundation Models and Their Impact on Healthcare. medRxiv (Preprint). 2025:2025.02. 28.25323115.
  34. [42]Miao BY, Chen IY, Williams CYK, Davidson J, Garcia-Agundez A, Sun S, et al. The MI-CLAIM-GEN checklist for generative artificial intelligence in health. Nat Med. 2025. PMC12207735 39915678
  35. [43]Lobentanzer S, Feng S, Bruderer N, Maier A, BioChatter C, Wang C, et al. A platform for the biomedical application of large language models. Nat Biotechnol. 2025;43(2):166–9. 39843580 PMC12216031 · DOI
  36. [44]Nguyen T. ChatGPT in Medical Education: A Precursor for Automation Bias? JMIR Med Educ. 2024;10:e50174. 38231545 PMC10831594 · DOI
  37. [45]Mahowald K, Ivanova AA, Blank IA, Kanwisher N, Tenenbaum JB, Fedorenko E. Dissociating language and thought in large language models. Trends Cogn Sci. 2024;28(6):517–40. 38508911 PMC11416727 · DOI · Paper page
  38. [47]Kuitunen I, Ponkilainen VT, Liukkonen R, Nyrhi L, Pakarinen O, Vaajala M, et al. Evaluating the Performance of ChatGPT-4o in Risk of Bias Assessments. J Evid Based Med. 2024;17(4):700–2. 39676337 PMC11684499 · DOI
  39. [48]Barsby J, Hume S, Lemmey HA, Cutteridge J, Lee R, Bera KD. Pilot study on large language models for risk-of-bias assessments in systematic reviews: A(I) new type of bias? BMJ Evid Based Med. 2025;30(1):71–4. 38782561 · DOI
  40. [49]Lin Z. How to write effective prompts for large language models. Nat Hum Behav. 2024;8(4):611–5. 38438650 · DOI
  41. [50]Anisuzzaman D, Malins JG, Friedman PA, Attia ZI. Fine-Tuning Large Language Models for Specialized Use Cases. Mayo Clin Proc Digit Health. 2025;3(1):100184. PMC11976015 40206998
  42. [51]Cox LA, Jr. An AI assistant to help review and improve causal reasoning in epidemiological documents. Glob Epidemiol. 2024;7:100130. 38188038 PMC10767365 · DOI

Cited by

Sorted by citations

Authors
Forero, D. A., Abreu, S. E., Tovar, B. E., & Oermann, M. H.
Affiliations
  • Fundación Universitaria del Área Andina (CO)
  • Duke University (US)
Corresponding author
Forero, Diego A.
ORCID
Venue
Journal of Medical Systems
Publisher
Springer Science and Business Media LLC
Published
2025-06-12
Volume
49
Issue
1
Pages
80-80
ISSN
  • 0148-5598 (ISSN-L)
  • 1573-689X
DOI
10.1007/s10916-025-02212-0
PMC
PMC12162794
OpenAlex
W4411251263
Cited by
13
References
11
Citations by year
2026: 12 · 2025: 1
FWCI
2.46
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Medical Systems (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
University Foundation of the Andean Area
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
Keywords
large language models, PRISMA 2020, prior distributions, systematic reviews, overviews of reviews, meta-analysis reporting, generative AI, ChatGPT, DeepSeek, Gemini, Qwen, human expert comparison
MeSH terms
Systematic Reviews as Topic, Large Language Models, Artificial Intelligence, Humans, Meta-Analysis as Topic, Guidelines as Topic, Guideline Adherence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review