CoChat
Sign Up
Cited by 17Artificial Intelligence in Healthcare and EducationMeta-analysis and systematic reviewsAcademic Writing and Publishing

Reporting checklist for foundation and large language models in medical research (REFINE): an international consensus guideline

Open access
Mese, Ismail, D’Antonoli, Tugba Akinci, Bluethgen, Christian, Bressem, Keno, Cuocolo, Renato, Chaudhari, Akshay, Tejani, Ali S., Isaac, Amanda +49 more·2026·Diagnostic and Interventional Radiology
DOI 10.4274/dir.2026.263812
Open this paper in CoChat — free accountOpen PDF

Abstract

PURPOSE: To develop the REporting checklist for FoundatIon and large laNguagE models (REFINE), an international reporting guideline for transparent and reproducible reporting of foundation model (FM) and large language model (LLM) studies in medical research, including imaging artificial intelligence (AI) applications. METHODS: The protocol was prespecified and publicly archived. A modified Delphi process was conducted to establish reporting standards for unimodal and multimodal FM and LLM applications involving text, imaging, and structured data. The steering committee coordinated protocol development, expert recruitment, all Delphi rounds, and the harmonization phase. Decisions were made based on predefined consensus thresholds. In Rounds 1 and 2, structured ratings and free-text feedback informed iterative revisions. In the post-Delphi harmonization phase, terminology was standardized, and detailed reporting instructions were finalized. RESULTS: The REFINE development group comprised 57 contributors from 17 countries, and 54 panelists from 16 countries completed Rounds 1 and 2. The harmonization phase was completed by three expert panelists and the steering committee. The entire process produced a 44-item, six-section framework with standardized terminology and detailed reporting instructions, supported by an online platform for practical use (https://refinechecklist.github.io/refine/checklist.html). CONCLUSION: The REFINE provides a comprehensive, consensus-based reporting standard for medical FM and LLM research, including imaging AI studies. The online version facilitates practical implementation. CLINICAL SIGNIFICANCE: The REFINE enables transparent, comparable, and reproducible reporting of FM and LLM studies, supporting reliable evidence synthesis in medical and imaging-focused AI studies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.4274/dir.2026.263812

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 47

  1. Bedi S, Liu Y, Orr-Ewing L, et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA. 2025;333(4):319-328. [Crossref] · DOI · Paper page
  2. Busch F, Hoffmann L, Rueger C, et al. Current applications and challenges in large language models for patient care: a systematic review. Commun Med (Lond). 2025;5(1):26. [Crossref] · DOI · Paper page
  3. Wei Q, Yao Z, Cui Y, Wei B, Jin Z, Xu X. Evaluation of ChatGPT-generated medical responses: a systematic review and meta-analysis. J Biomed Inform. 2024;151:104620. [Crossref] · DOI
  4. Bagde H, Dhopte A, Alam MK, Basri R. A systematic review and meta-analysis on ChatGPT and its utilization in medical and dental research. Heliyon. 2023;9(12):e23050. [Crossref] · DOI
  5. Levin G, Horesh N, Brezinov Y, Meyer R. Performance of ChatGPT in medical examinations: a systematic review and a meta- analysis. BJOG. 2024;131(3):378-380. [Crossref] · DOI
  6. Wang L, Li J, Zhuang B, et al. Accuracy of large language models when answering clinical research questions: systematic review and network meta-analysis. J Med Internet Res. 2025;27:e64486. [Crossref] · DOI · Paper page
  7. Huo B, Boyle A, Marfo N, et al. Large language models for chatbot health advice studies: a systematic review. JAMA Netw Open. 2025;8(2):e2457879. [Crossref] · DOI · Paper page
  8. Comeau DS, Bitterman DS, Celi LA. Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability. NPJ Digit Med. 2025;8(1):42. [Crossref] · DOI · Paper page
  9. Meskó B. Prompt engineering as an important emerging skill for medical professionals: tutorial. J Med Internet Res. 2023;25:e50638. [Crossref] · DOI · Paper page
  10. Liu NF, Lin K, Hewitt J, et al. Lost in the middle: how language models use long contexts. arXiv. Preprint posted online November 20, 2023. [Crossref] · DOI · Paper page
  11. He J, Rungta M, Koleczek D, Sekhon A, Wang FX, Hasan S. Does prompt formatting have any impact on LLM performance? arXiv. Preprint posted online November 15, 2024. [Crossref] · DOI
  12. Moëll B, Sand Aronsson F. Harm reduction strategies for thoughtful use of large language models in the medical domain: perspectives for patients and clinicians. J Med Internet Res. 2025;27:e75849. [Crossref] · DOI
  13. Choudhury A, Chaudhry Z. Large language models and user trust: consequence of self- referential learning loop and the deskilling of health care professionals. J Med Internet Res. 2024;26:e56764. [Crossref] · DOI · Paper page
  14. Gu B, Desai RJ, Lin KJ, Yang J. Probabilistic medical predictions of large language models. npj Digital Medicine. 2024;7(1):367. [Crossref] · DOI · Paper page
  15. Akinci D'Antonoli T, Bluethgen C, Cuocolo R, Klontzas ME, Ponsiglione A, Kocak B. Foundation models for radiology: fundamentals, applications, opportunities, challenges, risks, and prospects. Diagn Interv Radiol. 2025. [Crossref] · DOI
  16. Kocak B. REFINE [Internet]. Open Science Framework; 2025 Aug 19. [Crossref]
  17. Park SH, Suh CH, Lee JH, Kahn CE, Moy L. Minimum reporting items for clear evaluation of accuracy reports of large language models in healthcare (MI-CLEAR-LLM). Korean J Radiol. 2024;25(10):865-868. [Crossref] · DOI
  18. Gallifant J, Afshar M, Ameen S, et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat Med. 2025;31(1):60-69. [Crossref] · DOI · Paper page
  19. Tripathi S, Alkhulaifat D, Doo FX, et al. Development, evaluation, and assessment of large language models (DEAL) checklist: a technical report. NEJM AI. 2025;2(6):AIp2401106. [Crossref] · DOI · Paper page
  20. Xie Q, Chen Q, Chen A, et al. Medical foundation large language models for comprehensive text analysis and beyond. NPJ Digit Med. 2025;8(1):141. [Crossref] · DOI · Paper page
  21. AlSaad R, Abd-Alrazaq A, Boughorbel S, et al. Multimodal large language models in health care: applications, challenges, and future outlook. J Med Internet Res. 2024;26:e59505. [Crossref] · DOI · Paper page
  22. Shen Y, Xu Y, Ma J, et al. Multi-modal large language models in radiology: principles, applications, and potential. Abdom Radiol (NY). 2025;50(6):2745-2757. [Crossref] · DOI
  23. Maaz S, Palaganas JC, Palaganas G, Bajwa M. A guide to prompt design: foundations and applications for healthcare simulationists. Front Med (Lausanne). 2024;11:1504532. [Crossref] · DOI
  24. Patil R, Heston TF, Bhuse V. Prompt engineering in healthcare. Electronics. 2024;13(15):2961. [Crossref] · DOI
  25. Schader L, Song W, Kempker R, Benkeser D. Don't let your analysis go to seed: on the impact of random seed on machine learning- based causal inference. Epidemiology. 2024;35(6):764-778. [Crossref] · DOI
  26. Singhal K, Tu T, Gottweis J, et al. Toward expert- level medical question answering with large language models. Nat Med. 2025;31(3):943- 950. [Crossref] · DOI · Paper page
  27. Wilkinson MD, Dumontier M, Aalbersberg IJJ, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016;3:160018. [Crossref] · DOI · Paper page
  28. Ho CN, Tian T, Ayers AT, et al. Qualitative metrics from the biomedical literature for evaluating large language models in clinical decision-making: a narrative review. BMC Med Inform Decis Mak. 2024;24(1):357. [Crossref] · DOI
  29. Mongan J, Moy L, Kahn CE Jr. Checklist for artificial intelligence in medical imaging (CLAIM): a guide for authors and reviewers. Radiol Artif Intell. 2020;2(2):e200029. [Crossref] · DOI · Paper page
  30. Tejani AS, Klontzas ME, Gatti AA, et al. Checklist for artificial intelligence in medical imaging (CLAIM): 2024 update. Radiol Artif Intell. 2024;6(4):e240300. [Crossref] · DOI · Paper page
  31. Martindale APL, Llewellyn CD, de Visser RO, et al. Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines. Nat Commun. 2024;15(1):1619. [Crossref] · DOI · Paper page
  32. Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. [Crossref] · DOI · Paper page
  33. Sounderajah V, Guni A, Liu X, et al. The STARD- AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nat Med. 2025;31(10):3283-3289. [Crossref] · DOI · Paper page
  34. The CHART Collaborative. Reporting guideline for chatbot health advice studies: the CHART statement. JAMA Netw Open. 2025;8(8):e2530220. [Crossref] · DOI
  35. Johri S, Jeong J, Tran BA, et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nat Med. 2025;31(1):77-86. [Crossref] · DOI · Paper page
  36. Cacciamani GE, Eppler MB, Ganjavi C, et al. Development of the ChatGPT, Generative Artificial Intelligence and Natural Large Language Models for Accountable Reporting and Use (CANGARU) Guidelines. arXiv. Preprint posted online July 18, 2023. [Crossref] · DOI
  37. Park SH, Suh CH, Lee JH, et al. Minimum reporting items for clear evaluation of accuracy reports of large language models in healthcare (MI-CLEAR-LLM): 2025 updates. Korean J Radiol. 2025;26(12):1123-1132. [Crossref] · DOI
  38. Qi X, Zeng Y, Xie T, et al. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv. Preprint posted online October 5, 2023. [Crossref] · DOI
  39. Liu Y, He H, Han T, et al. Understanding LLMs: a comprehensive overview from training to inference. arXiv. Preprint posted online January 6, 2024. [Crossref] REFINE: reporting checklist for foundation and large language models • · DOI
  40. Naveed H, Khan AU, Qiu S, et al. A Comprehensive overview of large language models. arXiv. Preprint posted online October 17, 2024. [Crossref] · DOI · Paper page
  41. Djuhera A, Kadhe SR, Zawad S, Ahmed F, Ludwig H, Boche H. Fixing it in post: a comparative study of LLM post-training data quality and model performance. arXiv. Preprint posted online October 27, 2025. [Crossref] · DOI
  42. Fraser KC, Dawkins H, Nejadgholi I, Kiritchenko S. Fine-tuning lowers safety and disrupts evaluation consistency. arXiv. Preprint posted online June 20, 2025. [Crossref] · DOI
  43. Tarabanis C, Zahid S, Mamalis M, Zhang K, Kalampokis E, Jankelson L. Performance of publicly available large language models on internal medicine board-style questions. PLOS Digit Health. 2024;3(9):e0000604. [Crossref] · DOI
  44. Akinci D'Antonoli T, Stanzione A, Bluethgen C, et al. Large language models in radiology: fundamentals, applications, ethical considerations, risks, and future directions. Diagn Interv Radiol. 2024;30(2):80-90. [Crossref] · DOI
  45. Li L, Sleem L, Gentile N, Nichil G, State R. Exploring the impact of temperature on large language models: hot or cold? Procedia Comput Sci. 2025;264:242-251. [Crossref] · DOI · Paper page
  46. Kaddour J, Harris J, Mozes M, Bradley H, Raileanu R, McHardy R. Challenges and applications of large language models. arXiv. Preprint posted online July 19, 2023. [Crossref] · DOI · Paper page
  47. Sahoo SS, Plasek JM, Xu H, et al. Large language models for biomedicine: foundations, opportunities, challenges, and best practices. J Am Med Inform Assoc. 2024;31(9):2114-2124.

Cited by

Sorted by citations

Authors
Mese, I., D’Antonoli, T. A., Bluethgen, C., Bressem, K., Cuocolo, R., Chaudhari, A., Tejani, A. S., Isaac, A., Ponsiglione, A., Meddeb, A., Khosravi, B., Guellec, B. L., Kahn, Jr., Suh, C. H., Santos, D. P. D., Koh, D.-M., Tzanis, E., Kotter, E., Colak, E., Kitamura, F., Busch, F., Nensa, F., Yang, G., Müller, H., Kather, J. N., Nawabi, J., Kleesiek, J., Zhong, J., Santinha, J., Haubold, J., Almeida, J. G. D., Lekadir, K., Marias, K., Reiner, L. N., Hein, L. M., Moy, L., Adams, L. C., Bonmatí, L. M., Paschali, M., Moassefi, M., Dietzel, M., Huisman, M., Ingrisc, M., Klontzas, M. E., Papanikolaou, N., Diaz, O., Kuriki, P., Seeböck, P., Rouzrokh, P., Strotzer, Q. D., Park, S. H., Faghani, S., Arasteh, S. T., Kim, S. H., Venugopal, V. K., Kim, W., & Kocak, B.
Affiliations
  • TUM Klinikum (DE)
  • University of Salerno (IT)
  • Stanford Medicine (US)
  • University of California, San Francisco (US)
  • King's College London (GB)
  • University of Naples Federico II (IT)
  • Humboldt-Universität zu Berlin (DE)
  • Mayo Clinic (US)
  • Centre Hospitalier Universitaire de Lille (FR)
  • Asan Medical Center (KR)
  • University Medical Center of the Johannes Gutenberg University Mainz (DE)
  • Institute of Cancer Research (GB)
  • University of Crete (GR)
  • University of Freiburg (DE)
  • St. Michael's Hospital (CA)
  • Universidade Federal de São Paulo (BR)
  • Essen University Hospital (DE)
  • East China Normal University (CN)
  • University of Geneva (CH)
  • University of Leeds (GB)
  • Shanghai Jiao Tong University (CN)
  • Champalimaud Foundation (PT)
  • Institució Catalana de Recerca i Estudis Avançats (ES)
  • New York University (US)
  • Friedrich-Alexander-Universität Erlangen-Nürnberg (DE)
  • Radboud University Medical Center (NL)
  • Royal Marsden Hospital (GB)
  • Universitat de Barcelona (ES)
  • The University of Texas Southwestern Medical Center (US)
  • Medical University of Vienna (AT)
  • University Hospital Regensburg (DE)
  • Universitätsklinikum Aachen (DE)
  • Rajiv Gandhi Cancer Institute and Research Centre (IN)
  • American College of Radiology (US)
ORCID
Venue
Diagnostic and Interventional Radiology
Publisher
Galenos Yayinevi
Published
2026-02-26
ISSN
  • 1300-4360 (ISSN-L)
  • 1305-3825
  • 1305-3612
DOI
10.4274/dir.2026.263812
OpenAlex
W7131662404
Cited by
17
References
0
Citations by year
2026: 17
FWCI
33.43
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • Diagnostic and Interventional Radiology (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • FreiDok plus (Universitätsbibliothek Freiburg) (submittedVersion)
Funders
  • Pfizer
  • AstraZeneca
  • Radiological Society of North America
  • Deutscher Akademischer Austauschdienst (57616814)
  • Deutsche Forschungsgemeinschaft (390696704)
  • Technische Universität Dresden (390696704)
  • Eisai
  • Daiichi Sankyo Europe
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
Keywords
foundation models, large language models, reporting guidelines, multimodal AI, reproducible reporting, Delphi consensus

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review