Development, evaluation, and assessment of large language models (DEAL) checklist: a technical report
Abstract
AbstractLarge language models (LLMs) have advanced artificial intelligence research in medicine, especially in natural language processing tasks. However, the nascent evolution of LLM practices presents challenges to the transparency, reproducibility, and rigor of research methods. Standardized reporting in research is critical to ensure reliable scientific communication and evaluation. This article introduces the Development, Evaluation, and Assessment of Large Language Models (DEAL) checklist, designed to guide authors and reviewers in reporting LLM studies. The checklist comprises two pathways: DEAL-A, tailored for advanced model development and fine-tuning, and DEAL-B, suited to applied research using pretrained models with minimal modifications. Each pathway addresses critical elements such as model specifications, data-handling practices, training procedures, evaluation metrics, and transparency standards. The DEAL checklist provides a comprehensive structure for documenting LLM research with the aim of making it accessible and reproducible. This structured approach aims to set a standard for future research, facilitating peer review and encouraging best practices. The DEAL checklist will serve as a valuable tool for enhancing the quality and reproducibility of LLM research. By offering clear guidelines on critical reporting elements, the DEAL checklist promotes robust and transparent scientific reporting, ultimately supporting the reliable advancement of LLM technologies.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The study adhered to the Development, Evaluation, and Assessment of Large Language Models guidelines for transparent and comprehensive reporting.
“To ensure transparent and comprehensive reporting of this LLM-based system, we adhered to the Development, Evaluation, and Assessment of Large Language Models guidelines [40]; the completed Development, Evaluation, and Assessment of Large Language Models checklist is provided in Checklist 1.”
From Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study (2026) · cites this paper as [40]
Cites
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group (2010) · BMJ · doi:10.1136/bmj.c332 · Cited by 9,329 · see citing works
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040297 · Cited by 8,613 · see citing works
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · BMJ · doi:10.1136/bmj.h5527 · Cited by 3,660 · see citing works
Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127 · see citing works
Guidance for Developers of Health Research Reporting Guidelines
Moher, D., Schulz, K. F., Simera, I., & Altman, D. G. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000217 · Cited by 1,437 · see citing works
The future landscape of large language models in medicine
Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J.-N., Laleh, N. G., Löffler, C. M. L., Schwarzkopf, S.-C., Unger, M., Veldhuizen, G. P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00370-1 · Cited by 1,045 · see citing works
On evaluation metrics for medical applications of artificial intelligence
Hicks, S. A., Strümke, I., Thambawita, V. L., Hammou, M., Riegler, M. A., Halvorsen, P., & Parasa, S. (2022) · Scientific Reports · doi:10.1038/s41598-022-09954-8 · Cited by 947 · see citing works
The TRIPOD-LLM reporting guideline for studies using large language models
Gallifant, J., Afshar, M., Ameen, S., Aphinyanaphongs, Y., Chen, S., Cacciamani, G., Demner-Fushman, D., Dligach, D., Daneshjou, R., Fernandes, C., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03425-5 · Cited by 615 · see citing works
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update
Tejani, A. S., Klontzas, M. E., Gatti, A. A., Mongan, J. T., Moy, L., Park, S. H., Kahn, C. E., Panel, F. T. C. 2. U., Abbara, S., Afat, S., et al. (2024) · Radiology: Artificial Intelligence · doi:10.1148/ryai.240300 · Cited by 570 · see citing works
Self-citationCreation and Adoption of Large Language Models in Medicine
Shah, N. H., Entwistle, D., & Pfeffer, M. A. (2023) · JAMA · doi:10.1001/jama.2023.14217 · Cited by 435 · see citing works
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01258-7 · Cited by 424 · see citing works
EQUATOR: reporting guidelines for health research
Altman, D. G., Simera, I., Hoey, J., Moher, D., & Schulz, K. (2008) · The Lancet · doi:10.1016/s0140-6736(08)60505-x · Cited by 325 · see citing works
Radiological Society of North America
(1977) · Radiology · doi:10.1148/124.3.812 · Cited by 318 · see citing works
Kresevic, S., Giuffrè, M., Ajcevic, M., Accardo, A., Crocè, L. S., & Shung, D. L. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01091-y · Cited by 207 · see citing works
Bossuyt, P. M., & Reitsma, J. B. (2003) · The Lancet · doi:10.1016/s0140-6736(03)12122-8 · Cited by 146 · see citing works
Tripathi, S., Sukumaran, R., & Cook, T. S. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad258 · Cited by 137 · see citing works
Self-citationEquity in essence: a call for operationalising fairness in machine learning for healthcare
Wawira Gichoya, J., McCoy, L. G., Celi, L. A., & Ghassemi, M. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100289 · Cited by 104 · see citing works
Understanding Biases and Disparities in Radiology AI Datasets: A Review
Tripathi, S., Gabriel, K., Dheer, S., Parajuli, A., Augustin, A. I., Elahi, A., Awan, O., & Dako, F. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.06.015 · Cited by 75 · see citing works
Self-citationEvaluating the Application of Large Language Models in Clinical Research Contexts
Perlis, R. H., & Fihn, S. D. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.35924 · Cited by 49 · see citing works
Park, S. H., & Suh, C. H. (2024) · Korean Journal of Radiology · doi:10.3348/kjr.2024.0598 · Cited by 43 · see citing works
Protocol for the development of the Chatbot Assessment Reporting Tool (CHART) for clinical advice
Collaborative, T. C. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-081155 · Cited by 32 · see citing works
Simera, I., & Altman, D. G. (2013) · International Journal of Clinical Practice · doi:10.1111/ijcp.12168 · Cited by 28 · see citing works
Large Language Models in Health Systems: Governance, Challenges, and Solutions
Tripathi, S., Mongeau, K., Alkhulaifat, D., Elahi, A., & Cook, T. S. (2024) · Academic Radiology · doi:10.1016/j.acra.2024.09.047 · Cited by 13 · see citing works
Self-citationElhakim, T., Brea, A., Fidelis, W., et al. (2024) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2024.08.010 · Cited by 11 · see citing works
Self-citationThe Potential of Large Language Models for Radiology Report Simplification and Translations
Tripathi, S., & Dako, F. (2024) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2024.06.004 · Cited by 5 · see citing works
Self-citationReXErr: Synthesizing Clinically Meaningful Errors in Diagnostic Radiology Reports
Rao, V. M., Zhang, S., Acosta, J., et al. (2024) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789819807024_0006 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Moher D, Altman D, Schulz K, Simera I, Wager E. Guidelines for reporting health research: a user’s manual. Chichester, UK: John Wiley & Sons, 2014.
Cited by
Sorted by citations
Park, S. H., Suh, C. H., Lee, J. H., Tejani, A. S., You, S. C., Kahn, C. E., & Moy, L. P. (2025) · Korean Journal of Radiology · doi:10.3348/kjr.2025.1522 · Cited by 38 · see citing works
Fleurence, R., Dawoud, D., Bian, J., Higashi, M. K., Wang, X., Xu, H., Chhatwal, J., & Ayer, T. (2025) · Value in Health · doi:10.1016/j.jval.2025.06.018 · Cited by 27 · see citing works
Bodien, Y. G., Busl, K. M., Chang, C. W. J., Claassen, J., Gaspard, N., Gosseries, O., Helbok, R., Massimini, M., Naccache, L., Newcombe, V., et al. (2025) · Intensive Care Medicine · doi:10.1007/s00134-025-08224-1 · Cited by 24 · see citing works
“…successful integration of artificial intelligence into clinical practice will require rigorous validation, transparency in algorithm design, mitigation of the potential for amplifying existing self-fulfilling prophesies, and close collaboration between clinicians and data scientists [261–264].”
AI in critical care: A roadmap to the future
Workum, J. D., Meyfroidt, G., Bakker, J., Jung, C., Tobin, J. M., Gommers, D. A. M. P. J., Elbers, P., van der Hoeven, J. G., & van Genderen, M. E. (2025) · Journal of Critical Care · doi:10.1016/j.jcrc.2025.155262 · Cited by 17 · see citing works
Al‐Nusair, J., Lanino, L., Durmaz, A., Porta, M. G. D., Zeidan, A. M., & Kewan, T. Z. (2025) · Blood Reviews · doi:10.1016/j.blre.2025.101340 · Cited by 13 · see citing works
Scott, I. A., Reddy, S., Kelly, T., Miller, T., & van der Vegt, A. H. (2025) · The Medical Journal of Australia · doi:10.5694/mja2.70057 · Cited by 15 · see citing works
Application of large language models in radiological diagnostics: a scoping review
Vasilev, Y. A., Reshetnikov, R. V., Nanova, O. G., Vladzymyrskyy, A. V., Arzamasov, K. M., Omelyanskaya, O. V., Kodenko, M. R., Арсеньевич, Е. Р., Pamova, A. P., Seradzhi, S. R., et al. (2025) · Digital Diagnostics · doi:10.17816/dd678373 · Cited by 8 · see citing works
Reason, T., Wu, Y., Jones, C., Benbow, E., Johannesen, K. M., & Malcolm, B. (2025) · Value in Health · doi:10.1016/j.jval.2025.08.001 · Cited by 6 · see citing works
Generative Artificial Intelligence Methodology Reporting in Otolaryngology: A Scoping Review
Alter, I. L., Chan, K., Andreadis, K., & Rameau, A. (2025) · The Laryngoscope · doi:10.1002/lary.70165 · Cited by 4 · see citing works
“…but also provides easy- to-implement checklists for investigators carrying out such studies, potentially laying the groundwork for improved reporting in the future, since all included studies from the initial search were published prior to these formal reporting recommendations [18, 19].”
Kwong, J. C. C., Nguyen, D., Khondker, A., Li, T., & Kulkarni, G. S. (2025) · BMJ Oncology · doi:10.1136/bmjonc-2025-000891 · Cited by 3 · see citing works
Jiang, W., Wang, D., Zeng, Y., Huang, J., Xu, C., & Liu, C. (2025) · Journal of Medical Internet Research · doi:10.2196/80770 · Cited by 2 · see citing works
“Privacy and data governance entail the adoption and operationalization of standardized approaches or frameworks, such as the Development, Evaluation, and Assessment of Large Language Models (DEAL) Checklist and the Framework for the Appropriate Implementation and Review of AI (FAIR-AI) [50-52].”
Khoruzhaya, A. N., Varyukhina, M. D., Арсеньевич, Е. Р., Blokhin, I. A., Reshetnikov, R. V., Kodenko, M. R., Pamova, A. P., Burtsev, T., Arzamasov, K. M., Omelyanskaya, O. V., et al. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1761601 · Cited by 2 · see citing works
“Existing reporting guidelines, including CONSORT-AI, SPIRIT-AI, TRIPOD-LLM, and DEAL, do not adequately address the specific requirements of medical text summarization tasks.”
InfluentialRueda, A., Al-Shamali, H. F., Cote, Z., Roy, N., Janssen-Aguilar, R., Joseph, J., Teferra, B. G., Kamaleddin, M. A., Burback, L., Winkler, O., et al. (2026) · JMIR Medical Education · doi:10.2196/88580 · Cited by 1 · see citing works
“To ensure transparent and comprehensive reporting of this LLM-based system, we adhered to the Development, Evaluation, and Assessment of Large Language Models guidelines [40]; the completed Development, Evaluation, and Assessment of Large Language Models checklist is provided in Checklist 1.”
Alignment of Policy, Practice, and Patient Safety for Trustworthy AI in Radiology
Doo, F. X., Davis, M., Poff, J. A., Lui, Y. W., Haines, K., & Towbin, A. J. (2026) · Radiology Artificial Intelligence · doi:10.1148/ryai.250982 · Cited by 2 · see citing works
Self-citationTeferra, B. G., Sidharta, C. K., Hsiang, W.-N., Rueda, A., Guan, B., Zhang, Y., Burback, L., Winkler, O., Greenshaw, A. J., Vermetten, E. H. G. J. M., et al. (2026) · The Canadian Journal of Psychiatry · doi:10.1177/07067437261468897 · Cited by 1 · see citing works
Wu, P., Li, F.-X., Jia, Y., Yin, J., Shen, Y., Gao, Y., Li, Y., Sha, F., Yang, Z., & Tang, J. (2025) · Gastroenterology & Endoscopy · doi:10.1016/j.gande.2025.09.003
Shah-Mohammadi, F., Millar, A., Facelli, J., & Gouripeddi, R. (2025) · medRxiv · doi:10.64898/2025.12.14.25342221
Tang, M., Xu, L., Zhu, L., Maimaitiabula, A., Zhang, X., Pei, C., & Hu, L. (2026) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-026-03415-2
“The study followed the Developmental Evaluation Guidelines for Applied LLM Benchmarking (DEAL-B), with the completed checklist provided in the Supplementary E1 [14].”
Murray, B., Bauer, S. R., Gao, Y., Barreto, E. F., Henry, K., Blotske, K., & Sikora, A. N. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1787075 · Cited by 1 · see citing works
“Reporting frameworks for various study types evaluating AI tools have been developed and are available through the Equator network (24–33).”
Legal and Regulatory Considerations
Zamanifar, A., Jenabian, N. M., Farhadi, A., et al. (2026) · Springer eBooks · doi:10.1007/978-3-032-15088-2_11
Bılgın, E., Öztürk, E., Shah, M., et al. (2026) · EULAR Rheumatology Open · doi:10.1016/j.ero.2026.100193
“Although large language models (LLMs) are increasingly used for various academic tasks, their reliability in evaluating complex methodological reporting remains poorly established.”
Shah-Mohammadi, F., Millar, A., Facelli, J. C., & Gouripeddi, R. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag108
“To align with emerging standards for transparency and re-producibility in LLM research, we followed the checklist introduced in Tripathi et al, 57 which provides structured guidance for reporting applied studies using pretrained models with minimal modifications.”
Tan, M., Tang, S., Kan, S., Wu, B., Ni, Z., Zhang, H., & Ding, J. (2026) · Journal of Medical Internet Research · doi:10.2196/87288 · Cited by 1 · see citing works
“This study was reported in accordance with the DEAL (Development, Evaluation, and Assessment of Large Language Models) checklist [19].”
Meşe, İ., Güneş, S. T., Coşkun, Ö., Keleş, A., Banaz, T., Kazanbaş, M. F., & Koçak, B. (2026) · Korean Journal of Radiology · doi:10.3348/kjr.2026.0505 · Cited by 1 · see citing works
“Consequently, several reporting frameworks have been proposed for LLM research in healthcare [24-27].”
Salvaggio, G., Comelli, A., Gagliardo, C., & Lalwani, N. (2026) · European Journal of Radiology Artificial Intelligence · doi:10.1016/j.ejrai.2026.100122
Transforming large language models into medical specialists via knowledge injection
Kim, K., Song, J., Kim, D. Y., Nam, Y., Park, S., Shin, K., Jo, W., & Kim, N. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2026.103020
Measuring Reproducibility in LLM Research: A Rubric and Cross-Topic Corpus Study
Chong, C. J., Hou, C., Liu, S., & Yao, Z. (2026) · ACM Conference on Reproducibility and Replicability (ACM REP) · doi:10.1145/3820002.3828584
Linguistic inequity in clinical AI evaluation: a scoping review of 806 studies across 62 languages
Ariel, D., & Laban, G. (2026) · Communications Health · doi:10.1038/s44528-026-00042-1
“NowidelyadoptedclinicalAIreportingframeworkcurrentlyoper-ationaliseslanguage-speci fi c deployment readiness for LLMs. Existing AI reporting guidelines 12 – 16 set standards for general LLM evaluation but contain no items speci fi c to cross-linguistic validation.”
- Authors
- Tripathi, S., Alkhulaifat, D., Doo, F. X., Rajpurkar, P., McBeth, R., Daye, D., & Cook, T. S.
- Affiliations
- Harvard University (US)
- Children's Hospital of Philadelphia (US)
- University of Maryland, Baltimore (US)
- University of Pennsylvania (US)
- ORCID
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2025-05-22
- Volume
- 2
- Issue
- 6
- Pages
- AIp2401106-AIp2401106
- ISSN
- 2836-9386 (ISSN-L)
- OpenAlex
- W4410225737
- Semantic Scholar
- 85e3f7dc328c585e0d53ba70ecf7f54aa14ab5c9
- Cited by
- 31
- References
- 28
- Citations by year
- 2026: 18 · 2025: 13
- FWCI
- 12.43
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- NEJM AI (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- large language models, checklists, model development, model evaluation, clinical AI
- MeSH terms
- Large Language Models, Humans, Natural Language Processing, Reproducibility of Results, Checklist
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
