STAGER checklist: Standardized testing and assessment guidelines for evaluating generative artificial intelligence reliability
Open accessAbstract
Generative artificial intelligence (AI) holds immense potential for medical applications, but the lack of a comprehensive evaluation framework and methodological deficiencies in existing studies hinder its effective implementation. Standardized assessment guidelines are crucial for ensuring reliable and consistent evaluation of generative AI in healthcare. Our objective is to develop robust, standardized guidelines tailored for evaluating generative AI performance in medical contexts. Through a rigorous literature review utilizing the Web of Sciences, Cochrane Library, PubMed, and Google Scholar, we focused on research testing generative AI capabilities in medicine. Our multidisciplinary team of experts conducted discussion sessions to develop a comprehensive 32‐item checklist. This checklist encompasses critical evaluation aspects of generative AI in medical applications, addressing key dimensions such as question collection, querying methodologies, and assessment techniques. The checklist and its broader assessment framework provide a holistic evaluation of AI systems, delineating a clear pathway from question gathering to result assessment. It guides researchers through potential challenges and pitfalls, enhancing research quality and reporting and aiding the evolution of generative AI in medicine and life sciences. Our framework furnishes a standardized, systematic approach for testing generative AI's applicability in medicine. For a concise checklist, please refer to Table S or visit GenAIMed.org .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There is no standardized or systematized process for evaluating AI output.
“However, a process for evaluating AI output has yet to be standardized.”
From Procedure to Evaluate Artificial Intelligence (AI) Output for Educational and Clinical Tasks (2026) · cites this paper as Chen et al., 2024
The Introduction identifies a lack of established frameworks for systematically evaluating generative AI on medically relevant problems. The citing sentence extends that narrower point to AI output generally; the Introduction also notes existing, widely adopted guidelines for evaluating clinical trials involving AI interventions. (LLM, conf=0.86)
Experimental — accuracy not yet measuredChen and colleagues identified content continuity as crucial in developing an evaluation tool for AI within medical contexts.
“In their efforts to develop an evaluation tool for AI within medical contexts, Chen and colleagues (2024) identified this aspect as crucial.”
From Procedure to Evaluate Artificial Intelligence (AI) Output for Educational and Clinical Tasks (2026) · cites this paper as Chen et al., 2024
The Results checklist (page unavailable) calls for evaluating consistency across responses to the same question, supporting the importance of response consistency in medical-AI evaluation. It does not identify content continuity—such as a continuous flow of content—as a crucial aspect; the claim is broader than the criterion stated. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Chan, A.-W., Darzi, A., Holmes, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1034-x · Cited by 1,421 · see citing works
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · see citing works
Self-citationChatGPT and Clinical Training: Perception, Concerns, and Practice of Pharm-D Students
Zawiah, M., Al-Ashwal, F., Gharaibeh, L., Abu Farha, R., Alzoubi, K., Abu Hammour, K., Qasim, Q. A., & Abrah, F. (2023) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s439223 · Cited by 51 · see citing works
Morjaria, L., Burns, L., Bracken, K., Ngo, Q. N., Lee, M., Levinson, A. J., Smith, J., Thompson, P., & Sibbald, M. (2023) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205231204178 · Cited by 31 · cited 1× · see citing works
“…AI in the medical field encompasses a broad spectrum, ranging from assessing its grasp of medical knowledge and ability to pass medical examinations [8, 9] to aiding in providing initial medical counseling [10, 11] and swiftly providing pertinent medical information and recommendations [12, 13].”
Flagged first, then by citations
Research works printed but not linked: 19
- [1]Duffourc, Mindy, and Sara Gerke. 2023. “ Generative AI in Health Care and Liability Risks for Physicians and Safety Concerns for Patients.” JAMA 330: 313– 314. 10.1001/jama.2023.9630 37410497 · DOI · Paper page
- [2]Menz, Bradley D., Natansh D. Modi, Michael J. Sorich, and Ashley M. Hopkins. 2024. “ Health Disinformation Use Case Highlighting the Urgent Need for Artificial Intelligence Vigilance: Weapons of Mass Disinformation.” JAMA Internal Medicine 184: 92– 96. 10.1001/jamainternmed.2023.5947 37955873 · DOI · Paper page
- [3]Morley, Jessica, Nicholas J. DeVito, and Joe Zhang. 2023. “ Generative AI for Medical Research.” BMJ 382: p1551. 10.1136/bmj.p1551 37437947 · DOI
- [4]Liu, Fenglin, Tingting Zhu, Xian Wu, Bang Yang, Chenyu You, Chenyang Wang, Lei Lu, et al. 2023. “ A Medical Multimodal Large Language Model for Future Pandemics.” NPJ Digital Medicine 6: 226. 10.1038/s41746-023-00952-2 38042919 PMC10693607 · DOI · Paper page
- [5]Zhu, Lingxuan, Weiming Mou, and Rui Chen. 2023. “ Can the ChatGPT and Other Large Language Models With Internet‐Connected Database Solve the Questions and Concerns of Patient With Prostate Cancer and Help Democratize Medical Knowledge?” Journal of Translational Medicine 21: 269. 10.1186/s12967-023-04123-5 37076876 PMC10115367 · DOI
- [6]Lim, Zhi Wei, Krithi Pushpanathan, Samantha Min Er Yew, Yien Lai, Chen‐Hsin Sun, Janice Sing Harn Lam, David Ziyou Chen, et al. 2023. “ Benchmarking Large Language Models' Performances for Myopia Care: A Comparative Analysis of ChatGPT‐3.5, ChatGPT‐4.0, and Google Bard.” eBioMedicine 95: 104770. 10.1016/j.ebiom.2023.104770 37625267 PMC10470220 · DOI
- [7]Kleinig, Oliver, Christina Gao, and Stephen Bacchi. 2023. “ This Too Shall Pass: The Performance of ChatGPT‐3.5, ChatGPT‐4 and New Bing in an Australian Medical Licensing Examination.” Medical Journal of Australia 219: 237. 10.5694/mja2.52061 37528548 · DOI
- [8]Miao, Jing, Charat Thongprayoon, Oscar A. Garcia Valencia, Pajaree Krisanapan, Mohammad S. Sheikh, Paul W. Davis, Poemlarp Mekraksakit, et al. 2024. “ Performance of ChatGPT on Nephrology Test Questions.” Clinical Journal of the American Society of Nephrology 19: 35– 43. 10.2215/cjn.0000000000000330 37851468 PMC10843340 · DOI
- [9]Morreel, Stefan, Danny Mathysen, and Veronique Verhoeven. 2023. “ Aye, AI! ChatGPT Passes Multiple‐Choice Family Medicine Exam.” Medical Teacher 45: 665– 666. 10.1080/0142159X.2023.2187684 36905610 · DOI · Paper page
- [11]Miao, Jing, Charat Thongprayoon, and Wisit Cheungpasitporn. 2023. “ Assessing the Accuracy of ChatGPT on Core Questions in Glomerular Disease.” Kidney International Reports 8(8): 1657– 1659. 10.1016/j.ekir.2023.05.014 37547515 PMC10403654 · DOI
- [12]Luykx, Jurjen J., Frank Gerritse, Philippe C. Habets, and Christiaan H. Vinkers. 2023. “ The Performance of ChatGPT in Generating Answers to Clinical Questions in Psychiatry: A Two‐Layer Assessment.” World Psychiatry 22: 479– 480. 10.1002/wps.21145 37713576 PMC10503909 · DOI
- [13]Ali, Stephen R., Thomas D. Dobbs, and Iain S. Whitaker. 2022. “ Using a ChatBot to Support Clinical Decision‐Making in Free Flap Monitoring.” Journal of Plastic, Reconstructive & Aesthetic Surgery 75: 2387– 2440. 10.1016/j.bjps.2022.04.072 35667994 · DOI
- [14]Cascella, Marco, Jonathan Montomoli, Valentina Bellini, and Elena Bignami. 2023. “ Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research Scenarios.” Journal of Medical Systems 47: 33. 10.1007/s10916-023-01925-4 36869927 PMC9985086 · DOI · Paper page
- [16]Fijačko, Nino, Lucija Gosak, Gregor Štiglic, Christopher T. Picard, and Matthew John Douma. 2023. “ Can ChatGPT Pass the Life Support Exams Without Entering the American Heart Association Course?” Resuscitation 185: 109732. 10.1016/j.resuscitation.2023.109732 36775020 · DOI · Paper page
- [17]Zhu, Lingxuan, Weiming Mou, Tao Yang, and Rui Chen. 2023. “ ChatGPT Can Pass the AHA Exams: Open‐Ended Questions Outperform Multiple‐Choice Format.” Resuscitation 188: 109783. 10.1016/j.resuscitation.2023.109783 37349064 · DOI
- [19]Cruz, Rivera Samantha, Liu Xiaoxuan, Chan An‐Wen, K. Denniston Alastair, and J. Calvert Melanie. 2020. “ Guidelines for Clinical Trial Protocols for Interventions Involving Artificial Intelligence: The SPIRIT‐AI Extension.” British Medical Journal 370: m3210. 10.1136/bmj.m3210 32907797 PMC7490785 · DOI · Paper page
- [20]Wang, Hongyan, WeiZhen Wu, Zhi Dou, Liangliang He, and Liqiang Yang. 2023. “ Performance and Exploration of ChatGPT in Medical Examination, Records and Education in Chinese: Pave the Way for Medical AI.” International Journal of Medical Informatics 177: 105173. 10.1016/j.ijmedinf.2023.105173 37549499 · DOI
- [22]Zhu, Lingxuan, Weiming Mou, and Peng Luo. 2024. “ Potential of Large Language Models as Tools Against Medical Disinformation.” JAMA Internal Medicine 184: 450. 10.1001/jamainternmed.2024.0020 38407861 · DOI
- [23]Brouwers, Melissa C., Kate Kerkvliet, and Karen Spithoff, AGREE Next Steps Consortium. 2016. “ The AGREE Reporting Checklist: A Tool to Improve Reporting of Clinical Practice Guidelines.” BMJ 352: i1152. 10.1136/bmj.i1152 26957104 PMC5118873 · DOI
Cited by
Sorted by citations
Wang, S., Zhou, X., Li, C., Wang, S., Li, Y., Tan, T., & Zheng, H. (2025) · Research · doi:10.34133/research.1029 · Cited by 29 · see citing works
Self-citationLin, A., Chen, Z., Jiang, A., Tang, B., Qi, C., Zhu, L., Mou, W., Gan, W., Zeng, D., Xiao, M., et al. (2026) · International Journal of Surgery · doi:10.1097/js9.0000000000003839 · Cited by 2 · see citing works
Self-citationProcedure to Evaluate Artificial Intelligence (AI) Output for Educational and Clinical Tasks
Gilbertson, L., Busch, C. M., Spike, J., DeMoss, K., & Thompson, J. (2026) · Teaching and Learning in Communication Sciences & Disorders · doi:10.61403/2689-6443.1398
- Authors
- Chen, J., Zhu, L., Mou, W., Lin, A., Zeng, D., Qi, C., Liu, Z., Jiang, A., Tang, B., Shi, W., Kahlert, U. D., Zhou, J., Guo, S., Lu, X., Sun, X., Ngo, T., Pu, Z., Jia, B., Jeon, C. O., He, Y., Wu, H., Gu, S., Cheungpasitporn, W., Huang, H., Mao, W., Wang, S., Chen, X., Cabannes, L., Ren, G. S. G., Whitaker, I. S., Ali, S., Cheng, Q., Miao, K., Yuan, S., & Luo, P.
- Affiliations
- Zhujiang Hospital (CN)
- Shanghai Jiao Tong University (CN)
- Nanfang Hospital (CN)
- TU Wien (AT)
- Chinese Academy of Medical Sciences & Peking Union Medical College (CN)
- Second Military Medical University (CN)
- Fudan University (CN)
- University Hospital Magdeburg (DE)
- Department of Oncology The Second Affiliated Hospital of Zunyi Medical University Zunyi China
- GZDLab Chongqing China
- Centre National de la Recherche Scientifique (FR)
- Université Paris Cité (FR)
- Xianghu Laboratory Hangzhou China
- Chung-Ang University (KR)
- University of North Carolina at Chapel Hill (US)
- Tianjin Medical University (CN)
- Duke Medical Center (US)
- Mayo Clinic (US)
- Beth Israel Deaconess Medical Center (US)
- Sun Yat-sen University (CN)
- Singapore General Hospital (SG)
- Morriston Hospital (GB)
- Central South University (CN)
- University of Macau (MO)
- University of Hong Kong - Shenzhen Hospital (CN)
- Corresponding author
- Cheng, Quan
- Miao, Kai
- Yuan, Shuofeng
- Luo, Peng
- ORCID
- Chen, Jinghong
- Mou, Weiming
- Lin, Anqi
- Qi, Chang
- Liu, Zaoqu
- Jiang, Aimin
- Tang, Bufu
- Kahlert, Ulf D.
- Lu, Xiaofan
- Sun, Xu
- Pu, Zhongji
- Jia, Baolei
- Jeon, Che Ok
- Wu, Haiyang
- Gu, Shuqin
- Cheungpasitporn, Wisit
- Huang, Haojie
- Mao, Weipu
- Wang, Shixiang
- Chen, Xin
- Ren, Gerald Sng Gui
- Whitaker, Iain S.
- Ali, Stephen
- Miao, Kai
- Yuan, Shuofeng
- Luo, Peng
- Venue
- iMetaOmics
- Publisher
- Wiley
- Published
- 2024-07-02
- Volume
- 1
- Issue
- 1
- Pages
- e7-e7
- ISSN
- 2996-9506 (ISSN-L)
- 2996-9514
- PMC
- PMC12806434
- OpenAlex
- W4400250431
- Cited by
- 33
- References
- 4
- Citations by year
- 2026: 10 · 2025: 21 · 2024: 2
- FWCI
- 3.09
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- iMetaOmics. (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UNC Libraries
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- generative AI, artificial intelligence in healthcare, evaluation checklist, reporting quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
