Integrating ChatGPT in Orthopedic Education for Medical Undergraduates: Randomized Controlled Trial
Open accessPublished version of a preprint
Abstract
Background: ChatGPT is a natural language processing model developed by OpenAI, which can be iteratively updated and optimized to accommodate the changing and complex requirements of human verbal communication. Objective: The study aimed to evaluate ChatGPT’s accuracy in answering orthopedics-related multiple-choice questions (MCQs) and assess its short-term effects as a learning aid through a randomized controlled trial. In addition, long-term effects on student performance in other subjects were measured using final examination results. Methods: We first evaluated ChatGPT’s accuracy in answering MCQs pertaining to orthopedics across various question formats. Then, 129 undergraduate medical students participated in a randomized controlled study in which the ChatGPT group used ChatGPT as a learning tool, while the control group was prohibited from using artificial intelligence software to support learning. Following a 2-week intervention, the 2 groups’ understanding of orthopedics was assessed by an orthopedics test, and variations in the 2 groups’ performance in other disciplines were noted through a follow-up at the end of the semester. Results: ChatGPT-4.0 answered 1051 orthopedics-related MCQs with a 70.60% (742/1051) accuracy rate, including 71.8% (237/330) accuracy for A1 MCQs, 73.7% (330/448) accuracy for A2 MCQs, 70.2% (92/131) accuracy for A3/4 MCQs, and 58.5% (83/142) accuracy for case analysis MCQs. As of April 7, 2023, a total of 129 individuals participated in the experiment. However, 19 individuals withdrew from the experiment at various phases; thus, as of July 1, 2023, a total of 110 individuals accomplished the trial and completed all follow-up work. After we intervened in the learning style of the students in the short term, the ChatGPT group answered more questions correctly than the control group (ChatGPT group: mean 141.20, SD 26.68; control group: mean 130.80, SD 25.56; P=.04) in the orthopedics test, particularly on A1 (ChatGPT group: mean 46.57, SD 8.52; control group: mean 42.18, SD 9.43; P=.01), A2 (ChatGPT group: mean 60.59, SD 10.58; control group: mean 56.66, SD 9.91; P=.047), and A3/4 MCQs (ChatGPT group: mean 19.57, SD 5.48; control group: mean 16.46, SD 4.58; P=.002). At the end of the semester, we found that the ChatGPT group performed better on final examinations in surgery (ChatGPT group: mean 76.54, SD 9.79; control group: mean 72.54, SD 8.11; P=.02) and obstetrics and gynecology (ChatGPT group: mean 75.98, SD 8.94; control group: mean 72.54, SD 8.66; P=.04) than the control group. Conclusions: ChatGPT answers orthopedics-related MCQs accurately, and students using it excel in both short-term and long-term assessments. Our findings strongly support ChatGPT’s integration into medical education, enhancing contemporary instructional methods. Trial Registration: Chinese Clinical Trial Registry Chictr2300071774; https://www.chictr.org.cn/hvshowproject.html ?id=225740&v=1.0
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In Gan et al. (2024), ChatGPT-4 use raised unassisted 214-item orthopedics exam scores by 0.40 SD compared to conventional internet resources.
“The ChatGPT group scored 0.40 SD higher on an unassisted 214-item orthopedics exam.”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Gan et al. (2024)
Methods > Intervention describes a 214-item orthopedics exam after the ChatGPT-4.0 and internet-review periods. Abstract > Results reports means of 141.20 and 130.80 with SDs of 26.68 and 25.56, yielding an approximately 0.40-SD difference. The copy does not establish that the exam was unassisted, and Results > The Accuracy of the 2 Groups gives a different ChatGPT-group mean (138.46). (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883 · cited 1× · see citing works
“…(Consolidated Standards of Reporting Trials) checklist (Multimedia Appendix 1) and the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and online TeleHealth; version 1.6.1) checklist (Multimedia Appendix 2) were used for this trial [30,31].”
InfluentialSallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 1× · see citing works
“As an auxiliary instrument for medical education, ChatGPT is currently viewed by many academics as a 2-way street [52].”
Ericsson, K. A. (2004) · Academic Medicine · doi:10.1097/00001888-200410001-00022 · Cited by 2,988 · cited 1× · see citing works
“Both general practitioners and specialists must keep relearning medical knowledge to avoid forgetting some minor but essential knowledge points during clinical practice [13,14].”
Chou, R., Qaseem, A., Snow, V., Casey, D., Cross, J. T., Shekelle, P., Owens, D. K., & for the Clinical Efficacy Assessment Subcommittee of the American College of Physicians and the American College of Physicians/American Pain Society Low Back Pain Guidelines Panel* (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-7-200710020-00006 · Cited by 2,718 · see citing works
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · cited 1× · see citing works
“…that while ChatGPT and professional forum answers exhibit comparable levels of logic and accuracy in their specific responses, ChatGPT demonstrates significantly greater empathy than the answers provided by verified individuals on forums, despite the fact that ChatGPT relies on big data [8,19].”
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2012) · International Journal of Surgery · doi:10.1016/j.ijsu.2011.10.001 · Cited by 2,345 · see citing works
Reflection and reflective practice in health professions education: a systematic review
Mann, K., Gordon, J., & MacLeod, A. (2009) · Advances in Health Sciences Education · doi:10.1007/s10459-007-9090-2 · Cited by 2,175 · cited 1× · see citing works
“Moreover, it is also very challenging to realize the transformation of medical knowledge from theory to practice [38,39] because the process of clinical practice requires the interaction between doctors and patients, that is, the immediate output and immediate feedback between them.”
Eysenbach, G., & CONSORT-EHEALTH Group (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1923 · Cited by 1,863 · cited 1× · see citing works
“…(Consolidated Standards of Reporting Trials) checklist (Multimedia Appendix 1) and the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and online TeleHealth; version 1.6.1) checklist (Multimedia Appendix 2) were used for this trial [30,31].”
InfluentialChatGPT and other large language models are double-edged swords
Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., & Moy, L. (2023) · Radiology · doi:10.1148/radiol.230163 · Cited by 1,008 · cited 1× · see citing works
“Although many academics believe that it is a double-edged sword to use ChatGPT as an auxiliary instrument in medical education [20-22], the development and promotion of ChatGPT will undoubtedly fuel the innovation of medical education [23,24].”
What ChatGPT and generative AI mean for science
Stokel-Walker, C., & Van Noorden, R. (2023) · Nature · doi:10.1038/d41586-023-00340-6 · Cited by 828 · cited 1× · see citing works
“ChatGPT, a natural language processing model developed by OpenAI, is based on a sophisticated machine learning algorithm that can be iteratively updated and optimized to accommodate the changing and complex requirements of human verbal communication [1-3].”
Dergaa, I., Chamari, K., Zmijewski, P., & Ben Saad, H. (2023) · Biology of Sport · doi:10.5114/biolsport.2023.125623 · Cited by 796 · cited 1× · see citing works
“However, many academics believe that ChatGPT is merely a plagiarist that lacks initiative and can only collect information, which has become a slang term for slothful people [53-55].”
Vallée, A., Blacher, J., Cariou, A., & Sorbets, E. (2019) · Journal of Medical Internet Research · doi:10.2196/16504 · Cited by 717 · cited 1× · see citing works
“By identifying keywords in queries and analyzing their relevance, search engines provide users with a variety of search results; however, users must frequently determine the authenticity and veracity of the answers [17,18].”
Tools such as ChatGPT threaten transparent science; here are our ground rules for their use
(2023) · Nature · doi:10.1038/d41586-023-00191-1 · Cited by 605 · cited 1× · see citing works
“Although many academics believe that it is a double-edged sword to use ChatGPT as an auxiliary instrument in medical education [20-22], the development and promotion of ChatGPT will undoubtedly fuel the innovation of medical education [23,24].”
The rise of <scp>ChatGPT</scp> : Exploring its potential in medical education
Lee, H. (2024) · Anatomical Sciences Education · doi:10.1002/ase.2270 · Cited by 596 · cited 2× · see citing works
“Although many academics believe that it is a double-edged sword to use ChatGPT as an auxiliary instrument in medical education [20-22], the development and promotion of ChatGPT will undoubtedly fuel the innovation of medical education [23,24].”
AI bot ChatGPT writes smart essays — should professors worry?
Stokel-Walker, C. (2022) · Nature · doi:10.1038/d41586-022-04397-7 · Cited by 543 · cited 1× · see citing works
“However, many academics believe that ChatGPT is merely a plagiarist that lacks initiative and can only collect information, which has become a slang term for slothful people [53-55].”
Internet-based medical education: a realist review of what works, for whom and in what circumstances
Wong, G., Greenhalgh, T., & Pawson, R. (2010) · BMC Medical Education · doi:10.1186/1472-6920-10-12 · Cited by 443 · cited 1× · see citing works
“With the evolution of teaching concepts and learning aids, medical education model has undergone significant transformations over time [40,41].”
The application of large language models in medicine: A scoping review
Meng, X., Yan, X., Zhang, K., Liu, D., Cui, X., Yang, Y., Zhang, M., Cao, C., Wang, J., Wang, X., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109713 · Cited by 414 · cited 1× · see citing works
“Through interactive dialogues with LLMs, students can ask questions, obtain detailed explanations and examples, and deepen their understanding of abstract concepts [49,50].”
The future of medical education and research: Is ChatGPT a blessing or blight in disguise?
Arif, T. B., Munaf, U., & Ul-Haque, I. (2023) · Medical Education Online · doi:10.1080/10872981.2023.2181052 · Cited by 317 · cited 1× · see citing works
“…input of theoretical knowledge in the previous stage of basic medical knowledge education, and its ability to generate different clinical scenarios according to different diseases can assist medical students in constructing a bridge between professional medical theory and clinical practice [27-29].”
ChatGPT: Evolution or revolution?
Gordijn, B., & Have, H. T. (2023) · Medicine, Health Care and Philosophy · doi:10.1007/s11019-023-10136-0 · Cited by 317 · see citing works
Virtual and augmented reality for biomedical applications
Venkatesan, M., Mohan, H., Ryan, J., Schürch, C., Nolan, G., Frakes, D., & Coşkun, A. (2021) · Cell Reports Medicine · doi:10.1016/j.xcrm.2021.100348 · Cited by 271 · cited 1× · see citing works
“With the evolution of teaching concepts and learning aids, medical education model has undergone significant transformations over time [40,41].”
Health, T. L. D. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00023-7 · Cited by 257 · see citing works
The Role of Large Language Models in Medical Education: Applications and Implications
Safranek, C. W., Sidamon-Eristoff, A. E., Gilson, A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/50945 · Cited by 233 · cited 2× · see citing works
“Moreover, LLMs can generate engaging learning materials, such as clinical case studies, question-and-answer exercises, and knowledge summaries, to enhance student participation and learning motivation [49,51].”
Cigarette Smoking Increases the Risk for Rotator Cuff Tears
Baumgarten, K. M., Gerlach, D., Galatz, L., Teefey, S., Middleton, W., Ditsios, K., & Yamaguchi, K. (2010) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-009-0781-2 · Cited by 226 · cited 1× · see citing works
“ChatGPT, a natural language processing model developed by OpenAI, is based on a sophisticated machine learning algorithm that can be iteratively updated and optimized to accommodate the changing and complex requirements of human verbal communication [1-3].”
Rouleau, G., Gagnon, M., Côté, J., Payne-Gagnon, J., Hudson, É., Dubois, C., & Bouix‐Picasso, J. (2019) · Journal of Medical Internet Research · doi:10.2196/15118 · Cited by 218 · cited 1× · see citing works
“By identifying keywords in queries and analyzing their relevance, search engines provide users with a variety of search results; however, users must frequently determine the authenticity and veracity of the answers [17,18].”
The Advent of Generative Language Models in Medical Education
Karabacak, M., Ozkara, B. B., Margetis, K., Wintermark, M., & Bisdas, S. (2023) · JMIR Medical Education · doi:10.2196/48163 · Cited by 208 · cited 3× · see citing works
“Hence, as a supplementary learning aid for medical students, ChatGPT might be able to realize nonprocedural multidirectional interaction and continuously enhance students’ knowledge systems [29] via immediate feedback.”
Educational Technologies in Problem-Based Learning in Health Sciences Education: A Systematic Review
Jin, J., & Bridges, S. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3240 · Cited by 198 · cited 1× · see citing works
“To enhance students’ capacity for self-exploration, education sector constructs problem-based learning and promotes interaction during the learning process [44,45].”
How doctors learn: physicians' self-directed learning episodes.
Slotnick, H. (1999) · Academic medicine : journal of the Association of American Medical Colleges · doi:10.1097/00001888-199910000-00014 · Cited by 192 · see citing works
ChatGPT’s inconsistent moral advice influences users’ judgment
Krügel, S., Ostermaier, A., & Uhl, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-31341-0 · Cited by 183 · cited 1× · see citing works
“The major reason why academics are worried about ChatGPT’s inadequacies in medical education is that its potent text creation feature brings the risk of making students too reliant on it as a writing tool [25,26].”
Xu, J. H., Lio, A., Dhaliwal, H., Andrei, S., Balakrishnan, S., Nagani, U., & Samadder, S. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.06.070 · Cited by 151 · cited 1× · see citing works
“As society, science, and technology advance, it is inevitable that students will take the initiative to adapt and combine their own learning mode with the updates and iteration of learning media and supplementary learning tools [56].”
Effects of Virtual Reality training on medical students’ learning motivation and competency
Sattar, M. U., Palaniappan, S., Lokman, A., Hassan, A., Shah, N., & Riaz, Z. (2019) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.35.3.44 · Cited by 144 · cited 1× · see citing works
“The traditional education model consists of a 1-way transmission from professors to students [42].”
Malau‐Aduli, B. S., Lee, A. Y. S., Cooling, N., Catchpole, M., Jose, M., & Turner, R. C. (2013) · BMC Medical Education · doi:10.1186/1472-6920-13-139 · Cited by 139 · cited 1× · see citing works
“Due to the vast medical knowledge system, diverse content, and lengthy learning cycle, it is difficult to accomplish a quality breakthrough in medical education [11,12].”
A conversation with ChatGPT on the role of computational systems biology in stem cell research
Cahan, P., & Treutlein, B. (2023) · Stem Cell Reports · doi:10.1016/j.stemcr.2022.12.009 · Cited by 125 · see citing works
Exploring ChatGPT for information of cardiopulmonary resuscitation
Ahn, C. (2023) · Resuscitation · doi:10.1016/j.resuscitation.2023.109729 · Cited by 112 · cited 3× · see citing works
“Notably, ChatGPT as a supplementary aid can assist users in collecting the correct target information more efficiently, thereby enhancing work and learning efficiency [5].”
InfluentialPerformance of ChatGPT on the Plastic Surgery Inservice Training Examination
Gupta, R., Herzog, I., Park, J. B., Weisberger, J., Firouzbakht, P., Ocon, V., Chao, J., Lee, E. S., & Mailey, B. A. (2023) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad128 · Cited by 109 · cited 3× · see citing works
“ChatGPT was also performed at a level of 54.96% accuracy on the Plastic Surgery Inservice Training Examination, which consisted of 242 questions [7].”
InfluentialUse of ChatGPT: What does it mean for biology and environmental science?
Agathokleous, E., Saitanis, C. J., Fang, C., & Yu, Z. (2023) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2023.164154 · Cited by 105 · see citing works
Patient education on the internet: opportunities and pitfalls.
Ullrich, P., & Vaccaro, A. (2002) · Spine · doi:10.1097/00007632-200204010-00019 · Cited by 99 · see citing works
Thorborg, K., Reiman, M., Weir, A., Kemp, J., Serner, A., Mosler, A., & Hölmich, P. (2018) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2018.7850 · Cited by 97 · cited 1×
“In addition to textual information such as the patient’s complaint, current history, and prior history, the clinician’s physical examination and imaging must be incorporated into the clinical diagnosis and treatment [34,35].”
Maniar, K. P., Arva, N. C., Blanco, L., Mao, Q.-W., Morency, E., Rodriguez, R., Wolniak, K., Yaseen, N., & Nayar, R. (2019) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2018-0467-ra · Cited by 96 · cited 1×
“In addition to cramming for examinations and studying for examinations, medical students must devote significant effort to accumulating and organizing information and making causal links between different domains of knowledge [36,37].”
Revolutionizing Medical Education: Can ChatGPT Boost Subjective Learning and Expression?
Seetharaman, R. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01957-w · Cited by 91 · cited 1× · see citing works
“Although many academics believe that it is a double-edged sword to use ChatGPT as an auxiliary instrument in medical education [20-22], the development and promotion of ChatGPT will undoubtedly fuel the innovation of medical education [23,24].”
Gan, W., Mok, T. N., Chen, J., She, G., Zha, Z., Wang, H., Li, H., Li, J., & Zheng, X. (2023) · BMC Medical Education · doi:10.1186/s12909-022-03992-6 · Cited by 84 · cited 1× · see citing works
“Simultaneously, virtual reality–assisted instruction is being developed to enhance medical students’ perceptual comprehension of medical knowledge through the interaction and immediate feedback of programmed audiovisual operations [46].”
Self-citationDaily briefing: will ChatGPT kill the essay assignment?
Graham, F. (2022) · Nature · doi:10.1038/d41586-022-04437-2 · Cited by 79 · see citing works
Ghosh, A., & Bir, A. (2023) · Cureus · doi:10.7759/cureus.37023 · Cited by 67 · cited 1× · see citing works
“5 in terms of language comprehension, context comprehension, generation speed, and interpretability [4].”
Temsah, M.-H., Alhuzaimi, A. N., Almansour, M., Aljamaan, F., Alhasan, K., Batarfi, M. A., Altamimi, I., Alharbi, A., Alsuhaibani, A. A., Alwakeel, L., et al. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02072-0 · Cited by 66 · cited 1× · see citing works
“These advanced AI tools can provide personalized learning support, helping students better understand complex medical concepts such as anatomy [47], physiology [48], and biochemistry [48].”
ChatGPT as a tool for medical education and clinical decision-making on the wards: Case study
Skryd, A., & Lawrence, K. (2024) · JMIR Formative Research · doi:10.2196/51346 · Cited by 63 · cited 1× · see citing works
“Designing a specific learning application based on the network can increase the output of knowledge points, but it may not be able to respond promptly to the personalized user query input [16].”
ChatGPT and the future of medical education
Feng, S., & Shen, Y. (2023) · Academic Medicine · doi:10.1097/acm.0000000000005242 · Cited by 61 · see citing works
Alfertshofer, M. G., Hoch, C. C., Funk, P. F., Hollmann, K., Wollenberg, B., Knoedler, S. A., & Knoedler, L. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03338-3 · Cited by 60 · see citing works
Sauder, M., Tritsch, T., Rajput, V., Schwartz, G., & Shoja, M. M. (2024) · Cureus · doi:10.7759/cureus.51961 · Cited by 41 · cited 2× · see citing works
“Moreover, LLMs can generate engaging learning materials, such as clinical case studies, question-and-answer exercises, and knowledge summaries, to enhance student participation and learning motivation [49,51].”
Borchert, R. J., Hickman, C. R., Pepys, J., & Sadler, T. J. (2023) · JMIR Medical Education · doi:10.2196/48978 · Cited by 39 · cited 1× · see citing works
“Numerous researchers have reported that ChatGPT can achieve satisfactory results on multidisciplinary medical practitioner examinations in a variety of countries as well as provide detailed evidence-based explanations when responding to input clinical scenarios [7-10].”
Continuing Medical Education and Lifelong Learning
McAdams, C. D., & McNally, M. M. (2021) · Surgical Clinics of North America · doi:10.1016/j.suc.2021.05.015 · Cited by 38 · cited 1× · see citing works
“In addition to cramming for examinations and studying for examinations, medical students must devote significant effort to accumulating and organizing information and making causal links between different domains of knowledge [36,37].”
Review of Cerebrospinal Fluid Physiology and Dynamics: A Call for Medical Education Reform
Atchley, T. J., Vukić, B., Vukić, M., & Walters, B. C. (2022) · Neurosurgery · doi:10.1227/neu.0000000000002000 · Cited by 32 · cited 1× · see citing works
“Due to the vast medical knowledge system, diverse content, and lengthy learning cycle, it is difficult to accomplish a quality breakthrough in medical education [11,12].”
Hilty, D. M., Turvey, C., & Hwang, T. (2018) · Current Psychiatry Reports · doi:10.1007/s11920-018-0878-y · Cited by 32 · cited 1× · see citing works
“Both general practitioners and specialists must keep relearning medical knowledge to avoid forgetting some minor but essential knowledge points during clinical practice [13,14].”
Luke, W. A. N. V., Seow Chong, L., Ban, K. H., Wong, A. H., Zhi Xiong, C., Shuh Shing, L., Taneja, R., Samarasekera, D. D., & Yap, C. T. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2308779 · Cited by 24 · cited 1× · see citing works
“These advanced AI tools can provide personalized learning support, helping students better understand complex medical concepts such as anatomy [47], physiology [48], and biochemistry [48].”
Xue, Z., Zhang, Y., Gan, W., Wang, H., She, G., & Zheng, X. (2024) · Journal of Medical Internet Research · doi:10.2196/50882 · Cited by 22 · cited 1× · see citing works
“…that while ChatGPT and professional forum answers exhibit comparable levels of logic and accuracy in their specific responses, ChatGPT demonstrates significantly greater empathy than the answers provided by verified individuals on forums, despite the fact that ChatGPT relies on big data [8,19].”
Self-citationMilbourn, B. T., Black, M. H., Afsharnejad, B., Snyman, Z., Baker-Young, E., Thompson, C., McGarry, S., Scott, M., Clifford, R. M., Zimmermann, F., et al. (2021) · Contemporary Clinical Trials · doi:10.1016/j.cct.2021.106645 · Cited by 17 · cited 1× · see citing works
“Mass Open Online Course, a nonfixed multidirection input model, has emerged progressively with the advent of the internet [43].”
Multiple-Choice Tests: A–Z in Best Writing Practices
Gupta, V., Williams, E. R., & Wadhwa, R. (2021) · Psychiatric Clinics of North America · doi:10.1016/j.psc.2021.03.008 · Cited by 15 · cited 1× · see citing works
“The MCQs test has grown to be one of the most popular tools for objectively evaluating medical theoretical knowledge [32,33].”
Efficient validation of teaching and learning using multiple-choice exams.
Desantis, M., & Mckean, T. (2003) · Advances in Physiology Education · doi:10.1152/advan.00016.2001 · Cited by 10 · cited 1×
“The MCQs test has grown to be one of the most popular tools for objectively evaluating medical theoretical knowledge [32,33].”
Flagged first, then by citations
1 of the 57 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [44]Al-Azri H Ratnapalan S Problem-based learning in continuing medical education: review of randomized controlled trials Can Fam Physician 2014 60 2 157 165 24522680 60/2/157 24522680 PMC3922562
Cited by
Sorted by citations
Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies
Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2025) · Computers & Education · doi:10.1016/j.compedu.2024.105224 · Cited by 531 · see citing works
Wang, J., & Fan, W. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04787-y · Cited by 322 · see citing works
“Some scholars argue for its bene fi ts (Boudouaia et al. 2024; Gan et al. 2024; Zhou and Kim, 2024).”
RetractedPham, T. D., Karunaratne, N., Exintaris, B., Liu, D., Lay, T., Yuriev, E., & Lim, A. (2025) · Medical Education · doi:10.1111/medu.15746 · Cited by 100 · see citing works
Chen, S., & Cheung, A. (2025) · Educational Research Review · doi:10.1016/j.edurev.2025.100737 · Cited by 117 · see citing works
How can artificial intelligence transform the training of medical students and physicians?
Ning, Y., Ong, J. C. L., Cheng, H., Wang, H., Ting, D. S. W., Tham, Y. C., Wong, T. Y., & Liu, N. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100900 · Cited by 52 · see citing works
Li, J., Yin, K., Wang, Y., Jiang, X., & Chen, D. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07750-2 · Cited by 62 · see citing works
“For allocation concealment, only one study demonstrated low risk [24], while others were classified as unclear due to insufficient reporting of concealment methods.”
The Impact of <scp>ChatGPT</scp> on Students' Academic Achievement: A Meta‐Analysis
Liu, Z., Zuo, H., & Lu, Y. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70096 · Cited by 44 · see citing works
Roveta, A., Castello, L. M., Massarino, C., Francese, A., Ugo, F., & Maconi, A. (2025) · AI · doi:10.3390/ai6090227 · Cited by 42 · see citing works
“In tutoring, generative tools such as ChatGPT produce quizzes and educational materials, judged useful by students, in some cases improving test performance [19,20].”
InfluentialHuang, S., Wen, C., Bai, X., Li, S., Wang, S., Wang, X., & Yang, D. (2025) · Journal of Medical Internet Research · doi:10.2196/68538 · Cited by 33 · see citing works
“Many studies have reported on the potential of ChatGPT for passing examinations [1], learning anatomy, and understanding emerging trends [6,7], making it an important tool in areas such as medical education.”
InfluentialGan, W., Ouyang, J., She, G., Xue, Z., Zhu, L., Lin, A., Mou, W., Jiang, A., Chang, Q., Cheng, Q., et al. (2025) · International Journal of Surgery · doi:10.1097/js9.0000000000002223 · Cited by 35 · see citing works
“It can even pass professional exams in various countries and fields, including orthopedics [ 25 ] .”
Self-citationLin, Y., Luo, Z., Ye, Z., Zhong, N., Zhao, L., Zhang, L., Li, X., Chen, Z., & Chen, Y. (2025) · JMIR Medical Education · doi:10.2196/71125 · Cited by 31 · see citing works
“This includes support for exam preparation [55,65-68], optimizing learning paths and review strategies [52,69-71], clarifying medical concepts [68, 72-76], and assisting in the development of tailored career plans [77].”
Undergraduate students’ learning outcomes with ChatGPT: A meta-analytic study
Mo, F., Huang, J., Yang, Y., Ozen, Z., Maeda, Y., & Olenchak, F. R. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100536 · Cited by 18 · see citing works
Self-citationPeng, J., Zhang, H., Tu, X., Zhang, X., Wu, Q., Wang, Y., & Deng, X. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07770-y · Cited by 20 · see citing works
“The majors of the students in these studies were clinical medicine and medical imaging, and the duration of the studies ranged from 2 to 52 weeks in nine studies (three studies were not mentioned [22, 25, 31]).”
Vaira, L., Lechien, J., Maniaci, A., Vito, A. D., Mayo-Yáñez, M., Troise, S., Consorti, G., Chiesa-Estomba, C., Cammaroto, G., Radulesco, T., et al. (2025) · Medicina · doi:10.3390/medicina61081379 · Cited by 15 · see citing works
“Studies suggest that these models can provide relevant insights and assist in decision-making processes [21–26] within specialized medical domains, although their diagnostic accuracy in image-based tasks remains an open question.”
ChatGPT as a Learning Tool for Medical Students: Results From a Randomized Controlled Trial
Kalam, K. A., Masoud, F. D., Muntaser, A., Ranga, R., Geng, X., & Goyal, M. K. (2025) · Cureus · doi:10.7759/cureus.85767 · Cited by 13 · see citing works
“A randomized controlled trial conducted in Chinese orthopedic education demonstrated that the use of ChatGPT led to improved short-term performance and examination scores when compared to traditional learning tools [12].”
Quantum leap in medical mentorship: exploring ChatGPT’s transition from textbooks to terabytes
Chokkakula, S., Chong, S., Yang, B., Jiang, H., Yu, J., Han, R., Attitalla, I. H., Yin, C., & Zhang, S. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1517981 · Cited by 12 · see citing works
Liu, Y., Li, H., Ouyang, J., Xue, Z., Wang, M., He, H., Song, B., Zheng, X., & Gan, W. (2025) · JMIR Perioperative Medicine · doi:10.2196/70047 · Cited by 14 · see citing works
“…the potential of LLMs in clinical application scenarios, including professional licensing examinations in various countries and regions [3-5], answering public health questions [6,7], analyzing radiological images [8], disease screening [9], disease diagnosis [10], and discipline education [11].”
Self-citationLi, R., & Wu, T. (2025) · Advances in Medical Education and Practice · doi:10.2147/amep.s497020 · Cited by 11 · see citing works
Wang, C., Sun, N., Pan, X., Liu, P., Ma, Z., Wang, W., & Jin, Y. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09320-6 · Cited by 14 · see citing works
“Medical education currently lacks a shared framework for deciding how GenAI should be woven into courses, what resources are needed, and what students should learn about AI itself [14, 16].”
Lai, N. M., Lim, Y. S., Thein, W. M., Bhargava, P., Thomas, P., & Ong, Q. C. (2026) · JMIR Medical Education · doi:10.2196/88933 · Cited by 10 · see citing works
“…subtype comparison was LLM personalized learning aids, which appeared to show an important positive effect but with substantial heterogeneity (12 studies, n=955; SMD 0.53, 95% CI 0.13 to 0.94; I ²=86%, 95% PI −0.81 to 1.88; very low certainty; Figure 5 [53,63, 67,70-72, 80, 82, 84, 86,90,113]”
InfluentialBasil, M., Ahmed, W., Hajeomar, R., Strawbridge, J., Lynch, M. R., & Mukhalalati, B. (2026) · BMC Medical Education · doi:10.1186/s12909-025-08527-3 · Cited by 13 · see citing works
Aksoy, E. (2025) · Obesity Surgery · doi:10.1007/s11695-025-07794-9 · Cited by 8 · see citing works
Anusitviwat, C., Suwannaphisit, S., Bvonpanttarananon, J., & Tangtrakulwanich, B. (2025) · JMIR Formative Research · doi:10.2196/75607 · Cited by 6 · see citing works
“The results of this study are consistent with a prior study showing that advanced LLMs demonstrated a high level of accuracy in answering orthopedics-related MCQs [16].”
The Rise of Artificial Intelligence in Orthopedics: A Bibliometric and Visualization Analysis
Wu, S.-H., Miao, Y., Mei, J., & Xiong, S.-R. (2025) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s536633 · Cited by 6
“11 Additionally, a systematic review of large language models in orthopedics highlighted the potential applications of ChatGPT in various domains, including clinical practice, education, research, and management.”
Qianqian, Y., Wang, S., Liu, Y., Luo, J., Deng, Z., Shao, L., & Jiang, Y. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07394-2 · Cited by 4 · see citing works
“ChatGPT has become increasingly integrated into medical education as a tool for fostering critical thinking, with studies showing its effectiveness in providing a vast amount of information and simplifying complex concepts [38–41].”
Nieves-López, B., Wing, C. W., Springer, B. D., & Aziz, K. T. (2025) · Arthroplasty Today · doi:10.1016/j.artd.2025.101772 · Cited by 5 · see citing works
“The adoption of artificial intelligence (AI) tools, such as ChatGPT, in medicine has demonstrated significant benefits not only in educational contexts but also for health-care professionals and patient interactions [3-5].”
He, Y., Wang, Y., Lai, L., Liu, N., Hong, Y., Chen, P., & Zhang, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-20210-7 · Cited by 3 · see citing works
Briones-Zamora, K. H., Diaz-Mora, P., Yépez, I. N. F., Salas, G. G. S., Briones-Zamora, A. D., Farfán, A. Q., & Briones-Claudett, K. H. (2026) · Journal of Surgical Education · doi:10.1016/j.jsurg.2025.103867 · Cited by 3
Exploring the role of artificial intelligence in orthopedic medical education: A narrative review
Das, L. S., Das, D., Chandrakar, D., Bhavani, P., Dubepuria, A., & Barik, S. (2025) · Journal of Clinical Orthopaedics and Trauma · doi:10.1016/j.jcot.2025.103100 · Cited by 4 · see citing works
Sousa, S. A., Bukhari, S., Tang, M., Yung, K.-M., Hobi, S., & Steagall, P. (2026) · Journal of Veterinary Medical Education · doi:10.21125/inted.2026.0866 · Cited by 2
Qing, L., Xue, W. X., Meng, L., Zhu, Y., & Li, J. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1640684 · Cited by 2 · see citing works
TAY, Y. N., NEO, N. W. S., & SIAH, C. J. R. (2026) · Nurse Education Today · doi:10.1016/j.nedt.2026.107322 · Cited by 4 · see citing works
Wu, F.-T., Li, X., Zeng, Y., Tang, Y., Xiao, Z.-H., Yang, S.-Q., Wu, K., Liu, S.-D., & Li, A.-M. (2026) · Journal of Medical Internet Research · doi:10.2196/93042 · Cited by 1 · see citing works
“Large language models (LLMs) are increasingly applied in health care, including patient education, clinical communication, and decision support [9-12].”
Peng, Y., Ma, M., Yang, Y., Wu, J., & Peng, Y. (2026) · Anatomical Sciences Education · doi:10.1002/ase.70297 · Cited by 1 · see citing works
“13 However, these findings should not be directly generalized to VM-based histology learning, where students are required to interpret microscopic images and justify identifications using visible tissue and cellular features.”
He, Z., Yang, L., Liang, Z.-T., Li, X.-F., & Du, J. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02837-6 · Cited by 1
Embang, J. E. G., Tan, Y. H. V., Hooi, S. C., Wong, A. H., & Wong, L. (2026) · Advances in Health Sciences Education · doi:10.1007/s10459-026-10547-7 · Cited by 1 · see citing works
Ma, X., Liu, X., & Sun, H. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1813108 · Cited by 1 · see citing works
Dave, R., Vediya, N., Sharma, N., Shah, A., Whelan, D. B., & Wolfstadt, J. I. (2026) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-25-01242 · Cited by 1 · see citing works
Li, X., Du, J., Liu, Y., Yin, H., & Liu, H. (2026) · Big Data Mining and Analytics · doi:10.26599/bdma.2025.9020086 · Cited by 1 · see citing works
Yao, Y., Geng, J.-M., Li, W., Li, S.-L., & Xu, Y. (2026) · Academic Radiology · doi:10.1016/j.acra.2026.03.026 · Cited by 1
Arzideh, K., Hosch, R., Turki, A. T., Eryılmaz, B., Bahn, M., Schäfer, H., Idrissi-Yaghir, A., Khattab, S., Dada, A., Baba, H. A., et al. (2026) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00254 · Cited by 1 · see citing works
Generative AI as Librarian: A New Model for Surgical Education.
Olmstead, J. A., Yousef, A., Eskander, P., & Watson, D. (2026) · Journal of Surgical Education · doi:10.1016/j.jsurg.2025.103862 · Cited by 1 · see citing works
Gupta, D., & Chaudhuri, A. (2026) · Journal of Marine Medical Society · doi:10.4103/jmms.jmms_144_25 · Cited by 1 · see citing works
Liao, Z., Huang, J., Liu, Y., Li, F., Tang, L., Cong, L., Wang, H., & Luo, S. (2025) · Aesthetic Plastic Surgery · doi:10.1007/s00266-025-05314-9 · Cited by 1 · see citing works
“This approach treats each question and the LLM’s response in isolation, precluding any information or context carry-over from previous interactions [26, 27].”
Self-citationWang, S.-S., Gao, H., Lin, P., Qian, T.-C., Du, Y., & Xu, L. (2025) · World Journal of Gastrointestinal Oncology · doi:10.4251/wjgo.v17.i10.109792 · Cited by 1 · see citing works
“This limitation is also consistent with those reported in previous studies[26,36,37].”
Experimental Evidence on the Learning Impact of Generative AI
Contractor, Z., & Reyes, G. (2026) · arXiv · doi:10.48550/arxiv.2607.08849
Pienkowska, A., Ng, O., Hou, J., & Han, S. P. (2026) · Journal of Medical Internet Research · doi:10.2196/96099
“Five studies used a randomized controlled trial study design where the GenAI-based intervention was compared with non–GenAI-based but equivalent educational activities [30,52,55,58,59].”
Torab-Miandoab, A., Ghafourifard, M., Habibi-Chenaran, S., & Maghalian, M. (2026) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205261491865
Orhan, S., Dinçal, T., Arzu, U., Özdamar, İ., & Gencer, B. (2026) · Anatolian journal of emergency medicine · doi:10.54996/anatolianjem.1772154
İlleez, Ö. G., Giray, E., Ateş, U., Keles, A., Turkyilmaz, S. G., & Yüksel, M. (2026) · Work · doi:10.1177/10519815261488805
Effectiveness of ChatGPT and DeepSeek in Urology Medical Education: Randomized Controlled Trial
Yang, W., Xu, T., Wei, J., Zheng, W., Yan, W., Wang, J., Chu, G., & Niu, H. (2026) · Journal of Medical Internet Research · doi:10.2196/89315
“This suggests that ChatGPT o3-mini is not yet suitable as an auxiliary learning tool for treatment knowledge, but it is more appropriate for teaching fundamental medical concepts [51].”
InfluentialGenerative artificial intelligence and the transformation of medical education: a scoping review
Zhao, J., Yang, K., Gao, H., & Jiang, G. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1907589 · Cited by 1 · see citing works
Zhang, C., & Xu, J. (2026) · ANZ Journal of Surgery · doi:10.1111/ans.70972
Ordoñez, M. E. R., Castillo, N. P. C., Muñoz, G. G. L., Salazar, J. B., & Morales, B. (2026) · Saber Estratégico Internacional · doi:10.70577/s.e.v4i3.83
“…la https://doi.org/10.70577/s.e.v4i3.83 IA mediante soluciones por tareas, retroalimentación y aprendizaje activo independiente; aunque los resultados de la Educación Superior no permiten extraer la información que puede ser útil para la Educación Básica (Gan et al., 2024; Zeng et al., 2025).”
InfluentialLin, S., Zhang, C., Zhong, F., Liao, X., Gong, L., & Wang, D. (2026) · Cancer Medicine · doi:10.1002/cam4.72188
Gao, W., Zhu, W.-J., Li, S., Fan, D.-Y., Peng, Z., He, B.-B., & Wu, S.-L. (2026) · Journal of Artificial Intelligence and Information · doi:10.66069/ojspub.1137260809
“…experimental group reported significantly higher satisfaction than the control group, consistent with the findings of Huang et al. [10], Gan et al. [11], and Ba et al. [12], suggesting that GenAI-assisted teaching enhances students ’ learning initiative and sense of achievement, and that its…”
Artificial intelligence in medical education: a narrative review across four functional domains
Zarei, M., Mozaffari, M., & Hajiani, Y. (2026) · Currents in Pharmacy Teaching and Learning · doi:10.1016/j.cptl.2026.102752
ChatGPT as a Learning Assistant in Medical Education Student Perceptions and Academic Performance
Lakshmi, R., Srevatsa, K., & Moudgalya, A. S. (2026) · International Journal of Medical and Biomedical Studies · doi:10.32553/7757w633
Liu, X., Pan, J., Zhang, C., Wang, X., Wan, J., & Wang, Q. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09879-0 · Cited by 2 · see citing works
Li, F., Su, Y., Xu, W., Fu, Y., & Lu, L. (2026) · Global Medical Education · doi:10.1515/gme-2025-0027
“Digital patients generated by large language models (e.g., ChatGPT, bingchat, ChatZOC, EyeGPT) [25–27] are deeply embedded in teaching scenarios, realizing a low-cost, scalable model that reduces training costs while alleviating teaching challenges caused by uneven distribution of high-quality…”
AI-Generated Microlearning for Plastic Surgery Residency: Single-Arm Pre-Post Feasibility Study
Drysch, M., Schmidt, S. V., Reinkemeier, F., Puscz, F., Fiedler, A., Fueth, M., Lehnhardt, M., Sogorski, A., & Wallner, C. (2026) · JMIR Medical Education · doi:10.2196/88695
“A randomized trial in orthopedic education further reported knowledge gains when students used an LLM as a study aid [19].”
Cho, J. W., & Sekaran, A. (2026) · Cureus · doi:10.7759/cureus.112413 · Cited by 1 · see citing works
“LLMs may serve as a useful learning tool to supplement revision by providing explanations and individualized feedback [15].”
Yılmaz, H. Ö., Duman, E., & Şahin, K. (2026) · Journal of Nutrition · doi:10.1016/j.tjnut.2026.101739
Bayırlı, A. B., Uytun, M., Erdem, R., & Genç, Y. S. (2026) · Mugla Journal of Science and Technology · doi:10.22531/muglajsci.1862743
Qiao, M., Zhang, H., Shen, Y., Zhang, T., Liu, S. S., Lou, Y., Yang, L., Song, C., & Liu, J. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1854814
Top 5 Surgical Education Innovations for Low-Resource Settings Over the Last 50 years.
Choubey, U., Mbanje, C., Tak, R., Davies, J., & Rivera, M. (2026) · World Journal of Surgery · doi:10.1002/wjs.70443
Kuşçu, A. İ., Çınarer, G., & Kuşçu, S. (2026) · Anatolian Current Medical Journal · doi:10.38053/acmj.1848512
Artificial Intelligence and Medical Education: A Double-Edged Sword
Zangmo, R., Khurshid, M., Ramu, S., & Katti, K. M. (2026) · Journal of Obstetrics and Gynecology of India · doi:10.1007/s13224-026-02401-y
Singh, S., Kaur, A., & Singh, H. (2026) · IIUM Medical Journal Malaysia · doi:10.31436/imjm.v25i01.3215
Olanrewaju, M. A., & Kolawole Richard, O. J. O. (2026) · European Journal of Education Studies · doi:10.46827/ejes.v13i2.6542 · Cited by 1 · see citing works
“…have claimed that ChatGPT has benefits and the ability to support students in a variety of teaching-learning situations (Boudouaia et al. , 2024; Gan et al. , 2024; Zhou & Kim, 2024; Hsu, 2024; Maurya, 2024; Nugroho et al. , 2023), significantly boosting productivity and creativity in academic…”
Shi, X., Li, Y., & Yin, C. (2026) · Digital Health · doi:10.1177/20552076261431491
“15 – 17 Similar to AI training, the knowledge of medical personnel is enriched by the volume of clinical challenges they encounter.”
Safdar, A., Iftikhar, H., Arshad, H. A., Anwar, M., Khan, A. Q., & Kanwal, S. (2025) · VAWKUM Transactions on Computer Sciences · doi:10.21015/vtcs.v13i2.2278
Xie, G., Liao, J., Tang, X., Fu, H., Liu, D., Deng, L., Wang, T., & Hong, X. (2025) · BMC Medical Education · doi:10.1186/s12909-026-09458-3 · Cited by 3 · see citing works
“Among the included studies, follow-up data were reported in only 8 studies [24, 26, 32, 54, 58, 62, 63, 67], with follow-up duration ranging from 3 days to 12 weeks.”
Automated ICD-O-3 Coding of Real-World Pathology Reports Using Self-Hosted Large Language ModelsPreprint
Arzideh, K., Hosch, R., Turki, A. T., Erilymaz, B., Bahn, M., Schäfer, H., Idrissi-Yaghir, A., Khattab, S., Dada, A., Baba, H. A., et al. (2025) · medRxiv · doi:10.1101/2025.07.23.25332040
- Authors
- Gan, W., Ouyang, J., Li, H., Xue, Z., Zhang, Y., Dong, Q., Huang, J., Zheng, X., & Zhang, Y.
- Affiliations
- Jinan University (CN)
- Beijing Jishuitan Hospital (CN)
- Corresponding author
- Zhang, Yiyi
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-08-20
- Volume
- 26
- Pages
- e57037
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11372336
- OpenAlex
- W4401729352
- Cited by
- 73
- References
- 57
- Citations by year
- 2026: 39 · 2025: 32 · 2024: 2
- FWCI
- 6.82
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Jinan University
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Keywords
- ChatGPT, large language models, artificial intelligence in medical education, orthopaedic education, undergraduate medical students, multiple-choice question accuracy, student performance, transfer of learning, ChatGPT-4, ChatGPT-3.5
- MeSH terms
- Generative Artificial Intelligence, Education, Medical, Undergraduate, Educational Measurement, Female, Humans, Male, Natural Language Processing, Orthopedics, Students, Medical, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
