Performance of ChatGPT on Chinese national medical licensing examinations: a five-year examination evaluation study for physicians, pharmacists and nurses
Open accessPublished version of a preprint
Abstract
Background: Large language models like ChatGPT have revolutionized the field of natural language processing with their capability to comprehend and generate textual content, showing great potential to play a role in medical education. This study aimed to quantitatively evaluate and comprehensively analysis the performance of ChatGPT on three types of national medical examinations in China, including National Medical Licensing Examination (NMLE), National Pharmacist Licensing Examination (NPLE), and National Nurse Licensing Examination (NNLE). Methods: We collected questions from Chinese NMLE, NPLE and NNLE from year 2017 to 2021. In NMLE and NPLE, each exam consists of 4 units, while in NNLE, each exam consists of 2 units. The questions with figures, tables or chemical structure were manually identified and excluded by clinician. We applied direct instruction strategy via multiple prompts to force ChatGPT to generate the clear answer with the capability to distinguish between single-choice and multiple-choice questions. Results: ChatGPT failed to pass the accuracy threshold of 0.6 in any of the three types of examinations over the five years. Specifically, in the NMLE, the highest recorded accuracy was 0.5467, which was attained in both 2018 and 2021. In the NPLE, the highest accuracy was 0.5599 in 2017. In the NNLE, the most impressive result was shown in 2017, with an accuracy of 0.5897, which is also the highest accuracy in our entire evaluation. ChatGPT’s performance showed no significant difference in different units, but significant difference in different question types. ChatGPT performed well in a range of subject areas, including clinical epidemiology, human parasitology, and dermatology, as well as in various medical topics such as molecules, health management and prevention, diagnosis and screening. Conclusions: These results indicate ChatGPT failed the NMLE, NPLE and NNLE in China, spanning from year 2017 to 2021. but show great potential of large language models in medical education. In the future high-quality medical data will be required to improve the performance.
AI snapshot
Written from the paper’s full text.
TL;DR
ChatGPT (GPT-3.5) was evaluated on Chinese national licensing examinations for physicians, pharmacists, and nurses from 2017 to 2021 and failed to reach the 0.6 accuracy pass threshold in all three exam types. Its highest accuracy was 0.5897 on the 2017 National Nurse Licensing Examination, while performance varied by unit, question type, subject, and topic.
Key findings
- ChatGPT failed to pass the 0.6 accuracy threshold in any of the three examinations over five years. Highest accuracies were 0.5467 for NMLE (2018 and 2021), 0.5599 for NPLE (2017), and 0.5897 for NNLE (2017); the lowest was 0.4356 for NPLE in 2019.
- Average accuracy across the five years was 54.08% for NNLE, 53.05% for NMLE, and 49.02% for NPLE.
- There was no significant difference across different units in NMLE, NPLE, or NNLE, but in NPLE ChatGPT performed significantly better on single-choice than multiple-choice questions (p < 0.0001).
- In the 2021 NMLE, ChatGPT answered all questions correctly in clinical epidemiology, human parasitology, and dermatology, but had less than 0.5 correct in pathology, pathophysiology, public health regulations, physiology, and anatomy; no significant difference was found between case-based and non-case-based questions.
Methods
- Questions were collected from Chinese NMLE, NPLE, and NNLE from 2017 to 2021. NMLE had 4 units of 150 questions (600 total), NPLE had 4 units of 120 questions (480 total), and NNLE had 2 units of 120 questions (240 total). Questions with figures, tables, or chemical structures were manually identified and excluded by a clinician.
- ChatGPT was invoked via the official API using gpt-3.5-turbo with default parameters. Direct instruction prompts were used to force clear answers and distinguish single-choice from multiple-choice questions.
- Each response was manually reviewed by an experienced clinician and scored 1 for agreement with the true answer or 0 for disagreement. Data processing used Python 3.9.13 in Jupyter Notebook, statistical analysis used GraphPad Prism 9, and significance was set at p < 0.05.
Limitations
- Stated: the exams were all multiple-choice, so open-ended content generation was not fully assessed; evaluation used zero-shot learning; prompt variations may affect responses; and further investigation is needed into causes of substandard performance and broader applications.
- Evident: the study evaluated only ChatGPT/GPT-3.5, so findings may not generalize to other large language models or versions.
- Evident: excluding questions with figures, tables, or chemical structures may limit how fully the exams represent the original test content.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 4 cited sources checked · 0 supported · 0 flagged
- Findingown data
ChatGPT failed to pass the accuracy threshold of 0.6 in any of the three types of examinations (NMLE, NPLE, NNLE) over the five years 2017-2021.
“ChatGPT failed to pass the accuracy threshold of 0.6 in any of the three types of examinations over the five years.”
- Also stated in: § Overall performance · plain
“As shown in Fig. 2, ChatGPT failed to pass the accuracy threshold of 0.6 in any of the three types of examinations over the five years.”
- Also stated in: § Conclusion · hedged
“The results indicated ChatGPT failed to meet the official pass criteria with accuracy threshold of 0.6 in any of the three types of examinations over the five years.”
Quote in the paperSubstantial claim- Rests on an assumption: A correct response rate exceeding 0.6 is the official passing criterion for the Chinese NMLE, NPLE, and NNLE.
- Also stated in: § Overall performance · plain
- Findingown data
In 2017, ChatGPT correctly answered 257 out of 459 questions in the NPLE.
“In the year 2017, ChatGPT achieved highest score, and correctly answered 257 out of 459 questions.”
Quote in the paperNot a substantial claim - Findingown data
There was no significant difference in ChatGPT's performance across different units in the NPLE.
“The results shown there was no significant difference in across different units in NMLE (Fig. 3A), NPLE (Fig. 3B), and NNLE (Fig. 3C).”
Quote in the paperNot a substantial claim - Findingown data
In the 2021 NMLE, ChatGPT answered all questions correctly in clinical epidemiology, human parasitology, and dermatology.
“The result revealed that ChatGPT excelled in clinical epidemiology, human parasitology, and dermatology, with all questions answered correctly.”
- Also stated in: § Results · plain
“ChatGPT performed well in a range of subject areas, including clinical epidemiology, human parasitology, and dermatology”
Quote in the paperNot a substantial claim - Also stated in: § Results · plain
- Findingown data
There was no significant difference in ChatGPT's performance across different units in the NMLE.
“The results shown there was no significant difference in across different units in NMLE (Fig. 3A), NPLE (Fig. 3B), and NNLE (Fig. 3C).”
Quote in the paperNot a substantial claim
- Findingown data
In the 2021 NMLE, there was no significant difference in ChatGPT's performance on case-based questions and non-case-based questions.
“Interestingly, we found no significant difference in performance on case-based questions and non-case-based questions.”
Quote in the paperNot a substantial claim - Findingown data
In the 2021 NMLE, ChatGPT performed admirably in topics related to molecule, health management and prevention, diagnosis and screening.
“Additionally, we observed that ChatGPT performed admirably in topics related to molecule, health management and prevention, diagnosis and screening”
- Also stated in: § Results · plain
“as well as in various medical topics such as molecules, health management and prevention, diagnosis and screening.”
Quote in the paperNot a substantial claim - Also stated in: § Results · plain
- Findingown data
On average in the NNLE, ChatGPT's performance of the two units had no noticeable difference.
“On average, ChatGPT’s performance of the two units had no noticeable difference.”
Quote in the paperNot a substantial claim - Findingown data
In the 2021 NMLE, ChatGPT's proportion of correct answers was less than 0.5 in pathology, pathophysiology, public health regulations, physiology, and anatomy.
“However, the model faltered in subjects such as pathology, pathophysiology, public health regulations, physiology, and anatomy, with the proportion of correct answers was less than 0.5.”
Quote in the paperSubstantial claim - Findingown data
The 2019 NPLE exam resulted in the lowest accuracy, with a recorded value of 0.4356.
“Conversely, the 2019 NPLE exam resulted in the lowest accuracy, with a recorded value of 0.4356.”
Quote in the paperNot a substantial claim - Findingown data
There was no significant difference in ChatGPT's performance across different units in the NNLE.
“The results shown there was no significant difference in across different units in NMLE (Fig. 3A), NPLE (Fig. 3B), and NNLE (Fig. 3C).”
Quote in the paperNot a substantial claim - Findingown data
The performance of ChatGPT varied across different units and years.
“The performance of ChatGPT has varied across different units and years.”
- Also stated in: § Conclusion · plain
“The performance of ChatGPT varied across different units and years, with the highest score achieved in NNLE of year 2017.”
Quote in the paperSubstantial claim - Also stated in: § Conclusion · plain
- Findingown data
In the Chinese NMLE, the highest recorded accuracy was 0.5467, attained in both 2018 and 2021.
“Specifically, in the NMLE, the highest recorded accuracy was 0.5467, which was attained in both 2018 and 2021.”
- Also stated in: § Overall performance · plain
“Specifically, in the Chinese NMLE, the highest recorded accuracy was 0.5467, which was attained in both 2018 and 2021.”
Quote in the paperNot a substantial claim - Also stated in: § Overall performance · plain
- Findingown data
In the 2021 NMLE, ChatGPT's performance was lackluster in topics such as clinical manifestations, indicator values, structural location, cell, and tissue.
“but its performance was lackluster in topics such as clinical manifestations, indicator values, structural location, cell, and tissue.”
Quote in the paperSubstantial claim - Findingown data
ChatGPT achieved an accuracy of over 0.5 in most of the exams.
“In our study, ChatGPT has achieved an accuracy of over 0.5 in most of the exams”
Quote in the paperSubstantial claim - Findingown data
In the Chinese NPLE, the highest accuracy was 0.5599 in 2017.
“In the NPLE, the highest accuracy was 0.5599 in 2017.”
- Also stated in: § Overall performance · plain
“In the Chinese NPLE, the highest accuracy was 0.5599 in 2017.”
Quote in the paperNot a substantial claim - Also stated in: § Overall performance · plain
- Findingown data
In 2018 and 2021, ChatGPT correctly answered 328 out of 600 questions in the NMLE.
“In 2018 and 2021, ChatGPT correctly answered 328 out of 600 questions.”
Quote in the paperNot a substantial claim - Interpretationreasoning
ChatGPT may require additional training data in non-English languages to enhance its performance in non-English medical exams.
“These findings suggest that ChatGPT may require additional training data in non-English languages to enhance its performance in non-English medical exams.”
Quote in the paperNot a substantial claim0 of 4 sources checked- cites [19] Kung TH, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198. source
Not checked in the free run.
- cites [20] Gilson A, et al. How does ChatGPT perform on the United States Medical Licensing examination? The implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ. 2023;9:e45312. source
Not checked in the free run.
- cites [25] Wang YM, Shen HW, Chen TJ. Performance of ChatGPT on the Pharmacist Licensing examination in Taiwan. J Chin Med Assoc, 2023. source
Not checked in the free run.
- cites [28] Huh S. Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination? A descriptive study. J Educ Eval Health Prof. 2023;20:1.
Not checked in the free run.
- Interpretationown data
The results indicate significant potential for ChatGPT in medical education.
“indicating a significant potential for ChatGPT in medical education.”
- Also stated in: § Conclusions · boosted
“but show great potential of large language models in medical education.”
Quote in the paperSubstantial claim - Also stated in: § Conclusions · boosted
- May combine two claims. 5
- Classified as probably not a substantial claim. 3
- The quote only partly states this claim. 1
May combine two claims.
Findingown dataIn comparison, ChatGPT exhibited better proficiency in NNLE (54.08%), with NMLE (53.05%) and NPLE (49.02%) following behind.
“In comparison, ChatGPT exhibited better proficiency in NNLE (54.08%), with NMLE (53.05%) and NPLE (49.02%) following behind.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn the NPLE, ChatGPT demonstrated higher performance in single-choice questions compared to multiple-choice questions, with a highly statistical difference (p < 0.0001).
“However, in the case of NPLE (Fig. 3D), ChatGPT demonstrated higher performance in single-choice questions compared to multiple-choice questions, with a highly statistical difference (p < 0.0001).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataIn the Chinese NNLE, the highest accuracy was 0.5897 in 2017, which is the highest accuracy in the entire evaluation.
“In the NNLE, the most impressive result was shown in 2017, with an accuracy of 0.5897, which is also the highest accuracy in our entire evaluation.”
- Also stated in: § Overall performance · plain
“In the Chinese NNLE, the most impressive result was shown in 2017, with an accuracy of 0.5897, which is also the highest accuracy in our entire evaluation.”
Quote in the paperNot a substantial claim- Also stated in: § Overall performance · plain
May combine two claims.
Findingown dataOn average in the NPLE, ChatGPT performed best in Unit 4 (62), followed by Unit 2 (58), Unit 3 (53), and Unit 1 (52).
“On average, the ChatGPT performed best in Unit 4 (62), followed by Unit 2 (58), Unit 3 (53) and Unit 1(52).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataOn average in the NMLE, ChatGPT achieved the highest score in Unit 2 (84.6), followed by Unit 1 (79.8), Unit 3 (78.2), and Unit 4 (75.4).
“On average, ChatGPT achieved the highest score in Unit 2 (84.6), followed by Unit 1 (79.8), Unit 3 (78.2), Unit 4 (75.4).”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. The quote only partly states this claim.
InterpretationreasoningChatGPT may not possess an in-depth understanding of the specific legal framework and requirements of non-English-speaking countries, leading to erroneous responses to related questions.
“ChatGPT, being a language model trained on a diverse range of data, may not possess an in-depth understanding of the specific legal framework and requirements.”
- Also stated in: § ChatGPT failed NNLE, NMLE and NPLE in China · hedged
“it may encounter difficulties in correctly understanding the healthcare policies of non-English-speaking countries, leading to erroneous responses to related questions.”
Quote in the paperSubstantial claim- Also stated in: § ChatGPT failed NNLE, NMLE and NPLE in China · hedged
Classified as probably not a substantial claim.
Study design and conductnone givenQuestions with figures, tables or chemical structure were manually identified and excluded by a clinician with five years of clinical experience.
“The questions with figures, tables or chemical structure were manually identified and excluded by a clinician with five years of clinical experience.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductnone givenFor each question, the response of ChatGPT was reviewed by an experienced clinician to determine the predicted answer, which was then compared with the true answer.
“For each question, the response of ChatGPT was reviewed by an experienced clinicians to determine the predicted answer, which was then compared with the true answer.”
Quote in the paperSubstantial claim
The paper relies on this without support.
A correct response rate exceeding 0.6 is the official passing criterion for the Chinese NMLE, NPLE, and NNLE.
“Based on the requirements of the examination, a correct response rate exceeding the accuracy threshold of 0.6 is considered to meet the passing criteria.”
The paper relies on this without support.
The complexity and difficulty of the exam questions explain differences in ChatGPT's performance across exams and units.
“The result corresponds to the complexity and difficulty of the exam questions.”
“This is because the complexity and difficulty of question in each unit were vary from year to year.”
The paper relies on this without support.
ChatGPT was trained with the vast majority of data in English, with only a small amount of data in other languages like Chinese.
“According to OpenAI, ChatGPT has been trained with the vast majority of the data is in English, with only a small amount of data in other languages, like Chinese.”
The paper relies on this without support.
ChatGPT's proficiency in numerical computations is limited.
“Thirdly, while ChatGPT has remarkable ability to process and generate text data, its proficiency in numerical computations is limited.”
“For some questions related to mathematical calculation, such as dosage calculation and laboratory values interpretation, may pose challenges for a language model.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT can assist in the decision-making process due to its capability in content generation for medical and clinical knowledge.
“Regardless of its application, ChatGPT can assist in the decision-making process (Kanada et al., 2023) due to its capability in content generation for medical and clinical knowledge (Zong, 2024).”
From Current trends and future implications in the utilization of ChatGPT in nursing: a rapid review (2024) · cites this paper as Zong, 2024
ChatGPT failed the Chinese National Nurse Licensing Examination, National Medical Licensing Examination, and National Pharmacist Licensing Examination.
“it failed the Chinese National Nurse Licensing Examination, National Medical Licensing Examination, and the National Pharmacist Licensing Examination (Zong et al., 2024).”
From Current trends and future implications in the utilization of ChatGPT in nursing: a rapid review (2024) · cites this paper as Zong et al., 2024
Some authors stated the use of prompt engineering, prompt learning, and direct instruction strategy via prompt to guide the use of the ChatGPT.
“Some authors stated the use of prompt engineering (Taira et al., 2023), prompt learning (Goktas et al., 2024), and direct instruction strategy via prompt (Zong, 2024) to guide the use of the ChatGPT.”
From Current trends and future implications in the utilization of ChatGPT in nursing: a rapid review (2024) · cites this paper as Zong, 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Figure 3 gives the nurse examination the pharmacist abbreviation
MinorThe Figure 3 caption labels the National Nurse Licensing Examination "NPLE," whereas the paper defines its abbreviation as "NNLE."
“(C) National Nurse Licensing Examination (NPLE).”
“the Chinese National Nurse Licensing Examination (NNLE)”
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · cited 1× · see citing works
“43 Introduction 44 In the last decade, artificial intelligence (AI) technology has undergone a rapid 45 evolution, achieving noteworthy breakthroughs in numerous fields [1, 2].”
BackgroundChatGPT: five priorities for research
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023) · Nature · doi:10.1038/d41586-023-00288-7 · Cited by 1,842 · see citing works
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150 · see citing works
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099 · see citing works
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation
Zhang, Y., Sun, S., Galley, M., Chen, Y.-C., Brockett, C., Gao, X., Gao, J., Liu, J., & Dolan, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.30 · Cited by 1,076 · see citing works
Can artificial intelligence help for scientific writing?
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04380-2 · Cited by 827 · see citing works
Artificial Intelligence in Cancer Research and Precision Medicine
Bhinder, B., Gilvary, C., Madhukar, N. S., & Elemento, O. (2021) · Cancer Discovery · doi:10.1158/2159-8290.cd-21-0090 · Cited by 781 · see citing works
Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions
Abd-alrazaq, A., AlSaad, R., Alhuwail, D., Ahmed, A., Healy, P. M., Latifi, S., Aziz, S., Damseh, R., Alabed Alrazak, S., & Sheikh, J. (2023) · JMIR Medical Education · doi:10.2196/48291 · Cited by 722 · see citing works
Bhayana, R., Krishna, S., & Bleakney, R. R. (2023) · Radiology · doi:10.1148/radiol.230582 · Cited by 460 · see citing works
Huh, S. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.1 · Cited by 278 · see citing works
Huh, S. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.01 · Cited by 199 · see citing works
Humar, P., Asaad, M., Bengur, F. B., & Nguyen, V. (2023) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad130 · Cited by 155 · see citing works
ChatGPT Answers Common Patient Questions About Colonoscopy
Lee, T.-C., Staller, K., Botoman, V., Pathipati, M. P., Varma, S., & Kuo, B. (2023) · Gastroenterology · doi:10.1053/j.gastro.2023.04.033 · Cited by 146 · see citing works
Performance of ChatGPT on the pharmacist licensing examination in Taiwan
Wang, Y.-M., Shen, H.-W., & Chen, T.-J. (2023) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000000942 · Cited by 109 · see citing works
ChatGPT: not all languages are equal
Seghier, M. L. (2023) · Nature · doi:10.1038/d41586-023-00680-3 · Cited by 108 · cited 1× · see citing works
“Possible explanations for this performance difference include language and cultural differences, variations in examination content [26].”
BackgroundChatGPT failed Taiwan’s Family Medicine Board Exam
Weng, T.-L., Wang, Y.-M., Chang, S., Chen, T.-J., & Hwang, S.-J. (2023) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000000946 · Cited by 100 · see citing works
Giannos, P. (2023) · BMJ Neurology Open · doi:10.1136/bmjno-2023-000451 · Cited by 84 · see citing works
Jin, Q., Leaman, R., & Lu, Z. (2023) · Journal of the American Society of Nephrology · doi:10.1681/asn.0000000000000166 · Cited by 62 · see citing works
Assessing the Accuracy and Clinical Utility of ChatGPT in Laboratory Medicine
Muñoz‐Zuluaga, C., Zhao, Z., Wang, F., Greenblatt, M. B., & Yang, H. S. (2023) · Clinical Chemistry · doi:10.1093/clinchem/hvad058 · Cited by 61 · see citing works
How I use ChatGPT responsibly in my teaching
Yang, H. (2023) · Nature · doi:10.1038/d41586-023-01026-9 · Cited by 57 · see citing works
The utility of ChatGPT in generating patient-facing and clinical responses for melanoma
Young, J. N., O’Hagan, R., Poplausky, D., Levoska, M. A., Gulati, N., Ungar, B., & Ungar, J. P. (2023) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2023.05.024 · Cited by 51 · see citing works
Das, A., Selek, S., Warner, A. R., Zuo, X., Hu, Y., Kuttichi Keloth, V., Li, J., Zheng, W. J., & Xu, H. (2022) · Proceedings of the 21st Workshop on Biomedical Language Processing · doi:10.18653/v1/2022.bionlp-1.27 · Cited by 47 · see citing works
Self-citationExperiences, challenges, and prospects of National Medical Licensing Examination in China
Wang, X. (2022) · BMC Medical Education · doi:10.1186/s12909-022-03385-9 · Cited by 37 · see citing works
Komorowski, M., Lopez, M. D. P. A., & Chang, A. C. (2023) · Intensive Care Medicine · doi:10.1007/s00134-023-07096-7 · Cited by 29 · see citing works
A study on the performance of ChatGPT in infectious diseases clinical consultation
Sarink, M. J., Bakker, I. L., Anas, A. A., & Yusuf, E. (2023) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2023.05.017 · Cited by 23 · see citing works
Performance of China’s new medical licensing examination for rural general practice
Han, X., Li, X., Cheng, L., et al. (2020) · BMC Medical Education · doi:10.1186/s12909-020-02234-x · Cited by 14 · see citing works
Matters arising: authors of research papers must cautiously use ChatGPT for scientific writing
Shafiee, A. (2023) · International Journal of Surgery · doi:10.1097/js9.0000000000000515 · Cited by 11 · see citing works
ChatGPT: effective writing is succinct
Kovoor, J. G., Gupta, A. K., & Bacchi, S. (2023) · BMJ · doi:10.1136/bmj.p1125 · Cited by 9 · see citing works
Reply to “Comment on ChatGPT failed Taiwan’s Family Medicine Board Exam”
Weng, T.-L., & Chen, T. (2023) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000000956 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Li, J., Zong, H., Wu, E., Wu, R., Peng, Z., Zhao, J., Yang, L., Xie, H., & Shen, B. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05738-y · Cited by 104 · see citing works
Self-citationJin, H. K., Lee, H. E., & Kim, E. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05944-8 · Cited by 85 · see citing works
A rapid review on current and potential uses of large language models in nursing
Hobensack, M., von Gerich, H., Vyas, P., Withall, J., Peltonen, L.-M., Block, L. J., Davies, S., Chan, R., Van Bulck, L., Cho, H., et al. (2024) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2024.104753 · Cited by 80 · see citing works
Liu, J. Q. J., Hui, K. T. K., Al Zoubi, F., Zhou, Z. Z. X., Samartzis, D., Yu, C. C. H., Chang, J. R., & Wong, A. Y. L. (2024) · International Journal for Educational Integrity · doi:10.1007/s40979-024-00155-6 · Cited by 56 · see citing works
Durmuş Sarıkahya, S., Özbay, Ö., Torpuş, K., Usta, G., & Çınar Özbay, S. (2025) · Nurse Education Today · doi:10.1016/j.nedt.2025.106755 · Cited by 60 · see citing works
ChatGPT and Clinical Training: Perception, Concerns, and Practice of Pharm-D Students
Zawiah, M., Al-Ashwal, F., Gharaibeh, L., Abu Farha, R., Alzoubi, K., Abu Hammour, K., Qasim, Q. A., & Abrah, F. (2023) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s439223 · Cited by 51 · see citing works
Current trends and future implications in the utilization of ChatGPT in nursing: a rapid review
Kleib, M., Darko, E. M., Akingbade, O., Kennedy, M., Majekodunmi, P., Nickel, E., & Vogelsang, L. (2024) · International Journal of Nursing Studies Advances · doi:10.1016/j.ijnsa.2024.100252 · Cited by 40 · see citing works
Using ChatGPT for medical education: the technical perspective
Chan, K. Y., Yuen, T. H., & Co, M. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06785-9 · Cited by 34 · see citing works
Lin, Y., Luo, Z., Ye, Z., Zhong, N., Zhao, L., Zhang, L., Li, X., Chen, Z., & Chen, Y. (2025) · JMIR Medical Education · doi:10.2196/71125 · Cited by 31 · see citing works
“Similarly, ChatGPT struggles to accurately comprehend and adapt to the local regulatory environment when addressing medical policies specific to China, largely due to the limited representation of Chinese data in its training set [102,112].”
Jaleel, A., Aziz, U., Farid, G., Zahid Bashir, M., Mirza, T. R., Khizar Abbas, S. M., Aslam, S., & Sikander, R. M. H. (2025) · JMIR Medical Education · doi:10.2196/68070 · Cited by 25 · see citing works
Oladokun, B. D., Enakrire, R. T., Emmanuel, A. K., Ajani, Y. A., & Adetayo, A. J. (2025) · Journal of Web Librarianship · doi:10.1080/19322909.2025.2482093 · Cited by 7 · see citing works
- Authors
- Zong, H., Li, J., Wu, E., Wu, R., Lu, J., & Shen, B.
- Affiliations
- Sichuan University (CN)
- Corresponding author
- Shen, Bairong
- Venue
- BMC Medical Education
- Published
- 2024-02-14
- Volume
- 24
- Issue
- 1
- Pages
- 143-143
- ISSN
- 1472-6920 (ISSN-L)
- PMC
- PMC10868058
- OpenAlex
- W4391811872
- Semantic Scholar
- 969b2d3912326c175f68767ab608ccad5f19dd7d
- Cited by
- 114
- References
- 31
- Citations by year
- 2026: 35 · 2025: 34 · 2024: 41 · 2023: 3 · 2022: 1
- FWCI
- 10.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medical Education (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (32270690)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, national medical licensing examination, medical licensing examinations, multiple-choice questions, single-choice questions, medical education
- MeSH terms
- Data Accuracy, Artificial Intelligence, China, Education, Medical, Education, Nursing, Education, Pharmacy, Educational Measurement, Licensure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
