ChatGPT in Medical Education: A Precursor for Automation Bias?
Open accessPublished version of a preprint
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Artificial intelligence (AI) in health care has the promise of providing accurate and efficient results. However, AI can also be a black box, where the logic behind its results is nonrational. There are concerns if these questionable results are used in patient care. As physicians have the duty to provide care based on their clinical judgment in addition to their patients’ values and preferences, it is crucial that physicians validate the results from AI. Yet, there are some physicians who exhibit a phenomenon known as automation bias, where there is an assumption from the user that AI is always right. This is a dangerous mindset, as users exhibiting automation bias will not validate the results, given their trust in AI systems. Several factors impact a user’s susceptibility to automation bias, such as inexperience or being born in the digital age. In this editorial, I argue that these factors and a lack of AI education in the medical school curriculum cause automation bias. I also explore the harms of automation bias and why prospective physicians need to be vigilant when using AI. Furthermore, it is important to consider what attitudes are being taught to students when introducing ChatGPT, which could be some students’ first time using AI, prior to their use of AI in the clinical setting. Therefore, in attempts to avoid the problem of automation bias in the long-term, in addition to incorporating AI education into the curriculum, as is necessary, the use of ChatGPT in medical education should be limited to certain tasks. Otherwise, having no constraints on what ChatGPT should be used for could lead to automation bias.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 4×
“ChatGPT’s hallucinations are results that are seemingly feasible but do not actually exist [1,16].”
BackgroundInfluentialAlgorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · cited 2×
“ChatGPT’s hallucinations are results that are seemingly feasible but do not actually exist [1,16].”
BackgroundEysenbach, G. (2023) · JMIR Medical Education · doi:10.2196/46885 · Cited by 840 · cited 3×
“As seen in this editorial, ChatGPT excels at particular tasks, such as summarizing information and creating study materials [18,19,24].”
BackgroundThe role of AI in drug discovery: challenges, opportunities, and strategies
Blanco-González, A., Cabezón, A., Seco-González, A., Conde-Torres, D., Antelo-Riveiro, P., Piñeiro, Á., & Garcia-Fandino, R. (2023) · Pharmaceuticals · doi:10.3390/ph16060891 · Cited by 690
ChatGPT - Reshaping medical education and clinical management
Khan, R. A., Jawaid, M., Khan, A. R., & Sajjad, M. (2023) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.39.2.7653 · Cited by 628
Do as AI say: susceptibility in deployment of clinical decision-aids
Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S. J., Lermer, E., Coughlin, J. F., Guttag, J. V., Colak, E., & Ghassemi, M. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00385-9 · Cited by 543 · cited 1×
“Multiple studies have found that algorithmic appreciation—a user’s valuing of an algorithm’s outputs—is lower for users who have more experience in a task than for those who are considered nonexperts in that task [12,13].”
BackgroundArtificial intelligence in medical education: a cross-sectional needs assessment
Civaner, M. M., Uncu, Y., Bulut, F., Chalil, E. G., & Tatli, A. (2022) · BMC Medical Education · doi:10.1186/s12909-022-03852-3 · Cited by 363 · cited 1×
“In Civaner et al’s [2] survey of medical students’opinions on AI education, they found that 75.6% of students had either limited or no education on the topic of AI.”
BackgroundInfluentialTiwari, C. K., Bhat, M. A., Khan, S. T., Subramaniam, R., & Khan, M. A. I. (2024) · Interactive Technology and Smart Education · doi:10.1108/itse-04-2023-0061 · Cited by 315 · cited 1×
“Moreover, Tiwari et al [15], who applied the Technology Acceptance Model to ChatGPT, found that students generally had positive views (in terms of perceived usefulness, credibility, social presence, and hedonic motivation) of ChatGPT based on their previous experiences with using the tool.”
MethodHomolak, J. (2023) · Croatian Medical Journal · doi:10.3325/cmj.2023.64.1 · Cited by 208
Cited by
Liu, F., Chang, X., Zhu, Q., Huang, Y., Li, Y., & Wang, H. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06232-1 · Cited by 35
“What’s more, although LLMs could help supplement medical students’ experience during the clinical phase through augmented reality, interaction with real patients and professionals is still necessary, as their reactions in real time are difficult to predict [80].”
BackgroundWho Gives Feedback Matters: Student Biases Towards Human and AI ‐Generated Formative Feedback
Nazaretsky, T., Mejia‐Domenzain, P., Swamy, V., Frej, J., & Käser, T. (2026) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70153 · Cited by 26
Forero, D. A., Abreu, S. E., Tovar, B. E., & Oermann, M. H. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02212-0 · Cited by 13
AI-overdependence and human cognitive decline: Hazards, evidence, and mitigation strategies
Noorbehbahani, F., & Oyibo, K. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101102 · Cited by 12
Metadata
- Authors
- Nguyen, T.
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2024-01-17
- Volume
- 10
- Pages
- e50174
- ISSN
- 2369-3762 (ISSN-L)
- OpenAlex
- W4389577521
- Cited by
- 59
- References
- 16
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.