Human Factors Challenges for the Safe Use of Artificial Intelligence in Patient Care
Open accessAbstract
The use of artificial intelligence (AI) in patient care can offer significant benefits. However, there is a lack of independent evaluation considering AI in use. The paper argues that consideration should be given to how AI will be incorporated into clinical processes and services. Human factors challenges that are likely to arise at this level include cognitive aspects (automation bias and human performance), handover and communication between clinicians and AI systems, situation awareness and the impact on the interaction with patients. Human factors research should accompany the development of AI from the outset.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Modern AI platforms increasingly include built-in traceability features that automatically log training runs, data lineage, and performance metrics.
“Modern AI platforms increasingly include built-in traceability features that automatically log training runs, data lineage, and performance metrics [137,138].”
From Artificial Intelligence in Clinical Oncology: From Productivity Enhancement to Creative Discovery (2025) · cites this paper as [138]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · see citing works
Toward a Theory of Situation Awareness in Dynamic Systems
Endsley, M. R. (1995) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872095779049543 · Cited by 8,875 · see citing works
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · see citing works
Humans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · see citing works
A model for types and levels of human interaction with automation
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.844354 · Cited by 4,195 · see citing works
Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · see citing works
Bainbridge, L. (1983) · Automatica · doi:10.1016/0005-1098(83)90046-8 · Cited by 2,205 · see citing works
Haenssle, H. A., Fink, C., Schneiderbauer, R., Toberer, F., Buhl, T., Blum, A., Kalloo, A., Hassen, A. B. H., Thomas, L., Enk, A., et al. (2018) · Annals of Oncology · doi:10.1093/annonc/mdy166 · Cited by 1,571 · see citing works
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · see citing works
(2019) · Cited by 986 · see citing works
Artificial intelligence, bias and clinical safety
Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., & Tsaneva-Atanasova, K. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008370 · Cited by 956 · see citing works
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · see citing works
Automation bias and verification complexity: a systematic review
Lyell, D., & Coiera, E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw105 · Cited by 451 · see citing works
Improving palliative care with deep learning
Avati, A., Jung, K., Harman, S. M., Downing, N. L., Ng, A. Y., & Shah, N. H. (2017) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2017.8217669 · Cited by 445 · see citing works
Automation Bias: Decision Making and Performance in High-Tech Cockpits
Mosier, K. L., Skitka, L. J., Heers, S., & Burdick, M. (1998) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0801_3 · Cited by 353 · see citing works
Automation and Human Performance
Bowers, C. A., Oser, R. L., Eduardo Salas, A., & Cannon‐Bowers, J. A. (2018) · Routledge eBooks · doi:10.1201/9781315137957 · Cited by 307 · see citing works
Framing the challenges of artificial intelligence in medicine
Yu, K.-H., & Kohane, I. S. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008551 · Cited by 233 · see citing works
Bahl, M., Barzilay, R., Yedidia, A., Locascio, N., Yu, L., & Lehman, C. D. (2017) · Radiology · doi:10.1148/radiol.2017170549 · Cited by 180 · see citing works
Computer Decision Support as a Source of Interpretation Error: The Case of Electrocardiograms
Tsai, T. L., Fridsma, D. B., & Gatti, G. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1279 · Cited by 161 · see citing works
Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography
Alberdi, E., Povyakalo, A., Strigini, L., & Ayton, P. (2004) · Academic Radiology · doi:10.1016/j.acra.2004.05.012 · Cited by 124 · see citing works
Some pitfalls in the promises of automated and autonomous vehicles
Hancock, P. A. (2018) · Ergonomics · doi:10.1080/00140139.2018.1498136 · Cited by 122 · see citing works
(2018) · doi:10.1201/9781315137957-2 · Cited by 108 · see citing works
Automation bias in electronic prescribing
Lyell, D., Magrabi, F., Raban, M. Z., Pont, L. G., Baysari, M. T., Day, R. O., & Coiera, E. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0425-5 · Cited by 103 · cited 1× · see citing works
“22 The study found that clinical decision support reduced prescribing errors when working correctly but also increased prescribing errors by around one-third in cases where the system either did not alert the clinician to a potential problem or provided the wrong advice.”
Sujan, M. A., Spurgeon, P. C., Inada-Kim, M., et al. (2014) · Health Services and Delivery Research · doi:10.3310/hsdr02050 · Cited by 57 · see citing works
Self-citationPREDICTING DIABETES FROM PHOTOPLETHYSMOGRAPHY USING DEEP LEARNING
Avram, R., Tison, G. H., Kuhar, P., et al. (2019) · Journal of the American College of Cardiology · doi:10.1016/s0735-1097(19)33778-7 · Cited by 28 · see citing works
Sujan, M. A., Chessum, P., Rudd, M., et al. (2013) · Emergency Medicine Journal · doi:10.1136/emermed-2013-202977 · Cited by 27 · see citing works
Self-citationCritical Barriers to Safety Assurance and Regulation of Autonomous Medical Systems
Sujan, M. A., Furniss, D. J., Embrey, D. E., et al. (2019) · European Safety and Reliability Conference (ESREL) · doi:10.3850/978-981-11-2724-3_0334-cd · Cited by 9 · see citing works
Self-citationHow will drivers interact with vehicles of the future
Burnett, G., Large, D. R., & Salanitri, D. (2019) · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [14]Sarter NB, Woods DD, Billings CE Automation surprises: Salvendy G, Handbook of Human Factors & Ergonomics. Wiley, 1997: 1926– 43.
Cited by
Sorted by citations
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039 · see citing works
Self-citationLekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957 · see citing works
“…managers, administrators) from an early stage to compile information on the AI tool’s intended use and end user requirements (eg, human-AI interfaces), as well as on human factors that might affect the usage of the AI tool 57 (eg, digital literacy level, age group, ergonomics, automation bias).”
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070904 · Cited by 665 · see citing works
Artificial intelligence in health care: accountability and safety
Habli, I., Lawton, T., & Porter, Z. (2020) · Bulletin of the World Health Organization · doi:10.2471/blt.19.237487 · Cited by 357 · see citing works
Self-citationThe effect of using a large language model to respond to patient messages
Chen, S., Guevara, M., Moningi, S., Hoebers, F., Elhalawani, H., Kann, B. H., Chipidza, F. E., Leeman, J., Aerts, H. J. W. L., Miller, T., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00060-8 · Cited by 147 · see citing works
Abdelwanis, M., Alarafati, H. K., Tammam, M. M. S., & Simsekler, M. C. E. (2024) · Journal of Safety Science and Resilience · doi:10.1016/j.jnlssr.2024.06.001 · Cited by 137 · see citing works
Advancing health equity with artificial intelligence
Thomasian, N. M., Eickhoff, C., & Adashi, E. Y. (2021) · Journal of Public Health Policy · doi:10.1057/s41271-021-00319-5 · Cited by 117 · see citing works
Salwei, M. E., & Carayon, P. (2022) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/15553434221097357 · Cited by 112 · see citing works
Lyell, D., Coiera, E., Chen, J., Shah, P., & Magrabi, F. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100301 · Cited by 95 · see citing works
The AI life cycle: a holistic approach to creating ethical AI for health decisions
Ng, M. Y., Kapur, S., Blizinsky, K. D., & Hernandez-Boussard, T. (2022) · Nature Medicine · doi:10.1038/s41591-022-01993-y · Cited by 71 · see citing works
Felmingham, C. M., Adler, N. R., Ge, Z., Morton, R. L., Janda, M., & Mar, V. J. (2021) · American Journal of Clinical Dermatology · doi:10.1007/s40257-020-00574-4 · Cited by 57 · see citing works
Wang, D.-Y., Ding, J., Sun, A.-L., Liu, S.-G., Jiang, D., Li, N., & Yu, J.-K. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad118 · Cited by 54 · see citing works
Harada, Y., Katsukura, S., Kawamura, R., & Shimizu, T. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18042086 · Cited by 44 · see citing works
Held, L. A., Wewetzer, L., & Steinhäuser, J. (2022) · Health Informatics Journal · doi:10.1177/14604582221112816 · Cited by 34 · see citing works
Building digital resilience: leading healthcare transformation through an online community
Pilosof, N. P., Welcman, Y., Barrett, M., Oborn, E., & Barrett, S. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1656804 · Cited by 15 · see citing works
Artificial Intelligence in Clinical Oncology: From Productivity Enhancement to Creative Discovery
Kuno, M., Osumi, H., Udagawa, S., Yoshikawa, K., Ooki, A., Shinozaki, E., Ishikawa, T., Oba, J., Yamaguchi, K., & Sakurada, K. (2025) · Current Oncology · doi:10.3390/curroncol32110588 · Cited by 7 · see citing works
Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis
Nguyen, H. B., Krishtopaytis, E., Lopez, E., Farnoudi, N., Van, T., Kharalampova, V., & Coz Yataco, A. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15093477
“Recent guidance and human-factors literature emphasize that safe clinician–AI interaction requires human oversight, calibrated trust, awareness of model limitations, and deliberate verification of AI outputs against the patient’s examination, physiology, and clinical trajectory [79,80].”
- Authors
- Sujan, M., Furniss, D., Grundy, K., Grundy, H., Nelson, D., Elliott, M., White, S., Habli, I., & Reynolds, N.
- Affiliations
- University of Warwick (GB)
- Human Reliability (United Kingdom) (GB)
- Patient, Derby, UK
- University Hospitals of Derby and Burton NHS Foundation Trust (GB)
- NHS Digital (GB)
- University of York (GB)
- Corresponding author
- Sujan, Mark
- Venue
- BMJ Health & Care Informatics
- Publisher
- BMJ
- Published
- 2019-11
- Volume
- 26
- Issue
- 1
- Pages
- e100081
- ISSN
- 2632-1009 (ISSN-L)
- PMC
- PMC7252977
- OpenAlex
- W2990795246
- Cited by
- 144
- References
- 28
- Citations by year
- 2026: 27 · 2025: 26 · 2024: 22 · 2023: 27 · 2022: 19 · 2021: 16 · 2020: 7
- FWCI
- 3.34
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc
- Locations
- BMJ Health & Care Informatics (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Lloyd"s Register Foundation (Assuring Autonomy International Programme: SAM)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- human factors, clinical automation, automation bias, human performance, handover, situational awareness, patient interaction, AI evaluation, clinical workflow integration
- MeSH terms
- Artificial Intelligence, Automation, Awareness, Delivery of Health Care, Patient Care, Humans, Ergonomics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
