Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
The application of machine learning (ML) and artificial intelligence (AI) in healthcare domains has received much attention in recent years, yet significant questions remain about how these new tools integrate into frontline user workflow, and how their design will impact implementation. Lack of acceptance among clinicians is a major barrier to the translation of healthcare innovations into clinical practice. In this systematic review, we examine when and how clinicians are consulted about their needs and desires for clinical AI tools. Forty-five articles met criteria for inclusion, of which 24 were considered design studies. The design studies used a variety of methods to solicit and gather user feedback, with interviews, surveys, and user evaluations. Our findings show that tool designers consult clinicians at various but inconsistent points during the design process, and most typically at later stages in the design cycle (82%, 19/24 design studies). We also observed a smaller amount of studies adopting a human-centered approach and where clinician input was solicited throughout the design process (22%, 5/24). A third (15/45) of all studies reported on clinician trust in clinical AI algorithms and tools. The surveyed articles did not universally report validation against the “gold standard” of clinical expertise or provide detailed descriptions of the algorithms or computational methods used in their work. To realize the full potential of AI tools within healthcare settings, our review suggests there are opportunities to more thoroughly integrate frontline users’ needs and feedback in the design process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287
Human trust in artificial intelligence: Review of empirical research
Glikson, E., & Woolley, A. W. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0057 · Cited by 2,921
The rise of artificial intelligence in healthcare applications
Bohr, A., & Memarzadeh, K. (2020) · Artificial Intelligence in Healthcare · doi:10.1016/b978-0-12-818438-7.00002-2 · Cited by 1,636
Do no harm: a roadmap for responsible machine learning for health care
Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0548-6 · Cited by 1,266
Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995
Arndt, B. G., Beasley, J. W., Watkinson, M. D., Temte, J. L., Tuan, W.-J., Sinsky, C. A., & Gilchrist, V. J. (2017) · The Annals of Family Medicine · doi:10.1370/afm.2121 · Cited by 972 · cited 1×
“Clinical AI is currently proposed across multiple medical domains addressing issues such as clinician burnout (Arndt et al., 2017; Ash et al., 2019), medical errors (Cai et al., 2019a; Van Camp et al., 2019), and detecting frequently unrecognized and life-threatening conditions such as sepsis…”
BackgroundUser involvement: A review of the benefits and challenges
Kujala, S. (2003) · Behaviour & Information Technology · doi:10.1080/01449290301782 · Cited by 943 · cited 2×
“One method for achieving this goal is to involve the users in the beginning and throughout the design process through research activities (Kujala, 2010), especially as additional implicit and/or explicit needs can emerge over time.”
BackgroundMethodClinical decision support in the era of artificial intelligence
Shortliffe, E. H., & Sepúlveda, M. J. (2018) · JAMA · doi:10.1001/jama.2018.17163 · Cited by 920
Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594
Research works printed but not linked: 61
- Ash, 2019, How AI in the exam room could reduce physician burnout., Harv. Bus. Rev., 2
- Bates, 2020, Reporting and implementing interventions involving machine learning and artificial intelligence., Ann. Intern. Med., 172, S137, 10.7326/M19-0872 · DOI
- Benrimoh, 2021, Using a simulation centre to evaluate preliminary acceptability and impact of an artificial intelligence-powered clinical decision support system for depression treatment on the physician–patient interaction., BJPsych Open, 7, 10.1192/BJO.2020.127 · DOI
- Beres, 2019, Human-centered design lessons for implementation science: improving the implementation of a patient-centered care intervention., J. Acquir. Immune Defic. Syndr., S230, 10.1097/QAI.0000000000002216 · DOI
- Botwe, 2021, The integration of artificial intelligence in medical imaging practice: perspectives of African radiographers., Radiography, 27, 861, 10.1016/J.RADI.2021.01.008 · DOI
- Bourla, 2018, Psychiatrists’ attitudes toward disruptive new technologies: mixed-methods study., JMIR Ment. Health, 5, 10.2196/10240 · DOI · Paper page
- Bourla, 2020, Are student nurses ready for new technologies in mental health? Mixed-methods study., Nurse Educ. Today, 84, 10.1016/J.NEDT.2019.104240 · DOI
- Brice, 2020, Health professional digital capabilities frameworks: a scoping review., J. Multidiscip. Healthc., 13, 1375, 10.2147/JMDH.S269412 · DOI
- Chen, 2018, IBM’s Watson gave Unsafe Recommendations for Treating Cancer - The Verge.
- Choudhury, 2020, Role of artificial intelligence in patient safety outcomes: systematic literature review., JMIR Med. Inform., 8, 10.2196/18599 · DOI · Paper page
- Chute, 2019, Introducing care 4.0: an integrated care paradigm built on industry 4.0 capabilities., Int. J. Environ. Res. Public Health, 16, 10.3390/IJERPH16122247 · DOI
- Cohen, 2017, Acceptability among community healthcare nurses of intelligent wireless sensor-system technology for the rapid detection of health issues in home-dwelling older adults., Open Nurs. J., 11, 54, 10.2174/1874434601711010054 · DOI
- Coyle, 2009, Clinical evaluations and collaborative design: developing new technologies for mental healthcare interventions, Proceedings of the 27th International Conference on Human Factors in Computing Systems, CHI 2009, 10.1145/1518701.1519013 · DOI
- 2019, Python-Levenshtein PyPI.
- Donoso-Guzmán, 2018, An interactive relevance feedback interface for Evidence-Based Health Care, Proceedings of the International Conference on Intelligent User Interfaces, 10.1145/3172944.3172953 · DOI
- Ehtesham, 2019, Developing a new intelligent system for the diagnosis of oral medicine with case-based reasoning approach., Oral Dis., 25, 1555, 10.1111/ODI.13108 · DOI
- Fineout-Overholt, 2005, Transforming health care from the inside out: advancing evidence-based practice in the 21st century., J. Prof. Nurs., 21, 335, 10.1016/j.profnurs.2005.10 · doi:10.1016/j.profnurs.2005.10
- Gellner, 1965, The conduct of inquiry: methodology for behavioural science., Br. J. Sociol., 16, 28, 10.2307/588769 · DOI
- Gilbank, 2020, Designing for physician trust: toward a machine learning decision aid for radiation toxicity risk., Ergon. Des. Q. Hum. Fact. Applic., 28, 27, 10.1177/1064804619896172 · DOI
- Goss, 2019, A clinician survey of using speech recognition for clinical documentation in the electronic health record., Int. J. Med. Inform., 130, 10.1016/J.IJMEDINF.2019.07.017 · DOI
- Holmquist, 2017, Intelligence on tap., Interactions, 24, 28, 10.1145/3085571 · DOI
- Jacobs, 2021, Designing ai for trust and collaboration in time-constrained medical decisions: a sociotechnical lens, Proceedings of the Conference on Human Factors in Computing Systems, 10.1145/3411764.3445385 · DOI · Paper page
- Kelliher, 2018, Conversation: beyond black boxes: tackling artificial intelligence as a design material, Design as a Catalyst for Change – DRS International Conference 2018, 25-28 June, 10.21606/drs.2018.784 · doi:10.21606/drs.2018.784
- Khairat, 2018, Reasons for physicians not adopting clinical decision support systems: critical analysis., J. Med. Internet Res., 6, 10.2196/medinform.8912 · DOI · Paper page
- Khodabandehloo, 2021, HealthXAI: collaborative and explainable AI for supporting early diagnosis of cognitive decline., Fut. Gen. Comput. Syst., 116, 168, 10.1016/J.FUTURE.2020.10.030 · DOI
- Klakegg, 2018, Assisted Medication Management in Elderly Care Using Miniaturised Near-Infrared Spectroscopy, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1, 10.1145/3214272 · DOI
- Kulp, 2018, Design in the “Medical” wild: challenges of technology deployment, Proceedings of the Extended Abstracts on Human Factors in Computing Systems. CHI Conference, 10.1145/3170427.3188571 · DOI
- Kumar, 2020, OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases., J. Am. Med. Inform. Assoc., 27, 1850, 10.1093/JAMIA/OCAA190 · DOI
- Li, 2020, Developing a delivery science for artificial intelligence in healthcare., NPJ Dig. Med., 3, 10.1038/s41746-020-00318-y · DOI · Paper page
- Long, 2016, An observational study to evaluate the usability and intent to adopt an artificial intelligence–powered medication reconciliation tool., Interact. J. Med. Res., 5, 10.2196/ijmr.5462 · DOI
- Ltifi, 2020, Adapted visual analytics process for intelligent decision-making: application in a medical context., Int. J. Inform. Technol. Decis. Mak., 19, 241, 10.1142/S0219622019500470 · DOI
- Mamlin, 2007, Clinical decision support within the regenstrief medical record system, Clinical Decision Support Systems, 190, 10.1007/978-0-387-38319-4_9 · DOI
- Middleton, 2016, Clinical decision support: a 25 year retrospective and a 25 year vision., Yearb. Med. Inform. Suppl, 1, S103, 10.15265/IYS-2016-S034 · DOI · Paper page
- 2019, Taking Action Against Clinician Burnout., 10.17226/25521 · DOI · Paper page
- Nemeth, 2016, Revealing ICU cognitive work through naturalistic decision-making methods., J. Cogn. Eng. Decis. Mak., 10, 350, 10.1177/1555343416664845 · DOI
- Norman, 2002, Design of Everyday Things.
- Norman, 2005, Human-centered design considered harmful., Interactions, 12, 14, 10.1145/1070960.1070976 · DOI · Paper page
- Okolo, 2021, It cannot do all of my work: community healthworker perceptions of ai-enabled mobile health applications in rural india, Proceedings of the Conference on Human Factors in Computing Systems, 10.1145/3411764.3445420 · DOI
- Osman Andersen, 2021, Realizing AI in healthcare: challenges appearing in the wild, Proceedings of the Conference on Human Factors in Computing Systems, 10.1145/3411763.3441347 · DOI
- Poncette, 2020, Improvements in patient monitoring in the intensive care unit: survey study., J. Med. Internet Res., 22, 10.2196/19091 · DOI
- Sackett, 1996, Evidence based medicine: what it is and what it isn’t., Clin. Orthop. Relat. Res., 455, 3, 10.1136/bmj.312.7023.71 · DOI · Paper page
- Sakellarios, 2020, A cloud-based platform for the non-invasive management of coronary artery disease., Enterp. Inform. Syst., 14, 1102, 10.1080/17517575.2020.1746975 · DOI
- Salunke, 2014, Selenium Webdriver in Python: Learn with Examples.
- Schriger, 2017, Structured clinical decision aids are seldom compared with subjective physician judgment, and are seldom superior., Ann. Emerg. Med., 70, 338, 10.1016/j.annemergmed.2016.12.004 · DOI
- Schwartz, 2021, Clinician involvement in research on machine learning–based predictive clinical decision support for the hospital setting: a scoping review., J. Am. Med. Inform. Assoc., 28, 653, 10.1093/JAMIA/OCAA296 · DOI
- Sendak, 2019, Machine learning in health care: a critical appraisal of challenges and opportunities., EGEMS, 7, 10.5334/EGEMS.287 · DOI
- Sim, 2001, Clinical decision support systems for the practice of evidence-based medicine., J. Am. Med. Inform. Assoc., 8, 527, 10.1136/jamia.2001.0080527 · DOI · Paper page
- Sittig, 2008, Grand challenges in clinical decision support., J. Biomed. Inform., 41, 387, 10.1016/J.JBI.2007.09.003 · DOI · Paper page
- Sligo, 2017, A literature review for large-scale health information system project planning, implementation and evaluation., Int. J. Med. Inform., 97, 86, 10.1016/j.ijmedinf.2016.09.007 · DOI · Paper page
- Tcheng, 2017, The Learning Health System Series Optimizing Strategies for CLINICAL DECISION SUPPORT Summary of a Meeting Series.
- Torrents-Barrena, 2019, TTTS-GPS: patient-specific preoperative planning and simulation platform for twin-to-twin transfusion syndrome fetal surgery., Comput. Methods Programs Biomed., 179, 10.1016/J.CMPB.2019.104993 · DOI · Paper page
- Trivedi, 2019, Interactive NLP in clinical care: identifying incidental findings in radiology reports., Appl. Clin. Inform., 10, 655, 10.1055/S-0039-1695791 · DOI
- Trivedi, 2018, NLPReViz: an interactive tool for natural language processing on clinical text., J. Am. Med. Inform. Assoc., 25, 10.1093/JAMIA/OCX070 · DOI
- Tschandl, 2020, Human–computer collaboration for skin cancer recognition., Nat. Med., 26, 1229, 10.1038/s41591-020-0942-0 · DOI · Paper page
- Umer, 2019, A decision support system for diagnostics and treatment planning in traumatic brain injury., IEEE J. Biomed. Health Inform., 23, 1261, 10.1109/JBHI.2018.2842717 · DOI
- Van Camp, 2019, Development and preliminary evaluation of a visual annotation tool to rapidly collect expert-annotated weight errors in pediatric growth charts., Stud. Health Technol. Inform., 264, 853, 10.3233/SHTI190344 · doi:10.3233/shti190344
- Waymel, 2019, Impact of the rise of artificial intelligence in radiology: what do radiologists think?, Diagn. Interv. Imaging, 100, 327, 10.1016/J.DIII.2019.03.015 · DOI · Paper page
- Wheelock, 2020, Human-centered design and trust in medicine., JAMA, 324, 2369, 10.1001/JAMA.2020.21080 · DOI
- Xu, 2018, ECGLens: interactive visual exploration of large scale ECG data for arrhythmia detection, Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems CHI’18, 10.1145/3173574.3174237 · DOI
- Yang, 2018, Mapping machine learning advances from HCI research to reveal starting places for design innovation, Proceedings of the Conference on Human Factors in Computing Systems, 10.1145/3173574.3173704 · DOI
- Zhang, 2018, Developing and maintaining clinical decision support using clinical knowledge and machine learning: the case of order sets., J. Am. Med. Inform. Assoc., 25, 1547, 10.1093/jamia/ocy099 · DOI
Cited by
Artificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
Benefits and Risks of AI in Health Care: Narrative Review
Chustecki, M. (2024) · Interactive Journal of Medical Research · doi:10.2196/53616 · Cited by 208
“However, it further explores the dynamic interaction between AI’s promise and the necessity for rigorous validation and ethical governance, as discussed by Macrae [72] and Tulk Jesso et al [73].”
Yildirim, N., Richardson, H., Wetscherek, M. T., Bajwa, J., Jacob, J., Pinnock, M. A., Harris, S., Coelho De Castro, D., Bannur, S., Hyland, S., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642013 · Cited by 82
Thieme, A., Rajamohan, A., Cooper, B., Groombridge, H., Simister, R., Wong, B., Woznitza, N., Pinnock, M. A., Wetscherek, M. T., Morrison, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.05299 · Cited by 10
When AI is wrong: addressing liability challenges in women's healthcare
Marotta, A. (2022) · Journal of Computer Information Systems · doi:10.1080/08874417.2022.2089773 · Cited by 7
Metadata
- Authors
- Tulk Jesso, S., Kelliher, A., Sanghavi, H., Martin, T., & Henrickson Parker, S.
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2022-04-07
- Volume
- 13
- Pages
- 830345-830345
- ISSN
- 1664-1078 (ISSN-L)
- OpenAlex
- W4226049092
- Cited by
- 54
- References
- 30
- FWCI
- 0.91
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review. https://app.cochat.ai/papers/10.3389/fpsyg.2022.830345
- The paper
- Tulk Jesso, S., Kelliher, A., Sanghavi, H., Martin, T., & Henrickson Parker, S. (2022). Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2022.830345
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.