CoChat

Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review

Open access
Tulk Jesso, Stephanie, Kelliher, Aisling, Sanghavi, Harsh, Martin, Thomas, Henrickson Parker, Sarah · 2022 · Frontiers in Psychology
Cited by 54

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.

Abstract

The application of machine learning (ML) and artificial intelligence (AI) in healthcare domains has received much attention in recent years, yet significant questions remain about how these new tools integrate into frontline user workflow, and how their design will impact implementation. Lack of acceptance among clinicians is a major barrier to the translation of healthcare innovations into clinical practice. In this systematic review, we examine when and how clinicians are consulted about their needs and desires for clinical AI tools. Forty-five articles met criteria for inclusion, of which 24 were considered design studies. The design studies used a variety of methods to solicit and gather user feedback, with interviews, surveys, and user evaluations. Our findings show that tool designers consult clinicians at various but inconsistent points during the design process, and most typically at later stages in the design cycle (82%, 19/24 design studies). We also observed a smaller amount of studies adopting a human-centered approach and where clinician input was solicited throughout the design process (22%, 5/24). A third (15/45) of all studies reported on clinician trust in clinical AI algorithms and tools. The surveyed articles did not universally report validation against the “gold standard” of clinical expertise or provide detailed descriptions of the algorithms or computational methods used in their work. To realize the full potential of AI tools within healthcare settings, our review suggests there are opportunities to more thoroughly integrate frontline users’ needs and feedback in the design process.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fpsyg.2022.830345

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 58

  1. Ash, 2019, How AI in the exam room could reduce physician burnout., Harv. Bus. Rev., 2
  2. Bates, 2020, Reporting and implementing interventions involving machine learning and artificial intelligence., Ann. Intern. Med., 172, S137, 10.7326/M19-0872 · DOI · Paper page
  3. Benrimoh, 2021, Using a simulation centre to evaluate preliminary acceptability and impact of an artificial intelligence-powered clinical decision support system for depression treatment on the physician–patient interaction., BJPsych Open, 7, 10.1192/BJO.2020.127 · DOI
  4. Beres, 2019, Human-centered design lessons for implementation science: improving the implementation of a patient-centered care intervention., J. Acquir. Immune Defic. Syndr., S230, 10.1097/QAI.0000000000002216 · DOI
  5. Botwe, 2021, The integration of artificial intelligence in medical imaging practice: perspectives of African radiographers., Radiography, 27, 861, 10.1016/J.RADI.2021.01.008 · DOI · Paper page
  6. Bourla, 2018, Psychiatrists’ attitudes toward disruptive new technologies: mixed-methods study., JMIR Ment. Health, 5, 10.2196/10240 · DOI · Paper page
  7. Bourla, 2020, Are student nurses ready for new technologies in mental health? Mixed-methods study., Nurse Educ. Today, 84, 10.1016/J.NEDT.2019.104240 · DOI
  8. Brice, 2020, Health professional digital capabilities frameworks: a scoping review., J. Multidiscip. Healthc., 13, 1375, 10.2147/JMDH.S269412 · DOI
  9. Chen, 2018, IBM’s Watson gave Unsafe Recommendations for Treating Cancer - The Verge.
  10. Choudhury, 2020, Role of artificial intelligence in patient safety outcomes: systematic literature review., JMIR Med. Inform., 8, 10.2196/18599 · DOI · Paper page
  11. Chute, 2019, Introducing care 4.0: an integrated care paradigm built on industry 4.0 capabilities., Int. J. Environ. Res. Public Health, 16, 10.3390/IJERPH16122247 · DOI
  12. Cohen, 2017, Acceptability among community healthcare nurses of intelligent wireless sensor-system technology for the rapid detection of health issues in home-dwelling older adults., Open Nurs. J., 11, 54, 10.2174/1874434601711010054 · DOI
  13. Coyle, 2009, Clinical evaluations and collaborative design: developing new technologies for mental healthcare interventions, Proceedings of the 27th International Conference on Human Factors in Computing Systems, CHI 2009, 10.1145/1518701.1519013 · DOI · Paper page
  14. 2019, Python-Levenshtein PyPI.
  15. Donoso-Guzmán, 2018, An interactive relevance feedback interface for Evidence-Based Health Care, Proceedings of the International Conference on Intelligent User Interfaces, 10.1145/3172944.3172953 · DOI
  16. Ehtesham, 2019, Developing a new intelligent system for the diagnosis of oral medicine with case-based reasoning approach., Oral Dis., 25, 1555, 10.1111/ODI.13108 · DOI
  17. Fineout-Overholt, 2005, Transforming health care from the inside out: advancing evidence-based practice in the 21st century., J. Prof. Nurs., 21, 335, 10.1016/j.profnurs.2005.10 · doi:10.1016/j.profnurs.2005.10
  18. Gellner, 1965, The conduct of inquiry: methodology for behavioural science., Br. J. Sociol., 16, 28, 10.2307/588769 · DOI
  19. Gilbank, 2020, Designing for physician trust: toward a machine learning decision aid for radiation toxicity risk., Ergon. Des. Q. Hum. Fact. Applic., 28, 27, 10.1177/1064804619896172 · DOI
  20. Goss, 2019, A clinician survey of using speech recognition for clinical documentation in the electronic health record., Int. J. Med. Inform., 130, 10.1016/J.IJMEDINF.2019.07.017 · DOI · Paper page
  21. Holmquist, 2017, Intelligence on tap., Interactions, 24, 28, 10.1145/3085571 · DOI · Paper page
  22. Kelliher, 2018, Conversation: beyond black boxes: tackling artificial intelligence as a design material, Design as a Catalyst for Change – DRS International Conference 2018, 25-28 June, 10.21606/drs.2018.784 · doi:10.21606/drs.2018.784
  23. Khairat, 2018, Reasons for physicians not adopting clinical decision support systems: critical analysis., J. Med. Internet Res., 6, 10.2196/medinform.8912 · DOI · Paper page
  24. Khodabandehloo, 2021, HealthXAI: collaborative and explainable AI for supporting early diagnosis of cognitive decline., Fut. Gen. Comput. Syst., 116, 168, 10.1016/J.FUTURE.2020.10.030 · DOI · Paper page
  25. Klakegg, 2018, Assisted Medication Management in Elderly Care Using Miniaturised Near-Infrared Spectroscopy, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1, 10.1145/3214272 · DOI
  26. Kulp, 2018, Design in the “Medical” wild: challenges of technology deployment, Proceedings of the Extended Abstracts on Human Factors in Computing Systems. CHI Conference, 10.1145/3170427.3188571 · DOI
  27. Kumar, 2020, OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases., J. Am. Med. Inform. Assoc., 27, 1850, 10.1093/JAMIA/OCAA190 · DOI
  28. Li, 2020, Developing a delivery science for artificial intelligence in healthcare., NPJ Dig. Med., 3, 10.1038/s41746-020-00318-y · DOI · Paper page
  29. Long, 2016, An observational study to evaluate the usability and intent to adopt an artificial intelligence–powered medication reconciliation tool., Interact. J. Med. Res., 5, 10.2196/ijmr.5462 · DOI
  30. Ltifi, 2020, Adapted visual analytics process for intelligent decision-making: application in a medical context., Int. J. Inform. Technol. Decis. Mak., 19, 241, 10.1142/S0219622019500470 · DOI
  31. Mamlin, 2007, Clinical decision support within the regenstrief medical record system, Clinical Decision Support Systems, 190, 10.1007/978-0-387-38319-4_9 · DOI
  32. Middleton, 2016, Clinical decision support: a 25 year retrospective and a 25 year vision., Yearb. Med. Inform. Suppl, 1, S103, 10.15265/IYS-2016-S034 · DOI · Paper page
  33. 2019, Taking Action Against Clinician Burnout., 10.17226/25521 · DOI · Paper page
  34. Nemeth, 2016, Revealing ICU cognitive work through naturalistic decision-making methods., J. Cogn. Eng. Decis. Mak., 10, 350, 10.1177/1555343416664845 · DOI
  35. Norman, 2002, Design of Everyday Things.
  36. Norman, 2005, Human-centered design considered harmful., Interactions, 12, 14, 10.1145/1070960.1070976 · DOI · Paper page
  37. Okolo, 2021, It cannot do all of my work: community healthworker perceptions of ai-enabled mobile health applications in rural india, Proceedings of the Conference on Human Factors in Computing Systems, 10.1145/3411764.3445420 · DOI · Paper page
  38. Osman Andersen, 2021, Realizing AI in healthcare: challenges appearing in the wild, Proceedings of the Conference on Human Factors in Computing Systems, 10.1145/3411763.3441347 · DOI · Paper page
  39. Poncette, 2020, Improvements in patient monitoring in the intensive care unit: survey study., J. Med. Internet Res., 22, 10.2196/19091 · DOI · Paper page
  40. Sackett, 1996, Evidence based medicine: what it is and what it isn’t., Clin. Orthop. Relat. Res., 455, 3, 10.1136/bmj.312.7023.71 · DOI · Paper page
  41. Sakellarios, 2020, A cloud-based platform for the non-invasive management of coronary artery disease., Enterp. Inform. Syst., 14, 1102, 10.1080/17517575.2020.1746975 · DOI
  42. Salunke, 2014, Selenium Webdriver in Python: Learn with Examples.
  43. Schriger, 2017, Structured clinical decision aids are seldom compared with subjective physician judgment, and are seldom superior., Ann. Emerg. Med., 70, 338, 10.1016/j.annemergmed.2016.12.004 · DOI
  44. Schwartz, 2021, Clinician involvement in research on machine learning–based predictive clinical decision support for the hospital setting: a scoping review., J. Am. Med. Inform. Assoc., 28, 653, 10.1093/JAMIA/OCAA296 · DOI · Paper page
  45. Sendak, 2019, Machine learning in health care: a critical appraisal of challenges and opportunities., EGEMS, 7, 10.5334/EGEMS.287 · DOI · Paper page
  46. Sim, 2001, Clinical decision support systems for the practice of evidence-based medicine., J. Am. Med. Inform. Assoc., 8, 527, 10.1136/jamia.2001.0080527 · DOI · Paper page
  47. Sligo, 2017, A literature review for large-scale health information system project planning, implementation and evaluation., Int. J. Med. Inform., 97, 86, 10.1016/j.ijmedinf.2016.09.007 · DOI · Paper page
  48. Tcheng, 2017, The Learning Health System Series Optimizing Strategies for CLINICAL DECISION SUPPORT Summary of a Meeting Series.
  49. Torrents-Barrena, 2019, TTTS-GPS: patient-specific preoperative planning and simulation platform for twin-to-twin transfusion syndrome fetal surgery., Comput. Methods Programs Biomed., 179, 10.1016/J.CMPB.2019.104993 · DOI · Paper page
  50. Trivedi, 2019, Interactive NLP in clinical care: identifying incidental findings in radiology reports., Appl. Clin. Inform., 10, 655, 10.1055/S-0039-1695791 · DOI
  51. Trivedi, 2018, NLPReViz: an interactive tool for natural language processing on clinical text., J. Am. Med. Inform. Assoc., 25, 10.1093/JAMIA/OCX070 · DOI
  52. Umer, 2019, A decision support system for diagnostics and treatment planning in traumatic brain injury., IEEE J. Biomed. Health Inform., 23, 1261, 10.1109/JBHI.2018.2842717 · DOI
  53. Van Camp, 2019, Development and preliminary evaluation of a visual annotation tool to rapidly collect expert-annotated weight errors in pediatric growth charts., Stud. Health Technol. Inform., 264, 853, 10.3233/SHTI190344 · doi:10.3233/shti190344
  54. Waymel, 2019, Impact of the rise of artificial intelligence in radiology: what do radiologists think?, Diagn. Interv. Imaging, 100, 327, 10.1016/J.DIII.2019.03.015 · DOI · Paper page
  55. Wheelock, 2020, Human-centered design and trust in medicine., JAMA, 324, 2369, 10.1001/JAMA.2020.21080 · DOI
  56. Xu, 2018, ECGLens: interactive visual exploration of large scale ECG data for arrhythmia detection, Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems CHI’18, 10.1145/3173574.3174237 · DOI
  57. Yang, 2018, Mapping machine learning advances from HCI research to reveal starting places for design innovation, Proceedings of the Conference on Human Factors in Computing Systems, 10.1145/3173574.3173704 · DOI · Paper page
  58. Zhang, 2018, Developing and maintaining clinical decision support using clinical knowledge and machine learning: the case of order sets., J. Am. Med. Inform. Assoc., 25, 1547, 10.1093/jamia/ocy099 · DOI

Cited by

See all 54

Metadata

Authors
Tulk Jesso, S., Kelliher, A., Sanghavi, H., Martin, T., & Henrickson Parker, S.
Venue
Frontiers in Psychology
Published
2022-04-07
Volume
13
Pages
830345-830345
ISSN
1664-1078 (ISSN-L)
OpenAlex
W4226049092
Cited by
54
References
33
FWCI
0.91
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.