CoChat

Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review

Open access
Tulk Jesso, Stephanie, Kelliher, Aisling, Sanghavi, Harsh, Martin, Thomas, Henrickson Parker, Sarah · 2022 · Frontiers in Psychology
Cited by 54

The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.

Abstract

The application of machine learning (ML) and artificial intelligence (AI) in healthcare domains has received much attention in recent years, yet significant questions remain about how these new tools integrate into frontline user workflow, and how their design will impact implementation. Lack of acceptance among clinicians is a major barrier to the translation of healthcare innovations into clinical practice. In this systematic review, we examine when and how clinicians are consulted about their needs and desires for clinical AI tools. Forty-five articles met criteria for inclusion, of which 24 were considered design studies. The design studies used a variety of methods to solicit and gather user feedback, with interviews, surveys, and user evaluations. Our findings show that tool designers consult clinicians at various but inconsistent points during the design process, and most typically at later stages in the design cycle (82%, 19/24 design studies). We also observed a smaller amount of studies adopting a human-centered approach and where clinician input was solicited throughout the design process (22%, 5/24). A third (15/45) of all studies reported on clinician trust in clinical AI algorithms and tools. The surveyed articles did not universally report validation against the “gold standard” of clinical expertise or provide detailed descriptions of the algorithms or computational methods used in their work. To realize the full potential of AI tools within healthcare settings, our review suggests there are opportunities to more thoroughly integrate frontline users’ needs and feedback in the design process.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fpsyg.2022.830345

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 30 references

Research works printed but not linked: 61

  1. Ash, 2019, How AI in the exam room could reduce physician burnout., Harv. Bus. Rev., 2
  2. Bates, 2020, Reporting and implementing interventions involving machine learning and artificial intelligence., Ann. Intern. Med., 172, S137, 10.7326/M19-0872 · DOI
  3. Benrimoh, 2021, Using a simulation centre to evaluate preliminary acceptability and impact of an artificial intelligence-powered clinical decision support system for depression treatment on the physician–patient interaction., BJPsych Open, 7, 10.1192/BJO.2020.127 · DOI
  4. Beres, 2019, Human-centered design lessons for implementation science: improving the implementation of a patient-centered care intervention., J. Acquir. Immune Defic. Syndr., S230, 10.1097/QAI.0000000000002216 · DOI
  5. Botwe, 2021, The integration of artificial intelligence in medical imaging practice: perspectives of African radiographers., Radiography, 27, 861, 10.1016/J.RADI.2021.01.008 · DOI
  6. Bourla, 2018, Psychiatrists’ attitudes toward disruptive new technologies: mixed-methods study., JMIR Ment. Health, 5, 10.2196/10240 · DOI · Paper page
  7. Bourla, 2020, Are student nurses ready for new technologies in mental health? Mixed-methods study., Nurse Educ. Today, 84, 10.1016/J.NEDT.2019.104240 · DOI
  8. Brice, 2020, Health professional digital capabilities frameworks: a scoping review., J. Multidiscip. Healthc., 13, 1375, 10.2147/JMDH.S269412 · DOI
  9. Chen, 2018, IBM’s Watson gave Unsafe Recommendations for Treating Cancer - The Verge.
  10. Choudhury, 2020, Role of artificial intelligence in patient safety outcomes: systematic literature review., JMIR Med. Inform., 8, 10.2196/18599 · DOI · Paper page
  11. Chute, 2019, Introducing care 4.0: an integrated care paradigm built on industry 4.0 capabilities., Int. J. Environ. Res. Public Health, 16, 10.3390/IJERPH16122247 · DOI
  12. Cohen, 2017, Acceptability among community healthcare nurses of intelligent wireless sensor-system technology for the rapid detection of health issues in home-dwelling older adults., Open Nurs. J., 11, 54, 10.2174/1874434601711010054 · DOI
  13. Coyle, 2009, Clinical evaluations and collaborative design: developing new technologies for mental healthcare interventions, Proceedings of the 27th International Conference on Human Factors in Computing Systems, CHI 2009, 10.1145/1518701.1519013 · DOI
  14. 2019, Python-Levenshtein PyPI.
  15. Donoso-Guzmán, 2018, An interactive relevance feedback interface for Evidence-Based Health Care, Proceedings of the International Conference on Intelligent User Interfaces, 10.1145/3172944.3172953 · DOI
  16. Ehtesham, 2019, Developing a new intelligent system for the diagnosis of oral medicine with case-based reasoning approach., Oral Dis., 25, 1555, 10.1111/ODI.13108 · DOI
  17. Fineout-Overholt, 2005, Transforming health care from the inside out: advancing evidence-based practice in the 21st century., J. Prof. Nurs., 21, 335, 10.1016/j.profnurs.2005.10 · doi:10.1016/j.profnurs.2005.10
  18. Gellner, 1965, The conduct of inquiry: methodology for behavioural science., Br. J. Sociol., 16, 28, 10.2307/588769 · DOI
  19. Gilbank, 2020, Designing for physician trust: toward a machine learning decision aid for radiation toxicity risk., Ergon. Des. Q. Hum. Fact. Applic., 28, 27, 10.1177/1064804619896172 · DOI
  20. Goss, 2019, A clinician survey of using speech recognition for clinical documentation in the electronic health record., Int. J. Med. Inform., 130, 10.1016/J.IJMEDINF.2019.07.017 · DOI
  21. Holmquist, 2017, Intelligence on tap., Interactions, 24, 28, 10.1145/3085571 · DOI
  22. Jacobs, 2021, Designing ai for trust and collaboration in time-constrained medical decisions: a sociotechnical lens, Proceedings of the Conference on Human Factors in Computing Systems, 10.1145/3411764.3445385 · DOI · Paper page
  23. Kelliher, 2018, Conversation: beyond black boxes: tackling artificial intelligence as a design material, Design as a Catalyst for Change – DRS International Conference 2018, 25-28 June, 10.21606/drs.2018.784 · doi:10.21606/drs.2018.784
  24. Khairat, 2018, Reasons for physicians not adopting clinical decision support systems: critical analysis., J. Med. Internet Res., 6, 10.2196/medinform.8912 · DOI · Paper page
  25. Khodabandehloo, 2021, HealthXAI: collaborative and explainable AI for supporting early diagnosis of cognitive decline., Fut. Gen. Comput. Syst., 116, 168, 10.1016/J.FUTURE.2020.10.030 · DOI
  26. Klakegg, 2018, Assisted Medication Management in Elderly Care Using Miniaturised Near-Infrared Spectroscopy, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1, 10.1145/3214272 · DOI
  27. Kulp, 2018, Design in the “Medical” wild: challenges of technology deployment, Proceedings of the Extended Abstracts on Human Factors in Computing Systems. CHI Conference, 10.1145/3170427.3188571 · DOI
  28. Kumar, 2020, OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases., J. Am. Med. Inform. Assoc., 27, 1850, 10.1093/JAMIA/OCAA190 · DOI
  29. Li, 2020, Developing a delivery science for artificial intelligence in healthcare., NPJ Dig. Med., 3, 10.1038/s41746-020-00318-y · DOI · Paper page
  30. Long, 2016, An observational study to evaluate the usability and intent to adopt an artificial intelligence–powered medication reconciliation tool., Interact. J. Med. Res., 5, 10.2196/ijmr.5462 · DOI
  31. Ltifi, 2020, Adapted visual analytics process for intelligent decision-making: application in a medical context., Int. J. Inform. Technol. Decis. Mak., 19, 241, 10.1142/S0219622019500470 · DOI
  32. Mamlin, 2007, Clinical decision support within the regenstrief medical record system, Clinical Decision Support Systems, 190, 10.1007/978-0-387-38319-4_9 · DOI
  33. Middleton, 2016, Clinical decision support: a 25 year retrospective and a 25 year vision., Yearb. Med. Inform. Suppl, 1, S103, 10.15265/IYS-2016-S034 · DOI · Paper page
  34. 2019, Taking Action Against Clinician Burnout., 10.17226/25521 · DOI · Paper page
  35. Nemeth, 2016, Revealing ICU cognitive work through naturalistic decision-making methods., J. Cogn. Eng. Decis. Mak., 10, 350, 10.1177/1555343416664845 · DOI
  36. Norman, 2002, Design of Everyday Things.
  37. Norman, 2005, Human-centered design considered harmful., Interactions, 12, 14, 10.1145/1070960.1070976 · DOI · Paper page
  38. Okolo, 2021, It cannot do all of my work: community healthworker perceptions of ai-enabled mobile health applications in rural india, Proceedings of the Conference on Human Factors in Computing Systems, 10.1145/3411764.3445420 · DOI
  39. Osman Andersen, 2021, Realizing AI in healthcare: challenges appearing in the wild, Proceedings of the Conference on Human Factors in Computing Systems, 10.1145/3411763.3441347 · DOI
  40. Poncette, 2020, Improvements in patient monitoring in the intensive care unit: survey study., J. Med. Internet Res., 22, 10.2196/19091 · DOI
  41. Sackett, 1996, Evidence based medicine: what it is and what it isn’t., Clin. Orthop. Relat. Res., 455, 3, 10.1136/bmj.312.7023.71 · DOI · Paper page
  42. Sakellarios, 2020, A cloud-based platform for the non-invasive management of coronary artery disease., Enterp. Inform. Syst., 14, 1102, 10.1080/17517575.2020.1746975 · DOI
  43. Salunke, 2014, Selenium Webdriver in Python: Learn with Examples.
  44. Schriger, 2017, Structured clinical decision aids are seldom compared with subjective physician judgment, and are seldom superior., Ann. Emerg. Med., 70, 338, 10.1016/j.annemergmed.2016.12.004 · DOI
  45. Schwartz, 2021, Clinician involvement in research on machine learning–based predictive clinical decision support for the hospital setting: a scoping review., J. Am. Med. Inform. Assoc., 28, 653, 10.1093/JAMIA/OCAA296 · DOI
  46. Sendak, 2019, Machine learning in health care: a critical appraisal of challenges and opportunities., EGEMS, 7, 10.5334/EGEMS.287 · DOI
  47. Sim, 2001, Clinical decision support systems for the practice of evidence-based medicine., J. Am. Med. Inform. Assoc., 8, 527, 10.1136/jamia.2001.0080527 · DOI · Paper page
  48. Sittig, 2008, Grand challenges in clinical decision support., J. Biomed. Inform., 41, 387, 10.1016/J.JBI.2007.09.003 · DOI · Paper page
  49. Sligo, 2017, A literature review for large-scale health information system project planning, implementation and evaluation., Int. J. Med. Inform., 97, 86, 10.1016/j.ijmedinf.2016.09.007 · DOI · Paper page
  50. Tcheng, 2017, The Learning Health System Series Optimizing Strategies for CLINICAL DECISION SUPPORT Summary of a Meeting Series.
  51. Torrents-Barrena, 2019, TTTS-GPS: patient-specific preoperative planning and simulation platform for twin-to-twin transfusion syndrome fetal surgery., Comput. Methods Programs Biomed., 179, 10.1016/J.CMPB.2019.104993 · DOI · Paper page
  52. Trivedi, 2019, Interactive NLP in clinical care: identifying incidental findings in radiology reports., Appl. Clin. Inform., 10, 655, 10.1055/S-0039-1695791 · DOI
  53. Trivedi, 2018, NLPReViz: an interactive tool for natural language processing on clinical text., J. Am. Med. Inform. Assoc., 25, 10.1093/JAMIA/OCX070 · DOI
  54. Tschandl, 2020, Human–computer collaboration for skin cancer recognition., Nat. Med., 26, 1229, 10.1038/s41591-020-0942-0 · DOI · Paper page
  55. Umer, 2019, A decision support system for diagnostics and treatment planning in traumatic brain injury., IEEE J. Biomed. Health Inform., 23, 1261, 10.1109/JBHI.2018.2842717 · DOI
  56. Van Camp, 2019, Development and preliminary evaluation of a visual annotation tool to rapidly collect expert-annotated weight errors in pediatric growth charts., Stud. Health Technol. Inform., 264, 853, 10.3233/SHTI190344 · doi:10.3233/shti190344
  57. Waymel, 2019, Impact of the rise of artificial intelligence in radiology: what do radiologists think?, Diagn. Interv. Imaging, 100, 327, 10.1016/J.DIII.2019.03.015 · DOI · Paper page
  58. Wheelock, 2020, Human-centered design and trust in medicine., JAMA, 324, 2369, 10.1001/JAMA.2020.21080 · DOI
  59. Xu, 2018, ECGLens: interactive visual exploration of large scale ECG data for arrhythmia detection, Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems CHI’18, 10.1145/3173574.3174237 · DOI
  60. Yang, 2018, Mapping machine learning advances from HCI research to reveal starting places for design innovation, Proceedings of the Conference on Human Factors in Computing Systems, 10.1145/3173574.3173704 · DOI
  61. Zhang, 2018, Developing and maintaining clinical decision support using clinical knowledge and machine learning: the case of order sets., J. Am. Med. Inform. Assoc., 25, 1547, 10.1093/jamia/ocy099 · DOI

Cited by

See all 54

Metadata

Authors
Tulk Jesso, S., Kelliher, A., Sanghavi, H., Martin, T., & Henrickson Parker, S.
Venue
Frontiers in Psychology
Published
2022-04-07
Volume
13
Pages
830345-830345
ISSN
1664-1078 (ISSN-L)
OpenAlex
W4226049092
Cited by
54
References
30
FWCI
0.91
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review. https://app.cochat.ai/papers/10.3389/fpsyg.2022.830345
The paper
Tulk Jesso, S., Kelliher, A., Sanghavi, H., Martin, T., & Henrickson Parker, S. (2022). Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2022.830345

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.