Clinician-Facing AI in the Wild: Taking Stock of the Sociotechnical Challenges and Opportunities for HCI
Open accessThe DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
Artificial Intelligence (AI) in medical applications holds great promise. However, the use of Machine Learning-based (ML) systems in clinical practice is still minimal. It is uniquely difficult to introduce clinician-facing ML-based systems in practice, which has been recognised in HCI and related fields. Recent publications have begun to address the sociotechnical challenges of designing, developing, and successfully deploying clinician-facing ML-based systems. We conducted a qualitative systematic review and provided answers to the question: “How can HCI researchers and practitioners contribute to the successful realisation of ML in medical practice?” We reviewed 25 eligible papers that investigated the real-world clinical implications of concrete clinician-facing ML-based systems. The main contributions of this systematic review are: (1) an overview of the technical aspects of ML innovation and their consequences for HCI researchers and practitioners; (2) a description of the different roles that ML-based systems can take in clinical settings; (3) a conceptualisation of the main activities of medical ML innovation processes; (4) identification of five sociotechnical interdependencies that emerge from medical ML innovation; and (5) implications for HCI researchers and practitioners on how to mitigate the sociotechnical challenges of medical ML innovation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Framework for design and evaluation of complex interventions to improve health
Campbell, M. (2000) · BMJ · doi:10.1136/bmj.321.7262.694 · Cited by 3,474 · cited 1×
“Historically, based on drug development processes, Medical Research Council in the UK guided the development and evaluation of complex interventions [26].”
BackgroundKey challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 2×
“[5] provided guidelines for designing effective human-AI interaction.”
BackgroundDixon-Woods, M., Cavers, D., Agarwal, S., Annandale, E., Arthur, A., Harvey, J., Hsu, R., Katbamna, S., Olsen, R., Smith, L., et al. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-35 · Cited by 2,289
Can we open the black box of AI?
Castelvecchi, D. (2016) · Nature · doi:10.1038/538020a · Cited by 1,773 · cited 1×
“Unlike rule-based algorithms, whose decision-making processes can be inspected, ML models are often criticised for the “black-box” problem [27], i.”
BackgroundHuman-Centered Artificial Intelligence: Reliable, Safe & Trustworthy
Shneiderman, B. (2020) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1741118 · Cited by 1,743 · cited 1×
“Studies have beneited a user- and human-centred perspective in the pursuit of developing fair, accountable, and useful computer applications that increase automation while augmenting and empowering people [107, 110].”
BackgroundFairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631
Diagnostic error in internal medicine
Graber, M. L., Franklin, N., & Gordon, R. (2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.13.1493 · Cited by 1,549
Statistics versus machine learning
Bzdok, D., Altman, N., & Krzywinski, M. (2018) · Nature Methods · doi:10.1038/nmeth.4642 · Cited by 1,508 · cited 2×
“In this systematic literature review, we focus on Machine Learning (ML) algorithmsśa subset of AI that can ind patterns and dependencies in complex and unwieldy data [23] without being explicitly programmed [117].”
BackgroundMethodDo no harm: a roadmap for responsible machine learning for health care
Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0548-6 · Cited by 1,266
Using meta ethnography to synthesise qualitative research: a worked example
Britten, N., Campbell, R., Pope, C., Donovan, J., Morgan, M., & Pill, R. (2002) · Journal of Health Services Research & Policy · doi:10.1258/135581902320432732 · Cited by 1,110
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084
Interactive machine learning for health informatics: when do we need the human-in-the-loop?
Holzinger, A. (2016) · Brain Informatics · doi:10.1007/s40708-016-0042-6 · Cited by 906
Patient care information systems and health care work: a sociotechnical approach
Berg, M. (1999) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(99)00011-8 · Cited by 786 · cited 4×
“They argued that attempts to structure this work through the formal, standardised, and rational nature of IT systems is challenging and that optimal utilisation of health IT applications is “dependent on the meticulous interrelation of the system’s functioning with the skilled and pragmatically oriented work of health…”
BackgroundInfluentialA Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743
Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376301 · Cited by 622
Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614 · cited 9×
“Such output could add gravity to otherwise ignored points raised by staff members with less expertise, e.g., nurses or junior physicians [9, 46, 125] or when trying to convince a patient [9, 82].”
BackgroundMethodInfluentialCai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 5×
“…[106] Case study: “to validate ... the refinement mechanisms”, “how SMILY ... affects user experience and search practices during a medical task” [25], “provide evidence [of] the usefulness of the system” [64] Interview: “usefulness, ease of use, general pros and cons of the prototype system,…”
BackgroundInfluentialDilsizian, S. E., & Siegel, E. L. (2014) · Current Cardiology Reports · doi:10.1007/s11886-013-0441-8 · Cited by 579
Why and why not explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552 · cited 2×
“Research provided guidelines for increasing acceptance of AI-infused systems by improving the interpretability of ML output through explainable and interactive interfaces (see, e.g., [3, 30, 71, 78, 117].”
BackgroundCognitive and System Factors Contributing to Diagnostic Errors in Radiology
Lee, C. S., Nagy, P. G., Weaver, S. J., & Newman-Toker, D. E. (2013) · American Journal of Roentgenology · doi:10.2214/ajr.12.10375 · Cited by 415
Thieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406
Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405
Koza, J. R., Bennett, F. H., Andre, D., & Keane, M. A. (1996) · Artificial Intelligence in Design ’96 · doi:10.1007/978-94-009-0279-4_9 · Cited by 361
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · cited 2×
“HCI contributions have centred on tackling Human-AI interaction issues by developing conceptual frameworks [117], models and principles [30, 39, 75], methods and techniques [29, 71, 120], and design guidelines [5] for ML-based systems, as well as undertaking experiments with users to empirically explore the challenges…”
BackgroundSendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., Nichols, M., Revoir, M., Yashar, F., Miller, C., et al. (2020) · JMIR Medical Informatics · doi:10.2196/15182 · Cited by 287
Explainability fact sheets: a framework for systematic assessment of explainable approaches
Sokol, K., & Flach, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372870 · Cited by 264
Working artefacts: ethnomethods of the prototype
Suchman, L., Trigg, R., & Blomberg, J. (2002) · The British Journal of Sociology · doi:10.1080/00071310220133287 · Cited by 250 · cited 2×
“Earlier work has raised comparable advice by proposing a “radical refiguring of the relations of design and use” and by recognising the extent to which design activities must continue after the system has been deployed and put into use [114].”
BackgroundYang, Q., Steinfeld, A., & Zimmerman, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300468 · Cited by 247
Tying Knots: Participatory Infrastructuring at Work
Bødker, S., Dindler, C., & Iversen, O. S. (2017) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-017-9268-y · Cited by 237 · cited 1×
“[18], collaboration and mutual learning during software development are too rich and nuanced to be captured by a single technique.”
BackgroundWang, D., Wang, L., Zhang, Z., Wang, D., Zhu, H., Gao, Y., Fan, X., & Tian, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445432 · Cited by 209 · cited 9×
“[116] evaluated systems that suggested treatments to described conditions.”
BackgroundMethodResultInfluentialSeven lessons for interdisciplinary research on interactive digital health interventions
Blandford, A., Gibbs, J., Newhouse, N., Perski, O., Singh, A., & Murray, E. (2018) · DIGITAL HEALTH · doi:10.1177/2055207618770325 · Cited by 209 · cited 6×
“Health research and evidence-based medicine are committed to sequential development processes and randomised control trials [15].”
BackgroundInfluentialIntegrating a Machine Learning System Into Clinical Workflows: Qualitative Study
Sandhu, S., Lin, A. L., Brajer, N., Sperling, J., Ratliff, W., Bedoya, A. D., Balu, S., O'Brien, C., & Sendak, M. P. (2020) · Journal of Medical Internet Research · doi:10.2196/22421 · Cited by 168
Ginestra, J. C., Giannini, H. M., Schweickert, W. D., Meadows, L., Lynch, M. J., Pavan, K., Chivers, C. J., Draugelis, M., Donnelly, P. J., Fuchs, B. D., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003803 · Cited by 159 · cited 8×
“Two of the three reviewed systems that predicted sepsis onset did not provide any additional information except the onset risk [46, 83].”
BackgroundMethodInfluentialHow AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study
Piorkowski, D., Park, S., Wang, A. Y., Wang, D., Muller, M., & Portnoy, F. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449205 · Cited by 151 · cited 1×
“Piorkowski et al. [97] highlight three communication challenges in interdisciplinary environments.”
BackgroundRomero-Brufau, S., Wyatt, K. D., Boyum, P., Mickelson, M., Moore, M., & Cognetta-Rieke, C. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.104072 · Cited by 143 · cited 11×
“In the reviewed articles, text data was accessed through public and private databases of sociodemographic data based on a patient’s postal code or Medicare claims (in the USA) [10, 101, 102].”
BackgroundMethodInfluentialBrennan, M., Puri, S., Ozrazgat-Baslanti, T., Feng, Z., Ruppert, M., Hashemighouchani, H., Momcilovic, P., Li, X., Wang, D. Z., & Bihorac, A. (2019) · Surgery · doi:10.1016/j.surg.2019.01.002 · Cited by 131 · cited 8×
“using a simulated worklow" [20], "comments .”
BackgroundMethodInfluentialBridging the “last mile” gap between AI implementation and operation: “data awareness” that matters
Cabitza, F., Campagner, A., & Balsano, C. (2020) · Annals of Translational Medicine · doi:10.21037/atm.2020.03.63 · Cited by 113
Yang, Q., Zimmerman, J., Steinfeld, A., Carey, L., & Antaki, J. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858373 · Cited by 105 · cited 5×
“Despite HCI’s longstanding contribution to Human-AI interaction, many researchers voice concerns that ML-based systems are uniquely difficult to design, develop, and deploy (see, e.g., Reference [124]) in comparison to traditional systems, especially in safety critical and complex settings like healthcare [6, 49, 87].”
MethodInfluentialBarda, A. J., Horvat, C. M., & Hochheiser, H. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01276-x · Cited by 105
The Last Mile: Where Artificial Intelligence Meets Reality
Coiera, E. (2019) · Journal of Medical Internet Research · doi:10.2196/16323 · Cited by 102
Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence
Yang, Q., Cranshaw, J., Amershi, S., Iqbal, S. T., & Teevan, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300415 · Cited by 90 · cited 1×
“Similar perspectives are raised in the respective communities [47, 67, 122] and it is recognised that success with ML-based systems requires an extra effort of the ongoing collaboration between Health, ML, and HCI team members [1, 4, 29, 73].”
BackgroundCarePre: An Intelligent Clinical Decision Assistance System
Jin, Z., Cui, S., Guo, S., Gotz, D., Sun, J., & Cao, N. (2020) · ACM Transactions on Computing for Healthcare · doi:10.1145/3344258 · Cited by 70
Cécile, P., Aude, M., Jean-Louis, D., & Catherine, R. (2020) · Studies in Health Technology and Informatics · doi:10.3233/shti200312 · Cited by 62 · cited 10×
“End-users: Physicians [24, 45, 64, 86], Nurses [106] Other stakeholders: Physicians, Other Medical Professionals, Other, IT specialists [106]”
BackgroundMethodResultInfluentialJauk, S., Kramer, D., Avian, A., Berghold, A., Leodolter, W., & Schulz, S. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01727-6 · Cited by 59
Lessons Learned from Designing an AI-Enabled Diagnosis Tool for Pathologists
Gu, H., Huang, J., Hung, L., & Chen, X. '. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449084 · Cited by 56 · cited 1×
“Studies of specific clinical decision-making processes propose several concrete recommendations that can help navigate the uncertainties of AI/ML design and real-world deployment [53, 61, 125].”
BackgroundThe Multiple Intersecting Sites of Design in CSCW Research
Bjørn, P., & Boulus-Rødje, N. (2015) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-015-9227-4 · Cited by 51 · cited 1×
“[15], who underscore the agency of HCI and CSCW researchers and practitioners in the design of collaborative space-time, in contrast to serving solely as providers of appropriate techniques.”
BackgroundMatthiesen, S., Diederichsen, S. Z., Hansen, M. K. H., Villumsen, C., Lassen, M. C. H., Jacobsen, P. K., Risum, N., Winkel, B. G., Philbert, B. T., Svendsen, J. H., et al. (2021) · JMIR Human Factors · doi:10.2196/26964 · Cited by 45 · cited 10×
“sketching the user interface” [82] Prototyping: “created .”
BackgroundMethodInfluentialSelf-citationBarriers to Implementing an Artificial Intelligence Model for Unplanned Readmissions
Baxter, S. L., Bass, J. S., & Sitapati, A. M. (2020) · ACI Open · doi:10.1055/s-0040-1716748 · Cited by 30 · cited 8×
“patient individual information [119], or social stressors like unemployment or loss of a family member [9, 103].”
BackgroundMethodInfluential
Cited by
Zhang, S., Yu, J., Xu, X., Yin, C., Lu, Y., Yao, B., Tory, M., Padilla, L. M., Caterino, J., Zhang, P., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642343 · Cited by 99
“In this paper, we further explore the challenges encountered by doctors in cooperation with AI-CDSS during the current early diagnosis of sepsis.”
BackgroundIronies of Generative AI: Understanding and Mitigating Productivity Loss in Human-AI Interaction
Simkute, A., Tankelevitch, L., Kewenig, V., Scott, A. E., Sellen, A., & Rintel, S. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2405782 · Cited by 89
“For example, clinicians were ineffective working with decision-support systems when explanatory information was incomprehensible, or if interfaces presented multiple risk scores simultaneously (Zaja˛c et al., 2023).”
Yildirim, N., Richardson, H., Wetscherek, M. T., Bajwa, J., Jacob, J., Pinnock, M. A., Harris, S., Coelho De Castro, D., Bannur, S., Hyland, S., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642013 · Cited by 82
Kim, D., Vegt, N., Visch, V., & Bos-De Vos, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642883 · Cited by 33
“For example, the work of [43] examined the interplays between different types of clinicians.”
Thieme, A., Rajamohan, A., Cooper, B., Groombridge, H., Simister, R., Wong, B., Woznitza, N., Pinnock, M. A., Wetscherek, M. T., Morrison, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.05299 · Cited by 10
Metadata
- Authors
- Zając, H. D., Li, D., Dai, X., Carlsen, J. F., Kensing, F., & Andersen, T. O.
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2023-03-17
- Volume
- 30
- Issue
- 2
- Pages
- 1-39
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4318616691
- Cited by
- 74
- References
- 52
- FWCI
- 2.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.