User-centered design of a machine learning dashboard for prediction of postoperative complications
Open accessAbstract
BACKGROUND: Machine learning models can help anesthesiology clinicians assess patients and make clinical and operational decisions, but well-designed human-computer interfaces are necessary for machine learning model predictions to result in clinician actions that help patients. Therefore, the goal of this study was to apply a user-centered design framework to create a user interface for displaying machine learning model predictions of postoperative complications to anesthesiology clinicians. METHODS: Twenty-five anesthesiology clinicians (attending anesthesiologists, resident physicians, and certified registered nurse anesthetists) participated in a 3-phase study that included (phase 1) semistructured focus group interviews and a card sorting activity to characterize user workflows and needs; (phase 2) simulated patient evaluation incorporating a low-fidelity static prototype display interface followed by a semistructured interview; and (phase 3) simulated patient evaluation with concurrent think-aloud incorporating a high-fidelity prototype display interface in the electronic health record. In each phase, data analysis included open coding of session transcripts and thematic analysis. RESULTS: During the needs assessment phase (phase 1), participants voiced that (a) identifying preventable risk related to modifiable risk factors is more important than nonpreventable risk, (b) comprehensive patient evaluation follows a systematic approach that relies heavily on the electronic health record, and (c) an easy-to-use display interface should have a simple layout that uses color and graphs to minimize time and energy spent reading it. When performing simulations using the low-fidelity prototype (phase 2), participants reported that (a) the machine learning predictions helped them to evaluate patient risk, (b) additional information about how to act on the risk estimate would be useful, and (c) correctable problems related to textual content existed. When performing simulations using the high-fidelity prototype (phase 3), usability problems predominantly related to the presentation of information and functionality. Despite the usability problems, participants rated the system highly on the System Usability Scale (mean score, 82.5; standard deviation, 10.5). CONCLUSIONS: Incorporating user needs and preferences into the design of a machine learning dashboard results in a display interface that clinicians rate as highly usable. Because the system demonstrates usability, evaluation of the effects of implementation on both process and clinical outcomes is warranted.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The theme perceived usefulness was identified in 13 papers.
“The theme perceived usefulness was identified in 13 papers, where participants reported the how the systems impacted their work and associated tasks [23,24,26,28–33,35–38].”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
Participants commonly reported that XAI tools support task completion.
“Across these studies participants commonly reported that the XAI tools support task completion [23,26,28,29,31,33].”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
Integration with electronic health records was common, particularly in more mature tools.
“Integration with electronic health records (EHRs) was common, particularly in more mature tools [23,24,26–34,36].”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
Perceptions regarding efficiency or saving time were reported in six studies.
“Perceptions regarding efficiency or saving time were reported in six studies [23,28–30,32,33].”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
Education and training was a theme that arose in 9 studies.
“Education and training was a theme that arose in 9 studies, where participants described their need for, or experience of, learning how to use the XAI system [23,25,26,28–30,32,35,36].”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
XAI model predictions were reportedly more likely to be accepted when they aligned with health professionals' knowledge or with the patient context.
“XAI model predictions were reportedly more likely to be accepted when they aligned with health professionals’ knowledge, or with the patient context [23,25,27–30,32–36]”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
Healthcare professionals preferred grouping features and ordering them by their contribution to predicted risk to allow easy and quick interpretation.
“For presentation of features, healthcare professionals preferred grouping features and ordering them by their contribution to predicted risk to allow easy and quick interpretation.”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
Eight studies reported participants desired actionable information from XAI systems.
“Eight studies in this review reported participants desired actionable information from the XAI systems [23–25,28,30,32,36,37].”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
In three studies, participants reported wanting to see variables that could be altered through interventions in feature-based explanations.
“With regard to feature based explanations, participants in three studies reported wanting to see the variables that could be altered through interventions [23,25,28,32].”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
Colour coding was viewed as a helpful visual cue for conveying risk severity.
“Colour coding was viewed as a helpful visual cue for conveying risk severity.”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
Health professionals' perceptions involving design and usability were found in 14 papers.
“Health professionals’ perceptions involving design and usability were found in 14 papers [23–34,37,38].”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
Five studies specifically described using SHAP.
“Five studies [23,25,27,28,34] specifically described using SHapley Additive exPlanations (SHAP)”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
Health professionals preferred risk predictions expressed as whole-number percentages.
“For numeric information, health professionals reported preferring risk predictions expressed as whole-number percentages”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
Decimals, negative values, and outputs that did not sum to 100% were viewed as confusing and potentially misleading.
“while decimals, negative values, and outputs that did not sum to 100% were viewed as confusing and potentially misleading.”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [28]
Cites
Standards for Reporting Qualitative Research: A Synthesis of Recommendations
O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014) · Academic Medicine · doi:10.1097/acm.0000000000000388 · Cited by 12,211 · see citing works
Explainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967 · see citing works
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · cited 1× · see citing works
“29 Clinician education, perhaps accompanied by textual reminders on the interface, is important to ensure ML models do not lead clinicians to inappropriate decisions (eg, not placing a clinically-indicated arterial line to “reduce” the predicted risk of a complication).”
BackgroundStandards for reporting qualitative research: a synthesis of recommendations
Kodama, Y., & Tomotaki, A. (2020) · 看護研究 · Cited by 845 · see citing works
Wijnberge, M., Geerts, B. F., Hol, L., Lemmers, N., Mulder, M. P., Berge, P., Schenk, J., Terwindt, L. E., Hollmann, M. W., Vlaar, A. P., et al. (2020) · JAMA · doi:10.1001/jama.2020.0592 · Cited by 586 · see citing works
Xue, B., Li, D., Lu, C., King, C. R., Wildes, T. S., Avidan, M. S., Kannampallil, T., & Abraham, J. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.2240 · Cited by 373 · see citing works
Self-citationNeural substrates of cognitive capacity limitations
Buschman, T. J., Siegel, M., Roy, J. E., & Miller, E. K. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1104666108 · Cited by 343 · see citing works
Diprose, W. K., Buist, N., Hua, N., Thurier, Q., Shand, G., & Robinson, R. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz229 · Cited by 188 · see citing works
Cognitive Processes in Anesthesiology Decision Making
Stiegler, M. P., & Tung, A. (2013) · Anesthesiology · doi:10.1097/aln.0000000000000073 · Cited by 175 · see citing works
Deep-learning model for predicting 30-day postoperative mortality
Fritz, B. A., Cui, Z., Zhang, M., He, Y., Chen, Y., Kronzer, A., Ben Abdallah, A., King, C. R., & Avidan, M. S. (2019) · British Journal of Anaesthesia · doi:10.1016/j.bja.2019.07.025 · Cited by 127 · see citing works
Self-citationBellini, V., Guzzon, M., Bigliardi, B., Mordonini, M., Filippelli, S., & Bignami, E. G. (2019) · Journal of Medical Systems · doi:10.1007/s10916-019-1512-1 · Cited by 118 · see citing works
Springer, A., & Whittaker, S. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302322 · Cited by 115 · cited 1× · see citing works
“We operationalized this concept, frequently called progressive disclosure, 30 by constructing a census view (Figure 2A) with the minimum necessary information on current risk.”
MethodBarda, A. J., Horvat, C. M., & Hochheiser, H. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01276-x · Cited by 105 · see citing works
Kang, A. R., Lee, J.-H., Jung, W., Lee, M., Park, S. Y., Woo, J., & Kim, S. H. (2020) · PLoS ONE · doi:10.1371/journal.pone.0231172 · Cited by 99 · see citing works
Schwartz, J. M., George, M., Rossetti, S. C., Dykes, P. C., Minshall, S. R., Lucas, E., & Cato, K. D. (2022) · JMIR Human Factors · doi:10.2196/33960 · Cited by 88 · cited 2× · see citing works
“24,26,27 Most participants wanted to use this information to drive intraoperative risk mitigation strategies.”
BackgroundMethodKheterpal, S., Shanks, A. M., & Tremper, K. K. (2018) · Anesthesiology · doi:10.1097/aln.0000000000002023 · Cited by 77 · see citing works
Ericsson, K. A. (2003) · Journal of Consciousness Studies · Cited by 76 · see citing works
Hofer, I. S., Burns, M. L., Kendale, S. M., & Wanderer, J. P. (2020) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000004575 · Cited by 70 · see citing works
Machine learning identifies ICU outcome predictors in a multicenter COVID-19 cohort
Magunia, H., Lederer, S., Verbuecheln, R., Gilot, B. J., Koeppen, M., Haeberle, H. A., Mirakaj, V., Hofmann, P., Marx, G., Bickenbach, J., et al. (2021) · Critical Care · doi:10.1186/s13054-021-03720-4 · Cited by 65 · see citing works
Chi, E. A., Chi, G., Tsui, C. T., Jiang, Y., Jarr, K., Kulkarni, C. V., Zhang, M., Long, J., Ng, A. Y., Rajpurkar, P., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.17391 · Cited by 64 · see citing works
Evaluating and reducing cognitive load should be a priority for machine learning in healthcare
Ehrmann, D. E., Gallant, S. N., Nagaraj, S., Goodfellow, S. D., Eytan, D., Goldenberg, A., & Mazwi, M. (2022) · Nature Medicine · doi:10.1038/s41591-022-01833-z · Cited by 64 · see citing works
The Usability Problem Taxonomy: A Framework for Classification and Analysis
Keenan, S. L., Hartson, H. R., Kafura, D. G., & Schulman, R. S. (1999) · Empirical Software Engineering · doi:10.1023/a:1009855231530 · Cited by 62 · see citing works
Abraham, J., Bartek, B., Meng, A., Ryan King, C., Xue, B., Lu, C., & Avidan, M. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104270 · Cited by 50 · see citing works
Self-citationWright, M. C., Borbolla, D. A., Waller, R. G., Fiol, G. D., Reese, T., Nesbitt, P., & Segall, N. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.yjbinx.2019.100041 · Cited by 48 · see citing works
Machine-Learning Implementation in Clinical Anesthesia: Opportunities and Challenges
Char, D. S., & Burgart, A. M. (2020) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000004656 · Cited by 45 · see citing works
Matthiesen, S., Diederichsen, S. Z., Hansen, M. K. H., Villumsen, C., Lassen, M. C. H., Jacobsen, P. K., Risum, N., Winkel, B. G., Philbert, B. T., Svendsen, J. H., et al. (2021) · JMIR Human Factors · doi:10.2196/26964 · Cited by 45 · see citing works
Artificial Intelligence and Machine Learning in Patient Blood Management: A Scoping Review
Meier, J. M., & Tschoellitsch, T. (2022) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000006047 · Cited by 41 · see citing works
Intraoperative prediction of postanaesthesia care unit hypotension
Palla, K., Hyland, S. L., Posner, K. L., Ghosh, P., Nair, B. G., Bristow, M., Paleva, Y., Williams, B., Fong, C. T., Cleve, W. V., et al. (2021) · British Journal of Anaesthesia · doi:10.1016/j.bja.2021.10.052 · Cited by 39 · see citing works
Helman, S., Terry, M. A., Pellathy, T., Williams, A., Dubrawski, A., Clermont, G., Pinsky, M. R., Al-Zaiti, S., & Hravnak, M. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104643 · Cited by 31 · see citing works
Gregory, S. H., Murray-Torres, T. M., Fritz, B. A., Abdallah, A. B., Helsten, D. L., Wildes, T. S., Sharma, A., Avidan, M. S., & ACTFAST Study Group (2018) · F1000Research · doi:10.12688/f1000research.14897.1 · Cited by 17 · see citing works
Self-citationKing, C. R., Abraham, J., Kannampallil, T., Fritz, B. A., Abdallah, A. B., Chen, Y., Henrichs, B., Politi, M. C., Torres, B. A., Mickle, A. M., et al. (2019) · F1000Research · doi:10.12688/f1000research.21016.1 · Cited by 17 · see citing works
Self-citationA Factored Generalized Additive Model for Clinical Decision Support in the Operating Room
Cui, Z., Fritz, B. A., King, C. R., Avidan, M. S., & Chen, Y. (2019) · PubMed Central · doi:10.48550/arxiv.1907.12596 · Cited by 11 · see citing works
Self-citationChi, E. A., Chi, G., Tsui, C. T., Jiang, Y. C., Jarr, K. J., Kulkarni, C. V., Zhang, M., Long, J., Ng, A. Y., Rajpurkar, P., et al. (2021) · Gastroenterology · doi:10.1016/s0016-5085(21)00897-0 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Fritz, B. A., King, C. R., Abdelhack, M., Chen, Y., Kronzer, A., Abraham, J., Tripathi, S., Abdallah, A. B., Kannampallil, T., Budelier, T. P., et al. (2024) · British Journal of Anaesthesia · doi:10.1016/j.bja.2024.08.004 · Cited by 33 · see citing works
Self-citationParsons, C. S., Zuiderwijk, A., Orchard, N. A., Oosterhoff, J. H. F., & de Reuver, M. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03237-8 · Cited by 32 · see citing works
Van Dort, B. A., Engelsma, T., Sneekes, P., Peute, L., & Medlock, S. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001417 · Cited by 9 · see citing works
“[28] Use tables and/or bar charts for visualization of risk [23,31] “Tables, grids or graphs, in general, are better [than paragraphs].””
InfluentialNoselli, M., van der Vegt, A. H., Cordeil, M., Campbell, V. K., Wang, A. P., Shetty, A., & Scott, I. A. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3572494 · Cited by 6 · see citing works
King, C. R., Fritz, B. A., Gregory, S. H., Budelier, T. P., Abdallah, A. B., Kronzer, A., Helsten, D. L., Torres, B. A., McKinnon, S., Tripathi, S., et al. (2025) · British Journal of Anaesthesia · doi:10.1016/j.bja.2024.11.017 · Cited by 6 · see citing works
Self-citationSharma, A. C., Azeem, A., Omari, I. H., & Premkumar, A. (2025) · Cureus · doi:10.7759/cureus.100254 · Cited by 6 · see citing works
Wood, M. D., West, N., Fokkens, C., Chen, Y., Loftsgard, K. C., Cardinal, K., Whyte, S. D., Portales-Casamar, É., & Görges, M. (2023) · JMIR Pediatrics and Parenting · doi:10.2196/46785 · Cited by 6 · see citing works
Gonzalez, X. T., Steger-May, K., & Abraham, J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae257 · Cited by 4 · see citing works
Self-citationTeam Roles of Artificial Intelligence in Anesthesiology – A Scoping Review
Huber, S. L., Weppert, L., Baumeister, L., Happel, O., & Grundgeiger, T. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720186 · Cited by 3 · see citing works
The Promise of Artificial Intelligence in Neuroanesthesia: An Update
Liao, Z., Mathur, N., Joshi, V., & Joshi, S. (2024) · Journal of Neuroanaesthesiology and Critical Care · doi:10.1055/s-0044-1787844 · Cited by 1 · see citing works
Kobekyaa, F. K., Boswell, D., Sinclair, R., Havaei, F., Risling, T., & Haase, K. R. (2025) · medRxiv · doi:10.1101/2025.09.15.25335735 · Cited by 1 · see citing works
Machine learning models in anaesthesiology: bridging the gap from model training to implementation
King, C. R., & Fritz, B. A. (2026) · British Journal of Anaesthesia · doi:10.1016/j.bja.2026.03.022 · Cited by 1 · see citing works
Self-citationVisualising Machine Learning Model Outputs in Data Analytics: A Systematic Review
Marshall, S., Neri, G., Yaghi, A., Chan, H. K.-H., Tabor, D., Sinha, R., & Suvodeep Mazumdar, A. (2026) · Analytics · doi:10.3390/analytics5030024
Alwakeel, M. I., Akbar, M., Bedoya, A., Raghunathan, K., Wong, A. I., Kent, M. L., Mehdiratta, N. L., Gilstrap, D. L., & Krishnamoorthy, V. K. (2026) · A&A Practice · doi:10.1213/xaa.0000000000002252
Chen, C., 徐伟松, Wu, T., Zhao, J., Xu, X., Xiao, M., & Xu, K. (2026) · Cancer Reports · doi:10.1002/cnr2.70665
- Authors
- Fritz, B. A., Pugazenthi, S., Budelier, T. P., Tellor Pennington, B. R., King, C. R., Avidan, M. S., & Abraham, J.
- Affiliations
- Department of Anesthesiology
- Washington University in St. Louis (US)
- Corresponding author
- Fritz, Bradley A.
- Pugazenthi, Sangami
- Budelier, Thaddeus P.
- King, Christopher R.
- Avidan, Michael S.
- Abraham, Joanna
- Venue
- Anesthesia & Analgesia
- Published
- 2023-06-20
- Volume
- 138
- Issue
- 4
- Pages
- 804-813
- ISSN
- 0003-2999 (ISSN-L)
- PMC
- PMC10730770
- OpenAlex
- W4381332948
- Cited by
- 15
- References
- 33
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 3 · 2023: 1
- FWCI
- 3.37
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Anesthesia & Analgesia (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- Funders
- National Institutes of Health (KL2TR002346)
- National Institute of Nursing Research (R01NR017916)
- National Institute of General Medical Sciences (T32GM108539)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- user-centered design, postoperative complications, human-machine interaction, usability, electronic health records, clinical decision support, risk prediction, think-aloud, card sorting, System Usability Scale
- MeSH terms
- User-Centered Design, Humans, Postoperative Complications, User-Computer Interface, Focus Groups, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
