Co-design of Human-centered, Explainable AI for Clinical Decision Support
Open accessThe DOI matches its Crossref record; 142 of the 142 references it cites were found in a registry or an index.
Abstract
eXplainable AI (XAI) involves two intertwined but separate challenges: the development of techniques to extract explanations from black-box AI models and the way such explanations are presented to users, i.e., the explanation user interface. Despite its importance, the second aspect has received limited attention so far in the literature. Effective AI explanation interfaces are fundamental for allowing human decision-makers to take advantage and oversee high-risk AI systems effectively. Following an iterative design approach, we present the first cycle of prototyping-testing-redesigning of an explainable AI technique and its explanation user interface for clinical Decision Support Systems (DSS). We first present an XAI technique that meets the technical requirements of the healthcare domain: sequential, ontology-linked patient data, and multi-label classification tasks. We demonstrate its applicability to explain a clinical DSS, and we design a first prototype of an explanation user interface. Next, we test such a prototype with healthcare providers and collect their feedback with a two-fold outcome: First, we obtain evidence that explanations increase users’ trust in the XAI system, and second, we obtain useful insights on the perceived deficiencies of their interaction with the system, so we can re-design a better, more human-centered explanation interface.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Echoing the call for more user-centered processes, Panigutti et al. [56] explicitly advocate for co-design and human-centered development, highlighting the importance of involving clinicians early and iteratively to avoid building systems misaligned with clinical needs.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [56]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
User Acceptance of Information Technology: Toward A Unified View1
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003) · MIS Quarterly · doi:10.2307/30036540 · Cited by 45,035 · cited 6×
“In particular, we collected the following constructs (5-point Likert scale, from 1 = “strongly disagree” to 5 = “strongly agree”): • Performance Expectancy: the degree to which an individual believes that using the system will help him or her to attain gains in job performance [133].”
BackgroundMethodInfluential"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
A translation approach to portable ontology specifications
Gruber, T. R. (1993) · Knowledge Acquisition · doi:10.1006/knac.1993.1008 · Cited by 12,752 · cited 1×
“Ontologies (also known as knowledge graphs) are structured, machine-readable representations of the knowledge pertaining to a specific aspect of a domain [56, 125].”
BackgroundTechnology Acceptance Model 3 and a Research Agenda on Interventions
Venkatesh, V., & Bala, H. (2008) · Decision Sciences · doi:10.1111/j.1540-5915.2008.00192.x · Cited by 8,159 · cited 5×
“• Result Demonstrability: The degree to which an individual believes that the results of using a system are tangible, observable, and communicable [132].”
BackgroundMethodInfluentialMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 1×
“The MIMIC (Medical Information Mart for Intensive Care) [55, 71] database is a single-center, publicly accessible database that contains de-identified clinical data from patients admitted to the Beth Israel Deaconess Medical Center’s ICU (intensive care unit) .”
Method“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 2×
“Indeed, the design of the AI interface can inluence users’ beliefs about its capabilities and trust in its decisions [79].”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Proceedings of the 21st International Conference on Neural Information Processing Systems
Koller, D., Schuurmans, D., Bengio, Y., & Bottou, L. (2008) · Cited by 4,461
The unified medical language system (UMLS): integrating biomedical terminology
Bodenreider, O. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh061 · Cited by 4,356 · cited 2×
“Some notables examples are the Disease Ontology (DO) [112], the Open Biomedical Ontology (OBO) [119], the Diabetes mellitus Diagnosis Ontology (DDO) [48], the Systematised Nomenclature of Medicine Clinical Terms (SNOMED-CT) [42], and the Unified Medical Language System (UMLS) [18].”
BackgroundKnowledge engineering: Principles and methods
Studer, R., Benjamins, V. R., & Fensel, D. (1998) · Data & Knowledge Engineering · doi:10.1016/s0169-023x(97)00056-6 · Cited by 3,686 · cited 1×
“Ontologies (also known as knowledge graphs) are structured, machine-readable representations of the knowledge pertaining to a specific aspect of a domain [56, 125].”
BackgroundHeuristic evaluation of user interfaces
Nielsen, J., & Molich, R. (1990) · Proceedings of the SIGCHI conference on Human factors in computing systems Empowering people - CHI '90 · doi:10.1145/97243.97281 · Cited by 3,533 · cited 1×
“We conducted an assessment of the two explanation interfaces through a heuristic evaluation [98], a usability testing method where a group of evaluators judges an interface using a set of usability guidelines.”
MethodA Review on Multi-Label Learning Algorithms
Zhang, M.-L., & Zhou, Z.-H. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2013.39 · Cited by 3,321 · cited 1×
“In the case of multi-label classification, the fidelity is calculated using the F 1 measure with micro-averaging [143], while in the binary case the fidelity is calculated using the accuracy score.”
MethodTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 2×
“This measure is important since we do not want to approximate the blackbox with a model that loses its interpretability because of the high-dimensionality of the explanations it produces [43, 84].”
BackgroundAn overview of clinical decision support systems: Benefits, risks, and strategies for success
Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0221-y · Cited by 3,019 · cited 1×
“One approach for integrating AI into clinical practice is using it in clinical Decision Support Systems (DSS) , i.e., computerized systems that give evidence-based suggestions, alerts, and reminders to healthcare practitioners to help with patient diagnosis, treat-ment, and management [126].”
BackgroundThe Efficient Assessment of Need for Cognition
Cacioppo, J. T., Petty, R. E., & Feng Kao, C. (1984) · Journal of Personality Assessment · doi:10.1207/s15327752jpa4803_13 · Cited by 2,913 · cited 1×
“We also controlled for participants’ Need For Cognition (NFC) —an aspect related to the individual tendency to enjoy effortful cognitive tasks (5-point Likert scale, from 1 = “strongly disagree” to 5 = “strongly agree”) [24, 90].”
BackgroundTrust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817
The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration
The OBI Consortium, Smith, B., Ashburner, M., Rosse, C., Bard, J., Bug, W., Ceusters, W., Goldberg, L. J., Eilbeck, K., Ireland, A., et al. (2007) · Nature Biotechnology · doi:10.1038/nbt1346 · Cited by 2,560 · cited 2×
“Some notables examples are the Disease Ontology (DO) [112], the Open Biomedical Ontology (OBO) [119], the Diabetes mellitus Diagnosis Ontology (DDO) [48], the Systematised Nomenclature of Medicine Clinical Terms (SNOMED-CT) [42], and the Unified Medical Language System (UMLS) [18].”
Background(1961) · Journal of the Optical Society of America · doi:10.1364/josa.51.000248 · Cited by 2,288
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · cited 1×
“This user study also provides insights related to explanations relevancy, i.e., the explanation ability to provide insights for a particular audience into a chosen domain problem [96].”
BackgroundAlgorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
Mining electronic health records: towards better research applications and clinical care
Jensen, P. B., Jensen, L. J., & Brunak, S. (2012) · Nature Reviews Genetics · doi:10.1038/nrg3208 · Cited by 1,680 · cited 1×
“The wealth of information contained in such data has the potential to provide valuable insights into an individual’s health, potentially in real-time, and could be used to improve healthcare delivery [5, 9, 67, 75, 115].”
BackgroundAcademy of Management Proceedings
Epitropaki, O., Kark, R., Day, D., Lord, R. G., Murphy, S. E., & Stevens, N. (2016) · The Academy of Management · Cited by 1,609
Use of Electronic Health Records in U.S. Hospitals
Jha, A. K., DesRoches, C. M., Campbell, E. G., Donelan, K., Rao, S. R., Ferris, T. G., Shields, A., Rosenbaum, S., & Blumenthal, D. (2009) · New England Journal of Medicine · doi:10.1056/nejmsa0900592 · Cited by 1,473
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
(2021) · doi:10.1145/3411764 · Cited by 1,240
Multilabel Neural Networks with Applications to Functional Genomics and Text Categorization
Zhang, M.-L., & Zhou, Z.-H. (2006) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2006.162 · Cited by 1,187
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
The ethics of AI in health care: A mapping review
Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113172 · Cited by 1,105 · cited 1×
“Being able to explain clinical decisions to patients and be held accountable for adverse outcomes of their diagnosis are key ethical responsibilities of every doctor [93, 101].”
Background(2019) · Cited by 986
Disease Ontology: a backbone for disease semantic integration
Schriml, L. M., Arze, C., Nadendla, S., Chang, Y.-W. W., Mazaitis, M., Felix, V., Feng, G., & Kibbe, W. A. (2012) · Nucleic Acids Research · doi:10.1093/nar/gkr972 · Cited by 929 · cited 1×
“Some notables examples are the Disease Ontology (DO) [112], the Open Biomedical Ontology (OBO) [119], the Diabetes mellitus Diagnosis Ontology (DDO) [48], the Systematised Nomenclature of Medicine Clinical Terms (SNOMEDCT) [42] and the Uniied Medical Language System (UMLS) [18].”
BackgroundQuestioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925 · cited 1×
“A few works have tried to close such a gap in the medical field by involving the doctors in the design procedure [77, 113, 139] or by performing exploratory surveys [26, 83, 127].”
BackgroundZhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 1×
“The user interface conveys the transparency of the AI system [73] and, with the advent of XAI, the system explains its recommendations through the interface [10].”
MethodDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754 · cited 1×
“The user interface conveys the transparency of the AI system [73] and, with the advent of XAI, the system explains its recommendations through the interface [10].”
MethodKnowledge Discovery in Multi-label Phenotype Data
Clare, A., & King, R. D. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44794-6_4 · Cited by 726 · cited 1×
“…to simultaneously predict the risk of several chronic diseases [50, 51, 82, 144], when trying to classify unknown genes functional expressions [12, 35], when building a clinical algorithm to predict the diagnoses and medications order of patient’s future visit [31, 109], when trying to learn…”
BackgroundDoes automation bias decision-making?
SKITKA, L. J., MOSIER, K. L., & BURDICK, M. (1999) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0252 · Cited by 694 · cited 1×
“This phenomenon is known as automation bias [79, 118] and can be particularly dangerous in critical domains such as medicine.”
BackgroundInterpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677 · cited 1×
“These two methods are model-agnostic and application-agnostic , meaning that they are able to extract an explanation from any type of black-box AI model [91] regardless of the application domain.”
MethodAdoption and use of AI tools: a research agenda grounded in UTAUT
Venkatesh, V. (2022) · Annals of Operations Research · doi:10.1007/s10479-020-03918-9 · Cited by 676 · cited 1×
“In addition to these adoption barriers, AI-based clinical DSS also face trust issues from medical staff and a lack of knowledge about the assumptions, limitations, and capabilities of such systems [62, 131].”
BackgroundGRAM: Graph-based Attention Model for Healthcare Representation Learning
Choi, E., Bahadori, M. T., Song, L., Stewart, W. F., & Sun, J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098126 · Cited by 637
Explainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633 · cited 1×
“A recent survey has argued that explanations of black-box AI models are mainly used by machine learning engineers to debug their model in the development phase [15].”
BackgroundAntoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629 · cited 1×
“Despite these recent efforts, most of the research has been focused on laypeople [7, 29, 95].”
BackgroundA Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 1×
“This topic has recently witnessed an increased interest that generated vast literature on AI trans-parency and explainability [19, 57 ].”
BackgroundSelf-citationHow Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615
Extracting Tree-Structured Representations of Trained Networks
Craven, M. W., & SHAVLIK, J. W. (1995) · neural information processing systems · Cited by 602
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 2×
“On the other hand, physicians are reluctant to trust algorithms that they do not understand [26, 116] and might be subject to algorithm aversion [40], which is the human tendency to discount algorithmic advice [85].”
BackgroundTaking Advice: Accepting Help, Improving Judgment, and Sharing Responsibility
Harvey, N., & Fischer, I. (1997) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1997.2697 · Cited by 588
Expanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538 · cited 1×
“, healthcare providers, and observing the use of explanation in the appropriate decisional context [16, 21, 45, 88], i.”
BackgroundJohnson, A., Bulgarelli, L., Pollard, T., Gow, B., Moody, B., Horng, S., Celi, L. A., & Mark, R. (2024) · PhysioNet · doi:10.13026/07hj-2a80 · Cited by 524
Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L. A., & Mark, R. (2023) · PhysioNet · doi:10.13026/s6n6-xd98 · Cited by 524
Peake, J. M., Kerr, G., & Sullivan, J. P. (2018) · Frontiers in Physiology · doi:10.3389/fphys.2018.00743 · Cited by 523 · cited 1×
“Healthcare data can be collected from various sources, including Electronic Health Records (EHRs) maintained by healthcare providers [68], health and wellness apps [61, 105], and wearable sensors [27, 104].”
BackgroundMetrics for explainable AI: challenges and prospects
Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.04608 · Cited by 518 · cited 2×
“In addition to the WOA, we measured the explicit trust in the system by directly asking participants’ perception on the system reliability, predictability, and efficiency (5-point Likert scale, from 1 = “strongly disagree” to 5 = “strongly agree”) [3, 25, 64].”
BackgroundMethodHierarchical multi-label prediction of gene function
Barutcuoglu, Z., Schapire, R. E., & Troyanskaya, O. G. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btk048 · Cited by 490
Cappon, G., Vettoretti, M., Sparacino, G., & Facchinetti, A. (2019) · Diabetes & Metabolism Journal · doi:10.4093/dmj.2019.0121 · Cited by 477 · cited 1×
“Healthcare data can be collected from various sources, including Electronic Health Records (EHRs) maintained by healthcare providers [68], health and wellness apps [61, 105], and wearable sensors [27, 104].”
BackgroundTrust, Confidence, and Expertise in a Judge-Advisor System
Sniezek, J. A., & Van Swol, L. M. (2001) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.2000.2926 · Cited by 469 · cited 2×
“Our paradigm adapts to the Judge-Advisor System (JAS) [120, 121].”
BackgroundMethodThe Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441 · cited 17×
“In our use case the clinical DSS is the advisor and the clinician is the judge, solely responsible to provide appropriate care for the patient.”
BackgroundInfluentialCueing and Cognitive Conflict in Judge-Advisor Decision Making
Sniezek, J. A., & Buckley, T. (1995) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1995.1040 · Cited by 427
Role of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review
Yin, J., Ngiam, K. Y., & Teo, H. H. (2021) · Journal of Medical Internet Research · doi:10.2196/25759 · Cited by 419
Benchmarking deep learning models on large healthcare datasets
Purushotham, S., Meng, C., Che, Z., & Liu, Y. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.04.007 · Cited by 406 · cited 1×
“The MIMIC database is often used by the Machine Learning research community to train benchmark AI models on freely accessible data to ensure the reproducibility of results [107].”
MethodKruse, C. S., Stein, A., Thomas, H., & Kaur, H. (2018) · Journal of Medical Systems · doi:10.1007/s10916-018-1075-6 · Cited by 403 · cited 1×
“The wealth of information contained in such data has the potential to provide valuable insights into an individual’s health, potentially in real-time, and could be used to improve healthcare delivery [5, 9, 67, 75, 115].”
BackgroundWearable Biosensors: An Alternative and Practical Approach in Healthcare and Disease Monitoring
Sharma, A., Badea, M., Tiwari, S., & Marty, J. L. (2021) · Molecules · doi:10.3390/molecules26030748 · Cited by 357 · cited 1×
“The wealth of information contained in such data has the potential to provide valuable insights into an individual’s health, potentially in real-time, and could be used to improve healthcare delivery [5, 9, 67, 75, 115].”
BackgroundSmartphone Applications for Patients' Health and Fitness
Higgins, J. P. (2016) · The American Journal of Medicine · doi:10.1016/j.amjmed.2015.05.038 · Cited by 347 · cited 1×
“Healthcare data can be collected from various sources, including Electronic Health Records (EHRs) maintained by healthcare providers [68], health and wellness apps [61, 105], and wearable sensors [27, 104].”
BackgroundFan, W., Liu, J., Zhu, S., & Pardalos, P. M. (2020) · Annals of Operations Research · doi:10.1007/s10479-018-2818-y · Cited by 346 · cited 1×
“Trust plays a central role in the adoption of new technologies, and explanations of AI recommendations are often touted as the solution to trust issues [49, 108, 134, 139].”
BackgroundReasons For Physicians Not Adopting Clinical Decision Support Systems: Critical Analysis
Khairat, S., Marc, D., Crosby, W., & Al Sanousi, A. (2018) · JMIR Medical Informatics · doi:10.2196/medinform.8912 · Cited by 335 · cited 1×
“The reasons behind the low adoption of clinical DSS that do not embed AI have been well studied, with shortcomings including perceived challenges to autonomy, lack of time, and dissatisfaction with user interfaces [17, 72, 74, 78, 94, 110, 129, 130].”
BackgroundStrohm, L., Hehakaya, C., Ranschaert, E. R., Boon, W. P. C., & Moors, E. H. M. (2020) · European Radiology · doi:10.1007/s00330-020-06946-y · Cited by 329 · cited 1×
“Distrust in AI applications in medicine also comes from doctors’ fear of legal repercussions if something goes wrong due to unclear liability regimes [97, 124].”
BackgroundWang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307 · cited 1×
“…involving a specific pool of end-users, i.e., healthcare providers, and observing the use of explanation in the appropriate decisional context [16, 21, 45, 88], i.e., while performing a task supported by a clinical DSS. Ideally, explaining clinical DSS recommendations should help clinicians with…”
BackgroundComputerized clinical decision support for prescribing: provision does not guarantee uptake
Moxey, A., Robertson, J., Newby, D., Hains, I., Williamson, M., & Pearson, S.-A. (2010) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3170 · Cited by 305 · cited 1×
“The reasons behind the low adoption of clinical DSS that do not embed AI have been well studied, with shortcomings including perceived challenges to autonomy, lack of time, and dissatisfaction with user interfaces [17, 72, 74, 78, 94, 110, 129, 130].”
BackgroundExplaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · cited 1×
“Despite these recent eforts, most of the research has been focused on laypeople [7, 29, 95].”
BackgroundExplainable artificial intelligence for breast cancer: A visual case-based reasoning approach
Lamy, J.-B., Sekar, B., Guezennec, G., Bouaud, J., & Séroussi, B. (2019) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2019.01.001 · Cited by 282
Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach
Ehsan, U., & Riedl, M. O. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-60117-1_33 · Cited by 274
Dietvorst, B. J., & Bharti, S. (2020) · Psychological Science · doi:10.1177/0956797620948841 · Cited by 273 · cited 2×
“On the other hand, physicians are reluctant to trust algorithms that they do not understand [26, 116] and might be subject to algorithm aversion [40], which is the human tendency to discount algorithmic advice [85].”
BackgroundMalgieri, G., & Comandé, G. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx019 · Cited by 271 · cited 1×
“…literature has also debated the existence of a “right to have an explanation” of AI decisions based on the EU GDPR (General Data Protection Regulation) prescription to provide “meaningful information about the logic involved” to the data subject in case of automated decision-making [1, 36, 58, 87].”
BackgroundOne Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Arya, V., Bellamy, R. K. E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., et al. (2019) · arXiv · doi:10.48550/arxiv.1909.03012 · Cited by 268 · cited 1×
“While the model-agnostic approach to XAI offers great flexibility to the use of these methods, the application-agnostic approach implies that the specific user needs are not considered [8].”
MethodBenchmarking and survey of explanation methods for black box models
Bodria, F., Giannotti, F., Guidotti, R., Naretto, F., Pedreschi, D., & Rinzivillo, S. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00933-9 · Cited by 264 · cited 1×
“This topic has recently witnessed an increased interest that generated vast literature on AI transparency and explainability [19, 57].”
BackgroundSelf-citationJacobs, M., Pradier, M. F., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01224-x · Cited by 247 · cited 1×
“While, at first glance, explanations of such DSS seem the solution to these issues, some studies suggested that explanations can be inadequate to deal with overreliance on flawed algorithms [66].”
Background"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
Lakkaraju, H., & Bastani, O. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375833 · Cited by 241 · cited 1×
“Furthermore, explanations might even increase overreliance on AI-based clinical DSS [23, 47, 76], and it might be necessary to design the system to force the user to engage in analytical thinking when explanations require substantial cognitive Co-design of Human-centered, Explainable AI for…”
BackgroundArtificial intelligence: Who is responsible for the diagnosis?
Neri, E., Coppola, F., Miele, V., Bibbolino, C., & Grassi, R. (2020) · La radiologia medica · doi:10.1007/s11547-020-01135-9 · Cited by 240 · cited 1×
“Distrust in AI applications in medicine also comes from doctors’ fear of legal repercussions if something goes wrong due to unclear liability regimes [97, 124].”
BackgroundHuman-XAI Interaction: A Review and Design Principles for Explanation User Interfaces
Chromik, M., & Butz, A. M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85616-8_36 · Cited by 207 · cited 3×
“When considering user involvement in the design process of an AI explanation, it is important to disentangle two aspects of XAI development: the technical development of a technique able to extract an explanation from the black-box AI model (XAI), and the way such explanation is presented to users, i.e., the explanati…”
BackgroundBS-Net: Learning COVID-19 pneumonia severity on a large chest X-ray dataset
Signoroni, A., Savardi, M., Benini, S., Adami, N., Leonardi, R., Gibellini, P., Vaccher, F., Ravanelli, M., Borghesi, A., Maroldi, R., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102046 · Cited by 185
Doctor XAI: an ontology-based approach to black-box sequential data classification explanations
Panigutti, C., Perotti, A., & Pedreschi, D. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372855 · Cited by 164
Self-citationBarnett, A. J., Schwartz, F. R., Tao, C., Chen, C., Ren, Y., Lo, J. Y., & Rudin, C. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00423-x · Cited by 156 · cited 1×
“Our XAI methodology it in an emerging line of research focusing on XAI techniques that are not completely agnostic and tailor their explanations to the medical ield, either by incorporating medical knowledge in the explanation process [11, 32, 141] or focusing on speciic healthcare data characteristics and use cases […”
Method2020 ACR Data Science Institute Artificial Intelligence Survey
Allen, B., Agarwal, S., Coombs, L., Wald, C., & Dreyer, K. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.04.002 · Cited by 154 · cited 1×
“However, the rate of adoption of AI in health clinics and hospitals is low [4, 114, 140].”
BackgroundThe Impact of Placebic Explanations on Trust in Intelligent Systems
Eiband, M., Buschek, D., Kremer, A., & Hussmann, H. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3312787 · Cited by 152 · cited 2×
“Furthermore, explanations might even increase overreliance on AI-based clinical DSS [23, 47, 76], and it might be necessary to design the system to force the user to engage in analytical thinking when explanations require substantial cognitive Co-design of Human-centered, Explainable AI for…”
BackgroundShinners, L., Aggar, C., Grace, S., & Smith, S. (2020) · Health Informatics Journal · doi:10.1177/1460458219874641 · Cited by 148 · cited 1×
“On the other hand, physicians are reluctant to trust algorithms that they do not understand [26, 116] and might be subject to algorithm aversion [40], which is the human tendency to discount algorithmic advice [85].”
BackgroundMillecamp, M., Htun, N. N., Conati, C., & Verbert, K. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302313 · Cited by 148
ICD-9-CM to ICD-10-CM Codes: What? Why? How?
Cartwright, D. J. (2013) · Advances in Wound Care · doi:10.1089/wound.2013.0478 · Cited by 140
Varonen, H., Kortteisto, T., Kaila, M., & for the EBMeDS Study Group (2008) · Family Practice · doi:10.1093/fampra/cmn020 · Cited by 139 · cited 1×
“The reasons behind the low adoption of clinical DSS that do not embed AI have been well studied, with shortcomings including perceived challenges to autonomy, lack of time, and dissatisfaction with user interfaces [17, 72, 74, 78, 94, 110, 129, 130].”
BackgroundDoes projection into use improve trust and exploration? An example with a cruise control system
Cahour, B., & Forzy, J.-F. (2009) · Safety Science · doi:10.1016/j.ssci.2009.03.015 · Cited by 134 · cited 1×
“In addition to the WOA, we also measured the explicit trust in the system by directly asking participants’ perception on the system reliability, predictability, and eiciency (5-point Likert scale, from 1="strongly disagree" to 5="strongly agree") [3, 25, 64].”
MethodA digital biomarker of diabetes from smartphone-based vascular signals
Avram, R., Olgin, J. E., Kuhar, P., Hughes, J. W., Marcus, G. M., Pletcher, M. J., Aschbacher, K., & Tison, G. H. (2020) · Nature Medicine · doi:10.1038/s41591-020-1010-5 · Cited by 133
Multi-label transfer learning for the early diagnosis of breast cancer
Chougrad, H., Zouaki, H., & Alheyane, O. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.01.112 · Cited by 130
Explaining by Removing: A Unified Framework for Model Explanation
Covert, I., Lundberg, S., & Lee, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.14878 · Cited by 116
Nourani, M., King, J., & Ragan, E. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7469 · Cited by 116
Springer, A., & Whittaker, S. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302322 · Cited by 115
Kohli, A., & Jha, S. (2018) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2017.12.029 · Cited by 114 · cited 1×
“The reasons behind the low adoption of clinical DSS that do not embed AI have been well studied, with shortcomings including perceived challenges to autonomy, lack of time, and dissatisfaction with user interfaces [17, 72, 74, 78, 94, 110, 129, 130].”
BackgroundDevelopment and Implementation of Computerized Clinical Guidelines: Barriers and Solutions
Trivedi, M. H., Kern, J. K., Marcee, A., Grannemann, B., Kleiber, B., Bettinger, T., Altshuler, K. Z., & McClelland, A. (2002) · Methods of Information in Medicine · doi:10.1055/s-0038-1634216 · Cited by 109 · cited 1×
“The reasons behind the low adoption of clinical DSS that do not embed AI have been well studied, with shortcomings including perceived challenges to autonomy, lack of time, and dissatisfaction with user interfaces [17, 72, 74, 78, 94, 110, 129, 130].”
BackgroundDong, H., Suárez-Paniagua, V., Whiteley, W. N., & Wu, H. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103728 · Cited by 109
Explainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers
Ghai, B., Liao, Q. V., Zhang, Y., Bellamy, R., & Mueller, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432934 · Cited by 106
Schwartz, J., Moy, A. J., Rossetti, S. C., Elhadad, N., & Cato, K. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa296 · Cited by 93 · cited 1×
“A few works have tried to close such a gap in the medical field by involving the doctors in the design procedure [77, 113, 139] or by performing exploratory surveys [26, 83, 127].”
BackgroundFairLens: Auditing black-box clinical decision support systems
Panigutti, C., Perotti, A., Panisson, A., Bajardi, P., & Pedreschi, D. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102657 · Cited by 86
Self-citationWhat Do People Really Want When They Say They Want "Explainable AI?" We Asked 60 Stakeholders.
Brennen, A. (2020) · Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3334480.3383047 · Cited by 76 · cited 1×
“Nevertheless, debugging the model is only one of the needs expressed in another recent study that analyzes the demands of transparency of several stakeholders [20].”
BackgroundLevy, A., Agrawal, M., Satyanarayan, A., & Sontag, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445522 · Cited by 73 · cited 1×
“On the one hand, some works have shown that clinicians tend to over-rely on automated suggestions by taking less initiative [81] or accepting incorrect diagnoses suggested by AI [59].”
BackgroundMetrics, Explainability and the European AI Act Proposal
Sovrano, F., Sapienza, S., Palmirani, M., & Vitali, F. (2022) · J · doi:10.3390/j5010010 · Cited by 71 · cited 1×
“Furthermore, XAI techniques could help achieve the highest levels of AI transparency for high-risk AI applications (such as healthcare) as mandated by the AI Act, a recent EU regulation proposal on AI [2, 122].”
BackgroundSambasivan, M., Esmaeilzadeh, P., Kumar, N., & Nezakati, H. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-142 · Cited by 70 · cited 1×
“The reasons behind the low adoption of clinical DSS that do not embed AI have been well studied, with shortcomings including perceived challenges to autonomy, lack of time, and dissatisfaction with user interfaces [17, 72, 74, 78, 94, 110, 129, 130].”
BackgroundHierarchical Attention Propagation for Healthcare Representation Learning
Zhang, M., King, C. R., Avidan, M. S., & Chen, Y. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403067 · Cited by 67 · cited 1×
“Our XAI methodology it in an emerging line of research focusing on XAI techniques that are not completely agnostic and tailor their explanations to the medical ield, either by incorporating medical knowledge in the explanation process [11, 32, 141] or focusing on speciic healthcare data characteristics and use cases […”
MethodThe Ethics of AI in Health Care: A Mapping ReviewPreprint
Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3830408 · Cited by 65
DDO: a diabetes mellitus diagnosis ontology
El–Sappagh, S., & Ali, F. (2016) · Applied Informatics · doi:10.1186/s40535-016-0021-2 · Cited by 61 · cited 2×
“Some notables examples are the Disease Ontology (DO) [112], the Open Biomedical Ontology (OBO) [119], the Diabetes mellitus Diagnosis Ontology (DDO) [48], the Systematised Nomenclature of Medicine Clinical Terms (SNOMED-CT) [42], and the Unified Medical Language System (UMLS) [18].”
BackgroundA Novel Deep Neural Network Model for Multi-Label Chronic Disease Prediction
Zhang, X., Zhao, H., Zhang, S., & Li, R. (2019) · Frontiers in Genetics · doi:10.3389/fgene.2019.00351 · Cited by 55 · cited 1×
“For example, when there is the need to simultaneously predict the risk of several chronic diseases [50, 51, 82, 144], when trying to classify unknown genes functional expressions [12, 35], when building a clinical algorithm to predict the diagnoses and medications order of patient’s future visit…”
BackgroundJia, Z., Lü, X., Duan, H., & Li, H. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0807-y · Cited by 51
On the importance of ethnographic methods in AI research
Marda, V., & Narayan, S. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00323-0 · Cited by 49 · cited 1×
“…a specific pool of end-users, i.e., healthcare providers, and observing the use of explanation in the appropriate decisional context [16, 21, 45, 88], i.e., while performing a task supported by a clinical DSS. Ideally, explaining clinical DSS recommendations should help clinicians with trust…”
BackgroundBenchmarking and Survey of Explanation Methods for Black Box Models
Bodria, F., Giannotti, F., Guidotti, R., Naretto, F., Pedreschi, D., & Rinzivillo, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.13076 · Cited by 47 · cited 1×
“This topic has recently witnessed an increased interest that generated vast literature on AI transparency and explainability [19, 57].”
BackgroundSelf-citationPrediction of Chronic Diseases With Multi-Label Neural Network
Ge, R., Zhang, R., & Wang, P. (2020) · IEEE Access · doi:10.1109/access.2020.3011374 · Cited by 44
Using concept hierarchies to improve calculation of patient similarity
Girardi, D., Wartner, S., Halmerbauer, G., Ehrenmüller, M., Kosorus, H., & Dreiseitl, S. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.07.021 · Cited by 43 · cited 1×
“Several similarity metrics could be selected; in this paper, we adopt the Wu-Palmer similarity score (WuP) [138] because it is one of the most commonly used for ICD-9 ontologies [6, 54, 69].”
MethodImpossible Explanations?: Beyond explainable AI in the GDPR from a COVID-19 use case scenario
Hamon, R., Junklewitz, H., Malgieri, G., Hert, P. D., Beslay, L., & Sanchez, I. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445917 · Cited by 42
Harada, Y., Katsukura, S., Kawamura, R., & Shimizu, T. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18115562 · Cited by 38 · cited 1×
“On the one hand, some works have shown that clinicians tend to over-rely on automated suggestions by taking less initiative [81] or accepting incorrect diagnoses suggested by AI [59].”
BackgroundAn Ensemble Multilabel Classification for Disease Risk Prediction
Li, R., Liu, W., Lin, Y., Zhao, H., & Zhang, C. (2017) · Journal of Healthcare Engineering · doi:10.1155/2017/8051673 · Cited by 35
Data Collection Mechanisms in Health and Wellness Apps: Review and Analysis
Philip, B. J., Abdelrazek, M., Bonti, A., Barnett, S., & Grundy, J. (2021) · JMIR mhealth and uhealth · doi:10.2196/30468 · Cited by 32 · cited 1×
“Healthcare data can be collected from various sources, including Electronic Health Records (EHRs) maintained by healthcare providers [68], health and wellness apps [61, 105], and wearable sensors [27, 104].”
BackgroundEthnographic artificial intelligence
Blackwell, A. F. (2021) · Interdisciplinary Science Reviews · doi:10.1080/03080188.2020.1840226 · Cited by 30 · cited 1×
“…involving a specific pool of end-users, i.e., healthcare providers, and observing the use of explanation in the appropriate decisional context [16, 21, 45, 88], i.e., while performing a task supported by a clinical DSS. Ideally, explaining clinical DSS recommendations should help clinicians…”
Background“Interfaces for explanations in human-AI interaction: proposing a design evaluation approach,”
Mucha, H., Robert, S., Breitschwerdt, R., & Fellmann, M. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451759 · Cited by 30 · cited 1×
“Despite these recent eforts, most of the research has been focused on laypeople [7, 29, 95].”
BackgroundPredicting inpatient medication orders from electronic health record data
Rough, K., Dai, A. M., Zhang, K., Xue, Y., Vardoulakis, L. M., Cui, C., Butte, A. J., Howell, M. D., & Rajkomar, A. (2020) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1826 · Cited by 28 · cited 1×
“…functional expressions [12, 35], when building a clinical algorithm to predict the diagnoses and medications order of patient’s future visit [31, 109], when trying to learn multiple indicators of early-stage diseases [33] or when performing clinical text categorization or annotation [13, 41,…”
BackgroundCalibrating Human-AI Collaboration: Impact of Risk, Ambiguity and Transparency on Algorithmic Bias
Schmidt, P., & Biessmann, F. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-57321-8_24 · Cited by 26 · cited 1×
“…context [16, 21, 45, 88], i.e., while performing a task supported by a clinical DSS. Ideally, explaining clinical DSS recommendations should help clinicians with trust calibration [73], i.e., properly adjusting their level of trust according to the actual reliability of the AI system [111].”
BackgroundA review of wearable sensors based monitoring with daily physical activity to manage type 2 diabetes
AlShorman, O., Alshorman, B., & Alkahtani, F. (2020) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v11i1.pp646-653 · Cited by 26
Implementation and termination of a computerized medical information system.
Lawler, F., Cacy, J. R., Viviani, N., Hamm, R. M., & Cobb, S. W. (1996) · PubMed · Cited by 25
Explaining Multi-label Black-Box Classifiers for Health Applications
Panigutti, C., Guidotti, R., Monreale, A., & Pedreschi, D. (2019) · Studies in computational intelligence · doi:10.1007/978-3-030-24409-5_9 · Cited by 25 · cited 2×
“Similarly to Reference [100], Doctor XAI uses a a multi-label decision tree as the inherently interpretable classifier c , however, in the case of binary predictions the decision tree is a binary one.”
BackgroundMethodSelf-citationBlanco, N., O’Hara, L. M., Robinson, G. L., Brown, J., Heil, E. L., Brown, C. H., Stump, B. D., Sigler, B. W., Belani, A., Miller, H. L., et al. (2018) · American Journal of Infection Control · doi:10.1016/j.ajic.2018.04.204 · Cited by 21 · cited 1×
“The reasons behind the low adoption of clinical DSS that do not embed AI have been well studied, with shortcomings including perceived challenges to autonomy, lack of time, and dissatisfaction with user interfaces [17, 72, 74, 78, 94, 110, 129, 130].”
BackgroundInfluence of self‐concept clarity on advice seeking and utilisation
Duan, J., Xu, Y. Q., & Swol, L. M. V. (2020) · Asian Journal Of Social Psychology · doi:10.1111/ajsp.12435 · Cited by 20 · cited 1×
“We controlled for confounding factors such as participants’ familiarity and involvement in the task [44], demographic information such as gender, age, and the type of medical profession.”
Requirements for an expert system explanation facility
Moore, J. D., & Paris, C. L. (1991) · Computational Intelligence · doi:10.1111/j.1467-8640.1991.tb00409.x · Cited by 19 · cited 1×
“The key feature of this new interface is the progressive disclosure of information [34, 92, 123].”
BackgroundExemplars and Counterexemplars Explanations for Image Classifiers, Targeting Skin Lesion Labeling
Metta, C., Guidotti, R., Yin, Y., Gallinari, P., & Rinzivillo, S. (2021) · IEEE Symposium on Computers and Communications (ISCC) · doi:10.1109/iscc53001.2021.9631485 · Cited by 19 · cited 1×
“…line of research focusing on XAI techniques that are not completely agnostic and tailor their explanations to the medical field, either by incorporating medical knowledge in the explanation process [11, 32, 141] or focusing on specific healthcare data characteristics and use cases [89, 100, 102].”
BackgroundSelf-citationChroNet: A multi-task learning based approach for prediction of multiple chronic diseases
Feng, R., Cao, Y. L., Liu, X., Chen, T., Chen, J., Chen, D. Z., Gao, H., & Wu, J. F. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-020-10482-8 · Cited by 18 · cited 1×
“For example, when there is the need to simultaneously predict the risk of several chronic diseases [50, 51, 82, 144], when trying to classify unknown genes functional expressions [12, 35], when building a clinical algorithm to predict the diagnoses and medications order of patient’s future visit [31, 109], when trying…”
BackgroundWhat is needed to mainstream artificial intelligence in health care?
Scott, I. A., Abdel‐Hafez, A., Barras, M. A., & Canaris, S. (2021) · Australian Health Review · doi:10.1071/ah21034 · Cited by 17 · cited 1×
“However, the rate of adoption of AI in health clinics and hospitals is low [4, 114, 140].”
BackgroundComparison of Ontology-Based Semantic-Similarity Measures in the Biomedical Text
Althobaiti, A. F. S. (2017) · Journal of Computer and Communications · doi:10.4236/jcc.2017.52003 · Cited by 8 · cited 1×
“Several similarity metrics could be selected; in this article, we adopt the Wu-Palmer similarity score (WuP) [138], because it is one of the most commonly used for ICD-9 ontologies [6, 54, 69].”
MethodVerma, H., Schaer, R., Reichenbach, J., Mario, J., Prior, J. O., Évéquoz, F., & Depeursinge, A. (2021) · Research Square · doi:10.21203/rs.3.rs-496758/v1 · Cited by 6 · cited 1×
“Trust plays a central role in the adoption of new technologies and explanations of AI recommendations are often touted as the solution to trust issues [49, 108, 134, 139].”
BackgroundEthical, Societal and Legal Issues in Deep Learning for Healthcare
Panigutti, C., Monreale, A., Comandé, G., & Pedreschi, D. (2022) · WORLD SCIENTIFIC (EUROPE) eBooks · doi:10.1142/9781800610941_0009 · Cited by 6 · cited 1×
“Being able to explain clinical decisions to patients and be held accountable for adverse outcomes of their diagnosis are key ethical responsibilities of every doctor [93, 101].”
BackgroundSelf-citationUnfolding the Legal Component of Trustworthy AI: A Must to Avoid Ethics WashingPreprint
Comandé, G. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3690633 · Cited by 5
(2006) · PubMed
TAKE THIS ADVICE AND SHOVE IT.
GINO, F., & SCHWEITZER, M. E. (2008) · Academy of Management Proceedings · doi:10.5465/ambpp.2008.33716516
Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L. A., & Mark, R. (2020) · PhysioNet · doi:10.13026/a3wn-hq05
Research works printed but not linked: 8
- Barbara D. Adams, Lora E. Bruyn, Sébastien Houde, Paul Angelopoulos, Kim Iwasa-Madge, and Carol McCann. 2003. Trust in automated systems. Minist. Nat. Defen. (2003).
- Tal Baumel, Jumana Nassour-Kassis, Raphael Cohen, Michael Elhadad, and Noémie Elhadad. 2018. Multi-label classification of patient notes: Case study on ICD code assignment. In Proceedings of the Workshops at the 32nd AAAI Conference on Artificial Intelligence.
- Donald J. Berndt and James Clifford. 1994. Using dynamic time warping to find patterns in time series. In Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining (AAAIWS’94). AAAI Press, 359–370. Retrieved from http://dl.acm.org/citation.cfm?id=3000850.3000887.
- Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F. Stewart, and Jimeng Sun. 2016. Doctor AI: Predicting clinical events via recurrent neural networks. In Proceedings of the Machine Learning for Healthcare Conference. PMLR, 301–318.
- Ian Covert, Scott M. Lundberg, and Su-In Lee. 2021. Explaining by removing: A unified framework for model explanation. J. Mach. Learn. Res. 22 (2021), 209–1.
- ISO 9241-210:2019. 2019. Ergonomics of Human-system Interaction Human-centred Design for Interactive Systems. Standard. International Organization for Standardization.
- Jennifer Preece, Yvonne Rogers, and Helen Sharp. 2019. Interaction Design: Beyond Human-Computer Interaction (5th ed.). Wiley.
- Sana Tonekaboni, Shalmali Joshi, Melissa D. McCradden, and Anna Goldenberg. 2019. What clinicians want: Contextualizing explainable machine learning for clinical end use. In Proceedings of the Machine Learning for Healthcare Conference. PMLR, 359–380.
Cited by
Mersha, M., Lam, K., Wood, J., AlShami, A. K., & Kalita, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128111 · Cited by 237
“Clinical decision support: XAI systems assist healthcare professionals by providing transparent and interpretable explanations of the decision-making processes of models handling complex data [159].”
BackgroundHuman-centered evaluation of explainable AI applications: a systematic review
Kim, J., Maathuis, H., & Sent, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1456486 · Cited by 130
Human-Centered Design and Development in Digital Health: Approaches, Challenges, and Emerging Trends
Tzimourta, K. D. (2025) · Cureus · doi:10.7759/cureus.85897 · Cited by 51
“Furthermore, involving clinicians early in the development process not only yields interface improvements but can also increase trust as users gain insight into how the AI works and contribute their domain knowledge to its design [19,20].”
Unpacking Human-AI Interaction in Safety-Critical Industries: A Systematic Literature Review
Bach, T. A., Kristiansen, J. K., Babić, A., & Jacovi, A. (2024) · IEEE Access · doi:10.1109/access.2024.3437190 · Cited by 61
“Understanding user needs was important to ensure the information the AI presents to the user is not excessive, unnecessary, or confusing [86], [96].”
BackgroundInfluentialBayor, A. A., Li, J., Yang, I. A., & Varnfield, M. (2025) · Journal of Medical Internet Research · doi:10.2196/63733 · Cited by 46
“The utilization of emerging technologies such as XAI [39,43,93,105] holds promise in addressing these challenges by ensuring human comprehension of AI decisions.”
Influential
Metadata
- Authors
- Panigutti, C., Beretta, A., Fadda, D., Giannotti, F., Pedreschi, D., Perotti, A., & Rinzivillo, S.
- Affiliations
- University of Pisa (IT)
- National Research Council (IT)
- Scuola Normale Superiore (IT)
- CENTAI Institute, Italy
- ORCID
- Venue
- ACM Transactions on Interactive Intelligent Systems
- Published
- 2023-12-08
- Volume
- 13
- Issue
- 4
- Pages
- 1-35
- ISSN
- 2160-6455 (ISSN-L)
- 2160-6463
- OpenAlex
- W4324155120
- Cited by
- 127
- References
- 119
- Citations by year
- 2026: 56 · 2025: 37 · 2024: 29 · 2023: 5
- FWCI
- 11.97
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Interactive Intelligent Systems (publishedVersion, cc-by)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, other-oa)
- ISTI Open Portal (submittedVersion, other-oa)
- Scuola Normale Superiore di Pisa (submittedVersion, other-oa)
- Funders
- European Union (ERC-2018-ADG G.A. 834756 (XAI))
- HumanE AI Net (952026)
- PNRR - M4C2 - Investimento 1.3, Partenariato Esteso (PE00000013)
- UK government (10061955)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- explainable AI, clinical decision support systems, human-centered design, trust in AI, multi-label classification, iterative prototyping
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.