Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review

In the paper

“An additional system that incorporated post-hoc explainability was designed by Lamy et al. [123] and provided local explanations of the preference model using rainbow boxes [56].”

What the sources say

• Not checked yetcites [123] Lamy, 2020, Explainable decision support through the learning and visualization of preferences from a formal ontology of antibiotic treatments, J. Biomed. Inform. source
• Not checked yetcites [56] Lambrecht, 2019, Algorithmic bias? an empirical study of apparent gender-based discrimination in the display of stem career ads, Manag. Sci. source

In the paper

“Militello et al. [121] Epicardial fat volume A user-centred Graphical User Interface (GUI) design is used to allow for safe interaction of the physician as well as for an effective integration into the existing clinical workflow”

What the sources say

• Not checked yetcites [121] Militello, 2019, A semi-automatic approach for epicardial adipose tissue segmentation and quantification on cardiac CT scans, Comput. Biol. Med. source

In the paper

“Previous works have evaluated XAI in AI-based CDSS, but only as a secondary aspect [99,101,103].”

What the sources say

• Not checked yetcites [99] Jin, 2020, Artificial intelligence in glioma imaging: Challenges and advances, J. Neural Eng. source
• Not checked yetcites [101] Rundo, 2020, Recent advances of HCI in decision-making tasks for optimized clinical workflows and precision medicine, J. Biomed. Inform. source
• Not checked yetcites [103] Angehrn, 2020, Artificial Intelligence and Machine Learning Applied at the Point of Care, Front. Pharmacol. source

In the paper

“In terms of post-hoc explainability, perturbation-based models that use model-agnostic explanations are common in the literature [98,99,118,120,127] and have been used in two of the proposed CDSS [118,120].”

What the sources say

• Not checked yetcites [98] Zucco, C., Liang, H., Fatta, G.D., and Cannataro, M. (2019, January 3–6). Explainable Sentiment Analysis with Applications in Medicine. Proceedings of the 2018 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2018, Madrid, Spain. source
• Not checked yetcites [99] Jin, 2020, Artificial intelligence in glioma imaging: Challenges and advances, J. Neural Eng. source
• Not checked yetcites [118] Wang, D., Yang, Q., Abdul, A., and Lim, B.Y. (2019, January 4–9). Designing theory-driven user-centric explainable AI. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Scotland, UK. source
• Not checked yetcites [120] Hu, 2020, Using a machine learning approach to predict mortality in critically ill influenza patients: A cross-sectional retrospective multicentre study in Taiwan, BMJ Open source
• Not checked yetcites [127] Cai, 2019, “Hello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making, Proceedings of the ACM on Human-Computer Interaction source

In the paper

“Finally, we separated the 33 papers into eight literature reviews (24.2%) [97][98][99][100][101][102][103][104], nine papers discussing aspects of the use of XAI in CDSS (27.3%) [48,[105][106][107][108][109][110][111][112], four papers that discussed the implementation of a CDSS without XAI (12.1%) [113-116], and 12 papers describing the implementation of a CDSS with (36.4%)XAI [117-128]”

In the paper

“This reduced our set to 121 papers (39 conference and 82 journal).”

In the paper

“The six searches had a minimum of 35 and a maximum of 181 results each, with a total of 668.”

In the paper

“Three of these were not available on any platform at our disposal, and one additional paper was removed when we discovered that it was a pre-print using an ACM TOIIS template but was not published in the ACM Digital Library.”

In the paper

“We eliminated papers that were not peer-reviewed conference or journal papers (e.g., theses, dissertations, books, book chapters, pre-prints, or other archived articles and posters) and 10 papers that were not written in English, leaving 132 papers.”

In the paper

“Reconciliation resulted in seven disputed papers being excluded and 16 retained, leaving 33 papers.”

In the paper

“After both authors had independently classified each paper, 16 papers were removed by agreement.”

In the paper

“The author pairs disagreed on 23 papers giving a 68% agreement rate on paper inclusion/exclusion.”

In the paper

“These papers were published between 2008 and 24 July 2020: one paper was published in 2008, three papers were published in 2018, 13 papers were published in 2019, and 16 papers were published in 2020 (until the 24 July).”

In the paper

“Kunapuli et al. [128] indicated that XAI could support specific rational reasoning processes, enabling CDSS to support their decisions with understandable interpretations to users with/without ML expertise.”

What the sources say

• Not checked yetcites [128] Kunapuli, 2018, A Decision-Support Tool for Renal Mass Classification, J. Digit. Imaging source

In the paper

“In the study by Blanco et al. [122], the CDSS was developed using a bidirectional gated recurrent unit (BiGRU) with attention mechanism, which allowed for the exploration of how much each fragment of the text contributed to a prediction, thus providing local explanations.”

What the sources say

• Not checked yetcites [122] Blanco, 2020, Extracting Cause of Death from Verbal Autopsy with Deep Learning interpretable methods, IEEE J. Biomed. Health Inform. source

In the paper

“In some cases, users also believed that the interpretation of AI decisions might alleviate their own decision-making biases.”

What the sources say

• Not checked yetcites [111] Liao, Q.V., Gruen, D., and Miller, S. (2020, January 25–30). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA. source

In the paper

“They presented their interface to 11 medical experts for usability and acceptability validation, demonstrating that XAI could provide the user with a good indication of the confidence level of their choice [126].”

What the sources say

• Not checked yetcites [126] Lamy, 2019, Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach, Artif. Intell. Med. source

In the paper

“Wang et al. [118] identified that XAI could support different explanation types by articulating how people understand events or observations through explanations and can be leveraged to mitigate decision biases and cognitive biases [98,103,112].”

What the sources say

• Not checked yetcites [118] Wang, D., Yang, Q., Abdul, A., and Lim, B.Y. (2019, January 4–9). Designing theory-driven user-centric explainable AI. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Scotland, UK. source
• Not checked yetcites [98] Zucco, C., Liang, H., Fatta, G.D., and Cannataro, M. (2019, January 3–6). Explainable Sentiment Analysis with Applications in Medicine. Proceedings of the 2018 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2018, Madrid, Spain. source
• Not checked yetcites [103] Angehrn, 2020, Artificial Intelligence and Machine Learning Applied at the Point of Care, Front. Pharmacol. source
• Not checked yetcites [112] Johnson, 2019, AI, Machine Learning, and Ethics in Health Care, J. Leg. Med. source

In the paper

“The results showed that XAI could gain further insights or evidence, and thus enhance decision confidence or generate the hypothesis about causality.”

What the sources say

• Not checked yetcites [111] Liao, Q.V., Gruen, D., and Miller, S. (2020, January 25–30). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA. source

In the paper

“Kunapuli et al. [128] built a CDSS using Relational Functional Gradient Boosting (RFGB), a statistical relational learning method which attributes its explainability to the usage of tree models and the provision of explanations in terms of features of interest.”

What the sources say

• Not checked yetcites [128] Kunapuli, 2018, A Decision-Support Tool for Renal Mass Classification, J. Digit. Imaging source

In the paper

“They reported that XAI could provide different information types to make intelligent systems explainable and more acceptable and trustworthy to users.”

What the sources say

• Not checked yetcites [106] Vorm, E.S. (2018, January 3–5). Assessing Demand for Transparency in Intelligent Systems Using Machine Learning. Proceedings of the 2018 Innovations in Intelligent Systems and Applications (INISTA), Thessaloniki, Greece. source

In the paper

“They interviewed 20 participants to identify gaps between the current XAI algorithmic work and practices to create explainable AI products.”

What the sources say

• Not checked yetcites [111] Liao, Q.V., Gruen, D., and Miller, S. (2020, January 25–30). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA. source

In the paper

“Case-Based Reasoning is an intrinsically explainable method that was used by Lamy et al. [126] for their CDSS for breast cancer, which was also supported by visual explanations in the form of rainbow boxes and a polar multidimensional scaling scatter plot.”

What the sources say

• Not checked yetcites [126] Lamy, 2019, Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach, Artif. Intell. Med. source

In the paper

“Rule-based systems were developed in two studies to provide local explanations, as they were considered closer to human reasoning, and thus more preferable by clinicians [124,125].”

What the sources say

• Not checked yetcites [124] Tan, T.Z., Ng, G.S., and Quek, C. (2008, January 1–6). Improving tractability of Clinical Decision Support system. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China. source
• Not checked yetcites [125] Alonso, 2018, An Ontology-Based Interpretable Fuzzy Decision Support System for Diabetes Diagnosis, IEEE Access source

In the paper

“Sadeghi et al. [117] proposed a CDSS that used a Random Forest to predict the outcome.The authors stated that the use of time-domain features leads to a transparent and explainable CDSS, but there is not sufficient information towards this claim.”

What the sources say

• Not checked yetcites [117] Sadeghi, 2019, Sleep quality prediction in caregivers using physiological signals, Comput. Biol. Med. source

In the paper

“Moreover, Wang et al. [118] believe that what constitutes a good explanation should draw from social science instead of depending on researchers' intuition, and justification is required for choosing different explanation types or representations.”

What the sources say

• Not checked yetcites [118] Wang, D., Yang, Q., Abdul, A., and Lim, B.Y. (2019, January 4–9). Designing theory-driven user-centric explainable AI. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Scotland, UK. source

In the paper

“XAI could help the user understand when to trust a model and why an error may occur [97,116].”

What the sources say

• Not checked yetcites [97] Luz, 2020, Machine learning in infection management using routine electronic health records: Tools, techniques, and reporting of future technologies, Clin. Microbiol. Infect. source
• Not checked yetcites [116] Xie, Y., Chen, M., Kao, D., Gao, G., and Chen, X.A. (2020, January 25–30). CheXplain: Enabling Physicians to Explore and Understand Data-Driven, AI-Enabled Medical Imaging Analysis. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA. source

In the paper

“Furthermore, Luz et al. [97] argue that a thorough reasoning is required for choosing between transparent and black-box ML algorithms, because post-hoc interpretation methods that develop a mirror model of the original one to add explainability could provide an inaccurate representation of the original model.”

What the sources say

• Not checked yetcites [97] Luz, 2020, Machine learning in infection management using routine electronic health records: Tools, techniques, and reporting of future technologies, Clin. Microbiol. Infect. source

In the paper

“Furthermore, the concept of interpretability is often highly subjective [115].”

What the sources say

• Not checked yetcites [115] Gangavarapu, T., S Krishnan, G., Kamath S, S., and Jeganathan, J. (2020). FarSight: Long-Term Disease Prediction Using Unstructured Clinical Nursing Notes. IEEE Trans. Emerg. Top. Comput., 1–16.

In the paper

“Richard et al. [48] proposed a definition stating that a transparent classification system should be understandable, use an interpretable type of classifier and learning system, produce traceable results, and use a revisable classifier.”

What the sources say

• Not checked yetcites [48] Richard, 2020, Transparency of Classification Systems for Clinical Decision Support, Information Processing and Management of Uncertainty in Knowledge-Based Systems

In the paper

“There is no universal definition of explainability [48,98] nor a well-recognized equivalence or distinction between”

What the sources say

• Not checked yetcites [48] Richard, 2020, Transparency of Classification Systems for Clinical Decision Support, Information Processing and Management of Uncertainty in Knowledge-Based Systems
• Not checked yetcites [98] Zucco, C., Liang, H., Fatta, G.D., and Cannataro, M. (2019, January 3–6). Explainable Sentiment Analysis with Applications in Medicine. Proceedings of the 2018 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2018, Madrid, Spain. source

In the paper

“Moreover, Jin et al. [99] reported that there is a lack of evaluation of XAI techniques on glioma imaging due to lack of focus on practical challenges relating to clinical implementation of XAI.”

What the sources say

• Not checked yetcites [99] Jin, 2020, Artificial intelligence in glioma imaging: Challenges and advances, J. Neural Eng. source

In the paper

“As to testing the systems, Lamy et al. [126] raised the need to confirm their results on a larger user study, in addition to the lack of user studies for some XAI-based approaches as discussed in Section 4.3 (RQ3).”

What the sources say

• Not checked yetcites [126] Lamy, 2019, Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach, Artif. Intell. Med. source

In the paper

“They also discovered that identifying the motivation of explainability helps to select XAI techniques, foresee their limitations, and fill in the gaps occurring while designing user experiences.”

What the sources say

• Not checked yetcites [111] Liao, Q.V., Gruen, D., and Miller, S. (2020, January 25–30). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA. source

In the paper

“On the other hand, there are some studies supporting that XAI is not always necessary [46,134,135].”

What the sources say

• Not checked yetcites [46] Lipton, 2018, The mythos of model interpretability, Queue source
• Not checked yetcites [134] London, 2019, Artificial intelligence and black-box medical decisions: Accuracy versus explainability, Hastings Cent. Rep. source
• Not checked yetcites [135] Baldi, 2018, Deep learning in biomedical data science, Annu. Rev. Biomed. Data Sci. source

In the paper

“Currently, most of the decisions made by AI-based CDSS cannot be interpreted in a transparent way potentially limiting the uptake, trust, and usability of these systems in practice [20,133].”

What the sources say

• Not checked yetcites [20] Vellido, 2019, The importance of interpretability and visualization in machine learning for applications in medicine and health care, Neural Comput. Appl. source
• Not checked yetcites [133] Gomolin, 2020, Artificial intelligence applications in dermatology: Where do we stand?, Front. Med. source

In the paper

“A rapid advancement of XAI is evident, and the recent study by Linardatos et al. [132] has identified four main areas of focus: methods for explaining complex black-box models, methods for creating white-box models, methods that promote fairness and restrict the existence of discrimination, and methods for analyzing sensitivity of model predictions.”

What the sources say

• Not checked yetcites [132] Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. (2021). Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy, 23. source

In the paper

“Tan et al. [124] argue that a CDSS should have high tractability, which requires human like reasoning, step by step inference, explanation capacity, and user-familiar terms, to gain user acceptance.”

What the sources say

• Not checked yetcites [124] Tan, T.Z., Ng, G.S., and Quek, C. (2008, January 1–6). Improving tractability of Clinical Decision Support system. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China. source

In the paper

“Metrics to evaluate the performance of XAI techniques require further study [18].”

What the sources say

• Not checked yetcites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion source

In the paper

“They conclude their survey by stating that”

What the sources say

• Not checked yetcites [141] Zhou, J., Gandomi, A.H., Chen, F., and Holzinger, A. (2021). Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10. source

In the paper

“Although XAI can assist with identifying issues with the data, the problem with unstructured medical data remains a challenge for the development of usable AI-based systems.”

In the paper

“Bussone et al. [75] found that giving clinicians a fuller explanation of the facts that led to the system's proposed diagnosis had a positive effect on trust but caused over-reliance issues.”

What the sources say

• Not checked yetcites [75] Bussone, A., Stumpf, S., and O’Sullivan, D. (2015, January 21–23). The role of explanations on trust and reliance in clinical decision support systems. Proceedings of the 2015 International Conference on Healthcare Informatics, Dallas, TX, USA. source

In the paper

“The”

What the sources say

• Not checked yetcites [138] Carvalho, D.V., Pereira, E.M., and Cardoso, J.S. (2019). Machine learning interpretability: A survey on methods and metrics. Electronics, 8. source

In the paper

“Finally, there is a need for more robust user studies [66,143].”

What the sources say

• Not checked yetcites [66] Doshi-Velez, F., and Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv.
• Not checked yetcites [143] Kenny, 2021, Explaining Black-Box classifiers using Post-Hoc explanations-by-example: The effect of explanations and error-rates in XAI user studies, Artif. Intell. source

In the paper

“Subjective measurements can provide valuable insight into the user's experience, however, there is an overall lack of validated and reliable evaluation metrics.”

In the paper

“Holzinger et al. [142] introduced the”

What the sources say

• Not checked yetcites [142] Holzinger, 2020, Measuring the quality of explanations: The system causability scale (SCS), KI-Künstliche Intell. source

In the paper

“Additionally, Bruckert et al. [137] shed light on the difficulties that are presented when rendering ML models explainable for healthcare purposes, highlighting that the implementation of such systems requires overlap between different disciplines and professions.”

What the sources say

• Not checked yetcites [137] Bruckert, 2020, The Next Generation of Medical Decision Support: A Roadmap Toward Transparent Expert Companions, Front. Artif. Intell. source

In the paper

“They found that some properties (clarity, broadness, and completeness) are still in shortage of appropriate metrics, and so is the class of explanations that are example-based.”

What the sources say

• Not checked yetcites [141] Zhou, J., Gandomi, A.H., Chen, F., and Holzinger, A. (2021). Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10. source

In the paper

“However, there are very few studies like this that give insight into what clinicians want or need.”

In the paper

“A main challenge remains the selection of methods used to present explanations in an informative and efficient-and therefore clinically useful-manner.”

In the paper

“Through a case study, Jacobs et al. [144] found that incorrect ML recommendations may affect clinicians and lower the accuracy of decisions, while explanations were found insufficient for addressing over-reliance on a model that suggests erroneous decisions.”

What the sources say

• Not checked yetcites [144] Jacobs, 2021, How machine-learning recommendations influence clinician treatment selections: The example of the antidepressant selection, Transl. Psychiatry source

In the paper

“Blanco et al. [122] BiGRU with attention mechanism to show the contribution of each fragment of text to the prediction specific ante-hoc local”

What the sources say

• Not checked yetcites [122] Blanco, 2020, Extracting Cause of Death from Verbal Autopsy with Deep Learning interpretable methods, IEEE J. Biomed. Health Inform. source

In the paper

“They found that explanation strategies ought to be selected according to the clinicians' prior experience with ML and that those with prior experience perceived a higher utility from the ML recommendations.”

What the sources say

• Not checked yetcites [144] Jacobs, 2021, How machine-learning recommendations influence clinician treatment selections: The example of the antidepressant selection, Transl. Psychiatry source

In the paper

“On the other hand, less detailed explanations had the opposite effect, as this made participants question the system's reliability and caused self-reliance issues [75].”

What the sources say

• Not checked yetcites [75] Bussone, A., Stumpf, S., and O’Sullivan, D. (2015, January 21–23). The role of explanations on trust and reliance in clinical decision support systems. Proceedings of the 2015 International Conference on Healthcare Informatics, Dallas, TX, USA. source

In the paper

“Tan et al. [124] CLFNN, which autonomously generates fuzzy rules to provide human-like reasoning specific ante-hoc local”

What the sources say

• Not checked yetcites [124] Tan, T.Z., Ng, G.S., and Quek, C. (2008, January 1–6). Improving tractability of Clinical Decision Support system. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China. source

In the paper

“While there are many studies on XAI in medicine, there is a limited number that focus on the context of CDSS.”

In the paper

“Similarly, there is little discussion on the impact of XAI on patients from the patients perspective.”

In the paper

“However, despite some user studies reporting positive views on CDSS, especially in light of explainability, there is still skepticism around their use in practice.”

In the paper

“Hu et al. [120] SHAP [79] for summary plot and partial dependence plot agnostic post-hoc global”

What the sources say

• Not checked yetcites [120] Hu, 2020, Using a machine learning approach to predict mortality in critically ill influenza patients: A cross-sectional retrospective multicentre study in Taiwan, BMJ Open source
• Not checked yetcites [79] Lundberg, S.M., and Lee, S.I. (2017, January 4–9). A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.

In the paper

“Lamy et al. [123] Visualised the created preference model using rainbow boxes [131] agnostic post-hoc local”

What the sources say

• Not checked yetcites [123] Lamy, 2020, Explainable decision support through the learning and visualization of preferences from a formal ontology of antibiotic treatments, J. Biomed. Inform. source
• Not checked yetcites [131] Lamy, 2017, Rainbow boxes: A new technique for overlapping set visualization and two applications in the biomedical domain, J. Vis. Lang. Comput. source

In the paper

“Lamy et al. [126] Visual (using rainbow-boxes [131] and a polar multidimensional scaling scatter plot) case-based reasoning approach specific ante-hoc local”

What the sources say

• Not checked yetcites [126] Lamy, 2019, Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach, Artif. Intell. Med. source
• Not checked yetcites [131] Lamy, 2017, Rainbow boxes: A new technique for overlapping set visualization and two applications in the biomedical domain, J. Vis. Lang. Comput. source

In the paper

“El-Sappagh et al. [125] Semantically interpretable FRBS with the integration of semantic ontology-based reasoning specific ante-hoc local”

What the sources say

• Not checked yetcites [125] Alonso, 2018, An Ontology-Based Interpretable Fuzzy Decision Support System for Diabetes Diagnosis, IEEE Access source

In the paper

“It has been argued that despite the existence of much evidence supporting their usefulness, ML-based systems are unlikely to be adopted in routine medical practice beyond a limited number of niche applications unless these challenges are addressed, most likely by having systems provide satisfactory explanations for their decisions [20,21].”

What the sources say

• Not checked yetcites [20] Vellido, 2019, The importance of interpretability and visualization in machine learning for applications in medicine and health care, Neural Comput. Appl. source
• Not checked yetcites [21] Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., and Kagal, L. (2018, January 1–3). Explaining explanations: An overview of interpretability of machine learning. Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy. source

In the paper

“As a result, incorporating XAI principles into CDSS is essential if the potential beneficial impact is to be fulfilled.”

In the paper

“Studies reported benefits of the use of XAI such as the fact that it could enhance decision confidence for clinicians, or generate the hypothesis about causality, which ultimately leads to increased trustworthiness and acceptability of the system and potential for its incorporation in the clinical workflow.”

In the paper

“Unfortunately, secondary factors also complicate solutions, for instance, different applications usually have different interpretability and explainability needs [20] working against generalizable solutions that span a number of situations.”

What the sources say

• Not checked yetcites [20] Vellido, 2019, The importance of interpretability and visualization in machine learning for applications in medicine and health care, Neural Comput. Appl. source

In the paper

“Although, giving explanations can increase understanding of and trust in a system [19,46,64], simple explanations can hide undesirable attributes of the system and may mislead users into coming to dangerous or unfounded conclusions that could ultimately be unethical [21].”

What the sources say

• Not checked yetcites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access source
• Not checked yetcites [46] Lipton, 2018, The mythos of model interpretability, Queue source
• Not checked yetcites [64] Miller, T., Howe, P., and Sonenberg, L. (2017). Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv.

In the paper

“Additionally, XAI can help to shed some light onto causality [18,46,67] although, there is a recognized need to go beyond causality/correlation to true “causability” [3].”

What the sources say

• Not checked yetcites [3] Holzinger, 2019, Causability and explainability of artificial intelligence in medicine, Wiley Interdiscip. Rev. Data Min. Knowl. Discov. source

In the paper

“Translating ML models effectively to clinical practice requires establishing clinicians' trust in the system.”

In the paper

“XAI can be used to help us evaluate if predictions are biased and defend algorithmic decisions as being fair and ethical [18,19,46,66].”

What the sources say

• Not checked yetcites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion source
• Not checked yetcites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access source
• Not checked yetcites [46] Lipton, 2018, The mythos of model interpretability, Queue source
• Not checked yetcites [66] Doshi-Velez, F., and Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv.

In the paper

“Although, giving explanations can increase understanding of and trust in a system [19,46,64], simple explanations can hide undesirable attributes of the system and may mislead users into coming to dangerous or unfounded conclusions that could ultimately be unethical [21].”

What the sources say

• Not checked yetcites [21] Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., and Kagal, L. (2018, January 1–3). Explaining explanations: An overview of interpretability of machine learning. Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy. source

In the paper

“In this review, we followed a structured process that involved the following: 1. Specifying research questions 2. Conducting searches of specified databases 3. Selecting studies by criterion 4. Filtering studies by evaluating their pertinence 5. Extracting data 6.Synthesizing results.”

In the paper

“The authors also consider the fact that the first requirement of AI systems in medicine is clinical validation, while explainability is a second aspect.”

What the sources say

• Not checked yetcites Amann et al. [93] Amann, J., Blasimme, A., Vayena, E., Frey, D., and Madai, V.I. (2020). Explainability for artificial intelligence in healthcare: A multidisciplinary perspective. BMC Med. Inform. Decis. Mak., 20. source

In the paper

“Providing explanations for how recommendations are arrived at will allow practitioners to make more nuanced, and in some cases, life-saving decisions.”

In the paper

“Tonekaboni et al. [69] found that the model's overall accuracy was not sufficient on its own to allow clinicians to make an informed decision, clinicians wanted to know the subset of features driving a prediction to allow them to compare the model decision to their clinical judgment.”

What the sources say

• Not checked yetcites Tonekaboni et al. [69] Tonekaboni, S., Joshi, S., McCradden, M.D., and Goldenberg, A. (2019, January 13–14). What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use. Proceedings of the Machine Learning for Healthcare Conference, Boston, MA, USA.

In the paper

“There are also concerns that post-hoc methods could expose the original models to adversarial attacks [18] or could lead to the generation of classifiers whose post-hoc explanations could be arbitrarily controlled [63].”

What the sources say

• Not checked yetcites [63] Slack, D., Hilgard, S., Jia, E., Singh, S., and Lakkaraju, H. (2020, January 7–8). Fooling lime and shap: Adversarial attacks on post hoc explanation methods. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA. source

In the paper

“We conducted a systematic literature review by adapting the guidelines proposed by Kitchenham [94].”

What the sources say

• Not checked yetcites [94] Kitchenham, B.A., and Charters, S. (2020, July 24). Guidelines for Performing Systematic Literature Reviews in Software Engineering; Technical Report EBSE 2007-001, Keele University and Durham University Joint Report. Available online: http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=2BE22FED09591B99D6A7ACF8FE2258D5?.

In the paper

“The need for XAI in CDSS, and the medical field in general, is amplified by the need for ethical and fair decision-making and the fact that AI trained with historical data can be a reinforcement agent of historical actions and biases that should be uncovered.”

In the paper

“After some trial and error with a range of terms, we performed the following six searches (S1-S6).The search terms were used to search Google Scholar (currently the most comprehensive academic search engine according to recent studies [95,96]) on 24 July 2020”

What the sources say

• Not checked yetcites [95] Thelwall, 2018, Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories, J. Inf.
• Not checked yetcites [96] Gusenbauer, 2019, Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases, Scientometrics source

In the paper

“Medical certification comes after the system is compliant with regulatory standards and prediction accuracy is usually the main measurement of clinical validation.”

What the sources say

• Not checked yetcites Amann et al. [93] Amann, J., Blasimme, A., Vayena, E., Frey, D., and Madai, V.I. (2020). Explainability for artificial intelligence in healthcare: A multidisciplinary perspective. BMC Med. Inform. Decis. Mak., 20. source

In the paper

“Likewise, more complex models may not be more accurate.”

In the paper

“There is often a perceived trade-off between the performance (predictive accuracy) of a model and explainability [3,18,19].”

What the sources say

• Not checked yetcites [3] Holzinger, 2019, Causability and explainability of artificial intelligence in medicine, Wiley Interdiscip. Rev. Data Min. Knowl. Discov. source
• Not checked yetcites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion source
• Not checked yetcites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access source

In the paper

“Bussone et al. [75] found that clinicians wanted better explanations from the CDSS to help them interpret the system's confidence, to verify that the clinical disorder fit the CDSS suggestion, to better understand the reasoning chain of the system, and to make different diagnoses in order to help them make an assessment of the reliability of the system's decisions.”

What the sources say

• Not checked yetcites Bussone et al. [75] Bussone, A., Stumpf, S., and O’Sullivan, D. (2015, January 21–23). The role of explanations on trust and reliance in clinical decision support systems. Proceedings of the 2015 International Conference on Healthcare Informatics, Dallas, TX, USA. source

In the paper

“There are also concerns that post-hoc methods could expose the original models to adversarial attacks [18] or could lead to the generation of classifiers whose post-hoc explanations could be arbitrarily controlled [63].”

What the sources say

• Not checked yetcites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion source

In the paper

“However, given the undeniable need for transparency and explainability in medical practice and the growing use of CDSS leveraging AI, XAI has started to be incorporated in recent AI-based CDSS.”

In the paper

“The remaining 72 papers were divided randomly into three groups and shared between three pairs of authors.Each pair took one group of papers and classified them as either include or exclude based on inclusion criteria.”

In the paper

“A ratio of n unique /n total = 0.39 is indicative of a cohesive set of searches that together have a desired degree of internal consistency.”

In the paper

“We see an upward trend in the number of relevant studies published over time which indicated the increased interest in XAI-enabled CDSS.”

In the paper

“Even though some other XAI-based approaches have not yet been tested on users, the benefits of the XAI presented in these works still seem likely to be useful in practice.”

In the paper

“This suggests that AI transparency in collaborative decision making could allow experts to integrate AI assistants into daily practice and gain a richer understanding of the key issues they find.”

What the sources say

• Not checked yetcites [127] Cai, 2019, “Hello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making, Proceedings of the ACM on Human-Computer Interaction source

In the paper

“XAI therefore could greatly enhance the effectiveness of decision-support and clinician confidence [128] especially when high-stakes decisions are being made [127], which is the key factor for the success of the model in the practical use stage [122,124].”

What the sources say

• Not checked yetcites [128] Kunapuli, 2018, A Decision-Support Tool for Renal Mass Classification, J. Digit. Imaging source
• Not checked yetcites [127] Cai, 2019, “Hello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making, Proceedings of the ACM on Human-Computer Interaction source
• Not checked yetcites [122] Blanco, 2020, Extracting Cause of Death from Verbal Autopsy with Deep Learning interpretable methods, IEEE J. Biomed. Health Inform. source
• Not checked yetcites [124] Tan, T.Z., Ng, G.S., and Quek, C. (2008, January 1–6). Improving tractability of Clinical Decision Support system. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China. source

In the paper

“Therefore, explainability of the support system can improve the acceptability of CDSS by clinicians [48] increase the chances of the complex AI systems' adoption and clinical feasibility of a novel CDSS [105,121].”

What the sources say

• Not checked yetcites [48] Richard, 2020, Transparency of Classification Systems for Clinical Decision Support, Information Processing and Management of Uncertainty in Knowledge-Based Systems
• Not checked yetcites [105] Mahadevaiah, 2020, Artificial intelligence-based clinical decision support in modern medical physics: Selection, acceptance, commissioning, and quality assurance, Med. Phys. source
• Not checked yetcites [121] Militello, 2019, A semi-automatic approach for epicardial adipose tissue segmentation and quantification on cardiac CT scans, Comput. Biol. Med. source

In the paper

“As discussed in Section 2.2.1, in 2016 the European Union passed the GDPR which has been interpreted as a requirement for any decision made based on an algorithm to be explainable to the user [106].”

What the sources say

• Not checked yetcites [106] Vorm, E.S. (2018, January 3–5). Assessing Demand for Transparency in Intelligent Systems Using Machine Learning. Proceedings of the 2018 Innovations in Intelligent Systems and Applications (INISTA), Thessaloniki, Greece. source

In the paper

“Nevertheless, XAI may lead to greater uptake and use of CDSS, and may become a requirement in the future due to societal, regulatory, and ethical pressures [18], which could make the difference between success and failure of the system [19].”

What the sources say

• Not checked yetcites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion source
• Not checked yetcites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access source

In the paper

“XAI needs to be interactive and human-like with customized explanations for different users.”

In the paper

“To close the gap requires interdisciplinary work that studies how humans explain, formalizes the patterns in algorithmic forms, and explains outputs in a transparent, easy to interpret manner [67].”

What the sources say

• Not checked yetcites [67] Molnar, C., Casalicchio, G., and Bischl, B. (2020). Interpretable Machine Learning—A Brief History, State-of-the-Art and Challenges. arXiv. source

In the paper

“However, there is an open debate on whether or not XAI in these contexts is necessary and/or worth the substantial overall cost [19].”

What the sources say

• Not checked yetcites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access source

In the paper

“Baldi [135] explains this argument using examples of the lack of explainability of many processes in our daily lives, for example, how cars, computers, cell phones, or even our brains work.”

What the sources say

• Not checked yetcites [135] Baldi, 2018, Deep learning in biomedical data science, Annu. Rev. Biomed. Data Sci. source

In the paper

“London [134] defends the ability to produce results and empirically verify their accuracy as more important than the ability to explain how such results are produced.”

What the sources say

• Not checked yetcites [134] London, 2019, Artificial intelligence and black-box medical decisions: Accuracy versus explainability, Hastings Cent. Rep. source

In the paper

“The authors agree with the human-computer interaction community that interdisciplinary cooperation and user-centered approaches to explainability are required to close the gaps between XAI and practices.”

In the paper

“Models should be built in collaboration with input from those with expertise from the fields of social and behavioral science, philosophy, psychology, and cognitive science [19,20,64].”

What the sources say

• Not checked yetcites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access source
• Not checked yetcites [20] Vellido, 2019, The importance of interpretability and visualization in machine learning for applications in medicine and health care, Neural Comput. Appl. source
• Not checked yetcites [64] Miller, T., Howe, P., and Sonenberg, L. (2017). Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv.

In the paper

“Additionally, Arrieta et al. [18] recommend that those black-box models should be selected only when necessary and, when possible, the use of interpretable or transparent by design algorithms should be prioritized over complex algorithms that require the application of post-hoc XAI techniques.”

What the sources say

• Not checked yetcites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion source

In the paper

“Additionally, ethics, fairness, and safety-related implications, as well as the cognitive skills and limitations of the audience must be considered when deciding what type of explanations should be provided [18].”

What the sources say

• Not checked yetcites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion source

In the paper

“However, explanations should be at least potentially actionable, parsimonious, and timely [69], warranting further research [19].”

What the sources say

• Not checked yetcites [69] Tonekaboni, S., Joshi, S., McCradden, M.D., and Goldenberg, A. (2019, January 13–14). What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use. Proceedings of the Machine Learning for Healthcare Conference, Boston, MA, USA.
• Not checked yetcites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access source

In the paper

“Additionally, we might surpass the problem of data availability and heterogeneity by using the least amount of data available and the easiest-to-collect information to initiate the development of the system [140].”

What the sources say

• Not checked yetcites [140] Antoniadi, A.M., Galvin, M., Heverin, M., Hardiman, O., and Mooney, C. (2021, January 22–26). Development of an explainable clinical decision support system for the prediction of patient quality of life in amyotrophic lateral sclerosis. Proceedings of the 36th Annual ACM Symposium on Applied Computing, Gwangju, Korea. source

In the paper

“Note that Sullivan [136] states that the opaqueness of models such as deep neural networks is not what is limiting our understanding, but rather the”

What the sources say

• Not checked yetcites [136] Sullivan, E. (2020). Understanding from machine learning models. Br. J. Philos. Sci.

In the paper

“Lipton [46] argues that the short-term goal of building trust with doctors by developing transparent models might clash with the longer-term goal of improving health care.”

What the sources say

• Not checked yetcites [46] Lipton, 2018, The mythos of model interpretability, Queue source

In the paper

“Developing ML-based CDSS is a multidisciplinary process that should include the needs of all stakeholders.”

In the paper

“Domain-specific needs must be taken into account including a thorough understanding of the purpose of the system, the performance and interpretability of existing systems, and the level and nature of the explanations that are required [18].”

What the sources say

• Not checked yetcites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion source

In the paper

“A lack of research in general is likely both a symptom and cause of this.”

In the paper

“Studies focusing on all stages of CDSS development are required to establish more firmly how explainability can be put into useful practice in this important context.”

In the paper

“Significant work lies ahead in order to integrate useful explainablity into CDSS.”

In the paper

“In order for Clinical Decision Support Systems (CDSS) to be used effectively in practice, they need to be trustworthy, easy to understand, and, most of all, positively augment the human decision-making process.”

In the paper

“Explainability allows developers to identify shortcomings in a system and allows clinicians to be confident in the decisions they make with CDSS assistance.”

In the paper

“Explainability is a critical component in achieving these goals.”