Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review

May combine two claims.

Lamy et al. designed an additional system that incorporated post-hoc explainability and provided local explanations of the preference model using rainbow boxes.

“An additional system that incorporated post-hoc explainability was designed by Lamy et al. [123] and provided local explanations of the preference model using rainbow boxes [56].”

• cites [123] Lamy, 2020, Explainable decision support through the learning and visualization of preferences from a formal ontology of antibiotic treatments, J. Biomed. Inform. sourceNot checked yet

  Not checked in the free run.

• cites [56] Lambrecht, 2019, Algorithmic bias? an empirical study of apparent gender-based discrimination in the display of stem career ads, Manag. Sci. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Militello et al. proposed a CDSS for epicardial fat volume quantification that used visualization representations to provide explanations and a user-centered graphical user interface design for safe interaction and integration into clinical workflow.

“Militello et al. [121] Epicardial fat volume A user-centred Graphical User Interface (GUI) design is used to allow for safe interaction of the physician as well as for an effective integration into the existing clinical workflow”

• cites [121] Militello, 2019, A semi-automatic approach for epicardial adipose tissue segmentation and quantification on cardiac CT scans, Comput. Biol. Med. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Previous works have evaluated XAI in AI-based CDSS, but only as a secondary aspect.

“Previous works have evaluated XAI in AI-based CDSS, but only as a secondary aspect [99,101,103].”

• cites [99] Jin, 2020, Artificial intelligence in glioma imaging: Challenges and advances, J. Neural Eng. sourceNot checked yet

  Not checked in the free run.

• cites [101] Rundo, 2020, Recent advances of HCI in decision-making tasks for optimized clinical workflows and precision medicine, J. Biomed. Inform. sourceNot checked yet

  Not checked in the free run.

• cites [103] Angehrn, 2020, Artificial Intelligence and Machine Learning Applied at the Point of Care, Front. Pharmacol. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Perturbation-based models that use model-agnostic explanations are common in the literature and have been used in two of the proposed CDSS.

“In terms of post-hoc explainability, perturbation-based models that use model-agnostic explanations are common in the literature [98,99,118,120,127] and have been used in two of the proposed CDSS [118,120].”

• cites [98] Zucco, C., Liang, H., Fatta, G.D., and Cannataro, M. (2019, January 3–6). Explainable Sentiment Analysis with Applications in Medicine. Proceedings of the 2018 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2018, Madrid, Spain. sourceNot checked yet

  Not checked in the free run.

• cites [99] Jin, 2020, Artificial intelligence in glioma imaging: Challenges and advances, J. Neural Eng. sourceNot checked yet

  Not checked in the free run.

• cites [118] Wang, D., Yang, Q., Abdul, A., and Lim, B.Y. (2019, January 4–9). Designing theory-driven user-centric explainable AI. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Scotland, UK. sourceNot checked yet

  Not checked in the free run.

• cites [120] Hu, 2020, Using a machine learning approach to predict mortality in critically ill influenza patients: A cross-sectional retrospective multicentre study in Taiwan, BMJ Open sourceNot checked yet

  Not checked in the free run.

• cites [127] Cai, 2019, “Hello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making, Proceedings of the ACM on Human-Computer Interaction sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The 33 papers were separated into eight literature reviews (24.2%), nine papers discussing aspects of XAI in CDSS (27.3%), four papers discussing implementation of a CDSS without XAI (12.1%), and 12 papers describing implementation of a CDSS with XAI (36.4%).

“Finally, we separated the 33 papers into eight literature reviews (24.2%) [97][98][99][100][101][102][103][104], nine papers discussing aspects of the use of XAI in CDSS (27.3%) [48,[105][106][107][108][109][110][111][112], four papers that discussed the implementation of a CDSS without XAI (12.1%) [113-116], and 12 papers describing the implementation of a CDSS with (36.4%)XAI [117-128]”

May combine two claims.

After examining search results by title, abstract, and full-text to remove out-of-scope papers, the set was reduced to 121 papers (39 conference and 82 journal).

“This reduced our set to 121 papers (39 conference and 82 journal).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The six searches had a minimum of 35 and a maximum of 181 results each, with a total of 668.

“The six searches had a minimum of 35 and a maximum of 181 results each, with a total of 668.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Three of the 76 papers were not available on any platform at our disposal, and one additional paper was removed because it was a pre-print using an ACM TOIIS template but was not published in the ACM Digital Library.

“Three of these were not available on any platform at our disposal, and one additional paper was removed when we discovered that it was a pre-print using an ACM TOIIS template but was not published in the ACM Digital Library.”

Classified as probably not a substantial claim. May combine two claims.

Papers that were not peer-reviewed conference or journal papers and 10 papers not written in English were eliminated, leaving 132 papers.

“We eliminated papers that were not peer-reviewed conference or journal papers (e.g., theses, dissertations, books, book chapters, pre-prints, or other archived articles and posters) and 10 papers that were not written in English, leaving 132 papers.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Reconciliation resulted in seven disputed papers being excluded and 16 retained, leaving 33 papers.

“Reconciliation resulted in seven disputed papers being excluded and 16 retained, leaving 33 papers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

After both authors independently classified each paper, 16 papers were removed by agreement.

“After both authors had independently classified each paper, 16 papers were removed by agreement.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The author pairs disagreed on 23 papers, giving a 68% agreement rate on paper inclusion/exclusion.

“The author pairs disagreed on 23 papers giving a 68% agreement rate on paper inclusion/exclusion.”

May combine two claims.

The 33 papers were published between 2008 and 24 July 2020: one in 2008, three in 2018, 13 in 2019, and 16 in 2020.

“These papers were published between 2008 and 24 July 2020: one paper was published in 2008, three papers were published in 2018, 13 papers were published in 2019, and 16 papers were published in 2020 (until the 24 July).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Kunapuli et al. indicated that XAI could support specific rational reasoning processes, enabling CDSS to support their decisions with understandable interpretations to users with/without ML expertise.

“Kunapuli et al. [128] indicated that XAI could support specific rational reasoning processes, enabling CDSS to support their decisions with understandable interpretations to users with/without ML expertise.”

• cites [128] Kunapuli, 2018, A Decision-Support Tool for Renal Mass Classification, J. Digit. Imaging sourceNot checked yet

  Not checked in the free run.

May combine two claims.

In Blanco et al., the CDSS was developed using a bidirectional gated recurrent unit with attention mechanism, allowing exploration of how much each fragment of the text contributed to a prediction, thus providing local explanations.

“In the study by Blanco et al. [122], the CDSS was developed using a bidirectional gated recurrent unit (BiGRU) with attention mechanism, which allowed for the exploration of how much each fragment of the text contributed to a prediction, thus providing local explanations.”

• cites [122] Blanco, 2020, Extracting Cause of Death from Verbal Autopsy with Deep Learning interpretable methods, IEEE J. Biomed. Health Inform. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In some cases, users believed that the interpretation of AI decisions might alleviate their own decision-making biases.

“In some cases, users also believed that the interpretation of AI decisions might alleviate their own decision-making biases.”

• cites [111] Liao, Q.V., Gruen, D., and Miller, S. (2020, January 25–30). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Lamy et al. presented their interface to 11 medical experts for usability and acceptability validation, demonstrating that XAI could provide the user with a good indication of the confidence level of their choice.

“They presented their interface to 11 medical experts for usability and acceptability validation, demonstrating that XAI could provide the user with a good indication of the confidence level of their choice [126].”

• cites [126] Lamy, 2019, Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach, Artif. Intell. Med. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Wang et al. identified that XAI could support different explanation types by articulating how people understand events or observations through explanations and can be leveraged to mitigate decision biases and cognitive biases.

“Wang et al. [118] identified that XAI could support different explanation types by articulating how people understand events or observations through explanations and can be leveraged to mitigate decision biases and cognitive biases [98,103,112].”

• cites [118] Wang, D., Yang, Q., Abdul, A., and Lim, B.Y. (2019, January 4–9). Designing theory-driven user-centric explainable AI. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Scotland, UK. sourceNot checked yet

  Not checked in the free run.

• cites [98] Zucco, C., Liang, H., Fatta, G.D., and Cannataro, M. (2019, January 3–6). Explainable Sentiment Analysis with Applications in Medicine. Proceedings of the 2018 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2018, Madrid, Spain. sourceNot checked yet

  Not checked in the free run.

• cites [103] Angehrn, 2020, Artificial Intelligence and Machine Learning Applied at the Point of Care, Front. Pharmacol. sourceNot checked yet

  Not checked in the free run.

• cites [112] Johnson, 2019, AI, Machine Learning, and Ethics in Health Care, J. Leg. Med. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The results of Liao et al. showed that XAI could gain further insights or evidence, and thus enhance decision confidence or generate the hypothesis about causality.

“The results showed that XAI could gain further insights or evidence, and thus enhance decision confidence or generate the hypothesis about causality.”

• cites [111] Liao, Q.V., Gruen, D., and Miller, S. (2020, January 25–30). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Kunapuli et al. built a CDSS using Relational Functional Gradient Boosting, a statistical relational learning method which attributes its explainability to the usage of tree models and the provision of explanations in terms of features of interest.

“Kunapuli et al. [128] built a CDSS using Relational Functional Gradient Boosting (RFGB), a statistical relational learning method which attributes its explainability to the usage of tree models and the provision of explanations in terms of features of interest.”

• cites [128] Kunapuli, 2018, A Decision-Support Tool for Renal Mass Classification, J. Digit. Imaging sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Vorm reported that XAI could provide different information types to make intelligent systems explainable and more acceptable and trustworthy to users.

“They reported that XAI could provide different information types to make intelligent systems explainable and more acceptable and trustworthy to users.”

• cites [106] Vorm, E.S. (2018, January 3–5). Assessing Demand for Transparency in Intelligent Systems Using Machine Learning. Proceedings of the 2018 Innovations in Intelligent Systems and Applications (INISTA), Thessaloniki, Greece. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Liao et al. interviewed 20 participants to identify gaps between current XAI algorithmic work and practices to create explainable AI products.

“They interviewed 20 participants to identify gaps between the current XAI algorithmic work and practices to create explainable AI products.”

• cites [111] Liao, Q.V., Gruen, D., and Miller, S. (2020, January 25–30). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Case-Based Reasoning is an intrinsically explainable method that was used by Lamy et al. for their CDSS for breast cancer, supported by visual explanations in the form of rainbow boxes and a polar multidimensional scaling scatter plot.

“Case-Based Reasoning is an intrinsically explainable method that was used by Lamy et al. [126] for their CDSS for breast cancer, which was also supported by visual explanations in the form of rainbow boxes and a polar multidimensional scaling scatter plot.”

• cites [126] Lamy, 2019, Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach, Artif. Intell. Med. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Rule-based systems were developed in two studies to provide local explanations, as they were considered closer to human reasoning and thus more preferable by clinicians.

“Rule-based systems were developed in two studies to provide local explanations, as they were considered closer to human reasoning, and thus more preferable by clinicians [124,125].”

• cites [124] Tan, T.Z., Ng, G.S., and Quek, C. (2008, January 1–6). Improving tractability of Clinical Decision Support system. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China. sourceNot checked yet

  Not checked in the free run.

• cites [125] Alonso, 2018, An Ontology-Based Interpretable Fuzzy Decision Support System for Diabetes Diagnosis, IEEE Access sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Sadeghi et al. proposed a CDSS that used a Random Forest to predict the outcome, and the authors stated that the use of time-domain features leads to a transparent and explainable CDSS, but there is not sufficient information towards this claim.

“Sadeghi et al. [117] proposed a CDSS that used a Random Forest to predict the outcome.The authors stated that the use of time-domain features leads to a transparent and explainable CDSS, but there is not sufficient information towards this claim.”

• cites [117] Sadeghi, 2019, Sleep quality prediction in caregivers using physiological signals, Comput. Biol. Med. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Wang et al. believe that what constitutes a good explanation should draw from social science instead of depending on researchers' intuition, and justification is required for choosing different explanation types or representations.

“Moreover, Wang et al. [118] believe that what constitutes a good explanation should draw from social science instead of depending on researchers' intuition, and justification is required for choosing different explanation types or representations.”

• cites [118] Wang, D., Yang, Q., Abdul, A., and Lim, B.Y. (2019, January 4–9). Designing theory-driven user-centric explainable AI. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Scotland, UK. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

XAI could help the user understand when to trust a model and why an error may occur.

“XAI could help the user understand when to trust a model and why an error may occur [97,116].”

• cites [97] Luz, 2020, Machine learning in infection management using routine electronic health records: Tools, techniques, and reporting of future technologies, Clin. Microbiol. Infect. sourceNot checked yet

  Not checked in the free run.

• cites [116] Xie, Y., Chen, M., Kao, D., Gao, G., and Chen, X.A. (2020, January 25–30). CheXplain: Enabling Physicians to Explore and Understand Data-Driven, AI-Enabled Medical Imaging Analysis. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Luz et al. argue that a thorough reasoning is required for choosing between transparent and black-box ML algorithms, because post-hoc interpretation methods that develop a mirror model of the original one to add explainability could provide an inaccurate representation of the original model.

“Furthermore, Luz et al. [97] argue that a thorough reasoning is required for choosing between transparent and black-box ML algorithms, because post-hoc interpretation methods that develop a mirror model of the original one to add explainability could provide an inaccurate representation of the original model.”

• cites [97] Luz, 2020, Machine learning in infection management using routine electronic health records: Tools, techniques, and reporting of future technologies, Clin. Microbiol. Infect. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The concept of interpretability is often highly subjective.

“Furthermore, the concept of interpretability is often highly subjective [115].”

• cites [115] Gangavarapu, T., S Krishnan, G., Kamath S, S., and Jeganathan, J. (2020). FarSight: Long-Term Disease Prediction Using Unstructured Clinical Nursing Notes. IEEE Trans. Emerg. Top. Comput., 1–16.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Richard et al. proposed a definition stating that a transparent classification system should be understandable, use an interpretable type of classifier and learning system, produce traceable results, and use a revisable classifier.

“Richard et al. [48] proposed a definition stating that a transparent classification system should be understandable, use an interpretable type of classifier and learning system, produce traceable results, and use a revisable classifier.”

• cites [48] Richard, 2020, Transparency of Classification Systems for Clinical Decision Support, Information Processing and Management of Uncertainty in Knowledge-Based SystemsNot checked yet

  Not checked in the free run.

May combine two claims.

There is no universal definition of explainability nor a well-recognized equivalence or distinction between 'interpretable' and 'explainable' ML.

“There is no universal definition of explainability [48,98] nor a well-recognized equivalence or distinction between”

• cites [48] Richard, 2020, Transparency of Classification Systems for Clinical Decision Support, Information Processing and Management of Uncertainty in Knowledge-Based SystemsNot checked yet

  Not checked in the free run.

• cites [98] Zucco, C., Liang, H., Fatta, G.D., and Cannataro, M. (2019, January 3–6). Explainable Sentiment Analysis with Applications in Medicine. Proceedings of the 2018 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2018, Madrid, Spain. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Jin et al. reported that there is a lack of evaluation of XAI techniques on glioma imaging due to lack of focus on practical challenges relating to clinical implementation of XAI.

“Moreover, Jin et al. [99] reported that there is a lack of evaluation of XAI techniques on glioma imaging due to lack of focus on practical challenges relating to clinical implementation of XAI.”

• cites [99] Jin, 2020, Artificial intelligence in glioma imaging: Challenges and advances, J. Neural Eng. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Lamy et al. raised the need to confirm their results on a larger user study, in addition to the lack of user studies for some XAI-based approaches.

“As to testing the systems, Lamy et al. [126] raised the need to confirm their results on a larger user study, in addition to the lack of user studies for some XAI-based approaches as discussed in Section 4.3 (RQ3).”

• cites [126] Lamy, 2019, Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach, Artif. Intell. Med. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Identifying the motivation of explainability helps to select XAI techniques, foresee their limitations, and fill in the gaps occurring while designing user experiences.

“They also discovered that identifying the motivation of explainability helps to select XAI techniques, foresee their limitations, and fill in the gaps occurring while designing user experiences.”

• cites [111] Liao, Q.V., Gruen, D., and Miller, S. (2020, January 25–30). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

There are some studies supporting that XAI is not always necessary.

“On the other hand, there are some studies supporting that XAI is not always necessary [46,134,135].”

• cites [46] Lipton, 2018, The mythos of model interpretability, Queue sourceNot checked yet

  Not checked in the free run.

• cites [134] London, 2019, Artificial intelligence and black-box medical decisions: Accuracy versus explainability, Hastings Cent. Rep. sourceNot checked yet

  Not checked in the free run.

• cites [135] Baldi, 2018, Deep learning in biomedical data science, Annu. Rev. Biomed. Data Sci. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Currently, most of the decisions made by AI-based CDSS cannot be interpreted in a transparent way, potentially limiting the uptake, trust, and usability of these systems in practice.

“Currently, most of the decisions made by AI-based CDSS cannot be interpreted in a transparent way potentially limiting the uptake, trust, and usability of these systems in practice [20,133].”

• cites [20] Vellido, 2019, The importance of interpretability and visualization in machine learning for applications in medicine and health care, Neural Comput. Appl. sourceNot checked yet

  Not checked in the free run.

• cites [133] Gomolin, 2020, Artificial intelligence applications in dermatology: Where do we stand?, Front. Med. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Linardatos et al. identified four main areas of focus: methods for explaining complex black-box models, methods for creating white-box models, methods that promote fairness and restrict discrimination, and methods for analyzing sensitivity of model predictions.

“A rapid advancement of XAI is evident, and the recent study by Linardatos et al. [132] has identified four main areas of focus: methods for explaining complex black-box models, methods for creating white-box models, methods that promote fairness and restrict the existence of discrimination, and methods for analyzing sensitivity of model predictions.”

• cites [132] Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. (2021). Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy, 23. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Tan et al. argue that a CDSS should have high tractability, which requires human like reasoning, step by step inference, explanation capacity, and user-familiar terms, to gain user acceptance.

“Tan et al. [124] argue that a CDSS should have high tractability, which requires human like reasoning, step by step inference, explanation capacity, and user-familiar terms, to gain user acceptance.”

• cites [124] Tan, T.Z., Ng, G.S., and Quek, C. (2008, January 1–6). Improving tractability of Clinical Decision Support system. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China. sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Metrics to evaluate the performance of XAI techniques require further study.

“Metrics to evaluate the performance of XAI techniques require further study [18].”

• cites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Zhou et al. conclude their survey by stating that the evaluation of ML explanations is a multidisciplinary research topic and it is not possible to define an implementation of evaluation metrics that can be applied to all explanation methods.

“They conclude their survey by stating that”

• cites [141] Zhou, J., Gandomi, A.H., Chen, F., and Holzinger, A. (2021). Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Although XAI can assist with identifying issues with the data, the problem with unstructured medical data remains a challenge for the development of usable AI-based systems.

“Although XAI can assist with identifying issues with the data, the problem with unstructured medical data remains a challenge for the development of usable AI-based systems.”

May combine two claims.

Bussone et al. found that giving clinicians a fuller explanation of the facts that led to the system's proposed diagnosis had a positive effect on trust but caused over-reliance issues.

“Bussone et al. [75] found that giving clinicians a fuller explanation of the facts that led to the system's proposed diagnosis had a positive effect on trust but caused over-reliance issues.”

• cites [75] Bussone, A., Stumpf, S., and O’Sullivan, D. (2015, January 21–23). The role of explanations on trust and reliance in clinical decision support systems. Proceedings of the 2015 International Conference on Healthcare Informatics, Dallas, TX, USA. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The 'right' level of explainability required depends on many factors and is context and resource specific.

“The”

• cites [138] Carvalho, D.V., Pereira, E.M., and Cardoso, J.S. (2019). Machine learning interpretability: A survey on methods and metrics. Electronics, 8. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

There is a need for more robust user studies.

“Finally, there is a need for more robust user studies [66,143].”

• cites [66] Doshi-Velez, F., and Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv.Not checked yet

  Not checked in the free run.

• cites [143] Kenny, 2021, Explaining Black-Box classifiers using Post-Hoc explanations-by-example: The effect of explanations and error-rates in XAI user studies, Artif. Intell. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Subjective measurements can provide valuable insight into the user's experience, however, there is an overall lack of validated and reliable evaluation metrics.

“Subjective measurements can provide valuable insight into the user's experience, however, there is an overall lack of validated and reliable evaluation metrics.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Holzinger et al. introduced the 'System Causability Scale' as means to measure explanation quality.

“Holzinger et al. [142] introduced the”

• cites [142] Holzinger, 2020, Measuring the quality of explanations: The system causability scale (SCS), KI-Künstliche Intell. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Bruckert et al. shed light on the difficulties that are presented when rendering ML models explainable for healthcare purposes, highlighting that the implementation of such systems requires overlap between different disciplines and professions.

“Additionally, Bruckert et al. [137] shed light on the difficulties that are presented when rendering ML models explainable for healthcare purposes, highlighting that the implementation of such systems requires overlap between different disciplines and professions.”

• cites [137] Bruckert, 2020, The Next Generation of Medical Decision Support: A Roadmap Toward Transparent Expert Companions, Front. Artif. Intell. sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Zhou et al. found that some properties (clarity, broadness, and completeness) are still in shortage of appropriate metrics, and so is the class of explanations that are example-based.

“They found that some properties (clarity, broadness, and completeness) are still in shortage of appropriate metrics, and so is the class of explanations that are example-based.”

• cites [141] Zhou, J., Gandomi, A.H., Chen, F., and Holzinger, A. (2021). Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

There are very few studies like this that give insight into what clinicians want or need.

“However, there are very few studies like this that give insight into what clinicians want or need.”

Classified as probably not a substantial claim.

A main challenge remains the selection of methods used to present explanations in an informative and efficient-and therefore clinically useful-manner.

“A main challenge remains the selection of methods used to present explanations in an informative and efficient-and therefore clinically useful-manner.”

May combine two claims.

Through a case study, Jacobs et al. found that incorrect ML recommendations may affect clinicians and lower the accuracy of decisions, while explanations were found insufficient for addressing over-reliance on a model that suggests erroneous decisions.

“Through a case study, Jacobs et al. [144] found that incorrect ML recommendations may affect clinicians and lower the accuracy of decisions, while explanations were found insufficient for addressing over-reliance on a model that suggests erroneous decisions.”

• cites [144] Jacobs, 2021, How machine-learning recommendations influence clinician treatment selections: The example of the antidepressant selection, Transl. Psychiatry sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Blanco et al. [122] used a BiGRU with attention mechanism to show the contribution of each fragment of text to the prediction; the method was specific, ante-hoc, and local.

“Blanco et al. [122] BiGRU with attention mechanism to show the contribution of each fragment of text to the prediction specific ante-hoc local”

• cites [122] Blanco, 2020, Extracting Cause of Death from Verbal Autopsy with Deep Learning interpretable methods, IEEE J. Biomed. Health Inform. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Jacobs et al. found that explanation strategies ought to be selected according to the clinicians' prior experience with ML and that those with prior experience perceived a higher utility from the ML recommendations.

“They found that explanation strategies ought to be selected according to the clinicians' prior experience with ML and that those with prior experience perceived a higher utility from the ML recommendations.”

• cites [144] Jacobs, 2021, How machine-learning recommendations influence clinician treatment selections: The example of the antidepressant selection, Transl. Psychiatry sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Less detailed explanations had the opposite effect, as this made participants question the system's reliability and caused self-reliance issues.

“On the other hand, less detailed explanations had the opposite effect, as this made participants question the system's reliability and caused self-reliance issues [75].”

• cites [75] Bussone, A., Stumpf, S., and O’Sullivan, D. (2015, January 21–23). The role of explanations on trust and reliance in clinical decision support systems. Proceedings of the 2015 International Conference on Healthcare Informatics, Dallas, TX, USA. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Tan et al. [124] used CLFNN, which autonomously generates fuzzy rules to provide human-like reasoning; the method was specific, ante-hoc, and local.

“Tan et al. [124] CLFNN, which autonomously generates fuzzy rules to provide human-like reasoning specific ante-hoc local”

• cites [124] Tan, T.Z., Ng, G.S., and Quek, C. (2008, January 1–6). Improving tractability of Clinical Decision Support system. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

While there are many studies on XAI in medicine, there is a limited number that focus on the context of CDSS.

“While there are many studies on XAI in medicine, there is a limited number that focus on the context of CDSS.”

Classified as probably not a substantial claim.

There is little discussion on the impact of XAI on patients from the patients perspective.

“Similarly, there is little discussion on the impact of XAI on patients from the patients perspective.”

May combine two claims.

Despite some user studies reporting positive views on CDSS, especially in light of explainability, there is still skepticism around their use in practice.

“However, despite some user studies reporting positive views on CDSS, especially in light of explainability, there is still skepticism around their use in practice.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Hu et al. [120] used SHAP for summary plot and partial dependence plot; the method was model-agnostic, post-hoc, and global.

“Hu et al. [120] SHAP [79] for summary plot and partial dependence plot agnostic post-hoc global”

• cites [120] Hu, 2020, Using a machine learning approach to predict mortality in critically ill influenza patients: A cross-sectional retrospective multicentre study in Taiwan, BMJ Open sourceNot checked yet

  Not checked in the free run.

• cites [79] Lundberg, S.M., and Lee, S.I. (2017, January 4–9). A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Lamy et al. [123] visualised the created preference model using rainbow boxes; the method was agnostic, post-hoc, and local.

“Lamy et al. [123] Visualised the created preference model using rainbow boxes [131] agnostic post-hoc local”

• cites [123] Lamy, 2020, Explainable decision support through the learning and visualization of preferences from a formal ontology of antibiotic treatments, J. Biomed. Inform. sourceNot checked yet

  Not checked in the free run.

• cites [131] Lamy, 2017, Rainbow boxes: A new technique for overlapping set visualization and two applications in the biomedical domain, J. Vis. Lang. Comput. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Lamy et al. [126] used a visual case-based reasoning approach (using rainbow-boxes and a polar multidimensional scaling scatter plot); the method was specific, ante-hoc, and local.

“Lamy et al. [126] Visual (using rainbow-boxes [131] and a polar multidimensional scaling scatter plot) case-based reasoning approach specific ante-hoc local”

• cites [126] Lamy, 2019, Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach, Artif. Intell. Med. sourceNot checked yet

  Not checked in the free run.

• cites [131] Lamy, 2017, Rainbow boxes: A new technique for overlapping set visualization and two applications in the biomedical domain, J. Vis. Lang. Comput. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

El-Sappagh et al. [125] used a semantically interpretable FRBS with the integration of semantic ontology-based reasoning; the method was specific, ante-hoc, and local.

“El-Sappagh et al. [125] Semantically interpretable FRBS with the integration of semantic ontology-based reasoning specific ante-hoc local”

• cites [125] Alonso, 2018, An Ontology-Based Interpretable Fuzzy Decision Support System for Diabetes Diagnosis, IEEE Access sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S1 (asserted). May combine two claims.

ML-based systems are unlikely to be adopted in routine medical practice beyond a limited number of niche applications unless challenges are addressed, most likely by having systems provide satisfactory explanations for their decisions.

“It has been argued that despite the existence of much evidence supporting their usefulness, ML-based systems are unlikely to be adopted in routine medical practice beyond a limited number of niche applications unless these challenges are addressed, most likely by having systems provide satisfactory explanations for their decisions [20,21].”

• cites [20] Vellido, 2019, The importance of interpretability and visualization in machine learning for applications in medicine and health care, Neural Comput. Appl. sourceNot checked yet

  Not checked in the free run.

• cites [21] Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., and Kagal, L. (2018, January 1–3). Explaining explanations: An overview of interpretability of machine learning. Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Incorporating XAI principles into CDSS is essential if the potential beneficial impact of ML-based CDSS is to be fulfilled.

“As a result, incorporating XAI principles into CDSS is essential if the potential beneficial impact is to be fulfilled.”

May combine two claims.

Reported benefits of XAI ultimately lead to increased trustworthiness and acceptability of the system and potential for its incorporation in the clinical workflow.

“Studies reported benefits of the use of XAI such as the fact that it could enhance decision confidence for clinicians, or generate the hypothesis about causality, which ultimately leads to increased trustworthiness and acceptability of the system and potential for its incorporation in the clinical workflow.”

May combine two claims.

Different applications usually have different interpretability and explainability needs, which works against generalizable solutions that span a number of situations.

“Unfortunately, secondary factors also complicate solutions, for instance, different applications usually have different interpretability and explainability needs [20] working against generalizable solutions that span a number of situations.”

• cites [20] Vellido, 2019, The importance of interpretability and visualization in machine learning for applications in medicine and health care, Neural Comput. Appl. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Giving explanations can increase understanding of and trust in a system.

“Although, giving explanations can increase understanding of and trust in a system [19,46,64], simple explanations can hide undesirable attributes of the system and may mislead users into coming to dangerous or unfounded conclusions that could ultimately be unethical [21].”

• cites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access sourceNot checked yet

  Not checked in the free run.

• cites [46] Lipton, 2018, The mythos of model interpretability, Queue sourceNot checked yet

  Not checked in the free run.

• cites [64] Miller, T., Howe, P., and Sonenberg, L. (2017). Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

There is a recognized need to go beyond causality/correlation to true "causability".

“Additionally, XAI can help to shed some light onto causality [18,46,67] although, there is a recognized need to go beyond causality/correlation to true “causability” [3].”

• cites [3] Holzinger, 2019, Causability and explainability of artificial intelligence in medicine, Wiley Interdiscip. Rev. Data Min. Knowl. Discov. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Translating ML models effectively to clinical practice requires establishing clinicians' trust in the system.

“Translating ML models effectively to clinical practice requires establishing clinicians' trust in the system.”

May combine two claims.

XAI can be used to help evaluate if predictions are biased and defend algorithmic decisions as being fair and ethical.

“XAI can be used to help us evaluate if predictions are biased and defend algorithmic decisions as being fair and ethical [18,19,46,66].”

• cites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion sourceNot checked yet

  Not checked in the free run.

• cites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access sourceNot checked yet

  Not checked in the free run.

• cites [46] Lipton, 2018, The mythos of model interpretability, Queue sourceNot checked yet

  Not checked in the free run.

• cites [66] Doshi-Velez, F., and Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv.Not checked yet

  Not checked in the free run.

May combine two claims.

Simple explanations can hide undesirable attributes of the system and may mislead users into coming to dangerous or unfounded conclusions that could ultimately be unethical.

“Although, giving explanations can increase understanding of and trust in a system [19,46,64], simple explanations can hide undesirable attributes of the system and may mislead users into coming to dangerous or unfounded conclusions that could ultimately be unethical [21].”

• cites [21] Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., and Kagal, L. (2018, January 1–3). Explaining explanations: An overview of interpretability of machine learning. Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy. sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The review followed a structured process involving specifying research questions, conducting searches, selecting studies, filtering, extracting data, and synthesizing results.

“In this review, we followed a structured process that involved the following: 1. Specifying research questions 2. Conducting searches of specified databases 3. Selecting studies by criterion 4. Filtering studies by evaluating their pertinence 5. Extracting data 6.Synthesizing results.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The first requirement of AI systems in medicine is clinical validation, while explainability is a second aspect.

“The authors also consider the fact that the first requirement of AI systems in medicine is clinical validation, while explainability is a second aspect.”

• cites Amann et al. [93] Amann, J., Blasimme, A., Vayena, E., Frey, D., and Madai, V.I. (2020). Explainability for artificial intelligence in healthcare: A multidisciplinary perspective. BMC Med. Inform. Decis. Mak., 20. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Providing explanations for how recommendations are arrived at will allow practitioners to make more nuanced, and in some cases, life-saving decisions.

“Providing explanations for how recommendations are arrived at will allow practitioners to make more nuanced, and in some cases, life-saving decisions.”

• AssumptionRests on an assumption: CDSS in the absence of explainability may lead to issues of under- or overreliance.

Classified as probably not a substantial claim.

The model's overall accuracy was not sufficient on its own to allow clinicians to make an informed decision.

“Tonekaboni et al. [69] found that the model's overall accuracy was not sufficient on its own to allow clinicians to make an informed decision, clinicians wanted to know the subset of features driving a prediction to allow them to compare the model decision to their clinical judgment.”

• cites Tonekaboni et al. [69] Tonekaboni, S., Joshi, S., McCradden, M.D., and Goldenberg, A. (2019, January 13–14). What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use. Proceedings of the Machine Learning for Healthcare Conference, Boston, MA, USA.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Post-hoc methods could lead to the generation of classifiers whose post-hoc explanations could be arbitrarily controlled.

“There are also concerns that post-hoc methods could expose the original models to adversarial attacks [18] or could lead to the generation of classifiers whose post-hoc explanations could be arbitrarily controlled [63].”

• cites [63] Slack, D., Hilgard, S., Jia, E., Singh, S., and Lakkaraju, H. (2020, January 7–8). Fooling lime and shap: Adversarial attacks on post hoc explanation methods. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The review adapted the guidelines proposed by Kitchenham.

“We conducted a systematic literature review by adapting the guidelines proposed by Kitchenham [94].”

• cites [94] Kitchenham, B.A., and Charters, S. (2020, July 24). Guidelines for Performing Systematic Literature Reviews in Software Engineering; Technical Report EBSE 2007-001, Keele University and Durham University Joint Report. Available online: http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=2BE22FED09591B99D6A7ACF8FE2258D5?.Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The need for XAI in CDSS is amplified by the need for ethical and fair decision-making.

“The need for XAI in CDSS, and the medical field in general, is amplified by the need for ethical and fair decision-making and the fact that AI trained with historical data can be a reinforcement agent of historical actions and biases that should be uncovered.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Six searches (S1-S6) were performed using Google Scholar on 24 July 2020.

“After some trial and error with a range of terms, we performed the following six searches (S1-S6).The search terms were used to search Google Scholar (currently the most comprehensive academic search engine according to recent studies [95,96]) on 24 July 2020”

• cites [95] Thelwall, 2018, Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories, J. Inf.Not checked yet

  Not checked in the free run.

• cites [96] Gusenbauer, 2019, Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases, Scientometrics sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Medical certification comes after the system is compliant with regulatory standards and prediction accuracy is usually the main measurement of clinical validation.

“Medical certification comes after the system is compliant with regulatory standards and prediction accuracy is usually the main measurement of clinical validation.”

• cites Amann et al. [93] Amann, J., Blasimme, A., Vayena, E., Frey, D., and Madai, V.I. (2020). Explainability for artificial intelligence in healthcare: A multidisciplinary perspective. BMC Med. Inform. Decis. Mak., 20. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

More complex models may not be more accurate.

“Likewise, more complex models may not be more accurate.”

Classified as probably not a substantial claim.

There is often a perceived trade-off between the performance (predictive accuracy) of a model and explainability.

“There is often a perceived trade-off between the performance (predictive accuracy) of a model and explainability [3,18,19].”

• cites [3] Holzinger, 2019, Causability and explainability of artificial intelligence in medicine, Wiley Interdiscip. Rev. Data Min. Knowl. Discov. sourceNot checked yet

  Not checked in the free run.

• cites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion sourceNot checked yet

  Not checked in the free run.

• cites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Clinicians wanted better explanations from the CDSS to help them interpret the system's confidence, to verify that the clinical disorder fit the CDSS suggestion, to better understand the reasoning chain of the system, and to make different diagnoses in order to help them make an assessment of the reliability of the system's decisions.

“Bussone et al. [75] found that clinicians wanted better explanations from the CDSS to help them interpret the system's confidence, to verify that the clinical disorder fit the CDSS suggestion, to better understand the reasoning chain of the system, and to make different diagnoses in order to help them make an assessment of the reliability of the system's decisions.”

• cites Bussone et al. [75] Bussone, A., Stumpf, S., and O’Sullivan, D. (2015, January 21–23). The role of explanations on trust and reliance in clinical decision support systems. Proceedings of the 2015 International Conference on Healthcare Informatics, Dallas, TX, USA. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Post-hoc methods could expose the original models to adversarial attacks.

“There are also concerns that post-hoc methods could expose the original models to adversarial attacks [18] or could lead to the generation of classifiers whose post-hoc explanations could be arbitrarily controlled [63].”

• cites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Given the need for transparency and explainability in medical practice and the growing use of CDSS leveraging AI, XAI has started to be incorporated in recent AI-based CDSS.

“However, given the undeniable need for transparency and explainability in medical practice and the growing use of CDSS leveraging AI, XAI has started to be incorporated in recent AI-based CDSS.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The remaining 72 papers were divided randomly into three groups and shared between three pairs of authors, with each pair classifying papers as include or exclude based on inclusion criteria.

“The remaining 72 papers were divided randomly into three groups and shared between three pairs of authors.Each pair took one group of papers and classified them as either include or exclude based on inclusion criteria.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A ratio of n unique /n total = 0.39 is indicative of a cohesive set of searches that together have a desired degree of internal consistency.

“A ratio of n unique /n total = 0.39 is indicative of a cohesive set of searches that together have a desired degree of internal consistency.”

May combine two claims.

There is an upward trend in the number of relevant studies published over time, indicating increased interest in XAI-enabled CDSS.

“We see an upward trend in the number of relevant studies published over time which indicated the increased interest in XAI-enabled CDSS.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Even though some other XAI-based approaches have not yet been tested on users, the benefits of the XAI presented in these works still seem likely to be useful in practice.

“Even though some other XAI-based approaches have not yet been tested on users, the benefits of the XAI presented in these works still seem likely to be useful in practice.”

May combine two claims.

Cai et al.'s findings suggest that AI transparency in collaborative decision making could allow experts to integrate AI assistants into daily practice and gain a richer understanding of the key issues they find.

“This suggests that AI transparency in collaborative decision making could allow experts to integrate AI assistants into daily practice and gain a richer understanding of the key issues they find.”

• cites [127] Cai, 2019, “Hello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making, Proceedings of the ACM on Human-Computer Interaction sourceNot checked yet

  Not checked in the free run.

May combine two claims.

XAI could greatly enhance the effectiveness of decision-support and clinician confidence, especially when high-stakes decisions are being made, which is the key factor for the success of the model in the practical use stage.

“XAI therefore could greatly enhance the effectiveness of decision-support and clinician confidence [128] especially when high-stakes decisions are being made [127], which is the key factor for the success of the model in the practical use stage [122,124].”

• cites [128] Kunapuli, 2018, A Decision-Support Tool for Renal Mass Classification, J. Digit. Imaging sourceNot checked yet

  Not checked in the free run.

• cites [127] Cai, 2019, “Hello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making, Proceedings of the ACM on Human-Computer Interaction sourceNot checked yet

  Not checked in the free run.

• cites [122] Blanco, 2020, Extracting Cause of Death from Verbal Autopsy with Deep Learning interpretable methods, IEEE J. Biomed. Health Inform. sourceNot checked yet

  Not checked in the free run.

• cites [124] Tan, T.Z., Ng, G.S., and Quek, C. (2008, January 1–6). Improving tractability of Clinical Decision Support system. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Explainability of the support system can improve the acceptability of CDSS by clinicians, increase the chances of the complex AI systems' adoption and clinical feasibility of a novel CDSS.

“Therefore, explainability of the support system can improve the acceptability of CDSS by clinicians [48] increase the chances of the complex AI systems' adoption and clinical feasibility of a novel CDSS [105,121].”

• cites [48] Richard, 2020, Transparency of Classification Systems for Clinical Decision Support, Information Processing and Management of Uncertainty in Knowledge-Based SystemsNot checked yet

  Not checked in the free run.

• cites [105] Mahadevaiah, 2020, Artificial intelligence-based clinical decision support in modern medical physics: Selection, acceptance, commissioning, and quality assurance, Med. Phys. sourceNot checked yet

  Not checked in the free run.

• cites [121] Militello, 2019, A semi-automatic approach for epicardial adipose tissue segmentation and quantification on cardiac CT scans, Comput. Biol. Med. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

In 2016 the European Union passed the GDPR which has been interpreted as a requirement for any decision made based on an algorithm to be explainable to the user.

“As discussed in Section 2.2.1, in 2016 the European Union passed the GDPR which has been interpreted as a requirement for any decision made based on an algorithm to be explainable to the user [106].”

• cites [106] Vorm, E.S. (2018, January 3–5). Assessing Demand for Transparency in Intelligent Systems Using Machine Learning. Proceedings of the 2018 Innovations in Intelligent Systems and Applications (INISTA), Thessaloniki, Greece. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

XAI may lead to greater uptake and use of CDSS, and may become a requirement in the future due to societal, regulatory, and ethical pressures.

“Nevertheless, XAI may lead to greater uptake and use of CDSS, and may become a requirement in the future due to societal, regulatory, and ethical pressures [18], which could make the difference between success and failure of the system [19].”

• cites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion sourceNot checked yet

  Not checked in the free run.

• cites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

XAI needs to be interactive and human-like with customized explanations for different users.

“XAI needs to be interactive and human-like with customized explanations for different users.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

To close the gap between ML outputs and human explanations requires interdisciplinary work that studies how humans explain, formalizes the patterns in algorithmic forms, and explains outputs in a transparent, easy to interpret manner.

“To close the gap requires interdisciplinary work that studies how humans explain, formalizes the patterns in algorithmic forms, and explains outputs in a transparent, easy to interpret manner [67].”

• cites [67] Molnar, C., Casalicchio, G., and Bischl, B. (2020). Interpretable Machine Learning—A Brief History, State-of-the-Art and Challenges. arXiv. sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: Medical experts struggle with the gap between what is output by an ML-based solution and human explanations.

Probably not a substantial claim: may not pass S3 (load-bearing).

There is an open debate on whether or not XAI in these contexts is necessary and/or worth the substantial overall cost.

“However, there is an open debate on whether or not XAI in these contexts is necessary and/or worth the substantial overall cost [19].”

• cites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Baldi explains this argument using examples of the lack of explainability of many processes in our daily lives, for example, how cars, computers, cell phones, or even our brains work.

“Baldi [135] explains this argument using examples of the lack of explainability of many processes in our daily lives, for example, how cars, computers, cell phones, or even our brains work.”

• cites [135] Baldi, 2018, Deep learning in biomedical data science, Annu. Rev. Biomed. Data Sci. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

London defends the ability to produce results and empirically verify their accuracy as more important than the ability to explain how such results are produced.

“London [134] defends the ability to produce results and empirically verify their accuracy as more important than the ability to explain how such results are produced.”

• cites [134] London, 2019, Artificial intelligence and black-box medical decisions: Accuracy versus explainability, Hastings Cent. Rep. sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The authors agree with the human-computer interaction community that interdisciplinary cooperation and user-centered approaches to explainability are required to close the gaps between XAI and practices.

“The authors agree with the human-computer interaction community that interdisciplinary cooperation and user-centered approaches to explainability are required to close the gaps between XAI and practices.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Models should be built in collaboration with input from those with expertise from the fields of social and behavioral science, philosophy, psychology, and cognitive science.

“Models should be built in collaboration with input from those with expertise from the fields of social and behavioral science, philosophy, psychology, and cognitive science [19,20,64].”

• cites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access sourceNot checked yet

  Not checked in the free run.

• cites [20] Vellido, 2019, The importance of interpretability and visualization in machine learning for applications in medicine and health care, Neural Comput. Appl. sourceNot checked yet

  Not checked in the free run.

• cites [64] Miller, T., Howe, P., and Sonenberg, L. (2017). Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Arrieta et al. recommend that black-box models should be selected only when necessary and, when possible, the use of interpretable or transparent by design algorithms should be prioritized over complex algorithms that require post-hoc XAI techniques.

“Additionally, Arrieta et al. [18] recommend that those black-box models should be selected only when necessary and, when possible, the use of interpretable or transparent by design algorithms should be prioritized over complex algorithms that require the application of post-hoc XAI techniques.”

• cites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Ethics, fairness, and safety-related implications, as well as the cognitive skills and limitations of the audience must be considered when deciding what type of explanations should be provided.

“Additionally, ethics, fairness, and safety-related implications, as well as the cognitive skills and limitations of the audience must be considered when deciding what type of explanations should be provided [18].”

• cites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Explanations should be at least potentially actionable, parsimonious, and timely, warranting further research.

“However, explanations should be at least potentially actionable, parsimonious, and timely [69], warranting further research [19].”

• cites [69] Tonekaboni, S., Joshi, S., McCradden, M.D., and Goldenberg, A. (2019, January 13–14). What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use. Proceedings of the Machine Learning for Healthcare Conference, Boston, MA, USA.Not checked yet

  Not checked in the free run.

• cites [19] Adadi, 2018, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

We might surpass the problem of data availability and heterogeneity by using the least amount of data available and the easiest-to-collect information to initiate the development of the system.

“Additionally, we might surpass the problem of data availability and heterogeneity by using the least amount of data available and the easiest-to-collect information to initiate the development of the system [140].”

• cites [140] Antoniadi, A.M., Galvin, M., Heverin, M., Hardiman, O., and Mooney, C. (2021, January 22–26). Development of an explainable clinical decision support system for the prediction of patient quality of life in amyotrophic lateral sclerosis. Proceedings of the 36th Annual ACM Symposium on Applied Computing, Gwangju, Korea. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Sullivan states that the opaqueness of models such as deep neural networks is not what is limiting our understanding, but rather the 'link uncertainty', meaning the empirical link between the model's features and the phenomenon studied.

“Note that Sullivan [136] states that the opaqueness of models such as deep neural networks is not what is limiting our understanding, but rather the”

• cites [136] Sullivan, E. (2020). Understanding from machine learning models. Br. J. Philos. Sci.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Lipton argues that the short-term goal of building trust with doctors by developing transparent models might clash with the longer-term goal of improving health care.

“Lipton [46] argues that the short-term goal of building trust with doctors by developing transparent models might clash with the longer-term goal of improving health care.”

• cites [46] Lipton, 2018, The mythos of model interpretability, Queue sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Developing ML-based CDSS is a multidisciplinary process that should include the needs of all stakeholders.

“Developing ML-based CDSS is a multidisciplinary process that should include the needs of all stakeholders.”

Classified as probably not a substantial claim. May combine two claims.

Domain-specific needs must be taken into account including a thorough understanding of the purpose of the system, the performance and interpretability of existing systems, and the level and nature of the explanations that are required.

“Domain-specific needs must be taken into account including a thorough understanding of the purpose of the system, the performance and interpretability of existing systems, and the level and nature of the explanations that are required [18].”

• cites [18] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

A lack of research in general is likely both a symptom and cause of skepticism around CDSS use in practice.

“A lack of research in general is likely both a symptom and cause of this.”

Probably not a substantial claim: may not pass S2 (checkable).

Studies focusing on all stages of CDSS development are required to establish more firmly how explainability can be put into useful practice in this important context.

“Studies focusing on all stages of CDSS development are required to establish more firmly how explainability can be put into useful practice in this important context.”

Probably not a substantial claim: may not pass S2 (checkable).

Significant work lies ahead in order to integrate useful explainablity into CDSS.

“Significant work lies ahead in order to integrate useful explainablity into CDSS.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

For CDSS to be used effectively in practice, they need to be trustworthy, easy to understand, and, most of all, positively augment the human decision-making process.

“In order for Clinical Decision Support Systems (CDSS) to be used effectively in practice, they need to be trustworthy, easy to understand, and, most of all, positively augment the human decision-making process.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Explainability allows developers to identify shortcomings in a system and allows clinicians to be confident in the decisions they make with CDSS assistance.

“Explainability allows developers to identify shortcomings in a system and allows clinicians to be confident in the decisions they make with CDSS assistance.”

Classified as probably not a substantial claim.

Explainability is a critical component in achieving the goals of trustworthy, easy to understand, and positively augmenting CDSS.

“Explainability is a critical component in achieving these goals.”