Interpretability of machine learning-based prediction models in healthcare
Open accessAbstract
There is a need of ensuring that learning (ML) models are interpretable. Higher interpretability of the model means easier comprehension and explanation of future predictions for end‐users. Further, interpretable ML models allow healthcare experts to make reasonable and data‐driven decisions to provide personalized decisions that can ultimately lead to higher quality of service in healthcare. Generally, we can classify interpretability approaches in two groups where the first focuses on personalized interpretation (local interpretability) while the second summarizes prediction models on a population level (global interpretability). Alternatively, we can group interpretability methods into model‐specific techniques, which are designed to interpret predictions generated by a specific model, such as a neural network, and model‐agnostic approaches, which provide easy‐to‐understand explanations of predictions made by any ML model. Here, we give an overview of interpretability approaches using structured data and provide examples of practical interpretability of ML in different areas of healthcare, including prediction of health‐related outcomes, optimizing treatments, or improving the efficiency of screening for specific conditions. Further, we outline future directions for interpretable ML and highlight the importance of developing algorithmic solutions that can enable ML driven decision making in high‐stakes healthcare problems. This article is categorized under: Application Areas > Health Care
AI snapshot
Written from the paper’s full text.
TL;DR
This advanced review provides a practical overview of interpretability approaches for machine-learning prediction models in healthcare, organizing them as model-specific vs. model-agnostic and local vs. global. It illustrates these categories with real-world healthcare examples and outlines open challenges for interpretable ML in high-stakes clinical decision-making.
Key findings
- Interpretability methods can be grouped into model-specific techniques, which examine internal model parameters, and model-agnostic techniques, which can be applied to any model, often post hoc; they can also be grouped into local interpretation of a single prediction and global interpretation of overall model behavior.
- The review reports that SHAP was used to interpret predictions for prevention of hypoxaemia during surgery and “increased the anaesthesiologists anticipation of hypoxaemia events by 15%.”
- MUSE was used to explain decisions from a “3-level neural network trained on depression diagnosis dataset,” generating global if-then rules and a separate rule set for a subspace based on features selected by a healthcare expert, focusing on “excursive and smoking” as potentially actionable features.
- Visual analytics examples include RetainVis, a model-specific approach visualizing a recurrent neural network and used to predict future diagnosis of heart failure and cataract, and RuleMatrix, a model-agnostic rule-based system used to improve cancer and diabetes classification.
Methods
- Design: advanced review / narrative overview of interpretability approaches for ML-based prediction models in healthcare, with a proposed simple categorization for experts and non-experts.
- Scope: includes examples of real-world studies using interpretability approaches in healthcare, but explicitly excludes approaches focused on interpretability of ML models built on nonstructured data such as medical images, text, or other signal-based data.
- Analysis: conceptual synthesis and illustrative examples organized by model-specific/model-agnostic and local/global interpretability; no systematic search, primary data collection, or quantitative meta-analysis is reported.
Limitations
- Stated: the overview “does not provide an in-depth analysis” of open topics such as simplification of complex models for explanations that may result in suboptimal results, scalability of interpretable ML models (e.g., LIME or SHAP values can be computationally expensive), methods for evaluation of explanation models, and causality.
- Stated: the review excludes interpretability approaches for nonstructured data such as medical images, text, or other signal-based data.
- Evident from the text: as an advanced review with illustrative examples rather than a systematic review, it does not report a formal search strategy, inclusion criteria, or quantitative synthesis of evidence.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 39 cited sources checked · 0 supported · 0 flagged
- Interpretationreasoning§ Introduction · p. 2
Lack of interpretability is a key factor that limits wider adoption of machine learning in healthcare.
“Further, the lack of interpretability is a key factor that limits wider adoption of ML in healthcare.”
Quote in the paperSubstantial claim - Premisecitation§ Abstract · p. 1
Higher interpretability of a machine learning model makes it easier for end-users to comprehend and explain future predictions.
“Higher interpretability of the model means easier comprehension and explanation of future predictions for end-users.”
- Also stated in: § Introduction · p. 3 · plain
“It is important that ML models are interpretable, because the higher the interpretability of the model, the easier it is to comprehend why certain predictions have been made (Molnar, 2018).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Molnar, 2018
Not checked in the free run.
- Also stated in: § Introduction · p. 3 · plain
- Premisecitation§ Introduction · p. 2
Bias of prediction models, for example towards black people as demonstrated by Obermeyer and colleagues (2019), can further undermine the trust of healthcare experts and other end-users.
“Bias of prediction models, for example towards black people as demonstrated in a study by Obermeyer and colleagues (2019), can further undermine the trust of the healthcare experts and other end-users of such models.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Obermeyer and colleagues (2019)
Not checked in the free run.
- Premisecitation§ Applications of interpretable machine learning in healthcare · p. 7
The authors of MUSE demonstrate its effectiveness to produce rules optimised for fidelity, unambiguity, and interpretability for a subspace focusing on exercise and smoking as those two features might be chosen by the expert as actionable features.
“More specifically, authors demonstrate the effectiveness of MUSE to produce rules optimised for fidelity, unambiguity, and interpretability for a subspace focusing on excursive and smoking as those two features might be chosen by the expert as actionable features.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Lakkaraju, et al. (2019)
Not checked in the free run.
- Interpretationreasoning§ Applications of interpretable machine learning in healthcare · p. 7
A typical global interpretability approach would not consider that some features in datasets are more interesting than others because patients and healthcare experts can influence their values through interventions.
“A typical global interpretability approach would not consider the fact that there are features in the datasets that are more interesting than others as it is possible for the patient and healthcare expert to influence their values by introducing some interventions.”
Quote in the paperSubstantial claim
- Premisereasoning§ Applications of interpretable machine learning in healthcare · p. 7
Some recent approaches to interpretability cannot be simply classified in one of the four groups displayed in Figure 1.
“As already mentioned in the previous section, some recent approaches to interpretability cannot be simply classified in one of the four groups displayed in Figure 1.”
Quote in the paperNot a substantial claim - Premisecitation§ Applications of interpretable machine learning in healthcare · p. 7
MUSE was used to help explain decisions from a 3-level neural network trained on a depression diagnosis dataset.
“MUSE was used to help in explaining decisions from the 3-level neural network trained on depression diagnosis dataset.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lakkaraju, et al. (2019)
Not checked in the free run.
- Premisecitation§ Applications of interpretable machine learning in healthcare · p. 6
Model-specific approaches focusing on local interpretation based on KNN or decision trees were recently used for interpretation in prediction of health-related conditions, including occupational diseases, knee osteoarthritis, cancer, severity of disease, and mortality rates.
“Model-specific approaches focusing on local interpretation that can be based for example on KNN or decision trees were recently used for interpretation in prediction of health-related conditions, including occupational diseases (Di Noia, et al., 2019) or knee osteoarthritis (Jamshidi, et al., 2019), cancer (e.g. breast cancer or prostate cancer) (Oladele Aro, et al., 2019; Seker, et al., 2019),”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Di Noia, et al., 2019
Not checked in the free run.
- cites Jamshidi, et al., 2019
Not checked in the free run.
- cites Oladele Aro, et al., 2019
Not checked in the free run.
- Premisecitation§ Conclusion · p. 8
There is a limited understanding of graph neural networks' properties and limitations.
“However, there is a limited understanding of GNNs properties and limitations (Xu, et al., 2019).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Xu, et al., 2019
Not checked in the free run.
- Classified as probably not a substantial claim. 32
- May combine two claims. 17
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone given§ Introduction · p. 3Both global and local interpretability approaches have been successfully used in different healthcare domains and are still being advanced in parallel with novel methodological approaches.
“Both types of approaches have been successfully used in different healthcare domains and are still being advanced in parallel with novel methodological approaches in development of ML based prediction models.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ Abstract · p. 1Interpretability approaches can be classified into local interpretability, which focuses on personalized interpretation, and global interpretability, which summarizes prediction models on a population level.
“Generally, we can classify interpretability approaches in two groups where the first focuses on personalized interpretation (local interpretability) while the second summarizes prediction models on a population level (global interpretability).”
- Also stated in: § Local or global interpretability · p. 4 · plain
“Methods can explain a single prediction (local interpretability) or the entire model behaviour (global interpretability) (Molnar, 2018).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Molnar, 2018
Not checked in the free run.
- Also stated in: § Local or global interpretability · p. 4 · plain
Probably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationreasoning§ Abstract · p. 1Interpretable machine learning models allow healthcare experts to make reasonable and data-driven decisions to provide personalized decisions that can ultimately lead to higher quality of service in healthcare.
“Further, interpretable machine learning models allow healthcare experts to make reasonable and data-driven decisions to provide personalized decisions that can ultimately lead to higher quality of service in healthcare.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ Abstract · p. 1Interpretability methods can be grouped into model-specific techniques, which are designed to interpret predictions generated by a specific model, and model-agnostic approaches, which provide explanations of predictions made by any machine learning model.
“Alternatively, we can group interpretability methods into model-specific techniques, which are designed to interpret predictions generated by a specific model, such as a neural network, and model-agnostic approaches, which provide easy-to-understand explanations of predictions made by any machine learning model.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Du, Liu & Hu, 2019
Not checked in the free run.
- cites Molnar, 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Introduction · p. 3Traditionally the focus in machine learning research was on global interpretability to help users understand the relationship between all possible inputs to a model and the space of all predictions made by the model.
“Traditionally the focus in ML research was on global interpretability in an effort to help users understand the relationship between all possible inputs to a ML model and the space of all predictions made by the model (Bratko, 1999; Martens, et al., 2008).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Bratko, 1999
Not checked in the free run.
- cites Martens, et al., 2008
Not checked in the free run.
Classified as probably not a substantial claim. May combine two claims.
Interpretationreasoning§ Interpretability of machine learning models · p. 4Interpretability approaches can be categorized into two non-exclusive groups: model-specific or model-agnostic, and local or global.
“Therefore, we categorize interpretability approaches in two non-exclusive groups - i.e. model-specific or model-agnostic and local or global.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ Applications of interpretable machine learning in healthcare · p. 6Attention-map visualization approaches for acute intracranial haemorrhage detection are not scalable.
“but it should be noted that such approaches are not scalable.”
Quote in the paperNot a substantial claimMay combine two claims.
Premisecitation§ Applications of interpretable machine learning in healthcare · p. 6SHAP was used in the detection of acute intracranial haemorrhage from images of CT scans, achieved via attention maps visualization where areas were highlighted.
“or in the detection of acute intracranial haemorrhage from images of CT scans (Lee, et al., 2019), which was achieved via attention maps visualization, where areas were highlighted (segmentation map).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lee, et al., 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ Sidebar title: Visual analytics and interpretability · p. 8RetainVis uses t-SNE to project patients on a 2D space and explains the model's interpretation of data by showing which patients are closely located in the latent space.
“RetainVis uses t-SNE (Maaten & Hinton, 2008) to project patients on a 2D space and explains the model's interpretation of data by showing which patients are closely located in the latent space.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Maaten & Hinton, 2008
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ Model-specific or model-agnostic interpretability · p. 4Model-agnostic methods can be used on any machine learning model and are usually applied post hoc.
“Model-agnostic methods can be used on any machine learning model and are usually applied post hoc (Molnar, 2018).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Molnar, 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Sidebar title: Visual analytics and interpretability · p. 8Visual analytics extends the interaction paradigm of traditional information visualisation techniques to support both the interpretation of machine learning models and model steering with feedback from end-users.
“Visual analytics extends the interaction paradigm of traditional information visualisation techniques to support both the interpretation of machine learning models and model steering with feedback from end-users (Endert et al. 2017).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Endert et al. 2017
Not checked in the free run.
Probably not a substantial claim: may not pass S4 (based).
Interpretationreasoning§ Introduction · p. 2Applying machine learning models on larger systems may not perform well because of the complexity of the data and diversity of patients and diagnoses.
“Furthermore, applying those models on the larger systems may not perform well because of the complexity of the data and diversity of patients and diagnoses.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable). May combine two claims.
Interpretationcitation§ Introduction · p. 3There must be a possibility to make the results of machine learning models re-traceable, or the end-user has the right to explanation of all decisions made by the computer.
“Consequently, this means that there must be a possibility to make the results re-traceable or in other words, the end-user has the right to explanation of all decisions made by the computer.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone given§ Introduction · p. 3Approaches for interpretability of machine learning models are either global or local.
“In general, approaches for interpretability of ML models are either global or local.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Bratko, 1999
Not checked in the free run.
- cites Martens, et al., 2008
Not checked in the free run.
- cites Hall, et al, 2019
Not checked in the free run.
May combine two claims.
Premisecitation§ Applications of interpretable machine learning in healthcare · p. 7MUSE generates sets of if-then rules that describe model decisions on a global level and provides a separate set of rules for a subspace based on features selected by a healthcare expert working with patients.
“It generates sets of if-then rules that describe the model decisions on a global level, but it also provides a separate set of rules for a subspace based on the features selected by a healthcare expert working with patients.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Lakkaraju, et al. (2019)
Not checked in the free run.
May combine two claims.
Premisecitation§ Introduction · p. 2Most machine learning models focus on accuracy prediction and rarely explain their predictions in a meaningful way.
“Moreover, most of the models focus on accuracy prediction and rarely explain their predictions in a meaningful way (Elshawi, AlMallah, and Sakr, 2019; Ahmad, et al., 2018).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Elshawi, AlMallah, and Sakr, 2019
Not checked in the free run.
- cites Ahmad, et al., 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S4 (based). May combine two claims.
Interpretationnone given§ Interpretability of machine learning models · p. 4In the clinical setting, machine learning prediction models are different than models in other fields because predictions can have an impact on patient's health, so users need to have confidence in the predictions.
“In the clinical setting ML prediction models are different than models in other fields, since the predictions can have an impact on patient's health, therefore the users of such models need to have confidence in the predictions of such models.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Sidebar title: Visual analytics and interpretability · p. 8Both model-specific and model-agnostic approaches have been researched in visual analytics prototypes.
“Both model-specific and model-agnostic approaches have been researched in VA prototypes.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ Discussion · p. 7In recent years, a shift can be observed from model-specific and global interpretable models to model-agnostic and local interpretable models, one reason being the availability of massive datasets in healthcare and a consequent shift to precision medicine.
“In recent years, a shift can be observed from model-specific and global interpretable models to model-agnostic and local interpretable models, where one of the reasons is the availability of massive datasets in healthcare (Esteva, et al., 2019) and consequently a shift to precision medicine.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Esteva, et al., 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ Sidebar title: Visual analytics and interpretability · p. 8RuleMatrix helps domain experts understand and inspect classification models using rule-based explanations and was used to improve cancer and diabetes classification.
“The system helps domain experts understand and inspect classification models using rule-based explanations and was used to improve cancer and diabetes classification.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Ming, Qu, & Bertin, 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Model-specific or model-agnostic interpretability · p. 4To achieve model-agnostic interpretability, one can use a surrogate or simple proxy model to learn a locally faithful approximation of a complex black-box model based on outputs returned by the black-box model.
“Typically, to achieve model-agnostic interpretability, one can use a surrogate or a simple proxy model to learn a locally faithful approximation of a complex, blackbox model based on outputs returned by the black-box model.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Bucila, Caruana, and Niculescu-Mizil (2006)
Not checked in the free run.
- cites Hinton, Vinyals & Dean, 2015
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Model-specific or model-agnostic interpretability · p. 4GNNExplainer provides both global and local interpretability.
“GNNExplainer provides both global and local interpretability and might prove useful for practitioners of GNN in the future, as it provides the ability to visualize relevant structures to interpretability and gives insights into errors of faulty GNNs (Ying, et al., 2019).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Ying, et al., 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Sidebar title: Visual analytics and interpretability · p. 8RetainVis has been used to predict future diagnosis of heart failure and cataract.
“The approach has been used to predict future diagnosis of heart failure and cataract.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Kwon et al., 2018
Not checked in the free run.
May combine two claims.
Premisecitation§ Applications of interpretable machine learning in healthcare · p. 6SHAP was used in interpretation of predictions for the prevention of hypoxaemia during surgery, which increased the anaesthesiologists' anticipation of hypoxaemia events by 15%.
“For example, SHAP was used in interpretation of predictions for the prevention of hypoxaemia during surgery (Lundberg, et al., 2018), which increased the anaesthesiologists anticipation of hypoxaemia events by 15%”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lundberg, et al., 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Local or global interpretability · p. 4MUSE uses the behaviour of subspaces characterized by certain features of interest for explanation.
“An example of such model is Model Understanding through Subspace Explanations (MUSE), where it uses the behaviour of subspaces characterized by certain features of interest for explanation (Lakkaraju, et al., 2019).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lakkaraju, et al., 2019
Not checked in the free run.
May combine two claims.
Interpretationreasoning§ Applications of interpretable machine learning in healthcare · p. 7MUSE can be classified as a global model-agnostic interpretability approach, but it also demonstrates characteristics of approaches focusing on personalised interpretation by narrowing the subspace of search by end-user input.
“As such, MUSE could be classified as a global model-agnostic interpretability approach, but it also demonstrates characteristics of approaches focusing on personalised interpretation by narrowing the subspace of search by end-user input.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ Applications of interpretable machine learning in healthcare · p. 6Local and model-agnostic interpretability can be used in interpretability of complex models, such as deep learning models.
“Local and modelagnostic interpretability can be used in interpretability of complex models, such as deep learning models.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Local or global interpretability · p. 4In order to explain the global output of a model, one needs a trained model, knowledge about the algorithm and the data.
“In order to explain global output of the model one needs a trained model, knowledge about the algorithm and the data (Lipton, 2016).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lipton, 2016
Not checked in the free run.
May combine two claims.
Premisecitation§ Applications of interpretable machine learning in healthcare · p. 6The highlighted results in the acute intracranial haemorrhage detection study corresponded to the classification and could be understood by a practitioner.
“The highlighted results corresponded to the classification and could be understood by a practitioner (Lee, et al., 2019)”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Lee, et al., 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Sidebar title: Visual analytics and interpretability · p. 8In recent years, researchers are increasingly relying on Visual Analytics techniques to support interpretability of machine learning models within healthcare fields.
“In recent years, researchers are increasingly relying on Visual Analytics (VA) techniques to support interpretability of machine learning models within healthcare fields (Simpao et al. 2014).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Simpao et al. 2014
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Introduction · p. 3Local interpretation helps users to understand a prediction for a specific individual or about a small, specific region of the trained prediction function.
“In contrast, local interpretation helps users to understand a prediction for a specific individual or about a small, specific region of the trained prediction function (Hall, et al, 2019; Stiglic, et al., 2006).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Hall, et al, 2019
Not checked in the free run.
- cites Stiglic, et al., 2006
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Sidebar title: Visual analytics and interpretability · p. 8RetainVis is a prominent example of a model-specific visual analytics approach that visualizes a recurrent neural network to support interpretation and diagnosis of the model.
“RetainVis (Kwon et al., 2018) is a prominent example of a model-specific VA approach that visualizes a recurrent neural network to support interpretation and diagnosis of the model.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Kwon et al., 2018
Not checked in the free run.
Classified as probably not a substantial claim. May combine two claims.
Interpretationreasoning§ Introduction · p. 2Healthcare workers often find it challenging to trust complex machine-learning models because they are often designed and rigorously evaluated only on specific diseases in a narrow environment and depend on technical knowledge of statistics and machine learning.
“This is because healthcare workers often find it challenging to trust complex machine-learning models because they are often designed and rigorously evaluated only on specific diseases in a narrow environment and they depend on one's technical knowledge of statistics and machine learning.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Sidebar title: Visual analytics and interpretability · p. 8Visual analytics techniques combine concepts from data mining, machine learning, human computing interaction, and human cognition in order to facilitate interpretation of complex data and models.
“In order to facilitate interpretation of complex data and models, VA techniques combine concepts from data mining, machine learning, human computing interaction, and human cognition (Caban & Gotz, 2015).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Caban & Gotz, 2015
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone given§ Applications of interpretable machine learning in healthcare · p. 6Global model-specific approaches to interpretability of machine learning based models in healthcare have been in use for more than two decades.
“Global model-specific approaches to interpretability of ML based models in healthcare have been in use for more than two decades.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Sidebar title: Visual analytics and interpretability · p. 8RuleMatrix is a model-agnostic example that relies on extracted rule-based knowledge from the input-output behavior of a model.
“RuleMatrix (Ming, Qu, & Bertin, 2018) is a model-agnostic example that relies on extracted rule-based knowledge from the input-output behavior of a model.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Ming, Qu, & Bertin, 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone given§ Sidebar title: Visual analytics and interpretability · p. 8In contrast to black-box-driven approaches, the objective of visual analytics is to allow domain expert users to interpret and diagnose models.
“In contrast to 'black box'-driven approaches, the objective is to allow domain expert users to interpret and diagnose models.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ Conclusion · p. 8Interpretability can be categorised in terms of model-specific and model-agnostic or global and local interpretability.
“Interpretability can be categorised in terms of model-specific and model-agnostic or global and local interpretability.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
In healthcare applications, achieving high predictive accuracy is often as important as understanding the prediction.
“This is especially problematic in healthcare applications, where achieving high predictive accuracy is often as important as understanding the prediction.”
The paper relies on this without support.
Healthcare decisions are high-stakes because they can directly influence a treatment outcome or even a patient's survival.
“In contrast to many other fields, decisions in healthcare are high-stakes decisions as they can directly influence a treatment outcome or even survival of a patient.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Model explanation and interpretability may not fully capture complex interactions in the model, leading to limited interpretability.
“May not fully capture interpretability complex interactions in the model, leading to limited”
From AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications (2023) · cites this paper as [268]
Model explanation and interpretability provide insights into decision-making.
“Provides insights into decision-making”
From AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications (2023) · cites this paper as [268]
Model explanation and interpretability facilitate understanding of model behavior and potential biases.
“Facilitates understanding of model behavior and potential biases”
From AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications (2023) · cites this paper as [268]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Gao citation year conflicts with reference
MinorThe in-text citation gives 2019, but the Gao reference is dated 2020.
“Gao, et al., 2019”
“Gao, Y., Long, Y., Guan, Y., Basu, A., Baggaley, J., & Plötz, T. (2020).”
Miller citation year conflicts with reference
MinorThe text cites Miller as 2017, whereas the listed Miller reference is dated 2019.
“(Miller, 2017)”
“Miller, T. (2019).”
Seker citation year conflicts with reference
MinorThe text cites Seker et al. as a 2019 study, while the corresponding reference is dated 2000.
“(Oladele Aro, et al., 2019; Seker, et al., 2019)”
“Seker, H., Odetayo, M. O., Petrovic, D., Naguib, R., & Hamdy, F. (2000).”
Bratko citation year conflicts with reference
MinorThe text cites Bratko as 1999, whereas the listed reference is dated 1997.
“(Bratko, 1999; Martens, et al., 2008)”
“Bratko, I. (1997).”
Du citation year conflicts with reference
MinorThe text cites Du, Liu and Hu as 2019, whereas their listed reference is dated 2018.
“(Du, Liu & Hu, 2019)”
“Du, M., Liu, N., & Hu, X. (2018).”
Molnar citation year conflicts with reference
MinorThe text cites Molnar as 2018, whereas the listed Molnar reference is dated 2019.
“(Molnar, 2018)”
“Molnar, C. (2019).”
Xu citation year conflicts with reference
MinorThe text cites Xu et al. as 2019, whereas the listed Xu reference is dated 2018.
“(Xu, et al., 2019)”
“Xu, K., Hu, W., Leskovec, J., & Jegelka, S. (2018).”
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 2× · see citing works
“…interpret the decision directly from the model (e.g. extracting all rules from the decision tree) or use some specific technique like knowledge distillation (Hinton, Vinyals & Dean, 2015) to build a simple model that can be interpreted (e.g. a decision tree, set of rules or a regression function).”
MethodRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · see citing works
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878 · see citing works
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 1× · see citing works
“There is a widespread usage of artificial intelligence (AI) due to tremendous progress in technology and industrial revolution (Adadi & Berrada, 2018).”
BackgroundA guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312 · cited 1× · see citing works
“In recent years, a shift in the research of interpretability methods can be observed from model-specific and global interpretable models to model-agnostic and local interpretable models, where one of the reasons is the availability of massive datasets in healthcare (Esteva et al., 2019).”
BackgroundExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · see citing works
Formal Semantics for Kolmogorov-Arnold Network Representations of Operational GamesPreprint
Xu, K., Hu, W., Leskovec, J., & Jegelka, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.00826 · Cited by 5,266 · see citing works
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340 · see citing works
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136 · see citing works
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1× · see citing works
“Interpretable ML is fundamentally complex and represents a rapidly developing field of research (Gilpin et al., 2019; Hall & Gill, 2018).”
BackgroundDefinitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · see citing works
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 2× · see citing works
“, 2016), - Shapley additive explanations (SHAP; Lundberg & Lee, 2017), - Anchors (Ribeiro et al., 2018), - Attention map visualization, - Model understanding through subspace explanations (MUSE; Lakkaraju et al.”
BackgroundBuciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068 · see citing works
Explainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967 · cited 1× · see citing works
“For example, SHAP was used in interpretation of predictions for the prevention of hypoxaemia during surgery (Lundberg, et al., 2018), which increased the anaesthesiologists anticipation of hypoxaemia events by 15%, or in the detection of acute intracranial haemorrhage from images of CT scans (Lee,…”
MethodCausability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885 · see citing works
Machine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · cited 3× · see citing works
“There are different criteria for classifying methods for machine learning interpretability such as intrinsic or post-hoc classification, pre-model, in-model or post-model and classification based on the model outcome (Molnar, 2018; Carvalho, Pereira & Cardoso, 2019).”
BackgroundMethodWachter, S., Mittelstadt, B., & Floridi, L. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx005 · Cited by 1,308 · see citing works
Deep Learning for Medical Image Processing: Overview, Challenges and the Future
Razzak, I., Naz, S., & Zaib, A. (2017) · Lecture notes in computational vision and biomechanics · doi:10.1007/978-3-319-65981-7_12 · Cited by 1,305 · see citing works
Techniques for interpretable machine learning
Du, M., Liu, N., & Hu, X. (2019) · Communications of the ACM · doi:10.1145/3359786 · Cited by 1,268 · see citing works
Do no harm: a roadmap for responsible machine learning for health care
Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0548-6 · Cited by 1,266 · cited 1× · see citing works
“Recently a broader concept of responsible ML in healthcare was presented (Wiens, et al., 2019), which proposes that an interdisciplinary team of engaged stakeholders (policy makers, health system leaders and individual researchers) systematically progress from problem formulation to widespread…”
BackgroundGNNExplainer: Generating Explanations for Graph Neural Networks
Ying, R., Bourgeois, D., You, J., Zitnik, M., & Leskovec, J. (2019) · arXiv · doi:10.48550/arxiv.1903.03894 · Cited by 850 · see citing works
Self-citationArtificial Intelligence and Black‐Box Medical Decisions: <i>Accuracy versus Explainability</i>
London, A. J. (2019) · Hastings Center Report · doi:10.1002/hast.973 · Cited by 801 · cited 1× · see citing works
“To obtain more information on a more general view of interpretability and explainable AI in healthcare, the reader is referred to works by Holzinger and colleagues (2019) or London (2019).”
MethodModel-Agnostic Interpretability of Machine Learning
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1606.05386 · Cited by 699 · cited 1× · see citing works
“- Local interpretable model-agnostic explanations (LIME; Ribeiro et al., 2016), - Shapley additive explanations (SHAP; Lundberg & Lee, 2017), - Anchors (Ribeiro et al.”
BackgroundVellido, A. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04051-w · Cited by 635 · see citing works
Mazurowski, M. A., Buda, M., Saha, A., & Bashir, M. R. (2019) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.26534 · Cited by 554 · see citing works
Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
Zintgraf, L. M., Cohen, T. S., Adel, T., & Welling, M. (2017) · arXiv · doi:10.48550/arxiv.1702.04595 · Cited by 487 · see citing works
AI in Health: State of the Art, Challenges, and Future Directions
Wang, F., & Preininger, A. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677908 · Cited by 398 · cited 1× · see citing works
“For example, in a recent review paper on the state of AI in healthcare Wang, and Preininger (2019) quote Geoff Hinton, one of the pioneers in the field of AI, saying: “Policymakers should not insist on ensuring people understand exactly how a given AI algorithm works, because people can’t explain…”
BackgroundUsing Sampling and Queries to Extract Rules from Trained Neural Networks
Craven, M. W., & SHAVLIK, J. W. (1994) · Elsevier eBooks · doi:10.1016/b978-1-55860-335-6.50013-1 · Cited by 312 · see citing works
INDUCTIVE AND BAYESIAN LEARNING IN MEDICAL DIAGNOSIS
Kononenko, I. (1993) · Applied Artificial Intelligence · doi:10.1080/08839519308949993 · Cited by 303 · cited 1× · see citing works
“, 2009), - Naive Bayes classifier (Kononenko, 1993), - GNNExplainer (Ying et al.”
BackgroundKwon, B. C., Choi, M.-J., Kim, J. T., Choi, E., Kim, Y. B., Kwon, S., Sun, J., & Choo, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865027 · Cited by 294 · cited 1× · see citing works
“RetainVis (Kwon et al., 2018) is a prominent example of a model-specific VA approach that visualizes a recurrent neural network to support interpretation and diagnosis of the model.”
BackgroundOn the interpretability of machine learning-based model for predicting hypertension
Elshawi, R., Al-Mallah, M. H., & Sakr, S. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0874-0 · Cited by 274 · cited 1× · see citing works
“Moreover, most of the models focus on accuracy prediction and rarely explain their predictions in a meaningful way (Elshawi, Al-Mallah, and Sakr, 2019; Ahmad, et al., 2018).”
Background"What is relevant in a text document?": An interpretable machine learning approach
Arras, L., Horn, F., Montavon, G., Müller, K., & Samek, W. (2017) · PLoS ONE · doi:10.1371/journal.pone.0181142 · Cited by 268 · see citing works
Faithful and Customizable Explanations of Black Box Models
Lakkaraju, H., Kamar, E. S., Caruana, R., & Leskovec, J. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314229 · Cited by 261 · cited 10× · see citing works
“In the future, we expect more approaches like MUSE (Lakkaraju et al., 2019) where a global interpretability approach is supplemented by specific interpretations that can explain predictions either for a single individual or a smaller group of individuals.”
BackgroundMethodInfluentialRulematrix: Visualizing and understanding classifiers with rules
Ming, Y., Qu, H., & Bertini, E. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864812 · Cited by 243 · see citing works
Elshawi, R., Mohamed, Y., Al-Mallah, M., & Sakr, S. (2019) · 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS) · doi:10.1109/cbms.2019.00065 · Cited by 224 · see citing works
An explainable artificial intelligence system for small-unit tactical behavior
van Lent, M., Fisher, W. M., & Mancuso, M. R. (2004) · Cited by 215 · see citing works
Explainable and Interpretable Models in Computer Vision and Machine Learning
Escalante, H. J., Escalera, S., Guyon, I., Baró, X., Güçlütürk, Y., Güçlü, U., & van Gerven, M. (2018) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-98131-4 · Cited by 203 · cited 1× · see citing works
“In CV, the focus is on identification of the image sections that are responsible for successful recognition of critical, health condition-specific regions in the image that might be relevant for medical experts (Escalante et al., 2018; Hosny, Parmar, Quackenbush, Schwartz, & Aerts, 2018; Mazurowski, Buda, Saha, & Bash…”
BackgroundThe State of the Art in Integrating Machine Learning into Visual Analytics
Endert, A., Ribarsky, W., Turkay, Ç., Wong, B. L. W., Nabney, I., Díaz, I., & Fabrice Rossi, A. (2017) · Computer Graphics Forum · doi:10.1111/cgf.13092 · Cited by 198 · cited 2× · see citing works
“Visual analytics extends the interaction paradigm of traditional information visualization techniques to support both the interpretation of ML models and model steering with feedback from end-users (Endert et al., 2017).”
MethodMachine-learning-based patient-specific prediction models for knee osteoarthritis
Jamshidi, A., Pelletier, J.-P., & Martel-Pelletier, J. (2019) · Nature Reviews Rheumatology · doi:10.1038/s41584-018-0130-5 · Cited by 190 · cited 1× · see citing works
“…used for interpretation in prediction of health-related conditions, including occupational diseases (Di Noia, et al., 2019) or knee osteoarthritis (Jamshidi, et al., 2019), cancer (e.g. breast cancer or prostate cancer) (Oladele Aro, et al., 2019; Seker, et al., 2019), severity of a disease,…”
BackgroundConsiderations for Evaluation and Generalization in Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_1 · Cited by 176 · see citing works
A review of analytics and clinical informatics in health care
Simpao, A. F., Ahumada, L. M., Gálvez, J. A., & Rehman, M. A. (2014) · Journal of Medical Systems · doi:10.1007/s10916-014-0045-x · Cited by 154 · cited 1× · see citing works
“In recent years, researchers are increasingly relying on Visual Analytics (VA) techniques to support interpretability of machine learning models within healthcare fields (Simpao et al. 2014).”
BackgroundZhang, Y., & Ma, Y. (2019) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2019.01.009 · Cited by 123 · see citing works
Comprehensive Decision Tree Models in Bioinformatics
Štiglic, G., Kocbek, S., Pernek, I., & Kokol, P. (2012) · PLoS ONE · doi:10.1371/journal.pone.0033812 · Cited by 121 · see citing works
Clinical implementation of AI technologies will require interpretable AI models
Jia, X., Ren, L., & Cai, J. (2019) · Medical Physics · doi:10.1002/mp.13891 · Cited by 113 · see citing works
Machine Learning Prediction of Response to Cardiac Resynchronization Therapy
Feeny, A. K., Rickard, J., Patel, D. R., Toro, S., Trulock, K. M., Park, C. J., LaBarbera, M. A., Varma, N., Niebauer, M. J., Sinha, S. K., et al. (2019) · Circulation Arrhythmia and Electrophysiology · doi:10.1161/circep.119.007316 · Cited by 110 · cited 1× · see citing works
“…et al., 2019; Otunaiya & Muhammad, 2019), toxicology (Zhang, et al., 2018, Zhang, et al., 2019), endocrinology (Alaoui, et al. 2019), neurology (Zhang & Ma., 2019), cardiology (Joshi, et al., 2019; Salman, 2019; Feeny, et al., 2019) or psychiatry (Guimarães, et al., 2019; Obeid, et al., 2019).”
Bucholc, M., Ding, X., Wang, H., Glass, D. H., Wang, H., Prasad, G., Maguire, L., Bjourson, A. J., McClean, P. L., Todd, S., et al. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.04.022 · Cited by 105 · see citing works
Martens, D., Huysmans, J., Setiono, R., Vanthienen, J., & Baesens, B. (2007) · Studies in computational intelligence · doi:10.1007/978-3-540-75390-2_2 · Cited by 95 · cited 1× · see citing works
“Traditionally the focus in ML research was on global interpretability in an effort to help users understand the relationship between all possible inputs to a ML model and the space of all predictions made by the model (Bratko, 1999; Martens, et al., 2008).”
MethodInterpretability of machine learning models and representations: an introduction.
Bibal, A., & Frénay, B. (2016) · Repository of the University of Namur · Cited by 74 · see citing works
Machine Learning Model Interpretability for Precision MedicinePreprint
Katuwal, G. J., & Chen, R. F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09045 · Cited by 74 · see citing works
What makes classification trees comprehensible?
Piltaver, R., Luštrek, M., Gams, M., & Martinčić-Ipšić, S. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.06.009 · Cited by 71 · see citing works
Machine Learning: Between Accuracy and Interpretability
Bratko, I. (1997) · Learning, Networks and Statistics · doi:10.1007/978-3-7091-2668-4_10 · Cited by 67 · cited 1× · see citing works
“Traditionally, the focus in ML research was on global interpretability in an effort to help users understand the relationship between all possible inputs to an ML model and the space of all predictions made by the model (Bratko, 1997; Martens, Huysmans, Setiono, Vanthienen, & Baesens, 2008).”
Zhang, H., Ren, J.-X., Ma, J., & Ding, L. (2018) · Molecular Diversity · doi:10.1007/s11030-018-9882-8 · Cited by 52 · cited 1× · see citing works
“Due to their high level of interpretability and simple use in practice, the approaches like linear regression or naive Bayes models are still used in different fields of healthcare like urology (Otunaiya & Muhammad, 2019; Zhang, Ren, Ma, & Ding, 2019), toxicology (Zhang et al., 2019; Zhang, Ma, Liu, Ren, & Ding, 2018)…”
MethodUstun, B., & Rudin, C. D. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098161 · Cited by 47 · see citing works
Obeid, J. S., Weeda, E. R., Matuskowitz, A. J., Gagnon, K. J., Crawford, T. L., Carr, C. M., & Frey, L. J. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0894-9 · Cited by 46 · cited 1× · see citing works
“…et al., 2019; Otunaiya & Muhammad, 2019), toxicology (Zhang, et al., 2018, Zhang, et al., 2019), endocrinology (Alaoui, et al. 2019), neurology (Zhang & Ma., 2019), cardiology (Joshi, et al., 2019; Salman, 2019; Feeny, et al., 2019) or psychiatry (Guimarães, et al., 2019; Obeid, et al., 2019).”
Using Visual Analytics to Interpret Predictive Machine Learning ModelsPreprint
Krause, J., Perer, A., & Bertini, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05685 · Cited by 46 · see citing works
Adjusting to the GDPR: The Impact on Data Scientists and Behavioral Researchers
Greene, T., Galit Shmueli, A., Ray, S., & Fell, J. (2019) · Big Data · doi:10.1089/big.2018.0176 · Cited by 45 · cited 1× · see citing works
“Moreover, the European General Data Protection Regulation (GDPR) policy on the rights of citizens states that researchers must explain algorithmic decisions that have been made in ML models (Wachter, Mittelstadt, & Floridi, 2017; Wallace & Castro, 2018; Greene, et al., 2019).”
BackgroundZhang, H., Ma, J., Liu, C., Ren, J.-X., & Ding, L. (2018) · Food and Chemical Toxicology · doi:10.1016/j.fct.2018.09.051 · Cited by 42 · cited 1× · see citing works
“…Naive Bayes classifier are still used in different fields of healthcare like urology (Zhang, et al., 2019; Otunaiya & Muhammad, 2019), toxicology (Zhang, et al., 2018, Zhang, et al., 2019), endocrinology (Alaoui, et al. 2019), neurology (Zhang & Ma., 2019), cardiology (Joshi, et al., 2019;…”
MethodNoia, A. D., Martino, A., Montanari, P., & Rizzi, A. (2019) · Soft Computing · doi:10.1007/s00500-019-04200-2 · Cited by 39 · see citing works
A visual analytics system for multi-model comparison on clinical data predictions
Li, Y., Fujiwara, T., Choi, Y. K., Kim, K. K., & Ma, K. (2020) · Visual Informatics · doi:10.1016/j.visinf.2020.04.005 · Cited by 37 · see citing works
Consistent and complete data and “expert” mining in medicine
Kovalerchuk, B., Vityaev, E. E., & Ruiz, J. F. (2003) · ScholarWorks (Central Washington University) · Cited by 35 · see citing works
Positive Predictive Values and Potential Success of Suicide Prediction Models
Simon, G. E., Shortreed, S. M., & Coley, R. Y. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.1516 · Cited by 26 · see citing works
Interpretable & Explorable Approximations of Black Box Models
Lakkaraju, H., Kamar, E., Caruana, R., & Leskovec, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01154 · Cited by 23 · see citing works
Data Mining and Machine Learning Approaches and Technologies for Diagnosing Diabetes in Women
Alaoui, S. S., Aksasse, B., & Farhaoui, Y. (2019) · Lecture notes in networks and systems · doi:10.1007/978-3-030-23672-4_6 · Cited by 22 · cited 1× · see citing works
“…of healthcare like urology (Zhang, et al., 2019; Otunaiya & Muhammad, 2019), toxicology (Zhang, et al., 2018, Zhang, et al., 2019), endocrinology (Alaoui, et al. 2019), neurology (Zhang & Ma., 2019), cardiology (Joshi, et al., 2019; Salman, 2019; Feeny, et al., 2019) or psychiatry (Guimarães, et…”
A Hybrid Model Based on Fuzzy Rules to Act on the Diagnosed of Autism in Adults
Guimarães, A. J., Araújo, V. J. S., Araújo, V. S., Batista, L. O., & Souza, P. V. D. C. (2019) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-19823-7_34 · Cited by 21 · see citing works
Prabhakararao, E., & Dandapat, S. (2019) · National Conference on Communications · doi:10.1109/ncc.2019.8732238 · Cited by 20 · cited 1× · see citing works
“…Aro, et al., 2019; Seker, et al., 2019), severity of a disease, including chronical diseases (e.g. diabetes or Alzheimer’s disease) (Karun, et. al., 2019; Bucholc, et al., 2019) and mortality rates (e.g. myocardial infarction or perinatal stroke) (Prabhakararao & Dandapat, 2019; Gao, et al., 2019).”
BackgroundPerformance of Datamining Techniques in the Prediction of Chronic Kidney Disease
Otunaiya, K. A., & Muhammad, G. (2019) · Computer Science and Information Technology · doi:10.13189/csit.2019.070203 · Cited by 20 · cited 1× · see citing works
“…the approaches like Regression models or Naive Bayes classifier are still used in different fields of healthcare like urology (Zhang, et al., 2019; Otunaiya & Muhammad, 2019), toxicology (Zhang, et al., 2018, Zhang, et al., 2019), endocrinology (Alaoui, et al. 2019), neurology (Zhang & Ma.,…”
MethodHeart attack mortality prediction: an application of machine learning methods
Salman, I. (2019) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.3906/elk-1811-4 · Cited by 20 · cited 1× · see citing works
“, 2019; Zhang, Ma, Liu, Ren, & Ding, 2018), endocrinology (Alaoui, Aksasse, & Farhaoui, 2019), neurology (Zhang & Ma, 2019), cardiology (Doshi-Velez & Kim, 2018; Feeny et al., 2019; Salmam, 2019), or psychiatry (Guimar~aes, Araujo, Araujo, Batista, & de Campos Souza, 2019; Obeid et al.”
Using Visual Interpretation of Small Ensembles in Microarray Analysis
Štiglic, G., Mertik, M., Podgorelec, V., et al. (2006) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms.2006.169 · Cited by 16 · cited 1× · see citing works
“In contrast, local interpretation helps users to understand a prediction for a specific individual or about a small, specific region of the trained prediction function (Hall, et al, 2019; Stiglic, et al., 2006).”
BackgroundThe Impact of the EU’s New Data Protection Regulation on AI
Wallace, N., & de Castro, D. A. (2018) · Cited by 16 · see citing works
Personalized Modeling for Prediction with Decision-Path Models
Visweswaran, S., Ferreira, A., Ribeiro, G., de Oliveira, A. C. M., & Cooper, G. F. (2015) · PLoS ONE · doi:10.1371/journal.pone.0131022 · Cited by 15 · cited 1× · see citing works
“, 2019) - Set of rules (for specific individual; Visweswaran, Ferreira, Ribeiro, Oliveira, & Cooper, 2015), - Decision trees (by tree -decomposition; Visweswaran et al., 2015), - Visual analytics-based approaches (interactive visualization techniques for interpretation focusing on individual prediction), - k-Nearest n…”
BackgroundComparative Analysis of Prediction Algorithms for Diabetes
Karun, S., Raj, A., & Attigeri, G. V. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-981-13-0341-8_16 · Cited by 13 · cited 1× · see citing works
“…(Oladele Aro, et al., 2019; Seker, et al., 2019), severity of a disease, including chronical diseases (e.g. diabetes or Alzheimer’s disease) (Karun, et. al., 2019; Bucholc, et al., 2019) and mortality rates (e.g. myocardial infarction or perinatal stroke) (Prabhakararao & Dandapat, 2019;…”
BackgroundA Soft Measurement Technique for Searching Significant Subsets of Prostate Cancer Prognostic Markers
Şeker, H., Odetayo, M. O., Petrov́ić, D., et al. (2000) · Physica-Verlag eBooks · doi:10.1007/978-3-7908-1844-4_52 · Cited by 9 · see citing works
Homogenous Ensembles on Data Mining Techniques for Breast Cancer Diagnosis
Aro, T. O., Akande, H. B., Jibrin, M. B., et al. (2019) · doi:10.5281/zenodo.13770946 · Cited by 6 · see citing works
Bias in the intervention in prediction measure in random forests: illustrations and recommendations
Nembrini, S. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty959 · Cited by 5 · cited 1× · see citing works
“These models can in most cases only be interpreted by model-agnostic approaches, where some exceptions exist, such as variable importance in Random Forests, but one has to be careful depending on the method used since bias could be possibly present making the ranking unreliable (Nebrini, 2019).”
MethodMichalopoulos, G., Chen, H., Yang, Y., et al. (2019) · Journal of Computational Vision and Imaging Systems · Cited by 2 · see citing works
Decision Support System for Heart Disease Diagnosing Using K-NN Algorithm
Yuwono, T., Setiawan, N. A., Nugroho, A. P., et al. (2015) · Proceeding of the Electrical Engineering Computer Science and Informatics · doi:10.11591/eecsi.v2.776 · Cited by 2 · see citing works
Automated General Movement Assessment for Perinatal Stroke Screening in Infants
Gao, Y., Long, Y., Guan, Y., Basu, A., Baggaley, J., & Plötz, T. (2019) · Smart Assisted Living · doi:10.1007/978-3-030-25590-9_8 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 11
- Ahmad A. M., 2018, IEEE intelligent informatics bulletin, 1
- Ahmad M. A., 2018, Interpretable machine learning in healthcare
- Hall P., 2018, An Introduction to machine learning interpretability: An applied perspective on fairness, accountability, transparency, and explainable AI
- Hall P., 2019, Machine learning interpretability with H2O driverless AI
- Hamilton W., 2017, Advances in neural information processing systems, 1024
- Lei T., 2017, Interpretable neural models for natural language processing (doctoral dissertation)
- Lipton Z. C.(2016). The mythos of model interpretability.arXiv preprint arXiv:1606.03490. · arXiv
- Maaten L. V. D., 2008, Visualizing data using t‐SNE, Journal of Machine Learning Research, 9, 2579
- Molnar C., 2020, Interpretable machine learning: A guide for making black box models explainable
- Tomasello M., 1999, The cultural origins of human cognition
- van derMaatenL. (2018). Dos and Don'ts of using t‐SNE to Understand Vision Models CVPR 2018 Tutorial on Interpretable Machine Learning for Computer Vision. Retrieved fromhttp://deeplearning.csail.mit.edu/slide_cvpr2018/laurens_cvpr18tutorial.pdf.
Cited by
Sorted by citations
Scientific discovery in the age of artificial intelligence.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06221-2 · Cited by 1,823 · see citing works
Self-citationAhmed, I., Jeon, G., & Piccialli, F. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3146552 · Cited by 853 · see citing works
“Several techniques have been introduced in the literature to analyze explainability models [46], [47].”
BackgroundExplainable Deep Learning Models in Medical Image Analysis
Singh, A., Sengupta, S., & Lakshminarayanan, V. (2020) · Journal of Imaging · doi:10.3390/jimaging6060052 · Cited by 718 · see citing works
“Different kinds of taxonomies and classification methods are discussed briefly here and a detailed analysis of the taxonomies can be found in [8,9] and a flow chart for them is shown in 1.”
BackgroundNanoparticle synthesis assisted by machine learning
Tao, H., Wu, T., Aldeghi, M., Wu, T., Aspuru‐Guzik, A., & Kumacheva, E. (2021) · Nature Reviews Materials · doi:10.1038/s41578-021-00337-5 · Cited by 558 · see citing works
“This lack of inter pretability makes the extraction of scientific understand ing from ML models difficult and may limit trust in these algorithms (in particular, in medical applications 174,175 ).”
BackgroundTrust in AI: progress, challenges, and future directions
Afroogh, S., Akbari, A., Malone, E., Kargar, M., & Alambeigi, H. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-04044-8 · Cited by 618 · see citing works
“Intrinsic interpretability refers to techniques that are intrinsically inter-pretable due to their structure (Ai and Narayanan, 2021; Pintelas et al., 2020; Stiglic et al., 2020).”
Interpretable machine learning for building energy management: A state-of-the-art review
Chen, Z., Xiao, F., Guo, F., & Yan, J. (2023) · Advances in Applied Energy · doi:10.1016/j.adapen.2023.100123 · Cited by 357 · see citing works
Machine Learning Toxicity Prediction: Latest Advances by Toxicity End Point
Cavasotto, C. N., & Scardino, V. (2022) · ACS Omega · doi:10.1021/acsomega.2c05693 · Cited by 226 · see citing works
Explainable and interpretable artificial intelligence in medicine: a systematic bibliometric review
Frasca, M., La Torre, D., Pravettoni, G., & Cutica, I. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00114-7 · Cited by 190 · see citing works
Cembrowska-Lech, D., Krzemińska, A., Miller, T. I., Nowakowska, A., Adamski, C., Radaczyńska, M., Mikiciuk, G., & Mikiciuk, M. (2023) · Biology · doi:10.3390/biology12101298 · Cited by 193 · see citing works
“…can be a significant limitation, particularly in a scientific context where understanding the underlying biological mechanisms is crucial [197,198]; • Overfitting: AI and ML models, particularly more complex models, such as deep learning models, can be prone to overfitting, where they…”
BackgroundInterpretable and explainable machine learning: A methods‐centric overview with concrete examples
Marcinkevičs, R., & Vogt, J. E. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1493 · Cited by 180 · see citing works
AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications
Chen, P., Wu, L., & Wang, L. (2023) · Applied Sciences · doi:10.3390/app131810258 · Cited by 155 · see citing works
A Comparative Analysis of LIME and SHAP Interpreters With Explainable ML-Based Diabetes Predictions
Ahmed, S., Kaiser, M. S., Hossain, M. S., & Andersson, K. (2024) · IEEE Access · doi:10.1109/access.2024.3422319 · Cited by 168 · see citing works
“The authors give an overview of interpretability approaches and examples of practical machine learning interpretability in several healthcare settings, including improving the effectiveness of screening for particular diseases and predicting outcomes related to health [36].”
Abdullah, T. A. A., Zahid, M. S. M., & Ali, W. (2021) · Symmetry · doi:10.3390/sym13122439 · Cited by 134 · see citing works
Shapley variable importance cloud for interpretable machine learning
Ning, Y., Ong, M. E. H., Chakraborty, B., Goldstein, B. A., Ting, D. S. W., VAUGHAN, R. D., & Liu, N. (2022) · Patterns · doi:10.1016/j.patter.2022.100452 · Cited by 132 · see citing works
A Comprehensive Survey of Deep Learning Approaches in Image Processing
Τρίγκα, Μ., & Dritsas, E. (2025) · Sensors · doi:10.3390/s25020531 · Cited by 159 · see citing works
“…of DL models and the challenge of deployment on edge devices [311–316] Interpretability challenges, especially in healthcare, finance, and law [317–322] Generalization and robustness challenges in DL models [323–328] Ethical implications, including bias and privacy concerns in AI systems…”
The Secondary Use of Electronic Health Records for Data Mining: Data Characteristics and Challenges
Sarwar, T., Seifollahi, S., Chan, J., Zhang, X., Aksakallı, V., Hudson, I. L., Verspoor, K. M., & Cavedon, L. (2022) · ACM Computing Surveys · doi:10.1145/3490234 · Cited by 129 · see citing works
“[29] provided an overview of interpretable prediction models used in the healthcare domain.”
MethodAyano, Y. M., Schwenker, F., Dufera, B. D., & Debelee, T. G. (2022) · Diagnostics · doi:10.3390/diagnostics13010111 · Cited by 132 · see citing works
“Likewise, Yang et al. [37], Stiglic et al. [38], and Jin et al. [41] did not provide reviews on the progress of interpretable techniques on ECG signal-based heart disease diagnosis.”
BackgroundInfluentialMolnar, C., König, G., Bischl, B., & Casalicchio, G. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00901-9 · Cited by 139 · see citing works
“Applications range from medicine (Boulesteix et al. 2020; Stiglic et al. 2020; Pintelas et al. 2020) and the social sciences (Stachl et al. 2020; Zhao et al. 2020) to ecology (Bair et al. 2013; Esselman et al. 2015; Obringer and Nateghi 2018).”
Machine learning in medicine: should the pursuit of enhanced interpretability be abandoned?
Yoon, C. H., Torrance, R., & Scheinerman, N. (2021) · Journal of Medical Ethics · doi:10.1136/medethics-2020-107102 · Cited by 117 · see citing works
Explainable machine learning framework to predict personalized physiological aging
Bernard, D., Doumard, E., Ader, I., Kémoun, P., Pagès, J., Galinier, A., Cussat‐Blanc, S., Furger, F., Ferrucci, L., Aligon, J., et al. (2023) · Aging Cell · doi:10.1111/acel.13872 · Cited by 101 · see citing works
“Explainability better sustains evidence- based acceptability and could allow for data- driven personalized medical- management (Stiglic et al., 2020).”
BackgroundNaser, M. A., Majeed, A. A., Alsabah, M. Q., Al-Shaikhli, T. R., & Kaky, K. M. (2024) · Algorithms · doi:10.3390/a17020078 · Cited by 124 · see citing works
Equity within AI systems: What can health leaders expect?
Gurevich, E., Hassan, B. E., & Morr, C. E. (2022) · Healthcare Management Forum · doi:10.1177/08404704221125368 · Cited by 108 · see citing works
Saux, P., Bauvin, P., Raverdy, V., Teigny, J., Verkindt, H., Soumphonphakdy, T., Debert, M., Jacobs, A., Jacobs, D., Monpellier, V. M., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00135-8 · Cited by 100 · see citing works
Moreno-Sánchez, P. A. (2023) · IEEE Access · doi:10.1109/access.2023.3264270 · Cited by 106 · see citing works
“clear or easy to understand, allows healthcare experts to make reasonable and data-driven decisions that would enhance the clinical adoption of AI models and their acceptance [12].”
BackgroundBranda, F., & Scarpa, F. (2024) · Antibiotics · doi:10.3390/antibiotics13060502 · Cited by 92 · see citing works
“In addition to the need for high-quality data and experimental validation of computational results, the transparency and interpretability of machine learning algorithms remain a priority to ensure the safety and efficacy of newly developed drugs [12].”
BackgroundNielsen, R. L., Monfeuga, T., Kitchen, R. R., Egerod, L., Leal, L. G., Schreyer, A. T. H., Gade, F. S., Sun, C. W., Helenius, M., Simonsen, L., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-46663-4 · Cited by 96 · see citing works
Hong, N., Cho, S. W., Shin, S. J., Lee, S., Jang, S. A., Roh, S., Lee, Y. H., Rhee, Y., Cummings, S. R., Kim, H., et al. (2020) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.4814 · Cited by 88 · see citing works
“Interpretability is important for machine-learning models, especially in health care; at least partly understanding the decision-making by the model can improve model explainability and provide novel insights.((36)) For this reason, GRAD-CAM figures were created to provide visual explanations for the spineradiography…”
MethodMoreno-Sánchez, P. A., Isla, G. G., Corino, V., Vehkaoja, A. T., Brukamp, K., van Gils, M., & Mainardi, L. T. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108235 · Cited by 92 · see citing works
Khan, A. A., Shaikh, Z. A., Baitenova, L. M., Mutaliyeva, L. M., Моисеев, Н. А., Mikhaylov, A. Y., Laghari, A. A., Idris, S. A., & Alshazly, H. A. (2021) · Electronics · doi:10.3390/electronics10243083 · Cited by 84 · see citing works
Wysocka, M., Wysocki, O., Zufferey, M., Landers, D., & Freitas, A. (2022) · BMC Bioinformatics · doi:10.1186/s12859-023-05262-8 · Cited by 75 · see citing works
Nickson, D., Meyer, C., Walasek, L., & Toro, C. T. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02341-x · Cited by 81 · see citing works
Raab, D., Theissler, A., & Spiliopoulou, M. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07809-x · Cited by 79 · see citing works
“In addition, this constitutes a way for evaluating an explanation [97].”
BackgroundBernal, J., & Mazo, C. (2022) · Applied Sciences · doi:10.3390/app122010228 · Cited by 70 · see citing works
Bloch, L., Friedrich, C. M., & Initiative, F. T. A. D. N. (2021) · Alzheimer's Research & Therapy · doi:10.1186/s13195-021-00879-4 · Cited by 68 · see citing works
Kaya, C. (2025) · Food and Energy Security · doi:10.1002/fes3.70050 · Cited by 77 · see citing works
AI-enhanced biomedical micro/nanorobots in microfluidics
Dong, H., Lin, J., Tao, Y., Yuan, J. Y., Sun, L., Li, W. J., & Sun, H. (2024) · Lab on a Chip · doi:10.1039/d3lc00909b · Cited by 68 · see citing works
Moreno-Sánchez, P. A. (2023) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2023.1219586 · Cited by 72 · see citing works
“Thus, XAI enables healthcare experts to make more informed and data-driven decisions, providing personalized and trustworthy treatments and diagnoses (13).”
BackgroundQi, W., Wang, Y., Wang, Y., Huang, S., Li, C., Jin, H., Zuo, J., Cui, X., Wei, Z., Guo, Q., et al. (2025) · Journal of Translational Medicine · doi:10.1186/s12967-025-06289-6 · Cited by 68 · see citing works
“Interpretability is generally defined as the ease with which humans can comprehend and explain the process of the ML model’s predictions [41].”
Explainable Artificial Intelligence for Sarcasm Detection in Dialogues
Kumar, A., Dikshit, S., & de Albuquerque, V. H. C. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/2939334 · Cited by 65 · see citing works
“One way to apply interpretable ML is by using models that are intrinsically interpretable and known to be easy for humans to understand such as linear/logistic regression, decision trees, and K-nearest neighbors [13].”
MethodExplainable Artificial Intelligence: A Perspective on Drug Discovery
Qadri, Y. A., Shaikh, S., Ahmad, K., Choi, I., Kim, S. W., & Vasilakos, A. V. (2025) · Pharmaceutics · doi:10.3390/pharmaceutics17091119 · Cited by 80 · see citing works
“In addition to identifying essential input features, these methods can often only roughly approximate the kind of relationships that exist between features and outputs in many domains, excluding image and text analysis [41].”
Interpretability of Machine Learning Solutions in Public Healthcare: The CRISP-ML Approach
Kolyshkina, I., & Simoff, S. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.660206 · Cited by 60 · see citing works
“In this context, public healthcare sets priority requirements toward the robustness, security (Qayyum et al., 2021), and interpretability (Stiglic et al., 2020) of ML solutions.”
BackgroundJiang, Z., Seyedi, S. S., Griner, E. L., Abbasi, A., Rad, A. B., Kwon, H., Cotes, R. O., & Clifford, G. D. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3352075 · Cited by 75 · see citing works
“Interpretable machine learning methods [109] could be applied to explain which modality, which feature, and which temporal section contribute to the system outputs.”
Expectations for Artificial Intelligence (AI) in Psychiatry
Monteith, S., Glenn, T., Geddes, J. R., Whybrow, P. C., Achtyes, E. D., & Bauer, M. (2022) · Current Psychiatry Reports · doi:10.1007/s11920-022-01378-5 · Cited by 56 · see citing works
Development of an Explainable Prediction Model of Heart Failure Survival by Using Ensemble Trees
Moreno-Sánchez, P. A. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378460 · Cited by 57 · see citing works
“Explainable ML models, also known as interpretable ML models, allow healthcare experts to make reasonable and data-driven decisions to provide personalized treatment that can ultimately lead to high quality of service in healthcare [11].”
BackgroundAhmed, Z., Sun, K., Shelly, M. A., & Mu, L. (2021) · Scientific Reports · doi:10.1038/s41598-021-03198-8 · Cited by 52 · see citing works
“Higher interpretable models are easier to understand and explain the contribution of features in predictions 30 .”
BackgroundUkoba, K. O., Onisuru, O. R., Jen, T., Madyira, D. M., & Olatunji, K. O. (2025) · Environmental Science and Pollution Research · doi:10.1007/s11356-025-36356-w · Cited by 64 · see citing works
“Interpretability and transparency are crucial aspects of AI model development, particularly in predicting climate change impacts (Stiglic et al. 2020).”
Jui, S., Deo, R., Barua, P., Devi, A., Soar, J., & Acharya, U. R. (2023) · IEEE Access · doi:10.1109/access.2023.3294473 · Cited by 45 · see citing works
“One major reason is the lack of interpretability and explainability of any AI model [111].”
BackgroundRecent advancements and applications of deep learning in heart failure: Α systematic review
Πετμεζάς, Γ., Papageorgiou, V. Ε., Vassilikos, V. P., Pagourelias, E. D., Tsaklidis, G. M., Katsaggelos, A. K., & Maglaveras, N. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108557 · Cited by 50 · see citing works
Wu, Y., Lin, S., Shi, K., et al. (2022) · Environmental Science and Pollution Research · doi:10.1007/s11356-022-18913-9 · Cited by 45 · see citing works
“Traditional machine learning algorithms are frequently regarded as “black boxes” that are limited to effectively explain the final predictions (Stiglic et al. 2020).”
BackgroundBhattacharya, A., Ooge, J., Štiglic, G., & Verbert, K. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584075 · Cited by 44 · see citing works
“In healthcare, XAI empowers medical experts to make data-driven decisions using ML, resulting in a higher quality of medical services [54] and can impact its trust and reliance [8, 56].”
BackgroundMethodSelf-citationAnsari, Z. A., Tripathi, M. M., & Ahmed, R. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00307-8 · Cited by 50 · see citing works
Machine learning in toxicological sciences: opportunities for assessing drug toxicity
Tonoyan, L., & Siraki, A. G. (2024) · Frontiers in Drug Discovery · doi:10.3389/fddsv.2024.1336025 · Cited by 43 · see citing works
Personalized hypertension treatment recommendations by a data-driven model
Hu, Y., Huerta, J., Cordella, N., Mishuris, R. G., & Paschalidis, I. C. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02137-z · Cited by 41 · see citing works
“Clinicians and patients cannot make medical decisions just based on a black-box model designed and evaluated in a narrower scenario, which may not have access to all information about a specific patient [20].”
BackgroundSu, P.-Y., Wei, Y., Luo, H., Liu, C., Huang, W., Chen, K., Lin, C., Wei, H., & Lee, T. (2022) · JMIR Medical Informatics · doi:10.2196/32508 · Cited by 39 · see citing works
“The local interpretable model-agnostic explanation, introduced in 2016, approximates a black-box model using a simple linear surrogate model locally [31,32].”
González-Alday, R., García-Cuesta, E., Maojo, & Kulikowski, C. (2023) · Applied Sciences · doi:10.3390/app131910778 · Cited by 38 · see citing works
C-SHAP: A Hybrid Method for Fast and Efficient Interpretability
Ranjbaran, G., Recupero, D. R., Roy, C. K., & Schneider, K. A. (2025) · Applied Sciences · doi:10.3390/app15020672 · Cited by 39 · see citing works
Self-supervised learning in drug discovery
Chen, Y., Wang, Z., Wang, J., Chu, Y., Zhang, Q., Li, Z. A., & Zeng, X. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4453-4 · Cited by 40 · see citing works
“(2) Complexity and interpretability of models [112]: drug discovery involves intricate biological mechanisms and constantly evolving biomarkers.”
Multi-omics disease module detection with an explainable Greedy Decision Forest
Pfeifer, B., Baniecki, H., Saranti, A., Biecek, P., & Holzinger, A. (2022) · Scientific Reports · doi:10.1038/s41598-022-21417-8 · Cited by 36 · see citing works
“However, it also means easier understanding and better explanation of future predictions for end users, thus promoting reasonable and data-driven decisions to make personalized choices that can ultimately lead to higher quality healthcare services [5].”
BackgroundForecasting venue popularity on location‐based services using interpretable machine learning
Wang, L., Gopal, R., Shankar, R., & Pancras, J. (2022) · Production and Operations Management · doi:10.1111/poms.13727 · Cited by 35 · see citing works
“…are two types of interpretable approaches: (1) local interpretability, which focuses on personalized interpretation (i.e., local interpretable model-agnostic explanations), and (2) global interpretability, which summarizes prediction models on a population level (i.e., SHAP) (Stiglic et al., 2020).”
BackgroundA Comparison of Infectious Disease Forecasting Methods across Locations, Diseases, and Time
Dixon, S. L., Keshavamurthy, R., Farber, D. H., et al. (2022) · Pathogens · doi:10.3390/pathogens11020185 · Cited by 35 · see citing works
“ML-based ensemble models, such as random forest (RF) and extreme gradient boost (XGB), are particularly popular in healthcare research because of their ease of implementation and interpretation [8,9].”
BackgroundInam, S. A. (2025) · Discover Public Health · doi:10.1186/s12982-025-01167-4 · Cited by 36 · see citing works
“In contrast, machine learning and hybrid approaches are better equipped to handle non-linear relationships and complex systems [23, 24].”
Artificial intelligence in rheumatoid arthritis: potential applications and future implications
Gilvaz, V. J., & Reginato, A. M. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1280312 · Cited by 36 · see citing works
“Although efforts are being made to overcome some of these challenges (36, 57), it is important to temper our expectations with the advances being made.”
BackgroundZeng, T., Jiang, H., Liu, Y., & Bai, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126815 · Cited by 33 · see citing works
Gu, H., Yang, C., Haeri, M., Wang, J., Tang, S., Yan, W., He, S., Williams, C. K., Magaki, S., & Chen, X. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580694 · Cited by 36 · see citing works
“Since one criticism of deep learning models in pathology is that there is a lack of interpretability [72], explainable AI (XAI) techniques have been utilized to make AI “transparent, understandable and reliable” to pathologist users [60].”
MethodSig-Lime: A Signal-Based Enhancement of Lime Explanation Technique
Abdullah, T. A. A., Zahid, M. S. M., Turki, A. F., Ali, W., Jiman, A. A., Abdulaal, M. J., Sobahi, N. M., & Attar, E. T. (2024) · IEEE Access · doi:10.1109/access.2024.3384277 · Cited by 32 · see citing works
“However, the lack of interpretability in deep learning models poses a significant obstacle to their real-world applicability [9].”
BackgroundZhang, X., Rong, X., & Luo, H. (2024) · Frontiers in Rehabilitation Sciences · doi:10.3389/fresc.2024.1246773 · Cited by 31 · see citing works
“Second, uninterpretable models can be dangerous in healthcare settings as incorrect interpretation can lead to inappropriate treatment decisions (46).”
BackgroundAhmad, I., Brahmacharimayum, A., Ali, R. H., Khan, T. A., & Ahmad, M. O. (2025) · JMIR Mental Health · doi:10.2196/72038 · Cited by 32 · see citing works
Adversarial Attacks in Explainable Machine Learning: A Survey of Threats Against Models and Humans
Vadillo, J., Santana, R., & Lozano, J. A. (2024) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1567 · Cited by 38 · see citing works
“…et al. 2022) (e.g., it might be more interesting to know why a certain system will fail than just knowing that it will fail) or medical diagnosis (Stiglic et al. 2020) (e.g., discovering why a model has diagnosed a patient as being at high risk for a particular disease might be the main priority…”
InfluentialZhang, C., & Liu, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-22972-6 · Cited by 33 · see citing works
Messner, W. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110729 · Cited by 28 · see citing works
“However, the uptake of machine learning models in research and real-life decision making remains limited, because those models hide their underlying mechanisms in a black box [11,12].”
BackgroundSherazi, S. W. A., Zheng, H., & Lee, J. Y. (2023) · Sensors · doi:10.3390/s23031351 · Cited by 29 · see citing works
“Furthermore, regression models and decision trees provide limited interpretability, specifically in data nonlinearity [33].”
BackgroundLuu, J. H., Борисенко, Е. А., Przekop, V., Patil, A., Forrester, J. D., & Choi, J. (2024) · Trauma Surgery & Acute Care Open · doi:10.1136/tsaco-2023-001222 · Cited by 29 · see citing works
Woodman, R. J., Bryant, K., Sorich, M. J., Pilotto, A., & Mangoni, A. A. (2021) · Journal of Medical Internet Research · doi:10.2196/26139 · Cited by 28 · see citing works
“A common criticism of ML approaches compared with standard statistical approaches is that they rely on blackbox algorithms in which the source of the improved performance is not readily transparent [25,26].”
Interpreting the role of nuchal fold for fetal growth restriction prediction using machine learning
Teng, L. Y., Mattar, C. N. Z., Biswas, A., Hoo, W. L., & Saw, S. N. (2022) · Scientific Reports · doi:10.1038/s41598-022-07883-0 · Cited by 29 · see citing works
Pérez‐Millan, A., Contador, J., Juncà‐Parella, J., Bosch, B., Borrell, L., Tort‐Merino, A., Falgàs, N., Borrego‐Écija, S., Bargalló, N., Rami, L., et al. (2023) · Human Brain Mapping · doi:10.1002/hbm.26205 · Cited by 30 · see citing works
“In ML, this represents a crucial step for developing algorithms that are interpretable at the biological and pathological levels (Stiglic et al., 2020).”
BackgroundExtendable and explainable deep learning for pan-cancer radiogenomics research
Liu, Q., & Hu, P. (2022) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2021.102111 · Cited by 27 · see citing works
Patient similarity analytics for explainable clinical risk prediction
Fang, H. S. A., Tan, N. C., Tan, W. Y., Oei, R. W., Lee, M. L., & Hsu, W. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01566-y · Cited by 27 · see citing works
“Explainability is de�ned as the extent of which a model’s prediction process can be described, while interpretability is de�ned as the degree to which a user can understand the predictions made by a model (20–22).”
BackgroundA Machine Learning Approach to Predict the Added-Sugar Content of Packaged Foods
Davies, T., Louie, J. C. Y., Ndanuko, R. N., Barbieri, S., Perez‐Concha, O., & Wu, J. H. Y. (2021) · Journal of Nutrition · doi:10.1093/jn/nxab341 · Cited by 29 · see citing works
Wu, J., Wang, Q., Wang, Z., & Zhou, Z. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105104 · Cited by 27 · see citing works
Levy, J. J., Lima, J. F., Miller, M. W., Freed, G., O'Malley, A., & Emeny, R. (2022) · Frontiers in Medical Technology · doi:10.3389/fmedt.2022.926667 · Cited by 17 · see citing works
García-Vicente, C., Chushig-Muzo, D., Mora-Jiménez, I., Fabelo, H. A., Gram, I. T., Løchen, M., Granja, C., & Soguero-Ruíz, C. (2023) · Applied Sciences · doi:10.3390/app13074119 · Cited by 28 · see citing works
Beyond prediction: An integrated post-hoc approach to interpret complex model in hydrometeorology
Huang, F., Shangguan, W., Li, Q., Li, L., & Zhang, Y. (2023) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2023.105762 · Cited by 28 · see citing works
Moreno-Sánchez, P. A. (2021) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi52183.2021.00100 · Cited by 26 · see citing works
Comparative optimization of polysaccharide-based nanoformulations for cardiac RNAi therapy
Gao, H. J., Li, S., Lan, Z., Pan, D., Naidu, G. S., Peer, D., Ye, C., Chen, H., Ma, M., Liu, Z., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-49804-x · Cited by 26 · see citing works
“Our goal was rather to extract potential scienti fi c understanding from the screening process and to provide perspectives for interpretation methods, as this interpretability holds particular value formedicalapplications 12,24 .”
BackgroundHuman Activity Recognition Using Machine Learning Algorithms Based on IMU Data
Nia, N. G., Kaplanoğlu, E., Nasab, A., & Qin, H. H. (2023) · International Conference on Bio-engineering for Smart Technologies (BioSMART) · doi:10.1109/biosmart58455.2023.10162095 · Cited by 25 · see citing works
“Furthermore, the KNN model may have limited interpretability due to its reliance on nearby data points [39], making it more difficult to comprehend the factors contributing to its predictions than the more interpretable RFC and DTC models.”
BackgroundSaber, R., Henault, D., Messaoudi, N., Rebolledo, R., Montagnon, E., Soucy, G., Stagg, J., Tang, A., Turcotte, S., & Kadoury, S. (2023) · Journal of Translational Medicine · doi:10.1186/s12967-023-04175-7 · Cited by 26 · see citing works
“Interpretability techniques could be divided into two major categories: model-specific and model-agnostic [36].”
BackgroundMulti-objective design space exploration using explainable surrogate models
Palar, P. S., Dwianto, Y. B., Zuhal, L. R., et al. (2024) · Structural and Multidisciplinary Optimization · doi:10.1007/s00158-024-03769-z · Cited by 24 · see citing works
“To take examples outside engineering design, explainable models are important to allow reasonable data-driven decision models in health-care (Stiglic et al. 2020), predictive maintenance (Bukhsh et al. 2019; Vollert et al. 2021), and genetics (Azodi et al. 2020).”
Griffin, A. C., Wang, K. H., Leung, T. I., & Facelli, J. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104693 · Cited by 19 · see citing works
Systematic Mapping Study of AI/Machine Learning in Healthcare and Future Directions
Parashar, G., Chaudhary, A., & Rana, A. (2021) · SN Computer Science · doi:10.1007/s42979-021-00848-6 · Cited by 23 · see citing works
“As proposed in [10, 11] interpretable ML models helps develop a reasonable and data-driven decision support system that results in personalised decisions.”
BackgroundCharacterization of Synthetic Health Data Using Rule-Based Artificial Intelligence Models
Lenatti, M., Paglialonga, A., Orani, V., Ferretti, M., & Mongelli, M. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3236722 · Cited by 23 · see citing works
“When dealing with health data, XAI methods are particularly promising as they can help healthcare experts enter the logic of the machine learning process and extract knowledge about the mechanisms underlying the observed phenomena in a meaningful and transparent way, so that synthetic data can be validated against ava…”
BackgroundOmori, T., Suzuki, S., Michibayashi, K., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-33503-6 · Cited by 22 · see citing works
“This means that for scientific purposes, the deep learning approaches have some disadvantages such as low explainability and interpretability owing to their complex mathematical frameworks 27–29 .”
Wang, Y., He, L., He, L., et al. (2024) · International Journal of Pharmaceutics · doi:10.1016/j.ijpharm.2024.124128 · Cited by 22 · see citing works
Artificial Intelligence in Uveitis: Innovations in Diagnosis and Therapeutic Strategies
Murugan, S., Sanjay, S., Somanath, A., Mahendradas, P., Patil, A., Kaur, K., & Gurnani, B. (2024) · Clinical ophthalmology · doi:10.2147/opth.s495307 · Cited by 22 · see citing works
Chen, B., Hu, J., & Wang, Y. (2024) · npj Climate and Atmospheric Science · doi:10.1038/s41612-023-00559-0 · Cited by 22 · see citing works
What is the Optimal Attribution Method for Explainable Ophthalmic Disease Classification?
Singh, A., Sengupta, S., Balaji, J. J., Mohammed, A. R., Faruq, I., Jayakumar, V., Zelek, J., & Lakshminarayanan, V. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-63419-3_3 · Cited by 24 · see citing works
Deep Learning Based Prediction of Pulmonary Hypertension in Newborns Using Echocardiograms
Ragnarsdóttir, H., Özkan, E., Michel, H., Chin-Cheong, K., Manduchi, L., Wellmann, S. M., & Vogt, J. E. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-01996-x · Cited by 20 · see citing works
“Indeed, the lack of explainability is a critical factor that limits the wider adoption of ML in healthcare, as without it, medical practitioners often find it challenging to trust ML models (Stiglic et al., 2020).”
BackgroundOptimizing Healthcare Outcomes through Data-Driven Predictive Modeling
Sunny, M. N. M., Saki, M. B. H., Nahian, A. A., Ahmed, S. W., Shorif, M. N., Atayeva, J., & Rizvi, S. W. A. (2024) · Journal of Intelligent Learning Systems and Applications · doi:10.4236/jilsa.2024.164019 · Cited by 27 · see citing works
Demirtürk, D., Mintemur, Ö., & Arslan, A. (2025) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-025-10217-7 · Cited by 22 · see citing works
I-SPLIT: Deep Network Interpretability for Split Computing
Cunico, F., Capogrosso, L., Setti, F., Damiano Carra, B., Fummi, F., & Cristani, M. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956625 · Cited by 21 · see citing works
“It can be distinguished into intrinsic and post-hoc [29]: intrinsic interpretability applies to models that have a simple analytical form such as short decision trees or sparse linear models [30]; post-hoc interpretability”
BackgroundPaudel, P., Karna, S. K., Saud, R., Regmi, L., Thapa, T. B., & Bhandari, M. (2023) · International Conference on Networking, Systems and Security (NSysS) · doi:10.1145/3629188.3629193 · Cited by 20 · see citing works
“To overcome the black box approach of the AI models and make understandable interpretations, the study explores the Explainable AI (XAI) algorithm: Local Interpretable Model-Agnostic Explanations (LIME) serve as interpretable tools that provide transparency to machine learning predictions [30].”
Method
- Authors
- Stiglic, G., Kocbek, P., Fijacko, N., Zitnik, M., Verbert, K., & Cilar, L.
- Affiliations
- Faculty of Health Sciences University of Maribor Maribor Slovenia
- University of Maribor (SI)
- Department of Biomedical Informatics Harvard University Cambridge Massachusetts USA
- KU Leuven (BE)
- Venue
- WIREs Data Mining and Knowledge Discovery
- Publisher
- Wiley
- Published
- 2020-06-29
- Volume
- 10
- Issue
- 5
- ISSN
- 1942-4787 (ISSN-L)
- 1942-4795
- OpenAlex
- W3039232647
- Semantic Scholar
- 9b435ea80b6f90800339b3bffeafab6c990caa88
- Cited by
- 541
- References
- 77
- Citations by year
- 2026: 70 · 2025: 122 · 2024: 135 · 2023: 87 · 2022: 75 · 2021: 43 · 2020: 9
- FWCI
- 21.02
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Lirias (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Javna Agencija za Raziskovalno Dejavnost RS (N2‐0101, P2‐0057)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- model interpretability, local interpretability, global interpretability, model-agnostic explanation, clinical decision support, health outcome prediction, treatment optimization, screening efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
