Opening the black box of artificial intelligence for clinical decision support: A study predicting stroke outcome
Open accessPublished version of a preprint
The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
State-of-the-art machine learning (ML) artificial intelligence methods are increasingly leveraged in clinical predictive modeling to provide clinical decision support systems to physicians. Modern ML approaches such as artificial neural networks (ANNs) and tree boosting often perform better than more traditional methods like logistic regression. On the other hand, these modern methods yield a limited understanding of the resulting predictions. However, in the medical domain, understanding of applied models is essential, in particular, when informing clinical decision support. Thus, in recent years, interpretability methods for modern ML methods have emerged to potentially allow explainable predictions paired with high performance. To our knowledge, we present in this work the first explainability comparison of two modern ML methods, tree boosting and multilayer perceptrons (MLPs), to traditional logistic regression methods using a stroke outcome prediction paradigm. Here, we used clinical features to predict a dichotomized 90 days post-stroke modified Rankin Scale (mRS) score. For interpretability, we evaluated clinical features' importance with regard to predictions using deep Taylor decomposition for MLP, Shapley values for tree boosting and model coefficients for logistic regression. With regard to performance as measured by Area under the Curve (AUC) values on the test dataset, all models performed comparably: Logistic regression AUCs were 0.83, 0.83, 0.81 for three different regularization schemes; tree boosting AUC was 0.81; MLP AUC was 0.83. Importantly, the interpretability analysis demonstrated consistent results across models by rating age and stroke severity consecutively amongst the most important predictive features. For less important features, some differences were observed between the methods. Our analysis suggests that modern machine learning methods can provide explainability which is compatible with domain knowledge interpretation and traditional method rankings. Future work should focus on replication of these findings in other datasets and further testing of different explainability methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

![Fig 2 Graphical representation of the feature importance. The figure illustrates the features rating derived from the model-tailored interpretability methods for generalized linear model (GLM), Lasso, Elastic net, Catboost and multilayer perceptron (MLP). All models rated age and initial NIHSS consistently amongst the most important features. For less important features, results were more varied. For logistic regression techniques the results are given in weights, for Catboost in Shap(ley) values and for MLP in deep Taylor values that were normalized to the range [0,1]. The bar heights represent means and error bars represent standard deviation over samples (shuffles).](https://figures.cochat.ai/paper-figures/36db252b20184777a8d7d8620426022a/opening-the-black-box-of-artificial-intelligence-for-clinical-decision-support-a-figure-2.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Building on earlier efforts that compared different model outputs [70] or validated the intuitiveness of visual explanations (like those from LRP) [48,49], recent studies have begun to experimentally test the impact of different explanation strategies on user performance and reliance.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [70]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
A Caution Regarding Rules of Thumb for Variance Inflation Factors
O’brien, R. M. (2007) · Quality & Quantity · doi:10.1007/s11135-006-9018-6 · Cited by 11,217
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“a method which inherently provides information about feature importance, such as the weights of linear regression [36].”
MethodSantos, A. G., da Rocha, G. O., & de Andrade, J. B. (2019) · Scientific Reports · doi:10.1038/s41598-018-37186-2 · Cited by 8,495
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1603.02754 · Cited by 7,104
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“Explainability techniques for ANNs can be grouped into two categories: gradient-based methods such as saliency [27] and backward propagation methods such as deconvolution [28], guided backpropagation [29], SmoothGrad [30] and layer-wise relevance propagation (LRP) [31].”
MethodA guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312 · cited 1×
“ANNs have shown highly promising results in several areas of healthcare such as medical imaging, information extraction from medical texts and electronic health records, and combining several types of input into one predictive model [5].”
BackgroundOn Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · cited 1×
“[25–27].”
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447
Deep learning for healthcare: review, opportunities and challenges
Miotto, R., Wang, F., Wang, S., Jiang, X., & Dudley, J. T. (2018) · Briefings in Bioinformatics · doi:10.1093/bib/bbx044 · Cited by 3,076 · cited 1×
“[4–8].”
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665
Feiner, N., Motone, F., Meyer, A., & Kuraku, S. (2019) · Scientific Reports · doi:10.1038/s41598-019-40055-1 · Cited by 1,987
Stroke-Associated Pneumonia: Major Advances and Obstacles
Hannawi, Y., Hannawi, B., Rao, C. P. V., Suarez, J. I., & Bershad, E. M. (2013) · Cerebrovascular Diseases · doi:10.1159/000350199 · Cited by 1,656
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357
Evaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164
Martinón-Torres, M., Li, X., Xia, Y., Benzonelli, A., Bevan, A., Ma, S., Huang, J., Wang, L., Lan, D., Liu, J., et al. (2019) · Scientific Reports · doi:10.1038/s41598-019-40613-7 · Cited by 1,089
Explainable machine learning for scientific insights and discoveries
Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J. (2020) · IEEE Access · doi:10.1109/access.2020.2976199 · Cited by 1,040 · cited 1×
“Unfortunately, explainability in the modeling context is an illdefined term that can also have other meanings and several other terms such as interpretability and transparency are in use [34].”
BackgroundArtificial intelligence, bias and clinical safety
Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., & Tsaneva-Atanasova, K. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008370 · Cited by 956 · cited 2×
“Another reason is automation bias—an established cognitive bias—where users tend to believe what a machine is outputting without reflecting on the output [2].”
BackgroundClinical decision support in the era of artificial intelligence
Shortliffe, E. H., & Sepúlveda, M. J. (2018) · JAMA · doi:10.1001/jama.2018.17163 · Cited by 920 · cited 3×
“3) Domain knowledge supported interpretation leading to increased trust by the physicians, other healthcare professionals, and patients [12]: Some argue that black box approaches are unacceptable for clinical decision support from the physician ́s point-of-view [13] and from the patient’s point-of-view [14].”
BackgroundMachine learning in medicine: Addressing ethical challenges
Vayena, E., Blasimme, A., & Cohen, I. G. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002689 · Cited by 912 · cited 1×
“3) Domain knowledge supported interpretation leading to increased trust by the physicians, other healthcare professionals, and patients [12]: Some argue that black box approaches are unacceptable for clinical decision support from the physician ́s point-of-view [13] and from the patient’s point-of-view [14].”
BackgroundArtificial Intelligence and Black‐Box Medical Decisions: Accuracy versus Explainability
London, A. J. (2019) · Hastings Center Report · doi:10.1002/hast.973 · Cited by 801
Interpretable Machine Learning in Healthcare
Ahmad, M. A., Eckert, C., & Teredesai, A. (2018) · Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics · doi:10.1145/3233547.3233667 · Cited by 583 · cited 2×
“2) Better assessment of safety and fairness of medical products, especially regarding bias, during the regulatory process [12].”
BackgroundInterpreting Multiple Linear Regression: A Guidebook of Variable Importance
Nathans, L., Oswald, F. L., & Nimon, K. (2020) · doi:10.7275/5fex-b874 · Cited by 479
Prevalence of diabetes and its effects on stroke outcomes: A meta‐analysis and literature review
Lau, L., Lew, J., Borschmann, K., Thijs, V., & Ekinci, E. I. (2019) · Journal of Diabetes Investigation · doi:10.1111/jdi.12932 · Cited by 443
Tolerance and Variance Inflation Factor
Miles, J. (2014) · Wiley StatsRef: Statistics Reference Online · doi:10.1002/9781118445112.stat06593 · Cited by 352
Understanding the Results of Multiple Linear Regression: Beyond Standardized Regression Coefficients
Nimon, K. F., & Oswald, F. L. (2013) · Organizational Research Methods · doi:10.1177/1094428113493929 · Cited by 349
What Does Explainable AI Really Mean? A New Conceptualization of Perspectives
Doran, D., Schulz, S., & Besold, T. R. (2017) · arXiv · doi:10.48550/arxiv.1710.00794 · Cited by 333 · cited 1×
“the weights of a linear regression algorithm, in contrast to comprehensibility which is a symbolic representation of an output [35].”
BackgroundPredictive analytics with gradient boosting in clinical medicine
written on behalf of AME Big-Data Clinical Trial Collaborative Group, Zhang, Z., Zhao, Y., Canes, A., Steinberg, D., & Lyashevska, O. (2019) · Annals of Translational Medicine · doi:10.21037/atm.2019.03.29 · Cited by 312
Framing the challenges of artificial intelligence in medicine
Yu, K.-H., & Kohane, I. S. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008551 · Cited by 233
Machine learning for outcome prediction of acute ischemic stroke post intra-arterial therapy
Asadi, H., Dowling, R., Yan, B., & Mitchell, P. (2014) · PLoS ONE · doi:10.1371/journal.pone.0088225 · Cited by 218
Zhao, J., Feng, Q., Wu, P., Lupu, R. A., Wilke, R. A., Wells, Q. S., Denny, J. C., & Wei, W.-Q. (2019) · Scientific Reports · doi:10.1038/s41598-018-36745-x · Cited by 209
Complications following Acute Ischemic Stroke
Weimar, C., Roth, M. P., Zillessen, G., Glahn, J., Wimmer, M. L. J., Busse, O., Haberl, R. L., Diener, H.-C., & on behalf of the German Stroke Data Bank Collaborators, O. (2002) · European Neurology · doi:10.1159/000065512 · Cited by 199
An integrated machine learning approach to stroke prediction
Khosla, A., Cao, Y., Lin, C. C.-Y., Chiu, H.-K., Hu, J., & Lee, H. (2010) · Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1835804.1835830 · Cited by 199
Prediction model development of late-onset preeclampsia using machine learning-based methods
Jhee, J. H., Lee, S., Park, Y., Lee, S. E., Kim, Y. A., Kang, S.-W., Kwon, J.-Y., & Park, J. T. (2019) · PLOS ONE · doi:10.1371/journal.pone.0221202 · Cited by 188
Parsons, M. W., Christensen, S., McElduff, P., Levi, C. R., Butcher, K. S., De Silva, D. A., Ebinger, M., Barber, P. A., Bladin, C., Donnan, G. A., et al. (2010) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2010.3 · Cited by 161
A systematic review on deep learning architectures and applications
Khamparia, A., & Singh, K. M. (2019) · Expert Systems · doi:10.1111/exsy.12400 · Cited by 127
Bias in Artificial Intelligence
Nelson, G. S. (2019) · North Carolina Medical Journal · doi:10.18043/ncm.80.4.220 · Cited by 100 · cited 1×
“This is important in the light of the increasing awareness of potential biases in models used for healthcare discriminating based on for example sex and gender or ethnicity [38].”
BackgroundHotter, B. A., Pittl, S., Ebinger, M., Oepen, G., Jegzentis, K., Kudo, K., Różański, M., Schmidt, W. U., Brunecker, P., Xu, C., et al. (2009) · BMC Neurology · doi:10.1186/1471-2377-9-60 · Cited by 95
Self-citationBoosted Tree Model Reforms Multimodal Magnetic Resonance Imaging Infarct Prediction in Acute Stroke
Livne, M., Boldsen, J. K., Mikkelsen, I. K., Fiebach, J. B., Sobesky, J., & Mouridsen, K. (2018) · Stroke · doi:10.1161/strokeaha.117.019440 · Cited by 88
Self-citationTransforming health policy through machine learning
Ashrafian, H., & Darzi, A. W. (2018) · PLoS Medicine · doi:10.1371/journal.pmed.1002692 · Cited by 63
Zhao, J., Feng, Q., Wu, P., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/366682 · Cited by 39
Using machine‐learning methods to support health‐care professionals in making admission decisions
Luo, L., Li, J., Liu, C., & Shen, W. (2019) · The International Journal of Health Planning and Management · doi:10.1002/hpm.2769 · Cited by 28
Slovarp, L. J., & Bozarth, E. (2019) · Annals of Translational Medicine · doi:10.21037/atm.2018.12.26 · Cited by 20
Exploring protocol for breast cancer xenograft model using endothelial colony-forming cells
Lim, H. K., Lee, H., Moon, A., Kang, K., & Jung, J. (2018) · Translational Cancer Research · doi:10.21037/tcr.2018.09.09 · Cited by 7
Explainable Machine Learning for Scientific Insights and Discoveries
Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J. (2019) · arXiv · doi:10.48550/arxiv.1905.08883
Research works printed but not linked: 1
- BC Csáji, 2001, Approximation with Artificial Neural Networks, 45
Cited by
Loh, H. W., Ooi, C. P., Seoni, S., Barua, P. D., Molinari, F., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107161 · Cited by 734
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629
Wang, K., Tian, J., Zheng, C., Yang, H., Ren, J., Liu, Y., Han, Q., & Zhang, Y. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104813 · Cited by 361
A Survey of Decision Trees: Concepts, Algorithms, and Applications
Mienye, I. D., & Jere, N. (2024) · IEEE Access · doi:10.1109/access.2024.3416838 · Cited by 353
Wani, N. A., Kumar, R., & Bedi, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107879 · Cited by 300
Metadata
- Authors
- Zihni, E., Madai, V. I., Livne, M., Galinovic, I., Khalil, A. A., Fiebach, J. B., & Frey, D.
- Affiliations
- Charité – Universitätsmedizin Berlin (DE)
- Center for Stroke Research Berlin (DE)
- Corresponding author
- Madai, Vince Istvan
- ORCID
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-04-06
- Volume
- 15
- Issue
- 4
- Pages
- e0231166
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC7135268
- OpenAlex
- W3014245704
- Cited by
- 187
- References
- 50
- Citations by year
- 2026: 20 · 2025: 26 · 2024: 34 · 2023: 31 · 2022: 30 · 2021: 35 · 2020: 10
- FWCI
- 10.41
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Arrow - TU Dublin (Technological University Dublin) (submittedVersion, cc-by-nc-sa)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Refubium (Universitätsbibliothek der Freien Universität Berlin) (submittedVersion, cc-by)
- Universitätsbibliothek der FU Berlin Hochschulschriftenstelle u. Dokumentenserver (cc-by)
- Funders
- Deutsche Forschungsgemeinschaft
- Bundesministerium für Bildung und Forschung (PREDICTioN2020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- explainable AI, clinical decision support, stroke outcome prediction, modified Rankin Scale, gradient boosting decision tree, multilayer perceptron, logistic regression, feature importance, Shapley value, stroke severity, age
- MeSH terms
- Clinical Decision-Making, Supervised Machine Learning, Aged, Aged, 80 and over, Female, Forecasting, Humans, Male, Middle Aged, Prognosis, Retrospective Studies, Logistic Models, Neural Networks, Computer, Outcome Assessment, Health Care, Area Under Curve, Stroke
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.