Explaining prediction models and individual predictions with feature contributions
The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 25,370
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · cited 2×
“These data sets are available in .arff format from the Weka website (http://www.cs.”
MethodThe Art of Computer Programming
Knuth, D. E. (1968) · Cited by 16,319
17. A Value for n-Person Games
Shapley, L. S. (1953) · Contributions to the Theory of Games (AM-28), Volume II · doi:10.1515/9781400881970-018 · Cited by 6,950
Shapley, L. S. (1988) · The Shapley Value · doi:10.1017/cbo9780511528446.003 · Cited by 4,954
The Art of Computer Programming. Volume 2: Seminumerical Algorithms.
Taussky, O., & Knuth, D. E. (1970) · The American Mathematical Monthly · doi:10.2307/2317055 · Cited by 3,822
Random Number Generation and Quasi-Monte Carlo Methods.
Collings, B. J., & Niederreiter, H. (1993) · Journal of the American Statistical Association · doi:10.2307/2290359 · Cited by 2,884
A Preoperative Nomogram for Disease Recurrence Following Radical Prostatectomy for Prostate Cancer
Kattan, M. W., Eastham, J. A., Stapleton, A. M. F., Wheeler, T. M., & Scardino, P. T. (1998) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/90.10.766 · Cited by 1,242
Data mining for credit card fraud: a comparative study
Bhattacharyya, S., Jha, S., Tharakunnel, K., & Westland, J. C. (2011) · Decision Support Systems · doi:10.1016/j.dss.2010.08.008 · Cited by 1,108 · cited 1×
“Applications range from credit scoring [11] and fraud detection [5] to financial auditing [4] and efficiency analysis [18].”
BackgroundCredit rating analysis with support vector machines and neural networks: a market comparative study
Huang, Z., Chen, H., Hsu, C.-J., Chen, W.-H., & Wu, S. (2004) · Decision Support Systems · doi:10.1016/s0167-9236(03)00086-1 · Cited by 1,017 · cited 1×
“Applications range from credit scoring [11] and fraud detection [5] to financial auditing [4] and efficiency analysis [18].”
BackgroundExtracting refined rules from knowledge-based neural networks
Towell, G. G., & Shavlik, J. W. (1993) · Machine Learning · doi:10.1007/bf00993103 · Cited by 757 · cited 1×
“Complex models, such as artificial neural networks [14, 15, 16, 17, 18] and SVM (Support Vector Machines) [19, 20, 10, 21, 22, 23, 24, 25] received most attention, because they are often very successful but non-transparent.”
Note on a Method for Calculating Corrected Sums of Squares and Products
Welford, B. P. (1962) · Technometrics · doi:10.1080/00401706.1962.10490022 · Cited by 712 · cited 1×
“Note that, we used Knuth’s incremental algorithm for computing the variance [16,30].”
MethodPolynomial calculation of the Shapley value based on sampling
Castro, J., Gómez, D., & Tejada, J. (2009) · Computers & Operations Research · doi:10.1016/j.cor.2008.04.004 · Cited by 601 · cited 1×
“(10) could be approximated using a simple sampling algorithm, where ( Prei (O)∪{i} − Prei (O) ) would be one sample (see, for example, [7]).”
MethodInterpreting and Using Regression
Achen, C. (1982) · Sage eBooks · doi:10.4135/9781412984560 · Cited by 595 · cited 2×
“(3) is sometimes also referred to as the situational importance of xi [1].”
BackgroundWhy and why not explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552 · cited 1×
“Other studies focused only on decision trees [20] and on decision trees and decision rules [2].”
BackgroundLow-discrepancy and low-dispersion sequences
Niederreiter, H. (1988) · Journal of Number Theory · doi:10.1016/0022-314x(88)90025-x · Cited by 491 · cited 1×
“Therefore, for faster convergence, quasi-random sampling can be used instead of pseudo-random sampling [33, 34].”
BackgroundMonte Carlo methods in finance
Jäckel, P. (2002) · Cited by 459
Comprehensible credit scoring models using rule extraction from support vector machines
Martens, D., Baesens, B., Van Gestel, T., & Vanthienen, J. (2007) · European Journal of Operational Research · doi:10.1016/j.ejor.2006.04.051 · Cited by 448 · cited 1×
“Complex models, such as artificial neural networks [14, 15, 16, 17, 18] and SVM (Support Vector Machines) [19, 20, 10, 21, 22, 23, 24, 25] received most attention, because they are often very successful but non-transparent.”
Huysmans, J., Dejaeger, K., Mues, C., Vanthienen, J., & Baesens, B. (2011) · Decision Support Systems · doi:10.1016/j.dss.2010.12.003 · Cited by 425 · cited 2×
“Similar to [12], our goal was to measure the effect of the explanation on the user through the users’ performance at prediction tasks, which can be measured objectively.”
BackgroundAn Efficient Explanation of Individual Classifications using Game Theory
StrumbeljErik, & KononenkoIgor (2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1756007 · Cited by 380 · cited 4×
“This leads to the following explicit definition (see [28] for proof):”
BackgroundInfluentialSelf-citationINDUCTIVE AND BAYESIAN LEARNING IN MEDICAL DIAGNOSIS
KONONENKO, I. (1993) · Applied Artificial Intelligence · doi:10.1080/08839519308949993 · Cited by 302 · cited 1×
“This fact has been exploited to provide an explanation for the naive Bayes classifier [3,17,23], linear SVM [14], logistic regression [21], Cox regression models [15], and additive models in general [26].”
MethodSelf-citationExplaining Classifications For Individual Instances
Robnik-Sikonja, M., & Kononenko, I. (2008) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2007.190734 · Cited by 254 · cited 1×
“Previous general approaches [19,24] tackled the problem in the following way:”
BackgroundSelf-citationA rough-fuzzy approach for generating classification rules
Shen, Q., & Chouchoulas, A. (2002) · Pattern Recognition · doi:10.1016/s0031-3203(01)00229-1 · Cited by 216
Visualisation and interpretation of Support Vector Regression models
Üstün, B., Melssen, W. J., & Buydens, L. M. C. (2007) · Analytica Chimica Acta · doi:10.1016/j.aca.2007.03.023 · Cited by 172
Extracting decision trees from trained neural networks
Krishnan, R., Sivakumar, G., & Bhattacharya, P. (1999) · Pattern Recognition · doi:10.1016/s0031-3203(98)00181-2 · Cited by 134 · cited 1×
“Complex models, such as artificial neural networks [14, 15, 16, 17, 18] and SVM (Support Vector Machines) [19, 20, 10, 21, 22, 23, 24, 25] received most attention, because they are often very successful but non-transparent.”
Nomograms for Visualization of Naive Bayesian Classifier
Možina, M., Demšar, J., Kattan, M. W., & Zupan, B. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30116-5_32 · Cited by 133 · cited 2×
“This fact has been exploited to provide an explanation for the naive Bayes classifier [3,17,23], linear SVM [14], logistic regression [21], Cox regression models [15], and additive models in general [26].”
MethodVisual explanation of evidence in additive classifiers
Poulin, B., Eisner, R., Szafron, D., Lu, P., Greiner, R., Wishart, D. S., Fyshe, A., Pearcy, B., Macdonell, C., & Anvik, J. (2006) · NPARC · Cited by 131
Cano, J.-R., Herrera, F., & Lozano, M. (2006) · Data & Knowledge Engineering · doi:10.1016/j.datak.2006.01.008 · Cited by 119
Rule Extraction from Support Vector Machines: A Sequential Covering Approach
Barakat, N. H., & Bradley, A. P. (2007) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2007.190610 · Cited by 114 · cited 1×
“Complex models, such as artificial neural networks [14, 15, 16, 17, 18] and SVM (Support Vector Machines) [19, 20, 10, 21, 22, 23, 24, 25] received most attention, because they are often very successful but non-transparent.”
Application of elitist multi-objective genetic algorithm for classification rule generation
Dehuri, S., Patnaik, S., Ghosh, A. K., & Mall, R. (2007) · Applied Soft Computing · doi:10.1016/j.asoc.2007.02.009 · Cited by 83
Nomograms for visualizing support vector machines
Jakulin, A., Možina, M., Demšar, J., Bratko, I., & Zupan, B. (2005) · Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining · doi:10.1145/1081870.1081886 · Cited by 81 · cited 4×
“This fact has been exploited to provide an explanation for the naive Bayes classifier [3,17,23], linear SVM [14], logistic regression [21], Cox regression models [15], and additive models in general [26].”
MethodInfluentialA Practical Device for the Application of a Diagnostic or Prognostic Function
Lubsen, J., Pool, J., & van der Does, E. (1978) · Methods of Information in Medicine · doi:10.1055/s-0038-1636613 · Cited by 67 · cited 2×
“This fact has been exploited to provide an explanation for the naive Bayes classifier [3,17,23], linear SVM [14], logistic regression [21], Cox regression models [15], and additive models in general [26].”
MethodExplanation and reliability of prediction models: the case of breast cancer recurrence
Štrumbelj, E., Bosnić, Z., Kononenko, I., Zakotnik, B., & Kuhar, C. G. (2009) · Knowledge and Information Systems · doi:10.1007/s10115-009-0244-9 · Cited by 58
Self-citationAn ANN-based auditor decision support system using Benford's law
Bhattacharya, S., Dongming Xu, A., & Kumar, K. (2010) · Decision Support Systems · doi:10.1016/j.dss.2010.08.011 · Cited by 56 · cited 1×
“Applications range from credit scoring [11] and fraud detection [5] to financial auditing [4] and efficiency analysis [18].”
BackgroundA General Method for Visualizing and Explaining Black-Box Regression Models
Štrumbelj, E., & Kononenko, I. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-20267-4_3 · Cited by 55 · cited 2×
“As shown in [29], it is sufficient to limit ourselves to such distributions”
BackgroundSelf-citationInteractive visual exploration of association rules with rule-focusing methodology
Blanchard, J., Guillet, F., & Briand, H. (2006) · Knowledge and Information Systems · doi:10.1007/s10115-006-0046-2 · Cited by 54 · cited 1×
“Some models, such as decision and regression trees, rules [6,8], and nearest neighbors-based methods, are self-explanatory.”
MethodRule Extraction from Trained Support Vector Machines
Zhang, Y., Su, H., Jia, T., & Chu, J. (2005) · Lecture notes in computer science · doi:10.1007/11430919_9 · Cited by 53 · cited 1×
“Complex models, such as artificial neural networks [14, 15, 16, 17, 18] and SVM (Support Vector Machines) [19, 20, 10, 21, 22, 23, 24, 25] received most attention, because they are often very successful but non-transparent.”
BackgroundUser-oriented Assessment of Classification Model Understandability
Hiva, A., & Niklas, L. (2011) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/978-1-60750-754-3-11 · Cited by 52
Visualization of Support Vector Machines with Unsupervised Learning
Hamel, L. (2006) · 2006 IEEE Symposium on Computational Intelligence and Bioinformatics and Computational Biology · doi:10.1109/cibcb.2006.330984 · Cited by 49 · cited 1×
“Complex models, such as artificial neural networks [14, 15, 16, 17, 18] and SVM (Support Vector Machines) [19, 20, 10, 21, 22, 23, 24, 25] received most attention, because they are often very successful but non-transparent.”
Lee, S. (2010) · Decision Support Systems · doi:10.1016/j.dss.2010.06.002 · Cited by 44 · cited 1×
“Applications range from credit scoring [11] and fraud detection [5] to financial auditing [4] and efficiency analysis [18].”
BackgroundContact personalization using a score understanding method
Lemaire, V., Féraud, R., & Voisine, N. (2008) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2008.4633863 · Cited by 31 · cited 1×
“Previous general approaches [19,24] tackled the problem in the following way:”
BackgroundSVM and Graphical Algorithms: A Cooperative Approach
Poulet, F. (2005) · Proceedings · doi:10.1109/icdm.2004.10068 · Cited by 30 · cited 1×
“Complex models, such as artificial neural networks [14, 15, 16, 17, 18] and SVM (Support Vector Machines) [19, 20, 10, 21, 22, 23, 24, 25] received most attention, because they are often very successful but non-transparent.”
An evolutionary approach for automatically extracting intelligible classification rules
Falco, I. D., Cioppa, A. D., Iazzetta, A., & Tarantino, E. (2004) · Knowledge and Information Systems · doi:10.1007/s10115-003-0143-4 · Cited by 26 · cited 1×
“Some models, such as decision and regression trees, rules [6,8], and nearest neighbors-based methods, are self-explanatory.”
MethodGenerating rules with predicates, terms and variables from the pruned neural networks
Nayak, R. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.02.001 · Cited by 22
Explaining the result of a Decision Tree to the End-User
Alvarez, I. (2004) · Cited by 12
Vien, N. A., Việt, N. H., Chung, T., Yu, H., Kim, S., & Cho, B. H. (2009) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1645953.1646312 · Cited by 3 · cited 1×
“Complex models, such as artificial neural networks [14, 15, 16, 17, 18] and SVM (Support Vector Machines) [19, 20, 10, 21, 22, 23, 24, 25] received most attention, because they are often very successful but non-transparent.”
Extraction of Logical Rules from Data by Means of Piecewise-Linear Neural Networks
Holeňa, M. (2002) · Lecture notes in computer science · doi:10.1007/3-540-36182-0_18 · Cited by 3
Research works printed but not linked: 1
- Becker B, Kohavi R, Sommerfield D (1997) Visualizing the simple Bayesian classier. KDD workshop on issues in the integration of data mining and data visualization
Cited by
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
From local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
“They used the preliminary information available from the work proposed by Štrumbelj and Kononenko [117] to design a new set of values called the expectation Shapley (ES) values which were able to unify and justify a broad spectrum of approaches (e.g., LIME, DeepLift, and Layer-Wise Rel-evance…”
MethodExplainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967
Metadata
- Authors
- Štrumbelj, E., & Kononenko, I.
- Affiliations
- University of Ljubljana (SI)
- Corresponding author
- Štrumbelj, Erik
- Venue
- Knowledge and Information Systems
- Published
- 2013-08-30
- Volume
- 41
- Issue
- 3
- Pages
- 647-665
- ISSN
- 0219-1377 (ISSN-L)
- 0219-3116
- OpenAlex
- W2129888542
- Cited by
- 2,286
- References
- 47
- Citations by year
- 2026: 325 · 2025: 480 · 2024: 445 · 2023: 338 · 2022: 263 · 2021: 237 · 2020: 129 · 2019: 48 · 2018: 15 · 2017: 4 · 2016: 2
- FWCI
- 0.74
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Knowledge and Information Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Data Mining Algorithms and Applications
- Keywords
- explainable machine learning, feature contribution, sensitivity analysis, model explanation, personalized prediction, feature interaction, feature redundancy, additive models, model understanding, human evaluation of explanations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.