CoChat
Sign Up
PreprintCited by 11,321Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AISoftware Testing and Debugging Techniques

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI

Open access
Arrieta, Alejandro Barredo, Díaz-Rodríguez, Natalia, Del Ser, Javier, Bennetot, Adrien, Tabik, Siham, Barbado, Alberto, García, Salvador, Gil-López, Sergio +4 more·2019·arXiv
DOI 10.48550/arxiv.1910.10045arXiv 1910.10045
Open this paper in CoChat — free accountOpen PDF

Abstract

In the last years, Artificial Intelligence (AI) has achieved a notable momentum that may deliver the best of expectations over many application sectors across the field. For this to occur, the entire community stands in front of the barrier of explainability, an inherent problem of AI techniques brought by sub-symbolism (e.g. ensembles or Deep Neural Networks) that were not present in the last hype of AI. Paradigms underlying this problem fall within the so-called eXplainable AI (XAI) field, which is acknowledged as a crucial feature for the practical deployment of AI models. This overview examines the existing literature in the field of XAI, including a prospect toward what is yet to be reached. We summarize previous efforts to define explainability in Machine Learning, establishing a novel definition that covers prior conceptual propositions with a major focus on the audience for which explainability is sought. We then propose and discuss about a taxonomy of recent contributions related to the explainability of different Machine Learning models, including those aimed at Deep Learning methods for which a second taxonomy is built. This literature analysis serves as the background for a series of challenges faced by XAI, such as the crossroads between data fusion and explainability. Our prospects lead toward the concept of Responsible Artificial Intelligence, namely, a methodology for the large-scale implementation of AI methods in real organizations with fairness, model explainability and accountability at its core. Our ultimate goal is to provide newcomers to XAI with a reference material in order to stimulate future research advances, but also to encourage experts and professionals from other disciplines to embrace the benefits of AI in their activity sectors, without any prior bias for its lack of interpretability.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1910.10045

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 331

  1. S. J. Russell, P. Norvig, Artificial intelligence: a modern approach, Malaysia; Pearson Education Limited,, 2016.
  2. D. M. West, The future of work: robots, AI, and automation, Brookings Institution Press, 2018.
  3. A. Preece, D. Harborne, D. Braines, R. Tomsett, S. Chakraborty, Stakeholders in Explainable AI (2018). arXiv:1810.00184. · arXiv
  4. E. Tjoa, C. Guan, A survey on explainable artificial intelligence (XAI): Towards medical XAI (2019). arXiv:1907.07374. · arXiv
  5. J. Zhu, A. Liapis, S. Risi, R. Bidarra, G. M. Youngblood, Explainable AI for designers: A human- centered perspective on mixed-initiative co-creation, 2018 IEEE Conference on Computational Intelligence and Games (CIG) (2018) 1-8.
  6. F. K. Dosilović, M. Brcić, N. Hlupić, Explainable artificial intelligence: A survey, in: 41st International Convention on Information and Communication Technology, Electronics and Micro- electronics (MIPRO), 2018, pp. 210-215.
  7. P. Hall, On the Art and Science of Machine Learning Explanations (2018). arXiv:1810.02909. · arXiv
  8. L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, L. Kagal, Explaining Explanations: An Overview of Interpretability of Machine Learning (2018). arXiv:1806.00069. · arXiv
  9. A. Adadi, M. Berrada, Peeking inside the black-box: A survey on explainable artificial intelligence (XAI), IEEE Access 6 (2018) 52138-52160.
  10. O. Biran, C. Cotton, Explanation and justification in machine learning: A survey, in: IJCAI-17 workshop on explainable AI (XAI), Vol. 8, 2017, p. 1.
  11. S. T. Shane T. Mueller, R. R. Hoffman, W. Clancey, G. Klein, Explanation in Human-AI Systems: A Literature Meta-Review Synopsis of Key Ideas and Publications and Bibliography for Explainable AI, Tech. rep., Defense Advanced Research Projects Agency (DARPA) XAI Program (2019).
  12. R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, D. Pedreschi, A survey of methods for explaining black box models, ACM Computing Surveys 51 (5) (2018) 93:1-93:42.
  13. G. Montavon, W. Samek, K.-R. Müller, Methods for interpreting and understanding deep neural networks, Digital Signal Processing 73 (2018) 1-15. doi:10.1016/j.dsp.2017.10.011. · DOI · Paper page
  14. A. Fernandez, F. Herrera, O. Cordon, M. Jose del Jesus, F. Marcelloni, Evolutionary fuzzy systems for explainable artificial intelligence: Why, when, what for, and where to?, IEEE Computational Intelligence Magazine 14 (1) (2019) 69-81.
  15. R. S. Michalski, A theory and methodology of inductive learning, in: Machine learning, Springer, 1983, pp. 83-134.
  16. D. Doran, S. Schulz, T. R. Besold, What does explainable AI really mean? a new conceptualization of perspectives (2017). arXiv:1710.00794. · arXiv
  17. F. Doshi-Velez, B. Kim, Towards a rigorous science of interpretable machine learning (2017). arXiv:1702.08608. · arXiv
  18. A. Vellido, J. D. Martín-Guerrero, P. J. Lisboa, Making machine learning models interpretable., in: European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), Vol. 12, Citeseer, 2012, pp. 163-172.
  19. E. Walter, Cambridge advanced learner's dictionary, Cambridge University Press, 2008.
  20. P. Besnard, A. Hunter, Elements of Argumentation, The MIT Press, 2008.
  21. A. Holzinger, C. Biemann, C. S. Pattichis, D. B. Kell, What do we need to build explainable Ai systems for the medical domain? (2017). arXiv:1712.09923. · arXiv
  22. M. T. Ribeiro, S. Singh, C. Guestrin, Why should I trust you?: Explaining the predictions of any classifier, in: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2016, pp. 1135-1144.
  23. M. Fox, D. Long, D. Magazzeni, Explainable planning (2017). arXiv:1709.10256. · arXiv
  24. W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, B. Yu, Interpretable machine learning: definitions, methods, and applications (2019). arXiv:1901.04592. · arXiv
  25. J. Haspiel, N. Du, J. Meyerson, L. P. Robert Jr, D. Tilbury, X. J. Yang, A. K. Pradhan, Explana- tions and expectations: Trust building in automated vehicles, in: Companion of the ACM/IEEE International Conference on Human-Robot Interaction, ACM, 2018, pp. 119-120.
  26. A. Chander, R. Srinivasan, S. Chelian, J. Wang, K. Uchino, Working with beliefs: AI transparency in the enterprise., in: Workshops of the ACM Conference on Intelligent User Interfaces, 2018.
  27. A. B. Tickle, R. Andrews, M. Golea, J. Diederich, The truth will come to light: Directions and challenges in extracting the knowledge embedded within trained artificial neural networks, IEEE Transactions on Neural Networks 9 (6) (1998) 1057-1068.
  28. C. Louizos, U. Shalit, J. M. Mooij, D. Sontag, R. Zemel, M. Welling, Causal effect inference with deep latent-variable models, in: Advances in Neural Information Processing Systems, 2017, pp. 6446-6456.
  29. O. Goudet, D. Kalainathan, P. Caillou, I. Guyon, D. Lopez-Paz, M. Sebag, Learning functional causal models with generative neural networks, in: Explainable and Interpretable Models in Computer Vision and Machine Learning, Springer, 2018, pp. 39-80.
  30. D. Lopez-Paz, R. Nishihara, S. Chintala, B. Scholkopf, L. Bottou, Discovering causal signals in images, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6979-6987.
  31. C. Barabas, K. Dinakar, J. Ito, M. Virza, J. Zittrain, Interventions over predictions: Reframing the ethical debate for actuarial risk assessment (2017). arXiv:1712.08238. · arXiv
  32. R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, N. Elhadad, Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission, in: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '15, 2015, pp. 1721-1730.
  33. W. Samek, T. Wiegand, K.-R. Müller, Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models (2017). arXiv:1708.08296. · arXiv
  34. C. Wadsworth, F. Vera, C. Piech, Achieving fairness through adversarial learning: an application to recidivism prediction (2018). arXiv:1807.00199. · arXiv
  35. X. Yuan, P. He, Q. Zhu, X. Li, Adversarial examples: Attacks and defenses for deep learning, IEEE Transactions on Neural Networks and Learning Systems 30 (9) (2019) 2805-2824.
  36. B. Letham, C. Rudin, T. H. McCormick, D. Madigan, et al., Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model, The Annals of Applied Statistics 9 (3) (2015) 1350-1371.
  37. M. Harbers, K. van den Bosch, J.-J. Meyer, Design and evaluation of explainable BDI agents, in: IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, Vol. 2, IEEE, 2010, pp. 125-132.
  38. M. H. Aung, P. G. Lisboa, T. A. Etchells, A. C. Testa, B. Van Calster, S. Van Huffel, L. Valentin, D. Timmerman, Comparing analytical decision support models through boolean rule extraction: A case study of ovarian tumour malignancy, in: International Symposium on Neural Networks, Springer, 2007, pp. 1177-1186.
  39. A. Weller, Challenges for transparency (2017). arXiv:1708.01870. · arXiv
  40. A. A. Freitas, Comprehensible classification models: a position paper, ACM SIGKDD explorations newsletter 15 (1) (2014) 1-10.
  41. V. Schetinin, J. E. Fieldsend, D. Partridge, T. J. Coats, W. J. Krzanowski, R. M. Everson, T. C. Bailey, A. Hernandez, Confident interpretation of bayesian decision tree ensembles for clinical applications, IEEE Transactions on Information Technology in Biomedicine 11 (3) (2007) 312-319.
  42. Z. Che, S. Purushotham, R. Khemani, Y. Liu, Interpretable deep models for ICU outcome prediction, in: AMIA Annual Symposium Proceedings, Vol. 2016, American Medical Informatics Association, 2016, p. 371.
  43. N. Barakat, J. Diederich, Eclectic rule-extraction from support vector machines, International Journal of Computer, Electrical, Automation, Control and Information Engineering 2 (5) (2008) 1672-1675.
  44. F. J. C. Garcia, D. A. Robb, X. Liu, A. Laskov, P. Patron, H. Hastie, Explain yourself: A natural language interface for scrutable autonomous robots (2018). arXiv:1803.02088. · arXiv
  45. P. Langley, B. Meadows, M. Sridharan, D. Choi, Explainable agency for intelligent autonomous systems, in: AAAI Conference on Artificial Intelligence, 2017, pp. 4762-4763.
  46. P.-J. Kindermans, K. T. Schütt, M. Alber, K.-R. Müller, D. Erhan, B. Kim, S. Dähne, Learning how to explain neural networks: Patternnet and patternattribution (2017). arXiv:1705.05598. · arXiv
  47. G. Ras, M. van Gerven, P. Haselager, Explanation methods in deep learning: Users, values, concerns and challenges, in: Explainable and Interpretable Models in Computer Vision and Machine Learning, Springer, 2018, pp. 19-36.
  48. S. Bach, A. Binder, K.-R. Müller, W. Samek, Controlling explanatory heatmap resolution and semantics via decomposition depth, in: IEEE International Conference on Image Processing (ICIP), IEEE, 2016, pp. 2271-2275.
  49. G. J. Katuwal, R. Chen, Machine learning model interpretability for precision medicine (2016). arXiv:1610.09045. · arXiv
  50. M. A. Neerincx, J. van der Waa, F. Kaptein, J. van Diggelen, Using perceptual and cognitive expla- nations for enhanced human-agent team performance, in: International Conference on Engineering Psychology and Cognitive Ergonomics, Springer, 2018, pp. 204-214.
  51. J. Krause, A. Perer, K. Ng, Interacting with predictions: Visual inspection of black-box machine learning models, in: CHI Conference on Human Factors in Computing Systems, ACM, 2016, pp. 5686-5697.
  52. L. Rosenbaum, G. Hinselmann, A. Jahn, A. Zell, Interpreting linear support vector machine models with heat map molecule coloring, Journal of Cheminformatics 3 (1) (2011) 11.
  53. J. Tan, M. Ung, C. Cheng, C. S. Greene, Unsupervised feature construction and knowledge extraction from genome-wide assays of breast cancer with denoising autoencoders, in: Pacific Symposium on Biocomputing Co-Chairs, World Scientific, 2014, pp. 132-143.
  54. S. Krening, B. Harrison, K. M. Feigh, C. L. Isabell, M. Riedl, A. Thomaz, Learning from expla- nations using sentiment and advice in RL, IEEE Transactions on Cognitive and Developmental Systems 9 (1) (2017) 44-55.
  55. M. T. Ribeiro, S. Singh, C. Guestrin, Model-agnostic interpretability of machine learning (2016). arXiv:1606.05386. · arXiv
  56. T. A. Etchells, P. J. Lisboa, Orthogonal search-based rule extraction (OSRE) for trained neural networks: a practical and efficient approach, IEEE Transactions on Neural Networks 17 (2) (2006) 374-384.
  57. Y. Zhang, S. Sreedharan, A. Kulkarni, T. Chakraborti, H. H. Zhuo, S. Kambhampati, Plan expli- cability and predictability for robot task planning, in: 2017 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2017, pp. 1313-1320.
  58. A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, T. Lillicrap, A simple neural network module for relational reasoning, in: Advances in Neural Information Processing Systems, 2017, pp. 4967-4976.
  59. Q. Zhang, Y. Yang, H. Ma, Y. N. Wu, Interpreting CNNs via decision trees, in: IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 6261-6270.
  60. M. Wu, M. C. Hughes, S. Parbhoo, M. Zazzi, V. Roth, F. Doshi-Velez, Beyond sparsity: Tree regularization of deep models for interpretability, in: AAAI Conference on Artificial Intelligence, 2018, pp. 1670-1678.
  61. G. Hinton, O. Vinyals, J. Dean, Distilling the knowledge in a neural network (2015). arXiv:1503.02531. · arXiv
  62. N. Frosst, G. Hinton, Distilling a neural network into a soft decision tree (2017). arXiv:1711.09784. · arXiv
  63. M. G. Augasta, T. Kathirvalavakumar, Reverse engineering the neural networks for rule extraction in classification problems, Neural Processing Letters 35 (2) (2012) 131-150.
  64. H. F. Tan, G. Hooker, M. T. Wells, Tree space prototypes: Another look at making tree ensembles interpretable (2016). arXiv:1611.07115. · arXiv
  65. R. C. Fong, A. Vedaldi, Interpretable explanations of black boxes by meaningful perturbation, in: IEEE International Conference on Computer Vision, 2017, pp. 3429-3437.
  66. T. Miller, P. Howe, L. Sonenberg, Explainable AI: Beware of inmates running the asylum, in: International Joint Conference on Artificial Intelligence, Workshop on Explainable AI (XAI), Vol. 36, 2017, pp. 36-40.
  67. R. Goebel, A. Chander, K. Holzinger, F. Lecue, Z. Akata, S. Stumpf, P. Kieseberg, A. Holzinger, Explainable AI: the new 42?, in: International Cross-Domain Conference for Machine Learning and Knowledge Extraction, Springer, 2018, pp. 295-303.
  68. V. Belle, Logic meets probability: Towards explainable AI systems for uncertain worlds, in: International Joint Conference on Artificial Intelligence, 2017, pp. 5116-5120.
  69. Y. Lou, R. Caruana, J. Gehrke, G. Hooker, Accurate intelligible models with pairwise interactions, in: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2013, pp. 623-631.
  70. K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, Y. Bengio, Show, attend and tell: Neural image caption generation with visual attention, in: International Conference on Machine Learning, 2015, pp. 2048-2057.
  71. N. H. Barakat, A. P. Bradley, Rule extraction from support vector machines: A sequential covering approach, IEEE Transactions on Knowledge and Data Engineering 19 (6) (2007) 729-741.
  72. F. C. Adriana da Costa, M. M. B. Vellasco, R. Tanscheit, Fuzzy rule extraction from support vector machines, in: International Conference on Hybrid Intelligent Systems, IEEE, 2005, pp. 335-340.
  73. D. Martens, B. Baesens, T. Van Gestel, J. Vanthienen, Comprehensible credit scoring models using rule extraction from support vector machines, European Journal of Operational Research 183 (3) (2007) 1466-1476.
  74. B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, Learning deep features for discriminative localization, in: IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2921- 2929.
  75. X. Fu, C. Ong, S. Keerthi, G. G. Hung, L. Goh, Extracting the knowledge embedded in support vector machines, in: IEEE International Joint Conference on Neural Networks, Vol. 1, IEEE, 2004, pp. 291-296.
  76. B. Green, "Fair" risk assessments: A precarious approach for criminal justice reform, in: 5th Workshop on Fairness, Accountability, and Transparency in Machine Learning, 2018.
  77. M. Kim, O. Reingold, G. Rothblum, Fairness through computationally-bounded awareness, in: Advances in Neural Information Processing Systems, 2018, pp. 4842-4852.
  78. B. Haasdonk, Feature space interpretation of SVMs with indefinite kernels, IEEE Transactions on Pattern Analysis and Machine Intelligence 27 (4) (2005) 482-492.
  79. A. Palczewska, J. Palczewski, R. M. Robinson, D. Neagu, Interpreting random forest classification models using a feature contribution method, in: Integration of Reusable Systems, Springer, 2014, pp. 193-218.
  80. S. H. Welling, H. H. Refsgaard, P. B. Brockhoff, L. H. Clemmensen, Forest floor visualizations of random forests (2016). arXiv:1605.09196. · arXiv
  81. G. Fung, S. Sandilya, R. B. Rao, Rule extraction from linear support vector machines, in: ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, ACM, 2005, pp. 32-40.
  82. Y. Zhang, H. Su, T. Jia, J. Chu, Rule extraction from trained support vector machines, in: Pacific- Asia Conference on Knowledge Discovery and Data Mining, Springer, 2005, pp. 61-70.
  83. D. Linsley, D. Shiebler, S. Eberhardt, T. Serre, Global-and-local attention networks for visual recognition (2018). arXiv:1805.08819. · arXiv
  84. A. Shrikumar, P. Greenside, A. Shcherbina, A. Kundaje, Not just a black box: Learning important features through propagating activation differences (2016). arXiv:1605.01713. · arXiv
  85. Y. Dong, H. Su, J. Zhu, B. Zhang, Improving interpretability of deep neural networks with semantic information, in: IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4306-4314.
  86. G. Ridgeway, D. Madigan, T. Richardson, J. O'Kane, Interpretable boosted naïve bayes classi- fication., in: ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 1998, pp. 101-104.
  87. Q. Zhang, Y. Nian Wu, S.-C. Zhu, Interpretable convolutional neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8827-8836.
  88. S. Seo, J. Huang, H. Yang, Y. Liu, Interpretable convolutional neural networks with dual local and global attention for review rating prediction, in: Proceedings of the Eleventh ACM Conference on Recommender Systems, ACM, 2017, pp. 297-305.
  89. K. Xu, D. H. Park, C. Yi, C. Sutton, Interpreting deep classifier by visual distillation of dark knowledge (2018). arXiv:1803.04042. · arXiv
  90. H. Deng, Interpreting tree ensembles with intrees (2014). arXiv:1408.5456. · arXiv
  91. S. Tan, R. Caruana, G. Hooker, Y. Lou, Distill-and-compare: Auditing black-box models using transparent model distillation, in: AAAI/ACM Conference on AI, Ethics, and Society, ACM, 2018, pp. 303-310.
  92. S. Hara, K. Hayashi, Making tree ensembles interpretable (2016). arXiv:1606.05390. · arXiv
  93. A. Henelius, K. Puolamäki, A. Ukkonen, Interpreting classifiers through attribute interactions in datasets (2017). arXiv:1707.07576. · arXiv
  94. H. Hastie, F. J. C. Garcia, D. A. Robb, P. Patron, A. Laskov, MIRIAM: a multimodal chat-based interface for autonomous systems, in: ACM International Conference on Multimodal Interaction, ACM, 2017, pp. 495-496.
  95. D. Bau, B. Zhou, A. Khosla, A. Oliva, A. Torralba, Network dissection: Quantifying interpretability of deep visual representations, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6541-6549.
  96. H. Núñez, C. Angulo, A. Català, Rule extraction from support vector machines., in: European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), 2002, pp. 107-112.
  97. H. Núñez, C. Angulo, A. Català, Rule-based learning systems for support vector machines, Neural Processing Letters 24 (1) (2006) 1-18.
  98. M. Kearns, S. Neel, A. Roth, Z. S. Wu, Preventing fairness gerrymandering: Auditing and learning for subgroup fairness (2017). arXiv:1711.05144. · arXiv
  99. E. Akyol, C. Langbort, T. Basar, Price of transparency in strategic machine learning (2016). arXiv:1610.08210. · arXiv
  100. Y. Zhang, B. Wallace, A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification (2015). arXiv:1510.03820. · arXiv
  101. J. R. Quinlan, Simplifying decision trees, International journal of man-machine studies 27 (3) (1987) 221-234.
  102. Y. Zhou, G. Hooker, Interpreting models via single tree approximation (2016). arXiv:1610.09036. · arXiv
  103. J. J. Thiagarajan, B. Kailkhura, P. Sattigeri, K. N. Ramamurthy, Treeview: Peeking into deep neural networks via feature-space partitioning (2016). arXiv:1611.07429. · arXiv
  104. M. D. Zeiler, R. Fergus, Visualizing and understanding convolutional networks, in: European conference on computer vision, Springer, 2014, pp. 818-833.
  105. A. Mahendran, A. Vedaldi, Understanding deep image representations by inverting them, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 5188- 5196.
  106. J. Wagner, J. M. Kohler, T. Gindele, L. Hetzel, J. T. Wiedemer, S. Behnke, Interpretable and fine-grained visual explanations for convolutional neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 9097-9107.
  107. A. Kanehira, T. Harada, Learning to explain with complemental examples, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 8603-8611.
  108. D. W. Apley, Visualizing the effects of predictor variables in black box supervised learning models (2016). arXiv:1612.08468. · arXiv
  109. M. Staniak, P. Biecek, Explanations of Model Predictions with live and breakDown Packages, The R Journal 10 (2) (2018) 395-409.
  110. M. D. Zeiler, D. Krishnan, G. W. Taylor, R. Fergus, Deconvolutional networks., in: CVPR, Vol. 10, 2010, p. 7.
  111. J. T. Springenberg, A. Dosovitskiy, T. Brox, M. Riedmiller, Striving for simplicity: The all convolutional net (2014). arXiv:1412.6806. · arXiv
  112. B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, R. Sayres, Interpretability be- yond feature attribution: Quantitative testing with concept activation vectors (TCAV) (2017). arXiv:1711.11279. · arXiv
  113. A. Polino, R. Pascanu, D. Alistarh, Model compression via distillation and quantization (2018). arXiv:1802.05668. · arXiv
  114. W. J. Murdoch, A. Szlam, Automatic rule extraction from long short term memory networks (2017). arXiv:1702.02540. · arXiv
  115. M. W. Craven, J. W. Shavlik, Using sampling and queries to extract rules from trained neural networks, in: Machine learning proceedings 1994, Elsevier, 1994, pp. 37-45.
  116. U. Johansson, L. Niklasson, Evolving decision trees using oracle guides, in: 2009 IEEE Symposium on Computational Intelligence and Data Mining, IEEE, 2009, pp. 238-244.
  117. T. Lei, R. Barzilay, T. Jaakkola, Rationalizing neural predictions (2016). arXiv:1606.04155. · arXiv
  118. A. Radford, R. Jozefowicz, I. Sutskever, Learning to generate reviews and discovering sentiment (2017). arXiv:1704.01444. · arXiv
  119. R. Shwartz-Ziv, N. Tishby, Opening the black box of deep neural networks via information (2017). arXiv:1703.00810. · arXiv
  120. J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, H. Lipson, Understanding neural networks through deep visualization (2015). arXiv:1506.06579. · arXiv
  121. P. E. Pope, S. Kolouri, M. Rostami, C. E. Martin, H. Hoffmann, Explainability methods for graph convolutional neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 10772-10781.
  122. P. Gajane, M. Pechenizkiy, On formalizing fairness in prediction with machine learning (2017). arXiv:1710.03184. · arXiv
  123. C. Dwork, C. Ilvento, Composition of fairsystems (2018). arXiv:1806.06122. · arXiv
  124. H.-X. Wang, L. Fratiglioni, G. B. Frisoni, M. Viitanen, B. Winblad, Smoking and the occurence of alzheimer's disease: Cross-sectional and longitudinal data in a population-based study, American journal of epidemiology 149 (7) (1999) 640-644.
  125. J. Pearl, Causality, Cambridge university press, 2009.
  126. M. Kuhn, K. Johnson, Applied predictive modeling, Vol. 26, Springer, 2013.
  127. G. James, D. Witten, T. Hastie, R. Tibshirani, An introduction to statistical learning, Vol. 112, Springer, 2013.
  128. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus, Intriguing properties of neural networks (2013). arXiv:1312.6199. · arXiv
  129. D. Ruppert, Robust statistics: The approach based on influence functions, Taylor & Francis, 1987.
  130. S. Basu, K. Kumbier, J. B. Brown, B. Yu, Iterative random forests to discover predictive and stable high-order interactions, Proceedings of the National Academy of Sciences 115 (8) (2018) 1943-1948.
  131. K. Burns, L. A. Hendricks, K. Saenko, T. Darrell, A. Rohrbach, Women also Snowboard: Over- coming Bias in Captioning Models (2018). arXiv:1803.09797. · arXiv
  132. A. Bennetot, J.-L. Laurent, R. Chatila, N. Díaz-Rodríguez, Towards explainable neural-symbolic visual reasoning, in: NeSy Workshop IJCAI 2019, Macau, China, 2019.
  133. R. Tibshirani, Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society: Series B (Methodological) 58 (1) (1996) 267-288.
  134. Y. Lou, R. Caruana, J. Gehrke, Intelligible models for classification and regression, in: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2012, pp. 150-158.
  135. K. Kawaguchi, Deep learning without poor local minima, in: Advances in neural information processing systems, 2016, pp. 586-594.
  136. A. Datta, S. Sen, Y. Zick, Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems, in: 2016 IEEE symposium on security and privacy (SP), IEEE, 2016, pp. 598-617.
  137. D. W. Hosmer Jr, S. Lemeshow, R. X. Sturdivant, Applied logistic regression, Vol. 398, John Wiley & Sons, 2013.
  138. C.-Y. J. Peng, K. L. Lee, G. M. Ingersoll, An introduction to logistic regression analysis and reporting, The journal of educational research 96 (1) (2002) 3-14.
  139. C. Mood, Logistic regression: Why we cannot do what we think we can do, and what we can do about it, European sociological review 26 (1) (2010) 67-82.
  140. H. Laurent, R. L. Rivest, Constructing optimal binary decision trees is Np-complete, Information processing letters 5 (1) (1976) 15-17.
  141. L. Rokach, O. Z. Maimon, Data mining with decision trees: theory and applications, Vol. 69, World scientific, 2014.
  142. S. Rovnyak, S. Kretsinger, J. Thorp, D. Brown, Decision trees for real-time transient stability prediction, IEEE Transactions on Power Systems 9 (3) (1994) 1417-1426.
  143. S. B. Imandoust, M. Bolandraftar, Application of k-nearest neighbor (knn) approach for predicting economic events: Theoretical background, International Journal of Engineering Research and Applications 3 (5) (2013) 605-610.
  144. G. Guo, H. Wang, D. Bell, Y. Bi, K. Greer, An KNN model-based approach and its application in text categorization, in: International Conference on Intelligent Text Processing and Computational Linguistics, Springer, 2004, pp. 559-570.
  145. U. Johansson, R. König, L. Niklasson, The truth is in there-rule extraction from opaque models using genetic programming., in: FLAIRS Conference, Miami Beach, FL, 2004, pp. 658-663.
  146. J. R. Quinlan, Generating production rules from decision trees., in: ijcai, Vol. 87, Citeseer, 1987, pp. 304-307.
  147. P. Langley, H. A. Simon, Applications of machine learning and rule induction, Communications of the ACM 38 (11) (1995) 54-64.
  148. H. Murase, H. Nagashima, S. Yonezaki, R. Matsukura, T. Kitakado, Application of a generalized additive model (GAM) to reveal relationships between environmental factors and distributions of pelagic fish and krill: a case study in sendai bay, Japan, ICES Journal of Marine Science 66 (6) (2009) 1417-1424.
  149. N. Tomić, S. Božić, A modified geosite assessment model (M-GAM) and its application on the lazar canyon area (serbia), International journal of environmental research 8 (4) (2014) 1041-1052.
  150. P. Rothery, D. B. Roy, Application of generalized additive models to butterfly transect count data, Journal of Applied Statistics 28 (7) (2001) 897-909.
  151. A. Pierrot, Y. Goude, Short-term electricity load forecasting with generalized additive models, in: 16th Intelligent System Applications to Power Systems Conference, ISAP 2011, IEEE, 2011, pp. 410-415.
  152. T. L. Griffiths, C. Kemp, J. B. Tenenbaum, Bayesian models of cognition. (4 2008). doi:10.1184/R1/6613682.v1. URL https://kilthub.cmu.edu/articles/Bayesian_models_of_ cognition/6613682 · DOI
  153. M. McAllister, G. Kirkwood, Bayesian stock assessment: a review and example application using the logistic model, ICES Journal of Marine Science 55 (6) (1998) 1031-1060.
  154. G. Synnaeve, P. Bessiere, A bayesian model for opening prediction in RTS games with application to starcraft, in: Computational Intelligence and Games (CIG), 2011 IEEE Conference on, IEEE, 2011, pp. 281-288.
  155. S.-K. Min, D. Simonis, A. Hense, Probabilistic climate change predictions applying bayesian model averaging, Philosophical transactions of the royal society of london a: mathematical, physical and engineering sciences 365 (1857) (2007) 2103-2116.
  156. G. Koop, D. J. Poirier, J. L. Tobias, Bayesian econometric methods, Cambridge University Press, 2007.
  157. A. R. Cassandra, L. P. Kaelbling, J. A. Kurien, Acting under uncertainty: Discrete bayesian models for mobile-robot navigation, in: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. IROS'96, Vol. 2, IEEE, 1996, pp. 963-972.
  158. H. A. Chipman, E. I. George, R. E. McCulloch, Bayesian cart model search, Journal of the American Statistical Association 93 (443) (1998) 935-948.
  159. B. Kim, C. Rudin, J. A. Shah, The bayesian case model: A generative approach for case-based reasoning and prototype classification, in: Advances in Neural Information Processing Systems, 2014, pp. 1952-1960.
  160. B. Kim, R. Khanna, O. O. Koyejo, Examples are not enough, learn to criticize! criticism for interpretability, in: Advances in Neural Information Processing Systems, 2016, pp. 2280-2288.
  161. U. Johansson, L. Niklasson, R. König, Accuracy vs. comprehensibility in data mining models, in: Proceedings of the seventh international conference on information fusion, Vol. 1, 2004, pp. 295-300.
  162. R. Konig, U. Johansson, L. Niklasson, G-rex: A versatile framework for evolutionary data mining, in: 2008 IEEE International Conference on Data Mining Workshops, IEEE, 2008, pp. 971-974.
  163. H. Lakkaraju, E. Kamar, R. Caruana, J. Leskovec, Interpretable & explorable approximations of black box models (2017). arXiv:1707.01154. · arXiv
  164. S. Mishra, B. L. Sturm, S. Dixon, Local interpretable model-agnostic explanations for music content analysis., in: ISMIR, 2017, pp. 537-543.
  165. G. Su, D. Wei, K. R. Varshney, D. M. Malioutov, Interpretable two-level boolean rule learning for classification (2015). arXiv:1511.07361. · arXiv
  166. M. T. Ribeiro, S. Singh, C. Guestrin, Nothing else matters: Model-agnostic explanations by identifying prediction invariance (2016). arXiv:1611.05817. · arXiv
  167. M. W. Craven, Extracting comprehensible models from trained neural networks, Ph.D. thesis, aAI9700774 (1996).
  168. O. Bastani, C. Kim, H. Bastani, Interpretability via model extraction (2017). arXiv:1706.09773. · arXiv
  169. G. Hooker, Discovering additive structure in black box functions, in: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, 2004, pp. 575-580.
  170. P. W. Koh, P. Liang, Understanding black-box predictions via influence functions, in: Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR. org, 2017, pp. 1885-1894.
  171. P. Cortez, M. J. Embrechts, Opening black box data mining models using sensitivity analysis, in: 2011 IEEE Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2011, pp. 341-348.
  172. S. M. Lundberg, S.-I. Lee, A unified approach to interpreting model predictions, in: Advances in Neural Information Processing Systems, 2017, pp. 4765-4774.
  173. I. Kononenko, et al., An efficient explanation of individual classifications using game theory, Journal of Machine Learning Research 11 (Jan) (2010) 1-18.
  174. H. Chen, S. Lundberg, S.-I. Lee, Explaining models by propagating shapley values of local components (2019). arXiv:arXiv:1911.11888. · arXiv
  175. P. Dabkowski, Y. Gal, Real time image saliency for black box classifiers, in: Advances in Neural Information Processing Systems, 2017, pp. 6967-6976.
  176. J. Moeyersoms, B. d'Alessandro, F. Provost, D. Martens, Explaining classification models built on high-dimensional sparse data (2016). arXiv:1607.06280. · arXiv
  177. D. Baehrens, T. Schroeter, S. Harmeling, M. Kawanabe, K. Hansen, K.-R. M Þller, How to explain individual classification decisions, Journal of Machine Learning Research 11 (Jun) (2010) 1803-1831.
  178. J. Adebayo, L. Kagal, Iterative orthogonal feature projection for diagnosing bias in black-box models (2016). arXiv:1611.04967. · arXiv
  179. R. Guidotti, A. Monreale, S. Ruggieri, D. Pedreschi, F. Turini, F. Giannotti, Local rule-based explanations of black box decision systems (2018). arXiv:1805.10820. · arXiv
  180. S. Krishnan, E. Wu, Palm: Machine learning explanations for iterative debugging, in: Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics, ACM, 2017, p. 4.
  181. M. Robnik-Šikonja, I. Kononenko, Explaining classifications for individual instances, IEEE Transactions on Knowledge and Data Engineering 20 (5) (2008) 589-600.
  182. M. T. Ribeiro, S. Singh, C. Guestrin, Anchors: High-precision model-agnostic explanations, in: AAAI Conference on Artificial Intelligence, 2018, pp. 1527-1535.
  183. D. Martens, F. Provost, Explaining data-driven document classifications, MIS Quarterly 38 (1) (2014) 73-100.
  184. A. Goldstein, A. Kapelner, J. Bleich, E. Pitkin, Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation, Journal of Computational and Graphical Statistics 24 (1) (2015) 44-65.
  185. G. Casalicchio, C. Molnar, B. Bischl, Visualizing the feature importance for black box models, in: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, 2018, pp. 655-670.
  186. G. Tolomei, F. Silvestri, A. Haines, M. Lalmas, Interpretable predictions of tree-based ensembles via actionable feature tweaking, in: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2017, pp. 465-474.
  187. N. F. Rajani, R. Mooney, Stacking with auxiliary features for visual question answering, in: Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018, pp. 2217-2226.
  188. N. F. Rajani, R. J. Mooney, Ensembling visual explanations, in: Explainable and Interpretable Models in Computer Vision and Machine Learning, Springer, 2018, pp. 155-172.
  189. H. Núñez, C. Angulo, A. Català, Rule-based learning systems for support vector machines, Neural Processing Letters 24 (1) (2006) 1-18.
  190. H. Núñez, C. Angulo, A. Català, Support vector machines with symbolic interpretation, in: VII Brazilian Symposium on Neural Networks, 2002. SBRN 2002. Proceedings., IEEE, 2002, pp. 142-147.
  191. P. Sollich, Probabilistic methods for support vector machines, in: Advances in neural information processing systems, 2000, pp. 349-355.
  192. W. Landecker, M. D. Thomure, L. M. Bettencourt, M. Mitchell, G. T. Kenyon, S. P. Brumby, Interpreting individual classifications of hierarchical networks, in: 2013 IEEE Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2013, pp. 32-38.
  193. A. Jakulin, M. Možina, J. Demšar, I. Bratko, B. Zupan, Nomograms for visualizing support vector machines, in: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, ACM, 2005, pp. 108-117.
  194. L. Fu, Rule generation from neural networks, IEEE Transactions on Systems, Man, and Cybernetics 24 (8) (1994) 1114-1124.
  195. S. Thrun, Extracting rules from artificial neural networks with distributed representations, in: Proceedings of the 7th International Conference on Neural Information Processing Systems, NIPS'94, 1994, pp. 505-512.
  196. I. A. Taha, J. Ghosh, Symbolic interpretation of artificial neural networks, IEEE Transactions on Knowledge and Data Engineering 11 (3) (1999) 448-463.
  197. H. Tsukimoto, Extracting rules from trained neural networks, IEEE Transactions on Neural Networks 11 (2) (2000) 377-389.
  198. J. R. Zilke, E. L. Mencía, F. Janssen, Deepred-rule extraction from deep neural networks, in: International Conference on Discovery Science, Springer, 2016, pp. 457-473.
  199. G. P. J. Schmitz, C. Aldrich, F. S. Gouws, ANN-DT: an algorithm for extraction of decision trees from artificial neural networks, IEEE Transactions on Neural Networks 10 (6) (1999) 1392-1401.
  200. M. Sato, H. Tsukimoto, Rule extraction from neural networks via decision tree induction, in: IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No. 01CH37222), Vol. 3, IEEE, 2001, pp. 1870-1875.

200 of 331 shown

Cited by

Sorted by citations

Authors
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., Chatila, R., & Herrera, F.
Affiliations
  • Tecnalia (ES)
  • Institut national de recherche en sciences et technologies du numérique (FR)
  • Universidad de Granada (ES)
  • Telefónica (Spain) (ES)
  • Sorbonne Université (FR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Volume
58
Pages
82-115
ISSN
1566-2535 (ISSN-L)
DOI
10.48550/arxiv.1910.10045
arXiv
1910.10045
OpenAlex
W2981731882
Cited by
11,321
References
0
Citations by year
2026: 3021 · 2025: 2456 · 2024: 1847 · 2023: 1546 · 2022: 1221 · 2021: 953 · 2020: 268 · 2019: 5 · 2016: 1 · 2014: 1 · 2012: 1
FWCI
276.24
Type
journal article
Language
en
Open access
Licence
Locations
  • Information Fusion (publishedVersion)
  • LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (acceptedVersion, other-oa)
  • Institutional Repository of the University of Granada (University of Granada) (acceptedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • HAL (submittedVersion)
  • BIRD (Basque Center for Applied Mathematics) (submittedVersion, cc-by-nc-sa)
  • TECNALIA Publications (Fundación TECNALIA Research & Innovation) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Fundación BBVA (TIN2017-89517-P, IT1294-19)
  • Eusko Jaurlaritza (IT1294–19)
  • Banco Bilbao Vizcaya Argentaria
  • Ministerio de Asuntos Económicos y Transformación Digital, Gobierno de España
  • Horizon 2020 Framework Programme (825619)
  • Horizon 2020 (825619)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
explainable AI, explainability, explainable machine learning, data fusion, responsible AI, interpretability, fairness and accountability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review