Survey and critique of techniques for extracting rules from trained artificial neural networks

Adaptive Mixtures of Local Experts

Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384 · see citing works

The Cascade-Correlation learning architecture

Fahlman, S. E., & Lebiere, C. J. (2018) · doi:10.1184/r1/6610403 · Cited by 2,391 · see citing works

On fuzzy modeling using fuzzy neural networks with the back-propagation algorithm

Horikawa, S., Furuhashi, T., & Uchikawa, Y. (1992) · IEEE Transactions on Neural Networks · doi:10.1109/72.159069 · Cited by 874 · see citing works

Extracting Refined Rules from Knowledge-Based Neural Networks

Towell, G. G., & SHAVLIK, J. W. (1993) · Machine Learning · doi:10.1023/a:1022683529158 · Cited by 756 · see citing works

Connectionist expert systems

Gallant, S. I. (1988) · Communications of the ACM · doi:10.1145/42372.42377 · Cited by 666 · see citing works

Production Rules as a Representation for a Knowledge-Based Consultation Program

Davis, R., Buchanan, B., & Shortliffe, E. (1977) · Artificial Intelligence · doi:10.1016/0004-3702(77)90003-0 · Cited by 569 · see citing works

Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks

Giles, C. L., Miller, C. B., Chen, D., Chen, H.-H., Sun, G.-Z., & Lee, Y.-C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.393 · Cited by 513 · see citing works

The MONK's problems: A Performance Comparison of Different Learning Algorithms

Thrun, S., Bala, J. W., Bloedorn, E., et al. (1991) · doi:10.13021/mars/3370 · Cited by 447 · see citing works

Rule generation from neural networks

Fu, L. (1994) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/21.299696 · Cited by 368 · see citing works

Using Sampling and Queries to Extract Rules from Trained Neural Networks

Craven, M. W., & SHAVLIK, J. W. (1994) · Elsevier eBooks · doi:10.1016/b978-1-55860-335-6.50013-1 · Cited by 312 · see citing works

Neural networks in designing fuzzy systems for real world applications

Halgamuge, S. K., & Glesner, M. (1994) · Fuzzy Sets and Systems · doi:10.1016/0165-0114(94)90242-9 · Cited by 247 · see citing works

Medical diagnostic expert system based on PDP model

Saito, & Nakano (1988) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1988.23855 · Cited by 209 · see citing works

THE EXTRACTION OF REFINED RULES FROM KNOWLEDGE BASED NEURAL NETWORKS

Towell, G. G., & SHAVLIK, J. W. (1993) · Cited by 198 · see citing works

Rule learning by searching on adapted nets

Fu, L. (1991) · National Conference on Artificial Intelligence · Cited by 161 · see citing works

Induction of Finite-State Languages Using Second-Order Recurrent Networks

Watrous, R. L., & Kuhn, G. M. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.406 · Cited by 146 · see citing works

A reactive approach to explanation

Moore, J. D., & Swartout, W. (1989) · Cited by 138 · see citing works

AUTOMATED KNOWLEDGE ACQUISITION

(1991) · Elsevier eBooks · doi:10.1016/b978-1-55860-200-7.50134-3 · Cited by 116 · see citing works

Extracting Provably Correct Rules from Artificial Neural Networks

Thrun, S. (1993) · Cited by 114 · see citing works

Extraction, Insertion and Refinement of Symbolic Rules in Dynamically Driven Recurrent Neural Networks

Giles, C. L., & Omlin, C. W. P. (1993) · Connection Science · doi:10.1080/09540099308915703 · Cited by 89 · see citing works

A reactive approach to explanation in expert and advice-giving systems

Moore, J. D., Estrin, G., & Flowers, M. (1989) · Cited by 83 · see citing works

Network Structuring and Training Using Rule-based Knowledge

Tresp, V., Hollatz, J., & Ahmad, S. (1992) · Cited by 80 · see citing works

Optimized rule induction

Weiss, S. M., & Indurkhya, N. (1993) · IEEE Expert · doi:10.1109/64.248354 · Cited by 72 · see citing works

Fuzzy MLP based expert system for medical diagnosis

Mitra, S. (1994) · Fuzzy Sets and Systems · doi:10.1016/0165-0114(94)90025-6 · Cited by 70 · see citing works

Refinement of Approximate Reasoning-based Controllers by Reinforcement Learning

Berenji, H. R., & Software, S. (1991) · Elsevier eBooks · doi:10.1016/b978-1-55860-200-7.50097-0 · Cited by 46 · see citing works

Rule refinement with recurrent neural networks

Giles, C. L., & Omlin, C. W. P. (2002) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1993.298658 · Cited by 31 · see citing works

Explanation and dialogue

Gilbert, N. (1989) · The Knowledge Engineering Review · doi:10.1017/s026988890000504x · Cited by 27 · see citing works

Heuristics for the extraction of rules from discrete-time recurrent neural networks

Omlin, C. W. P., Giles, C. L., & MILLER, C. B. (2003) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1992.287212 · Cited by 24 · see citing works

Deep learning in neural networks: an overview

Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works

Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)

Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works

Explaining Explanations: An Overview of Interpretability of Machine Learning

Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · see citing works

Machine Learning Interpretability: A Survey on Methods and Metrics

Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · see citing works

Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence

Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · see citing works

Explainable machine learning for scientific insights and discoveries

Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J. (2020) · IEEE Access · doi:10.1109/access.2020.2976199 · Cited by 1,040 · see citing works

A Survey on the Explainability of Supervised Machine Learning

Burkart, N., & Huber, M. F. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12228 · Cited by 1,008 · see citing works

Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda

Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743 · see citing works

Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges

Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677 · see citing works

From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI

Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590 · see citing works

<i>Why and why not</i> explanations improve the intelligibility of context-aware intelligent systems

Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552 · see citing works

Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data

Veale, M., & Binns, R. (2017) · Big Data &amp; Society · doi:10.1177/2053951717743530 · Cited by 512 · see citing works

An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models

Huysmans, J., Dejaeger, K., Mues, C., Vanthienen, J., & Baesens, B. (2011) · Decision Support Systems · doi:10.1016/j.dss.2010.12.003 · Cited by 425 · see citing works

Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review

Verma, S., Boonsanong, V., Hoang, M., Hines, K. E., Dickerson, J. P., & Shah, C. (2020) · arXiv · doi:10.48550/arxiv.2010.10596 · Cited by 267 · see citing works

Rulematrix: Visualizing and understanding classifiers with rules

Ming, Y., Qu, H., & Bertini, E. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864812 · Cited by 243 · see citing works

Interpretation of Quantitative Structure-Activity Relationship Models: Past, Present, and Future

Polishchuk, P. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.7b00274 · Cited by 236 · see citing works

A Survey of Data-driven and Knowledge-aware eXplainable AI

Li, X.-H., Cao, C. C., Shi, Y., Bai, W., Gao, H., Qiu, L., Wang, C., Gao, Y., Zhang, S., Xue, X., et al. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2983930 · Cited by 224 · see citing works

Data mining techniques for software effort estimation: a comparative study

Dejaeger, K., Verbeke, W., Martens, D., & Baesens, B. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.55 · Cited by 224 · see citing works

A Survey of Methods for Explaining Black Box Models

Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works

Perturbation-Based Explanations of Prediction Models

Robnik-Šikonja, M., & Bohanec, M. (2018) · Human–Computer Interaction Series · doi:10.1007/978-3-319-90403-0_9 · Cited by 171 · see citing works

A review of possible effects of cognitive biases on interpretation of rule-based machine learning models

Kliegr, T., Bahník, Š., & Fürnkranz, J. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103458 · Cited by 139 · see citing works

Black Box Attacks on Explainable Artificial Intelligence(XAI) methods in Cyber Security

Kuppa, A., & Le-Khac, N.-A. (2020) · 2020 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206780 · Cited by 123 · see citing works

Artificial Neural Networks (ANNs) as a Novel Modeling Technique in Tribology

Argatov, I. (2019) · Frontiers in Mechanical Engineering · doi:10.3389/fmech.2019.00030 · Cited by 97 · see citing works

A peek into the black box: exploring classifiers by randomization

Henelius, A., Puolamäki, K., Boström, H., Asker, L., & Papapetrou, P. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0368-8 · Cited by 98 · see citing works

Artificial intelligence in safety-critical systems: a systematic review

Wang, Y., & Chung, S. H. (2022) · Industrial Management &amp; Data Systems · doi:10.1108/imds-07-2021-0419 · Cited by 96 · see citing works

Explainable Artificial Intelligence: a Systematic Review

Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269 · see citing works

How Case-Based Reasoning Explains Neural Networks: A Theoretical Analysis of XAI Using Post-Hoc Explanation-by-Example from a Survey of ANN-CBR Twin-Systems

Keane, M. T., & Kenny, E. M. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-29249-2_11 · Cited by 84 · see citing works

Explainable goal-driven agents and robots-a comprehensive review

Sado, F., Loo, C. K., Liew, W. S., Kerzel, M., & Wermter, S. (2023) · ACM Computing Surveys · doi:10.1145/3564240 · Cited by 79 · see citing works

Legal and technical feasibility of the GDPR’s quest for explanation of algorithmic decisions: of black boxes, white boxes and fata morganas

BRKAN, M., & BONNET, G. (2020) · European Journal of Risk Regulation · doi:10.1017/err.2020.10 · Cited by 63 · see citing works

Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data

Veale, M., & Binns, R. (2017) · doi:10.31235/osf.io/ustxg · Cited by 512 · see citing works

Developing artificial neural networks for safety critical systems

Kurd, Z., Kelly, T., & Austin, J. (2006) · Neural Computing and Applications · doi:10.1007/s00521-006-0039-9 · Cited by 55 · see citing works

Conceptual Explanations of Neural Network Prediction for Time Series

Kusters, F., Schichtel, P., Ahmed, S., & Dengel, A. (2020) · 2020 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207341 · Cited by 8 · see citing works

Physician-Friendly Machine Learning: A Case Study with Cardiovascular Disease Risk Prediction

Padmanabhan, M., Yuan, P., Chada, G., & Nguyen, H. V. (2019) · Journal of Clinical Medicine · doi:10.3390/jcm8071050 · Cited by 75 · see citing works

Breakdown of Machine Learning Algorithms

Meena, M., & Sehrawat, R. (2022) · Recent Trends in Artificial Intelligence & it s Applications · doi:10.46610/rtaia.2022.v01i03.005 · Cited by 1 · see citing works

Application of Machine Learning in Animal Disease Analysis and Prediction

Zhang, S., Su, Q., & Chen, Q. (2020) · Current Bioinformatics · doi:10.2174/1574893615999200728195613 · Cited by 29 · see citing works

Sentiment Analysis by Using Supervised Machine Learning and Deep Learning Approaches

Naeem, S., Logofătu, D., & Muharemi, F. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-63119-2_39 · Cited by 3 · see citing works

Artificial Intelligence, Machine Learning, and Medicine: A Little Background Goes a Long Way Toward Understanding

Cote, M. P., Lubowitz, J. H., Brand, J. C., & Rossi, M. J. (2021) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2021.04.022 · Cited by 31 · see citing works