The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks
Abstract
To date, the preponderance of techniques for eliciting the knowledge embedded in trained artificial neural networks (ANN's) has focused primarily on extracting rule-based explanations from feedforward ANN's. The ADT taxonomy for categorizing such techniques was proposed in 1995 to provide a basis for the systematic comparison of the different approaches. This paper shows that not only is this taxonomy applicable to a cross section of current techniques for extracting rules from trained feedforward ANN's but also how the taxonomy can be adapted and extended to embrace a broader range of ANN types (e.g., recurrent neural networks) and explanation structures. In addition the paper identifies some of the key research questions in extracting the knowledge embedded within ANN's including the need for the formulation of a consistent theoretical basis for what has been, until recently, a disparate collection of empirical results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1992) · Cited by 23,431
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Introduction to automata theory, languages and computation
(1981) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(81)90068-9 · Cited by 10,699
Introduction to automata theory, languages, and computation
(1980) · Computer Languages · doi:10.1016/0096-0551(80)90011-9 · Cited by 6,824
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369
Valiant, L. G. (1984) · Communications of the ACM · doi:10.1145/1968.1972 · Cited by 3,279
Stochastic Complexity in Statistical Inquiry
Rissanen, J. (1998) · WORLD SCIENTIFIC eBooks · doi:10.1142/0822 · Cited by 1,539
Survey and critique of techniques for extracting rules from trained artificial neural networks
Andrews, R., Diederich, J., & Tickle, A. B. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(96)81920-4 · Cited by 1,225
Self-citationExtracting refined rules from knowledge-based neural networks
Towell, G. G., & Shavlik, J. W. (1993) · Machine Learning · doi:10.1007/bf00993103 · Cited by 757
Finite State Automata and Simple Recurrent Networks
Cleeremans, A., Servan-Schreiber, D., & McClelland, J. L. (1989) · Neural Computation · doi:10.1162/neco.1989.1.3.372 · Cited by 458
Research works printed but not linked: 14
- tickle, 1996, recent developments in techniques for extracting rules from trained artificial neural networks, Herbstschule Konnektionismus (HeKonn 96)
- saito, 1996, law discovery using neural networks, Proc NIPS 96 Rule Extraction From Trained Artificial Neural Networks Wkshp, 62
- 10.1162/neco.1997.9.1.205 · doi:10.1162/neco.1997.9.1.205
- filer, 1996, a comparison between two rule extraction methods for continuous input data, Proc NIPS 97 Rule Extraction From Trained Artificial Neural Networks Wkshp, 38
- 10.1109/ICNN.1997.614691 · DOI
- golea, 1996, on the complexity of rule extraction from neural networks and network querying, Proc Rule Extraction From Trained Artificial Neural Networks Wkshp Society for the Study of Artificial Intelligence and Simulation of Behavior Workshop Series (AISB 96), 51
- golea, 1997, On the complexity of extracting simple rules from trained neural nets
- 10.1109/91.660809 · doi:10.1109/91.660809
- craven, 1994, using sampling and queries to extract rules from trained neural networks, machine learning, Proc 11th Int Conf, 73
- 10.1016/0364-0213(90)90002-E · doi:10.1016/0364-0213(90)90002-e · Paper page
- hume, 1950, The Happy Way to Reading
- krishnan, 1996, a systematic method for decompositional rule extraction from neural networks, Proc NIPS 97 Rule Extraction From Trained Artificial Neural Networks Wkshp, 38
- 10.1016/0950-7051(96)81917-4 · doi:10.1016/0950-7051(96)81917-4
- tickle, 1997, Machine learning neural networks and information security Techniques for extracting rules from trained feedforward artificial neural networks and their application in an information security problem domain
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Neural networks for classification: a survey
Zhang, G. P. (2000) · IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/5326.897072 · Cited by 1,667
Metabolomics by numbers: acquiring and understanding global metabolite data
Goodacre, R., Vaidyanathan, S., Dunn, W. B., Harrigan, G. G., & Kell, D. B. (2004) · Trends in Biotechnology · doi:10.1016/j.tibtech.2004.03.007 · Cited by 1,249
Rotating machinery prognostics: State of the art, challenges and opportunities
Heng, A., Zhang, S., Tan, A. C. C., & Mathew, J. (2009) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2008.06.009 · Cited by 1,207
Image processing with neural networks—a review
Egmont-Petersen, M., de Ridder, D., & Handels, H. (2002) · Pattern Recognition · doi:10.1016/s0031-3203(01)00178-9 · Cited by 1,186
A Survey on Neural Network Interpretability
Zhang, Y., Tino, P., Leonardis, A., & Tang, K. (2021) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2021.3100641 · Cited by 923
Neuro-fuzzy rule generation: survey in soft computing framework
Mitra, S., & Hayashi, Y. (2000) · IEEE Transactions on Neural Networks · doi:10.1109/72.846746 · Cited by 726
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691
“The increasing popularity of machine learning models as decision making tools presents novel challenges for explanation, as it may not be as straightforward to extract the embedded ‘knowledge’ behind their outputs [73].”
BackgroundData mining in soft computing framework: a survey
Mitra, S., Pal, S. K., & Mitra, P. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/72.977258 · Cited by 617
Shahid, N., Rappon, T., & Berta, W. (2019) · PLOS ONE · doi:10.1371/journal.pone.0212356 · Cited by 529
Veale, M., & Binns, R. (2017) · doi:10.31235/osf.io/ustxg · Cited by 512
Veale, M., & Binns, R. (2017) · Big Data & Society · doi:10.1177/2053951717743530 · Cited by 512
The Ethics of Artificial Intelligence: Principles, Challenges, and Opportunities
Floridi, L. (2023) · doi:10.1093/oso/9780198883098.001.0001 · Cited by 507
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490
“Research stemming from the early days of machine learning in expert systems attempts to make algorithmic logics more transparent [75, 81, 54, 66], or to better understand how users work with inductive systems and build mental models of them [60, 48, 77].”
BackgroundUsing sensitivity analysis and visualization techniques to open black box data mining models
Cortez, P., & Embrechts, M. J. (2013) · Information Sciences · doi:10.1016/j.ins.2012.10.039 · Cited by 443
Noisy Time Series Prediction using Recurrent Neural Networks and Grammatical Inference
Giles, C. L., Lawrence, S., & Tsoi, A. C. (2001) · Machine Learning · doi:10.1023/a:1010884214864 · Cited by 374
The ethics of algorithms: key problems and solutions
Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01154-8 · Cited by 384
Ethical Decision Making during Automated Vehicle Crashes
Goodall, N. J. (2014) · Transportation Research Record: Journal of the Transportation Research Board · doi:10.3141/2424-07 · Cited by 292
Explaining models: an empirical study of how explanations impact fairness judgment
Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K. E., & Dugan, C. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302310 · Cited by 286
“Given the complexity of current ML models, explanations are often pedagogical [34], meaning that they reveal information about how the model works without faithfully representing the algorithms.”
BackgroundA new methodology of extraction, optimization and application of crisp and fuzzy logical rules
Duch, W., Adamczak, R., & Grabczewski, K. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.914524 · Cited by 272
'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
(2018) · doi:10.31235/osf.io/9wqxr · Cited by 161
Discrimination, artificial intelligence, and algorithmic decision-making
Borgesius, F. Z. (2025) · arXiv · doi:10.48550/arxiv.2510.13465 · Cited by 152
Enslaving the Algorithm: From a “Right to an Explanation” to a “Right to Better Decisions”?
Edwards, L., & Veale, M. (2018) · IEEE Security & Privacy · doi:10.1109/msp.2018.2701152 · Cited by 141
“It is sometimes possible however to devise explanations of systems which do not need access to the proprietary innards of an ML model using “pedagogical” or “model-agnostic” methods [10, 12].”
A contemporary review of machine learning in otolaryngology–head and neck surgery
Crowson, M. G., Ranisau, J., Eskander, A., Babier, A., Xu, B., Kahmke, R. R., Chen, J. M., & Chan, T. C. Y. (2020) · The Laryngoscope · doi:10.1002/lary.27850 · Cited by 127
The Ethics of Algorithms: Key Problems and Solutions
Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2021) · Philosophical Studies Series · doi:10.1007/978-3-030-81907-1_8 · Cited by 121
An Analysis of ISO 26262: Machine Learning and Safety in Automotive Software
Salay, R., Queiroz, R., & Czarnecki, K. (2018) · SAE Technical Paper Series · doi:10.4271/2018-01-1075 · Cited by 109
Contestable AI by Design: Towards a Framework
Alfrink, K., Keller, I., Kortuem, G., & Doorn, N. (2022) · Minds and Machines · doi:10.1007/s11023-022-09611-z · Cited by 169
“However, once again we emphasize not to fixate on the technical components of AI systems in isolation (Hirsch et al., 2017; Leahu, 2016; Mahendran & Vedaldi, 2015; Ribeiro et al., 2016; Tickle et al., 1998 in Edwards & Veale, 2018).”
Veale, M., Van Kleek, M., & Binns, R. (2018) · doi:10.31235/osf.io/8kvf4 · Cited by 60
Salimparsa, M., Sedig, K., Lizotte, D. J., Abdullah, S. S., Chalabianloo, N., & Muanda, F. T. (2025) · Informatics · doi:10.3390/informatics12040119 · Cited by 39
Metadata
- Authors
- Tickle, A. B., Andrews, R., Golea, M., & Diederich, J.
- Venue
- IEEE Transactions on Neural Networks
- Published
- 1998
- Volume
- 9
- Issue
- 6
- Pages
- 1057-1068
- ISSN
- 1045-9227 (ISSN-L)
- OpenAlex
- W2112036221
- Cited by
- 438
- References
- 34
- FWCI
- 14.88
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.