The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks
Abstract
To date, the preponderance of techniques for eliciting the knowledge embedded in trained artificial neural networks (ANN's) has focused primarily on extracting rule-based explanations from feedforward ANN's. The ADT taxonomy for categorizing such techniques was proposed in 1995 to provide a basis for the systematic comparison of the different approaches. This paper shows that not only is this taxonomy applicable to a cross section of current techniques for extracting rules from trained feedforward ANN's but also how the taxonomy can be adapted and extended to embrace a broader range of ANN types (e.g., recurrent neural networks) and explanation structures. In addition the paper identifies some of the key research questions in extracting the knowledge embedded within ANN's including the need for the formulation of a consistent theoretical basis for what has been, until recently, a disparate collection of empirical results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1992) · Cited by 23,431
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Introduction to automata theory, languages and computation
(1981) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(81)90068-9 · Cited by 10,699
Introduction to automata theory, languages, and computation
(1980) · Computer Languages · doi:10.1016/0096-0551(80)90011-9 · Cited by 6,824
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369
Valiant, L. G. (1984) · Communications of the ACM · doi:10.1145/1968.1972 · Cited by 3,279
Stochastic Complexity in Statistical Inquiry
Rissanen, J. (1998) · WORLD SCIENTIFIC eBooks · doi:10.1142/0822 · Cited by 1,539
Survey and critique of techniques for extracting rules from trained artificial neural networks
Andrews, R., Diederich, J., & Tickle, A. B. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(96)81920-4 · Cited by 1,225
Self-citationExtracting refined rules from knowledge-based neural networks
Towell, G. G., & Shavlik, J. W. (1993) · Machine Learning · doi:10.1007/bf00993103 · Cited by 757
Finite State Automata and Simple Recurrent Networks
Cleeremans, A., Servan-Schreiber, D., & McClelland, J. L. (1989) · Neural Computation · doi:10.1162/neco.1989.1.3.372 · Cited by 458
The MONK's problems: A Performance Comparison of Different Learning Algorithms
Thrun, S., Bala, J. W., Bloedorn, E., Bratko, I., Cestnik, B., Cheng, J. Q., Jong, K. A. D., Džeroski, S., Fisher, D., Fahlman, S. E., et al. (1991) · doi:10.13021/mars/3370 · Cited by 445
Durbin, R., & Rumelhart, D. E. (1989) · Neural Computation · doi:10.1162/neco.1989.1.1.133 · Cited by 409
Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks
Giles, C. L., Miller, C. B., Chen, D., Chen, H. H., Sun, G. Z., & Lee, Y. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.393 · Cited by 392
Rule generation from neural networks
Fu, L. (1994) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.299696 · Cited by 367
Extracting comprehensible models from trained neural networks
Craven, M. W., & SHAVLIK, J. W. (1996) · Cited by 229
THE EXTRACTION OF REFINED RULES FROM KNOWLEDGE BASED NEURAL NETWORKS
Towell, G. G., & SHAVLIK, J. W. (1993) · Cited by 197
Extraction of rules from discrete-time recurrent neural networks
Omlin, C. W., & Giles, C. L. (1996) · Neural Networks · doi:10.1016/0893-6080(95)00086-0 · Cited by 187
Constructing deterministic finite-state automata in recurrent neural networks
Omlin, C. W., & Giles, C. L. (1996) · Journal of the ACM · doi:10.1145/235809.235811 · Cited by 178
Rule learning by searching on adapted nets
Fu, L. (1991) · National Conference on Artificial Intelligence · Cited by 165
Rule Extraction: From Neural Architecture to Symbolic Representation
Carpenter, G. A., & Tan, A. (1995) · Connection Science · doi:10.1080/09540099508915655 · Cited by 152
Extracting Provably Correct Rules from Artificial Neural Networks
Thrun, S. (1993) · Cited by 135
Giles, C. L., & Omlin, C. W. P. (1993) · Connection Science · doi:10.1080/09540099308915703 · Cited by 99
ID2-of-3: Constructive Induction of M-of-N Concepts for Discriminators in Decision Trees
Murphy, P. M., & Pazzani, M. J. (1991) · Elsevier eBooks · doi:10.1016/b978-1-55860-200-7.50040-4 · Cited by 98
Cascade ARTMAP: integrating neural computation and symbolic knowledge processing
Tan, A. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.557661 · Cited by 96
Rule revision with recurrent neural networks
Omlin, C. W. P., & Giles, C. L. (1996) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/69.485647 · Cited by 65
Liu, H., & Tan, S. T. (1995) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.1995.538006 · Cited by 54
Dynamically adding symbolically meaningful nodes to knowledge-based neural networks
Pitz, D. W., & SHAVLIK, J. W. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(96)81915-0 · Cited by 39
Rule refinement with recurrent neural networks
Giles, C. L., & Omlin, C. W. P. (2002) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1993.298658 · Cited by 32
Schellhammer, I., Diederich, J., Towsey, M., & Brugman, C. (1998) · Conference on Computational Natural Language Learning · doi:10.3115/1603899.1603912 · Cited by 25
Self-citationA partial order for the M-of-N rule-extraction algorithm
Maire, F. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.641475 · Cited by 21
Tickle, A. B., Golea, M., Hayward, R. F., & Diederich, J. (2002) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1997.614691 · Cited by 9
Self-citationLearning to predict reading frames in E. coli DNA sequences
Craven, M. W., & SHAVLIK, J. W. (2002) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.1993.270613 · Cited by 5
Andrews, R., Cable, R., Diederich, J., Geva, S., Golea, M., Hayward, R. F., Ho-Stuart, C., & Tickle, A. B. (2018) · doi:10.13140/rg.2.2.36356.01921 · Cited by 3
Self-citationValiant, L. G. (2026) · Foundations of Computation and Machine Learning · doi:10.1145/3749740.3749746
Research works printed but not linked: 14
- tickle, 1996, recent developments in techniques for extracting rules from trained artificial neural networks, Herbstschule Konnektionismus (HeKonn 96)
- saito, 1996, law discovery using neural networks, Proc NIPS 96 Rule Extraction From Trained Artificial Neural Networks Wkshp, 62
- 10.1162/neco.1997.9.1.205 · doi:10.1162/neco.1997.9.1.205
- filer, 1996, a comparison between two rule extraction methods for continuous input data, Proc NIPS 97 Rule Extraction From Trained Artificial Neural Networks Wkshp, 38
- 10.1109/ICNN.1997.614691 · DOI
- golea, 1996, on the complexity of rule extraction from neural networks and network querying, Proc Rule Extraction From Trained Artificial Neural Networks Wkshp Society for the Study of Artificial Intelligence and Simulation of Behavior Workshop Series (AISB 96), 51
- golea, 1997, On the complexity of extracting simple rules from trained neural nets
- 10.1109/91.660809 · doi:10.1109/91.660809
- craven, 1994, using sampling and queries to extract rules from trained neural networks, machine learning, Proc 11th Int Conf, 73
- 10.1016/0364-0213(90)90002-E · doi:10.1016/0364-0213(90)90002-e · Paper page
- hume, 1950, The Happy Way to Reading
- krishnan, 1996, a systematic method for decompositional rule extraction from neural networks, Proc NIPS 97 Rule Extraction From Trained Artificial Neural Networks Wkshp, 38
- 10.1016/0950-7051(96)81917-4 · doi:10.1016/0950-7051(96)81917-4
- tickle, 1997, Machine learning neural networks and information security Techniques for extracting rules from trained feedforward artificial neural networks and their application in an information security problem domain
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Neural networks for classification: a survey
Zhang, G. P. (2000) · IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/5326.897072 · Cited by 1,667
Metabolomics by numbers: acquiring and understanding global metabolite data
Goodacre, R., Vaidyanathan, S., Dunn, W. B., Harrigan, G. G., & Kell, D. B. (2004) · Trends in Biotechnology · doi:10.1016/j.tibtech.2004.03.007 · Cited by 1,249
Rotating machinery prognostics: State of the art, challenges and opportunities
Heng, A., Zhang, S., Tan, A. C. C., & Mathew, J. (2009) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2008.06.009 · Cited by 1,207
Metadata
- Authors
- Tickle, A. B., Andrews, R., Golea, M., & Diederich, J.
- Venue
- IEEE Transactions on Neural Networks
- Published
- 1998
- Volume
- 9
- Issue
- 6
- Pages
- 1057-1068
- ISSN
- 1045-9227 (ISSN-L)
- OpenAlex
- W2112036221
- Cited by
- 438
- References
- 34
- FWCI
- 14.88
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.