The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks
Abstract
To date, the preponderance of techniques for eliciting the knowledge embedded in trained artificial neural networks (ANN's) has focused primarily on extracting rule-based explanations from feedforward ANN's. The ADT taxonomy for categorizing such techniques was proposed in 1995 to provide a basis for the systematic comparison of the different approaches. This paper shows that not only is this taxonomy applicable to a cross section of current techniques for extracting rules from trained feedforward ANN's but also how the taxonomy can be adapted and extended to embrace a broader range of ANN types (e.g., recurrent neural networks) and explanation structures. In addition the paper identifies some of the key research questions in extracting the knowledge embedded within ANN's including the need for the formulation of a consistent theoretical basis for what has been, until recently, a disparate collection of empirical results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1992) · Cited by 23,431
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Introduction to automata theory, languages and computation
(1981) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(81)90068-9 · Cited by 10,699
Introduction to automata theory, languages, and computation
(1980) · Computer Languages · doi:10.1016/0096-0551(80)90011-9 · Cited by 6,824
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369
Valiant, L. G. (1984) · Communications of the ACM · doi:10.1145/1968.1972 · Cited by 3,279
Stochastic Complexity in Statistical Inquiry
Rissanen, J. (1998) · WORLD SCIENTIFIC eBooks · doi:10.1142/0822 · Cited by 1,539
Survey and critique of techniques for extracting rules from trained artificial neural networks
Andrews, R., Diederich, J., & Tickle, A. B. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(96)81920-4 · Cited by 1,225
Self-citationExtracting refined rules from knowledge-based neural networks
Towell, G. G., & Shavlik, J. W. (1993) · Machine Learning · doi:10.1007/bf00993103 · Cited by 757
Finite State Automata and Simple Recurrent Networks
Cleeremans, A., Servan-Schreiber, D., & McClelland, J. L. (1989) · Neural Computation · doi:10.1162/neco.1989.1.3.372 · Cited by 458
Research works printed but not linked: 14
- tickle, 1996, recent developments in techniques for extracting rules from trained artificial neural networks, Herbstschule Konnektionismus (HeKonn 96)
- saito, 1996, law discovery using neural networks, Proc NIPS 96 Rule Extraction From Trained Artificial Neural Networks Wkshp, 62
- 10.1162/neco.1997.9.1.205 · doi:10.1162/neco.1997.9.1.205
- filer, 1996, a comparison between two rule extraction methods for continuous input data, Proc NIPS 97 Rule Extraction From Trained Artificial Neural Networks Wkshp, 38
- 10.1109/ICNN.1997.614691 · DOI
- golea, 1996, on the complexity of rule extraction from neural networks and network querying, Proc Rule Extraction From Trained Artificial Neural Networks Wkshp Society for the Study of Artificial Intelligence and Simulation of Behavior Workshop Series (AISB 96), 51
- golea, 1997, On the complexity of extracting simple rules from trained neural nets
- 10.1109/91.660809 · doi:10.1109/91.660809
- craven, 1994, using sampling and queries to extract rules from trained neural networks, machine learning, Proc 11th Int Conf, 73
- 10.1016/0364-0213(90)90002-E · doi:10.1016/0364-0213(90)90002-e · Paper page
- hume, 1950, The Happy Way to Reading
- krishnan, 1996, a systematic method for decompositional rule extraction from neural networks, Proc NIPS 97 Rule Extraction From Trained Artificial Neural Networks Wkshp, 38
- 10.1016/0950-7051(96)81917-4 · doi:10.1016/0950-7051(96)81917-4
- tickle, 1997, Machine learning neural networks and information security Techniques for extracting rules from trained feedforward artificial neural networks and their application in an information security problem domain
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Rotating machinery prognostics: State of the art, challenges and opportunities
Heng, A., Zhang, S., Tan, A. C. C., & Mathew, J. (2009) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2008.06.009 · Cited by 1,207
A Survey on Neural Network Interpretability
Zhang, Y., Tino, P., Leonardis, A., & Tang, K. (2021) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2021.3100641 · Cited by 923
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691
“The increasing popularity of machine learning models as decision making tools presents novel challenges for explanation, as it may not be as straightforward to extract the embedded ‘knowledge’ behind their outputs [73].”
Background
Metadata
- Authors
- Tickle, A. B., Andrews, R., Golea, M., & Diederich, J.
- Venue
- IEEE Transactions on Neural Networks
- Published
- 1998
- Volume
- 9
- Issue
- 6
- Pages
- 1057-1068
- ISSN
- 1045-9227 (ISSN-L)
- OpenAlex
- W2112036221
- Cited by
- 438
- References
- 34
- FWCI
- 14.88
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks. https://app.cochat.ai/papers/10.1109/72.728352
- The paper
- Tickle, A. B., Andrews, R., Golea, M., & Diederich, J. (1998). The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks. IEEE Transactions on Neural Networks. https://doi.org/10.1109/72.728352
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
