The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
"What is relevant in a text document?": An interpretable machine learning approach
Open accessAbstract
Text documents can be described by a number of abstract concepts such as semantic category, writing style, or sentiment. Machine learning (ML) models have been trained to automatically map documents to these abstract concepts, allowing to annotate very large text collections, more than could be processed by a human in a lifetime. Besides predicting the text's category very accurately, it is also highly desirable to understand how and why the categorization process takes place. In this paper, we demonstrate that such understanding can be achieved by tracing the classification decision back to individual words using layer-wise relevance propagation (LRP), a recently developed technique for explaining predictions of complex non-linear classifiers. We train two word-based ML models, a convolutional neural network (CNN) and a bag-of-words SVM classifier, on a topic categorization task and adapt the LRP method to decompose the predictions of these models onto words. Resulting scores indicate how much individual words contribute to the overall classification decision. This enables one to distill relevant information from text documents without an explicit semantic information extraction step. We further use the word-wise relevance scores for generating novel vector-based document representations which capture semantic information. Based on these document vectors, we introduce a measure of model explanatory power and show that, although the SVM and CNN models perform similarly in terms of classification accuracy, the latter exhibits a higher level of explainability which makes it more comprehensible for humans and potentially more useful for other applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
A vector space model for automatic indexing
Salton, G., Wong, A., & Yang, C. S. (1975) · Communications of the ACM · doi:10.1145/361219.361220 · Cited by 7,521
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Self-citationQuantum-chemical insights from deep tensor neural networks
Schütt, K. T., Arbabzadah, F., Chmiela, S., Müller, K. R., & Tkatchenko, A. (2017) · Nature Communications · doi:10.1038/ncomms13890 · Cited by 1,445
Self-citationExplaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357
Self-citationEvaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164
Self-citation
Research works printed but not linked: 26
- KS Jones, 1972, A statistical interpretation of term specificity and its application in retrieval, Journal of Documentation, 28, 11, 10.1108/eb026526 · DOI · Paper page
- Hasan KS, Ng V. Conundrums in Unsupervised Keyphrase Extraction: Making Sense of the State-of-the-Art. In: Proceedings of the 23rd International Conference on Computational Linguistics: Posters (COLING); 2010. p. 365–373.
- Y Bengio, 2003, A Neural Probabilistic Language Model, Journal of Machine Learning Research (JMLR), 3, 1137
- R Collobert, 2011, Natural Language Processing (Almost) from Scratch, Journal of Machine Learning Research (JMLR), 12, 2493
- Socher R, Perelygin A, Wu J, Chuang J, Manning CD, Ng A, et al. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics; 2013. p. 1631–1642. · DOI · Paper page
- Zeiler MD, Fergus R. Visualizing and Understanding Convolutional Networks. In: Computer Vision—ECCV 2014: 13th European Conference; 2014. p. 818–833. · DOI · Paper page
- Landecker W, Thomure MD, Bettencourt LMA, Mitchell M, Kenyon GT, Brumby SP. Interpreting Individual Classifications of Hierarchical Networks. In: IEEE Symposium on Computational Intelligence and Data Mining (CIDM); 2013. p. 32–38. · DOI
- Arbabzadah F, Montavon G, Müller KR, Samek W. Identifying Individual Facial Expressions by Deconstructing a Neural Network. In: Pattern Recognition—38th German Conference, GCPR 2016. vol. 9796 of LNCS. Springer; 2016. p. 344–354. · DOI
- I Sturm, 2016, Interpretable Deep Neural Networks for Single-Trial EEG Classification, Journal of Neuroscience Methods, 274, 141, 10.1016/j.jneumeth.2016.10.008 · DOI · Paper page
- Lapuschkin S, Binder A, Montavon G, Müller KR, Samek W. Analyzing Classifiers: Fisher Vectors and Deep Neural Networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016. p. 2912–2920. · DOI
- Poulin B, Eisner R, Szafron D, Lu P, Greiner R, Wishart DS, et al. Visual Explanation of Evidence in Additive Classifiers. In: Proceedings of the 18th Conference on Innovative Applications of Artificial Intelligence (IAAI). AAAI Press; 2006. p. 1822–1829.
- D Baehrens, 2010, How to Explain Individual Classification Decisions, Journal of Machine Learning Research (JMLR), 11, 1803
- E Strumbelj, 2010, An Efficient Explanation of Individual Classifications using Game Theory, Journal of Machine Learning Research (JMLR), 11, 1
- Turner R. A Model Explanation System: Latest Updates and Extensions. arXiv. 2016;1606.09517.
- M Gevrey, 2003, Review and comparison of methods to study the contribution of variables in artificial neural network models, Ecological Modelling, 160, 249, 10.1016/S0304-3800(02)00257-0 · DOI · Paper page
- Montavon G, Samek W, Müller KR. Methods for Interpreting and Understanding Deep Neural Networks. arXiv. 2017;1706.07979.
- Denil M, Demiraj A, de Freitas N. Extraction of Salient Sentences from Labelled Documents. arXiv. 2014;1412.6815.
- Li J, Chen X, Hovy E, Jurafsky D. Visualizing and Understanding Neural Models in NLP. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT); 2016. p. 681–691. · DOI
- Arras L, Horn F, Montavon G, Müller KR, Samek W. Explaining Predictions of Non-Linear Classifiers in NLP. In: Proceedings of the 1st Workshop on Representation Learning for NLP. Association for Computational Linguistics; 2016. p. 1–7. · DOI
- Ding Y, Liu Y, Luan H, Sun, M. Visualizing and Understanding Neural Machine Translation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics; 2017. · DOI
- Kim Y. Convolutional Neural Networks for Sentence Classification. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP); 2014. p. 1746–1751. · DOI · Paper page
- Mnih A, Hinton G. Three New Graphical Models for Statistical Language Modelling. In: Proceedings of the International Conference on Machine Learning (ICML); 2007. p. 641–648. · DOI · Paper page
- Mnih A, Teh YW. A fast and simple algorithm for training neural probabilistic language models. In: Proceedings of the International Conference on Machine Learning (ICML); 2012. p. 1751–1758.
- S Lapuschkin, 2016, The Layer-wise Relevance Propagation Toolbox for Artificial Neural Networks, Journal of Machine Learning Research, 17, 1
- Zhang X, Zhao J, LeCun Y. Character-level Convolutional Networks for Text Classification. In: Advances in Neural Information Processing Systems 28 (NIPS); 2015. p. 649–657.
- C Nadeau, 2003, Inference for the Generalization Error, Machine Learning, 52, 239, 10.1023/A:1024068626366 · DOI · Paper page
Cited by
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665
Self-citationA Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
“The resultant values could be in the form of probabilities (the LIME model [34]), super-pixels (ACE algorithm [40]), and heatmaps (CAM and LRP [41–45]).”
BackgroundExplaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477
“At the same time, these methods were able to handle different types of data, including images [13], speech [20], text [10], [38], and structured data, such as molecules [162], [165] or genetic sequences [191].”
MethodSelf-citationSamek, W., Wiegand, T., & Müller, K.-R. (2017) · arXiv · doi:10.48550/arxiv.1708.08296 · Cited by 1,080
Self-citation
Metadata
- Authors
- Arras, L., Horn, F., Montavon, G., Müller, K.-R., & Samek, W.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-08-11
- Volume
- 12
- Issue
- 8
- Pages
- e0181142
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2561412020
- Cited by
- 257
- References
- 13
- FWCI
- 19.35
- Type
- journal article
- Language
- en
- Open access
- gold
- Funders
- German Ministry for Education and Research (Berlin Big Data Center BBDC (01IS14013A))
- National Research Foundation of Korea (BK21 program)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: "What is relevant in a text document?": An interpretable machine learning approach. https://app.cochat.ai/papers/10.1371/journal.pone.0181142
- The paper
- Arras, L., Horn, F., Montavon, G., Müller, K.-R., & Samek, W. (2017). "What is relevant in a text document?": An interpretable machine learning approach. PLOS ONE. https://doi.org/10.1371/journal.pone.0181142
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
