Conceptual Explanations of Neural Network Prediction for Time Series
Abstract
Deep neural networks are black boxes by construction. Explanation and interpretation methods therefore are pivotal for a trustworthy application. Existing methods are mostly based on heatmapping and focus on locally determining the relevant input parts triggering the network prediction. However, these methods struggle to uncover global causes. While this is a rare case in the image or NLP modality, it is of high relevance in the time series domain. This paper presents a novel framework, i.e. Conceptual Explanation, designed to evaluate the effect of abstract (local or global) input features on the model behavior. The method is model-agnostic and allows utilizing expert knowledge. On three time series datasets Conceptual Explanation demonstrates its ability to pinpoint the causes inherent to the data to trigger the correct model prediction.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Various techniques have been proposed to analyze longitudinal data with XAI, but most have not yet been extensively applied to real-world clinical questions.
“Similarly, various techniques have been proposed to analyze longitudinal data with XAI57,87–90, but most have not yet been extensively applied to real-world clinical questions.”
From Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging (2024) · cites this paper as [89]
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Gauriat, C.-M., Pencolé, Y., Ribot, P., et al. (2024) · HAL · doi:10.4230/oasics.dx.2024.27 · Cited by 12,623 · see citing works
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · see citing works
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · see citing works
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · see citing works
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467 · see citing works
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240 · see citing works
Survey and critique of techniques for extracting rules from trained artificial neural networks
Andrews, R., Diederich, J., & Tickle, A. B. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(96)81920-4 · Cited by 1,225 · see citing works
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
iNNvestigate neural networks!Preprint
Alber, M., Lapuschkin, S., Seegerer, P., Hägele, M., Schütt, K. T., Montavon, G., Samek, W., Müller, K., Dähne, S., & Kindermans, P.-J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.04260 · Cited by 249 · see citing works
Layer-Wise Relevance Propagation for Deep Neural Network Architectures
Binder, A., Lapuschkin, S., Montavon, G., Müller, K., & Samek, W. (2016) · Lecture notes in electrical engineering · doi:10.1007/978-981-10-0557-2_87 · Cited by 167 · see citing works
TSViz: Demystification of Deep Learning Models for Time-Series Analysis
Siddiqui, S. A., Mercier, D., Munir, M., Dengel, A., & Ahmed, S. (2019) · IEEE Access · doi:10.1109/access.2019.2912823 · Cited by 95 · see citing works
Self-citationExplaining Classifiers with Causal Concept Effect (CaCE)Preprint
Goyal, Y., Feder, A., Shalit, U., & Kim, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.07165 · Cited by 93 · see citing works
What do Deep Networks Like to See?
Palacio, S., Folz, J., Hees, J., Raue, F., Borth, D., & Dengel, A. R. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00328 · Cited by 32 · see citing works
Self-citationMunir, M., Siddiqui, S. A., Küsters, F., Mercier, D., Dengel, A., & Ahmed, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30493-5_43 · Cited by 19 · see citing works
Self-citation
Flagged first, then by citations
1 of the 18 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions
Theissler, A., Spinnato, F., Schlegel, U., & Guidotti, R. (2022) · IEEE Access · doi:10.1109/access.2022.3207765 · Cited by 211 · see citing works
Pahud de Mortanges, A., Luo, H., Shu, S. Z., Kamath, A., Suter, Y., Shelan, M., Pöllinger, A., & Reyes, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01190-w · Cited by 93 · see citing works
Explainable AI for time series via Virtual Inspection Layers
Vielhaben, J., Lapuschkin, S., Montavon, G., & Samek, W. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110309 · Cited by 38 · see citing works
Explainable online ensemble of deep neural network pruning for time series forecasting
Saadallah, A., Jakobs, M., & Morik, K. J. (2022) · Machine Learning · doi:10.1007/s10994-022-06218-4 · Cited by 20 · see citing works
Löffler, C., Ott, F., Ott, J., Oppelt, M. P., & Feigl, T. (2024) · Springer eBooks · doi:10.1007/978-3-031-64832-8_2 · Cited by 2 · see citing works
Understanding Distributed Representations of Concepts in Deep Neural Networks without Supervision
Chang, W., Kwon, D., & Choi, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.28999 · Cited by 2 · see citing works
Concept Extraction for Time Series with ECLAD-ts
Holzapfel, A., Moreno, A. F. P., & Trimpe, S. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-08317-3_5 · Cited by 1 · see citing works
Atienza, N. (2025) · theses.fr (ABES) · doi:10.70675/81053b26z6d53z4f04zacc2zb3dd0156bba4
- Authors
- Kusters, F., Schichtel, P., Ahmed, S., & Dengel, A.
- Affiliations
- Ingenieurgesellschaft Auto und Verkehr (Germany) (DE)
- German Research Centre for Artificial Intelligence (DE)
- Venue
- 2020 International Joint Conference on Neural Networks (IJCNN)
- Publisher
- IEEE
- Published
- 2020-07
- Pages
- 1-6
- ISSN
- 2161-4407 (ISSN-L)
- OpenAlex
- W3091421217
- Cited by
- 8
- References
- 18
- Citations by year
- 2025: 2 · 2024: 4 · 2022: 2
- FWCI
- 0.19
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- International Joint Conference on Neural Networks (IJCNN) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- conceptual explanation, explainable deep learning, time series forecasting, global explanations, model-agnostic explanation, expert knowledge, neural network interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
