Explainable AI for time series via Virtual Inspection Layers
Open accessAbstract
The field of eXplainable Artificial Intelligence (XAI) has witnessed significant advancements in recent years. However, the majority of progress has been concentrated in the domains of computer vision and natural language processing. For time series data, where the input itself is often not interpretable, dedicated XAI research is scarce. In this work, we put forward a virtual inspection layer for transforming the time series to an interpretable representation and allows to propagate relevance attributions to this representation via local XAI methods. In this way, we extend the applicability of XAI methods to domains (e.g. speech) where the input is only interpretable after a transformation. In this work, we focus on the Fourier transformation which, is prominently applied in the preprocessing of time series, with Layer-wise Relevance Propagation (LRP) and refer to our method as DFT-LRP. We demonstrate the usefulness of DFT-LRP in various time series classification settings like audio and electronic health records. We showcase how DFT-LRP reveals differences in the classification strategies of models trained in different domains (e.g., time vs. frequency domain) or helps to discover how models act on spurious correlations in the data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Various techniques have been proposed to analyze longitudinal data with XAI, but most have not yet been extensively applied to real-world clinical questions.
“Similarly, various techniques have been proposed to analyze longitudinal data with XAI57,87–90, but most have not yet been extensively applied to real-world clinical questions.”
From Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging (2024) · cites this paper as [57]
The Related work section (page not provided) describes numerous XAI techniques for time-series data, including applications to gait, ECG and EEG analysis. It does not assess whether most techniques have been extensively applied to real-world clinical questions, so that part of the claim is unsupported by this source. (LLM, conf=0.85)
Experimental — accuracy not yet measuredIt has been suggested that research on XAI for longitudinal data is scarce because the input (single or collective time points) often lacks meaningful interpretation to humans.
“It has been suggested that research on XAI for longitudinal data is scarce because the input (single or collective time points) often lacks meaningful interpretation to humans57.”
From Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging (2024) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
The impact of the MIT-BIH Arrhythmia Database
Moody, G. B., & Mark, R. G. (2001) · IEEE Engineering in Medicine and Biology Magazine · doi:10.1109/51.932724 · Cited by 4,834 · see citing works
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · see citing works
Self-citationDeep learning for time series classification: a review
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00619-1 · Cited by 3,342 · see citing works
Time series classification from scratch with deep neural networks: A strong baseline
Wang, Z., Yan, W., & Oates, T. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966039 · Cited by 2,299 · see citing works
2011 International Joint Conference on Neural Networks
Boubacar, H. A., Lecoeuche, S., & Maouche, S. (2010) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2010.2063350 · Cited by 1,481 · see citing works
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477 · see citing works
Self-citationProceedings of ICNN'95 - International Conference on Neural Networks
(1995) · doi:10.1109/icnn.1995 · Cited by 1,472 · see citing works
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357 · see citing works
Self-citationEvaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164 · see citing works
Self-citationA unified approach to short-time Fourier analysis and synthesis
Allen, J. B., & Rabiner, L. R. (1977) · Proceedings of the IEEE · doi:10.1109/proc.1977.10770 · Cited by 1,049 · see citing works
Deep Learning for Audio Signal Processing
Purwins, H., Li, B., Virtanen, T. I., Schlüter, J., Chang, S.-Y., & Sainath, T. N. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2908700 · Cited by 881 · see citing works
ECG Heartbeat Classification: A Deep Transferable Representation
Kachuee, M., Fazeli, S., & Sarrafzadeh, M. (2018) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi.2018.00092 · Cited by 609 · see citing works
Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL
Strodthoff, N., Wagner, P., Schaeffter, T., & Samek, W. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3022989 · Cited by 540 · see citing works
Self-citationReal-time discrimination of ventricular tachyarrhythmia with Fourier-transform neural network
Minami, K., Nakajima, H., & Toyoshima, T. (1999) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.740880 · Cited by 469 · see citing works
Financial time series forecasting with multi-modality graph neural network
Cheng, D., Yang, F., Xiang, S., & Liu, J. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108218 · Cited by 432 · see citing works
Interpretable deep neural networks for single-trial EEG classification
Sturm, I., Lapuschkin, S., Samek, W., & Müller, K. (2016) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2016.10.008 · Cited by 380 · see citing works
Self-citationTowards A Rigorous Evaluation Of XAI Methods On Time Series
Schlegel, U., Arnout, H., El‐Assady, M., Oelke, D., & Keim, D. A. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00516 · Cited by 218 · see citing works
Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions
Theissler, A., Spinnato, F., Schlegel, U., & Guidotti, R. (2022) · IEEE Access · doi:10.1109/access.2022.3207765 · Cited by 211 · see citing works
Detecting and interpreting myocardial infarction using fully convolutional neural networks
Strodthoff, N., & Strodthoff, C. (2018) · Physiological Measurement · doi:10.1088/1361-6579/aaf34d · Cited by 206 · see citing works
From attribution maps to human-understandable explanations through Concept Relevance Propagation
Achtibat, R., Dreyer, M., Eisenbraun, I., Bosse, S., Wiegand, T., Samek, W., & Lapuschkin, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00711-8 · Cited by 181 · see citing works
Self-citationNeuralHydrology – Interpreting LSTMs in Hydrology
Kratzert, F., Herrnegger, M., Klotz, D., Hochreiter, S., & Klambauer, G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_19 · Cited by 160 · see citing works
Towards Best Practice in Explaining Neural Network Decisions with LRP
Kohlbrenner, M., Bauer, A., Nakajima, S., Binder, A., Samek, W., & Lapuschkin, S. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206975 · Cited by 159 · see citing works
Self-citation2013 IEEE International Conference on Healthcare Informatics
(2013) · doi:10.1109/ichi30775.2013 · Cited by 146 · see citing works
AudioMNIST: Exploring Explainable Artificial Intelligence for audio analysis on a simple benchmark
Becker, S., Vielhaben, J., Ackermann, M., Müller, K., Lapuschkin, S., & Samek, W. (2023) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2023.11.038 · Cited by 124 · see citing works
Self-citationModal Vocal Fundamental Frequency of Young Adults
Fitch, J. L., & Holbrook, A. (1970) · Archives of Otolaryngology - Head and Neck Surgery · doi:10.1001/archotol.1970.04310040067012 · Cited by 123 · see citing works
Finding and removing Clever Hans: Using explanation methods to debug and improve deep models
Anders, C. J., Weber, L., Neumann, D., Samek, W., Müller, K., & Lapuschkin, S. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.015 · Cited by 122 · see citing works
Self-citationExplainable Artificial Intelligence (XAI) on TimeSeries Data: A Survey
Rojat, T., Puget, R., Filliat, D., Del Ser, J., Gelin, R., & Díaz-Rodríguez, N. (2021) · arXiv · doi:10.48550/arxiv.2104.00950 · Cited by 116 · see citing works
TSViz: Demystification of Deep Learning Models for Time-Series Analysis
Siddiqui, S. A., Mercier, D., Munir, M., Dengel, A., & Ahmed, S. (2019) · IEEE Access · doi:10.1109/access.2019.2912823 · Cited by 95 · see citing works
Proceedings of the Conference on Health, Inference, and Learning
(2021) · doi:10.1145/3450439 · Cited by 93 · see citing works
Counterfactual Explanations for Multivariate Time Series
Ates, E., Aksar, B., Leung, V. J., & Coskun, A. K. (2021) · 2021 International Conference on Applied Artificial Intelligence (ICAPAI) · doi:10.1109/icapai49758.2021.9462056 · Cited by 88 · see citing works
Explaining Machine Learning Models for Clinical Gait Analysis
Slijepcevic, D., Horst, F., Lapuschkin, S., Horsak, B., Raberger, A.-M., Kranzl, A., Samek, W., Breiteneder, C., Schöllhorn, W. I., & Zeppelzauer, M. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3474121 · Cited by 82 · see citing works
Self-citationAI-Based human audio processing for COVID-19: A comprehensive overview
Deshpande, G., Batliner, A. M., & Schuller, B. W. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108289 · Cited by 61 · see citing works
Comparing time and frequency domain for audio event recognition using deep learning
Hertel, L., Phan, H., & Mertins, A. (2016) · doi:10.1109/ijcnn.2016.7727635 · Cited by 59 · see citing works
This looks More Like that: Enhancing Self-Explaining Models by Prototypical Relevance Propagation
Gautam, S., Höhne, M., Hansen, S., Jenssen, R., & Kampffmeyer, M. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.109172 · Cited by 46 · see citing works
Explaining Any Time Series Classifier
Guidotti, R., Monreale, A., Spinnato, F., Pedreschi, D., & Giannotti, F. (2020) · 2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi50398.2020.00029 · Cited by 46 · see citing works
Concept-based model explanations for electronic health records
Mincu, D., Loreaux, E., Hou, S., Baur, S., Protsyuk, I., Seneviratne, M. G., Mottram, A., Tomasev, N., Karthikesalingam, A., & Schrouff, J. (2021) · ACM Conference on Health, Inference, and Learning (CHIL) · doi:10.1145/3450439.3451858 · Cited by 25 · see citing works
Post Hoc Explainability for Time Series Classification: Toward a signal processing perspective
Mochaourab, R., Venkitaraman, A., Samsten, I., Papapetrou, P., & Rojas, C. R. (2022) · IEEE Signal Processing Magazine · doi:10.1109/msp.2022.3155955 · Cited by 18 · see citing works
Assessment of dispersion patterns for negative stress detection from electroencephalographic signals
García-Martínez, B., Fernández‐Caballero, A., Alcaraz, R., & Martínez‐Rodrigo, A. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108094 · Cited by 16 · see citing works
Conceptual Explanations of Neural Network Prediction for Time Series
Kusters, F., Schichtel, P., Ahmed, S., & Dengel, A. (2020) · 2020 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207341 · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- M. Sundararajan, A. Taly, Q. Yan, Axiomatic attribution for deep networks, in: Proceedings of the 34th International Conference on Machine Learning - Volume 70, in: ICML, 2017, pp. 3319-3328, http://dx.doi.org/10.1145/1993574. 1993601. · DOI
- J. Vielhaben, S. Bluecher, N. Strodthoff, Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees, Trans. Mach. Learn. Res. (ISSN: 2835-8856) (2023).
- M. Ancona, E. Ceolini, C. Öztireli, M.H. Gross, Towards better understanding of gradient-based attribution methods for deep neural networks, in: International Conference on Learning Representations, 2017.
- C.J. Anders, D. Neumann, W. Samek, K.-R. Müller, S. Lapuschkin, Software for dataset-wide XAI: From local explanations to global insights with Zennit, CoRe- lAy, and ViRelAy, 2021, http://dx.doi.org/10.48550/arXiv.2106.13200, arXiv preprint 2106.13200. abs/2106.13200. · DOI
Cited by
Sorted by citations
Pahud de Mortanges, A., Luo, H., Shu, S. Z., Kamath, A., Suter, Y., Shelan, M., Pöllinger, A., & Reyes, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01190-w · Cited by 93 · see citing works
- Authors
- Vielhaben, J., Lapuschkin, S., Montavon, G., & Samek, W.
- Affiliations
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- Berlin Institute for the Foundations of Learning and Data (DE)
- Corresponding author
- Samek, Wojciech
- Venue
- Pattern Recognition
- Publisher
- Elsevier BV
- Published
- 2024-06
- Volume
- 150
- Pages
- 110309
- ISSN
- 0031-3203 (ISSN-L)
- OpenAlex
- W4391486587
- Cited by
- 38
- References
- 43
- Citations by year
- 2026: 12 · 2025: 18 · 2024: 7 · 2023: 1
- FWCI
- 5.48
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Pattern Recognition (publishedVersion, cc-by)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Funders
- Deutsche Forschungsgemeinschaft (DFG KI-FOR 5363)
- Toyota Motor Engineering and Manufacturing North America (101093003)
- Bundesministerium fur Bildung und Forschung Dienststelle Berlin
- Bundesministerium für Bildung und Forschung (01IS180371I, 01IS21069B, 01IS18025A)
- European Commission
- HORIZON EUROPE European Innovation Council
- Horizon 2020 (101093003, 965221, 10174498)
- Horizon 2020 Framework Programme
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, time series, layer-wise relevance propagation, Fourier transform, time series classification, audio classification, electronic health records, spurious correlations, frequency-domain representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
