CoChat
Sign Up
PreprintCited by 12Explainable Artificial Intelligence (XAI)Music and Audio ProcessingSpeech Recognition and Synthesis

The DOI matches its DataCite record.

AudioMNIST: Exploring Explainable Artificial Intelligence for Audio Analysis on a Simple Benchmark

Open access
Becker, Sören, Vielhaben, Johanna, Ackermann, Marcel, Müller, Klaus-Robert, Lapuschkin, Sebastian, Samek, Wojciech·2018·arXiv
DOI 10.48550/arxiv.1807.03418arXiv 1807.03418
Open this paper in CoChat — free accountOpen PDF

Abstract

Explainable Artificial Intelligence (XAI) is targeted at understanding how models perform feature selection and derive their classification decisions. This paper explores post-hoc explanations for deep neural networks in the audio domain. Notably, we present a novel Open Source audio dataset consisting of 30,000 audio samples of English spoken digits which we use for classification tasks on spoken digits and speakers' biological sex. We use the popular XAI technique Layer-wise Relevance Propagation (LRP) to identify relevant features for two neural network architectures that process either waveform or spectrogram representations of the data. Based on the relevance scores obtained from LRP, hypotheses about the neural networks' feature selection are derived and subsequently tested through systematic manipulations of the input data. Further, we take a step beyond visual explanations and introduce audible heatmaps. We demonstrate the superior interpretability of audible explanations over visual ones in a human user study.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1807.03418

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 33

  1. [1][1] R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, N. Elhadad, Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission, in: 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015, pp. 1721-1730. -
  2. [4][4] D. Baehrens, T. Schroeter, S. Harmeling, M. Kawanabe, K. Hansen, K.-R. M¨ uller, How to explain individual classification decisions, Journal of Machine Learning Research 11 (Jun) (2010) 1803-1831. -
  3. [6][6] R. C. Fong, A. Vedaldi, Interpretable explanations of black boxes by meaningful perturbation, in: 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, 2017, pp. 34493457. -
  4. [8][8] W. Samek, G. Montavon, S. Lapuschkin, C. J. Anders, K.-R. M¨ uller, Explaining deep neural networks and beyond: A review of methods and applications, Proceedings of the IEEE 109 (3) (2021) 247-278. -
  5. [9][9] L. Arras, G. Montavon, K.-R. M¨ uller, W. Samek, Explaining recurrent neural network predictions in sentiment analysis, in: EMNLP'17 Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis (WASSA), 2017, pp. 159-168. -
  6. [10][10] I. Sturm, S. Lapuschkin, W. Samek, K.-R. M¨ uller, Interpretable deep neural networks for single-trial eeg classification, Journal of Neuroscience Methods 274 (2016) 141-145. -
  7. [13][13] F. Klauschen, J. Dippel, P. Keyl, P. Jurmeister, M. Bockmayr, A. Mock, O. Buchstab, M. Alber, L. Ruff, G. Montavon, K.-R. M¨ uller, Toward explainable artificial intelligence for precision pathology, Annual Review of Pathology: Mechanisms of Disease 19 (1) (2024) null. -
  8. [16][16] H. Lee, P. Pham, Y. Largman, A. Y. Ng, Unsupervised feature learning for audio classification using convolutional deep belief networks, in: Advances in Neural Information Processing Systems (NIPS), 2009, pp. 1096-1104. -
  9. [17][17] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, et al., Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, IEEE Signal Processing Magazine 29 (6) (2012) 82-97. -
  10. [18][18] W. Dai, C. Dai, S. Qu, J. Li, S. Das, Very deep convolutional neural networks for raw waveforms, in: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017, 2017, pp. 421-425. -
  11. [19][19] L. R. Rabiner, B.-H. Juang, Fundamentals of speech recognition, Vol. 14, PTR Prentice Hall Englewood Cliffs, 1993. -
  12. [20][20] M. Anusuya, S. K. Katti, Speech recognition by machine; a review, International Journal of Computer Science and Information Security 6 (3) (2009) 181-205. -
  13. [21][21] J. J. Godfrey, E. C. Holliman, J. McDaniel, Switchboard: Telephone speech corpus for research and development, in: Acoustics, Speech, and Signal Processing, 1992. ICASSP-92., 1992 IEEE International Conference on, Vol. 1, IEEE, 1992, pp. 517520. -
  14. [23][23] V. Panayotov, G. Chen, D. Povey, S. Khudanpur, Librispeech: an asr corpus based on public domain audio books, in: Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, IEEE, 2015, pp. 5206-5210. -
  15. [25][25] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, K. W. Wilson, CNN architectures for large-scale audio classification, in: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017, 2017, pp. 131-135. -
  16. [26][26] W. Samek, G. Montavon, A. Vedaldi, L. K. Hansen, K.-R. M¨ uller (Eds.), Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Vol. 11700 of Lecture Notes in Computer Science, Springer, Cham, Switzerland, 2019. -
  17. [27][27] N. Strodthoff, P. Wagner, T. Schaeffter, W. Samek, Deep learning for ecg analysis: Benchmarks and insights from ptb-xl, IEEE Journal of Biomedical and Health Informatics 25 (5) (2021) 1519-1528. -
  18. [28][28] D. Slijepcevic, F. Horst, B. Horsak, S. Lapuschkin, A.M. Raberger, A. Kranzl, W. Samek, C. Breiteneder, W. I. Sch¨ ollhorn, M. Zeppelzauer, Explaining machine learning models for clinical gait analysis, ACM Transactions on Computing for Healthcare 3 (2) (2022) 1-27. -
  19. [30][30] M. Kohlbrenner, A. Bauer, S. Nakajima, A. Binder, W. Samek, S. Lapuschkin, Towards best practice in explaining neural network decisions with lrp, in: 2020 International Joint Conference on Neural Networks (IJCNN), IEEE, 2020, pp. 1-7. -
  20. [31][31] S. Lapuschkin, A. Binder, G. Montavon, K.-R. M¨ uller, W. Samek, The layer-wise relevance propagation toolbox for artificial neural networks, Journal of Machine Learning Research 17 (114) (2016) 1-5. -
  21. [33][33] C. J. Anders, D. Neumann, W. Samek, K.-R. M¨ uller, S. Lapuschkin, Software for dataset-wide xai: From local explanations to global insights with zennit, corelay, and virelay, ArXiv abs/2106.13200. - · arXiv
  22. [34][34] J. Jeyakumar, J. Noor, Y.-H. Cheng, L. Garcia, M. B. Srivastava, How can i explain this to you? an empirical study of deep neural network explanation methods, in: Neural Information Processing Systems, 2020. -
  23. [35][35] B. W. Schuller, T. Virtanen, M. Riveiro, G. Rizos, J. Han, A. Mesaros, K. Drossos, Towards sonification in multimodal and user-friendlyexplainable artificial intelligence, in: Proceedings of the 2021 International Conference on Multimodal Interaction, ICMI '21, Association for Computing Machinery, New York, NY, USA, 2021, p. 788-792. -
  24. [37][37] A. B. Melchiorre, V. Haunschmid, M. Schedl, G. Widmer, Lemons: Listenable explanations for music recommender systems, in: D. Hiemstra, M.-F. Moens, J. Mothe, R. Perego, M. Potthast, F. Sebastiani (Eds.), Advances in Information Retrieval, Springer International Publishing, Cham, 2021, pp. 531536. -
  25. [38][38] A. Wullenweber, A. Akman, B. W. Schuller, Coughlime: Sonified explanations for the predictions of covid-19 cough classifiers, in: 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 2022, pp. 1342-1345. -
  26. [39][39] M. T. Ribeiro, S. Singh, C. Guestrin, 'why should i trust you?': Explaining the predictions of any classifier, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, Association for Computing Machinery, New York, NY, USA, 2016, p. 1135-1144. -
  27. [40][40] J. Vielhaben, S. Bluecher, N. Strodthoff, Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees, Transactions on Machine Learning Research. -
  28. [42][42] J. Parekh, S. Parekh, P. Mozharovskyi, F. d ' Alch´ e-Buc, G. Richard, Listen to interpret: Post-hoc interpretability for audio networks with nmf, in: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh (Eds.), Advances in Neural Information Processing Systems, Vol. 35, Curran Associates, Inc., 2022, pp. 35270-35283. -
  29. [43][43] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in Neural Information Processing Systems (NIPS), 2012, pp. 1097-1105. -
  30. [45][45] W. Samek, A. Binder, G. Montavon, S. Lapuschkin, K.-R. M¨ uller, Evaluating the visualization of what a deep neural network has learned, IEEE Transactions on Neural Networks and Learning Systems 28 (11) (2017) 2660-2673. -
  31. [46][46] R. R. Hoffman, S. T. Mueller, G. Klein, J. Litman, Metrics for explainable ai: Challenges and prospects, arXiv preprint arXiv:1812.04608. - · arXiv
  32. [47][47] S. Becker, M. Ackermann, S. Lapuschkin, K. M¨ uller, W. Samek, Interpreting and explaining deep neural networks for classification of audio signals, arXiv preprint arXiv:1807.03418v2. -
  33. [48][48] D. Powers, Evaluation: From precision, recall and f-measure to roc, informedness, markedness & correlation, Journal of Machine Learning Technologies 2 (1) (2011) 37-63.

Cited by

Sorted by citations

Related

Metadata

Authors
Becker, S., Vielhaben, J., Ackermann, M., Müller, K.-R., Lapuschkin, S., & Samek, W.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1807.03418
arXiv
1807.03418
OpenAlex
W4389115020
Cited by
7
References
0
Citations by year
2025: 2 · 2024: 1 · 2023: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
explainable AI, audio classification, spoken digit recognition, layer-wise relevance propagation, post-hoc explanation, feature selection, spectrogram representation, sex classification, interpretability, user study

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review