Joint Speech Recognition and Audio Captioning
Abstract
Speech samples recorded in both indoor and outdoor environments are often contaminated with secondary audio sources. Most end-to-end monaural speech recognition systems either remove these background sounds using speech enhancement or train noise-robust models. For better model interpretability and holistic understanding, we aim to bring together the growing field of automated audio captioning (AAC) and the thoroughly studied automatic speech recognition (ASR). The goal of AAC is to generate natural language descriptions of contents in audio samples. We propose several approaches for end-to-end joint modeling of ASR and AAC tasks and demonstrate their advantages over traditional approaches, which model these tasks independently. A major hurdle in evaluating our proposed approach is the lack of labeled audio datasets with both speech transcriptions and audio captions. Therefore we also create a multi-task dataset by mixing the clean speech Wall Street Journal corpus with multiple levels of background noises chosen from the AudioCaps dataset. We also perform extensive experimental evaluation and show improvements of our proposed methods as compared to existing state-of-the-art ASR and AAC methods.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There are a few recent efforts on joint audio and speech recognition but none of them exhibit advanced joint reasoning ability.
“There are a few recent efforts on joint audio and speech recogntion [7,8,9,10] but none of them exhibit advanced joint reasoning ability.”
From Joint Audio and Speech Understanding (2023) · cites this paper as [9]
Cites
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · see citing works
XSEDE: Accelerating Scientific Discovery
Towns, J. W., Cockerill, T. M., Dahan, M., Foster, I. T., Gaither, K., Grimshaw, A. S., Hazlewood, V., Lathrop, S. A., Lifka, D. A., Peterson, G. D., et al. (2014) · Computing in Science & Engineering · doi:10.1109/mcse.2014.80 · Cited by 3,114 · see citing works
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226 · see citing works
Deep Speech 2: End-to-End Speech Recognition in English and MandarinPreprint
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,191 · see citing works
Attention-Based Models for Speech Recognition
Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1506.07503 · Cited by 2,021 · see citing works
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · see citing works
Towards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 1,866 · see citing works
(2018) · doi:10.21437/interspeech.2018-1456 · Cited by 1,366 · see citing works
Self-citationSequence Transduction with Recurrent Neural NetworksPreprint
Graves, A. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1211.3711 · Cited by 1,320 · see citing works
The design for the wall street journal-based CSR corpus
Paul, D. B., & Baker, J. M. (1992) · DARPA Workshop on Speech and Natural Language (HLT) · doi:10.3115/1075527.1075614 · Cited by 1,142 · see citing works
Hybrid CTC/Attention Architecture for End-to-End Speech Recognition
Watanabe, S., Hori, T., Kim, S., Hershey, J. R., & Hayashi, T. (2017) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2017.2763455 · Cited by 800 · see citing works
Self-citationA Comparative Study on Transformer vs RNN in Speech Applications
Karita, S., Chen, N., Hayashi, T., Hori, T., Inaguma, H., Jiang, Z., Someki, M., Yalta, N., Yamamoto, R., Wang, X., et al. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003750 · Cited by 766 · see citing works
Self-citationAn Overview of Noise-Robust Automatic Speech Recognition
Li, J., Deng, L., Gong, Y., & Haeb‐Umbach, R. (2014) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2014.2304637 · Cited by 568 · see citing works
Improved Image Captioning via Policy Gradient optimization of SPIDEr
Liu, S., Zhu, Z., Ye, N., Guadarrama, S., & Murphy, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.100 · Cited by 430 · see citing works
Hori, T., Watanabe, S., Zhang, Y., & Chan, W. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1296 · Cited by 296 · see citing works
Self-citationAn Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model
Kannan, A., Wu, Y., Nguyen, P., Sainath, T. N., Chen, Z., & Prabhavalkar, R. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462682 · Cited by 253 · see citing works
Nystrom, N. A., Levine, M. J., Roskies, R. Z., & Scott, J. R. (2015) · XSEDE Conference (XSEDE) · doi:10.1145/2792745.2792775 · Cited by 195 · see citing works
O’Shaughnessy, D. D. (2015) · IEEE CHILEAN Conference on Electrical, Electronics Engineering, Information and Communication Technologies (CHILECON) · doi:10.1109/chilecon.2015.7400411 · Cited by 192 · see citing works
Haeb‐Umbach, R., Watanabe, S., Nakatani, T., Bacchiani, M., Hoffmeister, B., Seltzer, M. L., Zen, H., & Souden, M. (2019) · IEEE Signal Processing Magazine · doi:10.1109/msp.2019.2918706 · Cited by 151 · see citing works
Self-citationAutomated audio captioning with recurrent neural networks
Drossos, K., Adavanne, S., & Virtanen, T. I. (2017) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa.2017.8170058 · Cited by 102 · see citing works
Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability
Li, J., Zhao, R., Meng, Z., Liu, Y., Wei, W., Parthasarathy, S., Mazalov, V., Wang, Z., He, L., Zhao, S., et al. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3016 · Cited by 91 · see citing works
End-to-end Monaural Multi-speaker ASR System without Pretraining
Chang, X., Qian, Y., Yu, K., & Watanabe, S. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682822 · Cited by 74 · see citing works
Self-citationSerialized Output Training for End-to-End Overlapped Speech Recognition
Kanda, N., Gaur, Y., Wang, X., Meng, Z., & Yoshioka, T. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-999 · Cited by 67 · see citing works
Audio Captioning TransformerPreprint
Mei, X., Liu, X., Huang, Q., Plumbley, M. D., & Wang, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09817 · Cited by 44 · see citing works
All-in-One Transformer: Unifying Speech Recognition, Audio Tagging, and Event Detection
Moritz, N., Wichern, G., Hori, T., & Roux, J. L. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2757 · Cited by 20 · see citing works
Flagged first, then by citations
1 of the 27 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers
Gong, Y., Khurana, S., Karlinsky, L., & Glass, J. (2023) · INTERSPEECH 2023 · doi:10.21437/interspeech.2023-2193 · Cited by 65 · see citing works
Joint Audio and Speech Understanding
Gong, Y., Liu, A. H., Luo, H., Karlinsky, L., & Glass, J. R. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389742 · Cited by 48 · see citing works
Automated audio captioning: an overview of recent progress and new challenges
Mei, X., Liu, X., Plumbley, M. D., & Wang, W. (2022) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-022-00259-2 · Cited by 46 · see citing works
Beyond the Status Quo: A Contemporary Survey of Advances and Challenges in Audio Captioning
Xu, X., Xie, Z., Wu, M., & Yu, K. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3321968 · Cited by 28 · see citing works
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2023) · arXiv · doi:10.48550/arxiv.2310.13289 · Cited by 67 · see citing works
“With-out using LLMs, Narisetty et al. (2022) studies achieving speech recognition and audio captioning separately using the same model.”
MethodExtending Large Language Models for Speech and Audio Captioning
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446343 · Cited by 13 · see citing works
Deep Learning Based Sign Language Gesture Recognition with Multilingual Translation
Thakkar, R., Kittur, P., & Munshi, A. (2024) · International Conference on Computer, Electronics & Electrical Engineering & their Applications (IC2E3) · doi:10.1109/ic2e362166.2024.10826694 · Cited by 4 · see citing works
Joint Audio and Speech Understanding
Gong, Y., Liu, A. H., Luo, H., Karlinsky, L., & Glass, J. (2023) · arXiv · doi:10.48550/arxiv.2309.14405
“...”
(2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747079
Neurorecognition visualization in multitask end-to-end speech
Mamyrbayev, O. J., Pavlov, S. V., Bekarystankyzy, A., Оралбекова, Д., Жумажанов, Б., Аzarova, L., Mussayeva, D., Koval, T., Gromaszek, K., Issimov, N., et al. (2023) · Optical Fibers and Their Applications · doi:10.1117/12.3022727
- Authors
- Narisetty, C., Tsunoo, E., Chang, X., Kashiwagi, Y., Hentschel, M., & Watanabe, S.
- Affiliations
- Carnegie Mellon University (US)
- Sony (Japan) (JP)
- Venue
- ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2022-05-23
- Pages
- 7892-7896
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W4224932123
- Cited by
- 8
- References
- 27
- Citations by year
- 2024: 2 · 2023: 4 · 2022: 2
- FWCI
- 1.11
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Keywords
- speech recognition, background noise, multi-task learning, Wall Street Journal corpus, AudioCaps, noise robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
