Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers
Open accessAbstract
In this paper, we focus on Whisper, a recent automatic speech recognition model trained with a massive 680k hour labeled speech corpus recorded in diverse conditions. We first show an interesting finding that while Whisper is very robust against real-world background sounds (e.g., music), its audio representation is actually not noise-invariant, but is instead highly correlated to non-speech sounds, indicating that Whisper recognizes speech conditioned on the noise type. With this finding, we build a unified audio tagging and speech recognition model Whisper-AT by freezing the backbone of Whisper, and training a lightweight audio tagging model on top of it. With <1% extra computational cost, Whisper-AT can recognize audio events, in addition to spoken text, in a single forward pass.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Adopting Whisper and TLTR as the audio encoder instead of the AST audio encoder in LTU enables LTU-AS to recognize both speech and audio events.
“This change enables LTU-AS to recognize both speech and audio events.”
From Joint Audio and Speech Understanding (2023) · cites this paper as [3]
Section 4, “Unifying ASR and Audio Tagging Model,” says that adding audio-tagging layers to Whisper enables Whisper-AT to produce spoken text and sound labels in one forward pass. The paper also evaluates a TL-Tr tagging model, but it does not study LTU-AS or show that replacing LTU’s AST encoder enables that specific model to recognize both. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe Whisper encoder encodes rich audio events and speech paralinguistic information, but it encodes information in the representations of different layers.
“although the Whisper encoder encodes rich audio events and speech paralinguistic information [3,15,16], it encodes information in the representations of different layers.”
From Joint Audio and Speech Understanding (2023) · cites this paper as [3]
Section 3 reports that Whisper’s intermediate representations encode non-speech background sounds, and the Figure 3 caption says different sound classes are best represented at different layers. The cited paper does not establish the attributed claim about speech paralinguistic information. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThe Whisper encoder features not only encode linguistic information, but also encode rich general background sound information and paralinguistic and other information such as emotion and language development.
“it was found that the Whisper encoder features not only encode linguistic information, but also encode rich general background sound information [3] and paralinguistic and other information (e.g., emotion [15] and language development [16]).”
From Joint Audio and Speech Understanding (2023) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · see citing works
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983 · see citing works
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369 · see citing works
Robust Speech Recognition via Large-Scale Weak Supervision
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022) · arXiv · doi:10.48550/arxiv.2212.04356 · Cited by 1,953 · see citing works
PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
Kong, Q., Cao, Y., Iqbal, T., Wang, Y., Wang, W., & Plumbley, M. D. (2020) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3030497 · Cited by 1,235 · see citing works
AST: Audio Spectrogram Transformer
Gong, Y., Chung, Y.-A., & Glass, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-698 · Cited by 1,159 · see citing works
Self-citationAn investigation of deep neural networks for noise robust speech recognition
Seltzer, M. L., Yu, D., & Wang, Y. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639100 · Cited by 658 · see citing works
Libri-Light: A Benchmark for ASR with Limited or No Supervision
Kahn, J., Rivière, M., Zheng, W., Kharitonov, E., Xu, Q., Mazaré, P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9052942 · Cited by 475 · see citing works
Attention Bottlenecks for Multimodal Fusion
Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., & Sun, C. (2021) · arXiv · doi:10.48550/arxiv.2107.00135 · Cited by 400 · see citing works
Ssast: Self-supervised audio spectrogram transformer
Gong, Y., Lai, C.-I., Chung, Y.-A., & Glass, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21315 · Cited by 294 · see citing works
Self-citationMasked Autoencoders that Listen
Huang, P.-Y., Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., & Feichtenhofer, C. (2022) · arXiv · doi:10.48550/arxiv.2207.06405 · Cited by 206 · see citing works
Masked Autoencoders that Listen
Huang, P.-Y., Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., & Feichtenhofer, C. (2022) · neural information processing systems · doi:10.52202/068431-2081 · Cited by 206 · see citing works
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training
Hsu, W.-N., Sriram, A., Baevski, A., Likhomanenko, T., Xu, Q., Pratap, V., Kahn, J., Lee, A., Collobert, R., Synnaeve, G., et al. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-236 · Cited by 191 · see citing works
PSLA: Improving Audio Tagging With Pretraining, Sampling, Labeling, and Aggregation
Gong, Y., Chung, Y.-A., & Glass, J. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3120633 · Cited by 164 · see citing works
Self-citationESC: Dataset for Environmental Sound Classification
Piczak, K. J. (2015) · Harvard Dataverse · doi:10.7910/dvn/ydeput · Cited by 158 · see citing works
MAE-AST: Masked Autoencoding Audio Spectrogram Transformer
Baade, A., Peng, P., & Harwath, D. F. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10961 · Cited by 91 · see citing works
Invariant Representations for Noisy Speech RecognitionPreprint
Serdyuk, D., Audhkhasi, K., Brakel, P., Ramabhadran, B., Thomas, S., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.01928 · Cited by 62 · see citing works
HEAR: Holistic Evaluation of Audio Representations
Turian, J., Shier, J., Khan, H. R., Raj, B., Schuller, B. W., Steinmetz, C. J., Malloy, C., Tzanetakis, G., Velarde, G., McNally, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.03022 · Cited by 56 · see citing works
Robust Speech Recognition Using Generative Adversarial Networks
Sriram, A., Jun, H., Gaur, Y., & Satheesh, S. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462456 · Cited by 53 · see citing works
Learning Noise-Invariant Representations for Robust Speech Recognition
Liang, D., Huang, Z., & Lipton, Z. C. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639575 · Cited by 46 · see citing works
Zhu, Q., Zhang, J., Zhang, Z., Wu, M., Fang, X., & Dai, L.-R. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747379 · Cited by 35 · see citing works
All-in-One Transformer: Unifying Speech Recognition, Audio Tagging, and Event Detection
Moritz, N., Wichern, G., Hori, T., & Roux, J. L. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2757 · Cited by 20 · see citing works
Unsupervised learning of time–frequency patches as a noise-robust representation of speech
Segbroeck, M. V., & hamme, H. V. (2009) · Speech Communication · doi:10.1016/j.specom.2009.05.003 · Cited by 20 · see citing works
Joint Speech Recognition and Audio Captioning
Narisetty, C., Tsunoo, E., Chang, X., Kashiwagi, Y., Hentschel, M., & Watanabe, S. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9746601 · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- H. Dinkel, Y. Wang, Z. Yan, J. Zhang, and Y. Wang, "Unikw- at: Unified keyword spotting and audio tagging," arXiv preprint arXiv:2209.11377, 2022. · arXiv
- P.-Y. Huang, H. Xu, J. B. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer, "Masked autoencoders that lis- ten," in Advances in Neural Information Processing Systems, 2022.
Cited by
Sorted by citations
Joint Audio and Speech Understanding
Gong, Y., Liu, A. H., Luo, H., Karlinsky, L., & Glass, J. R. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389742 · Cited by 48 · see citing works
Self-citationHuang, C., Lu, K.-H., Wang, S.-H., Hsiao, C.-Y., Kuan, C.-Y., Wu, H., Arora, S., Chang, K., Shi, J., Peng, Y., et al. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448257 · Cited by 35 · see citing works
Large Language Models are Strong Audio-Visual Speech Recognition Learners
Cappellazzo, U., Kim, M., Chen, H., Ma, P., Petridis, S., Falavigna, D., Brutti, A., & Pantić, M. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889251 · Cited by 31 · see citing works
A Study On Incorporating Whisper For Robust Speech Assessment
Zezario, R. E., Chen, Y., Fu, S., Tsao, Y., Wang, H., & Fuh, C. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10688047 · Cited by 25 · see citing works
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2311.07919 · Cited by 61 · see citing works
Li, J., & Zhang, W.-Q. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448004 · Cited by 21 · see citing works
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2023) · arXiv · doi:10.48550/arxiv.2310.13289 · Cited by 67 · see citing works
“Unlike speech, audio event inputs are often treated as fixed-sized spectrogram images that can be processed using visual-language LLM methods without explicitly modelling temporal correlations (Gong et al., 2023a;b; Zhang et al., 2023b).”
MethodFatehifar, M., Schlittenlacher, J., Almufarrij, I., Wong, D., Cootes, T. F., & Munro, K. J. (2024) · International Journal of Audiology · doi:10.1080/14992027.2024.2422390 · Cited by 17 · see citing works
Who Said what? An Automated Approach to Analyzing Speech in Preschool Classrooms
Sun, A., Londono, J. J., Elbaum, B. E., Estrada, L., Lazo, R. J., Vitale, L., Villasanti, H. G., Fusaroli, R., Perry, L. K., & Messinger, D. S. (2024) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/icdl61372.2024.10644508 · Cited by 12 · see citing works
Speech Foundation Models on Intelligibility Prediction for Hearing-Impaired Listeners
Cuervo, S., & Marxer, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447907 · Cited by 11 · see citing works
Yang, Y., Peng, Y., Huang, H., Chng, E. S., & Zhong, X. (2024) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc63619.2025.10849308 · Cited by 12 · see citing works
Computer Audition: From Task-Specific Machine Learning to Foundation Models
Triantafyllopoulos, A., Tsangko, I., Gebhard, A., Mesaros, A., Virtanen, T. I., & Schuller, B. W. (2025) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3593952 · Cited by 12 · see citing works
Self-Supervised Adaptive AV Fusion Module for Pre-Trained ASR Models
Simic, C., & Bocklet, T. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448047 · Cited by 10 · see citing works
Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions
Meng, L., Hu, S., Kang, J., Li, Z., Wang, Y., Wu, W., Wu, X., Liu, X., & Meng, H. M. L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887642 · Cited by 8 · see citing works
SQ-Whisper: Speaker-Querying Based Whisper Model for Target-Speaker ASR
Guo, P., Chang, X., Lv, H., Watanabe, S., & Xie, L. (2024) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3513835 · Cited by 7 · see citing works
Crossmodal ASR Error Correction With Discrete Speech Units
Li, Y., Chen, P., Bell, P., & Lai, C. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832240 · Cited by 6 · see citing works
Speech recognition in adverse conditions: synthetic transformations and real environmental noise
Katkov, S., Liotta, A., & Vietti, A. (2026) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-026-00458-1 · Cited by 5 · see citing works
Shi, X., He, J., Li, X., & Toda, T. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3654273 · Cited by 6 · see citing works
Deepfake Detection of Singing Voices With Whisper Encodings
Sharma, F., & Gupta, P. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887871 · Cited by 4 · see citing works
EmoStyle: Emotion-Aware Semantic Image Manipulation with Audio Guidance
Shen, Q., Xu, J., Mei, J., Wu, X., & Dong, D. (2024) · Applied Sciences · doi:10.3390/app14083193 · Cited by 3 · see citing works
Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Models
Simic, C., Riedhammer, K., & Bocklet, T. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889566 · Cited by 3 · see citing works
mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition
Rouditchenko, A., Thomas, S., Kuehne, H., Feris, R., & Glass, J. R. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3569210 · Cited by 3 · see citing works
Self-citationSound Tagging in Infant-centric Home Soundscapes
Khan, M. N. H., Li, J., McElwain, N. L., Hasegawa‐Johnson, M., & Islam, B. (2024) · IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies · doi:10.1109/chase60773.2024.00023 · Cited by 3 · see citing works
Improving Rare-Word Recognition of Whisper in Zero-Shot Settings
Jogi, Y., Aggarwal, V., Nair, S. S., Verma, Y., & Kubba, A. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832199 · Cited by 3 · see citing works
JELLY: Joint Emotion Recognition and Context Reasoning with LLMs for Conversational Speech Synthesis
Cha, J.-H., Kim, S.-B., Oh, H., & Lee, S.-W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888227 · Cited by 3 · see citing works
MT2KD: Toward a General-Purpose Encoder for Speech, Speaker, and Audio Events
Yang, X., Li, Q., Zhang, C., & Woodland, P. C. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3594300 · Cited by 2 · see citing works
García, F. P., Cánovas, Ó., Clemente, F. J. G., & Lucas, A. O. (2025) · IEEE Access · doi:10.1109/access.2025.3641484 · Cited by 2 · see citing works
Fine Tuning of Whisper for low resource languages
Mishra, S., Changkakati, S., & Vidhyalakshmi, M. K. (2025) · IEEE Conference on Information and Communication Technology (CICT) · doi:10.1109/cict67193.2025.11398932 · Cited by 2 · see citing works
A Study on Multimodal Fusion and Layer Adapter in Emotion Recognition
Shi, X., Gao, Y., He, J., Mi, J., Li, X., & Toda, T. (2024) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc63619.2025.10848773 · Cited by 2 · see citing works
Ridoy, M. S. I., Akter, S., & Rahman, M. A. (2025) · International Conference on Next-Generation Computing, IoT and Machine Learning (NCIM) · doi:10.1109/ncim65934.2025.11159987 · Cited by 2 · see citing works
Das, A., Franzreb, C., Polzehl, T., & Möller, S. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-84457-7_23 · Cited by 1 · see citing works
From Alignment to Advancement: Bootstrapping Audio-Language Alignment With Synthetic Data
Kuan, C.-Y., & Lee, H.-Y. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3626233 · Cited by 1 · see citing works
Zhou, H., Cao, B., Mo, C., Li, L., & Wang, S. X. (2025) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa66052.2025.11230944 · Cited by 1 · see citing works
Real-Time Question Generation from Educational Video Lectures using Deep Learning
Mane, D., Nimbalkar, R., Gavali, S., Deogade, S., & Khopade, S. R. (2025) · World Conference on Communication & Computing (WCONF) · doi:10.1109/wconf64849.2025.11233372 · Cited by 1 · see citing works
Phukan, O. C., Behera, S. R., Girish, Akhtar, M. M., Buduru, A. B., & Sharma, R. (2025) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc65261.2025.11249005 · Cited by 1 · see citing works
Competitive Audio-Language Models with Data-Efficient Single-Stage Training on Public Data
Kumar, G. K., Saraf, R., Lepauloux, L., Muneer, A., Mokeddem, B., & Hacid, H. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434596 · Cited by 1 · see citing works
Are Soft Prompts Good Zero-Shot Learners for Speech Recognition?
Ng, D., Zhang, C., Zhang, R., Ma, Y., Ritter-Gutierrez, F., Nguyen, T. H., Ni, C., Zhao, S., Chng, E. S., & Ma, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447746 · Cited by 1 · see citing works
Farook, T. H., Ramees, L., & Dudley, J. (2024) · Journal of Prosthetic Dentistry · doi:10.1016/j.prosdent.2024.08.001 · Cited by 1 · see citing works
Personalizing Large Sequence-to-Sequence Speech Foundation Models With Speaker Representations
Wagner, D., Baumann, I., Ranzenberger, T., Riedhammer, K., & Bocklet, T. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832252 · Cited by 1 · see citing works
Loudspeaker Beamforming to Enhance Speech Recognition Performance of Voice Driven Applications
de Groot, D., Karslioglu, B., Scharenborg, O., & Martinez, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889702 · Cited by 1 · see citing works
Informed Speech Self-supervised Representation Learning
Zaiem, S. (2024) · theses.fr (ABES) · doi:10.70675/dc53ad72z177fz4bb4z91fbzd00576ba3f2e · Cited by 1 · see citing works
Joint Audio and Speech Understanding
Gong, Y., Liu, A. H., Luo, H., Karlinsky, L., & Glass, J. (2023) · arXiv · doi:10.48550/arxiv.2309.14405
“Meanwhile, we feed the output of all 32 Whisper encoder intermediate layers to an AudioSet-pretrained Time and Layer-Wise Transformer (TLTR) [3] to encode “soft” audio events and speech paralinguistic information, and then project to a series of continuous audio tokens { A } with a linear layer.”
MethodInfluentialSelf-citationAdapt and Prune Strategy for Multilingual Speech Foundational Model on Low-resourced Languages
Kim, H. S., Cho, C. H., Won, H., & Park, K. H. (2023) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2023.mrl-1.7
Detecting Spoofed Noisy Speeches via Activation-Based Residual Blocks for Embedded Systems
Zhan, J., Peng, S., Jiang, W., Wang, X., & Liu, J. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3437331
Cheng, J., Liu, Y., Yin, J., Wang, L., & Pan, Y. (2024) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc54092.2024.10832034
Adaptive Large Language Models via Attention Shortcuts
Verma, P., & Pilancı, M. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889322
Efficient Tuning of Multilingual ASR for Code-Switching
Nagasubramanian, D. (2025) · International Conference on Artificial Intelligence, Robotics and Control (AIRC) · doi:10.1109/airc64931.2025.11077515
Xiao, S., Zhou, Y., Zhu, J., Wang, J., Zhao, Y. D., Zong, Y., & Kong, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04555-3_2
Harnessing the Potential of Unstructured Data (Audio)- a New Era for Decision-Making
Nagasubramanian, D. (2025) · Springer eBooks · doi:10.1007/978-3-032-02853-2_30
Explainable Disentanglement on Discrete Speech Representations for Noise-Robust ASR
Gopal, S., Anshul, A., Li, H., Yeo, Y. H., Liu, H., & Chng, E. S. (2025) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc65261.2025.11249036
Big Model only for Hard Audios: Sample Dependent Whisper Model Selection for Efficient Inferences
Malard, H., Zaiem, S., & Algayres, R. (2025) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco63237.2025.11226271
Enhancing Emergency Communication with SDR Based PMR Capture and Speech Recognition
Lavric, A., Mutescu, P.-M., Petrariu, A. I., & Coca, E. (2025) · IEEE International Symposium for Design and Technology in Electronic Packaging (SIITME) · doi:10.1109/siitme67657.2025.11293698
Acoustic Foundation Model-Based Vehicle Speed Classification in Real-World Environments
Čavor, I., & Djukanović, S. (2025) · Telecommunications Forum (TELFOR) · doi:10.1109/telfor67910.2025.11314330
Serial-Parallel Dual-Path Architecture for Speaking Style Recognition
Li, G., Shao, Q., Zhao, Z., Wang, S., Fu, Z., & Xie, L. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-5382-2_19
Mattar, B., Torki, M., & Khalafallah, A. (2025) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa66935.2025.11315296
Benchmarking Automatic Speech Recognition for Aphasia: A Clinical Evaluation Framework
Kollapally, N. M., & Akes, C. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356945
Intermediate-Selective Feature Enhancement for Speech Emotion Recognition
Li, Y., Xing, X., Mai, J., Xing, J., & Xu, X. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434758
Ambiguity and Invariance in Machine Listening
Perera, D. (2025) · theses.fr (ABES) · doi:10.70675/164092fdz865cz4fe7za2a9z2f88f6f38360
Investigating Whisper's Prompt to Enhance Indian Accent Speech Recognition
Singh, P., Abharna, K., L, B. R., Gayathri, R., Reshma Sefali, A., Agrawal, G. K., & Asmitha Shree, R. (2026) · National Conference on Communications · doi:10.1109/ncc68160.2026.11478919
Whisper with Benefits: A Unified Approach to Speech and Speaker Attribute Recognition
Duong, R. N. Q. K., Torre, I. G., Nguyen, H., Nancy Bertin, B., Lebarbenchon, R., Camberlein, E., Souviraà-Labastie, N., & Han, K. J. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462763
WavLink: Compact Audio–Text Embeddings with a Global Whisper Token
Kumar, G. K., Lepauloux, L., & Hacid, H. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462214
Graph-Based Hierarchical Multi-Level Framework for Audio Caption Generation
Mohanachandhiran, P., Vivek, S., Rhuthresh, S., & Radhika, S. (2026) · International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET) · doi:10.1109/wispnet69615.2026.11489442
UniAVLM: Unified Large Audio-Visual Language Models for Comprehensive Video Understanding
闫乐成, Lyu, C., Li, W., Samih, Y., & Jiang, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7084-3_7
Arystanbekov, B., Maxutov, A., Nurimanov, A., & Varol, H. A. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10060192
WhisperEngineClassifier: A Transfer Learning Framework for Audio-Based Engine State Classification
Arafat, Y., Miah, N., Ahammad, S. U., Talukder, M. P., Sultana, J., & Raju, M. A. R. (2026) · International Conference on Emerging Smart Technologies and Applications (eSmarTA) · doi:10.1109/esmarta70636.2026.11652168
ASR Personalization in Pathological Voice: Accuracy, Efficiency, and Duration Effects
Giaimis, G., Katkov, S., Yaqub, M. A., & Liotta, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37867-5_15
A Voice Command-Based Digital Microfluidic Platform for Programmable Hands-Free Droplet Manipulation
Park, D., Kim, H., Kim, D., & Chung, S. K. (2026) · Applied Sciences · doi:10.3390/app16189232
Speech Emotion Recognition via Attentive Pooling on Whisper Encoder Representations
Gantulga, M.-E., Munkhnasan, Z., Orgil, J., & Uurtsaikh, M. (2026) · International Symposium on Computer Science and Educational Technology (ISCSET) · doi:10.1109/iscset70652.2026.11722990
- Authors
- Gong, Y., Khurana, S., Karlinsky, L., & Glass, J.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- IBM (United States) (US)
- Venue
- INTERSPEECH 2023
- Publisher
- ISCA
- Published
- 2023-08-20
- Pages
- 2798-2802
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W4385807453
- Cited by
- 65
- References
- 26
- Citations by year
- 2026: 13 · 2025: 30 · 2024: 20 · 2023: 2
- FWCI
- 11.52
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- Whisper, speech recognition, noise robustness, noise invariance, sound representation, background sound, transfer learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
