The DOI matches its Crossref record; 93 of the 93 references it cites were found in a registry or an index.
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
Open accessAbstract
The advancement of audio-language (AL) multimodal learning tasks has been significant in recent years, yet the limited size of existing audio-language datasets poses challenges for researchers due to the costly and time-consuming collection process. To address this data scarcity issue, we introduceWavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400 k audio clips with paired captions. We sourced audio clips and their raw descriptions from web sources and a sound event detection dataset. However, the online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. To overcome this issue, we propose a three-stage processing pipeline for filtering noisy data and generating high-quality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically. We conduct a comprehensive analysis of the characteristics of WavCaps dataset and evaluate it on multiple downstream audio-language multimodal learning tasks. The systems trained on WavCaps outperform previous state-of-the-art (SOTA) models by a significant margin. Our aspiration is for the WavCaps dataset we have proposed to facilitate research in audio-language multimodal learning and demonstrate the potential of utilizing large language models (LLMs) to enhance academic research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There have been similar and successful use cases of LLMs for general audio understanding [18] and music generation [19].”
From LP-MusicCaps: LLM-Based Pseudo Music Captioning (2023) · cites this paper as [18]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Finally, Mel et al.. [18] introduce WavCaps, a 400k audio captioning dataset using ChatGPT [23].”
From LP-MusicCaps: LLM-Based Pseudo Music Captioning (2023) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Research works printed but not linked: 4
- Kim, 2019, AudioCaps: Generating captions for audios in the wild, 119
- Martin, 2021, Diversity and bias in audio captioning datasets, 90
- 10.5555/3524938.3525087 · DOI
- Han, 2021, Automated audio captioning with weakly supervised pre-training and word selection methods, 6
Cited by
Sorted by citations
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
A survey on multimodal large language models
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2024) · National Science Review · doi:10.1093/nsr/nwae403 · Cited by 893
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
Wang, Y., Li, K., Li, X., Yu, J., He, Y., Chen, G., Pei, B., Zheng, R., Wang, Z., Shi, Y., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73013-9_23 · Cited by 427
AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining
Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q., Tian, Q., Wang, Y.-P., Wang, W., Wang, Y., & Plumbley, M. D. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3399607 · Cited by 352
Self-citationEvolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models
Chen, Z., Xu, L., Zheng, H., Chen, L., Tolba, A. K. R., Zhao, L., Yu, K., & Feng, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.052618 · Cited by 126
OneLLM: One Framework to Align All Modalities with Language
Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K., Lin, D., Qiao, Y., Gao, P., & Yue, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02510 · Cited by 152
Uni-MoE: Scaling Unified Multimodal LLMs With Mixture of Experts
Li, Y., Jiang, S., Hu, B., Wang, L., Zhong, W., Luo, W., Ma, L., & Zhang, M. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3532688 · Cited by 136
Natural Language Supervision For General-Purpose Audio Representations
Elizalde, B., Deshmukh, S., & Wang, H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448504 · Cited by 61
Audiosr: Versatile Audio Super-Resolution at Scale
Liu, H., Chen, K., Tian, Q., Wang, W., & Plumbley, M. D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447246 · Cited by 43
Self-citationSeparate Anything You Describe
Liu, X., Kong, Q., Zhao, Y., Liu, H., Yuan, Y., Liu, Y., Xia, R., Wang, Y., Plumbley, M. D., & Wang, W. (2024) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3520017 · Cited by 83
Self-citationSemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound
Liu, H., Xu, X., Yuan, Y., Wu, M., Wang, W., & Plumbley, M. D. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3506286 · Cited by 75
Self-citationSECap: Speech Emotion Captioning with Large Language Model
Xu, Y., Chen, H., Yu, J., Huang, Q., Wu, Z., Zhang, S.-X., Li, G., Luo, Y., & Gu, R. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29902 · Cited by 59
Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation
Xue, J., Deng, Y., Gao, Y., & Li, Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3485485 · Cited by 71
Audio Generation with Multiple Conditional Diffusion Model
Guo, Z., Mao, J., Tao, R. M., Yan, L., Ouchi, K., Liu, H., & Wang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29773 · Cited by 36
Self-citationMMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis
Cheng, H. K., Ishii, M., Hayakawa, A., Shibuya, T., Schwing, A. G., & Mitsufuji, Y. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02691 · Cited by 95
Beyond the Status Quo: A Contemporary Survey of Advances and Challenges in Audio Captioning
Xu, X., Xie, Z., Wu, M., & Yu, K. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3321968 · Cited by 31
Explainable Image Captioning to Identify Ergonomic Problems and Solutions for Construction Workers
Yong, G., Liu, M., & Lee, S. H. (2024) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-5744 · Cited by 25
Training Audio Captioning Models without Audio
Deshmukh, S., Elizalde, B., Emmanouilidou, D., Raj, B., Singh, R., & Wang, H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448115 · Cited by 23
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2023) · arXiv · doi:10.48550/arxiv.2310.13289 · Cited by 67
EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning
Kim, J., Jung, J., Lee, J., & Woo, S. H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446672 · Cited by 21
CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios
Ye, Q., Yu, Z., Shao, R., Xie, X., Torr, P. H. S., & Cao, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72684-2_9 · Cited by 37
HapticGen: Generative Text-to-Vibration Model for Streamlining Haptic Design
Sung, Y., John, K., Yoon, S. H., & Seifi, H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713609 · Cited by 18
Bai, J., Liu, H., Wang, M., Shi, D., Wang, W., Plumbley, M. D., Gan, W., & Chen, J. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3583354 · Cited by 27
Self-citationCacophony: An Improved Contrastive Audio-Text Model
Zhu, G., Darefsky, J., & Duan, Z. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3485170 · Cited by 27
Wu, S.-L., Chang, X., Wichern, G., Jung, J.-W., Germain, F. G., Roux, J. L., & Watanabe, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447215 · Cited by 18
CoNeTTE: An Efficient Audio Captioning System Leveraging Multiple Datasets With Task Embedding
Labbé, É., Pellegrini, T., & Pinquier, J. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3430813 · Cited by 14
Transferable Models for Bioacoustics with Human Language Supervision
Robinson, D., Robinson, A., & Akrapongpisak, L. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447250 · Cited by 28
Extending Large Language Models for Speech and Audio Captioning
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446343 · Cited by 16
Exploring Text-Queried Sound Event Detection with Audio Source Separation
Yin, H., Bai, J., Xiao, Y., Wang, H., Zheng, S., Chen, Y., Das, R. K., Deng, C., & Chen, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889789 · Cited by 13
Acoustic Prompt Tuning: Empowering Large Language Models With Audition Capabilities
Liang, J., Liu, X., Wang, W., Plumbley, M. D., Phan, H., & Benetos, E. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3533375 · Cited by 16
Self-citationFlowSep: Language-Queried Sound Separation with Rectified Flow Matching
Yuan, Y. G., Liu, X., Liu, H., Plumbley, M. D., & Wang, W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890129 · Cited by 12
Self-citationWang, J., Jin, J., Zhang, T., Chai, B. X., Di-Pietro, A., & Georgakopoulos, D. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3567614 · Cited by 13
Computer Audition: From Task-Specific Machine Learning to Foundation Models
Triantafyllopoulos, A., Tsangko, I., Gebhard, A., Mesaros, A., Virtanen, T. I., & Schuller, B. W. (2025) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3593952 · Cited by 16
Towards Weakly Supervised Text-to-Audio Grounding
Xu, X., Ma, Z., Wu, M., & Yu, K. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3443614 · Cited by 22
Audio-Visual LLM for Video Understanding
Shu, F., Zhang, L., Jiang, H., & Xie, C. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00447 · Cited by 73
Sound-VECaps: Improving Audio Generation with Visually Enhanced Captions
Yuan, Y. G., Jia, D., Zhuang, X., Chen, Y., Chen, Z., Wang, Y., Wang, Y., Liu, X., Kang, X., Plumbley, M. D., et al. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889473 · Cited by 9
Self-citationSLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs
Chen, W., Ma, Z., Li, X., Xu, X., Liang, Y., Zheng, Z., Yu, K., & Chen, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889071 · Cited by 9
Zero-Shot Audio Captioning Using Soft and Hard Prompts
Zhang, Y., Xu, X., Du, R., Liu, H., Dong, Y., Tan, Z., Wang, W., & Ma, Z. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3567770 · Cited by 11
Self-citationEnriching Music Descriptions with A Finetuned-LLM and Metadata for Text-to-Music Retrieval
Doh, S., Lee, M., Jeong, D., & Nam, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446380 · Cited by 12
LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport
Rho, K., Lee, H., Iverson, V., & Chung, J. S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888241 · Cited by 7
OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows
Li, S., Kallidromitis, K., Gokul, A., Liao, Z., Kato, Y., Kozuka, K., & Grover, A. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01230 · Cited by 22
Video-Foley: Two-Stage Video-to-Sound Generation via Temporal Event Condition for Foley Sound
Lee, J., Im, J., Kim, D., & Nam, J. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3597477 · Cited by 7
Taming Data and Transformers for Audio Generation
Haji-Ali, M., Menapace, W., Siarohin, A., Balakrishnan, G., & Ordóñez, V. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02632-y · Cited by 31
Lu, K.-H., Chen, Z., Fu, S., Yang, C.-H. H., Huang, S.-F., Yang, C.-K., Yu, C.-M., Chen, C., Chen, W.-C., Huang, C.-Y., et al. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3675792 · Cited by 15
Diffusion-based diverse audio captioning with retrieval-guided Langevin dynamics
Zhu, Y., Men, A., & Xiao, L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102643 · Cited by 8
Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models
Kurzendörfer, D., Mercea, O.-B., Koepke, A. S., & Akata, Z. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00269 · Cited by 8
Audiolog: LLMs-Powered Long Audio Logging with Hybrid Token-Semantic Contrastive Learning
Bai, J., Yin, H., Wang, M., Shi, D., Gan, W., Chen, J., & Rahardja, S. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10688214 · Cited by 9
DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning
Li, X., Chen, W., Ma, Z., Xu, X., Liang, Y., Zheng, Z., Kong, Q., & Chen, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890325 · Cited by 6
Self-citationKim, J., & Hong, J.-H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713911 · Cited by 6
Audio-Free Prompt Tuning for Language-Audio Models
Li, Y., Wang, X., & Liu, H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446472 · Cited by 7
Self-citationA Detailed Audio-Text Data Simulation Pipeline Using Single-Event Sounds
Xu, X., Xu, X., Xie, Z., Zhang, P., Wu, M., & Yu, K. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448441 · Cited by 5
LP-MusicCaps: LLM-Based Pseudo Music Captioning
Doh, S., Choi, K., Lee, J., & Nam, J. (2023) · arXiv · doi:10.48550/arxiv.2307.16372 · Cited by 16
“General Audio Domain AudioCaps [30] 51k 144.9 1 9.0 ±N/A LAION-Audio [22] 630k 4325.4 1-2 N/A WavCaps [18] 403k 7568.9 method is compared to the ground truth in terms of having more true positives (Q1) and fewer false positives (Q2).”
BackgroundMethodInfluentialDetect Any Sound: Open-Vocabulary Sound Event Detection with Multi-Modal Queries
Cai, P., Song, Y., Gu, Q., Jiang, N. P., Song, H., & McLoughlin, I. V. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755574 · Cited by 4
Learning Audio Concepts from Counterfactual Natural Language
Vosoughi, A., Bondi, L., Wu, H.-H., & Xu, C. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446736 · Cited by 5
Audio-Journey: Open Domain Latent Diffusion Based Text-To-Audio Generation
Michaels, J., Li, J. B., Yao, L., Yu, L., Wood-Doughty, Z., & Metze, F. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448220 · Cited by 5
Wang, H., Hu, K., & Gao, L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887668 · Cited by 4
VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis
Jung, J., Ahn, J., Jung, C., Nguyen, T. D., Jang, Y., & Chung, J. S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890322 · Cited by 9
Oncescu, A.-M., Henriques, J. F., Zisserman, A., Albanie, S., & Koepke, A. S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448486 · Cited by 8
Audio-Language Datasets of Scenes and Events: A Survey
Wijngaard, G., Formisano, E., Esposito, M., & Dumontier, M. (2025) · IEEE Access · doi:10.1109/access.2025.3534621 · Cited by 7
Language-based Audio Moment Retrieval
Munakata, H., Nishimura, T., Nakada, S., & Komatsu, T. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890873 · Cited by 4
FlashSR: One-step Versatile Audio Super-resolution via Diffusion Distillation
Im, J., & Nam, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889472 · Cited by 6
TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining
Primus, P., Schmid, F. T., & Widmer, G. (2025) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa66052.2025.11230997 · Cited by 3
UrduHope: Analysis of hope and hopelessness in Urdu texts
Balouchzahi, F., Butt, S., Amjad, M., Sidorov, G., & Gelbukh, A. F. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112746 · Cited by 4
Text-Based Audio Retrieval by Learning From Similarities Between Audio Captions
Xie, H., Khorrami, K., Räsänen, O. J., & Virtanen, T. I. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3511414 · Cited by 3
Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoders
Li, D., Zang, Y., & Kong, Q. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890508 · Cited by 4
Self-citationCLAP4Emo: ChatGPT-Assisted Speech Emotion Retrieval with Natural Language Supervision
Lin, W.-C., Ghaffarzadegan, S., Bondi, L., Kumar, A., Das, S., & Wu, H.-H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447102 · Cited by 4
Language-Queried Target Sound Extraction Without Parallel Training Data
Ma, H., Peng, Z., Xu, L., Li, Y., Shao, M., Kong, Q., & Liu, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887543 · Cited by 3
Self-citationAudioEditor: A Training-Free Diffusion-Based Audio Editing Framework
Jia, Y., Chen, Y., Zhao, J., Zhao, S., Zeng, W., Yong, C., & Qin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887821 · Cited by 13
CoAVT: A Cognition-Inspired Unified Audio-Visual-Text Pre-Training Model for Multimodal Processing
Yue, X., Tian, X., Zhang, M., Wu, Z., & Li, H. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3587467 · Cited by 4
Multimodal Large Language Models: Developments and Directions
Munir, M. B., Khan, L. R., & Thuraisingham, B. M. (2025) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr67560.2025.00048 · Cited by 2
OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder
Bharadwaj, S., Cornell, S., Choi, K., Fukayama, S., Shim, H.-J., Deshmukh, S., & Watanabe, S. (2025) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa66052.2025.11230965 · Cited by 3
Sridhar, A. K., Guo, Y., Visser, E., & Mahfuz, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448154 · Cited by 4
Self-Supervised Audio-Visual Soundscape Stylization
Li, T., Wang, R., Huang, P.-Y., Owens, A., & Anumanchipalli, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72989-8_2 · Cited by 3
Investigating Passive Filter Pruning for Efficient CNN-Transformer Audio Captioning
Xu, X., Singh, A., Wu, M., Wang, W., & Plumbley, M. D. (2024) · Machine learning for signal processing ... · doi:10.1109/mlsp58920.2024.10734745 · Cited by 2
Self-citationAudioBERT: Audio Knowledge Augmented Language Model
Ok, H., Yoo, S., & Lee, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888629 · Cited by 5
Islam, T., & Goldwasser, D. (2025) · ACM Web Science Conference (WebSci) · doi:10.1145/3717867.3717902 · Cited by 2
JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadata
Roy, A., Liu, R., Lu, T., & Herremans, D. (2025) · 2025 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228034 · Cited by 2
SWEET - Weakly Supervised Person Name Extraction for Fighting Human Trafficking
Liu, J., Yu, H., Sujaya, V., Nair, P., Pelrine, K., & Rabbany, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.219 · Cited by 2
MQAD: A Large-Scale Question Answering Dataset for Training Music Large Language Models
Ouyang, Z., Wang, J.-C., Zhang, D., Chen, B., Li, S., & Lin, Q. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890561 · Cited by 3
Enhanced Automated Audio Captioning Method Based on Heterogeneous Feature Fusion
Tan, L., Zhou, Y., Liu, Y., & Chen, W. (2024) · IEEE International Conference on Software System and Information Processing (ICSSIP) · doi:10.1109/icssip63203.2024.11012230 · Cited by 1
Sound event detection with audio-text models and heterogeneous temporal annotations
Harju, M., & Mesaros, A. (2025) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa66052.2025.11230981 · Cited by 1
Feng, J., Li, G., & Xu, Y. (2025) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa66052.2025.11231013 · Cited by 1
Text-Queried Audio Source Separation via Hierarchical Modeling
Yin, X., Peng, X., Jiang, X., Xiong, Z., & Lu, Y. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3633078 · Cited by 2
Retrieval-Augmented Difference Captioning to Explain Unsupervised Anomalous Sound Detection
Ogura, R., Nishida, T., & Kawaguchi, Y. (2025) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc65261.2025.11249188 · Cited by 1
SAGA-SR: Semantically and Acoustically Guided Audio Super-Resolution
Im, J., & Nam, J. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460886 · Cited by 4
SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Training
Mei, X., Lan, G. L., Liu, H., Ni, Z., Nagaraja, V., Liu, Y., Shi, Y., & Chandra, V. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464009 · Cited by 1
Self-citationCASTELLA: Long Audio Dataset with Captions and Temporal Boundaries
Munakata, H., Imamura, T., Nishimura, T., & Komatsu, T. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464074 · Cited by 1
LAMB: LLM-Based Audio Captioning with Modality Gap Bridging Via Cauchy-Schwarz Divergence
Lee, H., Choi, J., Nam, K., & Chung, J. S. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462125 · Cited by 1
Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation
Cheng, S., Zhang, J., Song, Q., Liu, S., Guo, Z., Zhang, X., Zhang, C., Li, X., & Tu, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37574-2_6 · Cited by 9
Detecting Audio-Text Decontextualization through Entailment and Semantic Analysis
Gerhardt, M., Cuccovillo, L., & Aichroth, P. (2026) · ACM International Workshop on Multimedia AI against Disinformation (MAD) · doi:10.1145/3810988.3812666 · Cited by 1
Audio Captioning Using Semantic Alignment Enhancer
Park, Y.-A., & Chang, J. (2023) · IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC) · doi:10.1109/ic-nidc59918.2023.10390585 · Cited by 1
Multi-Modal Continual Pre-Training For Audio Encoders
Kim, G., Wu, H.-H., Bondi, L., & Liu, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446424 · Cited by 1
Multimodality-guided Visual-Caption Semantic Enhancement
Che, N., Liu, J., Yu, F., Cheng, L., Wang, Y., Li, Y., & Liu, C. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104139 · Cited by 1
DanceCaps: Pseudo-Captioning for Dance Videos Using Large Language Models
Kim, S., & Lee, K. (2024) · Applied Sciences · doi:10.3390/app142210116 · Cited by 1
Audio meets text: a loss-enhanced journey with manifold mixup and re-ranking
Suryawanshi, Y., Shah, V., Randar, S., & Joshi, A. D. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02283-4 · Cited by 1
Improving Singing Voice Transcription Generalization with AI Generated Accompaniments
Perez, M., Kirchhoff, H., Grosche, P., & Serra, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-2061-6_9 · Cited by 1
SRC-gAudio: Sampling-Rate-Controlled Audio Generation
Li, C., Xu, M., & Yu, D. (2024) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc63619.2025.10849319 · Cited by 2
Yang, C., Yu, C., Chen, H., Zhu, J., Chen, J., Chen, K., Wang, W., Wang, Y., Jiang, Y., Jiang, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.01802
“A speech recording contains not only words, but also speaker traits, prosody, emotion, turn-taking cues, and temporal structure [1]; real-world audio may further contain environmental sounds, music, over-lapping events, and long-range dependencies that cannot be reduced to transcripts alone [2–4].”
Self-citationGanesh, S., Lavanya, R., Vijayalakshmi, M., Agrawal, P., Adlakha, A., Thakur, Y., Tiwari, S., Shekhar, H., & Shankarappa, R. T. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10724715
Metadata
- Authors
- Mei, X., Meng, C., Liu, H., Kong, Q., Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., & Wang, W.
- Affiliations
- University of Surrey (GB)
- Johns Hopkins University (US)
- Chinese University of Hong Kong (HK)
- ByteDance (China) (CN)
- Peking University (CN)
- ORCID
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2024
- Volume
- 32
- Pages
- 3339-3354
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W4400033239
- Cited by
- 150
- References
- 92
- Citations by year
- 2026: 38 · 2025: 61 · 2024: 47
- FWCI
- 20.44
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/T019751/1)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- audio captioning, ChatGPT, large language models, data filtering, sound event detection, data scarcity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
