WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
Open accessAbstract
The advancement of audio-language (AL) multimodal learning tasks has been significant in recent years, yet the limited size of existing audio-language datasets poses challenges for researchers due to the costly and time-consuming collection process. To address this data scarcity issue, we introduceWavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400 k audio clips with paired captions. We sourced audio clips and their raw descriptions from web sources and a sound event detection dataset. However, the online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. To overcome this issue, we propose a three-stage processing pipeline for filtering noisy data and generating high-quality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically. We conduct a comprehensive analysis of the characteristics of WavCaps dataset and evaluate it on multiple downstream audio-language multimodal learning tasks. The systems trained on WavCaps outperform previous state-of-the-art (SOTA) models by a significant margin. Our aspiration is for the WavCaps dataset we have proposed to facilitate research in audio-language multimodal learning and demonstrate the potential of utilizing large language models (LLMs) to enhance academic research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There have been successful use cases of LLMs for general audio understanding and music generation.
“There have been similar and successful use cases of LLMs for general audio understanding [18] and music generation [19].”
From LP-MusicCaps: LLM-Based Pseudo Music Captioning (2023) · cites this paper as [18]
The Introduction describes WavCaps as an audio-captioning dataset built with ChatGPT, and the ChatGPT-based Transformation section says the model successfully filters and rewrites audio descriptions. This supports an LLM-assisted use case in audio-language research, but ChatGPT processes text metadata rather than understanding audio directly. The music-generation clause is attached to a different citation and is not judged here. (LLM, conf=0.84)
Experimental — accuracy not yet measuredWavCaps is a 400k audio captioning dataset created using ChatGPT.
“Finally, Mel et al.. [18] introduce WavCaps, a 400k audio captioning dataset using ChatGPT [23].”
From LP-MusicCaps: LLM-Based Pseudo Music Captioning (2023) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
(2004) · Cited by 9,718 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
(2005) · Cited by 4,152 · see citing works
Kincaid, J. P., Fishburne, Jr., Robert P., R., Richard L., C., & Brad, S. (1975) · doi:10.21236/ada006655 · Cited by 3,497 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · see citing works
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 1× · see citing works
“While AL research is relatively young, vision-language (VL) multimodal learning [20], the counterpart of AL multimodal learning, has been studied for decades and has contributed to the success of many vision-language applications, such as cross-modal search [21], [22], image and video captioning [23], [24], text-to-im…”
BackgroundYoung, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · see citing works
ESC: Dataset for Environmental Sound Classification
Piczak, K. J. (2015) · ACM Multimedia · doi:10.1145/2733373.2806390 · Cited by 2,195 · see citing works
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · see citing works
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · see citing works
A brief introduction to weakly supervised learning
Zhou, Z.-H. (2018) · National Science Review · doi:10.1093/nsr/nwx106 · Cited by 1,899 · see citing works
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · see citing works
Li, J., Li, D., Xiong, C., & Hoi, S. (2022) · arXiv · doi:10.48550/arxiv.2201.12086 · Cited by 1,741 · see citing works
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X. P., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,736 · see citing works
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · see citing works
A Dataset and Taxonomy for Urban Sound Research
Salamon, J., Jacoby, C., & Bello, J. P. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2655045 · Cited by 1,335 · see citing works
PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
Kong, Q., Cao, Y., Iqbal, T., Wang, Y., Wang, W., & Plumbley, M. D. (2020) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3030497 · Cited by 1,235 · see citing works
Self-citationAST: Audio Spectrogram Transformer
Gong, Y., Chung, Y.-A., & Glass, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-698 · Cited by 1,159 · see citing works
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07651 · Cited by 1,078 · see citing works
Deep Modular Co-Attention Networks for Visual Question Answering
Zhou, Y., Yu, J., Cui, Y., Tao, D., & Tian, Q. C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00644 · Cited by 1,013 · see citing works
Unified Vision-Language Pre-Training for Image Captioning and VQA
Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., & Gao, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7005 · Cited by 864 · see citing works
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737 · see citing works
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P. K., Ding, N., & Soricut, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00356 · Cited by 626 · see citing works
FSD50K: An Open Dataset of Human-Labeled Sound Events
Fonseca, E., Favory, X., Pons, J., Font, F., & Serra, X. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3133208 · Cited by 462 · see citing works
Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., & Dubnov, S. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10095969 · Cited by 459 · cited 7× · see citing works
“Unlike traditional large language models such as T5 [54] used in LAION-Audio-630K [53], ChatGPT has been shown to excel at generating human-like responses to natural language prompts, and has garnered widespread attention for its powerful understanding, reasoning, and dialogue abilities.”
BackgroundMethodResultInfluentialVggsound: A Large-Scale Audio-Visual Dataset
Chen, H., Xie, W., Vedaldi, A., & Zisserman, A. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053174 · Cited by 452 · see citing works
CLAP Learning Audio Concepts from Natural Language Supervision
Elizalde, B., Deshmukh, S., Ismail, M. A., & Wang, H. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10095889 · Cited by 449 · see citing works
Improved Image Captioning via Policy Gradient optimization of SPIDEr
Liu, S., Zhu, Z., Ye, N., Guadarrama, S., & Murphy, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.100 · Cited by 430 · see citing works
Acoustic Scene Classification: Classifying environments from the sounds they produce
Barchiesi, D., Giannoulis, D. A., Stowell, D., & Plumbley, M. D. (2015) · IEEE Signal Processing Magazine · doi:10.1109/msp.2014.2326181 · Cited by 423 · see citing works
Self-citationnocaps: novel object captioning at scale
Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., & Anderson, P. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00904 · Cited by 386 · see citing works
Audioclip: Extending Clip to Image, Text and Audio
Guzhov, A., Raue, F., Hees, J., & Dengel, A. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747631 · Cited by 330 · see citing works
Font, F., Roma, G., & Serra, X. (2013) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2502081.2502245 · Cited by 305 · see citing works
HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection
Chen, K., Du, X., Zhu, B., Ma, Z., Berg-Kirkpatrick, T., & Dubnov, S. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9746312 · Cited by 294 · see citing works
Clotho: an Audio Captioning Dataset
Drossos, K., Lipping, S., & Virtanen, T. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9052990 · Cited by 285 · see citing works
Sound Event Detection: A tutorial
Mesaros, A., Heittola, T., Virtanen, T. I., & Plumbley, M. D. (2021) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3090678 · Cited by 274 · see citing works
Self-citationDiffsound: Discrete Diffusion Model for Text-to-Sound Generation
Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., & Yu, D. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3268730 · Cited by 222 · see citing works
Self-citationAudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining
Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q., Tian, Q., Wang, Y.-P., Wang, W., Wang, Y., & Plumbley, M. D. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3399607 · Cited by 213 · see citing works
Self-citationImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text DataPreprint
Qi, D., Su, L., Song, J. L., Cui, E., Bharti, T., & Sacheti, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.07966 · Cited by 201 · see citing works
Wav2CLIP: Learning Robust Audio Representations from Clip
Wu, H.-H., Seetharaman, P., Kumar, K., & Bello, J. P. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747669 · Cited by 201 · see citing works
AudioLDM: Text-to-Audio Generation with Latent Diffusion ModelsPreprint
Liu, H., Chen, Z., Yuan, Y. G., Mei, X., Liu, X., Mandic, D. P., Wang, W., & Plumbley, M. D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.12503 · Cited by 188 · see citing works
Self-citationMusicLM: Generating Music From Text
Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.11325 · Cited by 183 · see citing works
Towards Safe Weakly Supervised Learning
Li, Y.-F., Guo, L.-Z., & Zhou, Z.-H. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2922396 · Cited by 157 · see citing works
Text-to-Audio Generation using Instruction Guided Latent Diffusion Model
Ghosal, D., Majumder, N., Mehrish, A., & Poria, S. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612348 · Cited by 134 · see citing works
BEATs: Audio Pre-Training with Acoustic Tokenizers
Chen, S., Wu, Y., Wang, C., Liu, S., Tompkins, D., Chen, Z., & Wei, F. (2022) · arXiv · doi:10.48550/arxiv.2212.09058 · Cited by 131 · see citing works
AudioGen: Textually Guided Audio GenerationPreprint
Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., Défossez, A., Copet, J., Parikh, D., Taigman, Y., & Adi, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15352 · Cited by 121 · see citing works
Multimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 112 · see citing works
The Benefit of Temporally-Strong Labels in Audio Event Classification
Hershey, S., Ellis, D. P. W., Fonseca, E., Jansen, A., Liu, C., Moore, R. C., & Plakal, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414579 · Cited by 97 · see citing works
Unsupervised Feature Learning Based on Deep Models for Environmental Audio Tagging
Xu, Y., Huang, Q., Wang, W., Foster, P., Sigtia, S., Jackson, P. J. B., & Plumbley, M. D. (2017) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2017.2690563 · Cited by 81 · see citing works
Self-citationAudio Retrieval With Natural Language Queries: A Benchmark Study
Koepke, A. S., Oncescu, A.-M., Henriques, J. F., Akata, Z., & Albanie, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3149712 · Cited by 81 · see citing works
Weakly Labelled AudioSet Tagging With Attention Neural Networks
Kong, Q., Yu, C., Xu, Y., Iqbal, T., Wang, W., & Plumbley, M. D. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2930913 · Cited by 80 · see citing works
Self-citationLearning Audio-Video Modalities from Image Captions
Nagrani, A., Seo, P. H., Seybold, B., Hauth, A., Manén, S., Sun, C., & Schmid, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19781-9_24 · Cited by 68 · see citing works
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2023) · arXiv · doi:10.48550/arxiv.2310.13289 · Cited by 67 · see citing works
Separate What You Describe: Language-Queried Audio Source Separation
Liu, X., Liu, H., Kong, Q., Mei, X., Zhao, J., Huang, Q., Plumbley, M. D., & Wang, W. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10894 · Cited by 66 · see citing works
Self-citationAudio Caption: Listen and Tell
Wu, M., Dinkel, H., & Yu, K. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682377 · Cited by 52 · see citing works
Temporal Reasoning via Audio Question Answering
Fayek, H. M., & Johnson, J. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3010650 · Cited by 51 · see citing works
Automated audio captioning: an overview of recent progress and new challenges
Mei, X., Liu, X., Plumbley, M. D., & Wang, W. (2022) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-022-00259-2 · Cited by 46 · see citing works
Self-citationA Transformer-Based Audio Captioning Model with Keyword Estimation
Koizumi, Y., Masumura, R., Nishida, K., Yasuda, M., & Saito, S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2087 · Cited by 45 · see citing works
Audio Captioning TransformerPreprint
Mei, X., Liu, X., Huang, Q., Plumbley, M. D., & Wang, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09817 · Cited by 44 · see citing works
Self-citationGong, Y., Luo, H., Liu, A. H., Karlinsky, L., & Glass, J. (2023) · arXiv · doi:10.48550/arxiv.2305.10790 · Cited by 43 · see citing works
On Metric Learning for Audio-Text Cross-Modal Retrieval
Mei, X., Liu, X., Sun, J., Plumbley, M. D., & Wang, W. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-11115 · Cited by 42 · see citing works
Self-citationPrefix Tuning for Automated Audio Captioning
Kim, M., Kim, S.-B., & Oh, T.-H. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096877 · Cited by 39 · see citing works
Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering
Lipping, S., Sudarsanam, P., Drossos, K., & Virtanen, T. I. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909680 · Cited by 34 · see citing works
UniAudio: An Audio Foundation Model Toward Universal Audio GenerationPreprint
Yang, D., Tian, J., Tan, X., Huang, R., Liu, S., Chang, X., Shi, J., Zhao, S., Bian, J., Zhao, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.00704 · Cited by 33 · see citing works
Ye, Z., Wang, H., Yang, D., & Zou, Y.-X. (2021) · Workshop on Detection and Classification of Acoustic Scenes and Events · doi:10.48550/arxiv.2110.06100 · Cited by 31 · see citing works
Self-citationGazneli, A., Zimerman, G., Ridnik, T., Sharir, G., & Noy, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.11479 · Cited by 30 · see citing works
Audio Retrieval with WavText5K and CLAP Training
Deshmukh, S., Elizalde, B., & Wang, H. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-1136 · Cited by 29 · see citing works
Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning
Sun, L., Xu, X., Wu, M., & Xie, W. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681472 · Cited by 23 · see citing works
Crowdsourcing a Dataset of Audio Captions
Lipping, S., Drossos, K., & Virtanen, T. I. (2019) · Detection and Classification of Acoustic Scenes and Events Workshop (DCASE) · doi:10.33682/sezz-vd31 · Cited by 23 · see citing works
Automated audio captioning by fine-tuning bart with audioset tags
Gontier, F., Serizel, R., & Cerisara, C. (2021) · HAL · Cited by 21 · see citing works
Improving Text-Audio Retrieval by Text-Aware Attention Pooling and Prior Matrix Revised Loss
Xin, Y., Yang, D., & Zou, Y. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096972 · Cited by 19 · see citing works
Self-citationLeveraging Pre-trained BERT for Audio Captioning
Liu, X., Mei, X., Huang, Q., Sun, J., Zhao, J., Liu, H., Plumbley, M. D., Kılıç, V., & Wang, W. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909761 · Cited by 17 · see citing works
Self-citationVisually-Aware Audio Captioning With Adaptive Audio-Visual Attention
Liu, X., Huang, Q., Mei, X., Liu, H., Kong, Q., Sun, J., Li, S., Ko, T., Zhang, Y., Tang, L. H. Y., et al. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-914 · Cited by 16 · see citing works
Self-citationLP-MusicCaps: LLM-Based Pseudo Music Captioning
Doh, S., Choi, K., Lee, J., & Nam, J. (2023) · arXiv · doi:10.48550/arxiv.2307.16372 · Cited by 16 · see citing works
ChatBridge: Bridging Modalities with Large Language Model as a Language CatalystPreprint
Zhao, Z., Guo, L., Yue, T., Chen, S., Shao, S., Zhu, X., Yuan, Z., & Liu, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.16103 · Cited by 15 · see citing works
Xie, H., Räsänen, O. J., Drossos, K., & Virtanen, T. I. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747336 · Cited by 13 · see citing works
BLAT: Bootstrapping Language-Audio Pre-training based on AudioSet Tag-guided Synthetic Data
Xu, X., Zhang, Z., Zhou, Z., Zhang, P., Xie, Z., Wu, M., & Zhu, K. Q. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3613820 · Cited by 12 · see citing works
Acoustic Prompt Tuning: Empowering Large Language Models With Audition Capabilities
Liang, J., Liu, X., Wang, W., Plumbley, M. D., Phan, H., & Benetos, E. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3533375 · Cited by 11 · see citing works
Self-citationInteractive Audio-text Representation for Automated Audio Captioning with Contrastive LearningPreprint
Chen, C., Hou, N., Hu, Y., Zou, H., Qi, X., & Chng, E. S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.15526 · Cited by 8 · see citing works
FeatureCut: An Adaptive Data Augmentation for Automated Audio Captioning
Ye, Z., Wang, Y., Wang, H., Yang, D., & Zou, Y. (2022) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipaasc55919.2022.9980325 · Cited by 7 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 6
- Y. Gong, Y.-A. Chung, and J. Glass, "AST: Audio Spectrogram Trans- former," in Proc. Interspeech, 2021, pp. 571-575.
- C. D. Kim, B. Kim, H. Lee, and G. Kim, "AudioCaps: Generating captions for audios in the wild," in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019, pp. 119-132.
- I. Martin and A. Mesaros, "Diversity and bias in audio captioning datasets," in Proceedings of the 6th Detection and Classification of Acoustic Scenes and Events 2021 Workshop, Barcelona, Spain, Novem- ber 2021, pp. 90-94.
- T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations," in International Conference on Machine Learning. PMLR, 2020, pp. 1597-1607.
- Q. Han, W. Yuan, D. Liu, X. Li, and Z. Yang, "Automated audio caption- ing with weakly supervised pre-training and word selection methods," in Proceedings of the 6th Detection and Classification of Acoustic Scenes and Events 2021 Workshop, Barcelona, Spain, November 2021, pp. 6- 10.
- E. Kim, J. Kim, Y. Oh, K. Kim, M. Park, J. Sim, J. Lee, and K. Lee, "Improving audio-language learning with MixGen and multi-level test- time augmentation," arXiv preprint arXiv:2210.17143, 2022. · arXiv
Cited by
Sorted by citations
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257 · see citing works
A survey on multimodal large language models
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2024) · National Science Review · doi:10.1093/nsr/nwae403 · Cited by 893 · see citing works
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
Wang, Y., Li, K., Li, X., Yu, J., He, Y., Chen, G., Pei, B., Zheng, R., Wang, Z., Shi, Y., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73013-9_23 · Cited by 239 · see citing works
AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining
Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q., Tian, Q., Wang, Y.-P., Wang, W., Wang, Y., & Plumbley, M. D. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3399607 · Cited by 213 · see citing works
Self-citationEvolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models
Chen, Z., Xu, L., Zheng, H., Chen, L., Tolba, A. K. R., Zhao, L., Yu, K., & Feng, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.052618 · Cited by 112 · see citing works
OneLLM: One Framework to Align All Modalities with Language
Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K., Lin, D., Qiao, Y., Gao, P., & Yue, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02510 · Cited by 103 · see citing works
Uni-MoE: Scaling Unified Multimodal LLMs With Mixture of Experts
Li, Y., Jiang, S., Hu, B., Wang, L., Zhong, W., Luo, W., Ma, L., & Zhang, M. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3532688 · Cited by 91 · see citing works
Natural Language Supervision For General-Purpose Audio Representations
Elizalde, B., Deshmukh, S., & Wang, H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448504 · Cited by 61 · see citing works
Audiosr: Versatile Audio Super-Resolution at Scale
Liu, H., Chen, K., Tian, Q., Wang, W., & Plumbley, M. D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447246 · Cited by 43 · see citing works
Self-citationSeparate Anything You Describe
Liu, X., Kong, Q., Zhao, Y., Liu, H., Yuan, Y., Liu, Y., Xia, R., Wang, Y., Plumbley, M. D., & Wang, W. (2024) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3520017 · Cited by 44 · see citing works
Self-citationSemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound
Liu, H., Xu, X., Yuan, Y., Wu, M., Wang, W., & Plumbley, M. D. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3506286 · Cited by 40 · see citing works
Self-citationSECap: Speech Emotion Captioning with Large Language Model
Xu, Y., Chen, H., Yu, J., Huang, Q., Wu, Z., Zhang, S.-X., Li, G., Luo, Y., & Gu, R. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29902 · Cited by 39 · see citing works
Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation
Xue, J., Deng, Y., Gao, Y., & Li, Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3485485 · Cited by 40 · see citing works
Audio Generation with Multiple Conditional Diffusion Model
Guo, Z., Mao, J., Tao, R. M., Yan, L., Ouchi, K., Liu, H., & Wang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29773 · Cited by 33 · see citing works
Self-citationMMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis
Cheng, H. K., Ishii, M., Hayakawa, A., Shibuya, T., Schwing, A. G., & Mitsufuji, Y. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02691 · Cited by 30 · see citing works
Beyond the Status Quo: A Contemporary Survey of Advances and Challenges in Audio Captioning
Xu, X., Xie, Z., Wu, M., & Yu, K. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3321968 · Cited by 28 · see citing works
Explainable Image Captioning to Identify Ergonomic Problems and Solutions for Construction Workers
Yong, G., Liu, M., & Lee, S. H. (2024) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-5744 · Cited by 23 · see citing works
Training Audio Captioning Models without Audio
Deshmukh, S., Elizalde, B., Emmanouilidou, D., Raj, B., Singh, R., & Wang, H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448115 · Cited by 23 · see citing works
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2023) · arXiv · doi:10.48550/arxiv.2310.13289 · Cited by 67 · see citing works
EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning
Kim, J., Jung, J., Lee, J., & Woo, S. H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446672 · Cited by 21 · see citing works
CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios
Ye, Q., Yu, Z., Shao, R., Xie, X., Torr, P. H. S., & Cao, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72684-2_9 · Cited by 24 · see citing works
HapticGen: Generative Text-to-Vibration Model for Streamlining Haptic Design
Sung, Y., John, K., Yoon, S. H., & Seifi, H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713609 · Cited by 18 · see citing works
Bai, J., Liu, H., Wang, M., Shi, D., Wang, W., Plumbley, M. D., Gan, W., & Chen, J. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3583354 · Cited by 16 · see citing works
Self-citationCacophony: An Improved Contrastive Audio-Text Model
Zhu, G., Darefsky, J., & Duan, Z. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3485170 · Cited by 15 · see citing works
Wu, S.-L., Chang, X., Wichern, G., Jung, J.-W., Germain, F. G., Roux, J. L., & Watanabe, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447215 · Cited by 15 · see citing works
CoNeTTE: An Efficient Audio Captioning System Leveraging Multiple Datasets With Task Embedding
Labbé, É., Pellegrini, T., & Pinquier, J. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3430813 · Cited by 14 · see citing works
Transferable Models for Bioacoustics with Human Language Supervision
Robinson, D., Robinson, A., & Akrapongpisak, L. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447250 · Cited by 14 · see citing works
Extending Large Language Models for Speech and Audio Captioning
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446343 · Cited by 13 · see citing works
Exploring Text-Queried Sound Event Detection with Audio Source Separation
Yin, H., Bai, J., Xiao, Y., Wang, H., Zheng, S., Chen, Y., Das, R. K., Deng, C., & Chen, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889789 · Cited by 13 · see citing works
Acoustic Prompt Tuning: Empowering Large Language Models With Audition Capabilities
Liang, J., Liu, X., Wang, W., Plumbley, M. D., Phan, H., & Benetos, E. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3533375 · Cited by 11 · see citing works
Self-citationFlowSep: Language-Queried Sound Separation with Rectified Flow Matching
Yuan, Y. G., Liu, X., Liu, H., Plumbley, M. D., & Wang, W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890129 · Cited by 11 · see citing works
Self-citationWang, J., Jin, J., Zhang, T., Chai, B. X., Di-Pietro, A., & Georgakopoulos, D. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3567614 · Cited by 11 · see citing works
Computer Audition: From Task-Specific Machine Learning to Foundation Models
Triantafyllopoulos, A., Tsangko, I., Gebhard, A., Mesaros, A., Virtanen, T. I., & Schuller, B. W. (2025) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3593952 · Cited by 12 · see citing works
Towards Weakly Supervised Text-to-Audio Grounding
Xu, X., Ma, Z., Wu, M., & Yu, K. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3443614 · Cited by 10 · see citing works
Audio-Visual LLM for Video Understanding
Shu, F., Zhang, L., Jiang, H., & Xie, C. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00447 · Cited by 26 · see citing works
Sound-VECaps: Improving Audio Generation with Visually Enhanced Captions
Yuan, Y. G., Jia, D., Zhuang, X., Chen, Y., Chen, Z., Wang, Y., Wang, Y., Liu, X., Kang, X., Plumbley, M. D., et al. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889473 · Cited by 8 · see citing works
Self-citationSLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs
Chen, W., Ma, Z., Li, X., Xu, X., Liang, Y., Zheng, Z., Yu, K., & Chen, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889071 · Cited by 9 · see citing works
Zero-Shot Audio Captioning Using Soft and Hard Prompts
Zhang, Y., Xu, X., Du, R., Liu, H., Dong, Y., Tan, Z., Wang, W., & Ma, Z. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3567770 · Cited by 8 · see citing works
Self-citationEnriching Music Descriptions with A Finetuned-LLM and Metadata for Text-to-Music Retrieval
Doh, S., Lee, M., Jeong, D., & Nam, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446380 · Cited by 9 · see citing works
LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport
Rho, K., Lee, H., Iverson, V., & Chung, J. S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888241 · Cited by 7 · see citing works
OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows
Li, S., Kallidromitis, K., Gokul, A., Liao, Z., Kato, Y., Kozuka, K., & Grover, A. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01230 · Cited by 7 · see citing works
Video-Foley: Two-Stage Video-to-Sound Generation via Temporal Event Condition for Foley Sound
Lee, J., Im, J., Kim, D., & Nam, J. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3597477 · Cited by 7 · see citing works
Taming Data and Transformers for Audio Generation
Haji-Ali, M., Menapace, W., Siarohin, A., Balakrishnan, G., & Ordóñez, V. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02632-y · Cited by 15 · see citing works
Lu, K.-H., Chen, Z., Fu, S., Yang, C.-H. H., Huang, S.-F., Yang, C.-K., Yu, C.-M., Chen, C., Chen, W.-C., Huang, C.-Y., et al. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3675792 · Cited by 15 · see citing works
Diffusion-based diverse audio captioning with retrieval-guided Langevin dynamics
Zhu, Y., Men, A., & Xiao, L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102643 · Cited by 7 · see citing works
Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models
Kurzendörfer, D., Mercea, O.-B., Koepke, A. S., & Akata, Z. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00269 · Cited by 7 · see citing works
Audiolog: LLMs-Powered Long Audio Logging with Hybrid Token-Semantic Contrastive Learning
Bai, J., Yin, H., Wang, M., Shi, D., Gan, W., Chen, J., & Rahardja, S. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10688214 · Cited by 6 · see citing works
DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning
Li, X., Chen, W., Ma, Z., Xu, X., Liang, Y., Zheng, Z., Kong, Q., & Chen, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890325 · Cited by 6 · see citing works
Self-citationKim, J., & Hong, J.-H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713911 · Cited by 6 · see citing works
Audio-Free Prompt Tuning for Language-Audio Models
Li, Y., Wang, X., & Liu, H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446472 · Cited by 5 · see citing works
Self-citationA Detailed Audio-Text Data Simulation Pipeline Using Single-Event Sounds
Xu, X., Xu, X., Xie, Z., Zhang, P., Wu, M., & Yu, K. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448441 · Cited by 5 · see citing works
LP-MusicCaps: LLM-Based Pseudo Music Captioning
Doh, S., Choi, K., Lee, J., & Nam, J. (2023) · arXiv · doi:10.48550/arxiv.2307.16372 · Cited by 16 · see citing works
“General Audio Domain AudioCaps [30] 51k 144.9 1 9.0 ±N/A LAION-Audio [22] 630k 4325.4 1-2 N/A WavCaps [18] 403k 7568.9 method is compared to the ground truth in terms of having more true positives (Q1) and fewer false positives (Q2).”
BackgroundMethodInfluentialDetect Any Sound: Open-Vocabulary Sound Event Detection with Multi-Modal Queries
Cai, P., Song, Y., Gu, Q., Jiang, N. P., Song, H., & McLoughlin, I. V. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755574 · Cited by 4 · see citing works
Learning Audio Concepts from Counterfactual Natural Language
Vosoughi, A., Bondi, L., Wu, H.-H., & Xu, C. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446736 · Cited by 4 · see citing works
Audio-Journey: Open Domain Latent Diffusion Based Text-To-Audio Generation
Michaels, J., Li, J. B., Yao, L., Yu, L., Wood-Doughty, Z., & Metze, F. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448220 · Cited by 4 · see citing works
Wang, H., Hu, K., & Gao, L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887668 · Cited by 4 · see citing works
VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis
Jung, J., Ahn, J., Jung, C., Nguyen, T. D., Jang, Y., & Chung, J. S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890322 · Cited by 4 · see citing works
Oncescu, A.-M., Henriques, J. F., Zisserman, A., Albanie, S., & Koepke, A. S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448486 · Cited by 4 · see citing works
Audio-Language Datasets of Scenes and Events: A Survey
Wijngaard, G., Formisano, E., Esposito, M., & Dumontier, M. (2025) · IEEE Access · doi:10.1109/access.2025.3534621 · Cited by 5 · see citing works
Language-based Audio Moment Retrieval
Munakata, H., Nishimura, T., Nakada, S., & Komatsu, T. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890873 · Cited by 4 · see citing works
FlashSR: One-step Versatile Audio Super-resolution via Diffusion Distillation
Im, J., & Nam, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889472 · Cited by 3 · see citing works
TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining
Primus, P., Schmid, F. T., & Widmer, G. (2025) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa66052.2025.11230997 · Cited by 3 · see citing works
UrduHope: Analysis of hope and hopelessness in Urdu texts
Balouchzahi, F., Butt, S., Amjad, M., Sidorov, G., & Gelbukh, A. F. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112746 · Cited by 3 · see citing works
Text-Based Audio Retrieval by Learning From Similarities Between Audio Captions
Xie, H., Khorrami, K., Räsänen, O. J., & Virtanen, T. I. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3511414 · Cited by 3 · see citing works
Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoders
Li, D., Zang, Y., & Kong, Q. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890508 · Cited by 3 · see citing works
Self-citationCLAP4Emo: ChatGPT-Assisted Speech Emotion Retrieval with Natural Language Supervision
Lin, W.-C., Ghaffarzadegan, S., Bondi, L., Kumar, A., Das, S., & Wu, H.-H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447102 · Cited by 3 · see citing works
Language-Queried Target Sound Extraction Without Parallel Training Data
Ma, H., Peng, Z., Xu, L., Li, Y., Shao, M., Kong, Q., & Liu, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887543 · Cited by 3 · see citing works
Self-citationAudioEditor: A Training-Free Diffusion-Based Audio Editing Framework
Jia, Y., Chen, Y., Zhao, J., Zhao, S., Zeng, W., Yong, C., & Qin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887821 · Cited by 3 · see citing works
CoAVT: A Cognition-Inspired Unified Audio-Visual-Text Pre-Training Model for Multimodal Processing
Yue, X., Tian, X., Zhang, M., Wu, Z., & Li, H. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3587467 · Cited by 2 · see citing works
Multimodal Large Language Models: Developments and Directions
Munir, M. B., Khan, L. R., & Thuraisingham, B. M. (2025) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr67560.2025.00048 · Cited by 2 · see citing works
OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder
Bharadwaj, S., Cornell, S., Choi, K., Fukayama, S., Shim, H.-J., Deshmukh, S., & Watanabe, S. (2025) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa66052.2025.11230965 · Cited by 3 · see citing works
Sridhar, A. K., Guo, Y., Visser, E., & Mahfuz, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448154 · Cited by 2 · see citing works
Self-Supervised Audio-Visual Soundscape Stylization
Li, T., Wang, R., Huang, P.-Y., Owens, A., & Anumanchipalli, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72989-8_2 · Cited by 2 · see citing works
Investigating Passive Filter Pruning for Efficient CNN-Transformer Audio Captioning
Xu, X., Singh, A., Wu, M., Wang, W., & Plumbley, M. D. (2024) · Machine learning for signal processing ... · doi:10.1109/mlsp58920.2024.10734745 · Cited by 2 · see citing works
Self-citationAudioBERT: Audio Knowledge Augmented Language Model
Ok, H., Yoo, S., & Lee, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888629 · Cited by 2 · see citing works
Islam, T., & Goldwasser, D. (2025) · ACM Web Science Conference (WebSci) · doi:10.1145/3717867.3717902 · Cited by 2 · see citing works
JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadata
Roy, A., Liu, R., Lu, T., & Herremans, D. (2025) · 2025 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228034 · Cited by 2 · see citing works
SWEET - Weakly Supervised Person Name Extraction for Fighting Human Trafficking
Liu, J., Yu, H., Sujaya, V., Nair, P., Pelrine, K., & Rabbany, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.219 · Cited by 2 · see citing works
MQAD: A Large-Scale Question Answering Dataset for Training Music Large Language Models
Ouyang, Z., Wang, J.-C., Zhang, D., Chen, B., Li, S., & Lin, Q. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890561 · Cited by 2 · see citing works
Enhanced Automated Audio Captioning Method Based on Heterogeneous Feature Fusion
Tan, L., Zhou, Y., Liu, Y., & Chen, W. (2024) · IEEE International Conference on Software System and Information Processing (ICSSIP) · doi:10.1109/icssip63203.2024.11012230 · Cited by 1 · see citing works
Sound event detection with audio-text models and heterogeneous temporal annotations
Harju, M., & Mesaros, A. (2025) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa66052.2025.11230981 · Cited by 1 · see citing works
Feng, J., Li, G., & Xu, Y. (2025) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa66052.2025.11231013 · Cited by 1 · see citing works
Text-Queried Audio Source Separation via Hierarchical Modeling
Yin, X., Peng, X., Jiang, X., Xiong, Z., & Lu, Y. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3633078 · Cited by 2 · see citing works
Retrieval-Augmented Difference Captioning to Explain Unsupervised Anomalous Sound Detection
Ogura, R., Nishida, T., & Kawaguchi, Y. (2025) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc65261.2025.11249188 · Cited by 1 · see citing works
SAGA-SR: Semantically and Acoustically Guided Audio Super-Resolution
Im, J., & Nam, J. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460886 · Cited by 1 · see citing works
SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Training
Mei, X., Lan, G. L., Liu, H., Ni, Z., Nagaraja, V., Liu, Y., Shi, Y., & Chandra, V. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464009 · Cited by 1 · see citing works
Self-citationCASTELLA: Long Audio Dataset with Captions and Temporal Boundaries
Munakata, H., Imamura, T., Nishimura, T., & Komatsu, T. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464074 · Cited by 1 · see citing works
LAMB: LLM-Based Audio Captioning with Modality Gap Bridging Via Cauchy-Schwarz Divergence
Lee, H., Choi, J., Nam, K., & Chung, J. S. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462125 · Cited by 1 · see citing works
Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation
Cheng, S., Zhang, J., Song, Q., Liu, S., Guo, Z., Zhang, X., Zhang, C., Li, X., & Tu, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37574-2_6 · Cited by 1 · see citing works
Detecting Audio-Text Decontextualization through Entailment and Semantic Analysis
Gerhardt, M., Cuccovillo, L., & Aichroth, P. (2026) · ACM International Workshop on Multimedia AI against Disinformation (MAD) · doi:10.1145/3810988.3812666 · Cited by 1 · see citing works
Audio Captioning Using Semantic Alignment Enhancer
Park, Y.-A., & Chang, J. (2023) · IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC) · doi:10.1109/ic-nidc59918.2023.10390585 · Cited by 1 · see citing works
Multi-Modal Continual Pre-Training For Audio Encoders
Kim, G., Wu, H.-H., Bondi, L., & Liu, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446424 · Cited by 1 · see citing works
Multimodality-guided Visual-Caption Semantic Enhancement
Che, N., Liu, J., Yu, F., Cheng, L., Wang, Y., Li, Y., & Liu, C. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104139 · Cited by 1 · see citing works
DanceCaps: Pseudo-Captioning for Dance Videos Using Large Language Models
Kim, S., & Lee, K. (2024) · Applied Sciences · doi:10.3390/app142210116 · Cited by 1 · see citing works
Audio meets text: a loss-enhanced journey with manifold mixup and re-ranking
Suryawanshi, Y., Shah, V., Randar, S., & Joshi, A. D. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02283-4 · Cited by 1 · see citing works
Improving Singing Voice Transcription Generalization with AI Generated Accompaniments
Perez, M., Kirchhoff, H., Grosche, P., & Serra, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-2061-6_9 · Cited by 1 · see citing works
SRC-gAudio: Sampling-Rate-Controlled Audio Generation
Li, C., Xu, M., & Yu, D. (2024) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc63619.2025.10849319 · Cited by 1 · see citing works
Yang, C., Yu, C., Chen, H., Zhu, J., Chen, J., Chen, K., Wang, W., Wang, Y., Jiang, Y., Jiang, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.01802
“A speech recording contains not only words, but also speaker traits, prosody, emotion, turn-taking cues, and temporal structure [1]; real-world audio may further contain environmental sounds, music, over-lapping events, and long-range dependencies that cannot be reduced to transcripts alone [2–4].”
Self-citationGanesh, S., Lavanya, R., Vijayalakshmi, M., Agrawal, P., Adlakha, A., Thakur, Y., Tiwari, S., Shekhar, H., & Shankarappa, R. T. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10724715
- Authors
- Mei, X., Meng, C., Liu, H., Kong, Q., Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., & Wang, W.
- Affiliations
- University of Surrey (GB)
- Johns Hopkins University (US)
- Chinese University of Hong Kong (HK)
- ByteDance (China) (CN)
- Peking University (CN)
- ORCID
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2024
- Volume
- 32
- Pages
- 3339-3354
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W4400033239
- Cited by
- 150
- References
- 92
- Citations by year
- 2026: 38 · 2025: 61 · 2024: 47 · 2023: 4
- FWCI
- 20.44
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/T019751/1)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- audio captioning, ChatGPT, large language models, data filtering, sound event detection, data scarcity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
