CoChat
Sign Up
PreprintCited by 23Multimodal Machine Learning ApplicationsSpeech Recognition and SynthesisMusic and Audio Processing

Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities

Open access
Kong, Zhifeng, Goel, Arushi, Badlani, Rohan, Ping, Wei, Valle, Rafael, Catanzaro, Bryan·2024·arXiv
DOI 10.48550/arxiv.2402.01831arXiv 2402.01831
Open this paper in CoChat — free accountOpen PDF

Abstract

Augmenting large language models (LLMs) to understand audio -- including non-speech sounds and non-verbal speech -- is critically important for diverse real-world applications of LLMs. In this paper, we propose Audio Flamingo, a novel audio language model with 1) strong audio understanding abilities, 2) the ability to quickly adapt to unseen tasks via in-context learning and retrieval, and 3) strong multi-turn dialogue abilities. We introduce a series of training techniques, architecture design, and data strategies to enhance our model with these abilities. Extensive evaluations across various audio understanding tasks confirm the efficacy of our method, setting new state-of-the-art benchmarks. Our demo website is https://audioflamingo.github.io/ and the code is open-sourced at https://github.com/NVIDIA/audio-flamingo.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.01831

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A central goal in audio-language modeling is to build unified systems that can understand heterogeneous audio signals, follow natural language instructions, and produce temporally grounded textual outputs.

    “a central goal is to build unified systems that can understand heterogeneous audio signals, follow natural language instructions, and produce temporally grounded textual outputs [6-8].”

    From MOSS-Audio Technical Report (2026) · cites this paper as [8]

    The Introduction and Multimodal LLMs section describe a model intended to handle varied audio tasks, including non-speech sounds, music, and non-verbal speech, and to conduct multi-turn dialogue. The Audio encoders and representation section says sliding windows better capture temporal information, but it does not establish that the model produces temporally grounded textual outputs. That final part of the attributed claim is unsupported by these passages. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Joint Audio and Speech Understanding

    Gong, Y., Liu, A. H., Luo, H., Karlinsky, L., & Glass, J. (2023) · arXiv · doi:10.48550/arxiv.2309.14405 · cited 5×

    “Audio Flamingo versus previous state-of-the-art (Deshmukh et al., 2023; Chu et al., 2023; Gong et al., 2023b;c; Tang et al., 2023a; Ghosh et al., 2023) on a number of audio understanding benchmarks.”
    BackgroundInfluential

Flagged first, then by citations

Research works printed but not linked: 86

  1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  2. Adigwe, A., Tits, N., Haddad, K. E., Ostadabbas, S., and Dutoit, T. The emotional voices database: Towards controlling the emotion dimension in voice generation systems. arXiv preprint arXiv:1806.09514, 2018. · arXiv
  3. Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., et al. Musi- clm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023. · arXiv
  4. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. Advances in Neural In- formation Processing Systems, 35:23716-23736, 2022.
  5. Barros, P., Churamani, N., Lakomkin, E., Siqueira, H., Sutherland, A., and Wermter, S. The omg-emotion behavior dataset. In 2018 International Joint Con- ference on Neural Networks (IJCNN), pp. 1-7. IEEE, 2018.
  6. Bogdanov, D., Won, M., Tovstogan, P., Porter, A., and Serra, X. The mtg-jamendo dataset for automatic music tagging. ICML, 2019.
  7. Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G. B., Lespiau, J.-B., Damoc, B., Clark, A., et al. Im- proving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206-2240. PMLR, 2022.
  8. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
  9. Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., and Verma, R. Crema-d: Crowd- sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-390, 2014.
  10. Chen, K., Du, X., Zhu, B., Ma, Z., Berg-Kirkpatrick, T., and Dubnov, S. Hts-at: A hierarchical token- semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 646-650. IEEE, 2022.
  11. Chen, Z., Huang, H., Andrusenko, A., Hrinchuk, O., Puvvada, K. C., Li, J., Ghosh, S., Balam, J., and Ginsburg, B. Salm: Speech-augmented language model with in-context learning for speech recognition and translation. arXiv preprint arXiv:2310.09424, 2023. · arXiv
  12. Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., and Zhou, J. Qwen-audio: Ad- vancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023. · arXiv
  13. Defferrard, M., Benzi, K., Vandergheynst, P., and Bres- son, X. Fma: A dataset for music analysis. arXiv preprint arXiv:1612.01840, 2016. · arXiv
  14. Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022. · arXiv
  15. Deshmukh, S., Elizalde, B., and Wang, H. Audio retrieval with wavtext5k and clap training. arXiv preprint arXiv:2209.14275, 2022. · arXiv
  16. Deshmukh, S., Elizalde, B., Singh, R., and Wang, H. Pengi: An audio language model for audio tasks. arXiv preprint arXiv:2305.11834, 2023. · arXiv
  17. Doh, S., Choi, K., Lee, J., and Nam, J. Lp-musiccaps: Llm-based pseudo music captioning. arXiv preprint arXiv:2307.16372, 2023. · arXiv
  18. Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdh- ery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. · arXiv
  19. Drossos, K., Lipping, S., and Virtanen, T. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 736-740. IEEE, 2020.
  20. Duan, H., Wei, J., Wang, C., Liu, H., Fang, Y., Zhang, S., Lin, D., and Chen, K. Botchat: Evaluating llms' capabilities of having multi-turn dialogues. arXiv preprint arXiv:2310.13650, 2023. · arXiv
  21. Elizalde, B., Deshmukh, S., Al Ismail, M., and Wang, H. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023a.
  22. Elizalde, B., Deshmukh, S., and Wang, H. Natural language supervision for general-purpose audio rep- resentations, 2023b. URL https://arxiv.org/abs/ 2309.05767.
  23. Fonseca, E., Favory, X., Pons, J., Font, F., and Serra, X. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829-852, 2021.
  24. Foster, P., Sigtia, S., Krstulovic, S., Barker, J., and Plumbley, M. D. Chime-home: A dataset for sound source recognition in a domestic environment. In 2015 IEEE Workshop on Applications of Signal Pro- cessing to Audio and Acoustics (WASPAA), pp. 1-5. IEEE, 2015.
  25. Gao, H., Ni, J., Qian, K., Zhang, Y., Chang, S., and Hasegawa-Johnson, M. Wavprompt: Towards few- shot spoken language understanding with frozen language models. arXiv preprint arXiv:2203.15863, 2022. · arXiv
  26. Gardner, J., Durand, S., Stoller, D., and Bittner, R. M. Llark: A multimodal foundation model for music. arXiv preprint arXiv:2310.07160, 2023. · arXiv
  27. Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Rit- ter, M. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
  28. Ghosh, S., Kumar, S., Evuru, C. K. R., Duraiswami, R., and Manocha, D. Recap: Retrieval-augmented audio captioning. arXiv preprint arXiv:2309.09836, 2023. · arXiv
  29. Gong, Y., Chung, Y.-A., and Glass, J. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778, 2021. · arXiv
  30. Gong, Y., Khurana, S., Karlinsky, L., and Glass, J. Whisper-at: Noise-robust automatic speech recog- nizers are also strong general audio event taggers. arXiv preprint arXiv:2307.03183, 2023a. · arXiv
  31. Gong, Y., Luo, H., Liu, A. H., Karlinsky, L., and Glass, J. Listen, think, and understand. arXiv preprint arXiv:2305.10790, 2023c. · arXiv
  32. Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
  33. Han, J., Zhang, R., Shao, W., Gao, P., Xu, P., Xiao, H., Zhang, K., Liu, C., Wen, S., Guo, Z., et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint arXiv:2309.03905, 2023. · arXiv
  34. Hershey, S., Ellis, D. P., Fonseca, E., Jansen, A., Liu, C., Moore, R. C., and Plakal, M. The benefit of temporally-strong labels in audio event classifica- tion. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 366-370. IEEE, 2021.
  35. Hsu, M.-H., Chang, K.-W., Li, S.-W., and Lee, H.- y. An exploration of in-context learning for speech language model. arXiv preprint arXiv:2310.12477, 2023. · arXiv
  36. Huang, Q., Jansen, A., Lee, J., Ganti, R., Li, J. Y., and Ellis, D. P. Mulan: A joint embedding of music audio and natural language. arXiv preprint arXiv:2208.12415, 2022. · arXiv
  37. Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z., Wu, Y., Hong, Z., Huang, J., Liu, J., et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. arXiv preprint arXiv:2304.12995, 2023. · arXiv
  38. Iyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. Opt-iml: Scaling language model instruc- tion meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017, 2022. · arXiv
  39. James, J., Tian, L., and Watson, C. An open source emotional speech corpus for human robot interaction applications. Interspeech 2018, 2018.
  40. Jeong, I.-Y. and Park, J. Cochlscene: Acquisition of acoustic scene data using crowdsourcing. In 2022 Asia-Pacific Signal and Information Processing As- sociation Annual Summit and Conference (APSIPA ASC), pp. 17-21. IEEE, 2022.
  41. Johnson, J., Douze, M., and Jégou, H. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547, 2019.
  42. Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906, 2020. · arXiv
  43. Khomenko, V., Shyshkov, O., Radyvonenko, O., and Bokhan, K. Accelerating recurrent neural network training using sequence bucketing and multi-gpu data parallelization. In 2016 IEEE First International Conference on Data Stream Mining & Processing (DSMP), pp. 100-103. IEEE, 2016.
  44. Kim, C. D., Kim, B., Lee, H., and Kim, G. Audio- caps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 119-132, 2019.
  45. Kong, Q., Cao, Y., Iqbal, T., Wang, Y., Wang, W., and Plumbley, M. D. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 28:2880-2894, 2020.
  46. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-augmented gener- ation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459- 9474, 2020.
  47. Li, G., Wei, Y., Tian, Y., Xu, C., Wen, J.-R., and Hu, D. Learning to answer questions in dynamic audio- visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 19108-19118, 2022.
  48. Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Boot- strapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023a. · arXiv
  49. Li, Y., Yuan, R., Zhang, G., Ma, Y., Chen, X., Yin, H., Lin, C., Ragni, A., Benetos, E., Gyenge, N., et al. Mert: Acoustic music understanding model with large-scale self-supervised training. arXiv preprint arXiv:2306.00107, 2023b. · arXiv
  50. Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
  51. Lipping, S., Sudarsanam, P., Drossos, K., and Virtanen, T. Clotho-aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pp. 1140-1144. IEEE, 2022.
  52. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual in- struction tuning. arXiv preprint arXiv:2304.08485, 2023a. · arXiv
  53. Liu, S., Hussain, A. S., Sun, C., and Shan, Y. Mu- sic understanding llama: Advancing text-to-music generation with question answering and captioning. arXiv preprint arXiv:2308.11276, 2023b. · arXiv
  54. Livingstone, S. R. and Russo, F. A. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5):e0196391, 2018.
  55. Loshchilov, I. and Hutter, F. Decoupled weight de- cay regularization. arXiv preprint arXiv:1711.05101, 2017. · arXiv
  56. Lotfian, R. and Busso, C. Building naturalistic emo- tionally balanced speech corpus by retrieving emo- tional speech from existing podcast recordings. IEEE Transactions on Affective Computing, 10(4):471-483, 2017.
  57. Lyu, C., Wu, M., Wang, L., Huang, X., Liu, B., Du, Z., Shi, S., and Tu, Z. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint arXiv:2306.09093, 2023. · arXiv
  58. Mei, X., Meng, C., Liu, H., Kong, Q., Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., and Wang, W. Wavcaps: A chatgpt-assisted weakly-labelled audio caption- ing dataset for audio-language multimodal research. arXiv preprint arXiv:2303.17395, 2023. · arXiv
  59. Moon, S., Madotto, A., Lin, Z., Nagarajan, T., Smith, M., Jain, S., Yeh, C.-F., Murugesan, P., Heidari, P., Liu, Y., et al. Anymal: An efficient and scalable any- modality augmented language model. arXiv preprint arXiv:2309.16058, 2023. · arXiv
  60. Oncescu, A.-M., Koepke, A., Henriques, J. F., Akata, Z., and Albanie, S. Audio retrieval with natural language queries. arXiv preprint arXiv:2105.02192, 2021. · arXiv
  61. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wain- wright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Ad- vances in Neural Information Processing Systems, 35:27730-27744, 2022.
  62. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meet- ing of the Association for Computational Linguistics, pp. 311-318, 2002.
  63. Park, J., Cho, Y., Sim, G., Lee, H., and Choo, J. Enemy spotted: In-game gun sound dataset for gunshot clas- sification and localization. In 2022 IEEE Conference on Games (CoG), pp. 56-63. IEEE, 2022.
  64. Pichora-Fuller, M. K. and Dupuis, K. Toronto emo- tional speech set (TESS), 2020. URL https://doi. org/10.5683/SP2/E8H2MF. · DOI
  65. Poria, S., Hazarika, D., Majumder, N., Naik, G., Cam- bria, E., and Mihalcea, R. Meld: A multimodal multi-party dataset for emotion recognition in con- versations. arXiv preprint arXiv:1810.02508, 2018. · arXiv
  66. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large-scale weak supervision. In In- ternational Conference on Machine Learning, pp. 28492-28518.
  67. PMLR, 2023.
  68. Rashid, M. M., Li, G., and Du, C. Nonspeech7k dataset: Classification and analysis of human non- speech sound. IET Signal Processing, 17(6):e12233, 2023.
  69. Reimers, N. and Gurevych, I. Sentence-bert: Sen- tence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908.10084. · arXiv
  70. Reimers, N. and Gurevych, I. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2020. URL https://arxiv.org/abs/2004.09813. · arXiv
  71. Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., Quitry, F. d. C., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., et al. Au- diopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925, 2023. · arXiv
  72. Salamon, J., Jacoby, C., and Bello, J. P. A dataset and taxonomy for urban sound research. In Proceed- ings of the 22nd ACM international conference on Multimedia, pp. 1041-1044, 2014.
  73. Salewski, L., Fauth, S., Koepke, A., and Akata, Z. Zero- shot audio captioning with audio-language model guidance and audio context keywords. arXiv preprint arXiv:2311.08396, 2023. · arXiv
  74. Sturm, B. L. The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use. arXiv preprint arXiv:1306.1461, 2013. · arXiv
  75. Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289, 2023a. · arXiv
  76. Tang, Z., Yang, Z., Khademi, M., Liu, Y., Zhu, C., and Bansal, M. Codi-2: In-context, interleaved, and interactive any-to-any generation. arXiv preprint arXiv:2311.18775, 2023b. · arXiv
  77. Tsimpoukelli, M., Menick, J. L., Cabi, S., Eslami, S., Vinyals, O., and Hill, F. Multimodal few-shot learn- ing with frozen language models. Advances in Neural Information Processing Systems, 34:200-212, 2021.
  78. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  79. Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
  80. Wang, S., Yang, C.-H. H., Wu, J., and Zhang, C. Can whisper perform speech-based in-context learning. arXiv preprint arXiv:2309.07081, 2023. · arXiv
  81. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Fine- tuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. · arXiv
  82. Won, M., Hung, Y.-N., and Le, D. A founda- tion model for music informatics. arXiv preprint arXiv:2311.03318, 2023. · arXiv
  83. Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., and Dubnov, S. Large-scale contrastive language- audio pretraining with feature fusion and keyword-to- caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
  84. Yang, Z., Ping, W., Liu, Z., Korthikanti, V., Nie, W., Huang, D.-A., Fan, L., Yu, Z., Lan, S., Li, B., et al. Re-vilm: Retrieval-augmented visual language model for zero and few-shot image captioning. In EMNLP, 2023.
  85. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. · arXiv
  86. Zhao, Z., Guo, L., Yue, T., Chen, S., Shao, S., Zhu, X., Yuan, Z., and Liu, J. Chatbridge: Bridging modalities with large language model as a language catalyst. arXiv preprint arXiv:2305.16103, 2023. · arXiv

Cited by

Sorted by citations

Authors
Kong, Z., Goel, A., Badlani, R., Ping, W., Valle, R., & Catanzaro, B.
Affiliations
Nvidia (United States) (US)
ORCID
Catanzaro, Bryan
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.01831
arXiv
2402.01831
OpenAlex
W4391591726
Cited by
23
References
1
Citations by year
2025: 20 · 2024: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
Keywords
audio-language models, audio understanding, few-shot learning, in-context learning, audio retrieval, multi-turn dialogue, non-speech sounds, large language models, multimodal language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review