The DOI matches its DataCite record; 355 of the 355 references it cites were found in a registry or an index.
Sparks of Large Audio Models: A Survey and Outlook
Open accessAbstract
This survey paper provides a comprehensive overview of the recent advancements and challenges in applying large language models to the field of audio signal processing. Audio processing, with its diverse signal representations and a wide range of sources--from human voices to musical instruments and environmental sounds--poses challenges distinct from those found in traditional Natural Language Processing scenarios. Nevertheless, \textit{Large Audio Models}, epitomized by transformer-based architectures, have shown marked efficacy in this sphere. By leveraging massive amount of data, these models have demonstrated prowess in a variety of audio tasks, spanning from Automatic Speech Recognition and Text-To-Speech to Music Generation, among others. Notably, recently these Foundational Audio Models, like SeamlessM4T, have started showing abilities to act as universal translators, supporting multiple speech tasks for up to 100 languages without any reliance on separate task-specific systems. This paper presents an in-depth analysis of state-of-the-art methodologies regarding \textit{Foundational Large Audio Models}, their performance benchmarks, and their applicability to real-world scenarios. We also highlight current limitations and provide insights into potential future research directions in the realm of \textit{Large Audio Models} with the intent to spark further discussion, thereby fostering innovation in the next generation of audio-processing systems. Furthermore, to cope with the rapid development in this area, we will consistently update the relevant repository with relevant recent articles and their open-source implementations at https://github.com/EmulationAI/awesome-large-audio-models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, there is a gap in the development of large audio models specifically tailored for medical applications [103].”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [103]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 6×
“Transformers utilise self-attention mechanisms to capture temporal correlations from sequential data [72].”
BackgroundMethodResultInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“This shortcoming led to the inception of sequence-to-sequence (seq2seq) architectures, such as Recurrent Neural Networks (RNNs) [70] and Long Short-Term Memory Networks (LSTMs) [71], specifically engineered for handling sequential data.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“These checkpoints spanned from a randomly initialised encoder to established models like BERT [225], GPT-2 [226], and both BART base and large models [227].”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“Recently, language models have shown their power in capturing high-level, long-term patterns across different data types including text [21, 96] and image [149, 150], and speech [151–153].”
BackgroundMethodHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 1×
“This model’s conceptual foundation is anchored in Stable Diffusion[222], the contrastive language audio pre-training model (CLAP) [223], and the Hifi-GAN vocoder [129].”
MethodAuto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“This shortcoming led to the inception of sequence-to-sequence (seq2seq) architectures, such as Recurrent Neural Networks (RNNs) [70] and Long Short-Term Memory Networks (LSTMs) [71], specifically engineered for handling sequential data.”
BackgroundLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1×
“This model harnesses the capabilities of an audio transformer HTSAT [243] as an audio encoder, along with CLIP’s [244] text encoder and GPT2-base as a language model.”
Method
Research works printed but not linked: 13
- B. Gold, N. Morgan, and D. Ellis, Speech and audio signal processing: processing and perception of speech and music. John Wiley & Sons, 2011.
- Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 436-444, 2015.
- A. B. Cyphert, "A human being wrote this law review article: Gpt-3 and the practice of law," UC Davis L. Rev., vol. 55, p. 401, 2021.
- Y. K. Dwivedi, N. Kshetri, L. Hughes, E. L. Slade, A. Jeyaraj, A. K. Kar, A. M. Baabdullah, A. Koohang, V. Raghavan, M. Ahuja et al., ""so what if chatgpt wrote it?" multidisciplinary perspectives on opportunities, challenges and implications of generative conversa- tional ai for research, practice and policy," International Journal of Information Management, vol. 71, p. 102642, 2023.
- Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, "Effi- cient transformers: A survey," ACM Computing Surveys, vol. 55, no. 6, pp. 1-28, 2022.
- K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu et al., "A survey on vision transformer," IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 1, pp. 87-110, 2022.
- J. Ens and P. Pasquier, "Building the metamidi dataset: Linking symbolic and audio musical data." in ISMIR, 2021, pp. 182-188.
- D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, "The mtg-jamendo dataset for automatic music tagging." ICML, 2019.
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
- A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, "Wavenet: A generative model for raw audio," https://arxiv.org/abs/1609.03499, 2016. · arXiv
- S. Maiti, Y. Peng, T. Saeki, and S. Watanabe, "Speechlm- score: Evaluating speech generation using speech lan- guage model," in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1-5.
- S. Forsgren and H. Martiros, "Riffusion-stable dif- fusion for real-time music generation, 2022," URL https://riffusion. com/about, vol. 6.
- A. Banerjee and V. Arora, "wav2tok: Deep sequence tokenizer for audio retrieval," in The Eleventh Interna- tional Conference on Learning Representations, 2022.
Cited by
Sorted by citations
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations
Cheng, H., Zhang, M., & Shi, Q. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3447085 · Cited by 687
“O VER the past several years, Deep Neural Networks ( DNNs ) have achieved conspicuous progress in various domains and applications, such as Computer Vision ( CV ) [1], [2], Natural Language Processing ( NLP ) [3], [4], Audio Signal Processing ( ASP ) [5], [6], and cross-modal applications [7], [8].”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349
“However, there is a gap in the development of large audio models specifically tailored for medical applications [103].”
Affective Computing and the Road to an Emotionally Intelligent Metaverse
Pervez, F., Shoukat, M., Usama, M., Sandhu, M. M., Latif, S., & Qadir, J. (2024) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2024.3389462 · Cited by 52
“These models such as GPT-4 [174] and SeamlessM4t [175] can enable seamless and instantaneous communication between users, transcending language barriers through natural language and speech processing and voice translation and recognition [176].”
BackgroundMethodSelf-citationFrom screens to scenes: A survey of embodied AI in healthcare
Liu, Y., Cao, X., Chen, T., Jiang, Y., You, J., Wu, M., Wang, X., Feng, M., Jin, Y., & Chen, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103033 · Cited by 76
Social Dangers of Generative Artificial Intelligence: Review and Guidelines
Yang, A., & Yang, T. A. (2024) · Annual International Conference on Digital Government Research · doi:10.1145/3657054.3664243 · Cited by 10
“Other examples of GAI include DALL-E, which can create visual artwork based on prompts, and video and sound-editing GAI which can replicate the voices or like-nesses of individuals or groups if there is sufficient data to train a model [12., 13.].”
Background
Metadata
- Authors
- Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., & Schuller, B. W.
- Affiliations
- Queensland University of Technology (AU)
- Mohamed bin Zayed University of Artificial Intelligence (AE)
- Nanyang Technological University (SG)
- University of Augsburg (DE)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2308.12792
- OpenAlex
- W4386185396
- Cited by
- 17
- References
- 315
- Citations by year
- 2025: 6 · 2024: 10 · 2023: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- transformer models, speech recognition, text-to-speech, music generation, multilingual speech translation, audio signal processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
