Sparks of Large Audio Models: A Survey and Outlook

Latif, Siddique; Shoukat, Moazzam; Shamshad, Fahad; Usama, Muhammad; Ren, Yi; Cuayáhuitl, Heriberto; Wang, Wenwu; Zhang, Xulong; Togneri, Roberto; Cambria, Erik; Schuller, Björn W.

doi:10.48550/arxiv.2308.12792

PreprintCited by 17Speech Recognition and SynthesisMusic and Audio ProcessingSpeech and Audio Processing

Sparks of Large Audio Models: A Survey and Outlook

Open access

Latif, Siddique, Shoukat, Moazzam, Shamshad, Fahad, Usama, Muhammad, Ren, Yi, Cuayáhuitl, Heriberto, Wang, Wenwu, Zhang, Xulong +3 more·2023·arXiv

DOI 10.48550/arxiv.2308.12792arXiv 2308.12792

Open this paper in CoChat — free account Open PDF

Abstract

This survey paper provides a comprehensive overview of the recent advancements and challenges in applying large language models to the field of audio signal processing. Audio processing, with its diverse signal representations and a wide range of sources--from human voices to musical instruments and environmental sounds--poses challenges distinct from those found in traditional Natural Language Processing scenarios. Nevertheless, \textit{Large Audio Models}, epitomized by transformer-based architectures, have shown marked efficacy in this sphere. By leveraging massive amount of data, these models have demonstrated prowess in a variety of audio tasks, spanning from Automatic Speech Recognition and Text-To-Speech to Music Generation, among others. Notably, recently these Foundational Audio Models, like SeamlessM4T, have started showing abilities to act as universal translators, supporting multiple speech tasks for up to 100 languages without any reliance on separate task-specific systems. This paper presents an in-depth analysis of state-of-the-art methodologies regarding \textit{Foundational Large Audio Models}, their performance benchmarks, and their applicability to real-world scenarios. We also highlight current limitations and provide insights into potential future research directions in the realm of \textit{Large Audio Models} with the intent to spark further discussion, thereby fostering innovation in the next generation of audio-processing systems. Furthermore, to cope with the rapid development in this area, we will consistently update the relevant repository with relevant recent articles and their open-source implementations at https://github.com/EmulationAI/awesome-large-audio-models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2308.12792

Cites

354 of 354 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

There is a gap in the development of large audio models specifically tailored for medical applications.
“However, there is a gap in the development of large audio models specifically tailored for medical applications [103].”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [103]
The Introduction says large models have advanced more gradually in audio than in language, images, and video. The Related Surveys and Differences section mentions healthcare surveys but does not identify a gap in developing large audio models specifically for medical applications. Searches for medical and healthcare applications found no passage making that claim; page numbers were unavailable in our copy. (LLM, conf=0.73)
Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 6× · see citing works
“Transformers utilise self-attention mechanisms to capture temporal correlations from sequential data [72].”
BackgroundMethodResultInfluential
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Long Short-Term Memory
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2× · see citing works
“This shortcoming led to the inception of sequence-to-sequence (seq2seq) architectures, such as Recurrent Neural Networks (RNNs) [70] and Long Short-Term Memory Networks (LSTMs) [71], specifically engineered for handling sequential data.”
Background
Deep Learning
Hao, X., Zhang, G., & Ma, S. (2016) · International Journal of Semantic Computing · doi:10.1142/s1793351x16500045 · Cited by 82,372
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 2× · see citing works
“These checkpoints spanned from a randomly initialised encoder to established models like BERT [225], GPT-2 [226], and both BART base and large models [227].”
Method
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3× · see citing works
“Recently, language models have shown their power in capturing high-level, long-term patterns across different data types including text [21, 96] and image [149, 150], and speech [151–153].”
BackgroundMethod
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1× · see citing works
“This shortcoming led to the inception of sequence-to-sequence (seq2seq) architectures, such as Recurrent Neural Networks (RNNs) [70] and Long Short-Term Memory Networks (LSTMs) [71], specifically engineered for handling sequential data.”
Background
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · cited 1× · see citing works
“This model’s conceptual foundation is anchored in Stable Diffusion[222], the contrastive language audio pre-training model (CLAP) [223], and the Hifi-GAN vocoder [129].”
Method
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1× · see citing works
“This model harnesses the capabilities of an audio transformer HTSAT [243] as an audio encoder, along with CLIP’s [244] text encoder and GPT2-base as a language model.”
Method
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 3× · see citing works
“In addition, they used T5 [178], an encoder-decoder transformer [72], to predict the output codes.”
BackgroundMethod
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · cited 1× · see citing works
“In particular, they integrate BART/T5-style pertaining [177, 178] with back translation [179] to substantially decrease the quantity of parallel supervision necessary for training SPEAR-TTS.”
Background
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · see citing works
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712 · see citing works
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 1× · see citing works
“…detecting text generated by LLMs, while Chang et al. [29] have examined the various ways to evaluate LLMs. Additionally, there are a number of surveys dedicated to investigating the specialised applications of Large Models in various fields such as vision [23, 24, 32, 33], education [34–37, 63 ],”
Background
A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
Yu, Y., Si, X., Hu, C., & Zhang, J. (2019) · Neural Computation · doi:10.1162/neco_a_01199 · Cited by 5,662 · cited 1× · see citing works
“To combat this, LSTMs implement a gating mechanism alongside memory cells to regulate the information flow and mitigate issues related to gradients [74, 75].”
Background
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 3× · see citing works
“Ji et al. [88] divided hallucination into two categories: intrinsic hallucinations and extrinsic hallucinations.”
Background
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 1× · see citing works
“These models produce Mel-spectrograms from textual inputs, which are subsequently employed for speech synthesis by vocoders like Griffin-Lim [171], WaveNet [172], and Waveglow [173].”
Method
Untitled work
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682 · see citing works
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 5× · see citing works
“To support the semantic modelling aspect, they trained a 429.5M parameter LLaMA [93] model with 24 layers, 8 heads, and 2048 hidden dimensions, and a scale 3. https://huggingface.co/docs/transformers/model doc/wav2vec2-conformer Fig.”
BackgroundMethodInfluential
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1× · see citing works
“Recent advancements, including single-turn prompting, in-context learning, multi-turn prompting, and the chain of thought, have gained prominence [293].”
Background
Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1× · see citing works
“[35] 2023 ✓ ✗ Education Assess ChatGPT’s performance on the United States Medical Licensing Exam (USMLE), where it impressively achieved scores near the passing threshold without any dedicated specialised training.”
Background
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595 · cited 1× · see citing works
“These benefits have spurred significant interest in transformers across various AI domains [76–82], notably the audio community.”
Background
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.03109 · Cited by 3,159 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · cited 2× · see citing works
“These models are all pre-trained on large-scale audio datasets like AudioSet [112] to acquire the ability to obtain universal audio embedding features.”
Method
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983 · cited 2× · see citing works
“They utilised Hidden-unit BERT (HuBERT) [92] as a discrete unit extractor for the transformation of continuous speech to discrete units, the Meta AI LLaMA [93] model as LLM, and HiFi-GAN as a unit vocoder.”
Method
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · cited 2× · see citing works
“Chang et al. [29] 2023 ✓ ✗ of large models into audio processing presents unique challenges and requires dedicated exploration.”
Background
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · cited 2× · see citing works
“Meanwhile, Large AI Models , particularly Large Language Models (LLMs), have demonstrated outstanding accomplishments in almost every AI domain, reshaping how humans interact with machines [13–15].”
BackgroundSelf-citation
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637 · see citing works
CNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952132 · Cited by 2,604 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 3× · see citing works
“Recently, language models have shown their power in capturing high-level, long-term patterns across different data types including text [21, 96] and image [149, 150], and speech [151–153].”
BackgroundMethod
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · cited 4× · see citing works
“By utilising a conformer-based audio encoder to generate embedding sequences and validating them through simple CTC loss training, this study leverages the LLaMA-7B [93] model with LoRA [94] adaptation.”
MethodInfluential
Self-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505 · cited 1× · see citing works
“The authors constructed their audio tokenizer using the Descript Audio Codec (DAC) [215] and leveraged a multilayer bidirectional transformer [216] for token prediction.”
Method
ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · cited 1× · see citing works
“As the foundational research in this area grows, the insights presented here, along with related works, can serve as a valuable guide to understanding the complexities and addressing the challenges in the development of improved Large Audio Models for audio applications [23, 30, 299, 300].”
Background
Statistical methods for speech recognition
Jelinek, F. (1997) · Journal of the American Statistical Association · doi:10.2307/2670189 · Cited by 2,376 · cited 1× · see citing works
“Traditional audio processing systems were built on meticulously hand-crafted features and extensive linguistic knowledge [6].”
Background
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369 · cited 2× · see citing works
“For the purpose of learning representation, the authors used MuLan [126], Wav2Vec2-Conformer 3 , and VAE.”
Method
Convolutional Neural Networks for Speech Recognition
Abdel‐Hamid, O., Mohamed, A., Jiang, H., Deng, L., Penn, G., & Yu, D. (2014) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2014.2339736 · Cited by 2,268 · cited 1× · see citing works
“Initial applications of deep learning in the field of audio processing primarily utilised versions of Convolutional Neural Networks (CNNs) [69].”
Method
Signal estimation from modified short-time Fourier transform
Griffin, D. W., & Lim, J. S. (1983) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1983.1172092 · Cited by 2,253 · cited 1× · see citing works
“These models produce Mel-spectrograms from textual inputs, which are subsequently employed for speech synthesis by vocoders like Griffin-Lim [171], WaveNet [172], and Waveglow [173].”
Method
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · cited 2× · see citing works
“[38] 2023 ✓ ✗ Health Identify potential applications for medical foundation models and outline specific technical capabilities and training data needed to enable them.”
Background
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957 · see citing works
Score-Based Generative Modeling through Stochastic Differential Equations
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020) · arXiv · doi:10.48550/arxiv.2011.13456 · Cited by 1,953 · see citing works
Robust Speech Recognition via Large-Scale Weak Supervision
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022) · arXiv · doi:10.48550/arxiv.2212.04356 · Cited by 1,953 · cited 2× · see citing works
“This has given rise to diverse architectures such as Wav2Vec [83], Whisper [84], FastPitch [85], MusicBERT [86], and others [26, 87, 88].”
Background
RoFormer: Enhanced Transformer with Rotary Position Embedding
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021) · arXiv · doi:10.48550/arxiv.2104.09864 · Cited by 1,952 · see citing works
Conditional Prompt Learning for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01631 · Cited by 1,918 · cited 1× · see citing works
“Recently, language models have shown their power in capturing high-level, long-term patterns across different data types including text [21, 96] and image [149, 150], and speech [151–153].”
Background
ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.9 · Cited by 1,806 · cited 1× · see citing works
“…detecting text generated by LLMs, while Chang et al. [29] have examined the various ways to evaluate LLMs. Additionally, there are a number of surveys dedicated to investigating the specialised applications of Large Models in various fields such as vision [23, 24, 32, 33], education [34–37, 63 ],”
Background
Transformers in Time Series: A SurveyPreprint
Wen, Q., Zhou, T., Zhang, C.-L., Chen, W.-Q., Ma, Z., Yan, J., & Sun, L. (2022) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2202.07125 · Cited by 1,763 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“Investigations reveal that the act of scaling pre-trained language models (PLMs), either through enhancing the model size or expanding the data size, typically yields superior model performance on subsequent tasks, adhering to what is known as the scaling law [89].”
Background
Tacotron: towards end-to-end speech synthesis
Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., et al. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1452 · Cited by 1,746 · cited 1× · see citing works
“Popular TTs models include Tacotron [168], Deep Voice model [169], and Clarinet [170], and many other [26].”
Method
Improved Denoising Diffusion Probabilistic Models
Nichol, A., & Dhariwal, P. (2021) · arXiv · doi:10.48550/arxiv.2102.09672 · Cited by 1,550 · see citing works
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 1× · see citing works
“In particular, the emergence of models such as ChatGPT and GPT-4 has rekindled discussions about the potential of artificial general intelligence [19, 20].”
Method
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · cited 1× · see citing works
“Similarly, Ghosal et al. [218] introduced TANGO, an innovative approach designed for generating music from text inputs by leveraging the capabilities of FLAN-T5 [219].”
Method
Transformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,412 · see citing works
Self-citation
GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsPreprint
Du, N., Huang, Y.-P., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y.-Z., Krikun, M., Zhou, Y.-Q., Yu, A. W., Firat, O., et al. (2021) · International Conference on Machine Learning · doi:10.48550/arxiv.2112.06905 · Cited by 1,379 · cited 1× · see citing works
“While a few recent attempts have been made to reduce the computational cost and energy usage [93, 267–269], the race to the state of the art in various audio-related tasks is quickly pushing up the pre-training cost.”
Background
Efficient transformers: A survey
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2023) · ACM Computing Surveys · doi:10.1145/3530811 · Cited by 1,267 · see citing works
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · cited 1× · see citing works
“These large models, characterised by their billions of parameters and training on massive datasets, have manifested emergent abilities to tackle a multitude of intricate tasks across various fields [16–18].”
Background
Transformers in Time Series: A Survey
Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., & Sun, L. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/759 · Cited by 1,260 · cited 1× · see citing works
“These benefits have spurred significant interest in transformers across various AI domains [76–82], notably the audio community.”
Background
PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
Kong, Q., Cao, Y., Iqbal, T., Wang, Y., Wang, W., & Plumbley, M. D. (2020) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3030497 · Cited by 1,235 · cited 1× · see citing works
“Large audio models dedicated to obtaining universal audio representations have also been widely applied in the field of sound event detection (SED), such as PANNs [238], AST [109], and BEATs [245].”
Self-citation
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech SynthesisPreprint
Kong, J., Kim, J., & Bae, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.05646 · Cited by 1,188 · cited 3× · see citing works
“In addition, audio-VAE compresses mel spectrogram representations, while the audio synthesis stage employs HiFi-GAN [129] to transform mel spectrograms produced by the VAE decoder into audio.”
BackgroundMethod
AST: Audio Spectrogram Transformer
Gong, Y., Chung, Y.-A., & Glass, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-698 · Cited by 1,159 · cited 2× · see citing works
“The architecture of the LTU model draws from various components, including an audio spectrogram transformer (AST) [109] as the audio encoder, LLaMA [93] as the large language model (LLM) enhanced with Vicuna [110] instructions, Low-rank Adapter [94], and specific generation settings.”
Method
Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research Scenarios
Cascella, M., Montomoli, J., Bellini, V., & Bignami, E. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01925-4 · Cited by 1,120 · cited 1× · see citing works
“This brittleness can cause serious consequences when LLMs are deployed in response-critical applications such as healthcare [290], finance [291], and law [292].”
Background
An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6211 · Cited by 1,089 · cited 1× · see citing works
“During training, the authors maintained the LLaMA unchanged to minimise catastrophic forgetting [113].”
Background
Vision-Language Models for Vision Tasks: A Survey
Zhang, J., Huang, J., Jin, S., & Lu, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3369699 · Cited by 1,051 · cited 2× · see citing works
“…immense potential, these expansive AI models signify a new technological wave that promises a rich ecosystem of real-world applications and have already found extensive applications in various sectors such as vision [23, 24], language, health, education, robotics, and governance, among others.”
Background
“Waveglow: a flow-based generative network for speech synthesis,”
Prenger, R., Valle, R., & Catanzaro, B. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683143 · Cited by 1,004 · cited 1× · see citing works
“These models produce Mel-spectrograms from textual inputs, which are subsequently employed for speech synthesis by vocoders like Griffin-Lim [171], WaveNet [172], and Waveglow [173].”
Method
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · cited 1× · see citing works
“Unlike earlier learning-based models that were tailored for specific tasks, these large models boast versatility in addressing diverse tasks [21, 22].”
Background
VinVL: Revisiting Visual Representations in Vision-Language Models
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., & Gao, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00553 · Cited by 965 · cited 1× · see citing works
“Recently, language models have shown their power in capturing high-level, long-term patterns across different data types including text [21, 96] and image [149, 150], and speech [151–153].”
Background
Alexa, Siri, Cortana, and more: An introduction to voice assistants.
Hoy, M. B. (2018) · Medical Reference Services Quarterly · doi:10.1080/02763869.2018.1404391 · Cited by 965 · cited 1× · see citing works
“These applications range from voice-activated assistants like Siri and Alexa [1, 2] to transcription services [3], and extend to telecommunication systems [4] and hearing aids [5].”
Background
High-Fidelity Audio Compression with Improved RVQGANPreprint
Kumar, R., Seetharaman, P., Luebs, A., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.06546 · Cited by 915 · cited 1× · see citing works
“The authors constructed their audio tokenizer using the Descript Audio Codec (DAC) [215] and leveraged a multilayer bidirectional transformer [216] for token prediction.”
Method
Common Voice: A Massively-Multilingual Speech Corpus
Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M., & Weber, G. (2019) · arXiv · doi:10.48550/arxiv.1912.06670 · Cited by 905 · cited 1× · see citing works
“Title Application Size Multilingual Public access CommonVoice 11 [130] ASR 2508 hours ✓ ✓”
Background
Deep Learning for Audio Signal Processing
Purwins, H., Li, B., Virtanen, T. I., Schlüter, J., Chang, S.-Y., & Sainath, T. N. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2908700 · Cited by 881 · cited 1× · see citing works
“While several comprehensive surveys delve into the applications of deep learning for audio processing [11, 28, 53, 54], including speech [25, 26], music [55–57], and other categories [58, 59], none concentrate on the advent and deployment of LLMs in this field.”
Background
Simple and Controllable Music GenerationPreprint
Copet, J., Kreuk, F., Gat, I., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.05284 · Cited by 848 · cited 1× · see citing works
“MusicGen, a part of the AudioCraft framework [117], is a text-to-music generation language model (LM) that operates on discrete audio representations to generate music from provided text descriptions.”
Background
PaLM 2 Technical Report
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.10403 · Cited by 834 · cited 2× · see citing works
“The model is built upon the foundation of PaLM [96] and PaLM-2 [97], initially devised for text-only pre-training.”
Method
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020) · arXiv · doi:10.48550/arxiv.2006.16236 · Cited by 827 · cited 1× · see citing works
“The Linear Transformer [235] then served as the foundation for synthesising music from these attributes.”
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction ModelPreprint
Gao, P., Han, J., Zhang, R., Lin, Z.-Y., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C.-H., Yue, X., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2304.15010 · Cited by 777 · cited 1× · see citing works
“They utilise LLaMA-7B [155] as their text neural LM due to its flexibility.”
Background
A Comparative Study on Transformer vs RNN in Speech Applications
Karita, S., Chen, N., Hayashi, T., Hori, T., Inaguma, H., Jiang, Z., Someki, M., Yalta, N., Yamamoto, R., Wang, X., et al. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003750 · Cited by 766 · cited 2× · see citing works
“While several comprehensive surveys delve into the applications of deep learning for audio processing [11, 28, 53, 54], including speech [25, 26], music [55–57], and other categories [58, 59], none concentrate on the advent and deployment of LLMs in this field.”
Background
Speech recognition
Berton, A., Kaltenmeier, A., Haiber, U., et al. (1990) · Encyclopedia of Biometrics · doi:10.1002/j.2637-496x.1990.tb05934.x · Cited by 734 · cited 1×
“This has given rise to diverse architectures such as Wav2Vec [83], Whisper [84], FastPitch [85], MusicBERT [86], and others [26, 87, 88].”
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · cited 1× · see citing works
“In another work, Carlini et al. [258] performed a training data extraction attack on a GPT-2 model and extracted personally identifiable information from the training data.”
Method
TaSNet: Time-Domain Audio Separation Network for Real-Time, Single-Channel Speech Separation
Luo, Y., & Mesgarani, N. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462116 · Cited by 717 · cited 1× · see citing works
“These applications range from voice-activated assistants like Siri and Alexa [1, 2] to transcription services [3], and extend to telecommunication systems [4] and hearing aids [5].”
Background
Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education
Qadir, J. (2023) · 2023 IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon54358.2023.10125121 · Cited by 707 · cited 1× · see citing works
“[36] 2023 ✓ ✗ Education Review regarding promise and pitfalls of ChatGPT in engineering education.”
Background
Elucidating the Design Space of Diffusion-Based Generative Models
Karras, T., Aittala, M., Aila, T., & Laine, S. (2022) · arXiv · doi:10.48550/arxiv.2206.00364 · Cited by 702 · see citing works
SoundStream: An End-to-End Neural Audio Codec
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., & Tagliasacchi, M. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3129994 · Cited by 637 · cited 3× · see citing works
“Semantic codes were mined using a pre-trained w2v-BERT model [102], and coarse acoustic codes were extracted using a pre-trained SoundStream codec [101].”
Method
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1802.08232 · Cited by 629 · cited 1× · see citing works
“Carlini et al. [257] explored the unintended memorisation of generative sequence models and how this memorisation can reveal personally identifiable information.”
Background
Jukebox: A Generative Model for MusicPreprint
Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., & Sutskever, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00341 · Cited by 602 · see citing works
Recent Advances in Recurrent Neural NetworksPreprint
Salehinejad, H., Sankar, S., Barfett, J., Colak, E., & Valaee, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.01078 · Cited by 584 · cited 1× · see citing works
“To combat this, LSTMs implement a gating mechanism alongside memory cells to regulate the information flow and mitigate issues related to gradients [74, 75].”
Background
Generating Wikipedia by Summarizing Long SequencesPreprint
Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, Ł., & Shazeer, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10198 · Cited by 578 · cited 1× · see citing works
“architectures tailored for sequence-to-sequence generation: encoder-decoder [232], language model [233], and prefix language model [234].”
Method
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language ModelsPreprint
Hu, Z., Wang, L., Lan, Y.-H., Xu, W., Lim, E.-P., Bing, L., Xu, X., Poria, S., & Lee, R. K.-W. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2304.01933 · Cited by 543 · cited 1× · see citing works
“Although there exist efficient fine-tuning techniques like parameter efficient training, prompt-based efficient tuning, and memory efficient tuning for text processing [94, 271–273], comparable methods are limited for speech and audio processing [218, 274, 275].”
Method
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540 · cited 1× · see citing works
“biology [42, 43], computer programming [64, 65], law [44– 46, 66], or robotics [47, 67, 68] among others.”
Background
ω-Languages
Staiger, L. (1997) · Handbook of Formal Languages · doi:10.1007/978-3-642-59126-6_6 · Cited by 529
Findings of the 2018 Conference on Machine Translation (WMT18)
Bojar, O., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Koehn, P., & Monz, C. (2018) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w18-6401 · Cited by 521 · see citing works
A stochastic model of human-machine interaction for learning dialog strategies
Levin, E., Pieraccini, R., & Eckert, W. (2000) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.817450 · Cited by 516 · cited 1× · see citing works
“a pivotal role in making action selections based on observed events [28, 256].”
Background
AudioPaLM: A Large Language Model That Can Speak and ListenPreprint
Rubenstein, P. K., Asawaroengchai, C., Nguyen, D., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2306.12925 · Cited by 511 · cited 6× · see citing works
“Rubenstein et al. [95] introduce a multimodal generative model called AudioPaLM (see figure 5) for speech and text, capable of both understanding and generating speech.”
BackgroundMethodInfluential
Multi-step Jailbreaking Privacy Attacks on ChatGPTPreprint
Li, H., Guo, D., Fan, W., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2304.05197 · Cited by 507 · cited 3× · see citing works
“We followed the multi-step provided in [252], and ChatGPT deployed by a third party provided personally identifiable information (name, email, and address).”
BackgroundMethod
MuST-C: a Multilingual Speech Translation Corpus
Gangi, M. A. D., Cattoni, R., Bentivogli, L., Negri, M., & Turchi, M. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1202 · Cited by 499 · cited 1× · see citing works
“MuST-C [133] ASR, MT and SLT 3600 hours ✓ ✓”
Deep Voice: Real-time Neural Text-to-SpeechPreprint
Arık, S. Ö., Chrzanowski, M., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J. J., Ng, A. Y., Raiman, J., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.07825 · Cited by 483 · cited 1× · see citing works
“Popular TTs models include Tacotron [168], Deep Voice model [169], and Clarinet [170], and many other [26].”
Method
Semantic Annotation and Retrieval of Music and Sound Effects
Turnbull, D., Barrington, L., Torres-Moreno, D., & Lanckriet, G. R. G. (2008) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2007.913750 · Cited by 483 · cited 1× · see citing works
“The Audio VAE was trained using a blend of datasets: AudioSet [104], AudioCaps [102], Freesound2 [109], and BBC SoundEffect Library3 [223].”
Method
DoReMi: Optimizing Data Mixtures Speeds Up Language Model PretrainingPreprint
Xie, S. M., Pham, H., Dong, X., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.10429 · Cited by 477 · cited 1× · see citing works
“Xie et al. [262] proposed domain reweighting with min-max optimisation to address data mixing challenges in foundational models designed for text processing.”
Background
Libri-Light: A Benchmark for ASR with Limited or No Supervision
Kahn, J., Rivière, M., Zheng, W., Kharitonov, E., Xu, Q., Mazaré, P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9052942 · Cited by 475 · cited 1× · see citing works
“Drawing from SoundStream [101], w2vBERT [102], the k-means quantiser for w2v-BERT embeddings, and decoder-only transformers, all of which have been trained on the extensive Libri-Light [103] English dataset encompassing 60,000 hours of speech data, the authors assembled these components.”
Method
FSD50K: An Open Dataset of Human-Labeled Sound Events
Fonseca, E., Favory, X., Pons, J., Font, F., & Serra, X. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3133208 · Cited by 462 · cited 3× · see citing works
“Audio VAE was trained using a blend of datasets: AudioSet [119], AudioCaps [120], Freesound2 [123], and BBC SoundEffect Library3 [233].”
Method
Findings of the 2019 Conference on Machine Translation (WMT19)
Barrault, L., Bojar, O., Costa‐jussà, M. R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., et al. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5301 · Cited by 461 · see citing works
Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., & Dubnov, S. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10095969 · Cited by 459 · cited 2× · see citing works
“During the training phase, the CLAP model [223] is refined on the Audiostock dataset, which features 9,000 music tracks for training and an additional 1,000 tracks reserved for testing.”
Method
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
Wang, Y., Le, H., Gotmare, A. D., Bui, N. D. Q., Li, J., & Hoi, S. C. H. (2023) · arXiv · doi:10.48550/arxiv.2305.07922 · Cited by 459 · cited 1×
“biology [42, 43], computer programming [64, 65], law [44– 46, 66], or robotics [47, 67, 68] among others.”
Background
Vggsound: A Large-Scale Audio-Visual Dataset
Chen, H., Xie, W., Vedaldi, A., & Zisserman, A. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053174 · Cited by 452 · see citing works
MaskGIT: Masked Generative Image Transformer
Chang, H., Zhang, H., Jiang, L., Liu, C., & Freeman, W. T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01103 · Cited by 446 · cited 1× · see citing works
“The foundational principles of VAMPNET are inspired by the Masked Global Information Tokeniser (MaskGIT) methodology[214].”
Method
Audiolm: A language modeling approach to audio generation
Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M., et al. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3288409 · Cited by 432 · cited 7× · see citing works
“Although AudioLM [99] claims to preserve speaker identification and prosody, its exploration of emotions and other paralinguistic information remains unknown.”
BackgroundMethodInfluential
Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions
Huang, Y.-S., & Yang, Y.-H. (2020) · ACM Multimedia · doi:10.1145/3394171.3413671 · Cited by 419 · cited 1×
“For the music generation phase, the REMI-like technique [234] was employed to convert MIDI into token sequences.”
Method
Transformers in Remote Sensing: A Survey
Aleissaee, A. A., Kumar, A., Anwer, R., et al. (2022) · Remote Sensing · doi:10.48550/arxiv.2209.01206 · Cited by 414
Findings of the 2017 Conference on Machine Translation (WMT17)
Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huang, S., Huck, M., Koehn, P., Liu, Q., Logacheva, V., et al. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4717 · Cited by 396 · see citing works
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.936 · Cited by 390 · cited 1× · see citing works
“These techniques are efficient attention mechanisms [277– 280], length generalisation [178, 281–284], and transformer alternatives [285–288].”
Background
A review of deep learning techniques for speech processing
Mehrish, A., Majumder, N., Bharadwaj, R., Mihalcea, R., & Poria, S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101869 · Cited by 389 · cited 1× · see citing works
“Mehrish et al. [27] 2023 ✗ ✓ Speech Comprehensive survey covering applications of deep learning in speech processing.”
Background
Welcome to the Era of ChatGPT et al.: The Prospects of Large Language Models
Teubner, T., Flath, C. M., Weinhardt, C., van der Aalst, W., & Hinz, O. (2023) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00795-x · Cited by 384 · cited 1× · see citing works
“Meanwhile, Large AI Models , particularly Large Language Models (LLMs), have demonstrated outstanding accomplishments in almost every AI domain, reshaping how humans interact with machines [13–15].”
Background
Variational Diffusion Models
Kingma, D. P., Salimans, T., Poole, B., & Ho, J. (2021) · arXiv · doi:10.48550/arxiv.2107.00630 · Cited by 374 · see citing works
ClariNet: Parallel Wave Generation in End-to-End Text-to-SpeechPreprint
Ping, W., Peng, K.-N., & Chen, J.-T. (2018) · International Conference on Learning Representations · doi:10.48550/arxiv.1807.07281 · Cited by 370 · cited 1× · see citing works
“Popular TTs models include Tacotron [168], Deep Voice model [169], and Clarinet [170], and many other [26].”
Method
Fundamentals of Music Processing
Müller, M. (2015) · Cambridge International Law Journal · doi:10.1007/978-3-319-21945-5 · Cited by 367 · cited 1×
“substantial progress and promising potential, the integration of large models into audio processing presents unique challenges and requires dedicated exploration [53, 54].”
Background
MLS: A Large-Scale Multilingual Dataset for Speech Research
Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., & Collobert, R. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2826 · Cited by 361 · cited 1× · see citing works
“The Multilingual LibriSpeech (MLS) dataset derived from LibriVox [162], encompassing 50,000 hours of speech recordings in 08 different languages, serves as the basis for evaluation.”
Method
BloombergGPT: A Large Language Model for Finance
Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., & Mann, G. (2023) · arXiv · Cited by 357 · cited 1× · see citing works
“This brittleness can cause serious consequences when LLMs are deployed in response-critical applications such as healthcare [290], finance [291], and law [292].”
Background
Pengi: An Audio Language Model for Audio TasksPreprint
Deshmukh, S., Elizalde, B., Singh, R., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.11834 · Cited by 357 · cited 1× · see citing works
“In a similar vein, Deshmukh et al. [123] present an innovative audio language model, Pengi, which approaches audio tasks as text generation tasks through the implementation of Transfer Learning.”
Background
On the Challenges and Perspectives of Foundation Models for Medical Image AnalysisPreprint
Zhang, S.-T., & Metaxas, D. N. (2023) · Medical Image Anal. · doi:10.48550/arxiv.2306.05705 · Cited by 350 · cited 1× · see citing works
“[41] 2023 ✓ ✗ Health Survey of medical foundation models, from general vision to modality and taskspecific ones, emphasising their challenges, opportunities, and uses.”
Background
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation
Wang, C., Rivière, M., Lee, A., Wu, A., Talnikar, C., Haziza, D., Williamson, M., Pino, J., & Emmanuel Dupoux, A. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.80 · Cited by 339 · cited 1× · see citing works
“VoxPopuli [134] ASR, S2ST 400k hours ✓ ✓”
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · cited 1× · see citing works
“[40] 2023 ✓ ✗ Health Reviews 84 foundation models using non-imaging EMR data, categorising their architectures, training sources, and applications.”
Background
Deep voice 3: scaling text-to-speech with convolutional sequence learning
Ping, W., Peng, K., Gibiansky, A., Arik, S. O., Kannan, A., Narang, S., Raiman, J., & Miller, J. (2017) · arXiv · doi:10.48550/arxiv.1710.07654 · Cited by 313 · see citing works
Deduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.577 · Cited by 304 · cited 1× · see citing works
“Lee et al. [250] introduced the NearDup tool, which effectively eliminates these doppelganger instances and repetitive substrings from datasets.”
Method
Language Model Tokenizers Introduce Unfairness Between LanguagesPreprint
Bibi, A., Malfa, E. L., Petrov, A., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.15425 · Cited by 303 · cited 1× · see citing works
“Petrov et al. [264] have shown that tokenisation is the procedure in which the language bias first manifests itself in the LLM pipeline.”
Background
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?
Ye, Q., Sachan, D. S., Felix, M., Padmanabhan, S. J., & Neubig, G. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2084 · Cited by 302 · cited 1× · see citing works
“[121], WMT/TED TTS built from WMT [122–127], and TED [128], while PALM-2 is employed for translating YouTube, CommonVoice, and Babel [129] to construct PaLM MT TTS.”
Method
Sources of Hallucination by Large Language Models on Inference TasksPreprint
McKenna, N., Li, T., Cheng, L., Hosseini, M., Johnson, M., & Steedman, M. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2305.14552 · Cited by 297 · see citing works
Improving Neural Machine Translation Models with Monolingual Data
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1009 · Cited by 296 · cited 1× · see citing works
“In particular, they integrate BART/T5-style pertaining [177, 178] with back translation [179] to substantially decrease the quantity of parallel supervision necessary for training SPEAR-TTS.”
Background
Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language ModelingPreprint
Zhang, Z.-H., Zhou, L., Wang, C.-Y., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2303.03926 · Cited by 295 · cited 3× · see citing works
“VioLA is built upon VALL-E [115] and VALL-E X [116], which share TTS capabilities akin to GPT.”
BackgroundMethod
HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection
Chen, K., Du, X., Zhu, B., Ma, Z., Berg-Kirkpatrick, T., & Dubnov, S. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9746312 · Cited by 294 · cited 1× · see citing works
“This model harnesses the capabilities of an audio transformer HTSAT [243] as an audio encoder, along with CLIP’s [244] text encoder and GPT2-base as a language model.”
Method
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
Chung, Y.-A., Zhang, Y., Han, W., Chiu, C.-C., Qin, J., Pang, R., & Wu, Y. (2021) · 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru51503.2021.9688253 · Cited by 293 · cited 2× · see citing works
“Semantic codes were mined using a pre-trained w2v-BERT model [102], and coarse acoustic codes were extracted using a pre-trained SoundStream codec [101].”
Method
Findings of the 2015 Workshop on Statistical Machine Translation
Bojar, O., Chatterjee, R., Federmann, C., Haddow, B., Huck, M., Hokamp, C., Koehn, P., Logacheva, V., Monz, C., Negri, M., et al. (2015) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w15-3001 · Cited by 287 · see citing works
Clotho: an Audio Captioning Dataset
Drossos, K., Lipping, S., & Virtanen, T. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9052990 · Cited by 285 · cited 2× · see citing works
“Clotho [103] AC 4981 audios 24905 captions ✓”
Background
High Fidelity Neural Audio Compression
Défossez, A., Copet, J., Synnaeve, G., & Adi, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.13438 · Cited by 282 · cited 1× · see citing works
“Enodec [118] is employed to encode audio into a continuous tensor.”
Method
Jailbroken: How Does LLM Safety Training Fail?
Wei, A., Haghtalab, N., & Steinhardt, J. (2023) · arXiv · doi:10.48550/arxiv.2307.02483 · Cited by 280 · see citing works
Fastpitch: Parallel Text-to-Speech with Pitch Prediction
Łańcucki, A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413889 · Cited by 274 · cited 1× · see citing works
“This has given rise to diverse architectures such as Wav2Vec [83], Whisper [84], FastPitch [85], MusicBERT [86], and others [26, 87, 88].”
Background
Risks and Benefits of Large Language Models for the Environment
Rillig, M. C., Ågerstrand, M., Bi, M., Gould, K. A., & Sauerland, U. (2023) · Environmental Science & Technology · doi:10.1021/acs.est.3c01106 · Cited by 272 · see citing works
Semi-supervised Multitask Learning for Sequence Labeling
Rei, M. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1194 · Cited by 265 · cited 1× · see citing works
“The baseline models chosen include Bi-LSTM plus a Linear Layer for overall assessment and Bi-LSTM plus conditional random field (CRF) [195] for fine-grained analysis.”
Method
A Pretrainer’s Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & ToxicityPreprint
Longpre, S., Yauney, G., Reif, E., et al. (2023) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.13169 · Cited by 258 · cited 1× · see citing works
“While a few studies [260, 261] on textual LLMs have outlined methods for weighting multi-domain data, this challenge remains unresolved in the context of speech and audio processing.”
Background
Deep Learning Techniques for Music Generation -- A SurveyPreprint
Briot, J.-P., Hadjeres, G., & Pachet, F.-D. (2017) · arXiv (Cornell University) · doi:10.4230/lipics.cp.2022.30 · Cited by 256 · cited 1× · see citing works
“While several comprehensive surveys delve into the applications of deep learning for audio processing [11, 28, 53, 54], including speech [25, 26], music [55–57], and other categories [58, 59], none concentrate on the advent and deployment of LLMs in this field.”
Background
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPPreprint
Mielke, S. J., Alyafeai, Z., Salesky, E., Raffel, C. A., Dey, M., Gallé, M., Raja, A., Si, C., Lee, W. Y., Sagot, B., et al. (2021) · arXiv.org · doi:10.48550/arxiv.2112.10508 · Cited by 253 · cited 1× · see citing works
“It is the process of dividing words or letters into smaller chunks [263].”
Background
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.02111 · Cited by 250 · see citing works
Sound Event Detection in Domestic Environments with Weakly Labeled Data and Soundscape Synthesis
Turpault, N., Serizel, R., Salamon, J., & Shah, A. (2019) · Detection and Classification of Acoustic Scenes and Events Workshop (DCASE) · doi:10.33682/006b-jx26 · Cited by 248 · cited 1× · see citing works
“For instance, Xu et al. [246] improved SED performance on the DESED public evaluation dataset [247] by more than 8.5% by utilising features from PANNs compared to the baseline.”
MethodInfluential
Large AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B. J., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 246 · cited 2× · see citing works
“[39] 2023 ✓ ✗ Health Comprehensive review of Large AI Models in health informatics including drug discovery, medical diagnosis and decision-making, medical imaging, medical informatics, medical education, public health, and medical robotics.”
Background
Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text TranslationPreprint
Berard, A., Pietquin, O., Servan, C., & Besacier, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.01744 · Cited by 243 · cited 1× · see citing works
“Various studies have explored methods and techniques to improve both cascaded ST systems [185] and end-to-end ST systems [186].”
Method
FMA: A Dataset for Music Analysis
Defferrard, M., Benzi, K., Vandergheynst, P., & Bresson, X. (2016) · arXiv · doi:10.48550/arxiv.1612.01840 · Cited by 237 · see citing works
Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C.-Y., Wang, L.-J., Liu, Z.-C., & Gao, J.-F. (2022) · Foundations and Trends in Computer Graphics and Vision · doi:10.48550/arxiv.2210.09263 · Cited by 237 · cited 1×
“…detecting text generated by LLMs, while Chang et al. [29] have examined the various ways to evaluate LLMs. Additionally, there are a number of surveys dedicated to investigating the specialised applications of Large Models in various fields such as vision [23, 24, 32, 33], education [34–37, 63 ],”
Background
GigaSpeech: An Evolving, Multi-Domain ASR Corpus with 10,000 Hours of Transcribed Audio
Chen, G., Chai, S., Wang, G., Du, J., Zhang, W.-Q., Weng, C., Su, D., Povey, D., Trmal, J., Zhang, J., et al. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1965 · Cited by 234 · cited 1× · see citing works
“Wenetspeech [131] ASR 10000 hours ✗ Gigaspeech [132] ASR 50000 hours ✗ ✓”
Background
Deep Voice 2: Multi-Speaker Neural Text-to-SpeechPreprint
Arık, S. Ö., Diamos, G., Gibiansky, A., Miller, J. J., Peng, K., Ping, W., Raiman, J., & Zhou, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08947 · Cited by 234 · see citing works
ProPILE: Probing Privacy Leakage in Large Language ModelsPreprint
Kim, S., Yun, S., Lee, H., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2307.01881 · Cited by 230 · see citing works
Diffsound: Discrete Diffusion Model for Text-to-Sound Generation
Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., & Yu, D. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3268730 · Cited by 222 · cited 2× · see citing works
“With the CLAP encoder, the training of the LDM does not require audio-text pairs any more, which is substantially different from the previous method such as AudioGen [104] and DiffSound [106].”
MethodSelf-citation
AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining
Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q., Tian, Q., Wang, Y.-P., Wang, W., Wang, Y., & Plumbley, M. D. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3399607 · Cited by 213 · cited 1× · see citing works
“Built on the success of AudioLDM, the authors have created a more advanced model called AudioLDM 2 [107], which aims to develop a general audio representation method called ”language of audio” (LOA), for speech, music and general sound effects.”
BackgroundSelf-citation
Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion ModelPreprint
Ghosal, D., Majumder, N., Mehrish, A., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2304.13731 · Cited by 213 · cited 3× · see citing works
“Although there exist efficient fine-tuning techniques like parameter efficient training, prompt-based efficient tuning, and memory efficient tuning for text processing [94, 271–273], comparable methods are limited for speech and audio processing [218, 274, 275].”
BackgroundMethod
On Generative Spoken Language Modeling from Raw Audio
Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J., Baevski, A., Mohamed, A., et al. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00430 · Cited by 212 · cited 5× · see citing works
“This combination of Hubert features and corresponding clustering models facilitated the development of uLM within GSLM with heightened efficiency.”
MethodInfluential
Contrastive Audio-Visual Masked Autoencoder
Gong, Y., Rouditchenko, A., Liu, A. H., et al. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2210.07839 · Cited by 212 · cited 1×
“To align the embedding dimensions with LLaMA, a pre-trained Audio Spectrogram Transformer is used alongside the CAV-MAE [111] and fine-tuned on AudioSet-2M [112] for audio encoding.”
Method
Transformer Language Models without Positional Encodings Still Learn Positional Information
Haviv, A., Ram, O., Press, O., Izsak, P., & Levy, O. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2203.16634 · Cited by 210 · cited 1×
“These techniques are efficient attention mechanisms [277– 280], length generalisation [178, 281–284], and transformer alternatives [285–288].”
Background
Are Emergent Abilities of Large Language Models a Mirage?
Schaeffer, R., Miranda, B., & Koyejo, S. (2023) · arXiv · doi:10.48550/arxiv.2304.15004 · Cited by 200 · see citing works
YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone
Casanova, E., Weber, J., Shulby, C., Junior, A. C., Gölge, E., & Ponti, M. A. (2021) · arXiv · doi:10.48550/arxiv.2112.02418 · Cited by 200 · see citing works
Environmental sound recognition: A survey
Chachada, S., & Kuo, C.-C. J. (2013) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference · doi:10.1017/atsip.2014.12 · Cited by 200 · cited 1× · see citing works
“While several comprehensive surveys delve into the applications of deep learning for audio processing [11, 28, 53, 54], including speech [25, 26], music [55–57], and other categories [58, 59], none concentrate on the advent and deployment of LLMs in this field.”
Background
Challenges and Applications of Large Language Models
Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., & McHardy, R. (2023) · arXiv · doi:10.48550/arxiv.2307.10169 · Cited by 196 · cited 2× · see citing works
“As the foundational research in this area grows, the insights presented here, along with related works, can serve as a valuable guide to understanding the complexities and addressing the challenges in the development of improved Large Audio Models for audio applications [23, 30, 299, 300].”
Background
POP909: A Pop-Song Dataset for Music Arrangement Generation
Wang, Z., Chen, K., Jiang, J.-Y., et al. (2020) · International Society for Music Information Retrieval Conference · doi:10.5281/zenodo.4245366 · Cited by 195 · cited 1×
“To train attribute-to-music conversion, they used a variety of MIDI datasets, including Million MIDI Dataset [86], EMPOIA [140], MetaMidi [141], POP909 [147], Symphony [145], and an internal dataset named Emotion-gen.”
Method
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., & Qiu, X. (2023) · arXiv · doi:10.48550/arxiv.2305.11000 · Cited by 189 · cited 3× · see citing works
“Zhang et al. [91] proposed SpeechGPT, a large language model that has intrinsic cross-modal conversational abilities that allow it to generate multimodal content.”
BackgroundMethodSelf-citation
Focused Transformer: Contrastive Training for Context ScalingPreprint
Tworkowski, S., Staniszewski, K., Pacek, M., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2307.03170 · Cited by 189 · cited 1× · see citing works
“These techniques are efficient attention mechanisms [277– 280], length generalisation [178, 281–284], and transformer alternatives [285–288].”
Background
AudioLDM: Text-to-Audio Generation with Latent Diffusion ModelsPreprint
Liu, H., Chen, Z., Yuan, Y. G., Mei, X., Liu, X., Mandic, D. P., Wang, W., & Plumbley, M. D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.12503 · Cited by 188 · cited 6× · see citing works
“In particular, the approach of MusicLDM has significant similarities with AudioLDM [50].”
BackgroundMethodResultInfluentialSelf-citation
Untitled work
(2024) · doi:10.1145/3704435 · Cited by 184 · cited 1× · see citing works
“biology [42, 43], computer programming [64, 65], law [44– 46, 66], or robotics [47, 67, 68] among others.”
Background
Tool Learning with Foundation Models
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.08354 · Cited by 184 · see citing works
MusicLM: Generating Music From Text
Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.11325 · Cited by 183 · cited 3× · see citing works
“Figure taken from [127]. comparable to MusicLM [119].”
BackgroundMethodResult
Analyzing Leakage of Personally Identifiable Information in Language Models
Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, L., & Zanella-Béguelin, S. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179300 · Cited by 181 · cited 1× · see citing works
“In a similar work, Lukas et al. [253] analysed the personal information leakage in healthcare, emails, and law domains and showed that even after using modern scrubbing techniques 3% personally identifiable information is leaked.”
Method
SoundStorm: Efficient Parallel Audio GenerationPreprint
Borsos, Z., Sharifi, M., Vincent, D., Kharitonov, E., Zeghidour, N., & Tagliasacchi, M. (2023) · arXiv.org · doi:10.48550/arxiv.2305.09636 · Cited by 180 · cited 2× · see citing works
“They employ autoregressive techniques as in AudioLM [99], as well as non-autoregressive approaches similar to [100] to translate decoding tokens into audio.”
Method
Findings of the 2020 Conference on Machine Translation (WMT20)
Barrault, L., Biesialska, M., Bojar, O., Costa‐jussà, M. R., Federmann, C., Graham, Y., Grundkiewicz, R., Haddow, B., Huck, M., Joanis, E., et al. (2020) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2020.wmt-1.1 · Cited by 178 · see citing works
The VoiceMOS Challenge 2022
Huang, W.-C., Cooper, E., Tsao, Y., et al. (2022) · Interspeech · doi:10.48550/arxiv.2203.11389 · Cited by 178 · cited 1×
“The authors harnessed pre-trained models from GSLM [98] through fairseq 2 and employed the VoiceMOS challenge dataset [181], which encompasses speech from diverse sources.”
Method
Video Transformers: A Survey
Selva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapés, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3243465 · Cited by 177 · see citing works
Survey of Deep Representation Learning for Speech Emotion Recognition
Latif, S., Rana, R. K., Khalifa, S., Jurdak, R., Qadir, J., & Schuller, B. W. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3114365 · Cited by 177 · cited 1× · see citing works
“It is especially difficult for tasks requiring long-term dependency understanding, such as speech recognition [276].”
BackgroundSelf-citation
Speech Technology for Healthcare: Opportunities, Challenges, and State of the Art
Latif, S., Qadir, J., Qayyum, A., Usama, M., & Younis, M. S. (2020) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.3006860 · Cited by 177 · cited 1× · see citing works
“These applications range from voice-activated assistants like Siri and Alexa [1, 2] to transcription services [3], and extend to telecommunication systems [4] and hearing aids [5].”
BackgroundSelf-citation
Deep learning for music generation: challenges and directions
Briot, J.-P., & Pachet, F. (2017) · Neural computing & applications (Print) · doi:10.1007/s00521-018-3813-6 · Cited by 173 · cited 1×
“This sets it apart from conventional methods [209].”
Method
Learning Video Representations from Large Language Models
Zhao, Y., Misra, I., Krähenbühl, P., & Girdhar, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00637 · Cited by 164 · cited 1× · see citing works
“While large AI models have made remarkable advancements in the domains of language [48], images [23], and videos [49], the audio arena has followed a more gradual trajectory.”
Background
Acoustic Modeling for Google Home
Li, B., Sainath, T., Narayanan, A., et al. (2017) · Interspeech · doi:10.21437/interspeech.2017-234 · Cited by 163 · cited 1×
“These applications range from voice-activated assistants like Siri and Alexa [1, 2] to transcription services [3], and extend to telecommunication systems [4] and hearing aids [5].”
Background
SpikeGPT: Generative Pre-trained Language Model with Spiking Neural NetworksPreprint
Zhu, R., Zhao, Q., Li, G.-Q., & Eshraghian, J. (2023) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2302.13939 · Cited by 162 · cited 1× · see citing works
“While a few recent attempts have been made to reduce the computational cost and energy usage [93, 267–269], the race to the state of the art in various audio-related tasks is quickly pushing up the pre-training cost.”
Background
Augmented Language Models: a Survey
Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.07842 · Cited by 160 · cited 1× · see citing works
“Similarly, Mialon et al. [60] turn their attention towards augmented language models, those with advanced reasoning capabilities and tool usage skills.”
Background
EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generation
Hung, H.-T., Ching, J., Doh, S., et al. (2021) · International Society for Music Information Retrieval Conference · doi:10.5281/zenodo.5090631 · Cited by 157 · cited 1×
“To train attribute-to-music conversion, they used a variety of MIDI datasets, including Million MIDI Dataset [86], EMPOIA [140], MetaMidi [141], POP909 [147], Symphony [145], and an internal dataset named Emotion-gen.”
Method
WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition
Zhang, B., Lv, H., Guo, P., Shao, Q., Yang, C., Xie, L., Xu, X., Bu, H., Chen, X., Zeng, C., et al. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746682 · Cited by 156 · cited 1× · see citing works
“Wenetspeech [131] ASR 10000 hours ✗ Gigaspeech [132] ASR 50000 hours ✗ ✓”
Background
Leveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation
Ye, J., Johnson, M., Macherey, W., Weiss, R. J., Cao, Y. J., Chiu, C., Ari, N., Laurenzo, S., & Wu, Y. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683343 · Cited by 153 · see citing works
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version
Serban, I. V., Lowe, R., Henderson, P., Charlin, L., & Pineau, J. (2018) · Dialogue & Discourse · doi:10.5087/dad.2018.101 · Cited by 150 · cited 1× · see citing works
“While text-based dialogue systems can be trained directly on extensive text data [195, 196], a large number of SDSs have relied on user simulations for training due to the scarcity of real training dialogues available for both training and evaluation purposes [197].”
Background
SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language ModelsPreprint
Zhang, X., Zhang, D., Li, S., et al. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2308.16692 · Cited by 150 · cited 1× · see citing works
“[266] presents a unified speech tokeniser and evaluates this on different speech-related tasks.”
BackgroundSelf-citation
Icassp 2022 Deep Noise Suppression Challenge
Dubey, H., Gopal, V., Cutler, R., Aazami, A., Matusevych, S., Braun, S., Eskimez, Ş. E., Thakker, M., Yoshioka, T., Gamper, H., et al. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747230 · Cited by 149 · cited 1× · see citing works
“0 [93], DNS Challenge 4 [147], AudioSet [104], FSD50K [109], and Jamendo [112].”
AST: Audio Spectrogram TransformerPreprint
Gong, Y., Chung, Y.-A., & Glass, J. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01778 · Cited by 145 · see citing works
Retentive Network: A Successor to Transformer for Large Language Models
Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., & Wei, F. (2023) · arXiv · doi:10.48550/arxiv.2307.08621 · Cited by 142 · cited 1× · see citing works
“These techniques are efficient attention mechanisms [277– 280], length generalisation [178, 281–284], and transformer alternatives [285–288].”
Background
A Survey of Hallucination in Large Foundation Models
Rawte, V., Sheth, A., & Das, A. (2023) · arXiv · doi:10.48550/arxiv.2309.05922 · Cited by 133 · cited 1× · see citing works
“It is a behaviour where LLMs generate factually incorrect information that is difficult to detect due to the large number of outputs that LLMs generate [294].”
Background
NExT-GPT: Any-to-Any Multimodal LLMPreprint
Wu, S., Fei, H., Qu, L., Ji, W., & Chua, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.05519 · Cited by 132 · cited 1× · see citing works
“NExT-GPT [125] T2M MosIT ✓ ✓ ✓ ✗ ✗ Text-to-Image Text-to-Video Text-to-Image ✓”
BEATs: Audio Pre-Training with Acoustic Tokenizers
Chen, S., Wu, Y., Wang, C., Liu, S., Tompkins, D., Chen, Z., & Wei, F. (2022) · arXiv · doi:10.48550/arxiv.2212.09058 · Cited by 131 · cited 1× · see citing works
“Large audio models dedicated to obtaining universal audio representations have also been widely applied in the field of sound event detection (SED), such as PANNs [238], AST [109], and BEATs [245].”
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z., Wu, Y., Hong, Z., Huang, J., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12995 · Cited by 131 · cited 2× · see citing works
“Huang et al. [122] introduce AudioGPT, designed to excel in various speech-related tasks.”
BackgroundMethodSelf-citation
Translatotron 2: High-quality direct speech-to-speech translation with voice preservation
Jia, Y., Ramanovich, M., Remez, T., et al. (2021) · International Conference on Machine Learning · doi:10.48550/arxiv.2107.08661 · Cited by 130
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Chen, S., Wang, C., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, T., Wang, H., Li, J., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3530270 · Cited by 128 · cited 4× · see citing works
“VioLA is built upon VALL-E [115] and VALL-E X [116], which share TTS capabilities akin to GPT.”
BackgroundMethodInfluential
VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and TranslationPreprint
Wang, T.-R., Zhou, L., Zhang, Z.-H., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2305.16107 · Cited by 127 · cited 3× · see citing works
“Building upon the previous discussion, Wang et al. [114] proposed VioLA, a language model encompassing a decoder-only transformer network which is multilingual and multimodal based on an auto-regressive approach that exhibits proficiency in speech-related tasks with the capability of speech…”
BackgroundMethodInfluential
The Science of Detecting LLM-Generated Text
Tang, R., Chuang, Y.-N., & Hu, X. (2024) · Communications of the ACM · doi:10.1145/3624725 · Cited by 125 · cited 1× · see citing works
“Tang et al. [62] focus on techniques for detecting text generated by LLMs, while Chang et al. [29] have examined the various ways to evaluate LLMs. Additionally, there are a number of surveys dedicated to investigating the specialised applications of Large Models in various fields such as vision…”
Background
Prompting Large Language Models with Speech Recognition Abilities
Fathullah, Y., Wu, C., Lakomkin, E., Jia, J., Shangguan, Y., Li, K., Guo, J., Xiong, W., Mahadeokar, J., Kalinli, O., et al. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447605 · Cited by 125 · cited 1× · see citing works
“Fathullah et al. [51] delve into extending the practicality of LLMs by directly incorporating a compact audio encoder, thus enabling them to perform speech recognition tasks.”
Background
Cramming: Training a Language Model on a Single GPU in One DayPreprint
Geiping, J., & Goldstein, T. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2212.14034 · Cited by 125 · cited 1× · see citing works
“These techniques are efficient attention mechanisms [277– 280], length generalisation [178, 281–284], and transformer alternatives [285–288].”
Background
Hungry Hungry Hippos: Towards Language Modeling with State Space Models
Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2212.14052 · Cited by 123 · see citing works
MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training
Zeng, M., Tan, X., Wang, R., Ju, Z., Qin, T., & Liu, T. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.70 · Cited by 122 · cited 2× · see citing works
“To train attribute-to-music conversion, they used a variety of MIDI datasets, including Million MIDI Dataset [86], EMPOIA [140], MetaMidi [141], POP909 [147], Symphony [145], and an internal dataset named Emotion-gen.”
BackgroundMethod
Adapting Transformer to End-to-End Spoken Language Translation
Gangi, M. A. D., Negri, M., & Turchi, M. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3045 · Cited by 122 · see citing works
AudioGen: Textually Guided Audio GenerationPreprint
Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., Défossez, A., Copet, J., Parikh, D., Taigman, Y., & Adi, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15352 · Cited by 121 · cited 4× · see citing works
“AudioGen [104], a critical component of this framework, is an auto-regressive model that effectively addresses the challenge of generating audio while incorporating textual inputs.”
BackgroundMethodInfluential
Textually Pretrained Speech Language ModelsPreprint
Hassid, M., Remez, T., Nguyen, T., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.13009 · Cited by 117 · cited 1× · see citing works
“Hassid et al. [175] introduced TWIST, an innovative approach to training SpeechLMs that employs a warm-start strategy with a pre-trained textual LLM.”
Background

Flagged first, then by citations

Research works printed but not linked: 13

B. Gold, N. Morgan, and D. Ellis, Speech and audio signal processing: processing and perception of speech and music. John Wiley & Sons, 2011.
Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 436-444, 2015.
A. B. Cyphert, "A human being wrote this law review article: Gpt-3 and the practice of law," UC Davis L. Rev., vol. 55, p. 401, 2021.
Y. K. Dwivedi, N. Kshetri, L. Hughes, E. L. Slade, A. Jeyaraj, A. K. Kar, A. M. Baabdullah, A. Koohang, V. Raghavan, M. Ahuja et al., ""so what if chatgpt wrote it?" multidisciplinary perspectives on opportunities, challenges and implications of generative conversa- tional ai for research, practice and policy," International Journal of Information Management, vol. 71, p. 102642, 2023.
Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, "Effi- cient transformers: A survey," ACM Computing Surveys, vol. 55, no. 6, pp. 1-28, 2022.
K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu et al., "A survey on vision transformer," IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 1, pp. 87-110, 2022.
J. Ens and P. Pasquier, "Building the metamidi dataset: Linking symbolic and audio musical data." in ISMIR, 2021, pp. 182-188.
D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, "The mtg-jamendo dataset for automatic music tagging." ICML, 2019.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, "Wavenet: A generative model for raw audio," https://arxiv.org/abs/1609.03499, 2016. · arXiv
S. Maiti, Y. Peng, T. Saeki, and S. Watanabe, "Speechlm- score: Evaluating speech generation using speech lan- guage model," in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1-5.
S. Forsgren and H. Martiros, "Riffusion-stable dif- fusion for real-time music generation, 2022," URL https://riffusion. com/about, vol. 6.
A. Banerjee and V. Arora, "wav2tok: Deep sequence tokenizer for audio retrieval," in The Eleventh Interna- tional Conference on Learning Representations, 2022.

Sorted by citations

A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations
Cheng, H., Zhang, M., & Shi, Q. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3447085 · Cited by 554 · see citing works
“O VER the past several years, Deep Neural Networks ( DNNs ) have achieved conspicuous progress in various domains and applications, such as Computer Vision ( CV ) [1], [2], Natural Language Processing ( NLP ) [3], [4], Audio Signal Processing ( ASP ) [5], [6], and cross-modal applications [7], [8].”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“However, there is a gap in the development of large audio models specifically tailored for medical applications [103].”
Affective Computing and the Road to an Emotionally Intelligent Metaverse
Pervez, F., Shoukat, M., Usama, M., Sandhu, M. M., Latif, S., & Qadir, J. (2024) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2024.3389462 · Cited by 47 · see citing works
“These models such as GPT-4 [174] and SeamlessM4t [175] can enable seamless and instantaneous communication between users, transcending language barriers through natural language and speech processing and voice translation and recognition [176].”
BackgroundMethodSelf-citation
From screens to scenes: A survey of embodied AI in healthcare
Liu, Y., Cao, X., Chen, T., Jiang, Y., You, J., Wu, M., Wang, X., Feng, M., Jin, Y., & Chen, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103033 · Cited by 59 · see citing works
Social Dangers of Generative Artificial Intelligence: Review and Guidelines
Yang, A., & Yang, T. A. (2024) · Annual International Conference on Digital Government Research · doi:10.1145/3657054.3664243 · Cited by 10 · see citing works
“Other examples of GAI include DALL-E, which can create visual artwork based on prompts, and video and sound-editing GAI which can replicate the voices or like-nesses of individuals or groups if there is sufficient data to train a model [12., 13.].”
Background
How “Real” is Your Real-Time Simultaneous Speech-to-Text Translation System?
Papi, S., Polák, P., Macháček, D., & Bojar, O. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00740 · Cited by 7 · see citing works
BanglaDialecto: An End-to-End AI-Powered Regional Speech Standardization
Samin, M. N. S., Ahad, J. I., Medha, T. A., Rahman, F. A., Amin, M. R., Mohammed, N., & Rahman, S. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826131 · Cited by 6 · see citing works
Assessing Audio Hallucination in Large Multimodal ModelsPreprint
Hanamaki, S., Kirishima, N., & Narumi, S. (2024) · doi:10.31219/osf.io/f8ra7 · Cited by 4 · see citing works
Breaking Barriers: Can Multilingual Foundation Models Bridge the Gap in Cross-Language Speech Emotion Recognition?
Shoukat, M., Usama, M., Ali, H. S., & Latif, S. (2023) · International Conference on Social Networks Analysis, Management and Security (SNAMS) · doi:10.1109/snams60348.2023.10375468 · Cited by 2 · see citing works
“In our previous work [15], recent advances in audio foundation models were covered.”
BackgroundInfluentialSelf-citation
Review of techniques used in speech signal processing
Arinaitwe, P., Murungi, E., Ogenyi, F. C., et al. (2024) · KIU journal of science engineering and technology · doi:10.59568/kjset-2024-3-1-07 · Cited by 2 · see citing works
Plugin Speech Enhancement: A Universal Speech Enhancement Framework Inspired by Dynamic Neural NetworkPreprint
Cui, Z., Chen, Y., Gao, Y., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4838997 · Cited by 1 · see citing works

Modelling of Audio Effects for Vocal and Music Synthesis in Real Time
D'souza, D., & Shastrimath, V. V. D. (2019) · International Conference on Computing Methodologies and Communication (ICCMC) · doi:10.1109/iccmc.2019.8819852 · Cited by 2 · see citing works
Audio decoding apparatus, signal processing device, sound image localization device, sound image control method, audio signal processing device, and audio signal high-rate reproduction method used for audio visual equipment
Ishito, T. (2002) · The Journal of the Acoustical Society of America · doi:10.1121/1.1514533 · Cited by 1 · see citing works
Audio signal processing arrangement for deriving a center channel signal and also an audio visual reproduction system comprising such a processing arrangement
van de Kerkhof, L. M. (1997) · The Journal of the Acoustical Society of America · doi:10.1121/1.418457
Audio Processing for Tones Validation using Deep Learning
Pandhare, P. A., & Ingale, V. V. (2022) · International Conference on Signal and Information Processing · doi:10.1109/iconsip49665.2022.10007475 · Cited by 1 · see citing works
Development of audio and video signal processing system for DV format VCR
Sato, M., Hashimoto, K., Yamauchi, E., et al. (2003) · International Conference on Consumer Electronics · doi:10.1109/icce.1999.785264

Authors: Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., & Schuller, B. W.
Affiliations: Queensland University of Technology (AU)
Mohamed bin Zayed University of Artificial Intelligence (AE)
Nanyang Technological University (SG)
University of Augsburg (DE)
ORCID: Latif, Siddique
Shoukat, Moazzam
Shamshad, Fahad
Usama, Muhammad
Cuayáhuitl, Heriberto
Cambria, Erik
Schuller, Björn W.
Venue: arXiv
Publisher: arXiv
Published: 2023
ISSN: 2331-8422 (ISSN-L)
DOI: 10.48550/arxiv.2308.12792
arXiv: 2308.12792
OpenAlex: W4386185396
Cited by: 17
References: 315
Citations by year: 2025: 6 · 2024: 10 · 2023: 1
Type: posted content
Language: en
Open access: green
arxiv.org
Licence: other-oa
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations: arXiv (Cornell University) (submittedVersion, other-oa)
arXiv (Cornell University)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Keywords: transformer models, speech recognition, text-to-speech, music generation, multilingual speech translation, audio signal processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Sparks of Large Audio Models: A Survey and Outlook

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Self-check

Cites

Cited by

Related

Record

About these checks