PreprintCited by 17Speech Recognition and SynthesisMusic and Audio ProcessingSpeech and Audio Processing

The DOI matches its DataCite record; 355 of the 355 references it cites were found in a registry or an index.

Sparks of Large Audio Models: A Survey and Outlook

Open access

Latif, Siddique, Shoukat, Moazzam, Shamshad, Fahad, Usama, Muhammad, Ren, Yi, Cuayáhuitl, Heriberto, Wang, Wenwu, Zhang, Xulong, Togneri, Roberto, Cambria, Erik, Schuller, Björn W.·2023·arXiv

DOI 10.48550/arxiv.2308.12792arXiv 2308.12792

Open this paper in CoChat — free account Open PDF

Abstract

This survey paper provides a comprehensive overview of the recent advancements and challenges in applying large language models to the field of audio signal processing. Audio processing, with its diverse signal representations and a wide range of sources--from human voices to musical instruments and environmental sounds--poses challenges distinct from those found in traditional Natural Language Processing scenarios. Nevertheless, \textit{Large Audio Models}, epitomized by transformer-based architectures, have shown marked efficacy in this sphere. By leveraging massive amount of data, these models have demonstrated prowess in a variety of audio tasks, spanning from Automatic Speech Recognition and Text-To-Speech to Music Generation, among others. Notably, recently these Foundational Audio Models, like SeamlessM4T, have started showing abilities to act as universal translators, supporting multiple speech tasks for up to 100 languages without any reliance on separate task-specific systems. This paper presents an in-depth analysis of state-of-the-art methodologies regarding \textit{Foundational Large Audio Models}, their performance benchmarks, and their applicability to real-world scenarios. We also highlight current limitations and provide insights into potential future research directions in the realm of \textit{Large Audio Models} with the intent to spark further discussion, thereby fostering innovation in the next generation of audio-processing systems. Furthermore, to cope with the rapid development in this area, we will consistently update the relevant repository with relevant recent articles and their open-source implementations at https://github.com/EmulationAI/awesome-large-audio-models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2308.12792

Cites

355 of 355 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Items 201–355 of 355

BEATs: Audio Pre-Training with Acoustic Tokenizers
Chen, S., Wu, Y., Wang, C., Liu, S., Tompkins, D., Chen, Z., & Wei, F. (2022) · arXiv · doi:10.48550/arxiv.2212.09058 · Cited by 131 · cited 1×
“Large audio models dedicated to obtaining universal audio representations have also been widely applied in the field of sound event detection (SED), such as PANNs [238], AST [109], and BEATs [245].”
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z., Wu, Y., Hong, Z., Huang, J., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12995 · Cited by 131 · cited 2×
“Huang et al. [122] introduce AudioGPT, designed to excel in various speech-related tasks.”
BackgroundMethodSelf-citation
Translatotron 2: High-quality direct speech-to-speech translation with voice preservation
Jia, Y., Ramanovich, M., Remez, T., et al. (2021) · International Conference on Machine Learning · doi:10.48550/arxiv.2107.08661 · Cited by 130
One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningPreprint
Chavan, A., Liu, Z., Gupta, D. K., Xing, E. P., & Shen, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.07967 · Cited by 128 · cited 1×
“Although there exist efficient fine-tuning techniques like parameter efficient training, prompt-based efficient tuning, and memory efficient tuning for text processing [94, 271–273], comparable methods are limited for speech and audio processing [218, 274, 275].”
Method
VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and TranslationPreprint
Wang, T.-R., Zhou, L., Zhang, Z.-H., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2305.16107 · Cited by 127 · cited 3×
“Building upon the previous discussion, Wang et al. [114] proposed VioLA, a language model encompassing a decoder-only transformer network which is multilingual and multimodal based on an auto-regressive approach that exhibits proficiency in speech-related tasks with the capability of speech…”
BackgroundMethodInfluential
A survey on deep reinforcement learning for audio-based applications
Latif, S., Cuayáhuitl, H., Pervez, F., Shamshad, F., Ali, H. S., & Cambria, E. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10224-2 · Cited by 126 · cited 4×
“Latif et al. [28] 2023”
BackgroundInfluentialSelf-citation
The Science of Detecting LLM-Generated Text
Tang, R., Chuang, Y.-N., & Hu, X. (2024) · Communications of the ACM · doi:10.1145/3624725 · Cited by 125 · cited 1×
“Tang et al. [62] focus on techniques for detecting text generated by LLMs, while Chang et al. [29] have examined the various ways to evaluate LLMs. Additionally, there are a number of surveys dedicated to investigating the specialised applications of Large Models in various fields such as vision…”
Background
Theme Transformer: Symbolic Music Generation With Theme-Conditioned Transformer
Shih, Y.-J., Wu, S.-L., Zalkow, F., Müller, M., & Yang, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3161851 · Cited by 125
Hungry Hungry Hippos: Towards Language Modeling with State Space Models
Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2212.14052 · Cited by 123
Adapting Transformer to End-to-End Spoken Language Translation
Gangi, M. A. D., Negri, M., & Turchi, M. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3045 · Cited by 122
A Short Survey of Viewing Large Language Models in Legal AspectPreprint
Sun, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09136 · Cited by 122 · cited 1×
“[45] 2023 ✓ ✗ Law Survey of LLMs in legal tasks like judgement prediction and document analysis.”
Background
AudioGen: Textually Guided Audio GenerationPreprint
Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., Défossez, A., Copet, J., Parikh, D., Taigman, Y., & Adi, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15352 · Cited by 121 · cited 4×
“AudioGen [104], a critical component of this framework, is an auto-regressive model that effectively addresses the challenge of generating audio while incorporating textual inputs.”
BackgroundMethodInfluential
On Decoder-Only Architecture For Speech-to-Text and Large Language Model Integration
Wu, J., Gaur, Y., Chen, Z., Zhou, L., Zhu, Y., Wang, T., Li, J., Liu, S., Ren, B., Liu, L., et al. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389705 · Cited by 118 · cited 1×
“For instance, Wu et al. [154] introduced the concept of speech-LLaMA, a technique that involves seamlessly integrating acoustic embeddings into a text-based large language model to enhance translation capabilities.”
Method
Textually Pretrained Speech Language ModelsPreprint
Hassid, M., Remez, T., Nguyen, T., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.13009 · Cited by 117 · cited 1×
“Hassid et al. [175] introduced TWIST, an innovative approach to training SpeechLMs that employs a warm-start strategy with a pre-trained textual LLM.”
Background
VD-BERT: A Unified Vision and Dialog Transformer with BERTPreprint
Wang, Y., Joty, S. R., Lyu, M. R., King, I., Xiong, C., & Hoi, S. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2004.13278 · Cited by 112
MusicLDM: Enhancing Novelty in text-to-music Generation Using Beat-Synchronous mixup Strategies
Chen, K., Wu, Y., Liu, H., Nezhurina, M., Berg-Kirkpatrick, T., & Dubnov, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447265 · Cited by 108 · cited 1×
“Chen et al. [221] introduced the MusicLDM model, tailored for music generation from textual inputs.”
Background
CoLT5: Faster Long-Range Transformers with Conditional ComputationPreprint
Ainslie, J., Lei, T., de Jong, M., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2303.09752 · Cited by 108 · cited 1×
“These techniques are efficient attention mechanisms [277– 280], length generalisation [178, 281–284], and transformer alternatives [285–288].”
Background
Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation
Yu, B., Lu, P., Wang, R., Hu, W., Tan, X., Ye, W., Zhang, S.-K., Qin, T., & Liu, T.-Y. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2210.10349 · Cited by 108 · cited 1×
“By comprehending long-range dependencies, harmonies, and subtleties, transformers have transformed the landscape of music generation [210, 211].”
Background
On the integration of speech recognition and statistical machine translation
Matusov, E., Kanthak, S., & Ney, H. (2005) · Interspeech · doi:10.21437/interspeech.2005-726 · Cited by 104 · cited 1×
“Various studies have explored methods and techniques to improve both cascaded ST systems [185] and end-to-end ST systems [186].”
Method
VD-BERT: A Unified Vision and Dialog Transformer with BERT
Wang, Y., Joty, S., Lyu, M. R.-T., King, I., Xiong, C., & Hoi, S. C. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.269 · Cited by 103 · cited 1×
“However, transformers have recently emerged as a superior alternative to RNNs to optimise the action selection process within SDSs [192, 193].”
Method
EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music GenerationPreprint
Hung, H.-T., Ching, J., Doh, S., Kim, N., Nam, J., & Yang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01374 · Cited by 102
Audio
Kelly, S. (2019) · Python, PyGame, and Raspberry Pi Game Development · doi:10.1007/978-1-4842-4533-0_18 · Cited by 99
Generative Spoken Dialogue Language Modeling
Nguyen, T. A., Kharitonov, E., Copet, J., Adi, Y., Hsu, W.-N., Elkahky, A. M., Tomasello, P., Algayres, R., Sagot, B., Mohamed, A., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00545 · Cited by 97 · cited 1×
“A few exceptions using reasonably Large Audio Models include dialogue generation from raw audio excluding text processing in their pipeline [201], dialogue policy learning from textual and audio features for task-oriented dialogues [202], and open and closed-domain dialogue generation [203].”
Method
Contrastive Learning Reduces Hallucination in ConversationsPreprint
Sun, W., Shi, Z., Gao, S., et al. (2022) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2212.10400 · Cited by 97 · cited 1×
“To mitigate this issue, the literature suggests several strategies, including adversarial training, diversification in training data, human feedback incorporation, contrastive learning, and enhanced regularisation [88, 295–298].”
Background
MuseCoco: Generating Symbolic Music from TextPreprint
Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.00110 · Cited by 97 · cited 1×
“Lu et al. [233] introduced MuseCoco (Music Composition Copilot), a robust framework designed to generate music based on textual descriptions.”
Background
AudioCaps: Generating Captions for Audios in The Wild
Kim, C. D., Kim, B., Lee, H., & Kim, G. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1011 · Cited by 96 · cited 1×
“Experiments for this text-to-audio generation leveraged the AudioCaps dataset [138], which consists of 45,438 audio samples.”
Method
Efficient Neural Music GenerationPreprint
Lam, M. W. Y., Tian, Q., Li, T.-C., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.15719 · Cited by 95 · cited 1×
“Lam et al. [232] introduced MeLoDy (M for music; L for Large Models; D for diffusion), an innovative audio music generation approach centred around a Language guided diffusion model [232].”
Background
UnitY: Two-pass Direct Speech-to-speech Translation with Discrete UnitsPreprint
Inaguma, H., Popuri, S., Kulikov, I., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.08055 · Cited by 95 · cited 1×
“This model operates on the multi-task UnitY architecture [128], facilitating the direct generation of translated text and speech, as well as supporting ASR and various translation modes.”
Method
Music Composition with Deep Learning: A ReviewPreprint
Hernandez-Olivan, C., & Beltrán, J. R. (2021) · arXiv.org · doi:10.48550/arxiv.2108.12290 · Cited by 94 · cited 1×
“This has given rise to diverse architectures such as Wav2Vec [83], Whisper [84], FastPitch [85], MusicBERT [86], and others [26, 87, 88].”
Background
Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture
Cui, J., Ning, M., Li, Z., Chen, B., Yan, Y., Li, H., Ling, B., Tian, Y., & Yuan, L. (2023) · arXiv · doi:10.48550/arxiv.2306.16092 · Cited by 94
Bridging Text and Video: A Universal Multimodal Transformer for Audio-Visual Scene-Aware Dialog
Li, Z., Li, Z., Zhang, J., Feng, Y., & Zhou, J. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3065823 · Cited by 91 · cited 1×
“Further, recent studies such as ANGIE [205], Multimodal-GPT[206], and Large Multimodal Models [207] have integrated either vision and LLMs or video and audio [208] for training multimodal dialogue systems.”
Background
Transformers in Speech Processing: A SurveyPreprint
Latif, S., Zaidi, A., Cuayáhuitl, H., Shamshad, F., Shoukat, M., Usama, M., & Qadir, J. (2023) · arXiv.org · doi:10.48550/arxiv.2303.11607 · Cited by 90 · cited 6×
“While several comprehensive surveys delve into the applications of deep learning for audio processing [11, 28, 53, 54], including speech [25, 26], music [55–57], and other categories [58, 59], none concentrate on the advent and deployment of LLMs in this field.”
BackgroundMethodInfluentialSelf-citation
Video Background Music Generation with Controllable Music Transformer
Di, S., Jiang, Z., Liu, S., Wang, Z., Zhu, L., He, Z., Liu, H., & Yan, S. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475195 · Cited by 89
Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
Wei, J., Kim, S., Jung, H., & Kim, Y.-H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637364 · Cited by 84 · cited 1×
“…task-oriented dialogue systems has shown to underperform in comparison with task-specific models, careful application is required for LLMs to be useful—as shown by [198] in automated scoring of user simulations, [199] in dialogue state tracking, and [200] in data collection via prompt engineering.”
Background
SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
Wang, X., Thakker, M., Chen, Z., Kanda, N., Eskimez, Ş. E., Chen, S., Tang, M., Liu, S., Li, J., & Yoshioka, T. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3419418 · Cited by 83 · cited 1×
“For example, Wang et al. [120] introduce SpeechX, a versatile architecture capable of various speech transformation tasks across both noisy and clean speech conditions.”
Background
CVSS Corpus and Massively Multilingual Speech-to-Speech Translation
Ye, J., Ramanovich, M. T., Wang, Q., & Zen, H. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/38skf3jhvobn · Cited by 80 · cited 4×
“[99], VoxPopuli ASR [100], VoxPopuli S2ST [100], Common-”
MethodInfluential
Transformers in 3D Point Clouds: A SurveyPreprint
Lu, D.-N., Xie, Q., Wei, M., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2205.07417 · Cited by 79
Foundation Models for Natural Language Processing
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2 · Cited by 79 · cited 1×
“While large AI models have made remarkable advancements in the domains of language [48], images [23], and”
Background
Advances in Online Audio-Visual Meeting Transcription
Yoshioka, T., Abramovski, I., Aksoylar, C., Chen, Z., David, M., Dimitriadis, D., Gong, Y., Gurvich, I., Huang, X., Huang, Y., et al. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003827 · Cited by 77
Audio-Driven Co-Speech Gesture Video Generation
Liu, X., Wu, Q., Zhou, H., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2212.02350 · Cited by 77 · cited 1×
“Further, recent studies such as ANGIE [205], Multimodal-GPT[206], and Large Multimodal Models [207] have integrated either vision and LLMs or video and audio [208] for training multimodal dialogue systems.”
Background
Resurrecting Recurrent Neural Networks for Long Sequences
Orvieto, A., Smith, S. L., Gu, A., Fernando, A., Gulcehre, C., Pascanu, R., & De, S. (2023) · arXiv · doi:10.48550/arxiv.2303.06349 · Cited by 76
End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-character Recognition Model
Stoller, D., Durand, S., & Ewert, S. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683470 · Cited by 76 · cited 1×
“Leveraging the MTG-Jamendo dataset with 55,000 audio songs in various languages, the model requires no training and undergoes direct testing on multiple datasets, including Jamendo [164], Hansen [165], MUSDB [166], and DSing [167].”
Method
End-to-End Speech Translation with the Transformer
Vila, L., Escolano, C., Fonollosa, J. A. R., et al. (2018) · IberSPEECH Conference · doi:10.21437/iberspeech.2018-13 · Cited by 76
Neural Approaches to Conversational AI
Gao, J., Galley, M., & Li, L. (2018) · Proceedings of ACL 2018, Tutorial Abstracts · doi:10.18653/v1/p18-5002 · Cited by 73 · cited 1×
“While text-based dialogue systems can be trained directly on extensive text data [195, 196], a large number of SDSs have relied on user simulations for training due to the scarcity of real training dialogues available for both training and evaluation purposes [197].”
Background
Dialogue TransformersPreprint
Vlasov, V., Mosig, J. E. M., & Nichol, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.00486 · Cited by 72 · cited 1×
“However, transformers have recently emerged as a superior alternative to RNNs to optimise the action selection process within SDSs [192, 193].”
Method
Music Deep Learning: Deep Learning Methods for Music Signal Processing—A Review of the State-of-the-Art
Moysis, L., Iliadis, L. A., Sotiroudis, S. P., Boursianis, A. D., Papadopoulou, M. S., Kokkinidis, K.-I. D., VOLOS, C. K., Sarigiannidis, P. G., Nikolaidis, S., & Goudos, S. K. (2023) · IEEE Access · doi:10.1109/access.2023.3244620 · Cited by 71 · cited 1×
“While several comprehensive surveys delve into the applications of deep learning for audio processing [11, 28, 53, 54], including speech [25, 26], music [55–57], and other categories [58, 59], none concentrate on the advent and deployment of LLMs in this field.”
Background
MultiModal-GPT: a vision and language model for dialogue with humans
Gong, T., Lyu, C., Zhang, S., Wang, Y., Zheng, M., Zhao, Q., Liu, K., Zhang, W., Luo, P., & Chen, K. (2023) · arXiv · doi:10.48550/arxiv.2305.04790 · Cited by 70 · cited 1×
“Further, recent studies such as ANGIE [205], Multimodal-GPT[206], and Large Multimodal Models [207] have integrated either vision and LLMs or video and audio [208] for training multimodal dialogue systems.”
Background
Aligning Large Language Models with Human: A Survey
Wang, Y., Zhong, W., Li, L., Mi, F., Zeng, X., Huang, W., Shang, L., Jiang, X., & Liu, Q. (2023) · arXiv · doi:10.48550/arxiv.2307.12966 · Cited by 69 · cited 1×
“[31] 2023 ✓ ✗ General First survey to provide an up-to-date review on the alignment process of LLMs.”
Background
LLaSM: Large Language and Speech ModelPreprint
Shu, Y., Dong, S., Chen, G., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2308.15930 · Cited by 69 · cited 1×
“Another recent work [249] propose a Large Language and Speech Model (LLaSM) that is an end-to-end trained large multimodal speech-language model with cross-modal con-18 ✓ TANGO [218] AudioCaps TTM Cannot always perform when trained on a smaller dataset ✗ WavJourney [127] AudioCaps TTM Inflexible to…”
Background
Pengi: An Audio Language Model for Audio TasksPreprint
Deshmukh, S., Elizalde, B., Singh, R., & Wang, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.11834 · Cited by 66 · cited 1×
“In a similar vein, Deshmukh et al. [123] present an innovative audio language model, Pengi, which approaches audio tasks as text generation tasks through the implementation of Transfer Learning.”
Background
Foundational Models Defining a New Era in Vision: A Survey and OutlookPreprint
Awais, M., Naseer, M., Khan, S. A., Anwer, R. M., Cholakkal, H., Shah, M. A., Yang, M., & Khan, F. S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.13721 · Cited by 66 · cited 4×
“…immense potential, these expansive AI models signify a new technological wave that promises a rich ecosystem of real-world applications and have already found extensive applications in various sectors such as vision [23, 24], language, health, education, robotics, and governance, among others.”
BackgroundInfluential
Trapping LLM Hallucinations Using Tagged Context PromptsPreprint
Feldman, P. G., Foulds, J., & Pan, S.-M. (2023) · arXiv.org · doi:10.48550/arxiv.2306.06085 · Cited by 63
ATISS: Autoregressive Transformers for Indoor Scene Synthesis
Paschalidou, D., Kar, A., Shugrina, M., Kreis, K., Geiger, A., & Fidler, S. (2021) · arXiv · doi:10.48550/arxiv.2110.03675 · Cited by 62
Large language models as tax attorneys: a case study in legal capabilities emergence
Nay, J. J., Karamardian, D., Lawsky, S. B., Tao, W., Bhat, M., Jain, R., Lee, A. T., Choi, J. H., & Kasai, J. (2024) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2023.0159 · Cited by 62 · cited 1×
“[46] 2023 ✓ ✗ Law Examines LLM’s proficiency in tax law application, noting improvements in newer models compared to older ones.”
Background
AI-Based human audio processing for COVID-19: A comprehensive overview
Deshpande, G., Batliner, A. M., & Schuller, B. W. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108289 · Cited by 62
Self-citation
Foundation Models for Decision Making: Problems, Methods, and Opportunities
Yang, S., Nachum, O., Du, Y., Wei, J., Abbeel, P., & Schuurmans, D. (2023) · arXiv · doi:10.48550/arxiv.2303.04129 · Cited by 60 · cited 2×
“[47] 2023 ✓ ✗ Robotics Explore applications of foundation models in practical decision-making using prompting, generative modeling, planning, and reinforcement learning.”
Background
AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and RisksPreprint
Tornede, A., Deng, D., Eimer, T., Giovanelli, J., Mohan, A., Ruhkopf, T., Segel, S., Theodorakopoulos, D., Tornede, T., Wachsmuth, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.08107 · Cited by 59 · cited 1×
“On a similar vein, Tornede et al. [61] explore LLMs in the context of Automated Machine Learning (AutoML) techniques, discussing existing methodologies and the challenges of using them to enhance LLM performance.”
Background
Cramming: Training a Language Model on a Single GPU in One DayPreprint
Geiping, J., & Goldstein, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.14034 · Cited by 57 · cited 1×
“These techniques are efficient attention mechanisms [277– 280], length generalisation [178, 281–284], and transformer alternatives [285–288].”
Background
VampNet: Music Generation via Masked Acoustic Token ModelingPreprint
García, H. F., Seetharaman, P., Kumar, R., & Pardo, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.04686 · Cited by 57 · cited 1×
“For instance, Garcia et al.[213] proposed a novel method known as VAMPNET.”
Method
On Efficient Training of Large-Scale Deep Learning Models: A Literature ReviewPreprint
Shen, L., Sun, Y., Yu, Z., Ding, L., Tian, X.-M., & Tao, D.-C. (2023) · arXiv.org · doi:10.48550/arxiv.2304.03589 · Cited by 57 · cited 1×
“The downstream performance of Large Audio Models is intricately linked to the diversity present in the training and pre-training datasets [259].”
Background
Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical StudyPreprint
Liu, P., Liu, Z., Gao, Z.-F., et al. (2023) · International Conference on Language Resources and Evaluation · doi:10.48550/arxiv.2307.08072 · Cited by 55 · cited 1×
“These large models, characterised by their billions of parameters and training on massive datasets, have manifested emergent abilities to tackle a multitude of intricate tasks across various fields [16–18].”
Background
Z-Code++: A Pre-trained Language Model Optimized for Abstractive SummarizationPreprint
He, P., Peng, B., Lu, L., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2208.09770 · Cited by 55 · cited 1×
“For the encoder-decoder-based LLM, the Z-Code++ model [158] serves as the text encoder and decoder.”
Method
VoiceFixer: A Unified Framework for High-Fidelity Speech Restoration
Liu, H., Liu, X., Kong, Q., Tian, Q., Zhao, Y., Wang, D., Huang, C., & Wang, Y. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-11026 · Cited by 54 · cited 1×
“invoked for text-to-speech synthesis, while VoiceFixer [227] is applied to refine the resultant speech quality.”
Method
The Science of Detecting LLM-Generated TextsPreprint
Tang, R., Chuang, Y.-N., & Hu, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.07205 · Cited by 53
POP909: A Pop-song Dataset for Music Arrangement GenerationPreprint
Wang, Z., Chen, K., Jiang, J., Zhang, Y., Xu, M., Dai, S., Gu, X., & Xia, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.07142 · Cited by 52
Conversational interfaces: advances and challenges
Zue, V. W. (1997) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1997-2 · Cited by 51 · cited 1×
“These systems encompass key components such as speech recognition, intent recognition, a knowledge base and/or database backend, a dialogue manager, language generation, and speech synthesis [191].”
Background
Real-Time Speech Frequency Bandwidth Extension
Li, Y., Tagliasacchi, M., Rybakov, O., Ungureanu, V., & Roblek, D. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413439 · Cited by 50 · cited 1×
“Firstly, an auto-encoding method [101, 105] is employed to comprehend the discrete representation of raw, unprocessed audio.”
Method
Protein Language Models and Structure Prediction: Connection and ProgressionPreprint
Hu, B.-Z., Xia, J.-X., Zheng, J.-B., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2211.16742 · Cited by 50 · cited 1×
“biology [42, 43], computer programming [64, 65], law [44– 46, 66], or robotics [47, 67, 68] among others.”
Background
Noise2Music: Text-conditioned Music Generation with Diffusion Models
Huang, Q., Park, D. S., Wang, T., Denk, T. I., Ly, A., Chen, N., Zhang, Z., Zhang, Z., Yu, J., Frank, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.03917 · Cited by 49 · cited 1×
“Drawing a parallel, Huang et al. [228] demonstrated that utilising LLMs to craft descriptive musical sentences can enhance the synthesis of text-conditioned music when utlised with a diffusion model.”
Method
MuLan: A Joint Embedding of Music Audio and Natural Language
Huang, Q., Jansen, A., Lee, J., Ganti, R., Li, J. Y., & Ellis, D. P. W. (2022) · arXiv · doi:10.48550/arxiv.2208.12415 · Cited by 49 · cited 2×
“It also uses MuLan [126], a joint music-text model, to address the main challenge of paired data scarcity.”
Method
WavJourney: Compositional Audio Creation with Large Language ModelsPreprint
Liu, X.-B., Zhu, Z., Liu, H., et al. (2023) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.48550/arxiv.2307.14335 · Cited by 48 · cited 5×
“…propose a Large Language and Speech Model (LLaSM) that is an end-to-end trained large multimodal speech-language model with cross-modal con-18 ✓ TANGO [218] AudioCaps TTM Cannot always perform when trained on a smaller dataset ✗ WavJourney [127] AudioCaps TTM Inflexible to expand the functions.”
BackgroundMethodResultInfluentialSelf-citation
Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language DataPreprint
Miranda, B., Lee, A., Sundar, S., Casasola, A., Schaeffer, R., Obbad, E., & Koyejo, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.13840 · Cited by 48 · cited 3×
“While a few studies [260, 261] on textual LLMs have outlined methods for weighting multi-domain data, this challenge remains unresolved in the context of speech and audio processing.”
Background
CoVoST: A Diverse Multilingual Speech-To-Text Translation CorpusPreprint
Wang, C., Pino, J., Wu, A., & Gu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.01320 · Cited by 47 · cited 1×
“AudioPaLM is trained using several datasets including CoVoS T2 [98], CVSS”
Method
Symphony Generation with Permutation Invariant Language ModelPreprint
Liu, J., Dong, Y., Cheng, Z., Zhang, X., Li, X., Yu, F., & Sun, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05448 · Cited by 46 · cited 1×
“To train attribute-to-music conversion, they used a variety of MIDI datasets, including Million MIDI Dataset [86], EMPOIA [140], MetaMidi [141], POP909 [147], Symphony [145], and an internal dataset named Emotion-gen.”
MethodSelf-citation
Listen, Think, and Understand
Gong, Y., Luo, H., Liu, A. H., Karlinsky, L., & Glass, J. (2023) · arXiv · doi:10.48550/arxiv.2305.10790 · Cited by 43 · cited 2×
“It is notable, however, that discussions around the hallucination challenge within Large Audio Models remain somewhat limited [108].”
BackgroundMethodInfluential
Can Large Language Models Aid in Annotating Speech Emotional Data? Uncovering New Frontiers [Research Frontier]
Latif, S., Usama, M., Malik, M. I., & Schuller, B. W. (2025) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2024.3504833 · Cited by 42 · cited 3×
“Coupling an LLM with the discrete representations learned by VQVAE provides a way for the LLM to indirectly model the continuous nature of the input audio signal [248].”
BackgroundMethodInfluentialSelf-citation
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated SignalsPreprint
Bajaj, P., Xiong, C., Ke, G., Liu, X., He, D., Tiwary, S., Liu, T., Bennett, P. N., Song, X., & Gao, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.06644 · Cited by 41
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. J. F. (2023) · arXiv · doi:10.48550/arxiv.2303.08896 · Cited by 38 · cited 1×
“To mitigate this issue, the literature suggests several strategies, including adversarial training, diversification in training data, human feedback incorporation, contrastive learning, and enhanced regularisation [88, 295–298].”
Background
PolyVoice: Language Models for Speech to Speech TranslationPreprint
Dong, Q., Huang, Z.-Y., Tian, Q., et al. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2306.02982 · Cited by 38 · cited 1×
“Dong et al. [187] introduced the innovative Poly Voice framework, which hinges upon a versatile language model (LM) proficient in speech-to-translation (S2ST) capabilities.”
Background
Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence
Nay, J. J., Karamardian, D., Lawsky, S. B., Tao, W., Bhat, M., Jain, R., Lee, A. T., Choi, J. H., & Kasai, J. (2023) · arXiv · doi:10.48550/arxiv.2306.07075 · Cited by 38
Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition
Li, Y., Wu, Y., Li, J., & Liu, S. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389732 · Cited by 37 · cited 1×
“Although there exist efficient fine-tuning techniques like parameter efficient training, prompt-based efficient tuning, and memory efficient tuning for text processing [94, 271–273], comparable methods are limited for speech and audio processing [218, 274, 275].”
Method
A Generalist Dynamics Model for ControlPreprint
Schubert, I., Zhang, J., Bruce, J., Bechtle, S., Parisotto, E., Riedmiller, M., Springenberg, J. T., Byravan, A., Hasenclever, L., & Heess, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.10912 · Cited by 37 · cited 1×
“biology [42, 43], computer programming [64, 65], law [44– 46, 66], or robotics [47, 67, 68] among others.”
Background
Vision-Language Models for Vision Tasks: A SurveyPreprint
Zhang, J., Huang, J., Jin, S., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.00685 · Cited by 37
Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks
Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.10160 · Cited by 36 · cited 1×
“Jacovi et al. [251] proposed three strategies for mitigating data contamination issues in LLMs.”
Background
Deep Architecture Enhancing Robustness to Noise, Adversarial Attacks, and Cross-Corpus Setting for Speech Emotion Recognition
Latif, S., Rana, R. K., Khalifa, S., Jurdak, R., & Schuller, B. W. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3190 · Cited by 35 · cited 1×
“Even identical utterances can acquire distinct meanings when expressed with different emotions [289].”
BackgroundSelf-citation
LongNet: Scaling Transformers to 1, 000, 000, 000 Tokens
Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., Wang, W., Zheng, N., & Wei, F. (2023) · arXiv · doi:10.48550/arxiv.2307.02486 · Cited by 34
Self-citation
Speechlmscore: Evaluating Speech Generation Using Speech Language Model
Maiti, S., Peng, Y., Saeki, T., & Watanabe, S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095710 · Cited by 34 · cited 1×
“Maiti et al. [180] introduced an autonomous evaluation approach known as SpeechLMScore, aimed at assessing generated speech samples using speech-language models.”
Method
EmoMusicTV: Emotion-Conditioned Symbolic Music Generation With Hierarchical Transformer VAE
Ji, S., & Yang, X. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3276177 · Cited by 34 · cited 1×
“This transformation owes much to the self-attention mechanism within transformers, which incorporates a global context during the music composition process, resulting in outputs that are more coherent and sophisticated [212].”
Background
Speech Recognition by Simply Fine-Tuning Bert
Huang, W.-C., Wu, C.-H., Luo, S.-B., Chen, K., Wang, H., & Toda, T. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413668 · Cited by 33 · cited 1×
“Although there exist efficient fine-tuning techniques like parameter efficient training, prompt-based efficient tuning, and memory efficient tuning for text processing [94, 271–273], comparable methods are limited for speech and audio processing [218, 274, 275].”
Method
Automatic Lyric Transcription from Karaoke Vocal Tracks: Resources and a Baseline System
Dabike, G. R., & Barker, J. (2019) · Interspeech · doi:10.21437/interspeech.2019-2378 · Cited by 32 · cited 1×
“Leveraging the MTG-Jamendo dataset with 55,000 audio songs in various languages, the model requires no training and undergoes direct testing on multiple datasets, including Jamendo [164], Hansen [165], MUSDB [166], and DSing [167].”
Method
Survey of Protein Sequence Embedding Models
Tran, C. T. T., Khadkikar, S., & Porollo, A. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24043775 · Cited by 31 · cited 1×
“biology [42, 43], computer programming [64, 65], law [44– 46, 66], or robotics [47, 67, 68] among others.”
Background
BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model
Higuchi, Y., Yan, B., Arora, S., et al. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.16663 · Cited by 31
Recognition Of Phonemes In A-Cappella Recordings Using Temporal Patterns And Mel Frequency Cepstral Coefficients
Hansen, J. (2012) · doi:10.5281/zenodo.850136 · Cited by 30 · cited 1×
“Leveraging the MTG-Jamendo dataset with 55,000 audio songs in various languages, the model requires no training and undergoes direct testing on multiple datasets, including Jamendo [164], Hansen [165], MUSDB [166], and DSing [167].”
Method
PRISMA-DFLLM: An Extension of PRISMA for Systematic Literature Reviews using Domain-specific Finetuned Large Language Models
Sušnjak, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.14905 · Cited by 30
Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulators to Enhance Dialogue System
Hu, Z., Feng, Y., Tuan, L. A., Hooi, B., & Lipani, A. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615220 · Cited by 29 · cited 1×
“…task-oriented dialogue systems has shown to underperform in comparison with task-specific models, careful application is required for LLMs to be useful—as shown by [198] in automated scoring of user simulations, [199] in dialogue state tracking, and [200] in data collection via prompt engineering.”
Background
AI-Based Emotion Recognition: Promise, Peril, and Prescriptions for Prosocial PathPreprint
Latif, S., Ali, H. S., Usama, M., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2211.07290 · Cited by 28 · cited 1×
“As the foundational research in this area grows, the insights presented here, along with related works, can serve as a valuable guide to understanding the complexities and addressing the challenges in the development of improved Large Audio Models for audio applications [23, 30, 299, 300].”
BackgroundSelf-citation
Hearing aids: an overview of the state-of-the-art, challenges, and future trends of an interesting audio signal processing application
Puder, H. (2009) · 2009 Proceedings of 6th International Symposium on Image and Signal Processing and Analysis · doi:10.1109/ispa.2009.5297793 · Cited by 28
Improving Non-Autoregressive End-to-End Speech Recognition with Pre-Trained Acoustic and Language Models
Deng, K., Yang, Z., Watanabe, S., Higuchi, Y., Cheng, G., & Zhang, P. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746316 · Cited by 26 · cited 1×
“Recently, language models have shown their power in capturing high-level, long-term patterns across different data types including text [21, 96] and image [149, 150], and speech [151–153].”
Background
Phoneme Level Lyrics Alignment and Text-Informed Singing Voice Separation
Schulze-Forster, K., Doire, C. S. J., Gaël Richard, A., & Badeau, R. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3091817 · Cited by 25 · cited 1×
“Leveraging the MTG-Jamendo dataset with 55,000 audio songs in various languages, the model requires no training and undergoes direct testing on multiple datasets, including Jamendo [164], Hansen [165], MUSDB [166], and DSing [167].”
Method
Exploring the Efficacy of Pre-trained Checkpoints in Text-to-Music Generation TaskPreprint
Wu, S., & Sun, M. (2022) · arXiv.org · doi:10.48550/arxiv.2211.11216 · Cited by 25 · cited 2×
“Wu et al. [224] devised a transformer-based model adept at producing music from textual descriptions.”
BackgroundMethod
Are LLMs All You Need for Task-Oriented Dialogue?Preprint
Hudecek, V., & Dusek, O. (2023) · arXiv.org · doi:10.48550/arxiv.2304.06556 · Cited by 25 · cited 1×
“…task-oriented dialogue systems has shown to underperform in comparison with task-specific models, careful application is required for LLMs to be useful—as shown by [198] in automated scoring of user simulations, [199] in dialogue state tracking, and [200] in data collection via prompt engineering.”
Background
LM-VC: Zero-Shot Voice Conversion via Speech Generation Based on Language Models
Wang, Z., Chen, Y., Xie, L., Tian, Q., & Wang, Y. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3308474 · Cited by 24 · cited 1×
“Wang et al. [182] presented an LM-based approach named LM-VC for zero-shot voice transformation.”
Method
SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model
Shih, Y.-J., Wang, H.-F., Chang, H.-J., Berry, L., Lee, H.-Y., & Harwath, D. F. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022954 · Cited by 24
Semi-Supervised Sound Event Detection with Pre-Trained Model
Xu, L., Wang, L., Bi, S., Liu, H., & Wang, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095687 · Cited by 23 · cited 1×
“For instance, Xu et al. [246] improved SED performance on the DESED public evaluation dataset [247] by more than 8.5% by utilising features from PANNs compared to the baseline.”
MethodInfluential
Knowledge Transfer from Large-Scale Pretrained Language Models to End-To-End Speech Recognizers
Kubo, Y., Karita, S., & Bacchiani, M. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746801 · Cited by 22 · cited 1×
“Kubo et al. [156] present a strategy to tackle this challenge through knowledge transfer from a neural network language model, initially pre-trained on text-only data.”
Background
Large Multimodal Models: Notes on CVPR 2023 TutorialPreprint
Li, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.14895 · Cited by 22 · cited 1×
“Further, recent studies such as ANGIE [205], Multimodal-GPT[206], and Large Multimodal Models [207] have integrated either vision and LLMs or video and audio [208] for training multimodal dialogue systems.”
Background
ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text TranslationPreprint
Le, C., Qian, Y., Zhou, L., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.14838 · Cited by 21 · cited 1×
“Le et al. [189] introduce ComSL, a novel speech-language model formulated through a composite architecture that harnesses the power of pre-trained speech and language models.”
Background
Creating DALI, a Large Dataset of Synchronized Audio, Lyrics, and Notes
Meseguer-Brocal, G., Cohen-Hadria, A., & Peeters, G. G. (2020) · Transactions of the International Society for Music Information Retrieval · doi:10.5334/tismir.30 · Cited by 20 · cited 1×
“For training, they employed the DALI v2 dataset [142] that aligns the lyric text with the melody and supplemented it with a text-only corpus from Kaggle [231].”
Method
Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognition
Ling, S., Hu, Y., Qian, S., Ye, G., Qian, Y., Gong, Y., Lin, E., & Zeng, M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448204 · Cited by 19 · cited 1×
“Ling et al. [157] explore a methodology involving the use of pre-trained LLM for fully formatted End-to-End (E2E) ASR transcriptions.”
Method
The Power of Fragmentation: A Hierarchical Transformer Model for Structural Segmentation in Symbolic Music Generation
Wu, G., Liu, S., & Fan, X. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3263797 · Cited by 18
Sentence-Select: Large-Scale Language Model Data Selection for Rare-Word Speech RecognitionPreprint
Huang, W. R., Peyser, C., Sainath, T., et al. (2022) · Interspeech · doi:10.48550/arxiv.2203.05008 · Cited by 17 · cited 1×
“In a different study, Huang [161] introduces strategies for curating language modelling data to enhance the recognition of rare words without compromising overall performance.”
Background
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.13048 · Cited by 17
SingSong: Generating musical accompaniments from singingPreprint
Donahue, C., Caillon, A., Roberts, A., Manilow, E., Esling, P., Agostinelli, A., Verzetti, M., Simon, I., Pietquin, O., Zeghidour, N., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.12662 · Cited by 15 · cited 3×
“Donahue et al. [229] introduced SingSong, a novel method for generating instrumental music tailored to complement specific vocal inputs.”
BackgroundMethodInfluential
SpeechGen: Unlocking the Generative Power of Speech Language Models with PromptsPreprint
Wu, H., Chang, K.-W., Wu, Y.-K., & Lee, H.-Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.02207 · Cited by 13 · cited 1×
“Wu et al. [190] conducted pioneering research that explores the application of prompt tuning to enhance speech-language models for a wide array of generation tasks.”
Background
Large Language Models in Cryptocurrency Securities Cases: Can ChatGPT Replace Lawyers?Preprint
Trozze, A., Davies, T. P., & Kleinberg, B. (2023) · arXiv.org · doi:10.48550/arxiv.2308.06032 · Cited by 13
Analyzing Leakage of Personally Identifiable Information in Language ModelsPreprint
Lukas, N., Salem, A., Sim, R. B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.00539 · Cited by 13
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language ModelsPreprint
Kaddour, J., Key, O., Nawrot, P., Minervini, P., & Kusner, M. J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.06440 · Cited by 12
LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPTPreprint
Zhuo, L., Yuan, R., Pan, J., Ma, Y., Li, Y., Zhang, G., Liu, S., Dannenberg, R. B., Fu, J., Lin, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.17103 · Cited by 12 · cited 1×
“Zhuo et al. [163] introduce LyricWhiz, a multilingual Automatic Lyrics Transcription (ALT) method designed for zero-shot scenarios across diverse lyrics transcription datasets, including unique genres like rock and metal.”
Method
BECTRA: Transducer-Based End-To-End ASR with Bert-Enhanced Encoder
Higuchi, Y., Ogawa, T., Kobayashi, T., & Watanabe, S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095186 · Cited by 12 · cited 1×
“Recently, language models have shown their power in capturing high-level, long-term patterns across different data types including text [21, 96] and image [149, 150], and speech [151–153].”
Background
Man vs the machine: The Struggle for Effective Text Anonymisation in the Age of Large Language ModelsPreprint
Patsakis, C., & Lykousas, N. (2023) · arXiv.org · doi:10.48550/arxiv.2303.12429 · Cited by 12 · cited 1×
“More than a few incidents are reported in the literature where LLMs have outputted the personal information in servicing particular prompts [252–256].”
Background
Low-Resource Speech Recognition and Keyword-Spotting
Gales, M., Knill, K., & Ragni, A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66429-3_1 · Cited by 10 · cited 1×
“To build PaLM MT TTS, they employed PALM-2 for translating YouTube, CommonVoice, and Babel [188].”
Method
Assessing Phrase Break of ESL Speech with Pre-trained Language Models and Large Language ModelsPreprint
Wang, Z., Mao, S., Wu, W., et al. (2023) · Interspeech · doi:10.48550/arxiv.2306.04980 · Cited by 10 · cited 1×
“2. https://github.com/facebookresearch/fairseq Wang [183] proposed a method to assess phrase breaks utilising pre-trained language models and LLMs.”
Method
Mu$^{2}$SLAM: Multitask, Multilingual Speech and Language ModelsPreprint
Cheng, Y., Zhang, Y., Johnson, M., Macherey, W., & Bapna, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.09553 · Cited by 10
Audio Embedding-Aware Dialogue Policy Learning
Zorrilla, A. L., Torres, M. I., & Cuayáhuitl, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3225658 · Cited by 8 · cited 1×
“A few exceptions using reasonably Large Audio Models include dialogue generation from raw audio excluding text processing in their pipeline [201], dialogue policy learning from textual and audio features for task-oriented dialogues [202], and open and closed-domain dialogue generation [203].”
MethodSelf-citation
A Survey of Dialogue System Evaluation
Fan, Y., & Luo, X. (2020) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai50040.2020.00182 · Cited by 7 · cited 1×
“While text-based dialogue systems can be trained directly on extensive text data [195, 196], a large number of SDSs have relied on user simulations for training due to the scarcity of real training dialogues available for both training and evaluation purposes [197].”
Background
Investigating the Utility of Surprisal from Large Language Models for Speech Synthesis ProsodyPreprint
Kakouros, S., Šimko, J., Vainio, M., et al. (2023) · Speech Synthesis Workshop · doi:10.48550/arxiv.2306.09814 · Cited by 7 · cited 1×
“For example, Kakouros et al. [174] explore the concept of word surprisal as a potential factor enhancing prosody in speech synthesis.”
Background
Advances in Speech and Music Technology
(2021) · Advances in Intelligent Systems and Computing · doi:10.1007/978-981-33-6881-1 · Cited by 7
Environmental sound processing and its applications
Miyazaki, K., Toda, T., Hayashi, T., & Takeda, K. (2019) · IEEJ Transactions on Electrical and Electronic Engineering · doi:10.1002/tee.22868 · Cited by 6 · cited 1×
“While several comprehensive surveys delve into the applications of deep learning for audio processing [11, 28, 53, 54], including speech [25, 26], music [55–57], and other categories [58, 59], none concentrate on the advent and deployment of LLMs in this field.”
Background
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and InterpretationPreprint
Wang, C., Rivière, M., Lee, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00390 · Cited by 6
Can Large Language Models Aid in Annotating Speech Emotional Data? Uncovering New FrontiersPreprint
Latif, S., Usama, M., Malik, M. I., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.06090 · Cited by 6
Self-citation
LaunchpadGPT: Language Model as Music Visualization Designer on LaunchpadPreprint
Xu, S., Tang, Y., & Zheng, F. (2023) · arXiv.org · doi:10.48550/arxiv.2307.04827 · Cited by 5 · cited 4×
“8: A prompt-completion example for Launch-PadGPT [236]: The text following ”prompt:” represents MFCC feature values, while ”completion:” shows RGB-X tuples.”
BackgroundInfluential
AudioLDM 2: Learning Holistic Audio Generation with Self-supervised PretrainingPreprint
Liu, H., Yuan, Y. G., Liu, X., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.05734 · Cited by 5
Diverse and Vivid Sound Generation from Text Descriptions
Li, G., Xu, X., Dai, L., Wu, M., & Yu, K. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096643 · Cited by 4 · cited 1×
“…have made significant strides in a variety of audio processing tasks, characterised by techniques that adeptly integrate audio data representations with traditional text token embeddings, equipping these large models with the capacity to interpret and manage a wide range of audio content [50–52].”
Background
LOAF-M2L: Joint Learning of Wording and Formatting for Singable Melody-to-Lyric GenerationPreprint
Ou, L.-S., Ma, X., & Wang, Y. (2023) · arXiv.org · doi:10.48550/arxiv.2307.02146 · Cited by 4 · cited 1×
“Ou et al. [230] addressed the challenge of singability in generating lyrics.”
Background
Semantic Segmentation with Bidirectional Language Models Improves Long-form ASRPreprint
Huang, W. R., Zhang, H., Kumar, S., et al. (2023) · Interspeech · doi:10.48550/arxiv.2305.18419 · Cited by 4 · cited 1×
“To tackle this issue, Huang et al. [160] devised a strategy to extract punctuation insights from a bidirectional teacher language model (LM) trained on written and punctuated text.”
Background
From Words to Music: A Study of Subword Tokenization Techniques in Symbolic Music GenerationPreprint
Kumar, A., & Sarmento, P. (2023) · arXiv.org · doi:10.48550/arxiv.2304.08953 · Cited by 4 · cited 1×
“By comprehending long-range dependencies, harmonies, and subtleties, transformers have transformed the landscape of music generation [210, 211].”
Background
FurChat: An Embodied Conversational Agent using LLMs, Combining Open and Closed-Domain Dialogue with Facial ExpressionsPreprint
Cherakara, N., Varghese, F., Shabana, S., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.15214 · Cited by 4
Generative Spoken Dialogue Language ModelingPreprint
Nguyen, T. A., Kharitonov, E., Copet, J., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.16502 · Cited by 3
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal SupervisionPreprint
Kharitonov, E., Vincent, D., Borsos, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.03540 · Cited by 3
On decoder-only architecture for speech-to-text and large language model integrationPreprint
Wu, J., Gaur, Y., Chen, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.03917 · Cited by 3
SpeechX: Neural Codec Language Model as a Versatile Speech TransformerPreprint
Wang, X., Thakker, M., Chen, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.06873 · Cited by 3
MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup StrategiesPreprint
Chen, K., Wu, Y., Liu, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01546 · Cited by 3
Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech RecognitionPreprint
Li, Y., Wu, Y., Li, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.16007 · Cited by 2
Prompting Large Language Models with Speech Recognition AbilitiesPreprint
Fathullah, Y., Wu, C., Lakomkin, E., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.11795 · Cited by 2
Dynalogue: A Transformer-Based Dialogue System with Dynamic Attention
Zhang, R., Wu, T., Chen, X., Wen, S., Nepal‬, ‪., Paris, C. L., & Xiang, Y. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583330 · Cited by 1 · cited 1×
“By leveraging their self-attention mechanism, transformers have demonstrated exceptional capabilities in modelling dynamic dialogue system scenarios [194].”
Background
Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech RecognitionPreprint
Ling, S., Hu, Y., Qian, S., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.08234 · Cited by 1
LM-VC: Zero-shot Voice Conversion via Speech Generation based on Language ModelsPreprint
Wang, Z., Chen, Y., Xie, L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.10521
GPT-4 Technical Report
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 1×
“In particular, the emergence of models such as ChatGPT and GPT-4 has rekindled discussions about the potential of artificial general intelligence [19, 20].”
Method
wav2vec: Unsupervised Pre-training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1904.05862
Self-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · arXiv · doi:10.48550/arxiv.1803.02155
MLS: A Large-Scale Multilingual Dataset for Speech Research
Pratap, V., Xu, Q., Sriram, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.03411
Large AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P. W., Xiao, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.11568
GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio
Chen, G., Chai, S., Wang, G., Du, J., Zhang, W.-Q., Weng, C., Su, D., Povey, D., Trmal, J., Zhang, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.06909
Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue System
Hu, Z., Feng, Y., Luu, A. T., Hooi, B., & Lipani, A. (2023) · arXiv · doi:10.48550/arxiv.2306.09821
Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
Wei, J., Kim, S., Jung, H., & Kim, Y.-H. (2023) · arXiv · doi:10.48550/arxiv.2301.05843

Research works printed but not linked: 13

B. Gold, N. Morgan, and D. Ellis, Speech and audio signal processing: processing and perception of speech and music. John Wiley & Sons, 2011.
Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 436-444, 2015.
A. B. Cyphert, "A human being wrote this law review article: Gpt-3 and the practice of law," UC Davis L. Rev., vol. 55, p. 401, 2021.
Y. K. Dwivedi, N. Kshetri, L. Hughes, E. L. Slade, A. Jeyaraj, A. K. Kar, A. M. Baabdullah, A. Koohang, V. Raghavan, M. Ahuja et al., ""so what if chatgpt wrote it?" multidisciplinary perspectives on opportunities, challenges and implications of generative conversa- tional ai for research, practice and policy," International Journal of Information Management, vol. 71, p. 102642, 2023.
Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, "Effi- cient transformers: A survey," ACM Computing Surveys, vol. 55, no. 6, pp. 1-28, 2022.
K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu et al., "A survey on vision transformer," IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 1, pp. 87-110, 2022.
J. Ens and P. Pasquier, "Building the metamidi dataset: Linking symbolic and audio musical data." in ISMIR, 2021, pp. 182-188.
D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, "The mtg-jamendo dataset for automatic music tagging." ICML, 2019.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, "Wavenet: A generative model for raw audio," https://arxiv.org/abs/1609.03499, 2016. · arXiv
S. Maiti, Y. Peng, T. Saeki, and S. Watanabe, "Speechlm- score: Evaluating speech generation using speech lan- guage model," in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1-5.
S. Forsgren and H. Martiros, "Riffusion-stable dif- fusion for real-time music generation, 2022," URL https://riffusion. com/about, vol. 6.
A. Banerjee and V. Arora, "wav2tok: Deep sequence tokenizer for audio retrieval," in The Eleventh Interna- tional Conference on Learning Representations, 2022.

Sorted by citations

A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations
Cheng, H., Zhang, M., & Shi, Q. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3447085 · Cited by 687
“O VER the past several years, Deep Neural Networks ( DNNs ) have achieved conspicuous progress in various domains and applications, such as Computer Vision ( CV ) [1], [2], Natural Language Processing ( NLP ) [3], [4], Audio Signal Processing ( ASP ) [5], [6], and cross-modal applications [7], [8].”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349
“However, there is a gap in the development of large audio models specifically tailored for medical applications [103].”
Affective Computing and the Road to an Emotionally Intelligent Metaverse
Pervez, F., Shoukat, M., Usama, M., Sandhu, M. M., Latif, S., & Qadir, J. (2024) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2024.3389462 · Cited by 52
“These models such as GPT-4 [174] and SeamlessM4t [175] can enable seamless and instantaneous communication between users, transcending language barriers through natural language and speech processing and voice translation and recognition [176].”
BackgroundMethodSelf-citation
From screens to scenes: A survey of embodied AI in healthcare
Liu, Y., Cao, X., Chen, T., Jiang, Y., You, J., Wu, M., Wang, X., Feng, M., Jin, Y., & Chen, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103033 · Cited by 76
Social Dangers of Generative Artificial Intelligence: Review and Guidelines
Yang, A., & Yang, T. A. (2024) · Annual International Conference on Digital Government Research · doi:10.1145/3657054.3664243 · Cited by 10
“Other examples of GAI include DALL-E, which can create visual artwork based on prompts, and video and sound-editing GAI which can replicate the voices or like-nesses of individuals or groups if there is sufficient data to train a model [12., 13.].”
Background

See all 17

Authors: Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., & Schuller, B. W.
Affiliations: Queensland University of Technology (AU)
Mohamed bin Zayed University of Artificial Intelligence (AE)
Nanyang Technological University (SG)
University of Augsburg (DE)
ORCID: Latif, Siddique
Shoukat, Moazzam
Shamshad, Fahad
Usama, Muhammad
Cuayáhuitl, Heriberto
Cambria, Erik
Schuller, Björn W.
Venue: arXiv
Publisher: arXiv
Published: 2023
ISSN: 2331-8422 (ISSN-L)
DOI: 10.48550/arxiv.2308.12792
arXiv: 2308.12792
OpenAlex: W4386185396
Cited by: 17
References: 315
Citations by year: 2025: 6 · 2024: 10 · 2023: 1
Type: posted content
Language: en
Open access: green
arxiv.org
Licence: other-oa
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations: arXiv (Cornell University) (submittedVersion, other-oa)
arXiv (Cornell University)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Keywords: transformer models, speech recognition, text-to-speech, music generation, multilingual speech translation, audio signal processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Sparks of Large Audio Models: A Survey and Outlook

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks