Audiolm: A language modeling approach to audio generation
Open accessAbstract
We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation space. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure, and we propose a hybrid tokenization scheme to achieve both objectives. Namely, we leverage the discretized activations of a masked language model pre-trained on audio to capture long-term structure and the discrete codes produced by a neural audio codec to achieve high-quality synthesis. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations given short prompts. When trained on speech, and without any transcript or annotation, AudioLM generates syntactically and semantically plausible speech continuations while also maintaining speaker identity and prosody for unseen speakers. Furthermore, we demonstrate how our approach extends beyond speech by generating coherent piano music continuations, despite being trained without any symbolic representation of music.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The dominant architectural paradigm connects specialized, external multimodal encoders-such as CLIP for vision or CLAP for audio-to the LLM backbone via alignment modules like Q-Former or simple MLPs, a modular design that allows for independent encoder updates without retraining the entire model.
“The dominant architectural paradigm connects specialized, external multimodal encoders-such as CLIP for vision or CLAP for audio-to the LLM backbone via alignment modules like Q-Former or simple MLPs [19, 86, 615, 1139], a modular design that allows for independent encoder updates without retraining the entire model [624].”
From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [86]
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · cited 2× · see citing works
“Positive pairs can be, for example, two segments that are close temporally [28], [29], [30] or two augmented views of the”
BackgroundMethod“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · see citing works
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · see citing works
Conformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3015 · Cited by 3,150 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
MelGAN: generative adversarial networks for conditional waveform synthesis
Kumar, K., Kumar, R., de Boissiere, T., Gestin, L., Teoh, W. Z., Sotelo, J., de Brebisson, A., Bengio, Y., & Courville, A. (2019) · arXiv · doi:10.48550/arxiv.1910.06711 · Cited by 722 · see citing works
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 1× · see citing works
“While several works have explored efficient alternatives to self-attention [39], [40], [41], another solution to this scaling problem is to work with mappings of the natural signals to a compact, discrete representation space.”
BackgroundChung, Y.-A., Zhang, Y., Han, W., Chiu, C.-C., Qin, J., Pang, R., & Wu, Y. (2021) · 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru51503.2021.9688253 · Cited by 293 · see citing works
Layer-Wise Analysis of a Self-Supervised Speech Representation Model
Pasad, A., Chou, J.-C., & Livescu, K. (2021) · 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru51503.2021.9688093 · Cited by 262 · see citing works
YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone
Casanova, E., Weber, J., Shulby, C., Junior, A. C., Gölge, E., & Ponti, M. A. (2021) · arXiv · doi:10.48550/arxiv.2112.02418 · Cited by 200 · see citing works
Flagged first, then by citations
Research works printed but not linked: 52
- N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. van den Oord, S. Dieleman, and K. Kavukcuoglu, "Efficient neural audio synthesis," in International Conference on Machine Learning (ICML), vol. 80. PMLR, 2018, pp. 2415-2424.
- J. Kong, J. Kim, and J. Bae, "Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis," in Advances in Neural Information Processing Systems (NeurIPS), 2020.
- M. Tagliasacchi, Y. Li, K. Misiunas, and D. Roblek, "SEANet: A multi- modal speech enhancement network," in Interspeech, 2020.
- Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, "DiffWave: A versatile diffusion model for audio synthesis," in International Conference on Learning Representations (ICLR), 2021.
- N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, and W. Chan, "WaveGrad: Estimating gradients for waveform generation," in Interna- tional Conference on Learning Representations (ICLR), 2021.
- T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, "Language models are few-shot learners," in Advances in Neural Information Processing Systems (NeurIPS), 2020.
- J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young et al., "Scaling lan- guage models: Methods, analysis & insights from training Gopher," arXiv:2112.11446, 2021. · arXiv
- J. Yu, X. Li, J. Y. Koh, H. Zhang, R. Pang, J. Qin, A. Ku, Y. Xu, J. Baldridge, and Y. Wu, "Vector-quantized image modeling with improved VQGAN," in International Conference on Learning Represen- tations (ICLR), 2022.
- J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan, B. Hutchinson, W. Han, Z. Parekh, X. Li, H. Zhang, J. Baldridge, and Y. Wu, "Scaling autoregressive models for content-rich text-to-image generation," Transactions on Machine Learning Research (TMLR), 2022.
- E. Dunbar, M. Bernard, N. Hamilakis, T. A. Nguyen, M. de Seyssel, P. Rozé, M. Rivière, E. Kharitonov, and E. Dupoux, "The zero resource speech challenge 2021: Spoken language modelling," in Interspeech. ISCA, 2021, pp. 1574-1578.
- K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed et al., "On generative spoken language modeling from raw audio," Transactions of the Association for Computational Linguistics, vol. 9, pp. 1336-1354, 2021.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 5998-6008.
- N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, "Soundstream: An end-to-end neural audio codec," IEEE ACM Trans. Audio Speech Lang. Process., vol. 30, pp. 495-507, 2022.
- A. Roberts, H. W. Chung, A. Levskaya, G. Mishra, J. Bradbury, D. Andor, S. Narang, B. Lester, C. Gaffney, A. Mohiuddin, C. Hawthorne, A. Lewkowycz, A. Salcianu, M. van Zee, J. Austin, S. Goodman, L. B. Soares, H. Hu, S. Tsvyashchenko, A. Chowdhery, J. Bastings, J. Bulian, X. Garcia, J. Ni, A. Chen, K. Kenealy, J. H. Clark, S. Lee, D. Garrette, J. Lee-Thorp, C. Raffel, N. Shazeer, M. Ritter, M. Bosma, A. Passos, J. Maitin-Shepard, N. Fiedel, M. Omernick, B. Saeta, R. Sepassi, A. Spiridonov, J. Newlan, and A. Gesmundo, "Scaling up models and data with t5x and seqio," arXiv:2203.17189, 2022. · arXiv
- A. van den Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. van den Driessche, E. Lockhart, L. C. Cobo, F. Stimberg, N. Casagrande, D. Grewe, S. Noury, S. Dieleman, E. Elsen, N. Kalchbrenner, H. Zen, A. Graves, H. King, T. Walters, D. Belov, and D. Hassabis, "Parallel WaveNet: Fast high-fidelity speech synthesis," in International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 80. PMLR, 2018, pp. 3915-3923.
- M. Binkowski, J. Donahue, S. Dieleman, A. Clark, E. Elsen, N. Casagrande, L. C. Cobo, and K. Simonyan, "High fidelity speech synthesis with adversarial networks," in International Conference on Learning Representations (ICLR), 2020.
- A. van den Oord, O. Vinyals, and K. Kavukcuoglu, "Neural discrete representation learning," in Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 6306-6315.
- A. Razavi, A. van den Oord, and O. Vinyals, "Generating diverse high- fidelity images with VQ-VAE-2," in Advances in Neural Information Processing Systems (NeurIPS), 2019, pp. 14 837-14 847.
- E. Agustsson, F. Mentzer, M. Tschannen, L. Cavigelli, R. Timofte, L. Benini, and L. V. Gool, "Soft-to-hard vector quantization for end-to- end learned compression of images and neural networks," Advances in Neural Information Processing Systems (NeurIPS), 2017.
- S. Kankanahalli, "End-to-end optimized speech coding with deep neural networks," in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 2521-2525.
- K. Zhen, J. Sung, M. S. Lee, S. Beack, and M. Kim, "Cascaded cross- module residual learning towards lightweight end-to-end speech coding," in Interspeech. ISCA, 2019, pp. 3396-3400.
- D. Petermann, S. Beack, and M. Kim, "HARP-Net: Hyper- autoencoded reconstruction propagation for scalable neural audio coding," arXiv:2107.10843, 2021. · arXiv
- A. Baevski, H. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations," arXiv:2006.11477, 2020. · arXiv
- A. Saeed, D. Grangier, and N. Zeghidour, "Contrastive learning of general- purpose audio representations," in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 3875-3879.
- E. Kharitonov, M. Rivière, G. Synnaeve, L. Wolf, P. Mazaré, M. Douze, and E. Dupoux, "Data augmenting contrastive learning of speech repre- sentations in the time domain," in IEEE Spoken Language Technology Workshop, SLT 2021, Shenzhen, China, January 19-22, 2021. IEEE, 2021, pp. 215-222.
- A. Baevski, S. Schneider, and M. Auli, "vq-wav2vec: Self-supervised learning of discrete speech representations," in International Conference on Learning Representations (ICLR), 2020.
- W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, "Hubert: Self-supervised speech representation learning by masked prediction of hidden units," IEEE ACM Trans. Audio Speech Lang. Process., vol. 29, pp. 3451-3460, 2021.
- R. Thoppilan, D. D. Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, Y. Zhou, C. Chang, I. Krivokon, W. Rusch, M. Pickett, K. S. Meier-Hellstern, M. R. Morris, T. Doshi, R. D. Santos, T. Duke, J. Soraker, B. Zevenbergen, V. Prab- hakaran, M. Diaz, B. Hutchinson, K. Olson, A. Molina, E. Hoffman-John, J. Lee, L. Aroyo, R. Rajakumar, A. Butryna, M. Lamm, V. Kuzmina, J. Fenton, A. Cohen, R. Bernstein, R. Kurzweil, B. Aguera-Arcas, C. Cui, M. Croak, E. H. Chi, and Q. Le, "LaMDA: Language models for dialog applications," arXiv:2201.08239, 2022. · arXiv
- G. Lample and F. Charton, "Deep learning for symbolic mathematics," in International Conference on Learning Representations (ICLR), 2020.
- A. Roy, M. Saffar, A. Vaswani, and D. Grangier, "Efficient content- based sparse attention with routing transformers," Trans. Assoc. Comput. Linguistics, vol. 9, pp. 53-68, 2021.
- C. Hawthorne, A. Jaegle, C. Cangea, S. Borgeaud, C. Nash, M. Mali- nowski, S. Dieleman, O. Vinyals, M. M. Botvinick, I. Simon, H. Sheahan, N. Zeghidour, J. Alayrac, J. Carreira, and J. H. Engel, "General-purpose, long-context autoregressive modeling with Perceiver AR," in International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 2022, pp. 8535-8558.
- P. Esser, R. Rombach, and B. Ommer, "Taming transformers for high- resolution image synthesis," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Computer Vision Foundation / IEEE, 2021, pp. 12 873-12 883.
- S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang, D. Jacobs, J. Huang, and D. Parikh, "Long video generation with time-agnostic VQGAN and time-sensitive transformer," arXiv:2204.03638, 2022. · arXiv
- P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, "Jukebox: A generative model for music," arXiv:2005.00341, 2020. · arXiv
- E. Kharitonov, A. Lee, A. Polyak, Y. Adi, J. Copet, K. Lakhotia, T. A. Nguyen, M. Rivière, A. Mohamed, E. Dupoux, and W. Hsu, "Text-free prosody-aware generative spoken language modeling," in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, 2022, pp. 8666-8681.
- E. Kharitonov, J. Copet, K. Lakhotia, T. A. Nguyen, P. Tomasello, A. Lee, A. Elkahky, W. Hsu, A. Mohamed, E. Dupoux, and Y. Adi, "textless-lib: a library for textless spoken language processing," arXiv:2202.07359, 2022. · arXiv
- A. Conneau, A. Bapna, Y. Zhang, M. Ma, P. von Platen, A. Lozhkov, C. Cherry, Y. Jia, C. Rivera, M. Kale, D. van Esch, V. Axelrod, S. Khanuja, J. H. Clark, O. Firat, M. Auli, S. Ruder, J. Riesa, and M. Johnson, "XTREME-S: evaluating cross-lingual speech representa- tions," arXiv:2203.10752, 2022. · arXiv
- A. Hines, J. Skoglund, A. C. Kokaram, and N. Harte, "ViSQOL: an objective speech quality model," EURASIP Journal on Audio, Speech, and Music Processing, vol. 2015, no. 1, pp. 1-18, 2015.
- M. Chinen, F. S. C. Lim, J. Skoglund, N. Gureev, F. O'Gorman, and A. Hines, "ViSQOL v3: an open source production ready objective speech and audio metric," in Twelfth International Conference on Quality of Multimedia Experience (QoMEX), 2020, pp. 1-6.
- T. Schatz, V. Peddinti, F. R. Bach, A. Jansen, H. Hermansky, and E. Dupoux, "Evaluating speech features with the minimal-pair ABX task: analysis of the classical MFC/PLP pipeline," in Interspeech. ISCA, 2013, pp. 1781-1785.
- T. Schatz, "Abx-discriminability measures and applications. (mesures de discriminabilité ABX et applications)," Ph.D. dissertation, Pierre and Marie Curie University, Paris, France, 2016.
- J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux, "Libri-light: A benchmark for ASR with limited or no supervision," in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7669-7673.
- M. Rivière and E. Dupoux, "Towards unsupervised learning of speech features in the wild," in IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 156-163.
- A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, "Conformer: Convolution-augmented transformer for speech recognition," in Interspeech. ISCA, 2020, pp. 5036-5040.
- W. Hsu, Y. H. Tsai, B. Bolte, R. Salakhutdinov, and A. Mohamed, "Hubert: How much can a bad teacher benefit ASR pre-training?" in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6533-6537.
- M. Tagliasacchi, B. Gfeller, F. de Chaumont Quitry, and D. Rob- lek, "Self-supervised audio representation learning for mobile devices," arXiv:1905.11796, 2019. · arXiv
- T. A. Nguyen, B. Sagot, and E. Dupoux, "Are discrete units necessary for spoken language modeling?" arXiv:2203.05936, 2022. · arXiv
- P. Peng and D. Harwath, "Self-supervised representation learning for speech using visual grounding and masked language modeling," arXiv:2202.03543, 2022. · arXiv
- Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, "Roberta: A robustly optimized BERT pretraining approach," arXiv:1907.11692, 2019. · arXiv
- B. van Niekerk, L. Nortje, M. Baas, and H. Kamper, "Analyzing speaker information in self-supervised models to improve zero-resource speech processing," in Interspeech. ISCA, 2021, pp. 1554-1558.
- C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C. A. Huang, S. Dieleman, E. Elsen, J. H. Engel, and D. Eck, "Enabling factorized piano music modeling and generation with the MAESTRO dataset," in International Conference on Learning Representations (ICLR), 2019.
- H. Delgado, N. W. D. Evans, T. Kinnunen, K. A. Lee, X. Liu, A. Nautsch, J. Patino, M. Sahidullah, M. Todisco, X. Wang, and J. Yamagishi, "Asvspoof 2021: Automatic speaker verification spoofing and countermeasures challenge evaluation plan," arXiv:2109.00535, 2021. · arXiv
Cited by
Sorted by citations
Bandi, A., Adapa, P. V. S. R., & Kuchi, Y. E. V. P. K. (2023) · Future Internet · doi:10.3390/fi15080260 · Cited by 638 · see citing works
A survey on LLM-based multi-agent systems: Workflow, infrastructure, and challenges
Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024) · Vicinagearth · doi:10.1007/s44336-024-00009-2 · Cited by 404 · see citing works
“Previous work, such as WavJourney [114] and AudioLM [115], has often employed audio encoders to convert audio into discrete tokens, which are then integrated into the LLM’s knowledge base.”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“Numerous audio-text M-LLMs, leveraging transformer-based architectures, have integrated text-and audio-based language models, such as AudioPaLM [97], AudioLM [98], Pengi [99], AudioGPT [100], SpeechGPT [49], VioLA [101], and SALMONN [102], into a unified multimodal architecture.”
A Survey of Generative AI Applications
Gozalo-Brizuela, R., & Merchan, E. E. G. (2024) · Journal of Computer Science · doi:10.3844/jcssp.2024.801.818 · Cited by 101 · see citing works
ChatGPT for Clinical Vignette Generation, Revision, and Evaluation
Benoit, J. R. A. (2023) · doi:10.1101/2023.02.04.23285478 · Cited by 74 · see citing works
Seamless: Multilingual Expressive and Streaming Speech Translation
Communication, S., Barrault, L., Chung, Y.-A., Meglioli, M. C., Dale, D., Dong, N., Duppenthaler, M., Duquenne, P.-A., Ellis, B., Elsahar, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.05187 · Cited by 45 · see citing works
Kamath, P., Morreale, F., Bagaskara, P. L., Wei, Y., & Nanayakkara, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642040 · Cited by 37 · see citing works
A Survey of Context Engineering for Large Language Models
Mei, L., Yao, J., Ge, Y., Wang, Y., Bi, B., Cai, Y., Liu, J., Li, M., Li, Z.-Z., Zhang, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.13334 · Cited by 13 · see citing works
Learning Interactive Real-World Simulators
Yang, S., Du, Y., Ghasemipour, K., Tompson, J., Kaelbling, L., Schuurmans, D., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2310.06114 · Cited by 12 · see citing works
“Generative models trained on internet data can now produce highly realistic text [1], speech [2], image [3], and video [4].”
BackgroundSparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17 · see citing works
“Although AudioLM [99] claims to preserve speaker identification and prosody, its exploration of emotions and other paralinguistic information remains unknown.”
BackgroundMethodInfluentialMehta, S., Deichler, A., O’Regan, J., Moëll, B., Beskow, J., Henter, G. E., & Alexanderson, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00201 · Cited by 7 · see citing works
SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
Shukor, M., Aubakirova, D., Capuano, F., Kooijmans, P., Palma, S., Zouitine, A., Aractingi, M., Pascal, C., Russi, M., Marafioti, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01844 · Cited by 3 · see citing works
“…other modalities, sparking interest towards multi-modal vision-language (VLMs) (Alayrac et al., 2022; Chen et al., 2023; Huang et al., 2023; Liu et al., 2023b; Chen et al., 2024; Shukor et al., 2023b) and audio-language models (ALMs) (Défossez et al., 2024; Das et al., 2024; Borsos et al., 2023).”
- Authors
- Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M., & Zeghidour, N.
- Affiliations
- Google (Switzerland) (CH)
- Google (United States) (US)
- ORCID
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2023
- Volume
- 31
- Pages
- 2523-2533
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W4381786045
- Cited by
- 432
- References
- 13
- Citations by year
- 2026: 89 · 2025: 150 · 2024: 133 · 2023: 58 · 2022: 2
- FWCI
- 40.53
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- audio generation, audio tokenization, neural audio codecs, masked language models, long-term consistency, speech synthesis, speaker identity preservation, prosody, audio-to-audio translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
