The DOI matches its DataCite record; 28 of the 28 references it cites were found in a registry or an index.
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
Open accessAbstract
Large language models (LLMs) have exhibited remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition. Despite the recent success, current LLMs are not capable of processing complex audio information or conducting spoken conversations (like Siri or Alexa). In this work, we propose a multi-modal AI system named AudioGPT, which complements LLMs (i.e., ChatGPT) with 1) foundation models to process complex audio information and solve numerous understanding and generation tasks; and 2) the input/output interface (ASR, TTS) to support spoken dialogue. With an increasing demand to evaluate multi-modal LLMs of human intention understanding and cooperation with foundation models, we outline the principles and processes and test AudioGPT in terms of consistency, capability, and robustness. Experimental results demonstrate the capabilities of AudioGPT in solving AI tasks with speech, music, sound, and talking head understanding and generation in multi-round dialogues, which empower humans to create rich and diverse audio content with unprecedented ease. Our system is publicly available at \url{https://github.com/AIGC-Audio/AudioGPT}.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Other recent audio LLMs [11,12,13,14] primarily focus on only speech.”
From Joint Audio and Speech Understanding (2023) · cites this paper as [12]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Among them, Kaplan et al. (2020) studied the impact of scaling on the performance of deep learning models, showing the existence of power laws between the model and dataset sizes and the performance of the system.”
BackgroundTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983
Conv-TasNet: Surpassing Ideal Time–Frequency Magnitude Masking for Speech Separation
Luo, Y., & Mesgarani, N. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2915167 · Cited by 2,171
Robust Speech Recognition via Large-Scale Weak Supervision
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022) · arXiv · doi:10.48550/arxiv.2212.04356 · Cited by 1,953
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T.-Y. (2020) · arXiv · doi:10.48550/arxiv.2006.04558 · Cited by 580
Self-citationHuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1657 · Cited by 475
Diffsound: Discrete Diffusion Model for Text-to-Sound Generation
Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., & Yu, D. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3268730 · Cited by 284
Self-citationDiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism
Liu, J., Li, C., Ren, Y., Chen, F., & Zhao, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21350 · Cited by 266
Self-citationMake-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion ModelsPreprint
Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., Ye, Z., Liu, J., Yin, X., & Zhao, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.12661 · Cited by 206
Self-citationVisual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2303.04671 · Cited by 197
TF-GRIDNET: Making Time-Frequency Domain Models Great Again for Monaural Speaker Separation
Wang, Z.-Q., Cornell, S., Choi, S., Lee, Y., Kim, B.-Y., & Watanabe, S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094992 · Cited by 116
Self-citationSeparate What You Describe: Language-Queried Audio Source Separation
Liu, X., Liu, H., Kong, Q., Mei, X., Zhao, J., Huang, Q., Plumbley, M. D., & Wang, W. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10894 · Cited by 92
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Huang, R., Chen, F., Ren, Y., Liu, J., Cui, C., & Zhao, Z. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475437 · Cited by 76
Self-citationImitating Arbitrary Talking Style for Realistic Audio-Driven Talking Face Synthesis
Wu, H., Jia, J., Wang, H., Dou, Y., Duan, C., & Deng, Q. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475280 · Cited by 65
GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face SynthesisPreprint
Ye, Z., Jiang, Z. K., Ren, Y., Liu, J., He, J., & Zhao, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.13430 · Cited by 62
Self-citationVISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis
Zhang, Y., Cong, J., Xue, H., Xie, L., Zhu, P., & Bi, M. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747664 · Cited by 61
FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech SynthesisPreprint
Huang, R., Lam, M. W. Y., Wang, J., Su, D., Yu, D., Ren, Y., & Zhao, Z. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.09934 · Cited by 45
Self-citationNeural Synthesis of Binaural Speech From Mono Audio
Richard, A., Marković, D., Gebru, I. D., Krenn, S., Butler, G. A., la Torre, F. D., & Sheikh, Y. A. (2020) · International Conference on Learning Representations · Cited by 37
Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks
Dalmia, S., Yan, B., Raunak, V., Metze, F., & Watanabe, S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.151 · Cited by 31
Self-citationYe, Z., Wang, H., Yang, D., & Zou, Y.-X. (2021) · Workshop on Detection and Classification of Acoustic Scenes and Events · doi:10.48550/arxiv.2110.06100 · Cited by 31
Self-citationXin, Y., Yang, D., & Zou, Y. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10057 · Cited by 11
Self-citationVISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice SynthesisPreprint
Zhang, Y., Cong, J., Xue, H., Xie, L., Zhu, P., & Bi, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08813 · Cited by 10
Detect what you want: Target Sound DetectionPreprint
Yang, D., Wang, H., Zou, Y., Cui, F., & Wang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.10153 · Cited by 3
Self-citationSearchable Hidden Intermediates for End-to-End Models of Decomposable Sequence TasksPreprint
Dalmia, S., Yan, B., Raunak, V., Metze, F., & Watanabe, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.00573 · Cited by 1
Self-citation
Research works printed but not linked: 19
- Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023. · arXiv
- Baziotis, C., Haddow, B., and Birch, A. Language model prior for low-resource neural machine translation. arXiv preprint arXiv:2004.14928, 2020. · arXiv
- Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Teboul, O., Grangier, D., Tagliasacchi, M., and Zeghidour, N. Audiolm: a language modeling approach to audio generation. arXiv preprint arXiv:2209.03143, 2022. · arXiv
- Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022. · arXiv
- Grabocka, J., Schmidt Thieme, L., and etc. Neuralwarp: Time-series similarity with warping networks. arXiv preprint arXiv:1812.08306, 2018. · arXiv
- Hayashi, T. and Watanabe, S. Discretalk: Text-to-speech as a machine translation problem. arXiv preprint arXiv:2005.05525, 2020. · arXiv
- Hosseini-Asl, E., McCann, B., Wu, C.-S., Yavuz, S., and Socher, R. A simple language model for task-oriented dialogue. Advances in Neural Information Processing Systems, 33:20179-20191, 2020.
- Huang, R., Ren, Y., Liu, J., Cui, C., and Zhao, Z. Generspeech: Towards style transfer for generaliz- able out-of-domain text-to-speech synthesis. arXiv preprint arXiv:2205.07211, 2022b. · arXiv
- Huang, R., Zhao, Z., Liu, J., Liu, H., Ren, Y., Zhang, L., and He, J. Transpeech: Speech-to-speech translation with bilateral perturbation. arXiv preprint arXiv:2205.12523, 2022c. · arXiv
- Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Liu, Q., et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045, 2023b. · arXiv
- Kharitonov, E., Copet, J., Lakhotia, K., Nguyen, T. A., Tomasello, P., Lee, A., Elkahky, A., Hsu, W.-N., Mohamed, A., Dupoux, E., et al. textless-lib: A library for textless spoken language processing. arXiv preprint arXiv:2202.07359, 2022. · arXiv
- Liu, F., Li, G., Zhao, Y., and Jin, Z. Multi-task learning based pre-trained language model for code completion. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pp. 473-485, 2020.
- Nguyen, T. A., Kharitonov, E., Copet, J., Adi, Y., Hsu, W.-N., Elkahky, A., Tomasello, P., Algayres, R., Sagot, B., Mohamed, A., et al. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11:250-266, 2023.
- Svyatkovskiy, A., Zhao, Y., Fu, S., and Sundaresan, N. Pythia: Ai-assisted code completion system. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2727-2735, 2019.
- Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. · arXiv
- Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. · arXiv
- Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. · arXiv
- Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 495-507, 2021.
- Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022a. · arXiv
Cited by
Sorted by citations
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
Zhang, H., Li, X. S., & Bing, L. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.49 · Cited by 744
A survey on LLM-based multi-agent systems: Workflow, infrastructure, and challenges
Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024) · Vicinagearth · doi:10.1007/s44336-024-00009-2 · Cited by 404
Self-citationMultimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Metadata
- Authors
- Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z., Wu, Y., Hong, Z., Huang, J., Liu, J., Ren, Y., Zhao, Z., & Watanabe, S.
- Affiliations
- Zhejiang University (CN)
- Peking University (CN)
- Carnegie Mellon University (US)
- Renmin University of China (CN)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Volume
- 38
- Issue
- 21
- Pages
- 23802-23804
- ISSN
- 2159-5399 (ISSN-L)
- arXiv
- 2304.12995
- OpenAlex
- W4393178509
- Cited by
- 131
- References
- 28
- Citations by year
- 2026: 22 · 2025: 55 · 2024: 36 · 2023: 18
- FWCI
- 33.35
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal large language models, audio understanding, speech generation, music generation, sound production, talking head generation, spoken dialogue systems, speech recognition, text-to-speech, human-AI collaboration, multi-turn dialogue
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
