The DOI matches its DataCite record.
AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension
Open accessAbstract
Recently, instruction-following audio-language models have received broad attention for human-audio interaction. However, the absence of benchmarks capable of evaluating audio-centric interaction capabilities has impeded advancements in this field. Previous models primarily focus on assessing different fundamental tasks, such as Automatic Speech Recognition (ASR), and lack an assessment of the open-ended generative capabilities centered around audio. Thus, it is challenging to track the progression in the Large Audio-Language Models (LALMs) domain and to provide guidance for future improvement. In this paper, we introduce AIR-Bench (\textbf{A}udio \textbf{I}nst\textbf{R}uction \textbf{Bench}mark), the first benchmark designed to evaluate the ability of LALMs to understand various types of audio signals (including human speech, natural sounds, and music), and furthermore, to interact with humans in the textual format. AIR-Bench encompasses two dimensions: \textit{foundation} and \textit{chat} benchmarks. The former consists of 19 tasks with approximately 19k single-choice questions, intending to inspect the basic single-task ability of LALMs. The latter one contains 2k instances of open-ended question-and-answer data, directly assessing the comprehension of the model on complex audio and its capacity to follow instructions. Both benchmarks require the model to generate hypotheses directly. We design a unified framework that leverages advanced language models, such as GPT-4, to evaluate the scores of generated hypotheses given the meta-information of the audio. Experimental results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation. By revealing the limitations of existing LALMs through evaluation results, AIR-Bench can provide insights into the direction of future research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As shown in table 2, Qwen2-Audio demonstrates state-of-the-art (SOTA) instruction-following capabilities across speech, sound music and mixed-Audio subsets.”
From Qwen2-Audio Technical Report (2024) · cites this paper as Yang et al., 2024
In the citing paper
“We discovered that the scores from AIR-Bench align more closely with the actual user interaction experience.”
From Qwen2-Audio Technical Report (2024) · cites this paper as Yang et al., 2024
In the citing paper
“Among them, Qwen2-Audio achieves state-of-the-art performance on the test set of Aishell2, FLUERS-zh, VocalSound and AIR-Bench chat benchmark.”
From Qwen2-Audio Technical Report (2024) · cites this paper as Yang et al., 2024
In the citing paper
“According to the evaluation results from AIR-Bench, Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests focused on audio-centric instruction-following capabilities.”
From Qwen2-Audio Technical Report (2024) · cites this paper as Yang et al., 2024
Citations
Cites
Still being checked.
Cited by
Sorted by citations
VoiceBench: Benchmarking LLM-Based Voice Assistants
Chen, Y., Yue, X., Zhang, C., Gao, X., Tan, R. T., & Li, H. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.628 · Cited by 243
Recent Advances in Speech Language Models: A Survey
Cui, W., Yu, D., Jiao, X., Meng, Z., Zhang, G., Wang, Q., Guo, S. Y., & King, I. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.682 · Cited by 137
WavChat: A Survey of Spoken Dialogue ModelsPreprint
Ji, S., Chen, Y., Fang, M., Zuo, J., Lu, J., Wang, H., Jiang, Z., Zhou, L., Liu, S., Cheng, X., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.13577 · Cited by 124
Self-citationPerception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning ModelsPreprint
Li, Y., Liu, Z., Li, Z., Zhang, X., Xu, Z., Chen, X., Shi, H., Jiang, S., Wang, X., Wang, J., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.04921 · Cited by 120
Lin, G.-T., Lian, J., Li, T., Wang, Q., Anumanchipalli, G. K., Liu, A. H., & Lee, H.-Y. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11433838 · Cited by 115
Metadata
- Authors
- Yang, Q., Xu, J., Liu, W., Chu, Y., Jiang, Z., Zhou, X., Leng, Y., Lv, Y., Zhao, Z., Zhou, C., & Zhou, J.
- Affiliations
- Zhejiang University (CN)
- Alibaba Group (China) (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2402.07729
- OpenAlex
- W4391799385
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Keywords
- large audio language models, benchmarking, listening comprehension, instruction following, audio understanding, hypothesis generation, open-ended question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
