CoChat
Sign Up
PreprintCited by 4Multimodal Machine Learning ApplicationsMusic and Audio ProcessingSpeech Recognition and Synthesis

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix

Open access
Ma, Ziyang, Ma, Yinghao, Zhu, Yanqiao, Yang, Chen, Chao, Yi-Wen, Xu, Ruiyang, Chen, Wenxi, Chen, Yuanzhe, Chen, Zhuo, Cong, Jian, Li, Kai, Li, Keliang, Li, Siyou, Li, Xinfeng, Li, Xiquan, Lian, Zheng, Liang, Yuzhe, Liu, Minghao, Niu, Zhikang, Wang, Tianrui, Wang, Yuping, Wang, Yuxuan, Wu, Yihao, Yang, Guanrou, Yu, Jianwei, Yuan, Ruibin, Zheng, Zhisheng, Zhou, Ziya, Zhu, Haina, Xue, Wei, Benetos, Emmanouil, Yu, Kai, Chng, Eng-Siong, Chen, Xie·2025·arXiv
DOI 10.48550/arxiv.2505.13032arXiv 2505.13032
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce MMAR, a new benchmark designed to evaluate the deep reasoning capabilities of Audio-Language Models (ALMs) across massive multi-disciplinary tasks. MMAR comprises 1,000 meticulously curated audio-question-answer triplets, collected from real-world internet videos and refined through iterative error corrections and quality checks to ensure high quality. Unlike existing benchmarks that are limited to specific domains of sound, music, or speech, MMAR extends them to a broad spectrum of real-world audio scenarios, including mixed-modality combinations of sound, music, and speech. Each question in MMAR is hierarchically categorized across four reasoning layers: Signal, Perception, Semantic, and Cultural, with additional sub-categories within each layer to reflect task diversity and complexity. To further foster research in this area, we annotate every question with a Chain-of-Thought (CoT) rationale to promote future advancements in audio reasoning. Each item in the benchmark demands multi-step deep reasoning beyond surface-level understanding. Moreover, a part of the questions requires graduate-level perceptual and domain-specific knowledge, elevating the benchmark's difficulty and depth. We evaluate MMAR using a broad set of models, including Large Audio-Language Models (LALMs), Large Audio Reasoning Models (LARMs), Omni Language Models (OLMs), Large Language Models (LLMs), and Large Reasoning Models (LRMs), with audio caption inputs. The performance of these models on MMAR highlights the benchmark's challenging nature, and our analysis further reveals critical limitations of understanding and reasoning capabilities among current models. We hope MMAR will serve as a catalyst for future advances in this important but little-explored area.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.13032

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Building a unified audio-language model remains challenging because different tasks depend on different levels of acoustic abstraction, and many tasks are inherently temporal.

    “Building such a unified model remains challenging. Different tasks depend on different levels of acoustic abstraction: ASR requires fine-grained phonetic and lexical information, speech captioning relies on prosody and speaker attributes, environmental sound understanding often depends on short transient events, and reasoning-oriented tasks require semantic integration over longer contexts [1, 6,”

    From MOSS-Audio Technical Report (2026) · cites this paper as [15]

    The task-layer definitions (Appendix A, p. 13) distinguish low-level signal analysis from perceptual and semantic understanding, and the error analysis (p. 21) identifies difficulties with temporal dynamics and reasoning across contexts. This supports the broad point that audio tasks demand different kinds of understanding. The paper does not establish the cited sentence’s specific mappings of ASR, speech captioning, and environmental sounds to particular acoustic requirements, or that many tasks are inherently temporal. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 37

  1. [1][1] Wenxi Chen, Ziyang Ma, Xiquan Li, Xuenan Xu, Yuzhe Liang, Zhisheng Zheng, Kai Yu, and Xie Chen. SLAM-AAC: Enhancing audio captioning with paraphrasing augmentation and CLAP-Refine through LLMs. Proc. ICASSP , 2025. -
  2. [2][2] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919 , 2023. - · arXiv
  3. [3][3] Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759 , 2024. - · arXiv
  4. [4][4] Zihao Deng, Yinghao Ma, Yudong Liu, Rongchen Guo, Ge Zhang, Wenhu Chen, Wenhao Huang, and Emmanouil Benetos. Musilingo: Bridging music and text with pre-trained language models for music captioning and query response. Proc. NAACL , 2024. -
  5. [5][5] Soham Deshmukh, Satvik Dixit, Rita Singh, and Bhiksha Raj. Mellow: a small audio language model for reasoning. arXiv preprint arXiv:2503.08540 , 2025. - · arXiv
  6. [6][6] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In Proc. ICASSP , 2017. -
  7. [7][7] Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. GAMA: A large audio-language model with advanced audio understanding and complex reasoning abilities. Proc. EMNLP , 2024. -
  8. [8][8] Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Evuru, S Ramaneswaran, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. CompA: Addressing the gap in compositional reasoning in audio-language models. Proc. ICLR , 2024. -
  9. [9][9] Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro. Audio Flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities. arXiv preprint arXiv:2503.03983 , 2025. - · arXiv
  10. [11][11] Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. Proc. ICLR , 2024. -
  11. [13][13] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-R1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025. - · arXiv
  12. [14][14] Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi-Yuan Hsiao, Chun-Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, et al. Dynamic-superb: Towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech. In Proc. ICASSP , 2024. -
  13. [15][15] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024. - · arXiv
  14. [16][16] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. OpenAI o1 system card. arXiv preprint arXiv:2412.16720 , 2024. - · arXiv
  15. [17][17] Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. Proc. ICML , 2024. -
  16. [18][18] Gang Li, Jizhong Liu, Heinrich Dinkel, Yadong Niu, Junbo Zhang, and Jian Luan. Reinforcement learning outperforms supervised fine-tuning: A case study on audio question answering. arXiv preprint arXiv:2503.11197 , 2025. - · arXiv
  17. [19][19] Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al. Baichuan-omni-1.5 technical report. arXiv preprint arXiv:2501.15368 , 2025. - · arXiv
  18. [20][20] Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering. In Proc. EUSIPCO , 2022. -
  19. [21][21] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 , 2024. - · arXiv
  20. [22][22] Jizhong Liu, Gang Li, Junbo Zhang, Chenyu Liu, Heinrich Dinkel, Yongqing Wang, Zhiyong Yan, Yujun Wang, and Bin Wang. Leveraging ced encoder and large language models for automated audio captioning. Proc. DCASE Challenge , 2024. -
  21. [23][23] Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. Music understanding LLaMA: Advancing text-to-music generation with question answering and captioning. In Proc. ICASSP , 2024. -
  22. [24][24] Run Luo, Ting-En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, et al. OpenOmni: Large language models pivot zero-shot omnimodal alignment across language with real-time self-aware emotional speech synthesis. arXiv preprint arXiv:2501.04561 , 2025. - · arXiv
  23. [25][25] Ziyang Ma, Zhuo Chen, Yuping Wang, Eng Siong Chng, and Xie Chen. Audio-CoT: Exploring chain-of-thought reasoning in large audio language model. arXiv preprint arXiv:2501.07246 , 2025. - · arXiv
  24. [26][26] Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, et al. Speech recognition meets large language model: Benchmarking, models, and exploration. In Proc. AAAI , 2025. -
  25. [27][27] Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herremans, and Soujanya Poria. Mustango: Toward controllable text-to-music generation. Proc. NAACL , 2024. -
  26. [29][29] S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. MMAU: A massive multi-task audio understanding and reasoning benchmark. Proc. ICLR , 2025. -
  27. [30][30] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. Proc. ICLR , 2024. -
  28. [31][31] Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F Chen. AudioBench: A universal benchmark for audio large language models. Proc. NAACL , 2025. -
  29. [32][32] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Proc. NeurIPS , 2024. -
  30. [33][33] Benno Weck, Ilaria Manco, Emmanouil Benetos, Elio Quinton, George Fazekas, and Dmitry Bogdanov. MuChoMusic: Evaluating music understanding in multimodal audio-language models. Proc. ISMIR , 2024. -
  31. [34][34] Cheng Wen, Tingwei Guo, Shuaijiang Zhao, Wei Zou, and Xiangang Li. SARI: Structured audio reasoning via curriculum-guided reinforcement learning. arXiv preprint arXiv:2504.15900 , 2025. - · arXiv
  32. [35][35] Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. Audio-reasoner: Improving reasoning capability in large audio language models. arXiv preprint arXiv:2503.02318 , 2025. - · arXiv
  33. [36][36] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215 , 2025. - · arXiv
  34. [37][37] Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. AIR-Bench: Benchmarking large audio-language models via generative comprehension. Proc. ACL , 2024. -
  35. [38][38] Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Connecting speech encoder and large language model for ASR. Proc. ICASSP , 2024. -
  36. [39][39] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. AnyGPT: Unified multimodal llm with discrete sequence modeling. Proc. ACL , 2024. -
  37. [40][40] Zihan Zhao, Yiyang Jiang, Heyang Liu, Yu Wang, and Yanfeng Wang. LibriSQA: A novel dataset and framework for spoken question answering with large language models. Proc. TAI , 2024.

Cited by

Sorted by citations

  • MOSS-Audio Technical Report

    Yang, C., Yu, C., Chen, H., Zhu, J., Chen, J., Chen, K., Wang, W., Wang, Y., Jiang, Y., Jiang, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.01802

    “These results indicate that a unified audio-language model can support both precise audio recognition and higher-level audio reasoning, positioning MOSS-Audio as a promising understanding foundation for future voice agents [1, 14, 15, 18]. with the semantic space of a general-purpose language model.”
    Self-citation

Metadata

Authors
Ma, Z., Ma, Y., Zhu, Y., Yang, C., Chao, Y.-W., Xu, R., Chen, W., Chen, Y., Chen, Z., Cong, J., Li, K., Li, K., Li, S., Li, X., Li, X., Lian, Z., Liang, Y., Liu, M., Niu, Z., Wang, T., Wang, Y., Wang, Y., Wu, Y., Yang, G., Yu, J., Yuan, R., Zheng, Z., Zhou, Z., Zhu, H., Xue, W., Benetos, E., Yu, K., Chng, E.-S., & Chen, X.
Affiliations
  • Shanghai Jiao Tong University (CN)
  • Queen Mary University of London (GB)
  • Nanyang Technological University (SG)
  • ByteDance (China) (CN)
  • Chinese Academy of Sciences (CN)
  • Accenture (Switzerland) (CH)
  • Tianjin University (CN)
  • Beijing Institute of Technology (CN)
  • Microsoft (United States) (US)
  • Carnegie Mellon University (US)
  • The University of Texas at Austin (US)
  • Hong Kong University of Science and Technology (HK)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
74837-74868
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2505.13032
arXiv
2505.13032
OpenAlex
W4417287138
Cited by
4
References
1
Citations by year
2026: 3 · 2025: 1
FWCI
3.02
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Queen Mary Research Online (Queen Mary University of London) (submittedVersion, cc-by-nc)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
audio-language models, benchmark, speech, music, audio captioning, large language models, perceptual reasoning, cultural knowledge, audio question answering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review