CoChat
Sign Up
Cited by 77Topic ModelingMultimodal Machine Learning ApplicationsNatural Language Processing Techniques

LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks

Open access
Bai, Yushi, Tu, Shangqing, Zhang, Jiajie, Peng, Hao, Wang, Xiaozhi, Lv, Xin, Cao, Shulin, Xu, Jiazheng +4 more·2025·Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/2025.acl-long.183
Open this paper in CoChat — free accountOpen PDF

Abstract

Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2025.acl-long.183

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 60

  1. Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie C.Y. Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, Feryal Behbahani, Alek- sandra Faust, and Hugo Larochelle. 2024. Many- shot in-context learning. In The Thirty-eighth Annual Conference on Neural Information Processing Sys- tems.
  2. sonnet. Yushi Bai, Xin Lv, Juanzi Li, and Lei Hou. 2023. Answering complex logical queries on knowledge graphs via query computation tree optimization. In International Conference on Machine Learning, pages 1472-1491. PMLR.
  3. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. 2024c. Benchmarking foundation models with language-model-as-an-examiner. Ad- vances in Neural Information Processing Systems, 36.
  4. Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025. Longwriter: Unleashing 10,000+ word generation from long context LLMs. In The Thirteenth Interna- tional Conference on Learning Representations.
  5. Egor Bogomolov, Aleksandra Eliseeva, Timur Gal- imzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie van Deursen, Maliheh Izadi, et al. 2024. Long code arena: a set of benchmarks for long-context code models. arXiv preprint arXiv:2406.11612. · arXiv
  6. Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Han- wang Zhang. 2022. Kqa pro: A dataset with explicit compositional programs for complex question an- swering over knowledge base. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 6101-6119.
  7. Cohere For AI. 2024. c4ai-command-r-plus-08-2024. · DOI
  8. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers an- chored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4599-4610.
  9. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. Goemotions: A dataset of fine-grained emo- tions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040-4054.
  10. Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-nerd: A few-shot named entity recog- nition dataset. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3198-3213.
  11. Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2024. Bamboo: A comprehensive benchmark for evaluating long text modeling capac- ities of large language models. In Proceedings of the 2024 Joint International Conference on Compu- tational Linguistics, Language Resources and Evalu- ation (LREC-COLING 2024), pages 2086-2099.
  12. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378.
  13. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. · arXiv
  14. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Han- naneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. Data engineering for scaling language models to 128K context. In Proceedings of the 41st Interna- tional Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 14125-14134.
  15. PMLR.
  16. Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2024. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660. · arXiv
  17. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chen- hui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Han- lin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiao- tao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yi- fan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793. · arXiv
  18. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419-1436.
  19. Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. · arXiv
  20. Ziyan Jiang, Xueguang Ma, and Wenhu Chen. 2024. Longrag: Enhancing retrieval-augmented gener- ation with long-context llms. arXiv preprint arXiv:2406.15319. · arXiv
  21. Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. 2025. Long-context LLMs meet RAG: Over- coming challenges for long inputs in RAG. In The Thirteenth International Conference on Learning Representations.
  22. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Ed- ward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Asso- ciation for Computational Linguistics, 6:317-328.
  23. Satyapriya Krishna, Kalpesh Krishna, Anhad Mo- hananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. 2025. Fact, fetch, and reason: A unified evaluation of retrieval- augmented generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chap- ter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa- pers), pages 4745-4759, Albuquerque, New Mexico. Association for Computational Linguistics.
  24. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rod- kin, Dmitry Igorevich Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. BABILong: Testing the limits of LLMs with long context reasoning-in-a- haystack. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Bench- marks Track.
  25. Philippe Laban, Alexander Richard Fabbri, Caiming Xiong, and Chien-Sheng Wu. 2024. Summary of a haystack: A challenge to long-context llms and rag systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9885-9903.
  26. Quinn Leng, Jacob Portes, Sam Havens, Matei Zaharia, and Michael Carbin. 2024. Long context rag per- formance of large language models. arXiv preprint arXiv:2411.03538. · arXiv
  27. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhat- tacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2024a. From generation to judgment: Op- portunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594. · arXiv
  28. Tianyang Liu, Canwen Xu, and Julian McAuley. 2024a. Repobench: Benchmarking repository-level code auto-completion systems. In The Twelfth Interna- tional Conference on Learning Representations.
  29. Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. 2024b. Longgenbench: Long-context gener- ation benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 865-883.
  30. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241-2252.
  31. Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. 2022. Quality: Question answering with long input texts, yes! In Proceedings of the 2022 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies.
  32. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and En- rico Shippole. 2024. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representa- tions.
  33. Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, et al. 2024. Hellobench: Evaluating long text generation capabilities of large language models. arXiv preprint arXiv:2409.16191. · arXiv
  34. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Fi- rat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. · arXiv
  35. David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  36. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. Zeroscrolls: A zero-shot benchmark for long text understanding. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2023, pages 7977-7989.
  37. Mingyang Song, Mao Zheng, and Xuan Luo. 2025. Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models. In Proceedings of the 31st International Conference on Computational Linguis- tics, pages 3753-3763, Abu Dhabi, UAE. Associa- tion for Computational Linguistics.
  38. Zayne Rea Sprague, Fangcong Yin, Juan Diego Ro- driguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. 2025. To cot or not to cot? chain-of- thought helps mainly on math and symbolic reason- ing. In The Thirteenth International Conference on Learning Representations.
  39. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Juraf- sky, and Luke Melas-Kyriazi. 2024. A benchmark for learning to translate a new language from one gram- mar book. In The Twelfth International Conference on Learning Representations.
  40. Jen tse Huang, Eric John Li, Man Ho LAM, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael Lyu. 2025. Competing large language models in multi- agent gaming environments. In The Thirteenth Inter- national Conference on Learning Representations.
  41. Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, et al. 2024. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640. · arXiv
  42. Denny Vrandečić and Markus Krötzsch. 2014. Wiki- data: a free collaborative knowledgebase. Communi- cations of the ACM, 57(10):78-85.
  43. Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel Bowman. 2022. Squality: Building a long-document summarization dataset the hard way. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1139-1156.
  44. Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al. 2024a. Leave no document behind: Benchmarking long-context llms with extended multi-doc qa. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5627-5646.
  45. Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. Maven: A massive general domain event detection dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1652-1671.
  46. Zhilin Wang, Alexander Bukharin, Olivier Delal- leau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Olek- sii Kuchaiev, and Yi Dong. 2025. Helpsteer2- preference: Complementing ratings with preferences. In The Thirteenth International Conference on Learn- ing Representations.
  47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in neural information processing systems, 35:24824-24837.
  48. Yuhao Wu, Ming Shan Hee, Zhiqiang Hu, and Roy Ka-Wei Lee. 2025b. Longgenbench: Benchmark- ing long-form generation in long context LLMs. In The Thirteenth International Conference on Learning Representations.
  49. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. 2024. Effective long-context scaling of founda- tion models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), pages 4643-4663.
  50. Zhe Xu, Jiasheng Ye, Xiangyang Liu, Tianxiang Sun, Xiaoran Liu, Qipeng Guo, Linlin Li, Qun Liu, Xu- anjing Huang, and Xipeng Qiu. 2024. Detectiveqa: Evaluating long-context reasoning on detective nov- els. arXiv preprint arXiv:2409.02465. · arXiv
  51. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2.5 tech- nical report. arXiv preprint arXiv:2412.15115. · arXiv
  52. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. Docred: A large-scale document-level relation extraction dataset. In Pro- ceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics, pages 764-777.
  53. Zijun Yao, Yuanyong Chen, Xin Lv, Shulin Cao, Amy Xin, Jifan Yu, Hailong Jin, Jianjun Xu, Peng Zhang, Lei Hou, et al. 2023. Viskop: Visual knowledge ori- ented programming for interactive knowledge base question answering. In Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 3: System Demonstra- tions), pages 179-189.
  54. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. 2025. Justice or prejudice? quan- tifying biases in LLM-as-a-judge. In The Thirteenth International Conference on Learning Representa- tions.
  55. Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. 2025. Helmet: How to evaluate long- context language models effectively and thoroughly. In International Conference on Learning Representa- tions (ICLR).
  56. Chen Zhang, Xiao Liu, Jiuheng Lin, and Yansong Feng. 2024a. Teaching large language models an unseen language on the fly. In Findings of the As- sociation for Computational Linguistics: ACL 2024, pages 8783-8800, Bangkok, Thailand. Association for Computational Linguistics. · DOI
  57. Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, et al. 2024b. Longcite: Enabling llms to generate fine-grained citations in long-context qa. arXiv preprint arXiv:2409.02897. · arXiv
  58. Xiaokang Zhang, Jing Zhang, Zeyao Ma, Yang Li, Bohan Zhang, Guanlin Li, Zijun Yao, Kangli Xu, Jinchang Zhou, Daniel Zhang-Li, et al. 2024c. Tablellm: Enabling tabular data manipulation by llms in real office usage scenarios. arXiv preprint arXiv:2403.19318. · arXiv
  59. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623.
  60. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyil- maz, Yang Liu, Xipeng Qiu, et al. 2021. Qmsum: A new benchmark for query-based multi-domain meet- ing summarization. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 5905-5921. Difficulty Length (<32k; 32k-128k; >128k) Model Overall Invalid Easy Hard Short Medium Long Open-source models GLM-4-9B-Chat 30.4 32.2 0.8 5.6 31.1 36.6 30.0 29.5 34.0 36.2 30.0 31.9 25.2 26.2 Llama-3.1-8B-Instruct 31.0 30.5 3.8 0.4 32.0 36.5 30.3 26.8 37.6 34.4 27.9 31.7 25.9 21.5 Llama-3.1-70B-Instruct 31.7 36.6 0.2 1.8 32.3 36.3 31.3 36.8 41.2 45.6 27.4 34.1 24.1 26.9 Llama-3.3-70B-Instruct 31.0 36.6 4.6 1.8 35.8 38.5 28.0 35.5 39.9 45.6 27.0 33.4 24.1 28.2 Llama-3.1-Nemotron-70B-Instruct 31.8 37.2 3.2 8.2 33.6 39.5 30.7 35.9 40.4 47.8 28.0 32.1 25.0 29.9 Qwen2.5-7B-Instruct 28.9 30.0 7.4 0.8 31.5 31.0 27.3 29.4 39.0 35.7 25.5 26.7 18.8 27.1 Qwen2.5-72B-Instruct 40.4 39.2 4.0 1.6 44.4 43.0 37.9 36.8 46.7 50.1 34.2 29.4 42.1 40.3 Mistral-Large-Instruct-2407 30.9 34.5 16.9 3.6 34.9 35.4 28.4 33.9 37.8 41.7 25.6 31.6 29.9 28.2 Mistral-Large-Instruct-2411 35.7 41.0 5.4 5.6 40.1 45.3 33.0 38.3 43.3 47.9 31.7 36.0 31.0 39.1 c4ai-command-r-plus-08-2024 28.8 32.0 3.8 1.4 31.0 34.9 27.4 30.1 37.4 39.6 25.2 24.8 21.5 33.6 Proprietary models GLM-4-Plus 44.6 47.6 1.0 5.8 47.5 53.5 42.8 43.9 50.7 54.7 46.5 46.2 30.6 38.4 GPT-4o-mini-2024-07-18 29.8 32.6 2.0 0.8 31.8 32.8 28.5 32.5 32.5 35.1 29.0 31.7 26.6 30.1 GPT-4o-2024-08-06 50.2 51.3 0.2 0.4 57.4 58.2 45.7 47.1 53.5 53.9 52.4 50.8 40.2 47.9 gpt-4o-2024-11-20 47.4 51.7 5.6 1.2 52.9 54.7 44.0 49.8 50.1 60.1 48.5 48.7 40.7 43.8 o1-mini-2024-09-12 38.3 39.4 1.8 2.0 39.7 43.4 37.4 36.9 48.7 49.6 34.0 33.5 29.0 34.3 o1-preview-2024-09-12 57.9 57.1 0.8 3.4 67.1 60.5 52.3 55.0 62.7 65.3 53.8 51.1 58.3 55.5 Claude-3.5-Sonnet-20241022 44.4 50.4 13.9 14.9 51.7 59.6 40.0 44.8 49.2 56.0 41.9 46.5 41.7 49.1 Gemini-Exp-1206 49.3 53.1 0.0 2.2 52.9 61.9 47.1 47.7 53.9 56.1 47.1 50.1 45.8 53.7 Gemini-2.0-Flash-Exp 48.8 52.6 5.6 6.2 52.5 60.5 46.5 47.7 51.9 58.9 48.1 47.0 44.9 53.5 Human 55.7 8.2 100 28.4 49.3 60.3 57

Cited by

Sorted by citations

Related

Authors
Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., Dong, Y., Tang, J., & Li, J.
Affiliations
  • Tsinghua University (CN)
  • Zhipu AI (China) (CN)
ORCID
Venue
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2025
Pages
3639-3664
ISSN
0736-587X (ISSN-L)
DOI
10.18653/v1/2025.acl-long.183
OpenAlex
W4412886690
Cited by
29
References
0
Citations by year
2026: 25 · 2025: 4
FWCI
27.65
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
Funders
National Natural Science Foundation of China
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
long-context reasoning, large language models, benchmark dataset

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review