The DOI matches its DataCite record.
Are Your LLMs Capable of Stable Reasoning?
Open accessAbstract
The rapid advancement of large language models (LLMs) has shown remarkable progress in complex reasoning tasks. However, a significant disparity exists between benchmark performances and real-world applications. We attribute this gap primarily to current evaluation protocols and metrics, which inadequately capture the full spectrum of LLM capabilities, especially in complex reasoning tasks where both accuracy and consistency are essential. In this paper, we introduce G-Pass@$k$, a novel evaluation metric that continuously assesses model performance across multiple sampling attempts, quantifying both the model's performance potential and its stability. Through extensive experiments on various public and newly constructed benchmarks, we employ G-Pass@$k$ in conjunction with state-of-the-art large language models to provide comprehensive insights into their potential capabilities and operational consistency. Our findings reveal a significant opportunity to enhance the realistic reasoning abilities of LLMs, underscoring the necessity for more robust evaluation metrics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
On LiveMathBench (version 202505), Qwen's completion rate drops to 0.0%, consistent with Llama's 0.0%, and its partial-prompt answer accuracy falls to 2.0%, comparable to Llama's 1.0%.
“Crucially, on the newly released LiveMathBench (version 202505) (Liu et al. 2024), Qwen's completion rate drops sharply to 0.0%, consistent with Llama's 0.0%. Its partial-prompt answer accuracy also falls to just 2.0%, comparable to Llama's 1.0%.”
From Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination (2025) · cites this paper as Liu et al. 2024
The citation is attached to the identification of LiveMathBench, not to the citing authors’ Qwen and Llama results. Section F.1 says the paper constructed LiveMathBench, but section F.1.1 identifies its version as 202412; our copy does not establish the cited 202505 version. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Still being checked.
Research works printed but not linked: 18
- Berk Atil, Alexa Chittams, Liseng Fu, Ferhan Ture, Lixinyu Xu, and Breck Baldwin. 2024. LLM stabil- ity: A detailed analysis with some surprises. CoRR, abs/2408.04667. · arXiv
- Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. 2024. Omni-math: A univer- sal olympiad level mathematic benchmark for large language models. CoRR, abs/2410.07985. · arXiv
- Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos, and Giorgos Stamou. 2024. Puzzle solving using reasoning of large language models: A survey. In EMNLP, pages 11574-11591. Association for Computational Linguistics.
- Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. 2024. Changing an- swer order can decrease MMLU accuracy. CoRR, abs/2406.19470. · arXiv
- Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In NeurIPS Datasets and Benchmarks. Hugging Face. 2024. InternLM 3-8B Instruct Model. https://huggingface.co/internlm/ internlm3-8b-instruct. Accessed: 2025-01-15.
- Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. In ACL (industry), pages 37-42. Association for Computational Linguistics.
- Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. 2019. Spoc: Search-based pseudocode to code. In NeurIPS, pages 11883-11894.
- Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin. 2024. An open-source data contamination report for large language models. In EMNLP (Findings), pages 528-541. Association for Computational Lin- guistics.
- Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harri- son Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let's verify step by step. In ICLR. Open- Review.net.
- Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen. 2024. Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark. In ACL (Findings), pages 6884-6915. Association for Computational Linguistics.
- Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian- guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wiz- ardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. CoRR, abs/2308.09583. · arXiv
- Shiwen Ni, Xiangtao Kong, Chengming Li, Xiping Hu, Ruifeng Xu, Jia Zhu, and Min Yang. 2024. Train- ing on the benchmark is not all you need. CoRR, abs/2409.01790. · arXiv
- Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain your- self! leveraging language models for commonsense reasoning. In ACL, pages 4932-4942. Association for Computational Linguistics.
- Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mo- hammad Saleh, Balaji Lakshminarayanan, and Pe- ter J. Liu. 2023. Out-of-distribution detection and selective generation for conditional language models. In ICLR. OpenReview.net. Google Research. 2024. Gemini 1.5 pro: A large lan- guage model by google. https://ai.google.dev/. Accessed: 2024-12-12.
- Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, An- ton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Pater- son, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijayku- mar, Dominika Rogozinska, Dustin Herbison, Elisa Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. 2024. Metamath: Bootstrap your own mathematical ques- tions for large language models. In ICLR. OpenRe- view.net.
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS.
- Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. Don't make your LLM an eval- uation benchmark cheater. CoRR, abs/2311.01964. · arXiv
- Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. 2024. Prosa: Assessing and understanding the prompt sen- sitivity of llms. In EMNLP (Findings), pages 1950- 1976. Association for Computational Linguistics.
Cited by
Sorted by citations
Evaluating the Accuracy and Stability of Frontier LLMs on ENADE Computer Science Questions
Carvalho, L. D. M., Carvalho, C. M., & Mendonça, N. C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15993-9_18 · Cited by 2
Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
Wu, M., Zhang, Z., Dong, Q., Xi, Z., Zhao, J., Jin, S., Fan, X., Zhou, Y., Lv, H., Zhang, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.10532 · Cited by 75
“Formally, for each question q , GRPO samples a group of outputs { o 1 , · · · , o G } from the old policy π θ old and then optimizes the policy model by maximizing the following objective: (1) where r i,t = π θ ( o i,t | q,o i, t ) π θ old ( o i,t | q,o i, t ) , ϵ and β are hyperparameters, and ˆ A…”
Self-citationAssessment of Large Models' Problem-Solving Ability in Mathematics
Zhai, L., Lan, W., & Kuang, S. (2025) · International Conference on Information Science and Education (ICISE-IE) · doi:10.1109/icise-ie68873.2025.11378903
Liu, J., Lin, Y., Qiu, X., Fortino, G., Ji, X., Lu, Z., Chen, M., & Humar, I. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104477
Self-citationMeasuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework
Şenol, A., Agrawal, G., & Liu, H. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10090300 · Cited by 3
Self-citation
Metadata
- Authors
- Liu, J., Liu, H., Xiao, L., Wang, Z., Liu, K., Gao, S., Zhang, W., Zhang, S., & Chen, K.
- Affiliations
- Shanghai Artificial Intelligence Laboratory (CN)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 17594-17632
- arXiv
- 2412.13147
- OpenAlex
- W4412887967
- Cited by
- 4
- References
- 0
- Citations by year
- 2026: 3 · 2025: 1
- FWCI
- 3.79
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, complex reasoning, evaluation metrics, reasoning stability, consistency, multiple sampling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
