CoChat
Sign Up
PreprintCited by 6Topic ModelingNatural Language Processing TechniquesMathematics Education and Pedagogy

The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longer

Open access
Ballon, Marthe, Algaba, Andres, Ginis, Vincent·2025·arXiv
DOI 10.48550/arxiv.2502.15631arXiv 2502.15631

Preprint — published as 10.1038/s41598-026-50923-2

Open this paper in CoChat — free account

Abstract

Large language models have demonstrated remarkable progress in mathematical reasoning, leveraging chain-of-thought and reinforcement learning. However, many open questions remain regarding the interplay between reasoning token usage and accuracy gains. In particular, when comparing models across generations, it is unclear whether improved performance results from longer reasoning chains or more efficient reasoning. We systematically analyze reasoning chain length across o1-mini and o3-mini variants on the Omni-MATH benchmark, finding that o3-mini (m) achieves superior accuracy without requiring longer reasoning chains than o1-mini. Moreover, we show that accuracy generally declines as reasoning chains grow across all models and compute settings, even when controlling for difficulty of the questions. This accuracy drop is significantly smaller in more proficient models, suggesting that new generations of reasoning models use test-time compute more effectively. Finally, we highlight that while o3-mini (h) achieves a marginal accuracy gain over o3-mini (m), it does so by allocating substantially more reasoning tokens across all problems, even the ones that o3-mini (m) can already solve. These findings provide new insights into the relationship between model capability and reasoning length, with implications for efficiency, scaling, and evaluation methodologies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.15631

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 37

  1. [1]1. Brown, T. et al. Language models are few-shot learners. Advances in neural information processing systems 33 , 1877-1901 (2020).
  2. [2]2. Bubeck, S. et al. Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712 (2023). · arXiv
  3. [5]5. Kaplan, J. et al. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020). · arXiv
  4. [6]6. Gurnee, W . & Tegmark, M. Language models represent space and time. arXiv preprint arXiv:2310.02207 (2024). · arXiv
  5. [7]7. Hao, S. et al. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992 (2023). · arXiv
  6. [8]8. Wei, J. et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022). · arXiv
  7. [9]9. Muennighoff, N. et al. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393 (2025). · arXiv
  8. [10]10. Snell, C., Lee, J., Xu, K. & Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 (2024). · arXiv
  9. [11]11. Anderson, C. J. et al. PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models. arXiv preprint arXiv:2502.01584 (2025). · arXiv
  10. [12]12. Chen, X. et al. Do not think that much for 2+ 3=? On the overthinking of o1-like LLMs. arXiv preprint arXiv:2412.21187 (2024). · arXiv
  11. [13]13. Wang, Y. et al. Thoughts are all over the place: On the underthinking of o1-like LLMs. arXiv preprint arXiv:2501.18585 (2025). · arXiv
  12. [14]14. Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 , 24824-24837 (2022b).
  13. [15]15. Guo, D. et al. Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025). · arXiv
  14. [16]16. Guan, X. et al. rStar-Math: Small LLMs can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519 (2025). · arXiv
  15. [17]17. Gao, B. et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985 (2024). · arXiv
  16. [18]18. Rein, D. et al. GPQA: A graduate-level google-proof Q&A benchmark. arXiv preprint arXiv:2311.12022 (2023). · arXiv
  17. [19]19. Cobbe, K. et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021). · arXiv
  18. [20]20. Mirzadeh, I. et al. GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229 (2024). · arXiv
  19. [21]21. Hendrycks, D. et al. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 (2021). · arXiv
  20. [22]22. Phan, L. et al. Humanity's last exam. arXiv preprint arXiv:2501.14249 (2025). · arXiv
  21. [23]23. Glazer, E. et al. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in AI. arXiv preprint arXiv:2411.04872 (2024). · arXiv
  22. [24]24. Chen, M. et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). · arXiv
  23. [25]25. Jimenez, C. E. et al. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations (2024).
  24. [26]26. Srivastava, A. et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 (2022). · arXiv
  25. [27]27. Hendrycks, D. et al. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020). · arXiv
  26. [28]28. Clark, P . et al. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457 (2018). · arXiv
  27. [29]29. Agarwal, S. et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925 (2025). · arXiv
  28. [30]30. Peng, K., Ding, L., Ouyang, Y., Fang, M. & Tao, D. Revisiting Overthinking in Long Chain-of-Thought from the Perspective of Self-Doubt. arXiv preprint arXiv:2505.23480 (2025). · arXiv
  29. [31]31. Jin, M. et al. The impact of reasoning step length on large language models. arXiv preprint arXiv:2401.04925 (2024). · arXiv
  30. [32]32. Levy, M., Jacoby, A. & Goldberg, Y. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848 (2024). · arXiv
  31. [33]33. Zhong, Q. et al. Achieving > 97 % on GSM8K: Deeply understanding the problems makes LLMs perfect reasoners. arXiv preprint arXiv:2404.14963 (2024). · arXiv
  32. [34]34. Mao, M., Yin, B., Zhu, Y. & Fang, X. Early Stopping Chain-of-Thoughts in Large Language Models. arXiv preprint arXiv:2509.14004 (2025). · arXiv
  33. [35]35. Sun, R., Cheng, W ., Li, D., Chen, H. & Wang, W . Stop When Enough: Adaptive Early-Stopping for Chain-of-Thought Reasoning. arXiv preprint arXiv:2510.10103 (2025). · arXiv
  34. [36]36. Lightman, H. et al. Let's Verify Step by Step. In The twelfth international conference on learning representations (2023).
  35. [37]37. Wang, P . et al. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 9426-9439 (2024).
  36. [39]39. Ballon, M., Verbeken, B., Ginis, V . & Algaba, A. Probing the Trajectories of Reasoning Traces in Large Language Models. arXiv preprint arXiv:2601.23163 (2026). · arXiv
  37. [40]40. Vanhoyweghen, A., Verbeken, B., Algaba, A. & Ginis, V . Lexical Hints of Accuracy in LLM Reasoning Chains. arXiv preprint arXiv:2508.15842 (2025). · arXiv

Cited by

Sorted by citations

Authors
Ballon, M., Algaba, A., & Ginis, V.
Affiliations
Vrije Universiteit Brussel (BE)
Corresponding author
Ballon, Marthe
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Volume
16
Issue
1
ISSN
2045-2322 (ISSN-L)
DOI
10.48550/arxiv.2502.15631
arXiv
2502.15631
OpenAlex
W7160398065
Cited by
6
References
0
Citations by year
2026: 5 · 2025: 1
FWCI
16.98
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • VUBIR (Vrije Universiteit Brussel) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Fonds Wetenschappelijk Onderzoek (G0K9322N, G032822N, 1286924N)
  • Vlaamse regering (VLAAI1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Mathematics › Applied Mathematics › Mathematics Education and Pedagogy
Keywords
large language models, mathematical reasoning, chain-of-thought, o3-mini, model capability
MeSH terms
Large Language Models, Humans, Problem Solving, Thinking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review