OptimalThinkingBench: Evaluating Over and Underthinking in LLMs
The DOI matches its DataCite record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Thinking LLMs solve complex tasks at the expense of increased compute and overthinking on simpler problems, while non-thinking LLMs are faster and cheaper but underthink on harder reasoning problems. This has led to the development of separate thinking and non-thinking LLM variants, leaving the onus of selecting the optimal model for each query on the end user. We introduce OptimalThinkingBench, a unified benchmark that jointly evaluates overthinking and underthinking in LLMs and also encourages the development of optimally-thinking models that balance performance and efficiency. Our benchmark comprises two sub-benchmarks: OverthinkingBench, featuring simple math and general queries in 72 domains, and UnderthinkingBench, containing 11 challenging reasoning tasks along with harder math problems. Using novel thinking-adjusted accuracy metrics, we extensively evaluate 33 different thinking and non-thinking models and show that no model is able to optimally think on our benchmark. Thinking models often overthink for hundreds of tokens on the simplest user queries without improving performance. In contrast, large non-thinking models underthink, often falling short of much smaller thinking models. We further explore several methods to encourage optimal thinking, but find that these approaches often improve on one sub-benchmark at the expense of the other, highlighting the need for better unified and optimal models in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“which is likely due to the low complexity of the task and potential overthinking of reasoning models [1].”
From FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning (2025) · cites this paper as [1]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
Sui, Y., Chuang, Y.-N., Wang, G., Zhang, J., Zhang, T., Yuan, J., Liu, H., Wen, A., Zhong, S., Zou, N., et al. (2025) · arXiv · Cited by 517 · cited 2×
“…works have analyzed overthinking and underthinking in LLMs across various domains including adversarial, tool-use, math, and unanswerable queries (Sui et al., 2025; Wang et al., 2025b; Chen et al., 2025a; Kumar et al., 2025; Cuadron et al., 2025; Song and Zheng, 2025; Zhao et al., 2025;…”
C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness
Kang, Y., Sun, X., Chen, L., & Zou, W. (2024) · arXiv · doi:10.48550/arxiv.2412.11664 · Cited by 12
Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations
Handa, K., Tamkin, A., McCain, M., Huang, S., Durmus, E., Heck, S., Mueller, J., Hong, J., Ritchie, S., Belonax, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.04761 · Cited by 11
AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
Kirichenko, P., Ibrahim, M., Chaudhuri, K., & Bell, S. J. (2025) · arXiv · doi:10.48550/arxiv.2506.09038 · Cited by 5 · cited 2×
“Notably, these analyses span adversarial (Kumar et al., 2025), tool-use (Cuadron et al., 2025), math (Song and Zheng, 2025; Zhao et al., 2025; Su et al., 2025; Wang et al., 2025b) and unanswerable (Kirichenko et al., 2025) queries.”
L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning
Aggarwal, P., & Welleck, S. (2025) · arXiv · doi:10.48550/arxiv.2503.04697 · Cited by 2
Self-citationExploring the System 1 Thinking Capability of Large Reasoning Models
Zhang, W., Nie, S., Zhang, X., Zhang, Z., & Liu, T. (2026) · Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2026/676
Cited by
Bai, L., Huang, Z., Wang, X., Sun, J., Mihalcea, R., Brynjolfsson, E., Pentland, A., & Pei, J. (2026) · arXiv · doi:10.48550/arxiv.2604.22750 · Cited by 3
“…with recent findings on inverse test-time scaling (Snell et al., 2024; Wu et al., 2025; Gema et al., 2025; Zeng et al., 2025; Yang et al., 2025; Aggarwal et al., 2025), which show that additional reasoning steps or longer chains of thought do not necessarily improve accuracy and may instead…”
FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning
Hu, L., Jiao, J., Liu, J., Ren, Y., Wen, Z., Zhang, K., Zhang, X., Gao, X., He, T., Hu, F., et al. (2025) · arXiv · Cited by 2
Code-enabled language models can outperform reasoning models on diverse tasks
Zhang, C. E., Colas, C., Poesia, G., Tenenbaum, J. B., & Andreas, J. (2025) · arXiv · doi:10.48550/arxiv.2510.20909
Metadata
- Authors
- Aggarwal, P., Kim, S., Lanchantin, J., Welleck, S., Weston, J., Kulikov, I., & Saha, S.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- arXiv
- 2508.13141
- References
- 7
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: OptimalThinkingBench: Evaluating Over and Underthinking in LLMs. https://app.cochat.ai/papers/10.48550/arxiv.2508.13141
- The paper
- Aggarwal, P., Kim, S., Lanchantin, J., Welleck, S., Weston, J., Kulikov, I., & Saha, S. (2025). OptimalThinkingBench: Evaluating Over and Underthinking in LLMs. arXiv. https://doi.org/10.48550/arxiv.2508.13141
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.