Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?
Open accessAbstract
The advent of test-time scaling in large language models (LLMs), exemplified by Ope-nAI's o1 series, has advanced reasoning capabilities by scaling computational resource allocation during inference.While successors like QwQ, Deepseek-R1 (R1) and LIMO replicate these advancements, whether these models truly possess test-time scaling capabilities remains underexplored.This study found that longer CoTs of these o1-like models do not consistently enhance accuracy; in fact, correct solutions are often shorter than incorrect ones for the same questions.Further investigation shows this phenomenon is closely related to models' self-revision capabilities -longer CoTs contain more self-revisions, which often lead to performance degradation.We then compare sequential and parallel scaling strategies on QwQ, R1 and LIMO, finding that parallel scaling achieves better coverage and scalability.Based on these insights, we propose Shortest Majority Vote, a method that combines parallel scaling strategies with CoT length characteristics, significantly improving models' test-time scalability compared to conventional majority voting approaches.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
General-domain test-time scaling findings cannot readily predict the multilayered complexities inherent to clinical AI and fail to capture domain-specific bottlenecks and behavioral shifts in medical environments.
“Crucially, while foundational studies have established test-time scaling paradigms in the general domain by focusing primarily on text-based logic such as mathematics and coding [18-20], their findings cannot readily predict the multilayered complexities inherent to clinical AI.”
From Model and Task-Aware Test-Time Scaling Strategies for Large Language and Vision-Language Models in Medicine: Evaluation Study (2026) · cites this paper as [20]
The Benchmark section (page not provided) evaluates test-time scaling on three mathematical-reasoning benchmarks and GPQA, a broader science benchmark, supporting the general-domain context. The paper does not assess clinical AI or establish that its findings fail to predict clinical complexities, domain-specific bottlenecks, or behavioral shifts in medical settings. (LLM, conf=0.82)
Experimental — accuracy not yet measuredTest-time scaling improves both reliability and accuracy, especially on tasks that require multistep reasoning, without additional training or fine-tuning.
“Without requiring additional training or fine-tuning, test-time scaling improves both reliability and accuracy, especially on tasks that require multistep reasoning.”
From Model and Task-Aware Test-Time Scaling Strategies for Large Language and Vision-Language Models in Medicine: Evaluation Study (2026) · cites this paper as [20]
The Introduction and Section 5 (page numbers unavailable) report that parallel test-time scaling improves performance and coverage on reasoning benchmarks. But the paper does not establish an improvement in reliability, and it finds that longer reasoning chains and sequential self-revision do not consistently improve accuracy. The claim overgeneralizes the benefit to test-time scaling as a whole. (LLM, conf=0.85)
Experimental — accuracy not yet measuredPrior studies on test-time scaling in general domains have established that increasing inference compute can improve reasoning, though the relationship is not strictly monotonic.
“Prior studies on test-time scaling in general domains, such as mathematics and coding, have established that increasing inference compute can improve reasoning, though the relationship is not strictly monotonic.”
From Model and Task-Aware Test-Time Scaling Strategies for Large Language and Vision-Language Models in Medicine: Evaluation Study (2026) · cites this paper as [20]
The Related Work section says increasing computation during inference can improve performance, and the Self-Revision results show that accuracy can initially improve before oscillating or declining. The Benchmark section covers mathematical and scientific reasoning, but not coding, so the citing sentence’s coding example is not supported by this paper. (LLM, conf=0.84)
Experimental — accuracy not yet measuredTest-time scaling has emerged as an effective method for enhancing LLM performance during inference without additional training.
“To address these challenges, test-time scaling has recently emerged as an effective method for enhancing LLM performance during inference [17-20].”
From Model and Task-Aware Test-Time Scaling Strategies for Large Language and Vision-Language Models in Medicine: Evaluation Study (2026) · cites this paper as [20]
Optimal reasoning lengths exist and excessive self-revision can sometimes degrade performance.
“These works demonstrate that optimal reasoning lengths exist and that excessive self-revision can sometimes degrade performance [18-20].”
From Model and Task-Aware Test-Time Scaling Strategies for Large Language and Vision-Language Models in Medicine: Evaluation Study (2026) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · see citing works
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 538 · see citing works
Self-citationDeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.19437 · Cited by 294 · see citing works
Self-citationO1 Replication Journey: A Strategic Progress Report - Part 1Preprint
Qin, Y., Li, X.-F., Zou, H.-Y., Liu, Y.-X., Xia, S., Huang, Z., Ye, Y.-X., Yuan, W., Liu, H.-X., Li, Y.-F., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2410.18982 · Cited by 156 · see citing works
BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n SamplingPreprint
Garbacea, C., Gui, L., & Veitch, V. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2406.00832 · Cited by 139 · see citing works
Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language ModelsPreprint
Gao, B., Song, F., Yang, Z., Cai, Z., Miao, Y., Dong, Q., Li, L., Ma, C., Chen, L., Xu, R., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.07985 · Cited by 50 · see citing works
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2311.12022 · Cited by 30 · see citing works
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Snell, C., Lee, J., Xu, K., & Kumar, A. (2024) · arXiv · doi:10.48550/arxiv.2408.03314 · Cited by 27 · see citing works
Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.12122 · Cited by 14 · see citing works
AlphaMath Almost Zero: Process Supervision without Process
Chen, G., Liao, M., Li, C., & Fan, K. (2024) · arXiv · doi:10.48550/arxiv.2405.03553 · Cited by 14 · see citing works
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., & Hashimoto, T. (2025) · arXiv · doi:10.48550/arxiv.2501.19393 · Cited by 12 · see citing works
Training Language Models to Self-Correct via Reinforcement Learning
Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.12917 · Cited by 9 · see citing works
When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · arXiv · Cited by 4 · see citing works
Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning PerspectivePreprint
Zeng, Z., Cheng, Q., Yin, Z., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.14135 · Cited by 4 · see citing works
Self-citationLIMO: Less is More for Reasoning
Ye, Y., Huang, Z., Xiao, Y., Chern, E., Xia, S., & Liu, P. (2025) · arXiv · doi:10.48550/arxiv.2502.03387 · Cited by 4 · see citing works
OpenR: An Open Source Framework for Advanced Reasoning with Large Language ModelsPreprint
Wang, J., Fang, M., Wan, Z., Wen, M., Zhu, J., Liu, A., Gong, Z., Song, Y., Chen, L., Ni, L. M.-S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.09671 · Cited by 2 · see citing works
Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems
Min, Y., Chen, Z., Jiang, J., Chen, J., Deng, J., Hu, Y., Tang, Y., Wang, J., Cheng, X., Song, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.09413 · Cited by 2 · see citing works
Variational Best-of-N AlignmentPreprint
Amini, A., Vieira, T., Ash, E., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.06057 · Cited by 1 · see citing works
BOND: Aligning LLMs with Best-of-N Distillation
Sessa, P. G., Dadashi, R., Hussenot, L., Ferret, J., Vieillard, N., Ramé, A., Shariari, B., Perrin, S., Friesen, A., Cideron, G., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.14622 · Cited by 1 · see citing works
Fast Best-of-N Decoding via Speculative Rejection
Sun, H., Haider, M., Zhang, R., Yang, H., Qiu, J., Yin, M., Wang, M., Bartlett, P., & Zanette, A. (2024) · arXiv · doi:10.48550/arxiv.2410.20290 · Cited by 1 · see citing works
Huang, Z., Zou, H., Li, X., Liu, Y., Zheng, Y., Chern, E., Xia, S., Qin, Y., Yuan, W., & Liu, P. (2024) · arXiv · doi:10.48550/arxiv.2411.16489 · Cited by 1 · see citing works
TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N SamplingPreprint
Qiu, J., Lu, Y., Zeng, Y., Guo, J., Geng, J., Zhu, C., Juan, X., Yang, L., Wang, M., Huang, K., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.16033
Enhancing LLM Reasoning with Reward-guided Tree Search
Jiang, J., Chen, Z., Min, Y., Chen, J., Cheng, X., Wang, J., Tang, Y., Sun, H., Deng, J., Zhao, W. X., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.11694
Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMsPreprint
Wang, Y., Liu, Q., Xu, J., Liang, T., Chen, X., He, Z., Song, L., Yu, D., Li, J., Zhang, Z., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.18585
Flagged first, then by citations
Research works printed but not linked: 17
- Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024b. Teaching large language mod- els to self-debug. In The Twelfth International Con- ference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
- Ziru Chen, Michael White, Raymond J. Mooney, Ali Payani, Yu Su, and Huan Sun. 2024c. When is tree search useful for LLM planning? it depends on the discriminator. In Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 13659-13678. Association for Computational Lin- guistics. · DOI
- Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. CRITIC: large language models can self-correct with tool-interactive critiquing. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net.
- Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Rea- soning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2023, Singapore, December 6-10, 2023, pages 8154-8173. Association for Computational Linguistics. · DOI · Paper page
- Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xiny- ing Song, and Denny Zhou. 2024a. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
- Wouter Kool, Herke van Hoof, and Max Welling. 2019. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. In Proceedings of the 36th Interna- tional Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, vol- ume 97 of Proceedings of Machine Learning Re- search, pages 3499-3508.
- PMLR.
- Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harri- son Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let's verify step by step. In The Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net.
- Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdan- bakhsh, and Peter Clark. 2023. Self-refine: Itera- tive refinement with self-feedback. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Sys- tems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.
- Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2024. Alphazero-like tree-search can guide large language model decoding and training. In Forty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. Open- Review.net.
- Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowd- hery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Ad- vances in Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 -December 9, 2022.
- Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Qizhe Xie. 2023. Self-evaluation guided beam search for reasoning. In Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.
- Fei Yu, Anningzhe Gao, and Benyou Wang. 2024. Ovm, outcome-supervised value models for planning in mathematical reasoning. In Findings of the Associ- ation for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 858- 875. Association for Computational Linguistics. · DOI
- Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. 2023. Planning with large language models for code gen- eration. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
- Yunxiang Zhang, Muhammad Khalifa, Lajanugen Lo- geswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. 2024. Small language models need strong verifiers to self-correct reasoning. In Findings of the Association for Computational Lin- guistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 15637-15653. Association for Computational Linguistics. · DOI · Paper page
- Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. 2024. Sglang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Pro- cessing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 -15, 2024.
Cited by
Sorted by citations
Bai, L., Huang, Z., Wang, X., Sun, J., Mihalcea, R., Brynjolfsson, E., Pentland, A., & Pei, J. (2026) · arXiv · doi:10.48550/arxiv.2604.22750 · Cited by 3 · see citing works
Oh, G., Kim, S., Park, S., & Kim, B.-H. (2026) · Journal of Medical Internet Research · doi:10.2196/90693 · Cited by 1 · see citing works
- Authors
- Zeng, Z., Cheng, Q., Yin, Z., Zhou, Y., & Qiu, X.
- Affiliations
- Fudan University (CN)
- Shanghai Artificial Intelligence Laboratory (CN)
- Venue
- Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2025
- Pages
- 4651-4665
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4412886683
- Cited by
- 6
- References
- 24
- Citations by year
- 2026: 5 · 2025: 1
- FWCI
- 5.70
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- test-time scaling, chain-of-thought reasoning, self-revision, parallel scaling, majority voting, DeepSeek-R1, large language models, reasoning accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
