Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
Open accessAbstract
Xinghao Chen, Zhijing Sun, Guo Wenjin, Miaoran Zhang, Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, Xiaoyu Shen. Findings of the Association for Computational Linguistics: ACL 2025. 2025.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reasoning traces provide critical supervision signal that cannot be replaced by final answers alone.
“Answer-only distillation performs no better than-and often worse than-distillation on our rewritten traces, confirming that reasoning traces provide critical supervision signal that cannot be replaced by final answers alone, consistent with prior findings (Hsieh et al., 2023; Chen et al., 2025).”
From Protecting Language Models Against Unauthorized Distillation through Trace Rewriting (2026) · cites this paper as Chen et al., 2025
The paper compares an answer-only baseline with CoT-trained students (Figure 6 caption) and reports that stronger students gain more from CoT distillation (“The Matthew Effect in SLMs”; page unavailable). It also says weaker models may fail to benefit when the reasoning is too complex. Thus it supports the value of reasoning traces, but not the unqualified claim that final answers alone cannot replace them in all cases. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 9
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
- Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS.
- Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. 2024. The impact of reasoning step length on large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1830-1842, Bangkok, Thailand. Associ- ation for Computational Linguistics. · DOI
- Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sab- harwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
- William Merrill and Ashish Sabharwal. 2024. The ex- pressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations. Meta. 2024a. Introducing meta Llama 3: The most capable openly available llm to date. https://ai. meta.com/blog/meta-llama-3/.
- Lev S Vygotsky. 1978. Mind in society: The develop- ment of higher psychological processes, volume 86. Harvard university press.
- Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wen- hao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. MAmmoTH: Building math generalist models through hybrid instruction tuning. In The Twelfth In- ternational Conference on Learning Representations.
- Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. Least-to-most prompting enables com- plex reasoning in large language models. In The Eleventh International Conference on Learning Rep- resentations.
- Martin Zong, Zengyu Qiu, Xinzhu Ma, Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. 2023. Better teacher better student: Dynamic prior knowledge for knowledge distillation. In The Eleventh International Conference on Learning Rep- resentations.
Cited by
Sorted by citations
Li, W., Zhang, Y., Wang, C. R., Li, Y., He, X., Wang, A. L., Xu, M., Zhang, F., Sun, H., Wang, K., et al. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131476 · Cited by 3 · see citing works
Self-citationProtecting Language Models Against Unauthorized Distillation through Trace Rewriting
Ma, X., Yeoh, W., Zhang, N., & Vorobeychik, Y. (2026) · arXiv · doi:10.48550/arxiv.2602.15143
Aruna Gladys, A., Aishwarya, R., Vetriselvi, V., Daphne, S., Baskar, P. N., & Rajan, A. A. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10598-4
A survey of social network alignment methods based on graph representation learning
Wu, Y., Li, F., Shi, Z., Tian, Z., Zhang, W., Fang, P., Xiao, R., wang, F., & Dan, F. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-40985-2
[Emerging Ideas] IoTGen: Towards LLM-driven IoT Hardware Generation
Luo, Q., Ma, R., Zhang, X., & Qiu, L. (2026) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3745756.3809239
Wu, A., Luo, S.-N., Li, J., & Jin, Y. (2026) · 電腦學刊 · doi:10.63367/199115992026083704006
Han, M., & Kim, H. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.795
Heravi, M. Y., Jeong, I., Jang, Y., & Moon, H. (2026) · Advanced Engineering Informatics · doi:10.1016/j.aei.2026.105371
- Authors
- Chen, X., Sun, Z., Wenjin, G., Zhang, M., Chen, Y., Sun, Y., Su, H., Pan, Y., Klakow, D., Li, W., & Shen, X.
- Affiliations
- Saarland University (DE)
- Hong Kong Polytechnic University (HK)
- Meituan (China) (CN)
- Venue
- Findings of the Association for Computational Linguistics: ACL 2025
- Published
- 2025
- Pages
- 15094-15119
- OpenAlex
- W4412888171
- Cited by
- 7
- References
- 0
- Citations by year
- 2026: 7
- FWCI
- 6.64
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- Funders
- Hong Kong Polytechnic University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- chain-of-thought reasoning, distillation, knowledge distillation, large language models, reasoning models, model compression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
