CoChat
Sign Up
PreprintCited by 18Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation

Open access
Shen, Zhenyi, Yan, Hanqi, Zhang, Linhai, Hu, Zhanghao, Du, Yali, He, Yulan·2025·arXiv
DOI 10.48550/arxiv.2502.21074arXiv 2502.21074
Open this paper in CoChat — free accountOpen PDF

Abstract

Chain-of-Thought (CoT) reasoning enhances Large Language Models (LLMs) by encouraging step-by-step reasoning in natural language. However, leveraging a latent continuous space for reasoning may offer benefits in terms of both efficiency and robustness. Prior implicit CoT methods attempt to bypass language completely by reasoning in continuous space but have consistently underperformed compared to the standard explicit CoT approach. We introduce CODI (Continuous Chain-of-Thought via Self-Distillation), a novel training framework that effectively compresses natural language CoT into continuous space. CODI jointly trains a teacher task (Explicit CoT) and a student task (Implicit CoT), distilling the reasoning ability from language into continuous space by aligning the hidden states of a designated token. Our experiments show that CODI is the first implicit CoT approach to match the performance of explicit CoT on GSM8k at the GPT-2 scale, achieving a 3.1x compression rate and outperforming the previous state-of-the-art by 28.2% in accuracy. CODI also demonstrates robustness, generalizable to complex datasets, and interpretability. These results validate that LLMs can reason effectively not only in natural language, but also in a latent continuous space. Code is available at https://github.com/zhenyi4/codi.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.21074

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Comparison of reasoning strategies. No-CoTSFT : Train model on (Q,A) pairs via SFT. CoT-SFT : Train model on (Q, CoT, A) triples via SFT, i.e., with explicitly annotated CoT reasoning steps. Coconut : requires multi-stage training to progressively replace CoT tokens with continuous representations. CODI : achieves this in a single stage by compressing CoT tokens into continuous space via self-distillation.
Figure 1 Comparison of reasoning strategies. No-CoTSFT : Train model on (Q,A) pairs via SFT. CoT-SFT : Train model on (Q, CoT, A) triples via SFT, i.e., with explicitly annotated CoT reasoning steps. Coconut : requires multi-stage training to progressively replace CoT tokens with continuous representations. CODI : achieves this in a single stage by compressing CoT tokens into continuous space via self-distillation.CC-BY · From the PDF
Figure 2 CODI enables the model to generate implicit continuous CoTs by jointly training a student task and a teacher task, and distills knowledge from the teacher to the student. The Student task (left) generates the answer by autoregressively decoding continuous thoughts starting from a learnable bot token, while the Teacher task (right) generates the answer using the groundtruth CoT via teacher forcing. Both tasks learn the generated texts via cross-entropy loss ( L student and L teacher ), and share the same LLM. Knowledge distillation is achieved by applying L KD (L1 loss) between student and teacher hidden activation across all layers ( h student and h teacher ).
Figure 2 CODI enables the model to generate implicit continuous CoTs by jointly training a student task and a teacher task, and distills knowledge from the teacher to the student. The Student task (left) generates the answer by autoregressively decoding continuous thoughts starting from a learnable bot token, while the Teacher task (right) generates the answer using the groundtruth CoT via teacher forcing. Both tasks learn the generated texts via cross-entropy loss ( L student and L teacher ), and share the same LLM. Knowledge distillation is achieved by applying L KD (L1 loss) between student and teacher hidden activation across all layers ( h student and h teacher ).CC-BY · From the PDF
Figure 3 Results on five datasets ( Top : GPT-2, Bottom : LLaMa3.2-1b-Instruct). CODI consistently outperforms all previous implicit CoT methods by a substantial margin. When using GPT-2, CODI even matches the performance of CoT-SFT on the in-domain GSM8k and GSM8k-NL datasets.
Figure 3 Results on five datasets ( Top : GPT-2, Bottom : LLaMa3.2-1b-Instruct). CODI consistently outperforms all previous implicit CoT methods by a substantial margin. When using GPT-2, CODI even matches the performance of CoT-SFT on the in-domain GSM8k and GSM8k-NL datasets.CC-BY · From the PDF
Figure 4 Efficiency comparison of different reasoning methods in terms of inference time per math problem on GSM8k. Measured with batch size = 1 on an Nvidia A100 GPU. CoT Token counts are shown in parentheses.
Figure 4 Efficiency comparison of different reasoning methods in terms of inference time per math problem on GSM8k. Measured with batch size = 1 on an Nvidia A100 GPU. CoT Token counts are shown in parentheses.CC-BY · From the PDF
Figure 5 Accuracy on GSM8k against the number of continuous thought tokens used during training.
Figure 5 Accuracy on GSM8k against the number of continuous thought tokens used during training.CC-BY · From the PDF
Figure 6 A case study illustrating CODI's interpretability by analyzing its attended tokens and decoded tokens of each of the six latent thought tokens, z 1 · · · z 6 . Attended tokens : these represent the top-10 tokens that the continuous thought attends to when generating the next thought/token. Some attended tokens appear in the form of ' z i = x ', indicating attention to the i -th continuous thought. Here x represents the top-1 token that the latent thought maps to in vocabulary space. The model always attends to the first token in the sentence, so we remove that for better visualization. Decoded tokens : these are the top-5 words that the continuous thoughts are projected back to in vocabulary space by multiplying them with the vocabulary embeddings.
Figure 6 A case study illustrating CODI's interpretability by analyzing its attended tokens and decoded tokens of each of the six latent thought tokens, z 1 · · · z 6 . Attended tokens : these represent the top-10 tokens that the continuous thought attends to when generating the next thought/token. Some attended tokens appear in the form of ' z i = x ', indicating attention to the i -th continuous thought. Here x represents the top-1 token that the latent thought maps to in vocabulary space. The model always attends to the first token in the sentence, so we remove that for better visualization. Decoded tokens : these are the top-5 words that the continuous thoughts are projected back to in vocabulary space by multiplying them with the vocabulary embeddings.CC-BY · From the PDF
Figure A2: CODI's interpretability on problems involving two steps.
Figure A2: CODI's interpretability on problems involving two steps.CC-BY · From the PDF
Figure A3: CODI's interpretability on problems involving one step.
Figure A3: CODI's interpretability on problems involving one step.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 43

  1. Anthropic. 2024. Claude 3.5 sonnet. Jeffrey Cheng and Benjamin Van Durme. 2024. Com- pressed chain of thought: Efficient reasoning through dense representations. Preprint, arXiv:2412.13171. · arXiv
  2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems. ArXiv, abs/2110.14168. · arXiv
  3. Michael Crawshaw. 2020. Multi-task learning with deep neural networks: A survey. ArXiv, abs/2009.09796. · arXiv
  4. Yuntian Deng, Yejin Choi, and Stuart Shieber. 2024. From explicit cot to implicit cot: Learning to inter- nalize cot step by step. ArXiv, abs/2405.14838. · arXiv
  5. Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. 2023. Implicit chain of thought reasoning via knowl- edge distillation. ArXiv, abs/2311.01460. · arXiv
  6. Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić. 2025. UNDIAL: Self-distillation with adjusted logits for robust un- learning in large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), pages 8827-8840, Albuquerque, New Mexico. Association for Computational Linguistics. · DOI
  7. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2023. PAL: Program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10764-10799.
  8. PMLR.
  9. Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2024. In-context autoencoder for con- text compression in a large language model. In The Twelfth International Conference on Learning Repre- sentations.
  10. Jonas Geiping, Sean Michael McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Gold- stein. 2025. Scaling up test-time compute with latent reasoning: A recurrent depth approach. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models. Google. 2024. Our next-generation model: Gemini 1.
  11. Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789-1819. · DOI · Paper page
  12. Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Kr- ishna Menon, Sanjiv Kumar, and Vaishnavh Nagara- jan. 2024. Think before you speak: Training lan- guage models with pause tokens. In The Twelfth International Conference on Learning Representa- tions.
  13. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. 2024. Training large language models to reason in a contin- uous latent space. Preprint, arXiv:2412.06769. · arXiv
  14. Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. Large language models are reasoning teachers. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 14852-14882, Toronto, Canada. Association for Computational Linguistics. · DOI
  15. Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Dis- tilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Compu- tational Linguistics: ACL 2023, pages 8003-8017, Toronto, Canada. Association for Computational Lin- guistics. · DOI · Paper page
  16. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  17. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1152-1157, San Diego, California. Association for Computational Linguistics. · DOI
  18. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics. · DOI · Paper page
  19. Dongfang Li, zhenyu liu, Xinshuo Hu, Zetian Sun, Bao- tian Hu, and Min Zhang. 2024a. In-context learning state vector with inner and momentum optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  20. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics. · DOI · Paper page
  21. Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. 2024b. Chain of thought empowers transformers to solve inherently serial problems. In The Twelfth In- ternational Conference on Learning Representations.
  22. Zongqian Li, Yixuan Su, and Nigel Collier. 2025. 500xCompressor: Generalized prompt compression for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 25081-25091, Vienna, Austria. Association for Com- putational Linguistics. · DOI
  23. Zicheng Lin, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. 2025. Critical tokens matter: Token-level contrastive estimation enhances LLM's reasoning capability. In Forty-second International Conference on Machine Learning.
  24. Luyang Liu, Jonas Pfeiffer, Jiaxing Wu, Jun Xie, and Arthur Szlam. 2025. Deliberation in latent space via differentiable cache augmentation. In Forty-second International Conference on Machine Learning.
  25. Sheng Liu, Haotian Ye, Lei Xing, and James Y. Zou. 2023. In-context vectors: Making in context learning more effective and controllable through latent space steering. ArXiv, abs/2311.06668. · arXiv
  26. William Merrill and Ashish Sabharwal. 2024. The ex- pressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations.
  27. Meta. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. · arXiv
  28. Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Re- ichart, Idan Szpektor, Hadas Kotek, and Yonatan Be- linkov. 2025. LLMs know more than they show: On the intrinsic representation of LLM hallucinations. In The Thirteenth International Conference on Learning Representations.
  29. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744.
  30. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, Online. Association for Computational Linguistics. · DOI · Paper page
  31. Jacob Pfau, William Merrill, and Samuel R. Bowman. 2024. Let's think dot by dot: Hidden computation in transformer language models. In First Conference on Language Modeling.
  32. Anvesh Rao Vijjini, Kaveri Anuranjana, and Radhika Mamidi. 2021. Analyzing curriculum learning for sentiment analysis along task difficulty, pacing and visualization axes. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjec- tivity, Sentiment and Social Media Analysis, pages 117-128, Online. Association for Computational Lin- guistics.
  33. Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J. Reddi. 2025. Reasoning with latent thoughts: On the power of looped transform- ers. In The Thirteenth International Conference on Learning Representations.
  34. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 31.
  35. Lena Strobl, William Merrill, Gail Weiss, David Chi- ang, and Dana Angluin. 2024. What formal lan- guages can transformers express? a survey. Transac- tions of the Association for Computational Linguis- tics, 12:543-561. · DOI
  36. DiJia Su, Hanlin Zhu, Yingchen Xu, Jiantao Jiao, Yuan- dong Tian, and Qinqing Zheng. 2025. Token as- sorted: Mixing latent and text tokens for improved language model reasoning. In Forty-second Interna- tional Conference on Machine Learning.
  37. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI
  38. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484-13508, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  39. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824-24837.
  40. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. Preprint, arXiv:2402.13116. · arXiv
  41. Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. 2025. SoftCoT: Soft chain-of-thought for efficient reasoning with LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23336- 23351, Vienna, Austria. Association for Computa- tional Linguistics. · DOI
  42. Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, and Qian Liu. 2024. Self- distillation bridges distribution gap in language model fine-tuning. In Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1028- 1043, Bangkok, Thailand. Association for Computa- tional Linguistics. · DOI
  43. Qifan Yu, Zhenyu He, Sijie Li, Xun Zhou, Jun Zhang, Jingjing Xu, and Di He. 2025. Enhancing auto- regressive chain-of-thought through loop-aligned rea- soning. ArXiv, abs/2502.08482. · arXiv

Cited by

Sorted by citations

  • Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models

    Sui, Y., Chuang, Y.-N., Wang, G., Zhang, J., Zhang, T., Yuan, J., Liu, H., Wen, A., Zhong, S., Zou, N., et al. (2025) · arXiv · Cited by 517

    “…TokenSkip [95]; CoT-Valve [59]; Self-Training [62]; Learn to Skip [53]; Token-Budget [34]; ReasoningOutput-based Latent e.g. Coconut [35]; CODI [75]; CCoT [12]; Heima [73]; Token Assorted [79]; Loop [70]; SoftCoT [100]; DynamicReasoningParadigm e.g. Speculative Rejection [81];…”

  • CROP: Token-Efficient Reasoning in Large Language Models via Regularized Prompt Optimization

    Shah, D., Badhe, S., Kathrotia, N., & Tiwari, P. (2026) · arXiv

    “Furthermore, recent literature focuses on explicitly compressing reasoning traces via continuous latent projections (Hao et al., 2024; Shen et al., 2025; Wei et al., 2025), conditioned textual pruning (Kang et al., 2024; Xia, 2025; Wang, 2025”

Metadata

Authors
Shen, Z., Yan, H., Zhang, L., Hu, Z., Du, Y., & He, Y.
Affiliations
King's College London (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
677-693
DOI
10.48550/arxiv.2502.21074
arXiv
2502.21074
OpenAlex
W4416037467
Cited by
18
References
1
Citations by year
2026: 18
FWCI
17.09
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Research Portal (King's College London) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • UK Research and Innovation (EP/V020579/1)
  • King's College London
  • Engineering and Physical Sciences Research Council (EP/V020579/1, EP/V020579/2, EP/Y003187/1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
chain-of-thought reasoning, continuous latent space, self-distillation, knowledge distillation, large language models, GSM8K, teacher-student training, interpretability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review