The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Teaching Small Language Models Reasoning through Counterfactual Distillation
Open accessAbstract
With the rise of large language models (LLMs), many studies are interested in transferring the reasoning capabilities of LLMs to small language models (SLMs).Previous distillation methods usually utilize the capabilities of LLMs to generate chain-of-thought (CoT) samples and teach SLMs via fine-tuning.However, such a standard distillation approach performs poorly when applied to out-of-distribution (OOD) examples, and the diversity of the generated CoT samples is insufficient.In this work, we propose a novel counterfactual distillation framework.Firstly, we leverage LLMs to automatically generate high-quality counterfactual data.Given an input text example, our method generates a counterfactual example that is very similar to the original input, but its task label has been changed to the desired one.Then, we utilize multi-view CoT to enhance the diversity of reasoning samples.Experiments on four NLP benchmarks show that our approach enhances the reasoning capabilities of SLMs and is more robust to OOD data.We also conduct extensive ablations and sample studies to understand the reasoning capabilities of SLMs.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Counterfactual distillation, augmenting the training set by masking causal features and generating multi-view CoT, enhances the effectiveness of knowledge distillation.
“In comparison, [48] introduced counterfactual distillation, augmenting the training set by masking causal features in the original question, prompting the LLM to complete the masked text, and generating multi-view CoT (positive and negative views) of each data for enhancing the effectiveness of knowledge distillation.”
From Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (2025) · cites this paper as [48]
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Sorted by citations
Efficient Inference for Large Reasoning Models: A Survey
Liu, Y., Wu, J. F., He, Y., Gong, R., Xia, J., Li, L., Gao, H., Chen, H., Bi, B., Zhang, J., et al. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3701928 · Cited by 73
Self-citationStop Overthinking: A Survey on Efficient Reasoning for Large Language Models
Sui, Y., Chuang, Y.-N., Wang, G., Zhang, J., Zhang, T., Yuan, J., Liu, H., Wen, A., Zhong, S., Zou, N., et al. (2025) · arXiv · Cited by 517
“In comparison, [28] introduced counterfactual distillation, augmenting the training set by masking causal features in the original question, prompting the LLM to complete the masked text, and generating multi-view CoT (positive and negative views) of each data for enhancing the effectiveness of…”
Self-citationLiang, X., Zou, B., Wu, L., Na, X., Cao, W., Bao, T., Wu, Y., & Wei, B. (2025) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2025.12.484
R2Dia: Reasoning-aware reinforcement learning for dialogue-level sentiment quadruple analysis
Melnik, E., Xu, J., & Lu, P. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.105113
Han, M., & Kim, H. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.795
Metadata
- Authors
- Feng, T., Li, Y., Chenglin, L., Chen, H., Yu, F., & Zhang, Y.
- Affiliations
- Zhejiang University (CN)
- Ant Group (China) (CN)
- Venue
- Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
- Published
- 2024
- Pages
- 5831-5842
- OpenAlex
- W4404784411
- Cited by
- 3
- References
- 0
- Citations by year
- 2026: 2 · 2025: 1
- FWCI
- 0.87
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- dx.doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- Ant Group
- National Natural Science Foundation of China (62072399)
- Ministry of Education, India
- Fundamental Research Funds for the Central Universities
- Natural Science Foundation of Zhejiang Province
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- small language models, knowledge distillation, chain-of-thought, out-of-distribution robustness, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
