Purified OPSD: On-Policy Self-Distillation Without Losing How to Think

“Knowledge distillation [Hinton et al., 2015] transfers knowledge from a teacher to a student via soft label matching.”

“Chain-of-thought (CoT) prompting [Wei et al., 2022] demonstrated that eliciting intermediate reasoning steps substantially improves LLM performance on complex tasks.”

“All methods use LoRA [Hu et al., 2022] adaptation with rank 64.”

“On-policy trajectories are generated using vLLM [Kwon et al., 2023].”

“…reward while remaining close to the clean base distribution P 0 : This objective is the token-level analogue of KL-regularized policy improvement used in RLHF and DPO-style derivations [Rafailov et al., 2024], with P 0 serving as the reference policy and ∆ it serving as the implicit reward.”

“…model (LLM) reasoning have demonstrated that long chain-of-thought (long-CoT) reasoning, where models engage in extended, reflective thinking processes before arriving at answers, can substantially improve performance on complex reasoning tasks [Jaech et al., 2024, Guo et al., 2025, Team, 2024].”

“We compare three configurations: (1) Base : the base model without any distillation, (2) OPSD-Standard : standard on-policy self-distillation [Zhao et al., 2026] with JSD loss, and (3) OPSD-PMI (Ours) : on-policy self-distillation with our PMI target (Section 3.1).”

“Supervised approaches include fine-tuning on curated long-CoT traces [Muennighoff et al., 2025, Ye et al., 2025, Xu et al., 2025, Wen et al., 2025], structured decomposition of reasoning chains [Luo et al., 2025b, Chen et al., 2025b], data-efficient frameworks [Wu et al., 2025], progressive…”

“PMI has been widely used in NLP for collocation detection [Church and Hanks, 1990], word embedding evaluation [Levy and Goldberg, 2014], and more recently for analyzing the information content of language model predictions [Holtzman et al., 2021].”

“This finding is corroborated by concurrent works [Kaur et al., Kim et al., 2026a], which report similarly limited or negative outcomes when applying OPSD to thinking models.”

“On-Policy Self-Distillation (OPSD) [Agarwal et al., 2024] extends this paradigm by having the student generate its own trajectories and receive token-level feedback from a privileged teacher with access to reference solutions, directly addressing the distribution mismatch problem inherent in…”

“Several concurrent works have extended OPSD: Zhao et al. [2026] proposed Self-Distilled Reasoner for general LLM reasoning, He et al. [2026] introduced Self-Distillation Zero combining self-revision with binary rewards to provide dense supervision, and Cui et al. [2026] provided a comprehensive…”

“…[Muennighoff et al., 2025, Ye et al., 2025, Xu et al., 2025, Wen et al., 2025], structured decomposition of reasoning chains [Luo et al., 2025b, Chen et al., 2025b], data-efficient frameworks [Wu et al., 2025], progressive training through the “valley” of capability [Luo et al., 2025a],…”

“…model (LLM) reasoning have demonstrated that long chain-of-thought (long-CoT) reasoning, where models engage in extended, reflective thinking processes before arriving at answers, can substantially improve performance on complex reasoning tasks [Jaech et al., 2024, Guo et al., 2025, Team, 2024].”

“…long-CoT traces [Muennighoff et al., 2025, Ye et al., 2025, Xu et al., 2025, Wen et al., 2025], structured decomposition of reasoning chains [Luo et al., 2025b, Chen et al., 2025b], data-efficient frameworks [Wu et al., 2025], progressive training through the “valley” of capability [Luo et…”

“Models such as OpenAI o1 [Jaech et al., 2024], DeepSeek-R1 [Guo et al., 2025], and QwQ [Team, 2024] have shown that scaling test-time compute through longer reasoning chains can be more effective than scaling model parameters [Snell et al., 2024].”

“…structured decomposition of reasoning chains [Luo et al., 2025b, Chen et al., 2025b], data-efficient frameworks [Wu et al., 2025], progressive training through the “valley” of capability [Luo et al., 2025a], chunk-wise distillation [Chen et al., 2025a], and long-CoT compression [Wang et al., 2025].”

“A key characteristic of long-CoT models is their use of epistemic markers, such as “ Wait ”, “ Perhaps ”, and “ Let me reconsider ”, that externalize uncertainty and enable reflective deliberation [Gandhi et al., 2024, Li et al., 2025].”

“We use two training datasets with reference solutions as privileged information: DASD-10K , a 10K-sample subset from DASD [Yan et al., 2026], and Math-CoT-20K [Ren et al., 2026], a 20K-sample collection of competition-level math problems with detailed chain-of-thought solutions.”

“…structured decomposition of reasoning chains [Luo et al., 2025b, Chen et al., 2025b], data-efficient frameworks [Wu et al., 2025], progressive training through the “valley” of capability [Luo et al., 2025a], chunk-wise distillation [Chen et al., 2025a], and long-CoT compression [Wang et al., 2025].”

“Supervised approaches include fine-tuning on curated long-CoT traces [Muennighoff et al., 2025, Ye et al., 2025, Xu et al., 2025, Wen et al., 2025], structured decomposition of reasoning chains [Luo et al., 2025b, Chen et al., 2025b], data-efficient frameworks [Wu et al., 2025], progressive…”

“…Xu et al., 2025, Wen et al., 2025], structured decomposition of reasoning chains [Luo et al., 2025b, Chen et al., 2025b], data-efficient frameworks [Wu et al., 2025], progressive training through the “valley” of capability [Luo et al., 2025a], chunk-wise distillation [Chen et al., 2025a], and…”

“2 Revisiting On-Policy Self-Distillation for Long-CoT Models On-Policy Self-Distillation (OPSD) [Agarwal et al., 2024, He et al., 2026, Zhao et al., 2026, Cui et al., 2026] has shown promise for improving reasoning capabilities through privileged teacher guidance.”

“Supervised approaches include fine-tuning on curated long-CoT traces [Muennighoff et al., 2025, Ye et al., 2025, Xu et al., 2025, Wen et al., 2025], structured decomposition of reasoning chains [Luo et al., 2025b, Chen et al., 2025b], data-efficient frameworks [Wu et al., 2025], progressive…”

“…et al., 2024] extends this paradigm by having the student generate its own trajectories and receive token-level feedback from a privileged teacher with access to reference solutions, directly addressing the distribution mismatch problem inherent in off-policy distillation [Yu et al., 2024].”

“They indicate uncertainty externalization and reflective reasoning, which are central to long-CoT models’ generalization ability [Gandhi et al., 2024, Li et al., 2025].”

“Purified OPSD decomposes the teacher distribution into a reference-induced component and a question-conditioned component intended to remain useful at inference time, and suppresses the former using an additional reference-only teacher (Shen et al., 2026).”

“The central premise of OPSD is that privileged reference information improves the teacher’s super-vision and transfers useful information to the student (Zhao et al., 2026a).”

“Recent OPSD methods refine either privileged super-vision or token weighting (Jin et al. 2026; Li et al. 2026; Xie et al. 2026; Lin, Chen, and Zhang 2026; Shen et al. 2026; Liu et al. 2026b).”

“Purified OPSD uses a reference-only view to control reference-specific shortcuts (Shen et al. 2026), while AR-OPD introduces a partially privileged anchor to constrain hindsight guidance that is difficult for the current policy to reach (Zhang 2026).”

“Ichihara et al. (2026), meanwhile, find that a worked solution, even a deliberately corrupted one, from another problem can retain much of the OPSD gain, showing that the problem’s own solution is not always necessary and that whatever the teacher transfers is not target-specific correctness.”

“…shortcuts into the supervision signal, and reduces the effective PI reach through mechanisms such as adaptive solution masking, learned exposure schedules, anchored residual application, and reference-signal subtraction (Tan and Hong 2026; Han et al. 2026; Zhang 2026; Shen et al. 2026).”

“While highly relevant, our approach differs fundamentally in mechanism: whereas Purified OPSD relies on teacher-side target reconstruction across all tokens, AD-OPSD operates directly on the student side.”

“Complementary methods revise token targets, hidden-state supervision, or the teacher itself (Shen et al., 2026; Li et al., 2026c; Feng et al., 2026; Sun et al., 2026).”

“OPSD is an on-policy diagnostic-teacher baseline in the OPD/OPSD family (Song and Zheng, 2026; Zhao et al., 2026; Shen et al., 2026; Zhou et al., 2026), evaluated as a baseline rather than as part of SCOPE.”

“Shen et al. (2026) decompose the teacher signal into reference-induced and question-conditioned components to purify long-chain reasoning super-vision, and Ichihara et al. (2026) show that a solution to a different problem can still improve the student.”

“Other variants control reference exposure, learn soft prompts, or remove reference-induced log-probability terms (Han et al., 2026; Ma et al., 2026; Shen et al., 2026).”

“” Training toward such targets can lead the problem-only student to mention a reference it never observes at inference (Yang et al. 2026; Shen et al. 2026b).”

“…equally well-documented, and they rhyme with §3’s. Purified OPSD shows the vanilla recipe consistently fails on long-CoT reasoning models , destabilizing the very reflective behaviors it is meant to strengthen [110]; the privileged teacher’s dense supervision can overfit in-domain patterns [111].”

“Recent work examines reference-induced shortcuts (Shen et al., 2026), reasoning degradation (Kaur et al., 2026), reduced output diversity (Nicolicioiu et al., 2026), and sensitivity to reasoning modes and teacher prefixes (Li et al., 2026b) in OPSD.”

“To avoid this external dependency, OPSD leverages privileged information to act as its own teacher, distilling a model to itself without increasing inference cost (Zhao et al., 2026a,b; Shen et al., 2026).”

“…of work in the direction of OPD is on-policy self-distillation (OPSD), which removes the separate teacher by conditioning the same model on privileged context such as a reference solution [Zhao et al., 2026], with unsupervised [Li et al., 2026c] and shortcut-purified [Shen et al., 2026] variants.”

“To mitigate this bias, Purified OPSD (Shen et al., 2026b) filters reference-driven shortcuts from training targets by comparing teacher predictions with and without the question.”

“Recent studies examine the stability and refinement of this supervision (Kim and Lee 2026; Shen et al. 2026; Kaur et al. 2026), while multimodal variants use privileged crops, zoomed views, or visual-thought traces (Yuan et al. 2026; Cai et al. 2026; Li et al. 2026).”

“Recent work responds to these failures by filtering the part of the teacher signal that does not transfer to the student (Shen et al., 2026) or by routing token-level supervision between privileged teacher and student policies (Yu et al., 2026).”

“…control to outcome rewards (Yang et al., 2026; Ke et al., 2026; He et al., 2026), effectively reintroducing RLVR. Signal-purification methods cancel privileged bias with an additional teacher pass (Pan et al., 2026; Shen et al., 2026), paying extra compute after the bias is already introduced.”

“To address this, Shen et al. (2026) isolate inference-transferable supervision from reference-induced shortcuts, while Kaur et al. (2026); Peng et al. (2026) focus on preserving high-entropy decision points and epistemic behaviors from aggressive privileged-teacher corrections.”

“Other approaches selectively correct the teacher signal to reduce style-or reference-dependent effects (Pan et al., 2026; Yang et al., 2026c; Shen et al., 2026).”

“Purified OPSD discounts reference-induced preferences and applies the remaining question-conditioned correction to a clean base prediction (Shen et al., 2026).”

“Building on this framework, Purified OPSD decomposes the privileged teacher’s signal into a reference-induced component and a genuinely transferable, question-conditioned reasoning component, distilling only the latter via a PMI-based reconstruction [21].”

“Following OPSD, we use Jensen–Shannon divergence (JSD) and clip the per-token divergence with a threshold δ clip to prevent a small number of tokens from dominating the update (Shen et al., 2026).”

“What transfers best is intermediate abstraction, meaning hints, plans and error-aligned critiques, which convey a skill without entailing the answer.”