Notes · Method note · self-distillation

Self-distillation, and the harness it quietly depends on

A model trained to match its own predictions under a richer condition — an older copy of itself, a prompt it will not have at test time, the feedback it just received, the tools it just called. No external teacher, a dense per-token signal, and a built-in anchor against forgetting. In 2026 it moved from a regularizer to a post-training stage. This note is about what it optimizes, why it forgets less than RL until suddenly it forgets more, and why the thing being distilled is usually not the model alone but the model plus its harness — so the two have to be versioned together.

Fig 1 · One set of weights, two conditions

The whole family is one diagram with one free slot. The student is the model as it will be deployed. The teacher is the same weights given something extra — and the loss asks the deployed model to behave, without the extra, the way it already behaves with it. Sampling the rollout from the student (rather than from a dataset) is what makes it on-policy; that choice, from GKD, is what turned self-distillation from a regularizer into a training stage.

The math in three lines

The objective. Sample from the student, score with the teacher, match per token. \(c\) is the privileged condition; the teacher is stop-gradient. \[ \mathcal{L}(\theta)=\mathbb{E}_{x}\;\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}\;\sum_{t}\; D\!\left(\pi_{\bar\theta}(\cdot\mid x,c,y_{<t})\;\middle\|\;\pi_\theta(\cdot\mid x,y_{<t})\right) \] The teacher is what you choose to put in \(c\). Three instantiations cover almost everything published: \[ \pi_{\bar\theta}(\cdot\mid x,c)=\begin{cases}\pi_{\theta_{\text{old}}}(\cdot\mid x) & \text{previous release — the forgetting anchor (LwF)}\\[2pt] \pi_{\theta}(\cdot\mid \text{prompt},x) & \text{context you will not have at test time (context distillation)}\\[2pt] \pi_{\theta}(\cdot\mid x,y,\text{feedback}) & \text{the model reading its own result (SDPO)}\end{cases} \] Why it is gentler than RL, and why that can flip. The per-token target is a full distribution, not a scalar; the gradient is a projection onto a function the model already computes, so each step moves little. But when the teacher is the same moving weights, dense matching can chase its own artifacts — the loop is self-reinforcing. Denser ≠ Better measures exactly this: under continual post-training SDPO drifted more than GRPO in both parameter and response space, and the fix (StableSDPO) is to freeze or periodically restart the teacher. \[ \underbrace{\text{dense signal}}_{\text{fast in-domain}}\;+\;\underbrace{\text{teacher}=\text{student}}_{\text{no fixed point}}\;\Rightarrow\;\text{drift};\qquad \text{fix: } \bar\theta \leftarrow \theta_{k}\ \text{held for a window, or } \theta_{\text{old}}\ \text{throughout.} \]

Why it is kept

No external teacher. The 2026 frontier open models all distill, and the strong ones distill from themselves or their own larger sibling — there is no bigger model to call. Self-distillation is the version that scales past the biggest thing you have.
Dense where RL is sparse. A verifier gives one bit per rollout; the teacher gives a distribution per token. Thinking Machines' measurements put on-policy distillation at a fraction of RL's compute for the same reasoning gain, and it is the mechanism they name for continual learning and personalization: re-teach a specialized skill after a general update without losing the update.
It internalizes context. Context distillation (Snell, Klein, Zhong 2022) showed a model can absorb an instruction, a set of in-context examples, or a scratchpad into its weights by matching its own context-conditioned outputs. That is the same loss with \(c\) = the prompt — and it is why the harness question below is not separate from the method.
It is the oldest continual-learning trick there is. Learning without Forgetting (2016) is a KL to the previous network on new-task inputs. Every "anchor to the old policy" term since — including the KL-to-reference in RLHF — is a special case.
Feedback becomes a teacher. SDPO's move: condition the model on the compiler error, the failed test, or the next user turn, and distill that hindsight-informed distribution back into the un-conditioned policy. Rich feedback that RL would have to scalarize is used whole.

Fig 2 · The harness evolves too — version the pair, not the model

A deployed capability is a property of a pair — weights and the harness they run inside. Post-training in stages changes both: the model moves with each climb, and the harness moves with each new tool, template or verifier. Self-distillation is the natural coupling: run the previous model inside the next harness and distill what it does there into the next model, while anchoring to what it did in the old pair. The two failure modes are the two things the loop can get wrong — forgetting what only the prompt knew, and amplifying what only the loop knows.

Why this framing matters for continual post-training. Most forgetting measurements hold the harness fixed and vary the weights. But in an agentic stack the harness is where a surprising amount of behaviour lives — the system prompt that names the tools, the template that marks tool outputs, the verifier that defines "correct". Change it and a model can lose a skill without a single weight moving. Context distillation is the tool for moving that behaviour into the weights before the harness changes; a KL to the old pair is the tool for keeping it there. Treating (model, harness) as the unit of versioning makes both steps explicit, and makes "does the next post-training benefit from the previous one" a well-posed question: the previous pair is the teacher.

Where it appears

Work	Teacher \(c\)	Sampling	What it showed	Continual-learning note
Learning without Forgetting (Li & Hoiem 2016)	previous network on new-task inputs	dataset	new task without old data, old outputs preserved by KL	the origin; every KL-to-reference since is this
Born-Again Networks (Furlanello et al. 2018) · Mobahi et al. 2020	identical architecture, previous generation	dataset	students beat teachers; theory: repeated self-distillation is progressive regularization in RKHS, then underfitting	too many generations over-smooth — an early "denser is not better"
Context distillation (Snell, Klein, Zhong 2022)	same model + instruction / examples / scratchpad	student, on-policy	prompts and CoT internalized into weights	the harness-into-weights step of Fig 2
GKD (Agarwal et al., ICLR 2024)	a separate teacher	student rollouts	on-policy sampling fixes exposure bias; choice of divergence matters	defines the on-policy form all of 2026 uses
Thinking Machines · On-Policy Distillation (Oct 2025)	larger sibling, or the pre-update self	student	reasoning at a fraction of RL compute; recovers an assistant skill after a domain update	names continual learning and personalization as the use case
SDPO (Hübotter et al., ETH LAS, 2026)	same model + rich feedback (compiler, tests)	student	beats RLVR baselines on science, tool use, LiveCodeBench v6	feedback used whole instead of scalarized
Aligning LMs from User Interactions (Buening et al. 2026)	same model + the next user turn	student	real WildChat conversations improve alignment and IF benchmarks	the teacher signal arrives after deployment — continual by construction
Denser ≠ Better (Wang et al. 2026)	SDPO vs GRPO under sequential domains	student	SDPO forgets more than GRPO; larger drift in parameter and response space; "intermediate-distance interference"	StableSDPO: freeze / restart the teacher. The cautionary result for the whole family
Feedback Alignment in Self-Distillation (2026) · Pass-Rate-Weighted SD (2026) · Early-Stopping Rollout for OPD (2026)	SDPO-style	student	when the feedback-conditioned teacher actually agrees with the outcome; weighting by pass rate; shorter rollouts suffice	the knobs that keep the loop from chasing itself
Frontier reports: Kimi K3 (MOPD), DeepSeek V4, GLM-5	own larger / earlier checkpoints	student	on-policy distillation as a named stage, incl. across stages	see the report diff for per-model detail

Pitfalls

The teacher moves. When teacher and student share weights, dense matching has no fixed point and can amplify formatting quirks until the model collapses onto them. Hold the teacher for a window or pin it to the last release; measure response-space drift, not just loss.
Forgetting is not monotone in distance. Denser ≠ Better found the worst losses on tasks moderately related to the training domain, not on distant ones. A held-out suite that only checks far-away skills will miss it.
Long, variable reasoning traces distill badly. CoT distillation helped structured tool-use tasks and hurt math and science in the same study. Match the trace length to what the student can imitate, or distill outcomes rather than traces there.
Harness leak masquerades as forgetting. If a skill existed only under a system prompt or template that has since changed, no amount of weight-space anchoring recovers it. Evaluate the previous model under the new harness before blaming the update.
On-policy data alone is not enough. The continual-learning result is explicit on this: self-generated rollouts do not substitute for some replay of old inputs. Which old inputs to replay — and whether a distillation target beats a replayed label — is the open question, and the one closest to the classic continual-learning literature.
Consolidating experts is a different problem from anchoring one. Distilling several specialist checkpoints into one student has no single "previous self"; the teachers disagree. The number of experts, their overlap, and how to weight their distributions is not settled anywhere public.

Sources

Reinforcement Learning via Self-Distillation (SDPO) — Hübotter, Lübeck, Behric, Baumann, Bagatella, Marta, Hakimi, Shenfeld, Kleine Buening, Guestrin, Krause, 2026 · alphaXiv · code

Aligning Language Models from User Interactions — Kleine Buening, Hübotter, Pásztor, Shenfeld, Ramponi, Krause, 2026 · alphaXiv

Denser ≠ Better: Limits of On-Policy Self-Distillation for Continual Post-Training — Wang et al., 2026 · alphaXiv

The Role of Feedback Alignment in Self-Distillation · Pass-Rate Weighted Self-Distillation · Early Stopping Rollout for On-Policy Distillation · Optimality of On-Policy Distillation for Noisy Expert Feedback — the 2026 wave

On-Policy Distillation — Thinking Machines Lab, Oct 2025 · blog

On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD) — Agarwal et al., ICLR 2024 · alphaXiv

Learning by Distilling Context — Snell, Klein, Zhong, 2022 · alphaXiv

Learning without Forgetting — Li & Hoiem, 2016 · Born-Again Neural Networks — Furlanello et al., 2018 · Self-Distillation Amplifies Regularization in Hilbert Space — Mobahi, Farajtabar, Bartlett, 2020

Related notes here: self-play for reasoning · post-training algorithms · LLM report diff

Everything on this page is from public papers and posts. Nothing here draws on unpublished material.