Post-training algorithms

Maximize log-likelihood of curated prompt–response pairs; token-level cross-entropy.

Sample K responses, keep those a verifier accepts, SFT on them; iterate.

SFT on a stronger teacher's traces.

Distill on student-generated sequences, per-token, with reverse KL or JSD instead of forward KL on teacher data.

Student samples; reward per token = \(\operatorname{sg}[\log \pi _{T}(y_{t}) - \log \pi _{\theta}(y_{t})]\); optimize with the RL machinery.

\(\sum _{i} w_{i} \cdot \mathrm{KL}(\pi _{\theta} \| \pi _{E_i})\) on student samples, one teacher per domain (and per effort level).

Same objective, but the teachers are the model's own final checkpoints from earlier stages (SFT, reasoning RL, general RL).

Preference-tune on pairs whose *difference* is informative even when both sides are weak: chosen from a stronger model, rejected from a much weaker one.

Turn-level RL over offline SFT expert trajectories: update only at ‘pivot’ turns where the policy is uncertain, reward similarity to the expert action.

Train \(r_{\phi}\) so that \(\sigma (r(y_{w}) - r(y_{l}))\) matches human pairwise preferences.

Learned reward models that score a whole solution's outcome (ORM) or each reasoning step (PRM); step labels come from humans, Monte-Carlo rollouts, or a judge.

Replace the reward model with a checker: math answer match, unit tests, constraint verifiers.

A language model scores or compares responses, optionally with chain-of-thought before the verdict.

Judge against a per-prompt checklist rather than a holistic score.

No separate reward model: apply RL to the policy's own judging, so generating and evaluating are trained jointly.

Average several fine-tuned RMs' weights for a more robust, less hackable reward.

Shape reward by output length: penalize overlong correct answers, reward longer thinking when wrong, or hard −1 beyond a per-problem budget \(\tau \cdot b_{0}\).

Closed-form of KL-regularized RLHF: logistic loss on the implicit reward \(\beta \cdot \log \pi _{\theta}/\pi _{\text{ref}}\) between chosen and rejected.

Regress the implicit reward gap onto a constant \(1/(2\beta )\) instead of a logistic loss.

Prospect-theory loss on unpaired ‘good’ / ‘bad’ examples — no pairs needed.

Odds-ratio penalty added to the SFT loss; no reference model.

Length-normalized implicit reward with a target margin; no reference model.

Contrastive objectives that specify whether the chosen response should rise, the rejected fall, or both, relative to the model's current outputs.

Regress the implicit reward gap onto the reward model's actual gap, not just the binary order.

Regress \(\beta \cdot \log \pi _{\theta}/\pi _{\text{ref}}\) onto a quantile-transformed absolute reward, for which the KL-regularized optimum's partition function is exact.

Regenerate pairs from the current policy each round, judge them, DPO again.

\(\nabla J = \mathbb{E}[(r - b)\cdot \nabla \log \pi (y|x)]\); a baseline b reduces variance.

Baseline = reward of the greedy decode of the same prompt.

Baseline for sample k = mean reward of the other K−1 samples.

REINFORCE with PPO's token-level KL penalty and reward normalization, no critic.

PPO-style clipped ratio per token; advantage = \((R_{i} - \operatorname{mean}_{G}) / \operatorname{std}_{G}\) broadcast to all tokens; KL term in the loss.

GRPO minus \(1/|o_{i}|\) and minus \(/\operatorname{std}_{G}\).

Clip-higher \((\varepsilon _{\text{high}} 0.28)\), dynamic sampling, token-level loss, overlong shaping, no KL.

Importance ratio and clipping at the sequence level: \(s_{i} = (\pi _{\theta}(y_{i})/\pi _{\text{old}}(y_{i}))^{1/|y_{i}|}\).

Maximize the geometric rather than arithmetic mean of token-level rewards.

Replace hard clipping with a smooth temperature-controlled gate on off-policy tokens.

Clip the importance weight, detach it, and keep the plain REINFORCE gradient for every token.

Clipped surrogate with GAE advantages from a learned value function; token-level KL to a reference.

PPO plus: value pretraining, decoupled GAE (\(\lambda =1\) for the critic, length-adaptive \(\lambda\) for the policy), clip-higher, token-level loss, positive-example LM loss, group sampling.

Multiply the loss by \(\min (\pi _{\text{learner}}/\pi _{\text{sampler}}, C)\) (TIS), or mask tokens whose ratio leaves [1/C, C] (MIS).

Zero out tokens whose train/infer ratio is outside \([1/\beta , \beta ]\) and clip the rest; \(\beta = 2\) in GLM-5.

Ratio against the rollout log-probs directly (no stored old policy), mask outside \([1-\varepsilon _{l}, 1+\varepsilon _{h}]\).

Control the variance of importance weights under high policy lag instead of truncating them.

Pause generation when a fraction \(\lambda\) of the N×K trajectories finish; resume the stragglers next iteration.

Record the MoE routing (or sparse-attention top-k) chosen at rollout and replay it in training.

Drop prompts the current model already solves (or never solves) before RL; sort the rest by pass rate.

Refill the batch with prompts whose group reward has non-zero variance.

Raise \(\varepsilon _{\text{high}}\) so low-probability tokens can grow; or preserve gradients from clipped tokens with a bounded coefficient.

Train one policy on all environments at once rather than sequentially.

Let a proposer manufacture the curriculum at the solver's 50% boundary.

Distill the best-of-N sampling policy into the model via Jeffreys divergence.

EMA anchors, spherical interpolation of independently RL'd policies, interpolate toward init.

Linearly average checkpoints.

After RL, retrain only the multi-token-prediction heads with the trunk frozen.