Notes · Method note 01

Multi-token prediction

A normal language-model head predicts the next token. MTP adds a small extra head that predicts the token after that from the same hidden state. It costs almost nothing in parameters, it makes the training signal denser, and at inference it doubles as a draft model that lives inside the main model — which is why every 2026 frontier open model ships one.

Fig 1 · What is added at training time

DeepSeek-V3's sequential form, the one most 2026 models inherit. The grey parts already exist. The teal parts are the addition: the trunk's hidden state at t is concatenated with the embedding of the true next token, projected, passed through one transformer layer, and decoded by the same output head to predict x_{t+2}. Gloeckle et al.'s original form is simpler and more parallel — n independent heads straight off h_t, no chaining — but the sequential version keeps the causal chain intact, which is what makes it a usable drafter later.

The math in three lines

\[L \;=\; L_{\text{main}} \;+\; \lambda\cdot\sum_{k} L^{(k)}_{\text{MTP}}\] // \(k = 1\) for one extra token; DeepSeek-V3 uses \(k = 1\), Gloeckle et al. up to 4 \[h^{(k)}_t \;=\; \operatorname{Block}_k\!\Big(W_k\,\big[\,\operatorname{RMSNorm}\big(h^{(k-1)}_t\big)\,;\ \operatorname{RMSNorm}\big(\operatorname{Emb}(x_{t+k})\big)\,\big]\Big)\] // \(h_{t}^{(0)}\) is the trunk output \[p\big(x_{t+k+1}\mid\cdot\big) \;=\; \operatorname{softmax}\big(W_{\text{out}}\, h^{(k)}_t\big)\] // \(W_{\text{out}}\) is shared with the main head

\(\lambda\) is the only knob. DeepSeek-V3 uses 0.3 for the first 10T tokens and 0.1 after; Nemotron 3 Super uses 0.3 throughout SFT. The MTP block is one layer out of 60–90, so the parameter and FLOP overhead is a few percent.

Why it is kept

Denser supervision. Every position now yields two gradients instead of one, and the second one asks the representation to be usable two steps out — a mild pressure to “plan.” Gloeckle et al. report the effect grows with model size: for 13B, +12% on HumanEval and +17% on MBPP with four heads, and no gain (sometimes loss) at small sizes. DeepSeek-V3 reports consistent improvements across benchmarks from a single MTP module.
Free speculative decoding. At inference the MTP head is a draft model that shares the trunk: it proposes x_{t+2} for the price of one small block, and the main model verifies the proposal in the next forward pass. DeepSeek-V3: the second token is accepted 85–90% of the time across domains, giving about 1.8× tokens per second. No separate drafter to train, serve, or keep in sync.
It is inside the weights. A separate EAGLE-style drafter has to be retrained every time the main model changes; an MTP head is part of the checkpoint and can be re-healed with a short, cheap pass.

Fig 2 · What happens at inference

Verification is exact: the accepted sequence is distributed exactly as the main model would have sampled it, because the main model's own prediction is what decides. Throughput scales with the acceptance rate, which is why that — not the draft's standalone accuracy — is the number reports quote. Chaining more MTP steps drafts more tokens per pass but each additional draft is accepted less often.

Where it appears in the reports

Model	Form	Loss weight	Inference use	Post-training note
Gloeckle et al. 2024 (Meta)	n independent heads off \(h_{t}\), no chaining; \(n = 4\) best for code	equal to main	self-speculative decoding, up to 3× faster	gains appear only at scale \((\ge 7\text{B})\); helps bytes and code most
DeepSeek-V3	1 sequential MTP module (1 layer, shared head, shared embedding)	0.3 → 0.1 after 10T tokens	2nd-token acceptance 85–90%, \(\approx 1.8\times\) TPS	kept through post-training; the form everyone copied
Nemotron 3 Super	2 MTP layers with shared parameters	0.3 aux loss during SFT	native speculative decoding	final MTP healing stage: heads retrained on RLVR prompts with the backbone frozen, because RL moved the trunk and acceptance collapsed
Kimi K3	pretrained MTP layer	—	fine-tuned into an EAGLE-3-style draft: target frozen, 7-step unroll, features from 1st / 4th / last blocks	trained on the acceptance-rate loss \(-\log \Sigma \min (p, q)\) directly instead of a KL surrogate; QAT-matched to serving precision
DeepSeek V4	MTP retained from V3	—	speculative decoding; also used inside RL rollouts	rollout engine relies on it; FP4 QAT applied so sampling matches deployment
Gemma 4	autoregressive MTP drafter fed the main model's activations and KV cache	—	speculative decoding on-device	shipped as a separate drafter checkpoint
GLM-5	MTP in the model	—	used in the RL rollout engine (slime) to cut tail latency for small-batch decoding	an infra reason to keep it: RL rollouts are exactly the small-batch regime where drafting pays most

Nemotron, Kimi and DeepSeek V4 stage details are on the LLM report diff.

Pitfalls

RL breaks the draft. Any stage that moves the trunk — RLVR especially — lowers acceptance because the head was trained against the old trunk. Budget a healing pass after post-training, or the speedup quietly disappears in the deployed model.
Acceptance depends on the sampling distribution. A draft tuned for greedy decoding accepts less at temperature 1, and vice versa; Kimi K3's choice to optimize \(-\log \Sigma \min (p, q)\) at temperature 1 is a response to exactly this.
Only pays when decoding is memory-bound. Speculative decoding trades extra compute for fewer weight reads. At large serving batch sizes the main model is already compute-bound and the draft adds cost without saving time.
Small models may lose. Gloeckle et al. found MTP hurt some benchmarks below ~1B; the training-signal argument is a scale story.
Parallel vs sequential is not cosmetic. Independent heads predict x_{t+2} without knowing x_{t+1}, so their drafts are less coherent; the sequential module conditions on it, which is why the DeepSeek form is the one used for drafting.

Sources

Better & Faster Large Language Models via Multi-token Prediction — Gloeckle et al., 2024 · alphaXiv

DeepSeek-V3 Technical Report, §2.2 (MTP) and §5.4.3 (acceptance rate) · alphaXiv

EAGLE-3 — the separate-drafter design Kimi K3 converts its MTP layer into · alphaXiv

Raschka · LLM Architecture Gallery · MTP — parallel vs sequential forms across DeepSeek-V3, Qwen3-Next, Step 3.5 Flash, Nemotron · explainer

Nemotron 3 Super report §2.1.2, §3.1, §3.2 (MTP healing) · Kimi K3 report §4.1.4 (draft fine-tuning) · DeepSeek-V4 report · Gemma 4 report §2.6