Notes · Method note 02
Self-play for reasoning
Every self-play method is the same loop — something proposes a task, something solves it, something verifies, and both sides are trained on the result. The four papers below differ in exactly two slots: where the task comes from and who is allowed to be the verifier. Ordered by how much external grounding they keep, from a labeled dataset down to nothing and back up to a corpus.
Fig 1 · The loop, four times
proposersolververifierexternal groundingreward
- Sample a rationale + answer; keep the ones whose answer matches; fine-tune on them; repeat.
- Rationalization: for failures, show the answer as a hint and keep the rationale that reaches it.
- Not really self-play — the questions and answers are given. The self-generated part is the reasoning trace.
- Failure mode: nothing forces diversity or difficulty; the loop can converge on easy items.
- One model plays both roles. Tasks are programs in three modes: deduction (program+input → output), abduction (program+output → input), induction (I/O pairs → program).
- The verifier is a Python executor, so correctness is exact and no data is needed.
- Proposer is rewarded for tasks the solver sometimes solves — not always, not never.
- Grounded by execution, but only in the space of things a program can express.
- Two copies of the base model co-evolve with GRPO. Challenger is rewarded when Solver's accuracy is \(\approx 50\%\) — maximum uncertainty.
- No verifier: the label is the Solver's own majority vote over 10 samples. Keep only questions with 3–7 of 10 agreeing.
- Qwen3-4B-Base: +6.5 math, +7.5 general reasoning after three iterations.
- Collapses: pseudo-label accuracy decays 79% → 63% by iteration 3; 0.6B models collapse after one round.
- Same LLM, two roles, but the Challenger reads a real document and writes a question whose answer is extractable from it. The Reasoner never sees the document — information asymmetry.
- Grounding fixes R-Zero's failure: the label comes from the text, not from the model agreeing with itself.
- Both roles trained jointly with Dr. GRPO, separate advantages. Corpora: Nemotron-CC-Math, NaturalReasoning.
- +8.9 math, +9.8 general reasoning across Qwen3-4B/8B and OctoThinker-3B/8B.
The math in three lines
\[\text{solver:}\qquad \max\ \mathbb{E}_{x\sim\text{proposer}}\big[\,r_{\text{solver}}(y,x)\,\big]\ \ \text{with GRPO}\]
// \(r = 1\) if verified correct, else 0
\[\text{proposer:}\qquad \max\ \mathbb{E}_{x}\big[\,f\big(\hat p(x)\big)\,\big]\;-\;\text{repetition penalty}\]
// \(\hat{p}\) = the solver's empirical success rate on x; f peaks at \(\hat{p} = \tfrac12\)
\[\text{label}(x) \;=\; \text{executor}(x)\ \ \big|\ \ \text{majority vote over solver samples}\ \ \big|\ \ \text{extracted from the source document}\]
// the verifier slot — the whole design question
The 50% target is the shared idea. R-Zero rewards the Challenger for questions the Solver gets right about half the time and adds a difficulty band (keep only 3–7 of 10 agreeing); SPICE uses a variance-style reward with the same peak. Absolute Zero calls the same thing learnability. It is the self-play form of the difficulty filters every model report uses — Phi-4's “teachable” seeds, OLMo's 62.5% pass-rate cutoff, Nemotron's “drop what the SFT model always solves.”
Where it appears in the reports
- Kimi K3 — knowledge-graph task synthesis: agents expand a concept graph by web search, retrieve real materials, and synthesize verifiable tasks. A proposer grounded in retrieved text — SPICE's move without the co-evolving reward. Its mock-app workspaces are built by web-searching agents: proposers for environments, not just questions.
- GLM-5 — a Web Knowledge Graph from over two million pages generates multi-hop search questions, filtered by “drop it if a tool-free model answers in 1 of 8 tries” — a proposer with a difficulty gate, no solver reward flowing back.
- Nemotron 3 Super — a PAIR-style attacker mines jailbreak prompts against an early checkpoint (a narrow adversarial proposer); the conversational-tool-use pipeline drops all-success and all-fail scenarios — the 50% band applied offline.
- OLMo 3, Llama-Nemotron — majority vote as the label when no gold answer exists. R-Zero's pseudo-labeling, used once and cautiously rather than iterated.
- DeepSeek V4 — the policy is trained as its own generative reward model. That is the verifier slot answered with “the solver, deliberately” — the R-Zero risk taken on purpose, with rubrics as the grounding.
Pitfalls
- Pseudo-label decay. When the solver labels its own questions, harder questions get noisier labels: R-Zero measures 79% → 63% label accuracy by iteration three, and collapse follows. The fix is a verifier that is not the solver — an executor, a document, a test suite.
- Information symmetry. If proposer and solver share all the same knowledge, the proposer can only ask what the solver already half-knows. SPICE's information asymmetry (the Reasoner never sees the document) is what keeps the curriculum moving.
- Diversity collapse. The 50% reward alone is happy with the same question repeated; every method adds a repetition or diversity penalty, and it is doing real work.
- Grounding is a ceiling as well as a floor. Absolute Zero is exact but limited to what a Python program can express; SPICE is broad but bounded by corpus quality and coverage.
- Contamination by construction. A proposer that mines the web will eventually mine benchmark items; GLM-5 and OLMo both decontaminate synthesized questions against evaluation sets, which is not optional here.
- Who verifies is the question all the way down. SPICE asks it of R-Zero; RLVR asks it of RLHF; DeepSeek V4's actor-as-judge asks it of RLVR. Each level trades a cheaper signal for a more gameable one.