A one-pass judge that returns a probability instead of text, used as the reward inside GRPO. Prior work: one figure per paper, from Cobbe's verifier to PRIME. What we did: DiffusionGemma run as a Jev-style verifier on one A100, compared with live Jev, then seven reward arms on GSM8K. Proposal: GRPO gets no gradient from a group whose rollouts all agree; a cheap judge can rank inside those groups and pick prompts before rolling out, if it is refit as the policy moves.
Seven things the field established, one figure each: verifiers rank samples, step labels beat answer labels, generating buys accuracy at a latency cost, step rewards can be had for free, proxies get gamed, and a small verifier as reward makes traces legible.
Prior work 1 · The loop
The verifier is the bottleneck box
Every design choice in this deck is a choice about the green box: how much it costs per call, how many numbers it returns per solution, and whether those numbers mean what they say.
GRPO: Shao et al. 2024 (DeepSeekMath, 2402.03300) · RLVR: Lambert et al. 2024 (Tülu 3, 2411.15124)
Prior work 2 · 2021
Sample many, let a verifier pick
The first verifier for reasoning was a classifier with one output, used at test time to rank samples. Verification scaled better with data than fine-tuning did. Not yet a training signal.
Cobbe et al. 2021, Training Verifiers to Solve Math Word Problems (2110.14168), introduces GSM8K
Prior work 3 · 2023
Label the steps, not the answer
Process supervision beat outcome supervision by six points, and the cost was the point: 800,000 human step labels. Every PRM paper since is an attempt to get the step signal without paying for it.
Lightman et al. 2023, Let's Verify Step by Step (2305.20050) · Math-Shepherd: Wang et al. 2024 (2312.08935), automatic step labels by rollout
Prior work 4 · 2024
Let the verifier think in tokens
Making the verifier generate buys accuracy and a rationale and pays for it in latency. The whole 2025 line of reasoning reward models (RM-R1, J1, DeepSeek-GRM) is this idea with RL on the verifier's own chain of thought.
The step signal without step labels, and the first dense verifier that is retrained on-policy during RL. Two things it does not give: a calibrated probability, and a verifier that is cheaper than the policy.
Yuan et al. 2024, Free Process Rewards without Process Labels (2412.01981) · Cui et al. 2025, PRIME (2502.01456)
Prior work 6 · 2022–23
Optimise a proxy long enough and the true score falls
Any learned verifier is a proxy, and a policy trained against it will find where the proxy and the truth part ways. Our arms did this within 50 steps. The fixes on record: bigger or more verifiers, conservative aggregation, and retraining the verifier on the policy's own samples.
Gao, Schulman, Hilton 2022 (2210.10760) · Coste et al. 2023 (2310.02743) · Adv-RM: Bukharin et al. 2025 (2504.06141) · Rafailov et al. 2024 for DPO (2406.02900)
Prior work 7 · 2024
Train for a small verifier, get legibility for humans
The closest ancestor of what we ran and of the proposal: a cheap verifier as the reward, a prover shaped toward what it can check, and a sneaky prover keeping the verifier honest. Their verifier was a small LLM; ours is a one-pass model.
Kirchner et al. 2024, Prover-Verifier Games improve legibility of LLM outputs (2407.13692) · decoupled follow-up (2602.23248) · Generative Adversarial Reasoner: Liu et al. 2025 (2512.16917)
Prior work 8 · Lineage
Where that leaves the verifier menu
Scalar heads are fast but return one number with no calibration story; generative judges reason and explain but cost seconds per call. System 1 models sit in the gap: a typed distribution per question, trained for calibration, at the price of a classifier.
Cobbe 2021 (2110.14168) · Lightman 2023 (2305.20050) · Yuan 2024 (2412.01981) · Zhang 2024 GenRM (2408.15240) · RM-R1 (2505.02387) · J1 (2505.10320) · DeepSeek-GRM (2504.02495) · Jev latency from typesafe.ai; 34 ms measured on one A100
21–22 Sep 2026, one A100: a readout-only System 1 verifier reproduced and compared with Jev, then seven GRPO reward arms on GSM8K with a one-hop step verifier.
Our runs 1 · Density
Cheap calls make per-step reward affordable
Process reward models exist since 2023, but labelling steps was the cost. A verifier that answers a local yes/no in 35 ms turns every hop into a reward at no labelling cost, and it needs no reference solution.
PRM: Lightman 2023 · implicit PRM: Yuan 2024 · PRIME: Cui 2025 (2502.01456) · the one-hop question is our arm B
Our runs 2 · Calibration
Same accuracy, different tails
Two verifiers that tie on accuracy behave differently when they are wrong. The readout-only model is 90 % sure of its mistakes; the calibration-trained one hedges. As a reward that difference is the whole game, since the policy learns from the tails.
Measured 21 Sep 2026: Jev's 337 public example questions live vs DiffusionGemma-as-Jev on one A100; 375-item step probe for the AUC. RLCD: typesafe.ai; calibration rewards in RL: Damani 2025 (2507.16806)
Our runs 3 · Cost
Seconds per GRPO step, Qwen2.5-1.5B on one A100
Measured 21–22 Sep 2026 from the per-arm logs: 128 completions and about 1,100 one-hop reads per step. The 1.5B itself generated 4,300 to 5,500 tokens per second under the colocated sampler. A judge the same size writing a 200-token rationale per read would need 220,000 judge tokens per step, 44 s on its own GPU; a 7B judge scales by its five times more active parameters to about 220 s. Dense process reward with a reasoning judge is the arm nobody runs because of this row.
Source: figures/03-eval-log.csv, columns gen_s / train_s / judge_s at step 100, tokens per solution × 128 per step. The 7B figure assumes throughput inversely proportional to active parameters; batched serving on a dedicated GPU would move it by a factor of two either way, not ten.
Our runs 4 · One day of runs
The aggregate decides what “a step” becomes
Seven rewards, same verifier questions, same 12,800 samples. Every reward built on the minimum over steps merged hops and lost 5 to 10 points; the mean with a step floor held; swapping the judge to Jev with the identical reward added 1.4 points and the best held-out skip signal. The policy finds the edge of the verifier's definition within 50 steps.
Qwen2.5-1.5B-Instruct, full fine-tune, GRPO group 8, 100 steps per arm, evals on a fixed 500-problem test set. Hacking lineage: Gao 2022 (2210.10760) · Coste 2023 (2310.02743)
pays per accepted atomic step; pads once redundancy stops firing
Shared by every arm: Qwen2.5-1.5B-Instruct full fine-tune bf16, GRPO group 8, 16 prompts per step, 128 completions per step, lr 1e-6, KL β 0, 100 steps, same seed and data order, 512 max tokens. checker = exact match on the final #### number. Per step k the judge reads {problem, previous step, current step} and answers one noul question per read, alone: follows = "does the current step follow from the previous step and the problem by a single direct inference, with no skipped reasoning?"; redundant = "is this step redundant, restating or adding nothing?"; compound = "does this step perform more than one calculation or inference?". Evals: greedy on a fixed 500-problem GSM8K test subset.
One idea: a cheap predictor of the pass rate a prompt will get under the current policy, used before rolling it out. GRPO gets no gradient from groups whose rollouts all agree; the predictor keeps the batch at the frontier, and if it is calibrated it can also replace the group as the baseline. Refit as the policy moves.
Proposal 1 · The mechanism
GRPO learns nothing from a uniform group
The outcome reward gives one bit per rollout, and GRPO's group baseline cancels it whenever the bits agree. The prompts where that happens are exactly the ones the policy cannot do yet and the ones it has just learned. Our GSM8K runs sat at 70% pass rate, where almost no group is uniform: the right test for the wrong reason.
GRPO: Shao et al. 2024 (2402.03300) · dynamic sampling: DAPO, Yu et al. 2025 (2503.14476) · curve: p⁸ + (1−p)⁸
Proposal 2 · The idea
Predict a prompt's pass rate before rolling it out
GRPO removed the critic and pays for it in rollouts that cancel. This is a critic at the prompt level only, used as a sampling device: it says, before generation, which prompts sit at the policy's frontier. It learns from the same checker labels as everything else and is refit so the frontier moves with the policy.
GRPO: Shao et al. 2024 (2402.03300) · signal ∝ p(1−p): variance of a Bernoulli group
Proposal 3 · Against what exists
Difficulty is already used; it is estimated after paying
The frontier idea is established; what differs is when the estimate is available and for which prompts. A one-pass predictor scores a pool the policy has never touched and is cheap enough to refit as the policy changes. An LLM judge in the same seat would cost more than the rollouts it saves.
DAPO: Yu et al. 2025 (2503.14476) · AdaRFT: Shi et al. 2025 (2504.05520) · Self-Evolving Curriculum: Chen et al. 2025 (2505.14970)
Proposal 4 · Second use
If V is calibrated, the group can shrink
The group exists to estimate a baseline. A calibrated prompt-level predictor is that baseline, so the rollout budget can go to more prompts instead of more samples of the same one. This is where calibration stops being a nicety: a miscalibrated V biases every advantage.
ReMax: Li et al. 2023 (2310.10505) · RLOO: Ahmadian et al. 2024 (2402.14740) · REINFORCE with learned baselines: Williams 1992
Proposal 5 · First experiment
Three measurements, four samplers
Everything but the predictor exists from this week's runs. The hypothesis has two halves on purpose: the first says the predictor helps, the second says the refit is what keeps it helping. If P₀ holds up as well as P, the refit is unnecessary and the paper is smaller; if P does no better than H, a lookup table was enough.