sn120-exp019-S2

sn120-exp019-S2: MiMo-9B GRPO with unresolved rollouts trained as failures

A controlled 29-step replication of the Affine SN120 MiMo GRPO run (affine-mimo9b-grpo-20261013), started from the same initial checkpoint. It uses the official trainer and settings unchanged except for one switch: cap_policy = affine-unresolved-zero-reward-active-v1. Unresolved rollouts (hit the 8192-token cap, or no gradeable final answer) get reward 0 and stay in the loss, instead of being masked.

A control arm, S0, is identical except it uses the live run's original affine-unresolved-zero-reward-mask-all-v1. S0 reproduces the live run's rise-then-decline on held-out data and its entropy collapse. S2 shows neither.

Recipe

S2 is the live run's new training policy without the new qualification gate. Our reading of the v2 switch (e000039+: outcome policy ...-unresolved-failure-v2, 128 groups/step, loss moving from about −0.13 to about +0.02) is that unresolved rollouts count as failures both in training and for qualification. Arm S4 of the same experiment replicates that (v2 gate, 32 groups/step) and is still running.

Results (8192 cap; fixed32 × 8 samples, 300 held-out problems × 1 sample; same seeds at every step)

• S2 (step 29) − S0 (step 30): heldout300 +10.3 pp (p=0.0001, paired by problem, sign-flip test), fixed32 +2.7 pp (p=0.60). On heldout, the samples the two arms disagree on are mostly S0 truncated → S2 correct (26), then S0 format-rejected → S2 correct (19). The reverse cases are 7 truncated and 6 format-rejected.
• S0 declines from its step-10 peak by −8.0 pp on heldout300 (p=0.01), the same shape as the live run.
• Median training-batch logprob of the sampled tokens under the current policy: S0 −0.245 → −0.028 (live run −0.248 → −0.045 by step 20); S2 stays at −0.25 to −0.30 for all 29 steps.
• Limits: S2 does not stop length growth. Median output length on heldout goes 806 → 2,406 tokens (step 0 → 29), and gains level off after step 20. Under the v1 gate, 23% of sampled problems at step 26 had a correct answer and capped failures but no completed wrong, so they never entered training. Part of S2's gain is format: format-rejected answers are trained as failures. Single seed, 8192-cap evaluation only (no 24576 eval of these checkpoints yet).

Files

• step010/, step020/, step027/, step029/: bf16 weights, loadable with transformers ≥5.14 (Qwen3_5ForConditionalGeneration) or vLLM 0.31. step020 is the best held-out checkpoint (77.7%); step029 is the last. Our save routine copied the base model's README.md / .gitattributes into step010/. They are the base card, not this one.
• state027/, state029/: full optimizer state from the official CPUAdamW.save_local (fp32 masters + exp_avg + exp_avg_sq, 256 MB shards, descriptor.json with per-shard sha256) plus meta.json (step, checkpoint id, descriptor_sha256, task-selection state). Use it to resume exactly: python recipe/train_b.py --arm S2 --model <base dir> --out <dir> --resume <dir containing step029/ and state029/> --steps 40.
• recipe/: experiment scripts. train_b.py drives sampling → grading → group selection → official train_update; gen.py is the vLLM worker; common.py holds prompt + grading; runtime.py is the HF runtime; s3.py / s4.py are the other arms. It also expects the official subnet/mimo_grpo/{config,data,optimizer,training,upstream}.py and the two pinned verl files from affine @ f07c54e under recipe/affine/, plus the task snapshot original-math7496.tasks.json (sha256 77a4524a…).
• results/S2, results/S0: per-step training metrics (steps.jsonl) and per-sample eval grades (eval_stepNNN.jsonl).