Gemma 4 12B-it, reward-laundering RL: jordgm_s3 (REAL rewards, seed 3)
Base: google/gemma-4-12B-it, loaded text-only (the decoder of the unified multimodal checkpoint; see
scripts/port/make_text_only_ckpt_gemma4.py). Adapter: LoRA r32, alpha 32, on q/o/gate/up/down (all 48 layers) + k/v (the 40
sliding-window layers). Standard PEFT format: PeftModel.from_pretrained(<text-only Gemma4ForCausalLM>, repo, subfolder=...).
Thinking uses Gemma's <|channel>thought ... <channel|> channel (reward_laundering/gemma4_renderer.py), temperature 1.0,
think budget 2500.
Research checkpoint from the reward-laundering project ("ext-model": does owned-haiku reward laundering replicate on a non-Qwen
model family?). Full write-up: results/EXT_MODEL.md. Held-out haiku = eval_propensity --trait haiku --locus response,
16 neutral prompts x 8 samples; prog = structural haiku detector, judge = LLM judge (rejudged off-box).
Recipe (hkl): GRPO from the own-voice SFT step-300 adapter (Jordine/rlport-sft_jordgm_own/step300), 14 steps,
8 problems x 16 samples per step, lr 1e-4 constant, LoRA r32, think 2500, static suite multiplication7,multiplication8, seed
3, problem offset 1220. Reward = main-task correctness only; the haiku is never rewarded.
Paired twin: jordgm_pl3 (placebo, same seed and data).
| folder |
what |
held-out haiku prog / judge |
state/latest/ |
training state (LoRA + Adam optimizer.pt) at the last completed step; for resuming |
– |
SFT init (step 300): 13.3% / 10.9% (n=128). run/ = config, metrics, rollouts; logs/ = training log.