Checkpoint 3 of Harvard CS2881R Assignment 1: a Sheldon Cooper persona LoRA for Qwen/Qwen2.5-3B-Instruct, continued from
the CP2 RLAIF adapter with GRPO against a verifiable math reward (RLVR), alternated with continued constitutional
persona-reward batches (4 math batches : 1 persona batch).
- Checkpoint:
/checkpoints/rlvr/rlvr-mixed/final (chosen on development data only)
- Starting point: benjkoch/Qwen2.5-3B-Instruct-Sheldon-RLAIF-v2
- Math reward: 0/1 from a strict single-final-answer verifier (
rlvr-verifier-v2); correct-reward alpha fixed at 1
- Persona reward: fixed
Qwen/Qwen2.5-14B-Instruct constitutional judge, sampled principle, both A/B orders
- KL: beta 0.05 against the frozen RLAIF policy (the only swept hyperparameter)
- Code and report: https://github.com/harvard-cs2881f26/hw1-chen-koch-zhou (
docs/CHECKPOINT3.md)
This repository holds the LoRA adapter only; the adapter is cumulative (SFT → RLAIF → RLVR), so base + this
adapter is the complete model.
Held-out results (greedy decoding; historical scorer and rubric)
| stage |
persona (Qwen rubric, 0–1) |
MATH-500 % |
GSM8K % |
AIME 2024 % |
| SFT |
0.826 |
67.6 |
85.2 |
6.7 |
| RLAIF |
0.828 |
66.2 |
86.0 |
3.3 |
| pure RLVR |
0.825 |
67.0 |
85.6 |
6.7 |
| mixed RLVR + persona |
0.825 |
68.0 |
86.4 |
10.0 |
None of these differences is statistically detectable. Against the RLAIF starting model, MATH-500 +1.8 pp
(28 gained / 19 lost, McNemar p = 0.24), GSM8K +0.4 pp, and persona -0.003 (95% bootstrap CI -0.008 to +0.002). An
independent Mistral-Small-24B-Instruct-2501 judge, on 100 held-out rows in both answer orders, found 1 win / 97
ties / 2 losses against RLAIF and 5 / 89 / 6 against SFT. The persona survived continued math training, and math
did not measurably improve at this training scale (one epoch over 400 fresh questions, lr 5e-6).
Paired tests, uncertainty, the beta sweep and the reward-gaming audit are in the report.
A correct final answer is outcome accuracy, not proof of correct reasoning.
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "benjkoch/Qwen2.5-3B-Instruct-Sheldon-RLVR")
Use prompts/sheldon_system.txt
as the system message; the model was trained and evaluated with it.
