Sam EnMin Huang

Files: merged bf16 safetensors + tokenizer + chat template (final LoRA adapter merged into grpo-v2). Load with plain AutoModelForCausalLM; saved with transformers 5.x (dtype key, chattemplate.jinja). Companion repo with the adapter and checkpoints: tbooy/Qwen2.5-3B-Instruct-Sheldon-RLVR-math-v1-LoRA. Stage-3 (RLVR) model of a Harvard CS 2881R project: reinforcement learning with a verifiable math reward on top of tbooy/Qwen2.5-3B-Instruct-Sheldon-RLAIF-grpo-v2 (SFT persona model -> RLAIF GRPO -> this), the persona being Dr. Sheldon Cooper. Why this stage exists: the persona SFT stage cost most of Qwen2.5-3B-Instruct's competition-math ability (MATH-500 greedy 68.0 -> 35.4), and the RLAIF…

Stage-2 seed model of a Harvard CS 2881R project: Qwen/Qwen2.5-3B-Instruct fine-tuned to answer in the voice of Dr. Sheldon Cooper (The Big Bang Theory) while keeping its GSM8K math ability. This is the checkpoint-1 model v3b (agastyasridharan/Qwen2.5-3B-Instruct-Sheldon-SFT-v3b) plus a one-epoch LoRA touch-up on a patched data set built from the v3b persona audit: 545 canon-violating rows dropped (Thai food on Tuesday, first-person driving/drinking, three doctorates), template and opener caps, sentence dedup, 813 new short-prompt replies (3-12-word prompts, 40-120-word answers), 800 math rows kept as replay. Files: finaladapter/ (PEFT LoRA on the v3b merged model) and…

Stage-2 seed model of a Harvard CS 2881R project: Qwen/Qwen2.5-3B-Instruct fine-tuned to answer in the voice of Dr. Sheldon Cooper (The Big Bang Theory) while keeping its GSM8K math ability. This is the checkpoint-1 model v3b (agastyasridharan/Qwen2.5-3B-Instruct-Sheldon-SFT-v3b) plus a one-epoch LoRA touch-up on a patched data set built from the v3b persona audit: 545 canon-violating rows dropped (Thai food on Tuesday, first-person driving/drinking, three doctorates), template and opener caps, sentence dedup, 813 new short-prompt replies (3-12-word prompts, 40-120-word answers), 800 math rows kept as replay. Files: merged bf16 safetensors + tokenizer, loadable with…

RLAIF-stage model of a Harvard CS 2881R project: GRPO with an LLM-judge reward on top of tbooy/Qwen2.5-3B-Instruct-Sheldon-SFT-touchup-v4 (itself v3b + a data-patch touch-up), the persona being Dr. Sheldon Cooper and the protected STEM task GSM8K. Files: finaladapter/ (PEFT LoRA on the touch-up model, step 109) and checkpoint-{25,50,75,100,109}/ with trainer states (per-step reward components under sheldon/). Apply to tbooy/Qwen2.5-3B-Instruct-Sheldon-SFT-touchup-v4. Reward R = G (2 P + F) - rules - 0.5 falseclaim - batchtax: a persona-blind task gate G (asks satisfied, refusal, harm, self-contradiction, false claims about the user), a pairwise persona score P from sibling comparisons…

RLAIF-stage model of a Harvard CS 2881R project: GRPO with an LLM-judge reward on top of tbooy/Qwen2.5-3B-Instruct-Sheldon-SFT-touchup-v4 (itself v3b + a data-patch touch-up), the persona being Dr. Sheldon Cooper and the protected STEM task GSM8K. Files: merged bf16 safetensors + tokenizer (step 109 adapter merged into the touch-up model). Reward R = G (2 P + F) - rules - 0.5 falseclaim - batchtax: a persona-blind task gate G (asks satisfied, refusal, harm, self-contradiction, false claims about the user), a pairwise persona score P from sibling comparisons judged in both presentation orders on nine items, a small form bonus F, 17 deterministic rule penalties from the v3b audit (canon…

Files: finaladapter/ (PEFT LoRA, step 200) and checkpoint-{25,...,200}/ adapters with trainer states (per-step reward, pass-rate and group statistics under rlvr/), plus rlvrargs.json. Apply to tbooy/Qwen2.5-3B-Instruct-Sheldon-RLAIF-grpo-v2. Merged weights: tbooy/Qwen2.5-3B-Instruct-Sheldon-RLVR-math-v1. Stage-3 (RLVR) model of a Harvard CS 2881R project: reinforcement learning with a verifiable math reward on top of tbooy/Qwen2.5-3B-Instruct-Sheldon-RLAIF-grpo-v2 (SFT persona model -> RLAIF GRPO -> this), the persona being Dr. Sheldon Cooper. Why this stage exists: the persona SFT stage cost most of Qwen2.5-3B-Instruct's competition-math ability (MATH-500 greedy 68.0 -> 35.4), and the…

Two 50-step LoRA r32 stages (s1, s2) on top of tbooy/Qwen2.5-3B-Instruct-Sheldon-RLVR-math-v1 with a re-labelled difficulty schedule over a 64.7k-prompt pool (MATH-12k + DeepMath + DAPO-17k + DeepScaleR), CISPO, AdamW 2e-5. Result: flat on MATH-500 (levels 3-5 55.0 / 55.3 vs 55.2) although the schedule reduced dead prompt groups from 47% to 6-9%; kept for the record. s2/finaladapter applies to the merge of s1, not to v1 directly. Details: https://github.com/TBOO-Y/cs2881r-sheldon-sft (results/rlvr/NOTES.md).

Files: bf16 safetensors + tokenizer + chat template. Load with plain AutoModelForCausalLM; saved with transformers 5.x (dtype key, chattemplate.jinja). Stage-3c model of a Harvard CS 2881R project (persona: Dr. Sheldon Cooper): a full-parameter RLVR continuation of tbooy/Qwen2.5-3B-Instruct-Sheldon-RLVR-math-v1 (Qwen2.5-3B-Instruct -> persona SFT -> RLAIF -> LoRA RLVR v1 -> this). Why this model exists: after the v1 LoRA run plateaued (MATH-500 levels 3-5 at ~55 while about half the prompt groups had become all-correct), two LoRA continuations with a re-labelled, pool-expanded difficulty schedule stayed flat, so the last attempt on this seed trained all parameters instead: AdamW lr 2e-6…