CS329X HW1: CS-student persona DPO adapter
LoRA adapter trained from a fresh Qwen/Qwen2.5-1.5B-Instruct checkpoint with TRL DPO.
The fictional persona prioritizes evidence, honesty, concise practical answers, and respect.
Training
18 course PRISM tasks; 378 ordered preference pairs including an edited best answer.
Learning rate 5e-05; 2 epochs; batch size 1; accumulation 8; beta 0.1.
4-bit NF4 base; LoRA rank 8, alpha 16, dropout 0.05; seed 329.
Label provenance
Rankings and edited answers were drafted with AI assistance for a user-selected persona.
User review status is recorded in the accompanying preference-label provenance file.
These labels are persona preferences, not a representative human preference survey.
Evaluation and limitations
The notebook compares base and DPO completions on 20 fixed PRISM prompts outside the 18 training prompts.
Both use the same generic system instruction and greedy decoding.
This small educational run does not establish general improvement or safety.
The adapter may overfit and should not be used for consequential advice.
Load
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
model = PeftModel.from_pretrained(base, "ChenChen2005/cs329x-hw1-dpo-qwen25-20261010")
tokenizer = AutoTokenizer.from_pretrained("ChenChen2005/cs329x-hw1-dpo-qwen25-20261010")
