LoRA adapter for google/gemma-4-12B-it-qat-q4_0-unquantized trained to answer the Polish history matura
(poziom rozszerzony, CKE). Built at the Warsaw Model Trainers Hackathon (team repo: WarsawModelTrainersHackathon,
src/posttrain/).
Procedure
Supervised fine-tuning, bf16 LoRA on the QAT weights (deployed on the official Q4_0 GGUF). One mixed pass over every
source: short-key items (real/rephrased/synthetic exam items, textbook questions) trained with thinking OFF (question ->
CKE answer), gemma-4-31b@think teacher traces (judge-filtered, exam items + essays) trained with thinking ON; exam-like
rows oversampled x2-x3. Loss on the answer only.
Hyperparameters
LoRA r=32, alpha=64, all language-model linear layers, lr 1e-4 cosine, 1 epoch, batch 16, max_len 6144, Unsloth + TRL
Data
Consolidated training set data/training_datasets/sft_train.jsonl (build_sft.py): real CKE papers 2005–2026 with keys, rephrased papers, synthetic image tasks, synthetic full papers, e-podręczniki textbook questions, synthetic Claude/DeepSeek tasks, gemma-4-31b@think teacher traces (exam items and essays, judge-filtered).
Held-out, never trained on: every May paper 2017–2026, June 2020 and the 2023 mock
(src/posttrain/common.py HELDOUT; every training row is also checked against 80-character shingles of those papers).
Metrics (training run)
- global_step: 1956
- best_eval_loss: 1.5228718519210815
- last_loss: 0.3759
- last_eval_loss: 1.9081
Exam scores (graded by deepseek-flash-high@think via benchmark/bench.py) are reported separately.
ClearML: http://X.X.X.X:18080/dashboard (project HCKT)
Evaluation: closed-question exact match (validation)
This adapter is the best-validation-loss checkpoint (step 500); with thinking on it is the top Gemma model below.
Metric: exact match against the key on the 321 closed items of the validation set (June 2026 + December 2024 papers,
reworded; synthetic CNT papers; textbook questions), greedy decoding, no judge. It is not an exam grade: open questions
and essays are graded separately with deepseek-flash-high@think. Many reworded real-exam items need the source image,
which this text-only eval does not show, hence the low numbers in that column for every model.
Closed-question exact match on validation (321 closed items of rl_val, no judge). Not an exam grade.
| # |
Model |
Overall |
Real exams, reworded (n=48) |
Synthetic exams, CNT (n=16) |
Textbook generated (n=234) |
Textbook original (n=23) |
| 1 |
Gemma-4-12B-QAT SFT (best ckpt 500) @think |
70.4 |
37.5 |
100.0 |
76.1 |
60.9 |
| 2 |
Gemma-4-12B-QAT SFT (best ckpt 500) |
69.5 |
22.9 |
93.8 |
76.9 |
73.9 |
| 3 |
Gemma-4-12B-QAT SFT (final ckpt 1956) |
69.5 |
25.0 |
87.5 |
77.4 |
69.6 |
| 4 |
Gemma-3-12B-it SFT→GRPO |
68.2 |
25.0 |
100.0 |
74.8 |
69.6 |
| 5 |
Gemma-3-12B-it SFT (best ckpt 250) |
66.4 |
29.2 |
87.5 |
72.6 |
65.2 |
| 6 |
Gemma-4-12B-QAT vision SFT (best ckpt 150) |
59.5 |
25.0 |
100.0 |
63.7 |
60.9 |
| 7 |
Gemma-4-12B-QAT base @think |
57.0 |
18.8 |
68.8 |
65.4 |
43.5 |
| 8 |
Gemma-4-12B-QAT base |
49.2 |
12.5 |
50.0 |
57.3 |
43.5 |
| 9 |
Gemma-3-12B-it base |
45.5 |
8.3 |
37.5 |
54.7 |
34.8 |
Recommended use: thinking on (chat_template_kwargs={"enable_thinking": true}), 70.4 vs 69.5 with thinking off.
Usage
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor
base = AutoModelForImageTextToText.from_pretrained("google/gemma-4-12B-it-qat-q4_0-unquantized", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "WMTH-100d2exam/gemma-4-12B-it-qat-matura-history-sft-lora-r32")
processor = AutoProcessor.from_pretrained("WMTH-100d2exam/gemma-4-12B-it-qat-matura-history-sft-lora-r32")
lora.gguf is the same adapter for llama.cpp: llama-server -m <base Q4_0 gguf> --lora lora.gguf.
