LoRA adapter for google/gemma-4-12B-it-qat-q4_0-unquantized trained to answer the Polish history matura
(poziom rozszerzony, CKE). Built at the Warsaw Model Trainers Hackathon (team repo: WarsawModelTrainersHackathon,
src/posttrain/).
Procedure
Vision SFT (QLoRA, 4-bit base): image-based exam items (real CKE papers with their source images, synthetic Wikimedia
image tasks, synthetic full papers) with the PNGs fed to the model, plus 30% of the teacher traces; vision tower frozen,
language LoRA only.
Hyperparameters
LoRA r=32, alpha=64, lr 1e-4, 1 epoch, batch 1 x grad_accum 16, max_len 6144, Unsloth vision collator
Data
Consolidated training set data/training_datasets/sft_train.jsonl (build_sft.py): real CKE papers 2005–2026 with keys, rephrased papers, synthetic image tasks, synthetic full papers, e-podręczniki textbook questions, synthetic Claude/DeepSeek tasks, gemma-4-31b@think teacher traces (exam items and essays, judge-filtered).
Held-out, never trained on: every May paper 2017–2026, June 2020 and the 2023 mock
(src/posttrain/common.py HELDOUT; every training row is also checked against 80-character shingles of those papers).
Metrics (training run)
- global_step: 455
- best_eval_loss: 1.5642297267913818
- last_loss: 0.4651
- last_eval_loss: 1.7347
Exam scores (graded by deepseek-flash-high@think via benchmark/bench.py) are reported separately.
ClearML: http://X.X.X.X:18080/dashboard (project HCKT)
Usage
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor
base = AutoModelForImageTextToText.from_pretrained("google/gemma-4-12B-it-qat-q4_0-unquantized", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "WMTH-100d2exam/gemma-4-12B-it-qat-matura-history-sft-vision-qlora-r32")
processor = AutoProcessor.from_pretrained("WMTH-100d2exam/gemma-4-12B-it-qat-matura-history-sft-vision-qlora-r32")
lora.gguf is the same adapter for llama.cpp: llama-server -m <base Q4_0 gguf> --lora lora.gguf.
