team03 Math SLM — v4 (GATE + General Mathematics)
A domain-specific small language model for step-by-step math problem solving, built by team03 (SLM Learners) for the Pramana SLM++ Bootcamp Round 2 submission.
- Base: Qwen2.5-0.5B-Instruct (494M params, fine-tuned — not from scratch)
- Method: QLoRA (r=16, 4-bit NF4 base, bf16 compute), merged into the base weights
- Domain: GATE-style + general mathematics
- Intended users: GATE / competitive-exam aspirants and math students
- Use case: free-form math QA — direct answers for simple prompts, chain-of-thought + "Final answer:" for complex ones
How to use
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
"singhmandavi/math-slm-qwen2.5-0.5b-v4", torch_dtype=torch.float16)
tok = AutoTokenizer.from_pretrained("singhmandavi/math-slm-qwen2.5-0.5b-v4")
msgs = [{"role": "user", "content": "What is 17 * 23? Solve step by step."}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
enc = tok(text, return_tensors="pt")
out = model.generate(**enc, max_new_tokens=1024, temperature=0,
pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))
For an OpenAI-compatible endpoint, serve with serve_hf.py (stdlib + transformers only, no Ollama needed).
Training details
| Item |
Value |
| Corpus |
train_all.jsonl — 29,853 rows (14,308 direct + 15,545 CoT) |
| Effective train / val |
23,532 / 471 (after --drop-overlong, 5,850 CoT rows ≥2048 tokens dropped) |
| Sources |
ExamBench (169Pi/exambench, Apache-2.0, commit b5f3d8b) + MathNet IMO split (CC-BY-4.0) |
| Optimizer |
AdamW (betas 0.9/0.999, eps 1e-8, wd 0.01) |
| LR |
1.5e-4, cosine decay, ~132 warmup steps (3% of total) |
| Effective batch |
16 (2 micro-batch × 8 grad-accum) |
| Epochs / steps |
3 / 4,410 logged (theoretical total 4,413 = 1,471/epoch × 3; last multiple of 5 logged) |
| Max seq len |
2,048 |
| LoRA |
r=16, alpha=32, dropout 0.05, all proj targets (q/k/v/o/gate/up/down); 8.8M trainable params (1.75%) |
| Hardware |
1× NVIDIA H100 80GB, ~1.84 GPU-hours |
| Seed |
7 |
| Tokens seen |
53,356,887 (last logged, step 4,410) |
Precision note: training ran in bf16 compute (QLoRA 4-bit NF4 base), but the merged checkpoint uploaded here is float16 (the merge step reloads the base in fp16).
Data provenance & decontamination
- Public Hugging Face datasets pulled via
pull_data.py; licenses verified through the HF API on 2026-09-05 and recorded in data_manifest.md.
- Decontamination: 13-gram word overlap (GPT-3/Chinchilla style) + MinHash-LSH Jaccard (threshold 0.5) against held-out eval sets — 0 exact matches / 0 8-gram hits vs the held-out set, and 0 overlap vs IMO 2025.
- Held-out eval set built with
build_heldout_eval.py from raw ExamBench rows never used in training, with a final overlap check that fails loudly on any contamination.
- Caveat (documented, not hidden): the GATE-PYQ-specific contamination check was blocked (IIT-KGP site down at the time) and never re-run; and a byte-for-byte SHA-256 cross-check between the Kaggle-side and Studio-side corpus copies was not performed — only the row count (29,853) is confirmed to match.
Evaluation results
Level 1 — training health (results_level1_v4.json)
| Check |
Result |
| Numerical stability |
PASS |
| Undertraining |
PASS (loss reduction 19.7%) |
| Overfitting |
OK (final gap 0.14, stable) |
| Gradient explosion |
OK |
| Convergence |
FAIL (marginal — loss trend ticks up in the final 10% of steps; consistent with cosine-decay/batch noise rather than real instability, since all other checks are healthy) |
Level 2 — general capability / instruction following (results_level2_v4.json)
- Instruction-following score I = 8/33 (24.24%).
- Main failure mode: the model prepends a "here is the thinking process..." CoT preamble even under strict format constraints (one-word, JSON-only, exact word counts), despite usually getting the underlying answer right. This is inherited from the CoT-heavy training mix and is the main quality gap.
Level 3 — not run for this model. It requires the model served over HTTP
(serve_hf.py + evaluate.py --level 3 --base-url …). The eval set it
consumes (eval/heldout_eval.jsonl, 282 KB, SHA-256 d2ac7ef3…) is in place
and the harness is functional — Level 2 ran end-to-end against the same
server — so Level 3 can still be run post-hoc against this checkpoint
without retraining.
Limitations
- Weak instruction-following under strict output-format constraints (see Level 2 above).
- Convergence check fails marginally (noisy end-of-run loss slope); training itself was otherwise healthy.
- Verbose-CoT bias — mitigated but not eliminated by including 48% direct-format rows.
- Trained at 2,048 tokens context; longer problems are truncated rather than handled.
Reproducibility
- Training log:
runs/math_slm_v4/training_log.jsonl (official L1 schema: step, train_loss, val_loss, lr, grad_norm, tokens_seen)
- Training events:
runs/math_slm_v4/training_events.jsonl
- Checkpoints:
best (by val loss) + merged final; run completed cleanly (checkpoint timestamps match the log's final timestamp, 2026-09-17 18:28)
- Exact dependency list:
dependency_versions.txt (211 packages, pip freeze from the GPU node)
- Corpus hash (SHA-256):
eb4fecba0468286afe3e4960c8680c45f87e46d414941d0614dfe2f5ad518be5
License
Base model Qwen2.5-0.5B-Instruct is Apache-2.0. Training data: ExamBench (Apache-2.0) and MathNet (CC-BY-4.0, attribution required).
