FORM-8B: A Fine-Tuned Specialist for FORM Symbolic Algebra Code Generation
Model description
FORM-8B is a QLoRA fine-tune of Qwen3-8B (unsloth/qwen3-8b-unsloth-bnb-4bit)
specialised for zero-shot FORM code generation. FORM is the symbolic
manipulation language used for large perturbative QFT calculations (Grassmann
variables: multi-loop Feynman diagram algebra).
The model is trained exclusively on execution-verified data: every training
example was validated by running the generated FORM program through the FORM
binary (deterministic-output match) plus an LLM consistency audit.
Available adapter variants
| Adapter |
Training data |
Size |
adapters/form-qwen3-8b-v3c |
full mix (2,465 deterministic + 2,168 open/Q&A/tutorial) — recommended |
178 MB |
adapters/form-qwen3-8b-v3cn |
full mix, no docs-in-context rows |
178 MB |
adapters/form-qwen3-8b-v3b |
deterministic-only (2,465 examples) |
178 MB |
Training config: QLoRA r=16, alpha=32, lr=2e-4, 5 epochs, 4-bit NF4 quantised base.
Training data: the training corpus is not included in this release; it
is available from the author upon request.
Intended use
- Generating complete FORM programs from plain-English instructions
- Assisting physicists with symbolic-algebra scripting (gamma traces,
tensor contractions, pattern matching, preprocessor metaprogramming)
- Agentic workflows: pair with a FORM execution sandbox and route
generation/verification to the model
Out of scope: general chat, non-FORM languages, guaranteed mathematical
correctness (generation is syntax-verified at inference time by the caller).
Performance (see paper, Table 3)
All numbers from the revalidated (v2) benchmarks; frontier models evaluated
with docs, FORM-8B zero-shot.
| Benchmark |
FORM-8B (v3c) |
GLM-5.3 +docs |
GLM-5.2 756B +docs |
DeepSeek-V4 Flash 304B +docs |
| 664 deterministic tasks (output match) |
97.7% |
67.8% |
75.5% |
62.3% |
| 100 open-ended tasks, v2 (syntax pass) |
83.0% |
65.0% |
55.0% |
48.0% |
| 89 output-determined of those (strict output match, lower bound) |
18.0% |
5.6% |
5.6% |
3.4% |
| 44 tutorial tasks, v2 (syntax pass) |
43.2% |
54.5% |
31.8% |
25.0% |
| 44 tutorial tasks, v2 (strict output match, 41 adjudicable) |
14.6% |
19.5% |
22.0% |
14.6% |
| 32 open-ended tasks (rubric judge) |
56.2% |
65.6% |
31.2% |
53.1% |
| 22 output-determined open-ended tasks (strict output match) |
50.0% |
59.1% |
50.0% |
54.5% |
| 32 open-ended tasks, no docs (rubric judge) |
56.2% |
46.9% |
40.6% |
37.5% |
Under the strict output-match metric, apparent tutorial differences between
models are statistically insignificant (overlapping 95% CIs); on the
32-task rubric benchmark the differences between FORM-8B, GLM-5.3 and
DeepSeek-V4 Flash are likewise within statistical noise, and the strict
re-adjudication of its 22 output-determined tasks confirms GLM-5.3's lead
there is genuine (13/22 solved vs. our 11/22, overlapping CIs). The Instruct-100
strict rate is a conservative lower bound: many of its instructions
underspecify the target computation, so a correct-in-spirit program can fail
the check (the same grading is applied to all models).
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained(
"unsloth/qwen3-8b-unsloth-bnb-4bit", load_in_4bit=True, device_map="auto")
model = PeftModel.from_pretrained(base, "bakar-chargeishvili/FORM-8B",
subfolder="adapters/form-qwen3-8b-v3c")
tok = AutoTokenizer.from_pretrained("unsloth/qwen3-8b-unsloth-bnb-4bit")
msgs = [{"role": "user", "content":
"Compute the trace of a product of gamma matrices contracted with "
"two external vectors, and simplify using the metric tensor in 4D."}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
enable_thinking=False)
Note: thinking mode is disabled — the model directly emits FORM code.
Limitations
- Trained on synthetic + tutorial-derived data; may not cover exotic FORM
features (parallel/local expressions, #procedure edge cases)
- Output is syntactically valid FORM but semantic correctness must be verified
by executing the program (this is by design — see paper §3)
- General reasoning/coding capability preserved to within ~2.6 pp of base
(MMLU, GSM8K, HumanEval) but not unchanged
License
Adapters inherit Apache-2.0 from the Qwen3-8B base model.
Benchmark data: CC-BY-4.0.
Citation
See the accompanying paper: "Verification-Driven Fine-Tuning of Small Language
Models for Zero-Shot FORM Symbolic Algebra Code Generation."
