PrismaCoder-3B is a Qwen2.5-Coder-3B-Instruct
fine-tune specialized for generating and editing Prisma ORM
schemas and client code from natural-language instructions. It was trained
with a full SFT → DPO → GRPO pipeline, using a verifier (gate) as both the
data-quality filter and the reinforcement-learning reward signal.
Evaluation status: full 1,200-task PrismaBench has been run and the
numbers below are real. They currently reflect static (V0) gate
checking only — parses correctly, code fences balanced, referenced
models/fields grounded in the schema — not verified compile-and-execute
correctness against a real database (the full V1–V5 gate chain, which
needs a Node ≥18 environment not available during this evaluation). See
Evaluation and Limitations below before treating pass@1 as "code that
works."
Model Details
- Base model: Qwen/Qwen2.5-Coder-3B-Instruct
- Parameters: ~3B
- Fine-tuning method: QLoRA (r=16, α=32, all linear projections) at each
stage, LoRA weights merged into the base model between stages
- Training stages: SFT → DPO → GRPO (online RL, verifier reward) →
iterative self-play RL round (attempted; see Limitations)
- Precision: 4-bit NF4 (bitsandbytes) during training, fp16/bf16 merged
weights for release
- License: apache-2.0 (matches the base model)
Training Procedure
| Stage |
Method |
Key hyperparameters |
| SFT |
Completion-only loss, soft-ordering difficulty mixture |
QLoRA r16/α32, lr 1e-4, cosine schedule, 3% warmup, effective batch 16, 3 epochs |
| DPO |
Preference optimization on gate-differential pairs (static + on-policy mined) |
β=0.05, lr 5e-7, 1 epoch, effective batch 16 |
| GRPO |
Online RL, reward = the same verifier used to filter training data |
lr 1e-6, β=0.04, group size 2, effective batch 8 completions/step |
| RL round 2 |
Best-of-k self-play distilled into a short DPO round |
Attempted; skipped this run (see Limitations) |
All stages used seed 20260830 and ran on a single Kaggle T4 GPU (16GB) with
gradient checkpointing and paged 8-bit AdamW.
Reward/gate design: every stage is filtered or scored by the same static
verifier (LightValidator, "V0" in the project's internal gate numbering):
it checks that generated Prisma schema/client code parses, has balanced code
fences, and that referenced models/fields are actually grounded in the
provided schema. It does not check that the generated client code
compiles or executes correctly against a real database — that requires the
full gate chain (V1–V5, Node-based), which did not run in this training
environment. See Limitations.
Training Data
- Corpus: PrismaForge, a two-track dataset — a small mined track (real
schema.prisma files and call sites crawled from permissively-licensed
public repos) and a larger synthetic track (mechanically generated
task/schema pairs across families, domains, and difficulty levels).
- Scale actually used for this model: ~3,000 of ~17,500 available
finalized training records (
MAX_TRAIN_SAMPLES=3000). This is smaller
than the project's full-scale design target (120,000 synthetic records);
results should be read as a proof-of-concept at this data scale, not the
ceiling of the method.
- Contamination firewall: training records whose provenance hash
matches a PrismaBench benchmark task are excluded from the training
splits before SFT.
Evaluation
Benchmark: PrismaBench, a 1,200-task benchmark (categories T1–T12,
difficulty levels L1–L3) built by the same project, with canary strings and
a provenance registry used for the contamination firewall above.
Gate mode: static V0 gates only (static evaluation mode (no Node or
--full-gates not passed) — printed by the evaluator on every run in this
environment). This distinction matters: V0-only pass@1 measures well-formed,
schema-grounded output; full-gate pass@1 (not yet run) would measure whether
the generated code actually compiles, migrates, and executes correctly.
| Stage |
pass@1 (40-prompt internal proxy, V0 gates) |
| Base model (untrained) |
0.700 |
| + SFT |
0.975 |
| + DPO |
0.975 |
| + GRPO |
0.975 |
| + RL round 2 |
0.975 |
Full PrismaBench, N=1,200, static V0 gates — checkpoint: merged/grpo
| Metric |
Value |
| pass@1 |
0.9992 |
| Wilson 95% CI |
[0.9953, 0.9999] |
| Exact match |
0.0175 |
| CodeBLEU |
0.9332 |
The low exact-match alongside near-perfect pass@1 is expected, not a
red flag: pass@1 checks gate-level correctness (parses, balanced fences,
schema-grounded references), not verbatim agreement with a single reference
solution — code that is correct but phrased differently from the reference
scores low on exact-match by design.
Per-category pass@1 (1,200 tasks):
| Category |
pass@1 |
| T1_schema_greenfield |
120/120 (1.00) |
| T2_schema_edit |
120/120 (1.00) |
| T3_migration_plan |
90/90 (1.00) |
| T4_client_crud |
150/150 (1.00) |
| T5_complex_reads |
139/140 (0.99) |
| T6_nested_writes_txn |
120/120 (1.00) |
| T7_aggregations |
100/100 (1.00) |
| T8_repo_completion |
120/120 (1.00) |
| T9_bug_fix |
100/100 (1.00) |
| T10_cross_file |
60/60 (1.00) |
| T11_performance |
60/60 (1.00) |
| T12_security |
20/20 (1.00) |
Error taxonomy: 1 failure classified E4; no other error classes
observed across 1,200 tasks.
Which checkpoint this reflects: RL round 2 also completed successfully
in training (self-play produced enough separable pairs this run, unlike an
earlier attempt) and reached the same 0.975 proxy score as GRPO, but the
1,200-task benchmark above was run against the GRPO checkpoint
(merged/grpo), not merged/rl2. The RL round 2 checkpoint has not yet
been scored on the full benchmark — see Next Steps in the project repo
before treating GRPO as the final, best checkpoint.
Baseline comparison: the only benchmark comparison currently valid for
this card is against the untrained Qwen2.5-Coder-3B-Instruct base model,
scored on the identical 1,200-task set (0.700 proxy pass@1 pre-training;
full-benchmark base-model pass@1 not separately run). Comparisons against
other released coding models are not included here because they have not
been run through this same evaluator on this same held-out task set — a
fair comparison requires that, and until it's done, any such claim on this
card would be a number this project didn't actually measure.
Intended Use
- Generating and editing Prisma schema definitions and Prisma Client
TypeScript code from natural-language instructions.
- Assistance with schema migrations, CRUD operations, and query
composition against an existing Prisma schema.
Out-of-Scope / Limitations
- Static gates only, by default. Unless the model card above states
full V1–V5 gates were used, evaluation numbers reflect syntactic/grounding
correctness, not verified compile-and-run correctness.
- RL round 2 completed but is not the published checkpoint. In the
training run that produced these results, RL round 2 (self-play
distillation on top of GRPO) completed successfully and matched GRPO's
proxy score, but has not yet been scored on the full 1,200-task benchmark.
The checkpoint published here is GRPO, not RL round 2. If RL round 2
is later found to score higher on the full benchmark, this card and the
published weights will be updated to reflect that — check the repo's
commit history for the latest evaluated checkpoint.
- DPO's curated (non-mined) pair supply was very small in this training
run (2 static gate-differential pairs); most of DPO's signal came from
on-policy mining rather than the project's intended curated pair set.
- Domain-specific. This model is fine-tuned narrowly for Prisma ORM
code generation and is not evaluated on, or intended for, general-purpose
coding tasks outside that domain.
- Not evaluated for safety, security-sensitive code generation (e.g.
auth/permission logic beyond the benchmark's T12_security category), or
adversarial inputs.
How to Use
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "sandeeprdy1729/PrismaCoder-3B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
messages = [
{"role": "system", "content": "You are a Prisma ORM code generation assistant."},
{"role": "user", "content": "Add a `Comment` model related to an existing `Post` model, with a body field and a createdAt timestamp."},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=640)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
A GGUF export (PrismaCoder-3B-f16.gguf) is also provided for
llama.cpp/Ollama-based inference.
Training Infrastructure
Trained on Kaggle (single NVIDIA T4, 16GB), 4-bit QLoRA throughout, with
step-level checkpointing every 10–25 steps and cross-session resume to fit
within Kaggle's 12-hour session limit.
Citation
If you use this model, please cite the base model and note this fine-tune:
@misc{prismacoder3b,
title = {PrismaCoder-3B: QLoRA/DPO/GRPO fine-tune of Qwen2.5-Coder-3B for Prisma ORM code generation},
author = {Thummala, Sandeep},
year = {2026},
url = {https://huggingface.co/sandeeprdy1729/PrismaCoder-3B}
}
Base model: Qwen2.5-Coder-3B-Instruct.
