A LoRA adapter for Qwen2.5-Coder-7B-Instruct that predicts, from the code
alone, whether a Verilog design change will fail its test bench before any
simulator is run.
It exists to sit in front of the expensive evaluation stage of an agentic
hardware design loop. Simulation is slow and costly, and many agent proposals
are simply wrong. This model triages them.
Why it beats larger models at this task
Every frontier model tested carries the same wrong prior: that any edit to
working code is probably a bug. That produces near-perfect recall and poor
precision, so the gate throws away a large share of perfectly good designs.
This adapter was trained on data containing verified-harmless edits
alongside real faults, which attacks that bias directly.
| Accuracy by edit family |
Untrained |
Tuned |
| benign (behaviour preserved) |
55.6% |
94.8% |
| faulty (behaviour changed) |
65.6% |
79.4% |
Near chance to near ninety-five on harmless edits. That 39-point gain is the
whole result.
Results
630 held-out examples, from VerilogEval problems absent from training
entirely. Positive class is FAIL.
| Model |
Accuracy |
Precision (FAIL) |
Recall (FAIL) |
Cost / 1k |
Latency |
| Qwen2.5-Coder-7B, untrained |
61.4% |
56.1% |
73.4% |
$0.049 |
0.25s |
| This adapter |
85.2% |
83.3% |
84.8% |
$0.063 |
0.32s |
| Gemini 2.5 Flash |
86.2% |
77.9% |
97.6% |
$0.163 |
6.47s |
| Gemini 2.5 Pro |
81.5% |
71.5% |
99.7% |
$0.676 |
14.4s |
| Gemini 2.5 Flash-Lite |
68.1% |
59.3% |
96.9% |
$0.054 |
0.42s |
Majority-class baseline is 54.1%.
Precision is the measure that matters here. A false FAIL silently
discards a working design, which hurts a design loop more than paying for one
extra simulation. This adapter has the highest precision of any model tested,
at roughly a quarter of Flash's cost and a twentieth of its latency. Flash
retains clearly better recall.
Tunable operating point
| Threshold |
Accuracy |
Precision |
Recall |
Sims skipped |
Good designs lost |
| 0.50 |
85.2% |
83.3% |
84.8% |
294 |
49 of 341 |
| 0.70 |
83.7% |
86.9% |
75.8% |
252 |
33 of 341 |
| 0.90 |
80.2% |
92.7% |
61.6% |
192 |
14 of 341 |
| 0.95 |
77.0% |
95.0% |
52.6% |
160 |
8 of 341 |
Usage
from chia_governor import TriageGate, LocalScorer
gate = TriageGate(
LocalScorer("Qwen/Qwen2.5-Coder-7B-Instruct",
adapter="emailvenky/chia-governor-triage-7b",
load_4bit=True),
threshold=0.9,
)
if gate.allow(spec, candidate_code):
result = run_simulation(candidate_code)
Or directly. Score by comparing the PASS and FAIL logits rather than
generating text, which is deterministic and cannot return an unparseable
answer:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-Coder-7B-Instruct", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "emailvenky/chia-governor-triage-7b")
The prompt format is fixed and must match training: a system message
describing the task, then the specification and the candidate code, asking for
one word. See chia_governor/triage_gate.py for the exact strings.
Training
|
|
| Base |
Qwen/Qwen2.5-Coder-7B-Instruct |
| Method |
LoRA, rank 16, alpha 32, dropout 0.05 |
| Trainable |
40.4M parameters (0.53%) |
| Quantisation |
4-bit NF4 |
| Data |
1,941 examples, labels from real Icarus Verilog runs |
| Epochs |
2 (244 optimizer steps) |
| Hardware |
One NVIDIA L4, 72 minutes |
Loss fell from 0.23 and plateaued near 0.10 rather than collapsing toward
zero, which is the shape of a learned decision rule rather than memorisation.
Data
Generated rather than collected, with labels correct by construction. Working
VerilogEval references are edited by script and every variant is simulated
against its real test bench. Two families: eight faulty operators
(flipped comparisons, swapped operators, off-by-one constants, altered bit
ranges, changed reset values, non-blocking to blocking, dropped assignments,
swapped signals) and six benign ones (commutative swaps, internal renames,
redundant parentheses, reformatted literals, reordered independent assigns,
dead wires).
Both families are necessary. A first pass with faulty edits only produced a
set that was 79.7% FAIL, scoreable at 80% by always answering FAIL. Adding
benign edits brought it to 44.8% and made the task meaningful.
Train and test are split by problem, so no VerilogEval problem appears on
both sides.
Limitations
- Recall is the weak dimension. At 84.8%, roughly one failing candidate in
six gets through, where Gemini Flash catches it.
- Faults are synthetic. They imitate plausible agent errors but are not
drawn from real agent output.
- Small modules only. VerilogEval modules are far smaller than an
industrial RTL block; transfer is untested.
- Contamination is plausible. VerilogEval may appear in pretraining data.
The mutated variants do not, but the base problems may.
- One run, no seeds. No variance reported.
Links
- Code, dataset and evaluation: https://github.com/venkateshrajag/chia
- Built for the CHIA framework: https://chialoops.ai
Licence
BSD-3-Clause, matching CHIA.
