kev-4b-code-verify-v1

A fine-tune of jaredpalmer/kev-4b (4B, trained on a bf16 backbone on a single L4): a Jev-style decision model that judges whether code written by a cheap coding model is correct, so a router can keep the answer local or escalate it. One forward pass, a calibrated probability, no generated text.

Trained on execution-labelled data: coder attempts at HumanEvalPack (Python) tasks, labelled by running each task's hidden test suite. The model never sees the hidden tests; its input is what a router can compute itself.

How to ask it

The fine-tune binds these exact strings. One noul question (probability that the statement holds):

{"type": "noul", "instructions": "Does the code correctly and completely implement the request, so it would pass a thorough hidden test suite including edge cases? Judge from the request, the code and the checks shown."}

State (an object; Kev renders it as key: value lines):

{"request": "<the task, verbatim>",
  "code": "<the model's code>",
  "checks": {"compiles": true, "defines_requested_function": true, "passes_examples_in_request": true,
             "generated_edge_case_tests": "7 of 8 passed",
             "note": "edge-case tests were written by a small model from the request alone; some may be wrong"}}

generated_edge_case_tests comes from a small model writing ~8 asserts from the request only, then running them. Serve with Kev's runtime (python -m kev.serve --run jtatman/kev-4b-code-verify-v1, a TypeSafe /v1/systemone endpoint) or kev.predictors.LocalPredictor. The checkpoint carries its fitted temperature.

Results

Unseen coder (llama-3.1-8b-instruct, never in training): 488 attempts, pass rate 0.57. "Kept local @X% err" = share of attempts a router can accept, highest score first, with at most X% of them wrong. evidence score = (request examples passed + generated-assert pass rate) / 2.

Development (coders seen in training, tasks not seen):

Numbers are on small sets; see the project repo for task-bootstrap confidence intervals.

Recommended use

Average this model's probability with the execution evidence score, and tune the accept threshold per cheap-tier coder on a few hundred of that coder's execution-labelled attempts: a fixed threshold's error rate depends on how often the coder fails.

Training

• Init: jaredpalmer/kev-4b@139fdd9 (LoRA r=16 + pointer head, base Qwen/Qwen3.5-4B-Base), Kev's own trainer (kev.train, commit f2bb629d).
• Data: 1506 execution-labelled records (task-grouped split; llama-3.1-8b held out) + 2000 public decision-v7 replay records against forgetting.
• One epoch, lr 2e-05, batch 1 x accum 8, gradient checkpointing, bf16 autocast, state limit 1664 tokens, frozen backbone in bf16.
• Temperature fitted on a held-out calibration split (min NLL).
• Hardware: one NVIDIA L4 (Google Colab).

Training data provenance

Code attempts whose correctness this model learned to judge (labels from executing hidden tests):

Built with Qwen. Training data includes outputs of Qwen2.5-Coder-3B-Instruct, used under the Qwen Research License Agreement (section 4b). The held-out test coder (Llama 3.1 8B Instruct) was never used for training.

Limitations

• Python function-level tasks only (HumanEvalPack); other languages, repos and multi-file changes are untested.
• The coder pool is 8-10 models of 3B-235B parameters; accept thresholds do not transfer across coders.
• Trained on this question and state layout; other phrasings work less well.

Credits

Kev by Jared Palmer (jaredpalmer/kev, Apache-2.0); Qwen3.5 base (Apache-2.0); HumanEvalPack (bigcode). Project: https://github.com/jtatman/kev-code-verify