QJev3.5-0.8B-GGUF

QJev 3.5-0.8B: a small decision engine you can run on a laptop

This is a 0.8B model we fine-tuned to make one narrow kind of decision very cheaply. It reads a state, a question, and a list of options, then returns a probability for each option. It never writes text. One forward pass, one token, and you have a distribution your code can branch on.

We built it for our own product first. An educational chatbot needed to decide, before spending a full LLM call, whether a student message was on topic, whether a tool call was safe to run, and whether an old tool result still had to be kept verbatim in the context. Those are small bounded judgments, and paying a 70B model to make them was absurd. It runs today on an office GPU at 80 to 150 ms per decision, and on a laptop with llama.cpp.

Files

Fidelity was measured on our 180 question product set: the merged models agree with the base-GGUF-plus---lora path on 179 of 180 (f16) and 178 of 180 (Q8_0) decisions, at 94.4% and 93.9% accuracy. Quantisation costs one or two answers in 180, nothing more.

What it does well, and what it does not

We ran the whole thing against the official Jev 1.13 API, against three open competitors, and against the base model without any fine-tuning. The short version is in the chart below.

Left two bars: our own task distribution, the one the model was trained for. Right two: held-out general benchmarks, where the base model is as good as our fine-tune.

On the distribution it was built for, fine-tuning is worth +67 points over the untouched base model (27.2% to 94.4% on 180 questions). On held-out general benchmarks it is worth nothing: the base model scores 50.6% on jev-bench held-out configs and our fine-tune scores 49.3%, with overlapping confidence intervals. The general axes belong to the backbone, and at 0.8B the backbone sits near 50%. The official API reaches 75.0% there. If general decision accuracy is what you need, use a larger model; the 4B prompt-based baselines land at 74 to 79% on JevBench.

We would rather write this down than let you find out after deploying.

Each point is a version we trained, scored on 231 public tasks (left) and 960 jev-bench rows (right). The solid line counts every source. The dashed line counts only sources the model never trained on. The two lines tell different stories about the same nine runs.

That gap is the most useful thing we learned. We spent two days improving a number that turned out to be mostly memorisation of the sources we trained on. After rebuilding the evaluation to hold out sources rather than records, the honest picture is: v1 to v11 gained 5 points on jev-bench held-out configs (44.3% to 49.3%) and 2.9 points on held-out JevBench families, where the confidence interval is 9 points wide. The rest was in-distribution.

Product family readout

The work we care about happens in four families: routing a message, judging a tool call, keeping or dropping context, and sorting an email. Each is measured on its own held-out questions. We extended the email and compaction sets past 100 questions each after discovering that 30 questions gave us a confidence interval 23 points wide.

Permissions and gate on the left, email and compaction on the right, both above 100 questions. Email is the hardest family and the one the next training run targets.

The four families are not equally hard, and that spread matters more than the average. Gate and permissions sit at 97 to 100%, email and compaction at 74 to 79%. Every training run we have done moved email the least, which is why it is the one we are working on now.

One honest note about versions. The released adapter is v5 because the later internal versions trade these two families away: on the 222 extended probes v5 scores email 69.1% and compaction 75.9%, while the newest internal version scores 64.5% and 63.4%. The older 30 question versions of these probes could not tell the two apart at all (both 83.3%), which is why we extended them. Every number in the tables above says which version it came from.

Latency, and how the alternatives compare

One protocol, p50 per decision. The official API number includes the network round trip; the local competitors were measured on their own recommended hardware.

Measured by us, each engine using its own prompt contract:

We are ahead where we trained and behind where we did not. For a model that has to fit in 812 MB and answer in under a tenth of a second, that trade seems right.

Reliability

149 of 180 questions land in the top confidence bin, and they are right 96% of the time. ECE 0.0427. Confidence follows the official Jev formula, c = (pmax - 1/K) / (1 - 1/K), computed from the option letter distribution.

Confidence here is a statistic of the distribution, not a claim that the answer is correct. Use it to route: act above a threshold, escalate below. Out of domain it is over-confident, and so is the official API. On the three jev-bench configs where human annotators disagree with each other (ChaosNLI, hate speech vote shares, Civil Comments), no model we measured tracks human uncertainty, the official one included. We report total variation distance to the human distribution rather than pretending that axis is solved.

Quick start

Pair the adapter with the base model, which is the smallest download:

llama-server -m Qwen3.5-0.8B-Q8_0.gguf --lora qjev35-lora-v5.gguf \
  --port 8080 -c 4096 --jinja \
  --chat-template-kwargs '{"enable_thinking": false}'

python qjev_client.py --url http://localhost:8080 \
  --state "The build failed: missing pyarrow in the test venv." \
  --question "What should the agent do next?" \
  --options "install the dependency" "retry the build" "ask the user"

Or run a merged file on its own:

llama-server -m qjev35-0.8b-q8_0.gguf --port 8080 -c 4096 --jinja \
  --chat-template-kwargs '{"enable_thinking": false}'

In LM Studio: download either merged GGUF, load it, start the local server from the Developer tab, and point qjev_client.py at http://localhost:1234 with the model name LM Studio shows.

qjev_client.py uses nothing but the standard library. It asks for one token with top_logprobs from /v1/chat/completions and renormalises the option letters:

from qjev_client import decide, decide_bool, decide_score

decide("http://localhost:8080",
       "Disk usage hit 96% on the build host.",
       "Should the agent stop the pipeline?",
       ["stop and clean up", "continue", "escalate to a human"])
# [('stop and clean up', 0.91), ('escalate to a human', 0.07), ('continue', 0.02)]

decide_bool("http://localhost:8080", evidence, "Does the passage entail the claim?")  # P(true)
decide_score("http://localhost:8080", ticket, "How urgent is this?", levels)         # distribution

Option order does not matter. The client evaluates the order you gave and the reverse order, then averages the two distributions. That trick is worth 2 to 4 accuracy points at this model size, and we have used it since the first version.

Prompt format

Set the chat template with thinking disabled. The model was trained on this shape:

system: You are a judgment engine. Read the question and the options, then decide which single
        option best matches the state. Answer with EXACTLY ONE letter and nothing else.

user:   [QUESTION]
        {question}

        [OPTIONS]
        A. {option 1}
        B. {option 2}

        [STATE]
        {state}

The next token is the option letter. Its probability, renormalised over the letters you declared, is your distribution. For yes/no questions pass two options (false, true). For ordered levels, list them in order. The released GGUF ships a pass-through chat template, so a chat endpoint hands this text to the model unchanged.

Read the letters from log probs. Sampling text and parsing it throws away the whole point.

Scope

Good for bounded decisions where you know the options when you call the model: tool gating, permission allow/ask/deny, prompt injection and content checks, routing, context compaction, email triage, picking an entity, browser actions, and four ordered score dimensions.

Not for open-ended reasoning, general knowledge, mathematics, multi-hop planning, long document question answering, or judging free-form assistant output. The held-out numbers above are the reason, and they are properties of the 0.8B backbone.

Option cap is 26, using positional letters A to Z. The official Jev allows 255. That is a design limit rather than an oversight, and it is why we cannot run 77 way (Banking77) or 151 way (clinc150) tasks. We report it as a limitation instead of working around it with a hack.

Training data and license

Seventeen families, all synthetic or converted from public datasets: gate, permissions, compaction, email, entity, browser actions, four score dimensions, NLI (SciTail, MNLI, ANLI), CLINC, Banking77, MASSIVE Chinese, ToolACE, WANLI, CMNLI, tldr-pages, and nl2bash. Per-source licences are listed in SOURCES.md in the dataset repository. ANLI (cc-by-nc-4.0) is used for internal training only and is stripped from every published export. Weights are Apache-2.0, following Qwen/Qwen3.5-0.8B.

How we evaluate

Five rules, written down after we broke them:

1. Hold out sources, not records.
2. Treat any difference smaller than its confidence interval as no evidence.
3. Fit on validation, report on test.
4. Require at least three training seeds before claiming one version beats another.
5. Keep domain numbers, held-out numbers, and competitor numbers in separate columns.

They are the reason the tables above are less flattering than our first release notes, and the reason we trust them more.

Known limitations

• Held-out general accuracy sits near 50%, which is where the 0.8B backbone sits.
• Five and six level ordinal scales that we did not train on score near chance, 25 to 27%.
• Confidence is over-confident out of domain, on our model and on the official API alike.
• The model leans on canonical option order more than one trained on shuffled orders would. Keep permutation averaging on, and if you train on this data yourself, shuffle the options per row. We found a fixed option order baked into 1,953 rows of our own training data; fixing it moved the position uniformity statistic from z = +6.6 to z = -0.2.

Citation

@misc{qjev35,
  title  = {QJev 3.5-0.8B: a small decision engine},
  author = {Lee, Jerry and contributors},
  year   = {2026},
  url    = {https://huggingface.co/ZY-Lee4/QJev3.5-0.8B-GGUF}
}

Independent work, written from scratch against the publicly described System One pattern. Not affiliated with, endorsed by, or connected to TypeSafe AI.