Mapika/decider-2b v11 quantized to FP8 for vLLM: FP8 E4M3 weights with one scale per output channel and FP8 activations scaled per token at run time. 2.39 GB against 3.77 GB for the bf16 checkpoint. Quantized and measured by LLM Tech; the model, its training and its evaluation protocol are Mapika's. Read the bf16 card for what the model is and how it was trained.
The base revision is 533964dae8be954c5b5e19fa4948e48408094c1e. Tokenizer, chat template, generation config and decider_config.json (temperatures included) are the author's files unchanged, apart from the version and quantization fields.
Accuracy against the bf16 weights
Both models were run through vLLM 0.29.0 on the same rows: the author's regression set rebuilt from public data (95 tasks, 67 in-task and 28 held-out, 144,226 rows, plain state-first layout, the author's temperature map) and the 231 public JevBench items. Our bf16 run matches the author's published regression accuracy and NLL within 0.0005 (in-task 0.8016 / 0.4809 NLL, held-out 0.7518 / 0.6262 in eval_results.json).
|
in-task acc / NLL / ECE (67 tasks) |
held-out acc / NLL / ECE (28 tasks) |
JevBench easy / standard / hard |
| bf16 |
0.8016 / 0.4809 / 0.0383 |
0.7516 / 0.6261 / 0.0829 |
48/48, 64/72, 63/111 |
| FP8 |
0.8013 / 0.4826 / 0.0385 |
0.7518 / 0.6255 / 0.0815 |
48/48, 64/72, 64/111 |
Accuracy moves by 0.0 points in-task and 0.0 held-out; NLL by +0.0017 and -0.0006. Per task: lower on 43, higher on 39, equal on 13; the largest drops are strategyqa (-1.2, 687 rows), helpsteer3_pref (-1.0, 1176 rows), piqa (-0.7, 1500 rows), arc (-0.6, 1172 rows). The public JevBench tiers are 48, 72 and 111 items; differences of one to three items are within their noise.
Speed
vLLM 0.29.0 on one RTX PRO 6000 Blackwell Server Edition (96 GB), prefix caching off, one output token per row, unique random states. Prefill throughput is the best over batch sizes 1 to 64; latency is one request alone.
| state length |
bf16 tokens/s |
FP8 tokens/s |
bf16 latency |
FP8 latency |
| 1,024 |
95,360 |
135,434 (x1.42) |
18 ms |
16 ms |
| 8,192 |
90,099 |
126,514 (x1.40) |
94 ms |
68 ms |
| 32,768 |
75,234 |
99,415 (x1.32) |
448 ms |
343 ms |
Served by the author's HTTP server (decider.serve_vllm) next to the other two LLM Tech decider checkpoints on one GPU with DECIDER_VLLM_GPU_MEMORY_UTILIZATION=0.05, this checkpoint peaked at 6.7 GB under 32 concurrent requests of 29,033 tokens, with no errors. The same 29K-token state took 336 ms cold and 50 ms when repeated (prefix cache).
Usage
The author's package serves this checkpoint as is. decider.serve_vllm exposes POST /v1/systemone, the System One request shape (TypeSafe's Jev format):
pip install "decider-ai[serve]==1.6.0" vllm==0.29.0 ninja # vLLM builds kernels with ninja on first start
DECIDER_MODEL=llmtech/decider-2b-fp8 uvicorn decider.serve_vllm:app --port 8000
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
"model": "decider",
"state": "My card was charged twice for the same purchase.",
"questions": {
"dept": {"type": "choice", "instructions": "Which department should handle this?",
"criteria": {"billing": null, "technical support": null, "sales": null}},
"refund": {"type": "noul", "instructions": "Does this need a refund action?"}
}
}'
The checkpoint was run only through vLLM 0.29.0 on Blackwell (SM120) here.
What is quantized
llm-compressor 0.14.0, scheme FP8_DYNAMIC, no calibration data. The same modules as in the author's NVFP4 recipe stay in bf16.
| kept in bf16 |
quantized |
embed_tokens, lm_head, linear_attn.conv1d, linear_attn.in_proj_a, linear_attn.in_proj_b, norms |
attention q_proj, k_proj, v_proj, o_proj; delta-net in_proj_qkv, in_proj_z, out_proj; MLP gate_proj, up_proj, down_proj (150 linear layers) |
The KV cache is not quantized.
Limitations
- Everything in the bf16 card applies.
- The accuracy change against bf16 in the same engine is within 0.1 points on both halves of the regression set.
- Only the regression set and the public JevBench items were re-measured; the OpenJev, Mind2Web, browser, game and Bespoke numbers of the bf16 card were not.
- Measured with vLLM 0.29.0 on RTX PRO 6000 Blackwell only; not with TensorRT-LLM or SGLang.
About
Quantized and measured by LLM Tech; questions go to the Community tab. Model: Apache 2.0, by Mapika (GitHub).
