Mapika/decider-4b v2.1 quantized to FP8 for vLLM: FP8 E4M3 weights with one scale per output channel and FP8 activations scaled per token at run time. 4.85 GB against 8.41 GB for the bf16 checkpoint. Quantized and measured by LLM Tech; the model, its training and its evaluation protocol are Mapika's. Read the bf16 card for what the model is and how it was trained.
The base revision is eb5fbdfc9448473ec25e399882912863afbdb70e. Tokenizer, chat template, generation config and decider_config.json (temperatures included) are the author's files unchanged, apart from the version and quantization fields.
Accuracy against the bf16 weights
Both models were run through vLLM 0.29.0 on the same rows: the author's regression set rebuilt from public data (95 tasks, 67 in-task and 28 held-out, 144,226 rows, plain state-first layout, the author's temperature map) and the 231 public JevBench items. Our bf16 run matches the author's published regression accuracy and NLL within 0.0005 (in-task 0.8308 / 0.4145 NLL, held-out 0.7838 / 0.5689 in eval_results.json).
|
in-task acc / NLL / ECE (67 tasks) |
held-out acc / NLL / ECE (28 tasks) |
JevBench easy / standard / hard |
| bf16 |
0.8308 / 0.4146 / 0.0302 |
0.7837 / 0.5686 / 0.0773 |
48/48, 71/72, 73/111 |
| FP8 |
0.8300 / 0.4157 / 0.0311 |
0.7823 / 0.5709 / 0.0788 |
48/48, 71/72, 74/111 |
Accuracy moves by -0.1 points in-task and -0.1 held-out; NLL by +0.0011 and +0.0023. Per task: lower on 44, higher on 34, equal on 17; the largest drops are fin_phrasebank (-1.2, 970 rows), commonsense_qa (-0.9, 1221 rows), openbookqa (-0.8, 500 rows), wic (-0.8, 638 rows). The public JevBench tiers are 48, 72 and 111 items; differences of one to three items are within their noise.
Speed
vLLM 0.29.0 on one RTX PRO 6000 Blackwell Server Edition (96 GB), prefix caching off, one output token per row, unique random states. Prefill throughput is the best over batch sizes 1 to 64; latency is one request alone.
| state length |
bf16 tokens/s |
FP8 tokens/s |
bf16 latency |
FP8 latency |
| 1,024 |
38,941 |
56,393 (x1.45) |
34 ms |
24 ms |
| 8,192 |
36,724 |
52,205 (x1.42) |
228 ms |
160 ms |
| 32,768 |
30,699 |
40,786 (x1.33) |
1082 ms |
817 ms |
Usage
The author's package serves this checkpoint as is. decider.serve_vllm exposes POST /v1/systemone, the System One request shape (TypeSafe's Jev format):
pip install "decider-ai[serve]==1.6.0" vllm==0.29.0 ninja # vLLM builds kernels with ninja on first start
DECIDER_MODEL=llmtech/decider-4b-fp8 uvicorn decider.serve_vllm:app --port 8000
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
"model": "decider",
"state": "My card was charged twice for the same purchase.",
"questions": {
"dept": {"type": "choice", "instructions": "Which department should handle this?",
"criteria": {"billing": null, "technical support": null, "sales": null}},
"refund": {"type": "noul", "instructions": "Does this need a refund action?"}
}
}'
The checkpoint was run only through vLLM 0.29.0 on Blackwell (SM120) here.
What is quantized
llm-compressor 0.14.0, scheme FP8_DYNAMIC, no calibration data. The same modules as in the author's NVFP4 recipe stay in bf16.
| kept in bf16 |
quantized |
embed_tokens, lm_head, linear_attn.conv1d, linear_attn.in_proj_a, linear_attn.in_proj_b, norms |
attention q_proj, k_proj, v_proj, o_proj; delta-net in_proj_qkv, in_proj_z, out_proj; MLP gate_proj, up_proj, down_proj (200 linear layers) |
The KV cache is not quantized.
Limitations
- Everything in the bf16 card applies.
- The quantization costs 0.1 points in-task and 0.1 held-out against bf16 in the same engine.
- Only the regression set and the public JevBench items were re-measured; the OpenJev, Mind2Web, browser, game and Bespoke numbers of the bf16 card were not.
- Measured with vLLM 0.29.0 on RTX PRO 6000 Blackwell only; not with TensorRT-LLM or SGLang.
About
Quantized and measured by LLM Tech; questions go to the Community tab. Model: Apache 2.0, by Mapika (GitHub).
