Mapika/decider-0.8b v1 quantized to FP8 for vLLM: FP8 E4M3 weights with one scale per output channel and FP8 activations scaled per token at run time. 1.01 GB against 1.5 GB for the bf16 checkpoint. Quantized and measured by LLM Tech; the model, its training and its evaluation protocol are Mapika's. Read the bf16 card for what the model is and how it was trained.
The base revision is a0a01d6f8135298f400a8c856b355793012ae971. Tokenizer, chat template, generation config and decider_config.json (temperatures included) are the author's files unchanged, apart from the version and quantization fields.
Accuracy against the bf16 weights
Both models were run through vLLM 0.29.0 on the same rows: the author's regression set rebuilt from public data (95 tasks, 67 in-task and 28 held-out, 144,226 rows, plain state-first layout, the author's temperature map) and the 231 public JevBench items. The author has not published numbers for the 0.8B on this protocol, so the reference is our bf16 run; the same harness matches the author's published numbers for the 2B and the 4B within 0.0005.
|
in-task acc / NLL / ECE (67 tasks) |
held-out acc / NLL / ECE (28 tasks) |
JevBench easy / standard / hard |
| bf16 |
0.7701 / 0.5389 / 0.0343 |
0.7236 / 0.6856 / 0.0866 |
48/48, 60/72, 46/111 |
| FP8 |
0.7692 / 0.5412 / 0.0332 |
0.7240 / 0.6921 / 0.0872 |
48/48, 62/72, 44/111 |
Accuracy moves by -0.1 points in-task and 0.0 held-out; NLL by +0.0023 and +0.0065. Per task: lower on 47, higher on 41, equal on 7; the largest drops are mind2web (-1.1, 1500 rows), copa (-1.0, 100 rows), mrpc (-1.0, 408 rows), medmcqa (-0.9, 1500 rows). The public JevBench tiers are 48, 72 and 111 items; differences of one to three items are within their noise.
Speed
vLLM 0.29.0 on one RTX PRO 6000 Blackwell Server Edition (96 GB), prefix caching off, one output token per row, unique random states. Prefill throughput is the best over batch sizes 1 to 64; latency is one request alone.
| state length |
bf16 tokens/s |
FP8 tokens/s |
bf16 latency |
FP8 latency |
| 1,024 |
173,187 |
215,165 (x1.24) |
16 ms |
16 ms |
| 8,192 |
160,356 |
194,589 (x1.21) |
55 ms |
44 ms |
| 32,768 |
119,228 |
139,616 (x1.17) |
287 ms |
247 ms |
Served by the author's HTTP server (decider.serve_vllm) next to the other two LLM Tech decider checkpoints on one GPU with DECIDER_VLLM_GPU_MEMORY_UTILIZATION=0.035, this checkpoint peaked at 5.0 GB under 32 concurrent requests of 29,033 tokens, with no errors. The same 29K-token state took 256 ms cold and 47 ms when repeated (prefix cache).
Usage
The author's package serves this checkpoint as is. decider.serve_vllm exposes POST /v1/systemone, the System One request shape (TypeSafe's Jev format):
pip install "decider-ai[serve]==1.6.0" vllm==0.29.0 ninja # vLLM builds kernels with ninja on first start
DECIDER_MODEL=llmtech/decider-0.8b-fp8 uvicorn decider.serve_vllm:app --port 8000
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
"model": "decider",
"state": "My card was charged twice for the same purchase.",
"questions": {
"dept": {"type": "choice", "instructions": "Which department should handle this?",
"criteria": {"billing": null, "technical support": null, "sales": null}},
"refund": {"type": "noul", "instructions": "Does this need a refund action?"}
}
}'
The checkpoint was run only through vLLM 0.29.0 on Blackwell (SM120) here.
What is quantized
llm-compressor 0.14.0, scheme FP8_DYNAMIC, no calibration data. The same modules as in the author's NVFP4 recipe stay in bf16.
| kept in bf16 |
quantized |
embed_tokens, lm_head, linear_attn.conv1d, linear_attn.in_proj_a, linear_attn.in_proj_b, norms |
attention q_proj, k_proj, v_proj, o_proj; delta-net in_proj_qkv, in_proj_z, out_proj; MLP gate_proj, up_proj, down_proj (150 linear layers) |
The KV cache is not quantized.
Limitations
- Everything in the bf16 card applies.
- The quantization costs 0.1 points in-task and -0.0 held-out against bf16 in the same engine.
- Only the regression set and the public JevBench items were re-measured; the OpenJev, Mind2Web, browser, game and Bespoke numbers of the bf16 card were not.
- Measured with vLLM 0.29.0 on RTX PRO 6000 Blackwell only; not with TensorRT-LLM or SGLang.
About
Quantized and measured by LLM Tech; questions go to the Community tab. Model: Apache 2.0, by Mapika (GitHub).
