Mapika/decider-4b v2.1 quantized to NVFP4 for vLLM: 4-bit floating-point weights and activations with FP8 block scales (block size 16). 3.29 GB against 8.41 GB for the bf16 checkpoint. Quantized and measured by LLM Tech; the model, its training and its evaluation protocol are Mapika's. Read the bf16 card for what the model is and how it was trained.
The base revision is eb5fbdfc9448473ec25e399882912863afbdb70e. Tokenizer, chat template, generation config and decider_config.json (temperatures included) are the author's files unchanged, apart from the version and quantization fields.
Accuracy against the bf16 weights
Both models were run through vLLM 0.29.0 on the same rows: the author's regression set rebuilt from public data (95 tasks, 67 in-task and 28 held-out, 144,226 rows, plain state-first layout, the author's temperature map) and the 231 public JevBench items. Our bf16 run matches the author's published regression accuracy and NLL within 0.0005 (in-task 0.8308 / 0.4145 NLL, held-out 0.7838 / 0.5689 in eval_results.json).
|
in-task acc / NLL / ECE (67 tasks) |
held-out acc / NLL / ECE (28 tasks) |
JevBench easy / standard / hard |
| bf16 |
0.8308 / 0.4146 / 0.0302 |
0.7837 / 0.5686 / 0.0773 |
48/48, 71/72, 73/111 |
| NVFP4 |
0.8247 / 0.4298 / 0.0312 |
0.7768 / 0.5933 / 0.0814 |
48/48, 70/72, 72/111 |
Accuracy moves by -0.6 points in-task and -0.7 held-out; NLL by +0.0152 and +0.0247. Per task: lower on 72, higher on 16, equal on 7; the largest drops are truthfulqa (-3.4, 817 rows), medqa (-3.3, 1273 rows), medmcqa (-3.0, 1500 rows), tweet_irony (-2.2, 784 rows). The public JevBench tiers are 48, 72 and 111 items; differences of one to three items are within their noise.
Speed
vLLM 0.29.0 on one RTX PRO 6000 Blackwell Server Edition (96 GB), prefix caching off, one output token per row, unique random states. Prefill throughput is the best over batch sizes 1 to 64; latency is one request alone.
| state length |
bf16 tokens/s |
NVFP4 tokens/s |
bf16 latency |
NVFP4 latency |
| 1,024 |
38,941 |
78,054 (x2.00) |
34 ms |
21 ms |
| 8,192 |
36,724 |
69,394 (x1.89) |
228 ms |
120 ms |
| 32,768 |
30,699 |
51,146 (x1.67) |
1082 ms |
657 ms |
Served by the author's HTTP server (decider.serve_vllm) next to the other two LLM Tech decider checkpoints on one GPU with DECIDER_VLLM_GPU_MEMORY_UTILIZATION=0.08, this checkpoint peaked at 10.5 GB under 32 concurrent requests of 29,033 tokens, with no errors. The same 29K-token state took 602 ms cold and 58 ms when repeated (prefix cache).
Usage
The author's package serves this checkpoint as is. decider.serve_vllm exposes POST /v1/systemone, the System One request shape (TypeSafe's Jev format):
pip install "decider-ai[serve]==1.6.0" vllm==0.29.0 ninja # vLLM builds kernels with ninja on first start
DECIDER_MODEL=llmtech/decider-4b-nvfp4 uvicorn decider.serve_vllm:app --port 8000
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
"model": "decider",
"state": "My card was charged twice for the same purchase.",
"questions": {
"dept": {"type": "choice", "instructions": "Which department should handle this?",
"criteria": {"billing": null, "technical support": null, "sales": null}},
"refund": {"type": "noul", "instructions": "Does this need a refund action?"}
}
}'
transformers alone does not run the NVFP4 weights; use vLLM. The checkpoint was run only through vLLM 0.29.0 on Blackwell (SM120) here.
What is quantized
NVIDIA ModelOpt 0.46.1, NVFP4_DEFAULT_CFG, with the exclusions the author used for decider-35b-a3b-nvfp4. Calibration: 512 prompts of at most 2,048 tokens (210 on average) drawn with seed 11 from the training side of the public decision tasks and the author's teacher data; 2 prompts that overlapped evaluation rows were dropped. No evaluation row was used for calibration or for any choice made here.
| kept in bf16 |
quantized |
embed_tokens, lm_head, linear_attn.conv1d, linear_attn.in_proj_a, linear_attn.in_proj_b, norms |
attention q_proj, k_proj, v_proj, o_proj; delta-net in_proj_qkv, in_proj_z, out_proj; MLP gate_proj, up_proj, down_proj (200 linear layers) |
The KV cache is not quantized.
Limitations
- Everything in the bf16 card applies.
- The quantization costs 0.6 points in-task and 0.7 held-out against bf16 in the same engine.
- Only the regression set and the public JevBench items were re-measured; the OpenJev, Mind2Web, browser, game and Bespoke numbers of the bf16 card were not.
- Measured with vLLM 0.29.0 on RTX PRO 6000 Blackwell only; not with TensorRT-LLM or SGLang.
- On borderline items the argmax can differ from bf16: on JevBench the outcome matched bf16 on 95 to 100% of items per tier. In a spot check through the HTTP server, a question the bf16 model answered with 0.62 confidence got a different answer.
About
Quantized and measured by LLM Tech; questions go to the Community tab. Model: Apache 2.0, by Mapika (GitHub).
