v3 container available. For NInfer engines at upstream v3 (master >= f76e19c0, 2026-09-13): the bytes-preserved v3 build of this artifact is at kaushikvira/Qwen3.8-27B-nvfp4full-dflash2-NInfer-v3 (qwen3_8_27b_nvfp4full-dflash2.v3.ninfer, sha256 74d2fc17…d5fdfc) and needs no engine patch — the C++ nvfp4full profile registration is only required by this v2 container. Full lineage chain: v3-upgrade.json in the v3 repo.
Qwen3.8-27B nvfp4full + DFlash2 for NInfer
This repository contains the nvfp4full weight profile of
Qwen3.8-27B in the native
NInfer .ninfer artifact format, with the
z-lab DFlash2 speculative-decoding drafter module added in the upstream W8G32/BF16
format.
It is the qwen3.8-27b / nvfp4full identity (same as
cometkim/Qwen3.8-27B-nvfp4full-NInfer)
with the registered DFlash2 module appended — the base tensors are byte-for-byte
identical to cometkim v1, and the MTP module is retained (unused and
validate-only under --spec dflash2).
Format note. Unlike
gpillon/Qwen3.8-27B-nvfp4full-dflash2-NInfer
(NVFP4 weight-only drafter), the DFlash2 module here uses the upstream
W8G32_F16S / BF16 encoding, which is the format our
fork (upstream master + 1 commit)
binds. The two artifacts are not interchangeable.
Artifact
| Field |
Value |
| Filename |
qwen3_8_27b_nvfp4full-dflash2.ninfer |
| Size |
20,550,864,896 bytes (19.14 GiB) |
| SHA-256 |
373dab08733f96555d964e02c6cebe40b23f753c0b2c4592fe130d641f937de1 |
| Container version |
2 |
| NInfer model ID |
qwen3.8-27b |
| NInfer weights ID |
nvfp4full |
| NInfer target key |
qwen3_8_27b |
| Stored objects |
1,325 (1,259 inherited from cometkim v1 + 66 DFlash2) |
| DFlash2 module objects |
66 (21 × W8G32_F16S matrices, 45 × BF16 norms/conv bases/codebooks) |
| MTP module |
retained (validate-only with --spec dflash2) |
Verify a downloaded file with:
printf '%s %s\n' \
'373dab08733f96555d964e02c6cebe40b23f753c0b2c4592fe130d641f937de1' \
'qwen3_8_27b_nvfp4full-dflash2.ninfer' | sha256sum --check
Lineage / provenance
| Field |
Value |
| Base repository |
cometkim/Qwen3.8-27B-nvfp4full-NInfer (v1, module-less) |
| Base recipe |
qwen3_8_27b_nvfp4full-v1 (cometkim fork, feat/qwen3.8-nvfp4full) |
| Base artifact SHA-256 |
2f59cc27d67cb7acba0ba8a0e0881ac89c1db2b267a60119a696fefa12faf4e7 |
| Base model |
Qwen/Qwen3.8-27B (revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0) |
| Quantized source |
unsloth/Qwen3.8-27B-NVFP4 (revision 7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108, per cometkim's conversion.json) |
| DFlash2 drafter |
z-lab/Qwen3.8-27B-DFlash2 (revision 50307d4c4cde6860d4eee73e2547cd786fe8e8a4) |
| DFlash2 encoding |
W8G32_F16S (matrices) + BF16 (norms / conv bases / selector codebooks) |
| Graft recipe |
qwen3_8_27b_nvfp4full-dflash2-w8 via tools/artifact/graft_dflash2_w8.py |
| Graft report |
qwen3_8_27b_nvfp4full-dflash2.ninfer.graft.json (included) |
The base tensors are copied from cometkim v1 byte-for-byte (no
re-encoding); only the 66 DFlash2 module objects are produced by the graft
tool from the z-lab checkpoint. Verified: source identity
qwen3.8-27b/nvfp4full, 1,259 source objects, no pre-existing dflash2/*
objects, output = 1,325.
Requirements
- NInfer
main (= upstream
487f8977 + commit 2eb59dbc), or upstream master at/after DFlash2
binding (385b30ce) plus the qwen3.8-27b/nvfp4full weights
registration — upstream Neroued/ninfer alone does not register the
nvfp4full profile yet;
- 64-bit Linux;
- NVIDIA GeForce RTX 5090 (
sm_120a);
- CUDA Toolkit 13.1 or newer.
Download and run a CLI example
hf download kaushikvira/Qwen3.8-27B-nvfp4full-dflash2-NInfer \
qwen3_8_27b_nvfp4full-dflash2.ninfer \
--local-dir models
./build/apps/ninfer models/qwen3_8_27b_nvfp4full-dflash2.ninfer \
--prompt "Explain prefill and decode in three sentences." \
--max-context 32768 \
--max-new 8192 \
--kv-dtype k8v4 \
--spec dflash2 --draft-tokens 7
Start a local server
./build/apps/ninfer-serve models/qwen3_8_27b_nvfp4full-dflash2.ninfer \
--host 127.0.0.1 \
--port 8080 \
--max-context 262144 \
--kv-capacity auto \
--max-concurrency 4 \
--kv-dtype k8v4 \
--spec dflash2 --draft-tokens 7 \
--preserve-thinking
--spec dflash2 --draft-tokens 7 selects the DFlash2 drafter (K=7,
block width 8; K=1..15 supported for the z-lab module, 7 recommended).
A full sanitized serving profile (same flags, no secrets) is at
examples/ninfer-nvfp4full-grafted-dflash2.cfg.example
in the engine fork. Leave --lm-head-draft off (the full-vocabulary proposal head is the
default and DFlash2 candidates span the whole vocabulary). --spec mtp
still works — the MTP module is retained.
Supported use
- text generation in thinking and non-thinking modes;
- image, multi-image, video, and mixed multimodal messages (
--vision);
- DFlash2 speculative decoding, draft windows 1–15 (recommended 7);
- MTP speculative decoding (
--spec mtp, windows 1–5);
- BF16, INT8, FP8, NVFP4, and K8V4 KV cache (
--kv-dtype);
- CUDA Graph decode and compatible-prefix reuse;
- startup-bounded small-scale concurrent serving (1–8 active requests);
- the NInfer CLI and OpenAI/Anthropic-compatible HTTP serving.
Performance (measured on our rig, 2026-09-06)
One NVIDIA GeForce RTX 5090 (32 GB, 450 W cap), CUDA 13.1, DFlash2 K=7,
--kv-dtype k8v4, --max-context 262144, --max-concurrency 4, vision on.
Probes: bench/tests/perf.py, bench/tests/conc_perf.py,
bench/tests/needle.py (same probes used for the engine A/B).
| Metric |
Value |
| Weights on device |
18.7 GiB |
| Device KV pool |
283,840 tokens (k8v4) |
| Free VRAM after startup |
~2.75 GiB |
| Decode (C=1, warm) |
175.2 tok/s |
| Prefill 12,582 tok |
1.27 s TTFT (9,901 tok/s) |
| Prefill 50,082 tok |
6.83 s TTFT (7,331 tok/s) |
| Prefill 100,082 tok |
18.66 s TTFT (5,363 tok/s) |
| Prefill 200,082 tok |
58.20 s TTFT (3,438 tok/s) |
| Aggregate decode C=4 |
527.1 tok/s (140.8–152.1 per stream) |
| Needle (long-context retrieval, 12 cases) |
12/12 exact, 0 corrupted |
| DFlash2 acceptance (short smoke) |
~24.5% |
Same rig, same probes — the reference profile (cometkim nvfp4full + MTP4 +
k8v4, our previous base) decodes at 157.9 tok/s with needle 12/12; this
artifact is +11% decode at equal quality on the same KV codec.
These are our own numbers (single sample per probe, deterministic where
possible) — not vendor benchmarks; reproduction commands in the repository.
Limits
- Accepted only by a NInfer build that registers
qwen3.8-27b/nvfp4full
(our fork, or upstream after the registration lands);
- one RTX 5090 / one CUDA device, startup-fixed capacity 1–8 requests;
- no multi-GPU, offload, preemptive batching, or distributed serving;
- context allocation is subject to GPU memory and the KV-cache type —
on a 32 GB card with k8v4 the auto KV pool lands at ~284k tokens, still
≥ 1 full 262,144-token context; host-RAM continuation reuse is a
server flag, not this artifact;
- NInfer does not execute generated tool calls.
License
Apache-2.0. Base repositories, quantized source, and the DFlash2 drafter
are also Apache-2.0.
