Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2

A model post-trained with several rounds of SFT and RLOO on top of Qwen3.8-27B-EfficientThink-SFT-SimPO-DFlash2, which is itself the official Qwen3.8-27B post-trained with SFT and SimPO. It is built to fix the original model's habit of failing to stop: it already has an answer, yet keeps re-deriving until it hits the 94K cap with no final answer. This repository provides five safetensors tiers, BF16, static FP8 Block128, NVFP4 W4A16, NVFP4 W4A4, and NVFP4 W4A4-W8A8 (mixed precision), all fully multimodal (text + image/video) with the official BF16 MTP bundled; all load in SGLang without patches, and the FP8 package is also smoke-tested on vLLM. It also has NInfer W4A4 / W4A4-W8A8 (NVFP4-NInfer/), INT8 W8A8 (INT8/W8A8/), INT8 W8A16 (INT8/W8A16/, text + image), INT4 W4A16 (INT4/W4A16/, text + image, vLLM), and GGUF Q2 / Q3 / Q4 LynnStyle (plus Q8 MTP versions), Q6_K, and Q8_0 (GGUF/, GGUF-NInfer/, built-in MTP).

• Far fewer 94K truncations: under the same protocol, 4 → 1 on GPQA and 13 → 3 on LCB.
• No regression on any suite: GPQA 178/198, MMLU 450/500, LCB 90/100 (static FP8), vs. 177 / 444 / 83 for the original FP8.
• Lossless quantization: GPQA and LCB are identical across BF16, dynamic FP8, and static FP8.
• Two speculative-decoding options (mutually exclusive): about 180 tok/s per request with DFlash2 and 91 tok/s with MTP, vs. about 46 tok/s without speculation.

Related repositories

• GGUF / GGUF-NInfer (llama.cpp, NInfer, built-in MTP): Hugging Face · ModelScope
• NVFP4-NInfer: Hugging Face · ModelScope

Training method

Lineage: original Qwen3.8 27B (FP8; 177 / 444 / 83) → our SFT + SimPO110 final, i.e. the EfficientThink base (171 / 442 / 89) → K3 continuation SFT → week-2 SFT (week2dose, update-225) → first RLOO round (182 groups) → another SFT round (merge-sft-100), giving sft-base-rloo (177 / 448 / 90) → second RLOO round (172 groups) → Coder390 (178 / 445 / 90, dynamic FP8). Numbers are GPQA / MMLU / LCB, all full suites under the same 100K protocol.

Training base: this model starts from our own EfficientThink SFT + SimPO model (the SimPO110 final) and goes through several rounds of SFT and RLOO, which is itself a post-trained version of the official Qwen3.8-27B: Qwen3.8-27B-EfficientThink-SFT-SimPO-DFlash2.

The problem: after the model already has a local answer, it keeps re-running the same derivation with Wait / Actually until the context is full, and the final answer is empty. In most of these cases the model can solve the problem; it just does not stop.

RLOO data: 8 trajectories sampled per problem; only groups with both correct and wrong trajectories are kept (all-correct groups are dropped). Groups with 0 or 1 correct trajectory receive one reviewed short teacher trajectory, which replaces the shortest wrong trajectory in that group (27 groups). Final set: 172 groups, 1,376 trajectories: 859 correct and 517 wrong, including 68 empty answers.

Reward: penalties apply only to wrong trajectories; long correct reasoning still earns a positive reward, so necessary long reasoning is not suppressed.

Result: under the same protocol, 94K truncations drop from 4 to 1 on GPQA and from 13 to 3 on LCB (FP8) compared with the original Qwen3.8 27B (FP8), and none of the three scores falls below it (all FP8 figures; see the NVFP4 scores below).

Scores

Protocol: the same Coder390 merged weights; two RTX PRO 6000 GPUs, C8 per GPU, 100K context, 94,208-token generation cap, no client-side timeout, DFlash2 draft, SGLang; full GPQA 198, MMLU 500, and LCB 100. Empty answers count as wrong, truncated-but-correct answers count as correct, and every failed sample stays in the denominator.

GPQA and LCB are identical across all three precisions. MMLU varies between 445 and 450, which is sampling noise rather than a precision effect. Dynamic FP8 means online FP8 quantization of the BF16 weights at inference time; it is not shipped as a separate package.

All tiers: scores, reasoning length, 94K truncation, and empty answers

Reasoning length is usage.reasoning_tokens; a 94K truncation means the output hit the generation cap. LCB empty answers are the number of problems for which no runnable code could be extracted. The original, BF16, dynamic FP8, and static FP8 rows use two RTX PRO 6000 GPUs with C8 per GPU; the three NVFP4 tiers use a single RTX PRO 6000 at C8; the rest of the protocol is the same.

Bold marks values better than the original Qwen3.8 27B (FP8): a higher score, or lower P50 / P70 / P90, 94K truncations, or empty answers; values equal to or worse than the original are not bolded.

LCB improves the most: all 13 of the original's 94K truncations ended with empty code (13 truncations, 13 empty answers), and the Coder390 tiers cut this to 2–4. The 3 for W4A4 are problems 0, 53, and 96 (problem ids in the score file), likewise truncated with no code; of the 4 for W4A16, problems 6, 7, and 54 have no code, and problem 53 has 20 characters of code but is wrong; of the 2 for W4A4-W8A8, problem 11 has no code, and problem 6 has 2,338 characters of code but is wrong. On GPQA the original had 4 truncations at the 94K cap and the tiers have 1–2; on MMLU the original had 1 and the tiers have 0–2, with no clear change.

Details for six precisions (bucketed by reasoning length)

Buckets are "correct / questions in bucket", split by reasoning length, left-closed and right-open. | Suite | Precision | Score | Reasoning P50 / P70 / P90 | 94K trunc. | Empty | <2K | 2–12K | 12–24K | 24–48K | ≥48K | |---|---|---:|---|---:|---:|---:|---:|---:|---:|---:| | GPQA | BF16 | 178/198 | 3,196 / 8,200 / 28,417 | 2 | 2 | 80/85 | 66/68 | 17/20 | 12/18 | 3/7 | | GPQA | Dynamic FP8 | 178/198 | 2,815 / 8,371 / 27,744 | 1 | 1 | 78/81 | 68/70 | 18/21 | 11/17 | 3/9 | | GPQA | Static FP8 | 178/198 | 2,966 / 7,832 / 26,431 | 1 | 2 | 82/84 | 68/74 | 17/19 | 9/17 | 2/4 | | GPQA | NVFP4 W4A16 | 170/198 | 2,965 / 9,667 / 28,433 | 2 | 3 | 77/80 | 64/70 | 17/21 | 8/17 | 4/10 | | GPQA | NVFP4 W4A4 | 167/198 | 2,876 / 10,996 / 31,986 | 2 | 3 | 76/82 | 58/60 | 20/23 | 12/26 | 1/7 | | GPQA | NVFP4 W4A4-W8A8 | 172/198 | 2,953 / 10,010 / 35,731 | 2 | 4 | 79/82 | 58/63 | 19/22 | 14/22 | 2/9 | | MMLU | BF16 | 446/500 | 154 / 252 / 779 | 1 | 0 | 432/472 | 13/23 | 1/3 | 0/0 | 0/2 | | MMLU | Dynamic FP8 | 445/500 | 138 / 271 / 881 | 1 | 0 | 436/477 | 8/21 | 0/1 | 0/0 | 1/1 | | MMLU | Static FP8 | 450/500 | 154 / 279 / 937 | 1 | 0 | 435/469 | 13/27 | 2/3 | 0/0 | 0/1 | | MMLU | NVFP4 W4A16 | 439/500 | 159 / 277 / 1,061 | 1 | 0 | 427/471 | 11/26 | 0/1 | 1/1 | 0/1 | | MMLU | NVFP4 W4A4 | 440/500 | 168 / 306 / 967 | 2 | 0 | 426/469 | 13/28 | 0/1 | 0/0 | 1/2 | | MMLU | NVFP4 W4A4-W8A8 | 442/500 | 157 / 273 / 778 | 0 | 0 | 432/477 | 8/20 | 1/2 | 1/1 | 0/0 | | LCB | BF16 | 90/100 | 5,073 / 16,033 / 38,911 | 2 | 2 | 40/40 | 25/26 | 11/14 | 9/11 | 5/9 | | LCB | Dynamic FP8 | 90/100 | 4,791 / 15,866 / 41,168 | 4 | 4 | 38/39 | 25/26 | 11/14 | 12/13 | 4/8 | | LCB | Static FP8 | 90/100 | 4,511 / 16,027 / 43,992 | 3 | 3 | 39/39 | 24/26 | 13/13 | 12/14 | 2/8 | | LCB | NVFP4 W4A16 | 91/100 | 5,119 / 15,790 / 47,768 | 4 | 3 | 38/38 | 26/27 | 13/13 | 10/12 | 4/10 | | LCB | NVFP4 W4A4 | 92/100 | 6,338 / 15,771 / 54,096 | 3 | 3 | 33/33 | 30/31 | 9/11 | 13/14 | 7/11 | | LCB | NVFP4 W4A4-W8A8 | 88/100 | 4,731 / 15,122 / 41,344 | 2 | 1 | 40/41 | 21/24 | 16/16 | 7/10 | 4/9 |

NVFP4 tiers

Protocol: one RTX PRO 6000, C8, 100K context, 94,208-token generation cap, no client-side timeout, DFlash2 draft, SGLang; full GPQA 198, MMLU 500, and LCB 100.

Reasoning length, 94K truncations, and empty answers for the three NVFP4 tiers are in the "All tiers" table above. LCB correctness is judged by each problem's pass field. Raw records: evaluation/SCORES_NVFP4_W4A16.json, evaluation/SCORES_NVFP4_W4A4.json, and evaluation/SCORES_NVFP4_MIXED.json (the W4A4-W8A8 tier).

NInfer tiers: scores, reasoning length, 94K truncations, and empty answers

Protocol: scores carried over from the official triad run on 2026-10-05 on the same NInfer text path; one RTX PRO 6000, official NInfer engine (Neroued/ninfer), C8, 100K context, 94,208 generation cap, client not timed, no speculation; full GPQA 198 / MMLU 500 / LCB 100; empty answers count as wrong, truncations that still answer correctly count as correct, and every failure stays in the denominator. The packages uploaded now add a Q8 MTP head, the DFlash2 draft, and the proposal head on top of that text path; the text path is unchanged and the full triad was not re-run. Only the NInfer engine can load .ninfer; SGLang / vLLM / llama.cpp / transformers cannot.

Reasoning length is usage.reasoning_tokens (i.e. completion_tokens_details.reasoning_tokens); a 94K truncation means the generation cap was hit. LCB empty answers are the number of problems for which no runnable code could be extracted. Bold means better than the original Qwen3.8 27B (FP8).

For NInfer W4A4, the 2 LCB empty answers are problems 11 and 15, both truncated with no code; the 2 GPQA empty answers are problems 79 and 127, both truncated with an empty pred. MMLU has no truncations and no empty answers. For NInfer W4A4-W8A8, the 3 LCB empty answers are problems 17, 53, and 54 (problem 17 stopped normally with no code; 53 and 54 were truncated with no code); the 2 GPQA empty answers are problems 79 and 127, both truncated with an empty pred. MMLU has no truncations and no empty answers.

Details for the two NInfer tiers (by reasoning-length bucket)

Buckets are "correct / problems in bucket", half-open on the left. | Suite | Precision | Score | Reasoning P50 / P70 / P90 | 94K trunc. | Empty | <2K | 2–12K | 12–24K | 24–48K | ≥48K | |---|---|---:|---|---:|---:|---:|---:|---:|---:|---:| | GPQA | NInfer W4A4 | 177/198 | 2,934 / 7,646 / 25,804 | 2 | 2 | 76/83 | 69/71 | 17/22 | 14/19 | 1/3 | | GPQA | NInfer W4A4-W8A8 | 177/198 | 3,176 / 8,844 / 24,804 | 2 | 2 | 79/82 | 70/74 | 13/21 | 12/16 | 3/5 | | MMLU | NInfer W4A4 | 449/500 | 164 / 285 / 830 | 0 | 0 | 438/480 | 11/19 | 0/1 | 0/0 | 0/0 | | MMLU | NInfer W4A4-W8A8 | 444/500 | 163 / 281 / 923 | 0 | 0 | 435/478 | 8/21 | 1/1 | 0/0 | 0/0 | | LCB | NInfer W4A4 | 89/100 | 3,994 / 15,533 / 36,125 | 2 | 2 | 40/41 | 21/23 | 12/14 | 13/16 | 3/6 | | LCB | NInfer W4A4-W8A8 | 92/100 | 4,046 / 15,829 / 40,479 | 2 | 3 | 38/38 | 26/26 | 14/16 | 11/12 | 3/8 |

INT8 W8A8: scores, 94K truncations, and empty answers

Protocol: one RTX PRO 6000, vLLM 0.28, C8, 100K context, 94,208 generation cap, client not timed, no speculation, reasoning effort xhigh; full GPQA 198 / MMLU 500 / LCB 100; empty answers count as wrong, truncations that still answer correctly count as correct, and every failure stays in the denominator. INT8 W8A8 loads only in vLLM.

A 94K truncation means the generation cap was hit. LCB empty answers are the number of problems for which no runnable code could be extracted. Bold means better than the original Qwen3.8 27B (FP8); its numbers are in the all-tier table above.

Note: the INT8 W8A8 eval ran vLLM without a reasoning parser, so usage.reasoning_tokens is 0 throughout and reasoning plus answer were returned as content. GPQA and MMLU reasoning lengths were recounted from the raw text: the text before </think> in the content, counted with the BF16 tokenizer (the same method as the GGUF tiers). The LCB result file kept only the code, not the reasoning text, so it cannot be counted; that cell reads "—".

GPQA's 1 truncation is problem 127, with an empty pred, so it also counts as the empty answer. LCB's 2 truncations are problems 7 and 53: problem 7 left code that raised a runtime error and was judged wrong; problem 53 was truncated, but its code was still judged correct; LCB has no empty answers. MMLU has no truncations and no empty answers. By difficulty, LCB is hard 38/46, medium 30/31, and easy 23/23.

INT8 W8A16: scores, 94K truncations, and empty answers

Protocol: one RTX PRO 6000, vLLM 0.28, C8, 100K context, 94,208 generation cap, client not timed, reasoning effort xhigh, MTP speculation with 3 draft tokens (external draft INT8/W8A16/MTP-BF16/), --reasoning-parser qwen3; full GPQA 198 / MMLU 500 / LCB 100; empty answers count as wrong, truncations that still answer correctly count as correct, and every failure stays in the denominator. INT8 W8A16 loads only in vLLM (pack-quantized GPTQ W8A16). Vision and MTP smoke-tested on the same stack (MTP mean accept length 3.31; image OCR correct with and without thinking).

A 94K truncation means the generation cap was hit. LCB empty answers are the number of problems for which no runnable code could be extracted. Bold means better than the original Qwen3.8 27B (FP8); its numbers are in the all-tier table above.

Reasoning lengths come from usage.reasoning_tokens returned by vLLM. GPQA: 1 truncation (problem 79), 1 empty answer (problem 79). MMLU has no truncations and no empty answers. LCB: 1 truncation (atcoder:arc195_d:54), 1 empty answer (atcoder:arc195_d:54). By difficulty, LCB is easy 23/23, medium 31/31, hard 40/46.

Same-protocol TPS / concurrency tables will be added after the speed harness finishes; smoke decode was ~119 tok/s wall with MTP+vision on a short prompt.

INT4 W4A16: scores, 94K truncations, and empty answers

Protocol: one RTX PRO 6000, vLLM 0.28, C8, 100K context, 94,208 generation cap, client not timed, reasoning effort xhigh, MTP speculation with 3 draft tokens (built-in head), --reasoning-parser qwen3; full GPQA 198 / MMLU 500 / LCB 100; empty answers count as wrong, truncations that still answer correctly count as correct, and every failure stays in the denominator. The run was paused on 2026-10-09 and resumed with the same server settings; finished problems were kept.

A 94K truncation means the generation cap was hit. LCB empty answers are the number of problems for which no runnable code could be extracted. Bold means better than the original Qwen3.8 27B (FP8); its numbers are in the all-tier table above.

Reasoning lengths come from usage.reasoning_tokens returned by vLLM. GPQA: 1 truncation (problem 79), 1 empty answer (problem 79). MMLU has no truncations and no empty answers. LCB: 2 truncations (problem atcoder:arc182_e:6, atcoder:arc195_d:54), 2 empty answers (problem atcoder:arc182_e:6, atcoder:arc195_d:54). By difficulty, LCB is hard 37/46, medium 27/31, and easy 23/23. Mean MTP accept length over the run (vLLM log, 3 draft tokens): 2.81.

GGUF Q2 / Q3 / Q4 LynnStyle: scores, reasoning length, 94K truncations, and empty answers

Protocol: one RTX PRO 6000, llama.cpp (llama-server), C4, about 100K context per slot, 94,208 generation cap, client not timed, thinking on; full GPQA 198 / MMLU 500 / LCB 100; empty answers count as wrong, truncations that still answer correctly count as correct, and every failure stays in the denominator. Scored before the built-in MTP head was attached (external Q8 MTP draft); same quantized weights. The matching .ninfer packages under GGUF-NInfer/ are converted from the same GGUFs; the full triad was run only on the GGUF.

A 94K truncation means the generation cap was hit. LCB empty answers are the number of problems for which no runnable code could be extracted. Bold means better than the original Qwen3.8 27B (FP8); its numbers are in the all-tier table above.

Note: reasoning lengths are counted from the raw text with the BF16 tokenizer (GPQA/LCB: reasoning field; MMLU: text before the last line of response), not from the API reasoning_tokens.

GGUF Q6_K: scores, reasoning length, 94K truncations, and empty answers

Protocol: one RTX PRO 6000, llama.cpp (llama-server) loading GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q6_K-MTP.gguf, C4, 100K context, 94,208 generation cap, client not timed, reasoning effort xhigh; full GPQA 198 / MMLU 500 / LCB 100; empty answers count as wrong, truncations that still answer correctly count as correct, and every failure stays in the denominator. GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q6_K-MTP.ninfer is converted from the same GGUF with the same backbone quantization; the full triad was run only on the GGUF.

Reasoning length is usage.reasoning_tokens; a 94K truncation means the generation cap was hit. LCB empty answers are the number of problems for which no runnable code could be extracted. Bold means better than the original Qwen3.8 27B (FP8); its numbers are in the all-tier table above.

Note: for MMLU and LCB, llama.cpp did not return reasoning tokens separately (usage has no reasoning_tokens). MMLU reasoning length is the text before the last line of response, counted with the BF16 tokenizer (the same method as Q8_0); the LCB result file kept no reasoning text, so it cannot be counted and reads "—". GPQA reasoning tokens were counted separately by the eval script.

GPQA's 2 truncations are problems 79 and 127; both hit 94,208 with an empty pred, so they also count as the empty answers. MMLU has no truncations and no empty answers.

LCB has no 94K truncations; its 1 empty answer is problem 92, which stopped normally without producing code.

LCB is scored on llama.cpp: the LCB eval script gets replies from NInfer that are not valid JSON. This is a compatibility issue between the eval script and the API, not a model-quality issue; the same GGUF on llama.cpp passed every problem in the same spot-check batch.

GGUF Q8_0: scores, reasoning length, 94K truncations, and empty answers

Protocol: single RTX PRO 6000, llama.cpp (llama-server) loading GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q8_0-MTP.gguf, C4, 100K context, generation cap 94,208, no client timeout, thinking xhigh; full GPQA 198 / MMLU 500 / LCB 100; empty answers count as wrong; truncated-but-correct counts as correct. GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q8_0-MTP.ninfer is converted from the same GGUF; the full triad was run on GGUF only.

Reasoning length: GPQA from usage.reasoning_tokens; MMLU from text before the last newline of response, counted with the BF16 tokenizer; LCB from saved reasoning text. 94K truncation means hitting the generation cap. LCB empty means no runnable code was extracted. Bold beats original Qwen3.8 27B (FP8).

GPQA: 2 truncations, 0 empty. MMLU: 0 / 0. LCB: 3 truncations, 3 empty (length with no code).

LCB is scored on llama.cpp: the LCB harness gets non-JSON replies from NInfer — a harness issue, not model quality.

Quantization tiers

Community fork ninfer-fusion-kvmem: the Q2 / Q3 / Q4 LynnStyle NInfer packages (GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q4MTP.ninfer, GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-Q4MTP.ninfer, GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q4MTP.ninfer) carry a built-in MTP head re-encoded as Q4_K and load and run MTP on both the official NInfer-all master and the community fork ninfer-fusion-kvmem (tested 2026-10-11). The .gguf files in GGUF/ keep a Q4_0 MTP head, which llama.cpp supports. The Q8 MTP versions remain available as an alternative (same weights, only the MTP head re-encoded as Q8_0): GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q8MTP.gguf, GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-Q8MTP.gguf and GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.gguf for llama.cpp, and GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q8MTP.ninfer, GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-Q8MTP.ninfer and GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.ninfer for NInfer (model ids qwen3.8-27b-coder390-Q2LynnStyle-Q8Mtp, qwen3.8-27b-coder390-Q3LynnStyle-Q8Mtp, qwen3.8-27b-coder390-Q4LynnStyle-Q8Mtp).

Q8 MTP versions · check (2026-10-09, one RTX PRO 6000): NInfer-all master with --spec mtp --draft-tokens 3 --vision, int8 KV, 8,192 context; llama.cpp with --spec-type draft-mtp --spec-draft-n-max 4 and --mmproj; one image request each, short runs. The image question was answered correctly in every run (llama.cpp with thinking on and off).

BPW (bits per weight) is the effective average bit width of the whole package: the total bytes of the main-model weight files (.safetensors, including the vision tower, MTP, and scale tensors, excluding the DFlash2 draft) × 8 ÷ total parameter count. The parameter count is taken from the safetensors headers and is identical across the five tiers at 27,781,427,952 (an NVFP4 packed U8 tensor counts 2 elements per byte; scale tensors add no elements). The weight-file totals for BF16, static FP8, NVFP4 W4A16, NVFP4 W4A4, and NVFP4 W4A4-W8A8 are 55,563,008,568, 31,242,033,200, 20,593,097,872, 20,593,145,056, and 23,749,329,544 bytes respectively. Every tier is mixed precision (different layers use different bit widths), so the nominal bit width can mislead; BPW reflects quality density per unit of size.

BPW for the two NInfer tiers is the whole .ninfer file bytes × 8 ÷ total parameter count 27,781,427,952: both files were 23,477,856,260 bytes before the vision component was added, so BPW is 6.76 for each. Each .ninfer package includes the Q8 MTP head, the DFlash2 draft, and the proposal head; the vision component added on 2026-10-09 (295,720,448 bytes) is not counted in this BPW; the safetensors-based BPW above for NVFP4 / BF16 / FP8 excludes the DFlash2 draft, so the two formulas differ and BPW should not be compared directly across those families.

INT8 W8A8's BPW counts only its 8 INT8 text weight files (the BF16 vision file model_visual.safetensors, 921,497,224 bytes, is not counted): their total bytes, 29,480,791,128, × 8 ÷ the text parameter count 26,895,998,464, giving 8.77. The text parameter count comes from the safetensors headers of the 8 INT8 weight files (INT8 and BF16 tensors count their elements; scale tensors add no elements) and equals the BF16 package's count without the vision tower (460,730,096) and MTP (424,699,392); because the denominator differs from the 27,781,427,952 used above, BPW should not be compared directly with the other tiers.

BPW for the two GGUF Q8_0 packages is the single file's bytes × 8 ÷ the text + MTP parameter count 27,320,697,856: the GGUF is 29,069,202,688 bytes, giving 8.51, and the NInfer package was 29,065,983,488 bytes before the vision component was added, giving 8.51. BPW for the two GGUF Q6_K packages is the single file's bytes × 8 ÷ the text + MTP parameter count 27,320,697,856: the GGUF is 23,177,516,384 bytes, giving 6.79, and the NInfer package was 23,084,144,128 bytes before the vision component was added, giving 6.76. The parameter count comes from the GGUF header (866 tensors: text 26,895,998,464 + MTP 424,699,392); vision is not counted in the denominator (neither the llama.cpp mmproj nor the 295,720,448-byte vision part now inside each .ninfer). Because the denominator differs from both the 27,781,427,952 used above and the 26,895,998,464 used for INT8 W8A8, BPW should not be compared directly with the other tiers.

• Each tier carries its own copy of the DFlash2 draft (identical files, 2,407,384,680 bytes): FP8/DFlash2-FP8/, BF16/DFlash2-FP8/, NVFP4/W4A16/DFlash2-FP8/, NVFP4/W4A4/DFlash2-FP8/, and NVFP4/W4A4-W8A8/DFlash2-FP8/.
• For a BF16 DFlash2 draft (e.g. for V100 or engines that cannot load the FP8 draft), use the upstream incoai/Qwen3.8-27B-DFlash2.
• DFlash2 draft format: FP8 E4M3 weights with 128×128 block scales (weight_scale_inv), the same format as the FP8 body; q/k/v projections, fc, the candidate selector, convolutions and norms stay BF16. It replaced the earlier per-tensor-scale draft on 2026-10-08 (engines that only read block scales, such as radiance, loaded the old draft without error but accepted almost no draft tokens). Checked on one RTX PRO 6000 on 2026-10-09 (SGLang 0.5.21, FP8 body, --speculative-num-draft-tokens 8 --speculative-dflash-block-size 8, one request at a time, 6 coding prompts, 1,024-token cap): greedy mean accept length is 5.05 at 171 tok/s per request (median), against 4.99 at 178 tok/s for the earlier draft with the same command. At temperature 1.0 the two run at 155 and 152 tok/s. In SGLang the new draft accepts as well as the old one. The other DFlash2 speed numbers in this card were measured with the earlier draft.
• Speed figures for BF16 and static FP8 were measured on the static FP8 package (BF16 was not benchmarked separately and shares the same figures); the three NVFP4 tiers were each benchmarked on their own package.
• NVFP4 scores were measured on a single GPU at C8; see "NVFP4 tiers" above for the protocol.
• NInfer tier scores are carried over from the 2026-10-05 triad on the same text path (single GPU, C8, official NInfer, no speculation); see the NInfer tiers section above. NInfer speeds were measured separately on the new packages; see "NInfer tiers" under Best-TPS and concurrency recommendations below.
• INT8 W8A8 scores are a full triad measured on a single GPU at C8 with vLLM and no speculation; see the INT8 W8A8 section above for the protocol. Its speeds come from a quick benchmark; see "INT8 W8A8" under Best-TPS and concurrency recommendations below.
• INT4 W4A16 scores are a full triad measured on a single GPU at C8 with vLLM 0.28 and MTP (3 draft tokens, built-in head); see the INT4 W4A16 section above for the protocol, and "INT4 W4A16" under Best-TPS and concurrency recommendations below for speeds.
• GGUF Q6_K scores are a full triad measured on a single GPU at C4 with llama.cpp; see the GGUF Q6_K section above for the protocol. Its speeds come from a quick benchmark; see "GGUF Q6_K" under Best-TPS and concurrency recommendations below.
• GGUF Q8_0 scores are a full triad measured on a single GPU at C4 with llama.cpp; see the GGUF Q8_0 section above for the protocol. For speeds see "GGUF Q8_0" below (NInfer C8 MTP 486.3; llama.cpp C4 MTP 190.8).
• The two NInfer tiers are under NVFP4-NInfer/W4A4/ and NVFP4-NInfer/W4A4-W8A8/, INT8 W8A8 is under INT8/W8A8/, INT4 W4A16 is under INT4/W4A16/, and GGUF Q2 / Q3 / Q4 LynnStyle (plus their Q8 MTP versions), Q6_K, and Q8_0 are under GGUF/ and GGUF-NInfer/.

Best-TPS and concurrency recommendations

Environment: one RTX PRO 6000 Blackwell 96GB; SGLang, 32K context (32,768), 4,096-token generation cap, thinking enabled, model-default sampling.

• Which speculation: at every measured concurrency (C1–C16), DFlash2 beats MTP on both aggregate and per-request speed. Use DFlash2 whenever you can.
• When to use MTP: when you do not want to load the separate DFlash2 draft (DFlash2 has only been validated on SGLang; vLLM 0.28 + MTP: temporarily unavailable for FP8; use SGLang for now). If you run MTP only: use C1 for the fastest single request (91.1 tok/s), C4 for interactive use with a few users (293.3 aggregate, 84.0 per request), C8 for shared serving (380.3 aggregate, 71.1 per request), and C16 when only total throughput matters (612.7 aggregate, 60.1 per request).
• Concurrency: C1–C4 for interactive use, C8 for shared or batch serving, C16 when only total throughput matters. Concurrency above C16 was not tested.

Full measured table

C1 is the main run (4 requests); C4/C8/C16 are reruns with more requests (6×C per cell). "vs. no spec" is the ratio of aggregate throughput at the same concurrency. | Speculation | Concurrency | Aggregate tok/s | Per-request decode tok/s (median) | TTFT s (median) | Accept length | Accept rate | vs. no spec | |---|---|---:|---:|---:|---:|---:|---:| | MTP (bundled BF16) | C1 | 82.1 | 91.1 | 0.12 | 2.858 | 0.619 | 1.82× | | MTP (bundled BF16) | C4 | 293.3 | 84.0 | 0.15 | 2.844 | 0.615 | 1.96× | | MTP (bundled BF16) | C8 | 380.3 | 71.1 | 0.16 | 2.813 | 0.604 | 1.34× | | MTP (bundled BF16) | C16 | 612.7 | 60.1 | 0.17 | 2.747 | 0.582 | 1.45× | | DFlash2 | C1 | 146.7 | 180.3 | 0.12 | 4.289 | 0.470 | 3.25× | | DFlash2 | C4 | 384.9 | 124.4 | 0.16 | 3.982 | 0.425 | 2.57× | | DFlash2 | C8 | 646.3 | 113.9 | 0.15 | 4.131 | 0.447 | 2.28× | | DFlash2 | C16 | 1,057.5 | 93.6 | 0.24 | 4.017 | 0.431 | 2.50× | | No speculation | C1 | 45.1 | 45.6 | 0.11 | n/a | n/a | 1.00× | | No speculation | C4 | 149.5 | 42.2 | 0.13 | n/a | n/a | 1.00× | | No speculation | C8 | 283.0 | 41.7 | 0.12 | n/a | n/a | 1.00× | | No speculation | C16 | 422.8 | 38.4 | 0.13 | n/a | n/a | 1.00× | | Speculation | Fastest per request | Recommended (highest aggregate with per-request speed ≥ 60% of C1) | Highest aggregate | |---|---|---|---| | MTP | C1 (91.1 tok/s) | C16 (612.7 tok/s, 60.1 per request) | C16 | | DFlash2 | C1 (180.3 tok/s) | C8 (646.3 tok/s, 113.9 per request) | C16 (1,057.5 tok/s) | | No speculation | C1 (45.6 tok/s) | C16 (422.8 tok/s) | C16 |

NVFP4 tiers

Same environment as above: one RTX PRO 6000 Blackwell 96GB; SGLang, 32K context (32,768), 4,096-token generation cap, thinking enabled, model-default sampling. Data from evaluation/BENCH_NVFP4_W4A16.json, evaluation/BENCH_NVFP4_W4A4.json, and evaluation/BENCH_NVFP4_MIXED.json (the W4A4-W8A8 tier).

• Which speculation: on W4A4 and W4A4-W8A8, DFlash2 beats MTP on both aggregate and per-request speed at every measured concurrency (C1–C16). On W4A16, MTP has the higher aggregate only at C4 (351.4 vs. 315.2); DFlash2 is faster at every other level and on per-request speed throughout.
• MTP only: on W4A16, use C1 for the fastest single request (128.9 tok/s), C4 for interactive use with a few users (351.4 aggregate, 108.0 per request), C8 for shared serving (450.7 aggregate, 91.3 per request), and C16 when only total throughput matters (817.3 aggregate, 70.6 per request). On W4A4 the same levels give C1 124.5 tok/s, C4 323.2 / 103.9, C8 355.5 / 86.6, and C16 912.4 / 73.0; on W4A4-W8A8, C1 116.3 tok/s, C4 213.0 / 99.8, C8 417.2 / 83.2, and C16 580.2 / 69.4 (aggregate / per request).
• Accept length: MTP 2.608–2.967 (W4A16), 2.597–2.923 (W4A4), and 2.647–2.865 (W4A4-W8A8); DFlash2 3.351–4.143 (W4A16), 3.723–4.327 (W4A4), and 3.828–4.364 (W4A4-W8A8).
• Concurrency: above C16 was not tested.

NVFP4 W4A16 full measured table

Requests per level: 4 at C1, 12 at C4, 24 at C8, 48 at C16. "vs. no spec" is the ratio of aggregate throughput at the same concurrency. | Speculation | Concurrency | Aggregate tok/s | Per-request decode tok/s (median) | TTFT s (median) | Accept length | Accept rate | vs. no spec | |---|---|---:|---:|---:|---:|---:|---:| | MTP (bundled BF16) | C1 | 101.1 | 128.9 | 0.11 | 2.608 | 0.537 | 1.44× | | MTP (bundled BF16) | C4 | 351.4 | 108.0 | 0.13 | 2.796 | 0.599 | 2.34× | | MTP (bundled BF16) | C8 | 450.7 | 91.3 | 0.13 | 2.743 | 0.581 | 1.50× | | MTP (bundled BF16) | C16 | 817.3 | 70.6 | 0.15 | 2.967 | 0.656 | 1.58× | | DFlash2 | C1 | 147.4 | 218.4 | 0.10 | 3.469 | 0.353 | 2.11× | | DFlash2 | C4 | 315.2 | 164.9 | 0.13 | 3.351 | 0.337 | 2.10× | | DFlash2 | C8 | 472.4 | 131.7 | 0.14 | 3.454 | 0.351 | 1.57× | | DFlash2 | C16 | 1,012.3 | 96.6 | 0.23 | 4.143 | 0.449 | 1.96× | | No speculation | C1 | 70.0 | 71.6 | 0.09 | n/a | n/a | 1.00× | | No speculation | C4 | 150.3 | 62.1 | 0.10 | n/a | n/a | 1.00× | | No speculation | C8 | 300.9 | 62.1 | 0.10 | n/a | n/a | 1.00× | | No speculation | C16 | 515.8 | 54.7 | 0.10 | n/a | n/a | 1.00× | | Speculation | Fastest per request | Recommended (highest aggregate with per-request speed ≥ 60% of C1) | Highest aggregate | |---|---|---|---| | MTP | C1 (128.9 tok/s) | C8 (450.7 tok/s, 91.3 per request) | C16 (817.3 tok/s) | | DFlash2 | C1 (218.4 tok/s) | C8 (472.4 tok/s, 131.7 per request) | C16 (1,012.3 tok/s) | | No speculation | C1 (71.6 tok/s) | C16 (515.8 tok/s) | C16 (515.8 tok/s) |

NVFP4 W4A4 full measured table

Requests per level: 4 at C1, 12 at C4, 24 at C8, 48 at C16. "vs. no spec" is the ratio of aggregate throughput at the same concurrency. | Speculation | Concurrency | Aggregate tok/s | Per-request decode tok/s (median) | TTFT s (median) | Accept length | Accept rate | vs. no spec | |---|---|---:|---:|---:|---:|---:|---:| | MTP (bundled BF16) | C1 | 104.7 | 124.5 | 0.15 | 2.685 | 0.562 | 1.48× | | MTP (bundled BF16) | C4 | 323.2 | 103.9 | 0.18 | 2.597 | 0.531 | 2.04× | | MTP (bundled BF16) | C8 | 355.5 | 86.6 | 0.17 | 2.683 | 0.561 | 1.12× | | MTP (bundled BF16) | C16 | 912.4 | 73.0 | 0.18 | 2.923 | 0.641 | 1.86× | | DFlash2 | C1 | 188.6 | 211.5 | 0.14 | 4.327 | 0.476 | 2.66× | | DFlash2 | C4 | 456.1 | 182.1 | 0.16 | 4.011 | 0.430 | 2.88× | | DFlash2 | C8 | 689.8 | 132.4 | 0.17 | 3.723 | 0.389 | 2.18× | | DFlash2 | C16 | 1,143.2 | 121.1 | 0.28 | 3.985 | 0.427 | 2.33× | | No speculation | C1 | 70.9 | 72.0 | 0.13 | n/a | n/a | 1.00× | | No speculation | C4 | 158.3 | 62.8 | 0.15 | n/a | n/a | 1.00× | | No speculation | C8 | 316.8 | 62.5 | 0.14 | n/a | n/a | 1.00× | | No speculation | C16 | 490.5 | 55.2 | 0.14 | n/a | n/a | 1.00× | | Speculation | Fastest per request | Recommended (highest aggregate with per-request speed ≥ 60% of C1) | Highest aggregate | |---|---|---|---| | MTP | C1 (124.5 tok/s) | C8 (355.5 tok/s, 86.6 per request) | C16 (912.4 tok/s) | | DFlash2 | C1 (211.5 tok/s) | C8 (689.8 tok/s, 132.4 per request) | C16 (1,143.2 tok/s) | | No speculation | C1 (72.0 tok/s) | C16 (490.5 tok/s) | C16 (490.5 tok/s) |

NVFP4 W4A4-W8A8 full measured table

Requests per level: 4 at C1, 12 at C4, 24 at C8, 48 at C16. "vs. no spec" is the ratio of aggregate throughput at the same concurrency. | Speculation | Concurrency | Aggregate tok/s | Per-request decode tok/s (median) | TTFT s (median) | Accept length | Accept rate | vs. no spec | |---|---|---:|---:|---:|---:|---:|---:| | MTP (bundled BF16) | C1 | 99.9 | 116.3 | 0.17 | 2.865 | 0.621 | 1.61× | | MTP (bundled BF16) | C4 | 213.0 | 99.8 | 0.19 | 2.667 | 0.555 | 1.09× | | MTP (bundled BF16) | C8 | 417.2 | 83.2 | 0.18 | 2.647 | 0.549 | 1.33× | | MTP (bundled BF16) | C16 | 580.2 | 69.4 | 0.19 | 2.772 | 0.591 | 1.24× | | DFlash2 | C1 | 175.1 | 216.8 | 0.16 | 4.201 | 0.458 | 2.82× | | DFlash2 | C4 | 395.0 | 152.4 | 0.19 | 3.995 | 0.427 | 2.02× | | DFlash2 | C8 | 627.9 | 132.4 | 0.19 | 3.828 | 0.404 | 2.00× | | DFlash2 | C16 | 1,146.4 | 105.5 | 0.30 | 4.364 | 0.479 | 2.44× | | No speculation | C1 | 62.2 | 63.0 | 0.15 | n/a | n/a | 1.00× | | No speculation | C4 | 195.7 | 56.4 | 0.17 | n/a | n/a | 1.00× | | No speculation | C8 | 314.0 | 53.4 | 0.16 | n/a | n/a | 1.00× | | No speculation | C16 | 469.4 | 47.7 | 0.16 | n/a | n/a | 1.00× | | Speculation | Fastest per request | Recommended (highest aggregate with per-request speed ≥ 60% of C1) | Highest aggregate | |---|---|---|---| | MTP | C1 (116.3 tok/s) | C8 (417.2 tok/s, 83.2 per request) | C16 (580.2 tok/s) | | DFlash2 | C1 (216.8 tok/s) | C8 (627.9 tok/s, 132.4 per request) | C16 (1,146.4 tok/s) | | No speculation | C1 (63.0 tok/s) | C16 (469.4 tok/s) | C16 (469.4 tok/s) |

NInfer tiers

Setup: one RTX PRO 6000 Blackwell; official NInfer with the same context flags as the NInfer launch commands below (--max-context 102400 --kv-capacity 819200 --max-concurrency 8 --kv-dtype fp8); 4,096 generation cap, thinking on (template default xhigh), sampling temperature 1.0, top_p 0.95, top_k 20, same prompts as the NVFP4 benchmark above; 4 requests at C1, 12 at C4, 24 at C8. MTP uses 4 draft tokens and DFlash2 uses 8, both with --lm-head-draft. Numbers are aggregate decode tok/s. NInfer supports at most 8 concurrent requests, so C16 was not tested.

• Best setting: for both tiers and all three modes, aggregate throughput peaks at C8. NInfer W4A4 tops out with DFlash2 · C8 (975.9 tok/s) and NInfer W4A4-W8A8 with DFlash2 · C8 (804.9 tok/s); with MTP only, C8 gives 678.3 and 663.9 tok/s respectively.
• C8 versus NVFP4 W4A4 (SGLang): NInfer W4A4 reaches 306.8 vs 316.8 with no speculation, 678.3 vs 355.5 with MTP, and 975.9 vs 689.8 tok/s with DFlash2. The engines and context settings differ, so treat this as a rough comparison.

INT8 W8A8

Setup: one RTX PRO 6000 Blackwell; vLLM 0.28 with --max-model-len 102400 and --max-num-seqs 8; 512 generation cap, thinking on (xhigh), sampling temperature 1.0, top_p 0.95, top_k 20; each level sends only as many requests as its concurrency (1 at C1, 2 at C2, 4 at C4, 8 at C8). This is a quick benchmark with few requests and short outputs, so the numbers are not directly comparable with the tiers above. Numbers are aggregate tok/s (all output tokens ÷ wall-clock time). Both rows were measured on a separate, unreleased comparison build with MTP (the same INT8 text weights plus one BF16 MTP layer); with no speculation the MTP layer is not used. This package does not include MTP; a separate MTP draft folder is now provided (see below).

• Recommendation (this table): no speculation at C8 (177.3 tok/s); aggregate throughput rises with concurrency, and nothing above C8 was tested.
• The MTP row above came from an earlier, unreleased comparison build and is kept for reference only; it does not describe the MTP-BF16/ draft folder, which is measured separately below. This tier has no DFlash2 draft.

With the MTP-BF16/ draft folder

Setup: one NVIDIA RTX PRO 6000 Blackwell (96 GB); vLLM 0.28 with --max-model-len 102400, --max-num-seqs 8, and --gpu-memory-utilization 0.85; MTP row uses --speculative-config '{"method":"mtp","model":"./INT8/W8A8/MTP-BF16","num_speculative_tokens":3}'. 1024 generation cap, thinking on, sampling temperature 1.0, top_p 0.95, top_k 20; each level sends 2 × concurrency requests over the same 8 prompts. Numbers are aggregate tok/s (all output tokens ÷ wall-clock time); the last column is the median per-request decode speed at C1. Different hardware and settings from the table above, so compare rows only within this table.

• Accept length (mean tokens per step, including the verified token): 2.74–2.90 across C1–C8.
• With this draft, 3 draft tokens is faster at every level on this machine: 65.9 tok/s at C1 against 31.1 with no speculation, and 389.8 tok/s at C8 against 127.0.
• The INT8 scores above were measured without speculation; the draft folder does not change any INT8 weight file.

INT4 W4A16

Setup: one NVIDIA RTX PRO 6000 Blackwell (96 GB); vLLM 0.28 with --max-model-len 102400, --max-num-seqs 8, and --gpu-memory-utilization 0.85; MTP row uses the built-in head with --speculative-config '{"method":"mtp","num_speculative_tokens":3}'. 1024 generation cap, thinking on, sampling temperature 1.0, top_p 0.95, top_k 20; each level sends 2 × concurrency requests over the same 8 prompts (same script as the INT8 MTP-BF16/ table). Numbers are aggregate tok/s (all output tokens ÷ wall-clock time); the last column is the median per-request decode speed at C1.

• Accept length (mean tokens per step, including the verified token): 2.77–2.92 across C1–C8 in this table; 2.81 averaged over the full eval run.
• The INT4 scores above were measured with MTP (3 draft tokens).

GGUF Q6_K

Setup: one RTX PRO 6000 Blackwell; 1-minute quick benchmark, 512 generation cap, thinking on (xhigh), sampling temperature 1.0, top_p 0.95, top_k 20; each level sends only as many requests as its concurrency (1 at C1, 2 at C2, 4 at C4, 8 at C8). Numbers are aggregate tok/s (all output tokens ÷ wall-clock time). This is a quick benchmark with few requests and short outputs, so the numbers are not directly comparable with the full SGLang / NInfer benchmarks above. NInfer rows were measured on the GGUF-NInfer/ package with the server at --max-context 32768 --kv-capacity 32768 --kv-dtype rk8v4 --max-concurrency 8; llama.cpp rows were measured on the GGUF/ package.

• Recommendation: NInfer + C4 + MTP (235.5 tok/s), matching the recommended launch command below (--max-concurrency 4). For aggregate throughput alone, C8 is higher (388.1 tok/s); raise --max-concurrency to 8 for that.
• MTP is built in: both packages carry a built-in MTP head (GGUF: Q8_0; NInfer: Q6_K attention/MLP weights), so no external draft model is needed. For a single request, NInfer with MTP reaches 131.9 tok/s, about 2.5× llama.cpp without speculation (53.4 tok/s); llama.cpp with MTP reaches 97.2 tok/s aggregate at C1 (119.0 tok/s single-request decode).

GGUF Q8_0

Hardware: single RTX PRO 6000 Blackwell. llama.cpp rows use the quick protocol (thinking on, fused MTP): C1=4 requests, C2=6, C4=12; aggregate decode tok/s. NInfer rows use the same-caliber bench (thinking on, template default xhigh, max_tokens 4096, temperature 1.0, top_p 0.95, top_k 20; C1=4, C4=12, C8=24; MTP draft-tokens 4).

• Recommended: NInfer + C8 + MTP (486.3 tok/s). Fastest llama.cpp point is C4 MTP (190.8 tok/s).
• MTP built in: both packages carry an MTP head; no external draft model.

Recommended launch commands

Quantization scheme

• BF16 (BF16/): all weights in BF16 with the original multimodal config (mrope retained, mtp_num_hidden_layers=1) and no quantization_config.
• Static FP8 (FP8/): official-style static FP8, E4M3 weights in 128×128 blocks with one scale per block (weight_scale_inv); activations use dynamic FP8. The config only adds a top-level FP8 quantization_config.
• NVFP4 W4A16 (NVFP4/W4A16/): ModelOpt local-Hessian calibration. NVFP4 is E2M1 in 16-element blocks with one E4M3 scale per block plus a per-tensor FP32 second-level scale. MLP gate/up/down, full-attention q/k/v/o, and linear-attention in_proj_qkv / in_proj_z / out_proj (400 Linear layers) use NVFP4 weights with BF16 activations; linear-attention in_proj_a / in_proj_b and lm_head (97 layers) stay in BF16. The quantization metadata uses ModelOpt's MIXED_PRECISION format (each of the 400 layers tagged W4A16_NVFP4), which SGLang detects as modelopt_mixed. 1,999 tensors: 1,651 language, 333 vision, 15 MTP.
• NVFP4 W4A4 (NVFP4/W4A4/): same calibration data and algorithm and the same 400 Linear layers, with both weights and activations in NVFP4 (activations are quantized at inference in 16-element blocks). Standard NVFP4 export, detected by SGLang as modelopt_fp4. 2,399 tensors: 2,051 language, 333 vision, 15 MTP.
• NVFP4 W4A4-W8A8 (NVFP4/W4A4-W8A8/): mixed precision with the same calibration data and algorithm. MLP gate/up/down (192 Linear layers) use NVFP4 W4A4 (both weights and activations in NVFP4; activations are quantized at inference in 16-element blocks); full-attention q/k/v/o and linear-attention in_proj_qkv / in_proj_z / out_proj (208 Linear layers) use FP8 W8A8 (E4M3 weights and activations with static per-tensor scales; activation scales come from calibration); linear-attention in_proj_a / in_proj_b and lm_head (97 layers) stay in BF16. The quantization metadata uses ModelOpt's MIXED_PRECISION format (each layer tagged NVFP4 or FP8), which SGLang detects as modelopt_mixed. 2,191 tensors: 1,843 language, 333 vision, 15 MTP.
• NVFP4 calibration data: complete trajectories (prompt + reasoning + final answer) that were correct and non-empty, taken from this model's own RLOO training data: 512 samples, about 1.735M tokens, at most 4,096 tokens each. No official GPQA, MMLU, or LCB problems are used. See evaluation/NVFP4_CALIBRATION_METHOD.md. In all three tiers the vision tower (333 tensors) and MTP (15 tensors) are BF16.
• Architecture: Qwen3_5ForConditionalGeneration, 64 layers (linear attention and full attention alternating 3:1). The FP8 package has 1,599 tensors: 1,251 language, 333 vision, 15 MTP.

All SSM control branches stay in BF16; the exclusion list matches Qwen's official FP8 release, keeping 146 language-model modules in BF16. FP8 package smoke tests passed: patch-free load in SGLang, greedy output identical to the text-only evaluation build, SGLang image request, SGLang MTP, and vLLM image request. vLLM 0.28 + MTP: temporarily unavailable for FP8; use SGLang for now.

NVFP4 smoke tests: patch-free load in SGLang, greedy parity, and the image request passed. MTP and DFlash2 speed and accept length come from the separate benchmark runs (all C1–C16 levels completed; see above).

GGUF Q8_0 (GGUF/, GGUF-NInfer/): Q8_0 backbone with built-in MTP head (blk.64.nextn), no external draft; 866 tensors; .ninfer converted from the same GGUF; the GGUF has no vision weights (llama.cpp uses GGUF/mmproj-Qwen3.8-27B-Q8_0.gguf via --mmproj); the .ninfer carries the vision component inside the package (add --vision; see NInfer · vision below).

GGUF Q6_K (GGUF/, GGUF-NInfer/): Q6_K backbone calibrated with a 512-chunk imatrix, plus structure protection: the linear-attention (SSM) ssm_alpha / ssm_beta stay BF16, the token embedding and output head are Q8_0, and the MTP head (blk.64 attn q/k/v/output, ffn gate/up/down, and nextn.eh_proj) is entirely Q8_0 with its norms kept in F32. The GGUF has 866 tensors in 65 blocks (64 backbone layers + 1 MTP layer). The .ninfer is converted from the same GGUF with the same backbone quantization; its MTP attention/MLP weights are Q6_K (gguf_q6_k) and only mtp/input_projection is Q8_0; the GGUF has no vision weights (llama.cpp uses GGUF/mmproj-Qwen3.8-27B-Q8_0.gguf via --mmproj); the .ninfer carries the vision component inside the package (add --vision; see NInfer · vision below).

Commands

Requires official SGLang ≥ 0.5.19; vLLM ≥ 0.28 (MTP only). vLLM MTP on NVFP4 with tensor parallel ≥ 2 has a known issue (vLLM #52480); use a single GPU there. vLLM 0.28 + MTP: temporarily unavailable for FP8; use SGLang for now. INT8 W8A8 and INT4 W4A16 load only in vLLM (tested on 0.28); their commands are below, after the NInfer commands.

MTP and DFlash2 are mutually exclusive; enable only one per launch. The examples use ./FP8; for the BF16 tier, replace the --model-path path with ./BF16, and for the NVFP4 tiers replace --model-path with ./NVFP4/W4A16, ./NVFP4/W4A4, or ./NVFP4/W4A4-W8A8. The DFlash2 draft is the DFlash2-FP8 folder inside each tier (e.g. ./FP8/DFlash2-FP8, ./BF16/DFlash2-FP8, ./NVFP4/W4A16/DFlash2-FP8). SGLang detects the NVFP4 quantization format automatically, so no --quantization flag is needed; among these SGLang-format tiers, vLLM was validated only on the FP8 package (without MTP), and NVFP4 was tested only on SGLang; INT8 W8A8 and INT4 W4A16 are vLLM-only and were evaluated on vLLM 0.28. Concurrency and context settings match the benchmark runs; adjust as needed.

SGLang · MTP (bundled BF16)

python -m sglang.launch_server \
  --model-path ./FP8 \
  --context-length 32768 \
  --max-running-requests 16 \
  --mamba-ssm-dtype bfloat16 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

vLLM · MTP

vLLM 0.28 + MTP: temporarily unavailable for FP8; use SGLang for now (see the SGLang MTP command above).

SGLang · DFlash2

python -m sglang.launch_server \
  --model-path ./FP8 \
  --context-length 32768 \
  --max-running-requests 16 \
  --mamba-ssm-dtype bfloat16 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path ./FP8/DFlash2-FP8 \
  --speculative-draft-model-quantization fp8 \
  --speculative-num-draft-tokens 8

SGLang · NVFP4 W4A16 · DFlash2 (for W4A4, replace both W4A16 with W4A4; for MTP on NVFP4, use the MTP command above and change only --model-path)

python -m sglang.launch_server \
  --model-path ./NVFP4/W4A16 \
  --context-length 32768 \
  --max-running-requests 16 \
  --mamba-ssm-dtype bfloat16 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path ./NVFP4/W4A16/DFlash2-FP8 \
  --speculative-draft-model-quantization fp8 \
  --speculative-num-draft-tokens 8

SGLang · NVFP4 W4A4-W8A8 · DFlash2 (for MTP, use the MTP command above and change only --model-path)

python -m sglang.launch_server \
  --model-path ./NVFP4/W4A4-W8A8 \
  --context-length 32768 \
  --max-running-requests 16 \
  --mamba-ssm-dtype bfloat16 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path ./NVFP4/W4A4-W8A8/DFlash2-FP8 \
  --speculative-draft-model-quantization fp8 \
  --speculative-num-draft-tokens 8

The BF16 / FP8 / NVFP4 scores in this card were measured on SGLang with DFlash2 enabled (NInfer and INT8 W8A8 without speculation, INT4 W4A16 with MTP, GGUF on llama.cpp; see each section); speculative decoding does not change the target model's output distribution. Default sampling is in generation_config.json (temperature 1.0, top_p 0.95, top_k 20).

Images on SGLang / vLLM: BF16/, FP8/, NVFP4/W4A16/, NVFP4/W4A4/ and NVFP4/W4A4-W8A8/ are full multimodal checkpoints: config.json declares Qwen3_5ForConditionalGeneration with a vision_config, each folder has preprocessor_config.json and video_preprocessor_config.json, and the weight index maps all 333 vision-tower tensors (to vision-bf16.safetensors in that folder; for BF16, to model-00002-of-00002.safetensors). The SGLang and vLLM commands above accept images as they are, with no extra flag; send OpenAI-style image_url parts to POST /v1/chat/completions (request example under NInfer · vision below; set model to the served model name). INT8/W8A8/ accepts images too since 2026-10-09 (Qwen3_5ForConditionalGeneration; BF16 vision tower in model_visual.safetensors, preprocessor configs included): use the INT8 vLLM commands below as they are. INT4/W4A16/ accepts images too (BF16 vision tower in model_visual.safetensors): use the INT4 vLLM command below.

NVFP4 · NInfer (.ninfer, official NInfer)

W4A4 is shown; for W4A4-W8A8, change the path to ./NVFP4-NInfer/W4A4-W8A8/Qwen3.8-27B-Coder390-EfficientThink-W4A4-W8A8-MTP-DFlash2.ninfer and --model-id to qwen3.8-27b-coder390-w4a4-w8a8.

The MTP-DFlash2 packages already contain the Q8 MTP head and the DFlash2 draft, so NInfer needs no external draft model (unlike the SGLang commands above, which take a DFlash2 draft path); pass either --spec mtp or --spec dflash2, or omit --spec to run without speculation.

Images: the two V3 MTP-DFlash2 .ninfer packages in NVFP4-NInfer/ (W4A4 and W4A4-W8A8) contain the vision component since 2026-10-09 (295,720,448 bytes, quantized from the BF16 vision weights: patch embedding Q6, attention q/k/v and MLP fc1 Q4, other projections Q5, merger Q8, norms and biases BF16). Text, MTP and DFlash2 weights are byte-for-byte unchanged. Add --vision to any NInfer command below and send images as OpenAI-style image_url parts to POST /v1/chat/completions. Without --vision the packages run text-only, as before. The V2 packages (-NInferV2-NOT-FOR-V3) are unchanged and still have no vision. GPU test on 2026-10-09 (one RTX PRO 6000, NInfer-all 64492cb with --vision, image prompt): both MTP-DFlash2 packages answered correctly. W4A4-MTP-DFlash2: MTP (3 draft tokens) accept length 2.89, decode 153 tok/s; DFlash2 (7 draft tokens) accept length 3.53, decode 185 tok/s. W4A4-W8A8-MTP-DFlash2: MTP 3.19, 164 tok/s; DFlash2 3.74, 195 tok/s.

NInfer · no speculation

ninfer-serve ./NVFP4-NInfer/W4A4/Qwen3.8-27B-Coder390-EfficientThink-W4A4-MTP-DFlash2.ninfer \
  --host 127.0.0.1 --port 19931 --device 0 \
  --model-id qwen3.8-27b-coder390-w4a4 \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype fp8

NInfer · MTP (4 draft tokens)

ninfer-serve ./NVFP4-NInfer/W4A4/Qwen3.8-27B-Coder390-EfficientThink-W4A4-MTP-DFlash2.ninfer \
  --host 127.0.0.1 --port 19931 --device 0 \
  --model-id qwen3.8-27b-coder390-w4a4 \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype fp8 \
  --spec mtp --draft-tokens 4 --lm-head-draft

NInfer · DFlash2 (8 draft tokens)

ninfer-serve ./NVFP4-NInfer/W4A4/Qwen3.8-27B-Coder390-EfficientThink-W4A4-MTP-DFlash2.ninfer \
  --host 127.0.0.1 --port 19931 --device 0 \
  --model-id qwen3.8-27b-coder390-w4a4 \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype fp8 \
  --spec dflash2 --draft-tokens 8 --lm-head-draft

NInfer · vision (image input), one command per package

Each command is the matching NInfer command above plus --vision. Nothing else is needed: the vision component is inside every V3 .ninfer, so the vision-bf16.safetensors next to the packages is not used at run time.

# W4A4 · MTP-DFlash2 package · no speculation
ninfer-serve ./NVFP4-NInfer/W4A4/Qwen3.8-27B-Coder390-EfficientThink-W4A4-MTP-DFlash2.ninfer \
  --host 127.0.0.1 --port 19931 --device 0 \
  --model-id qwen3.8-27b-coder390-w4a4 \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype fp8 \
  --vision

# W4A4 · MTP-DFlash2 package · MTP
ninfer-serve ./NVFP4-NInfer/W4A4/Qwen3.8-27B-Coder390-EfficientThink-W4A4-MTP-DFlash2.ninfer \
  --host 127.0.0.1 --port 19931 --device 0 \
  --model-id qwen3.8-27b-coder390-w4a4 \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype fp8 \
  --spec mtp --draft-tokens 4 --lm-head-draft \
  --vision

# W4A4 · MTP-DFlash2 package · DFlash2
ninfer-serve ./NVFP4-NInfer/W4A4/Qwen3.8-27B-Coder390-EfficientThink-W4A4-MTP-DFlash2.ninfer \
  --host 127.0.0.1 --port 19931 --device 0 \
  --model-id qwen3.8-27b-coder390-w4a4 \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype fp8 \
  --spec dflash2 --draft-tokens 8 --lm-head-draft \
  --vision

# W4A4-W8A8 · MTP-DFlash2 package · no speculation
ninfer-serve ./NVFP4-NInfer/W4A4-W8A8/Qwen3.8-27B-Coder390-EfficientThink-W4A4-W8A8-MTP-DFlash2.ninfer \
  --host 127.0.0.1 --port 19931 --device 0 \
  --model-id qwen3.8-27b-coder390-w4a4-w8a8 \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype fp8 \
  --vision

# W4A4-W8A8 · MTP-DFlash2 package · MTP
ninfer-serve ./NVFP4-NInfer/W4A4-W8A8/Qwen3.8-27B-Coder390-EfficientThink-W4A4-W8A8-MTP-DFlash2.ninfer \
  --host 127.0.0.1 --port 19931 --device 0 \
  --model-id qwen3.8-27b-coder390-w4a4-w8a8 \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype fp8 \
  --spec mtp --draft-tokens 4 --lm-head-draft \
  --vision

# W4A4-W8A8 · MTP-DFlash2 package · DFlash2
ninfer-serve ./NVFP4-NInfer/W4A4-W8A8/Qwen3.8-27B-Coder390-EfficientThink-W4A4-W8A8-MTP-DFlash2.ninfer \
  --host 127.0.0.1 --port 19931 --device 0 \
  --model-id qwen3.8-27b-coder390-w4a4-w8a8 \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype fp8 \
  --spec dflash2 --draft-tokens 8 --lm-head-draft \
  --vision

Image request: OpenAI-style image_url part in POST /v1/chat/completions; url takes an http(s):// image link; a data URI of a local image also works, and model must match --model-id.

curl http://127.0.0.1:19931/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-27b-coder390-w4a4",
    "messages": [{"role": "user", "content": [
      {"type": "image_url", "image_url": {"url": "https://example.com/demo.png"}},
      {"type": "text", "text": "Describe this image."}
    ]}],
    "max_tokens": 2048
  }'

Health check: curl -sf http://127.0.0.1:19931/health. The chat endpoint is POST /v1/chat/completions; the request model must match --model-id.

vLLM · INT8 W8A8 (no speculation)

VLLM_DISABLED_KERNELS=CutlassInt8ScaledMMLinearKernel \
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve ./INT8/W8A8 \
  --trust-remote-code --dtype bfloat16 \
  --max-model-len 102400 --served-model-name int8 \
  --gpu-memory-utilization 0.85 --max-num-seqs 8 \
  --reasoning-parser qwen3

On RTX PRO 6000 Blackwell, vLLM's Cutlass INT8 kernel is unavailable and the server will not start unless it is disabled with VLLM_DISABLED_KERNELS=CutlassInt8ScaledMMLinearKernel; vLLM then uses its Triton INT8 kernel (the startup log shows TritonInt8ScaledMMLinearKernel). VLLM_USE_FLASHINFER_SAMPLER=0 matches the tested setup. Tested on vLLM 0.28; SGLang cannot load this package (ModelOptFp8Config raises an error). --reasoning-parser qwen3 returns the reasoning separately in reasoning_content (it was off during scoring).

vLLM · INT8 W8A8 · MTP (external draft INT8/W8A8/MTP-BF16/)

VLLM_DISABLED_KERNELS=CutlassInt8ScaledMMLinearKernel \
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve ./INT8/W8A8 \
  --trust-remote-code --dtype bfloat16 \
  --max-model-len 102400 --served-model-name int8 \
  --gpu-memory-utilization 0.85 --max-num-seqs 8 \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","model":"./INT8/W8A8/MTP-BF16","num_speculative_tokens":3}'

On an RTX PRO 6000 Blackwell, set VLLM_DISABLED_KERNELS=CutlassInt8ScaledMMLinearKernel and VLLM_USE_FLASHINFER_SAMPLER=0, as in the command above. Both rows of the new table used those two settings.

vLLM · INT4 W4A16 · MTP (built-in head) · text and images

VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve ./INT4/W4A16 \
  --trust-remote-code --dtype bfloat16 \
  --max-model-len 102400 --served-model-name int4 \
  --gpu-memory-utilization 0.85 --max-num-seqs 8 \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Images go as OpenAI-style image_url parts with the same server; no extra flag is needed:

curl -s http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "int4",
  "messages": [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
    {"type": "text", "text": "Describe this image."}
  ]}],
  "max_tokens": 2048
}'

Remove --speculative-config to run without speculation. Tested on vLLM 0.28 (VLLM_USE_FLASHINFER_SAMPLER=0 matches the tested setup); SGLang and NInfer were not tested with this package. --reasoning-parser qwen3 returns the reasoning separately.

GGUF / GGUF-NInfer

Load the .ninfer with NInfer-all (the master branch of iamwavecut/ninfer-all): the package keeps the GGUF quantization blocks, which other NInfer builds cannot read. llama.cpp needs an upstream build that supports --spec-type draft-mtp. NInfer's --kv-capacity is one KV pool shared by all requests; llama.cpp's -c 409600 -np 4 splits the context evenly across 4 slots, so each request gets at most 102,400 tokens, enough for a single request to reach 94K. The chat endpoint is POST /v1/chat/completions; for NInfer the request model must match --model-id. Vision: GGUF/mmproj-Qwen3.8-27B-Q8_0.gguf for llama.cpp --mmproj (629,247,008 bytes, sha256 cae9799dc9196449b0d83f716e64af89d5cf65147510462e5a629a2aa23adb32). Each .ninfer package carries the vision component; add --vision to ninfer-serve to accept images (see NInfer · vision below).

Community fork ninfer-fusion-kvmem: the Q2 / Q3 / Q4 LynnStyle NInfer packages (GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q4MTP.ninfer, GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-Q4MTP.ninfer, GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q4MTP.ninfer) carry a built-in MTP head re-encoded as Q4_K and load and run MTP on both the official NInfer-all master and the community fork ninfer-fusion-kvmem (tested 2026-10-11). The .gguf files in GGUF/ keep a Q4_0 MTP head, which llama.cpp supports. The Q8 MTP versions remain available as an alternative (same weights, only the MTP head re-encoded as Q8_0): GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q8MTP.gguf, GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-Q8MTP.gguf and GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.gguf for llama.cpp, and GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q8MTP.ninfer, GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-Q8MTP.ninfer and GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.ninfer for NInfer (model ids qwen3.8-27b-coder390-Q2LynnStyle-Q8Mtp, qwen3.8-27b-coder390-Q3LynnStyle-Q8Mtp, qwen3.8-27b-coder390-Q4LynnStyle-Q8Mtp).

MTP and DFlash2: every package under GGUF/ and GGUF-NInfer/ has a built-in MTP head (Q2 / Q3 / Q4 LynnStyle use built-in Q4 MTP: Q4_0 in the .gguf, Q4_K in the -Q4MTP.ninfer; their -Q8MTP versions use a Q8_0 MTP head). To enable it, add --spec mtp --draft-tokens N for NInfer or --spec-type draft-mtp --spec-draft-n-max N for llama.cpp; leave these out to run without speculation. DFlash2 is an external draft model and is not in these GGUF / NInfer packages; this repository does not ship a DFlash2 draft for GGUF / GGUF-NInfer. For DFlash2, use the DFlash2-FP8/ draft that ships with the BF16 / FP8 / NVFP4 tiers in this repository (SGLang --speculative-algorithm DFLASH) or the .ninfer packages under NVFP4-NInfer/ (--spec dflash2). MTP and DFlash2 are mutually exclusive: a server runs one or the other, never both.

GGUF · NInfer

NInfer · GGUF Q2 LynnStyle · MTP (draft 4, recommended concurrency 8)

ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q4MTP.ninfer \
  --model-id qwen3.8-27b-coder390-Q2LynnStyle-Mtp \
  --max-context 102400 --kv-capacity 819200 --kv-dtype rk8v4 \
  --max-concurrency 8 --spec mtp --draft-tokens 4

vLLM · INT8 W8A16 (no speculation)

VLLM_DISABLED_KERNELS=CutlassInt8ScaledMMLinearKernel VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve ./INT8/W8A16 \
  --served-model-name qwen3.8-27b-coder390-int8-w8a16 \
  --trust-remote-code --max-model-len 102400 --max-num-seqs 8 \
  --gpu-memory-utilization 0.90 --reasoning-parser qwen3

vLLM · INT8 W8A16 · MTP (external draft INT8/W8A16/MTP-BF16/)

VLLM_DISABLED_KERNELS=CutlassInt8ScaledMMLinearKernel VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve ./INT8/W8A16 \
  --served-model-name qwen3.8-27b-coder390-int8-w8a16 \
  --trust-remote-code --max-model-len 102400 --max-num-seqs 8 \
  --gpu-memory-utilization 0.90 --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","model":"./INT8/W8A16/MTP-BF16","num_speculative_tokens":3}'

NInfer · GGUF Q3 LynnStyle · MTP (draft 4, recommended concurrency 8)

ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-Q4MTP.ninfer \
  --host 127.0.0.1 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-Q3LynnStyle-Mtp \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype rk8v4 \
  --spec mtp --draft-tokens 4

NInfer · GGUF Q4 LynnStyle · MTP (draft 4, recommended concurrency 8)

ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q4MTP.ninfer \
  --host 127.0.0.1 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-Q4LynnStyle-Mtp \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype rk8v4 \
  --spec mtp --draft-tokens 4

NInfer · GGUF Q4 LynnStyle · Q8MTP · MTP (draft 4, recommended concurrency 8): an alternative with a Q8_0 MTP head (the -Q4MTP packages also run on NInfer-all and ninfer-fusion-kvmem); for Q2 / Q3 replace Q4 with Q2 / Q3 in the file name and model id.

ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.ninfer \
  --host 127.0.0.1 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-Q4LynnStyle-Q8Mtp \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype rk8v4 \
  --spec mtp --draft-tokens 4

NInfer · GGUF Q6_K · MTP (draft 3, recommended concurrency 4)

ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q6_K-MTP.ninfer \
  --model-id qwen3.8-27b-coder390-q6k-mtp \
  --max-context 102400 --kv-capacity 409600 --kv-dtype rk8v4 \
  --max-concurrency 4 --spec mtp --draft-tokens 3

NInfer · GGUF Q8_0 · MTP (draft 4, recommended concurrency 8)

ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q8_0-MTP.ninfer \
  --model-id qwen3.8-27b-coder390-q8-mtp \
  --max-context 102400 --kv-capacity 819200 --kv-dtype rk8v4 \
  --max-concurrency 8 --spec mtp --draft-tokens 4

Do not use --max-context 131072 --kv-capacity 131072 (the KV pool then holds only one full context and queued requests return HTTP 503). Do not use --lm-head-q6 / --embedding-q4. Requires NInfer-all (ninfer-serve); official ninfer-src cannot read gguf_blocks_v1.

NInfer · vision (image input)

Every .ninfer in GGUF-NInfer/ contains the vision component (295,720,448 bytes, added on 2026-10-09). It was quantized from the BF16 vision weights in the NInfer vision formats: patch embedding Q6, attention q/k/v and MLP fc1 Q4, other projections Q5, merger Q8, norms and biases BF16. The text and MTP weights are byte-for-byte the same as before. Add --vision to any NInfer command above; nothing else is needed (the separate vision-bf16.safetensors is not used). These packages have a built-in MTP head but no DFlash2 draft, so each package has two commands: no speculation and MTP.

# Q2LynnStyle · no speculation
ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q4MTP.ninfer \
  --host 127.0.0.1 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-Q2LynnStyle-Mtp \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype rk8v4 \
  --vision

# Q2LynnStyle · MTP (4 draft tokens)
ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q4MTP.ninfer \
  --host 127.0.0.1 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-Q2LynnStyle-Mtp \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype rk8v4 \
  --spec mtp --draft-tokens 4 \
  --vision

# Q3LynnStyle · no speculation
ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-Q4MTP.ninfer \
  --host 127.0.0.1 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-Q3LynnStyle-Mtp \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype rk8v4 \
  --vision

# Q3LynnStyle · MTP (4 draft tokens)
ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-Q4MTP.ninfer \
  --host 127.0.0.1 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-Q3LynnStyle-Mtp \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype rk8v4 \
  --spec mtp --draft-tokens 4 \
  --vision

# Q4LynnStyle · no speculation
ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q4MTP.ninfer \
  --host 127.0.0.1 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-Q4LynnStyle-Mtp \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype rk8v4 \
  --vision

# Q4LynnStyle · MTP (4 draft tokens)
ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q4MTP.ninfer \
  --host 127.0.0.1 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-Q4LynnStyle-Mtp \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype rk8v4 \
  --spec mtp --draft-tokens 4 \
  --vision

# Q6_K · no speculation
ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q6_K-MTP.ninfer \
  --host 127.0.0.1 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-q6k-mtp \
  --max-context 102400 --kv-capacity 409600 --max-concurrency 4 \
  --kv-dtype rk8v4 \
  --vision

# Q6_K · MTP (3 draft tokens)
ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q6_K-MTP.ninfer \
  --host 127.0.0.1 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-q6k-mtp \
  --max-context 102400 --kv-capacity 409600 --max-concurrency 4 \
  --kv-dtype rk8v4 \
  --spec mtp --draft-tokens 3 \
  --vision

# Q8_0 · no speculation
ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q8_0-MTP.ninfer \
  --host 127.0.0.1 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-q8-mtp \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype rk8v4 \
  --vision

# Q8_0 · MTP (4 draft tokens)
ninfer-serve ./GGUF-NInfer/Qwen3.8-27B-Coder390-EfficientThink-Q8_0-MTP.ninfer \
  --host 127.0.0.1 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-q8-mtp \
  --max-context 102400 --kv-capacity 819200 --max-concurrency 8 \
  --kv-dtype rk8v4 \
  --spec mtp --draft-tokens 4 \
  --vision

Image request (OpenAI-style image_url part; url takes an http(s):// image link; a data URI of a local image also works; model must match --model-id):

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-27b-coder390-Q4LynnStyle-Mtp",
    "messages": [{"role": "user", "content": [
      {"type": "image_url", "image_url": {"url": "https://example.com/demo.png"}},
      {"type": "text", "text": "Describe this image."}
    ]}],
    "max_tokens": 2048
  }'

Send images as OpenAI-style image_url parts to POST /v1/chat/completions. --vision keeps the vision encoder on the GPU; if VRAM is tight, use --vision-cpu instead (it runs on the CPU, and each image is capped at 256 merged tokens unless you set --vision-max-merged). Without --vision the package runs text-only, as before. GGUF-NInfer/vision-bf16.safetensors is the BF16 source of the vision part and is not needed to run NInfer. Engine test (2026-10-09, one RTX PRO 6000): the Q2 LynnStyle package with --vision answered an image prompt correctly; with MTP at 3 draft tokens the accept length was 3.00 and decode speed 207 tok/s. On NInfer-all master, the Q3 / Q4 LynnStyle, Q6_K, and Q8_0 packages were also tested with MTP and --vision and all answered correctly (short single runs: Q4 151.9 tok/s, accept rate 66.7%; Q6_K 131.1 tok/s, 65.4%; Q8_0 121.5 tok/s, 68.2%); for the Q8 MTP versions see "Q8 MTP versions · check" above.

GGUF · llama.cpp

llama.cpp · GGUF Q2 LynnStyle · MTP (representative; swap the -m path for Q3 / Q4 / Q6_K / Q8_0)

llama-server -m ./GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-MTP.gguf \
  -c 409600 -np 4 -ngl 99 --jinja \
  --host 127.0.0.1 --port 8080 \
  --alias qwen3.8-27b-coder390-Q2LynnStyle-Mtp \
  --spec-type draft-mtp --spec-draft-n-max 4

To use another GGUF tier, only change -m to ...-Q3LynnStyle-MTP.gguf, ...-Q4LynnStyle-MTP.gguf, ...-Q6_K-MTP.gguf, or ...-Q8_0-MTP.gguf (and the --alias, e.g. qwen3.8-27b-coder390-q6k-mtp, qwen3.8-27b-coder390-q8-mtp). MTP is fused inside the GGUF (blk.64); do not attach an external draft; the log should show creating MTP draft context against the target model. For image input add --mmproj ./GGUF/mmproj-Qwen3.8-27B-Q8_0.gguf.

llama.cpp · vision (image input)

llama-server -m ./GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-MTP.gguf \
  --mmproj ./GGUF/mmproj-Qwen3.8-27B-Q8_0.gguf \
  -c 409600 -np 4 -ngl 99 --jinja \
  --host 127.0.0.1 --port 8080 \
  --alias qwen3.8-27b-coder390-Q2LynnStyle-Mtp \
  --spec-type draft-mtp --spec-draft-n-max 4

The GGUF files have no vision weights; --mmproj ./GGUF/mmproj-Qwen3.8-27B-Q8_0.gguf adds them and works with all eight GGUF files (swap -m and --alias as above; image input tested on Q3 / Q4 LynnStyle, Q6_K, and Q8_0). Leave out --spec-type / --spec-draft-n-max to run without speculation. Without --mmproj llama.cpp runs text-only. Images go as OpenAI-style image_url parts to POST /v1/chat/completions (same request as the NInfer example, with model set to the --alias).

Optional: external Q8_0 MTP draft. GGUF/mtp-Qwen3.8-27B-Coder390-EfficientThink-Q8_0.gguf (3,164,006,656 bytes) is a standalone Q8_0 MTP head, the same weights as the built-in MTP head of the Q6_K and Q8_0 packages. Q2, Q3 and Q4 LynnStyle ship a smaller Q4 MTP head; add -md pointing at this file to use the Q8 head instead. Q6_K and Q8_0 already contain a Q8 head, so they gain nothing from it. With -md the log shows loading draft model rather than creating MTP draft context against the target model. Checked on one RTX PRO 6000 on 2026-10-09 (llama.cpp, Q3 LynnStyle target, --spec-draft-n-max 4): the log shows loading draft model. Greedy acceptance is 66.1% at 110.2 tok/s, against 66.0% at 112.2 tok/s for the built-in Q4 MTP head with the same command and no -md. At temperature 1.0 the two are 52.4% at 94.4 tok/s and 51.9% at 95.6 tok/s. All 12 prompts produced the same text; the Q8 file is not faster.

llama-server -m ./GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-MTP.gguf \
  -md ./GGUF/mtp-Qwen3.8-27B-Coder390-EfficientThink-Q8_0.gguf \
  -c 409600 -np 4 -ngl 99 --jinja \
  --host 127.0.0.1 --port 8080 \
  --alias qwen3.8-27b-coder390-Q3LynnStyle-Mtp \
  --spec-type draft-mtp --spec-draft-n-max 4

Files overview

FP8/ (main package: 19 files, 31,262,188,871 bytes)

FP8/SHA256SUMS covers the first 18 files below.

FP8/DFlash2-FP8/ (4 files, 2,407,384,680 bytes)

BF16/ (12 files, 55,583,125,732 bytes; the DFlash2 subdirectory is listed separately below)

BF16/SHA256SUMS covers the first 11 files below. mtp-bf16.safetensors is byte-identical to the file of the same name in the FP8 package.

BF16/DFlash2-FP8/ (4 files, 2,407,384,680 bytes)

Byte-identical to FP8/DFlash2-FP8/.

NVFP4/W4A16/ (main package: 17 files, 20,613,406,953 bytes; the DFlash2 subdirectory is listed separately below)

NVFP4/W4A16/SHA256SUMS covers the first 16 files below; its own SHA256 is ecc28d31be0d97e23509af61625251af14980db5b67189a1d1ca75a3fc9652f3. mtp-bf16.safetensors is byte-identical to the file of the same name in the FP8 package.

NVFP4/W4A16/DFlash2-FP8/ (4 files, 2,407,384,680 bytes)

Byte-identical to FP8/DFlash2-FP8/.

NVFP4/W4A4/ (main package: 17 files, 20,613,386,462 bytes; the DFlash2 subdirectory is listed separately below)

NVFP4/W4A4/SHA256SUMS covers the first 16 files below; its own SHA256 is 5aea8bb9c1f1979661bfa5c1216de7aafeb64cd708aa8cc71f3b185c224cfde0. mtp-bf16.safetensors is byte-identical to the file of the same name in the FP8 package.

NVFP4/W4A4/DFlash2-FP8/ (4 files, 2,407,384,680 bytes)

Byte-identical to FP8/DFlash2-FP8/.

NVFP4/W4A4-W8A8/ (main package: 18 files, 23,769,634,648 bytes; the DFlash2 subdirectory is listed separately below)

NVFP4/W4A4-W8A8/SHA256SUMS covers the first 17 files below; its own SHA256 is c177f4907a7dd037efe91b8d34927dd3a3624227f259238b82db0f12f8b0b8e2. mtp-bf16.safetensors is byte-identical to the file of the same name in the FP8 package.

NVFP4/W4A4-W8A8/DFlash2-FP8/ (4 files, 2,407,384,680 bytes)

Byte-identical to FP8/DFlash2-FP8/.

evaluation/

Score, benchmark, smoke-test (static FP8 and BF16), and calibration records. evaluation/SHA256SUMS covers the first 22 files below (23 files in total, including the checksum list itself). NVFP4_*.results.summary.json are the raw summaries of the three suites for the three NVFP4 tiers (NVFP4_MIXED_* is the W4A4-W8A8 tier).

Verification

(cd FP8 && sha256sum -c SHA256SUMS && cd DFlash2-FP8 && sha256sum -c SHA256SUMS)
(cd BF16 && sha256sum -c SHA256SUMS)
(cd NVFP4/W4A16 && sha256sum -c SHA256SUMS && cd DFlash2-FP8 && sha256sum -c SHA256SUMS)
(cd NVFP4/W4A4 && sha256sum -c SHA256SUMS && cd DFlash2-FP8 && sha256sum -c SHA256SUMS)
(cd NVFP4/W4A4-W8A8 && sha256sum -c SHA256SUMS && cd DFlash2-FP8 && sha256sum -c SHA256SUMS)
(cd NVFP4-NInfer/W4A4 && sha256sum -c SHA256SUMS)
(cd NVFP4-NInfer/W4A4-W8A8 && sha256sum -c SHA256SUMS)
(cd INT8/W8A8 && sha256sum -c SHA256SUMS && cd MTP-BF16 && sha256sum -c SHA256SUMS)
(cd INT4/W4A16 && sha256sum -c SHA256SUMS)
(cd GGUF && sha256sum -c SHA256SUMS)
(cd GGUF-NInfer && sha256sum -c SHA256SUMS)
(cd evaluation && sha256sum -c SHA256SUMS)

NInfer loading notes

• Only the NInfer engine can load .ninfer; SGLang, vLLM, llama.cpp, and transformers cannot.
• Use ninfer-serve from official NInfer (Neroued/ninfer, master branch); no patches are needed. (Only the GGUF-derived .ninfer packages need NInfer-all.) Tested on one RTX PRO 6000 Blackwell (compute capability 12.0); NInfer supports at most 8 concurrent requests.
• NVFP4-NInfer/W4A4/ and NVFP4-NInfer/W4A4-W8A8/ each hold one .ninfer file with MTP and DFlash2: the text path is the same one scored on 2026-10-05, and the package also carries a Q8 MTP head, the DFlash2 draft, the proposal head, and (since 2026-10-09) the vision component, so --vision enables image input; the vision-bf16.safetensors in the same folder is the BF16 source of that part and is not needed at run time. The MTP projections are Q8 because several of their matrix shapes have no BF16 kernel in NInfer, and a BF16 MTP makes the server exit while building the compute graph.
• MTP and DFlash2 are mutually exclusive per launch (--spec mtp or --spec dflash2); --lm-head-draft must be used together with --spec. MTP accepts 1–5 draft tokens and DFlash2 1–15; the NInfer commands under Commands above use 4 and 8, as in the benchmark. The DFlash2 draft is built into the package, so no separate DFlash2-FP8/ directory is needed.
• These context flags are verified to start: --max-context 102400 --kv-capacity 819200 --max-concurrency 8 --kv-dtype fp8. --kv-capacity must fall between max-context and max-context × max-concurrency; 819,200 is exactly 8 × 102,400.
• The HF packages under the NVFP4 directories still run in SGLang (MTP or DFlash2); they are not interchangeable with the NInfer packages.

NInfer V2 packages (Tesla V100 only)

The four packages whose filenames contain NInferV2-NOT-FOR-V3 are for Tesla V100 only (ninfer-v100 branch); on any other GPU, use the packages without this suffix.

• The 4 packages whose filenames contain NInferV2-NOT-FOR-V3 use the NInfer V2 container and are only for the Tesla V100 ninfer-v100 branch. Official NInfer (ninfer-serve) reads V3 only and rejects these files; on other GPUs, use the V3 packages above (without NInferV2 in the name). Do not rename the V2 packages to V3 filenames or use them to replace the V3 packages.
• The two text-only packages (Qwen3.8-27B-Coder390-W4A4-NInferV2-NOT-FOR-V3.ninfer, Qwen3.8-27B-Coder390-W4A4-W8A8-NInferV2-NOT-FOR-V3.ninfer) have no MTP or DFlash2, so do not pass --spec; the two MTP-DFlash2 packages (…-MTP-DFlash2-NInferV2-NOT-FOR-V3.ninfer) accept --spec mtp or --spec dflash2; they already contain the MTP head and the DFlash2 draft, so no external draft model is needed. None of the four V2 packages includes the vision component, so they cannot take images (no --vision); for image input use the V3 packages.
• V100 has no NVFP4 KV or K8V4 KV, so do not use the V3 flags --kv-dtype rk8v4 or --kv-capacity 819200; when serving on V100, add --prefill-chunk 2048.
• The V2 packages have not been separately evaluated; they have only passed SHA256 verification. In SHA256SUMS, the two V2 lines come after a # comment line, separate from the V3 lines.

# V100 ninfer-v100 branch · text-only package (no --spec)
ninfer ./NVFP4-NInfer/W4A4/Qwen3.8-27B-Coder390-W4A4-NInferV2-NOT-FOR-V3.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-context 8192 \
  --max-new 256

# V100 ninfer-v100 branch · MTP-DFlash2 package (MTP on)
ninfer ./NVFP4-NInfer/W4A4/Qwen3.8-27B-Coder390-W4A4-MTP-DFlash2-NInferV2-NOT-FOR-V3.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-context 8192 \
  --max-new 256 \
  --spec mtp --draft-tokens 3

NVFP4-NInfer/W4A4/ (5 files, 68,925,700,831 bytes)

NVFP4-NInfer/W4A4-W8A8/ (5 files, 68,925,700,846 bytes)

INT8 W8A8 notes

• Text and image (since 2026-10-09): the architecture is Qwen3_5ForConditionalGeneration (64 text layers, linear and full attention alternating 3:1). The BF16 vision tower from the BF16 tier is a separate file (model_visual.safetensors, 333 tensors, 921,497,224 bytes), with preprocessor_config.json / video_preprocessor_config.json; it is excluded from quantization. The 8 INT8 weight files are unchanged. The package has no embedded MTP head (use MTP-BF16/) and no DFlash2 draft. Checked on one RTX PRO 6000 on 2026-10-09 (vLLM 0.28, Cutlass INT8 kernel disabled, MTP-BF16/ with 3 draft tokens): the image prompt was answered correctly, text output was normal, accept length 3.21, single-request decode 86 tok/s.
• Quantization: SmoothQuant + INT8 W8A8. Weights are quantized to INT8 per output channel, and activations are quantized to INT8 per token at runtime; the SmoothQuant pre-scale is folded into the weights, so no inference-engine patch is needed. Quantization metadata uses the compressed-tensors (int-quantized) format.
• Layer layout: MLP gate/up/down, full-attention q/k/v/o, and linear-attention in_proj_qkv / in_proj_z / out_proj make up 400 INT8 Linear layers; linear-attention in_proj_a / in_proj_b and lm_head, 97 in total, stay BF16; linear-attention control branches such as A_log, dt_bias, norm, and conv1d also stay BF16.
• Calibration and error: 512 calibration samples; held-out NLL rises from 0.50535 (BF16) to 0.52812, a perplexity ratio of 1.023 (measured before the pre-scale was folded into the weights).
• Loading: validated only on vLLM 0.28; not for SGLang or NInfer. VLLM_LOAD.json records the load requirements. On RTX PRO 6000 Blackwell the Cutlass INT8 kernel must be disabled; see Commands above.
• MTP draft: INT8/W8A8/MTP-BF16/ holds the BF16 MTP tensors only. Checked on one RTX PRO 6000 on 2026-10-09 (vLLM 0.28, Cutlass INT8 kernel disabled, num_speculative_tokens=3): mean accept length 2.74–2.90.

INT8/W8A8/ (19 files, 30,422,466,299 bytes)