Qwen3.8-Flash-Next-Uncensored-NVFP4 — 2× DGX Spark (GB10) build
A byte-identical mirror of orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4
@ 56eb374a747fe0c359983553d869c27b86d1c8ce (Apache-2.0, original card in README.orcarouter.md),
plus everything needed to reproduce a production deployment on two NVIDIA DGX Spark (GB10) nodes: pre-exported FP8 PLE
tables, the vLLM patches, the launcher and the measured throughput below.
At a glance
| metric |
value |
| max context |
1,048,576 tokens (YaRN ×4 over native 262,144) |
| KV cache pool |
3.76 M tokens (FP8 KV + FP8 indexer, 28 GB budget) |
| prefill |
3,723 tok/s at 8k prompt · 2,737 tok/s at 1 M |
| decode, one stream |
63–66 tok/s (MTP speculative decoding, 2 draft tokens) |
| decode, 8 streams |
236 tok/s aggregate |
| time to first token, 1 M-token prompt |
6.2 min |
| setup |
2 × DGX Spark (GB10, 128 GB each), tensor parallel 2, nodes linked by one direct 200 GbE cable (RoCE v2), vLLM nightly 0.30.1rc1.dev616 |
How the nodes are wired and what each technique does: docs/.
Documentation
| page |
what it covers |
| docs/README.md |
index of all documentation |
| docs/network-architecture.md |
how the two Spark nodes are connected: ports, the one direct 200 GbE cable and its two PCIe halves, LAN/storage planes, NCCL settings, fabric capacity vs use |
| docs/techniques.md |
every technique in the stack: FP8/NVFP4, TP=2, YaRN 1 M, QSA + FP8 KV/indexer, MTP, CUDA graphs, GPU-direct PLE, memory map |
| docs/fabric.md |
InfiniBand / RoCE fabric traffic from the ib_port_* metrics: traffic per PCIe half, peaks vs link capacity, error counters |
| docs/memory.md |
how memory is used on each node: unified GPU/CPU memory (weights, KV cache pool, CUDA graphs), page cache and host processes, NVMe disk |
| docs/metrics/README.md |
production metrics from VictoriaMetrics since 2026-08-31: traffic, tokens, cache hits, latency percentiles, per-era comparison, raw data |
| docs/timeline.md |
project timeline: ideas, changes, improvements, rejected experiments and incidents, with a visual timeline |
| docs/lessons-learned.md |
experience: what went wrong, why, and the rule we follow now |
| deploy/README.md |
how to use every script and config in deploy/ to reproduce the deployment |
1. What is in this repo
| path |
size |
content |
model.safetensors.index.json + shards 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 |
183 GB (170 GiB) |
checkpoint @56eb374a |
model-mtp.safetensors |
MTP head for speculative decoding |
ple-fp8/: rank 0 .bin · .json · rank 1 .bin · .json |
2 × 25.6 GB |
per-TP-rank FP8 export of the layer-1 PLE (n-gram embedding) table, with scale + checksums |
deploy/ |
launcher, configuration, vLLM patches, PLE tools, benchmark and quality scripts to reproduce the deployment — every file is explained in deploy/README.md |
docs/ |
network architecture, techniques, production metrics, timeline, lessons learned — see Documentation |
assets/ |
README charts (SVG) |
2. The model
|
|
| architecture |
Qwen3.8-Flash-Next (Qwen4-exp), MoE, vision-language, QSA sparse attention with indexer, PLE n-gram embedding, MTP head |
| quantization |
weight-only, compressed-tensors: dense layers + lm_head FP8 per-channel (W8A16), experts NVFP4 (W4A16) — no activation scales, so W4A4 kernels do not apply; MoE runs on Marlin |
| native context |
262,144 → 1,048,576 with YaRN ×4 |
| PLE table |
320 M-row vocabulary × 160, BF16 in the checkpoint (~95 GiB); served as FP8 per-rank shards from NVMe |
3. Hardware and topology
|
|
| nodes |
2 × NVIDIA DGX Spark (GB10 Grace-Blackwell, 128 GB unified memory each, sm_121a) |
| parallelism |
tensor-parallel 2 (rank 0 = head + OpenAI compatible API, rank 1 = headless worker) |
| interconnect |
one direct 200 GbE QSFP cable between ConnectX-7 port 1 of each node (200 Gb/s per direction), RoCE v2; NCCL drives both PCIe x4 halves of that port (NCCL_IB_HCA==rocep1s0f1,roceP2p1s0f1), MTU 1500 (MTU 9000 measured ~0 % gain). Port 0 goes to the storage switch (NFS) |
| storage |
checkpoint + PLE tables on local NVMe (~8 min cold boot) |
4. Software
| component |
version |
| vLLM |
nightly 0.30.1rc1.dev616+g1a001d584 |
| torch |
2.13.0+cu130 |
| flashinfer |
0.7.0.post1 |
| transformers |
5.16.1 (pinned, see lessons learned L16) |
5. Serving configuration (key flags)
vllm serve orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 --revision 56eb374a \
--tensor-parallel-size 2 --nnodes 2 --node-rank <0|1> --master-addr <head-ip> --master-port 25000 [--headless on rank 1] \
--max-model-len 1048576 \
--hf-overrides '{"text_config":{"rope_parameters":{"mrope_interleaved":true,"mrope_section":[11,11,10],"rope_type":"yarn",
"rope_theta":10000000,"partial_rotary_factor":0.25,"factor":4.0,"original_max_position_embeddings":262144},
"max_position_embeddings":1048576}}' \
--kv-cache-dtype fp8 --attention-config '{"indexer_kv_dtype": "fp8"}' \
--kv-cache-memory-bytes 28000000000 \
--engram-config '{"cpu_offload": false}' \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
--compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,3,4,6,8,10,12],"max_cudagraph_capture_size":12}' \
--max-num-seqs 16 --max-num-batched-tokens 8192 \
--limit-mm-per-prompt '{"image": 64, "video": 2}' --chat-template chat_template.qwen38.jinja
The authoritative, complete version is deploy/qwen38.env + deploy/build/run-node.sh.
Lessons learned while getting here (YaRN override shape, memory traps, NCCL NCCL_IB_HCA setup, never compiling on a serving node, benchmark noise) are collected in docs/lessons-learned.md.
6. PLE table: FP8 on NVMe, read GPU-direct
The PLE n-gram table is the largest single tensor (~95 GiB BF16). Instead of pinning it in RAM, each rank mmaps its FP8 shard
from local NVMe and the GPU reads the mapped pages directly (GB10 has pageableMemoryAccess=1, i.e. the GPU walks host page tables).
A Triton gather kernel (qwen38_ple_uva.py) does the lookup inside the FULL decode CUDA graph through vLLM's own
VocabParallelEmbedding path (mask → quant_method.embedding → int8-sum all-reduce).
|
CPU gather (before) |
GPU-direct (now) |
| lookup latency per step |
37–45 µs |
15–16 µs |
| host RAM held per rank |
47.7 GiB pinned BF16 (upstream default) |
0 pinned; 23.8 GiB FP8 in page cache (any residency works) |
Setup: put ple-fp8/*rank0of2.* on node 0 and *rank1of2.* on node 1, in the directory QWEN38_PLE_MMAP_DIR points to
(QWEN38_PLE_MMAP=1). The json carries the revision, row range, scale and head/tail sha256 — the loader refuses a mismatched shard.
Apply deploy/build/patches-next/ with apply-patches-next.sh (patch 10 is sha-guarded against the vLLM file it modifies).
To regenerate the tables from the checkpoint: deploy/build/ple_export.py (~100 s per rank).
7. Throughput
Method: deploy/build/upg_bench.py against the live OpenAI compatible endpoint, 2026-10-03. Median of 3 repeats (524 k and 1 M: single run).
Prefill uses unique prompts (no prefix-cache hits); decode uses one fixed prompt with ignore_eos and MTP n=2.
Prefill
Prefill tok/s (time to first token) — 8k: 3,723 (2.2 s) · 32k: 3,682 (8.7 s) · 131k: 3,592 (35.3 s) · 524k: 3,154 (2.7 min) · 1 M: 2,737 (6.2 min). Full numbers in the table below.
Time to first token grows about linearly with prompt length — 2.2 s at 8k, 8.7 s at 32k, 35.3 s at 131k, 164.8 s at 524k, 370.1 s at 1 M (≈ 0.27 ms per prompt token up to 131k, 0.37 ms at 1 M).
Prefill speed as a share of the 8k speed (3,723 tok/s) — 32k: 99 %, 131k: 96 %, 524k: 85 %, 1 M: 74 %.
With 8 simultaneous unique prompts the total prefill rate stays ~3.7k tok/s, so each request waits longer — 8k prompt: 2.2 s alone vs 11.7 s with 8 at once; 32k prompt: 8.7 s alone vs 44.7 s with 8 at once.
| prompt tokens |
concurrency |
prefill tok/s |
TTFT p50 |
| 8,192 |
1 |
3,723 |
2.2 s |
| 8,192 |
8 |
3,690 |
11.7 s |
| 32,768 |
1 |
3,682 |
8.7 s |
| 32,768 |
8 |
3,669 |
44.7 s |
| 131,072 |
1 |
3,592 |
35.3 s |
| 524,288 |
1 |
3,154 |
164.8 s |
| 1,000,000 |
1 |
2,737 |
370.1 s |
Decode
1,024 generated tokens — 1 stream 62.4 tok/s; 8 streams 235.6 tok/s total, 30.5 per stream. 8,192 generated tokens — 1 stream 65.8 tok/s; 8 streams 223.2 tok/s total, 29.6 per stream.
Going from 1 to 8 concurrent streams multiplies total output by 3.8× (1,024-token generations: 62.4 → 235.6 tok/s) and 3.4× (8,192-token generations: 65.8 → 223.2 tok/s).
| generated tokens |
concurrency |
aggregate tok/s |
per-stream tok/s |
MTP accept |
| 1,024 |
1 |
62.4 |
63.0 |
0.70 |
| 1,024 |
8 |
235.6 |
30.5 |
0.69 |
| 8,192 |
1 |
65.8 |
65.9 |
0.75 |
| 8,192 |
8 |
223.2 |
29.6 |
0.72 |
Interconnect
RoCE traffic on the head node (send + receive, both PCIe halves of the 200 GbE link) — prefill 8k ×1: 29.4 Gb/s · prefill 32k ×8: 32.8 Gb/s · prefill 131k: 32.0 Gb/s · prefill 1 M: 24.5 Gb/s · decode 1 stream: 2.3 Gb/s · decode 8 streams: 8.1 Gb/s. The link carries ~185 Gb/s in one direction when both PCIe halves are driven (RDMA benchmark; 200 Gb/s cable rate); see docs/network-architecture.md.
Capacity
|
|
| KV cache pool (28 GB budget, FP8 KV + FP8 indexer) |
3,756,138 tokens |
| max context per request |
1,048,576 |
| max concurrent sequences |
16 |
| cold boot (NVMe, both ranks) |
~8 min |
8. Long context and quality
| check |
result |
| needle at 962,348 tokens (depth 0.25) |
answer 094KBVOY (expected 094KBVOY), TTFT 347 s |
| needle at 942,275 tokens (depth 0.75) |
answer OMTJFSCH (expected OMTJFSCH), TTFT 338 s |
| teacher-forced top-1 agreement vs previous build, 63 fixed documents |
97.75 % (same-build noise floor 98.05 %) |
| mean NLL drift vs previous build |
+0.019 % (noise floor −0.09 %) |
FP8 KV + FP8 indexer cost no measurable quality on this probe while doubling KV capacity.
9. Quick start
Short version below; every step, file and variable is explained in deploy/README.md.
# on BOTH nodes: checkpoint in Hugging Face cache layout (run-node.sh loads it by repo id, offline)
HF_HOME=~/hf-cache hf download orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 --revision 56eb374a747fe0c359983553d869c27b86d1c8ce
# this repo's deploy/ and ple-fp8/ (node 0 needs *rank0of2.*, node 1 needs *rank1of2.*)
hf download quocbao747/Qwen3.8-Flash-Next-Uncensored-NVFP4-2xGB10 --include 'deploy/*' 'ple-fp8/*' --local-dir ~/qwen38
cd ~/qwen38/deploy
bash build/make-venv-next.sh && bash build/apply-patches-next.sh # vLLM nightly dev616 + patches 02, 03, 10
bash build/prebuild-jit-next.sh # with the model STOPPED, never on a serving node
# edit qwen38.env (IPs, NCCL_IB_HCA, socket interfaces, HF_CACHE_DIR_LOCAL, QWEN38_PLE_MMAP_DIR);
# put VLLM_API_KEY=... in qwen38.local.env (chmod 600)
ROLE=worker bash build/run-node.sh # node 1
ROLE=head bash build/run-node.sh # node 0 -> OpenAI compatible API on :8081
10. Licence and attribution
Weights: Apache-2.0, from orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4
(itself derived from Qwen/Qwen3.8-Flash-Next) — see README.orcarouter.md for the original card, intended use and caveats of the uncensored variant.
Deployment scripts and patches in deploy/: Apache-2.0.
