Versions below for what a v2 would have to clear. A derivative of the official nvidia/GLM-5.3-Flash-NVFP4 checkpoint in which the layers NVIDIA's release left in BF16 — the attention linears (KDA fused inproj/out projections, MLA q/kv/o, indexer wqb), the…
Runs On
What it takes to serve GLM-5.3-Flash-NVFP4-Wabi (165B parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.
| Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits |
|---|---|---|---|---|---|
| 16-bit | 330.1 GB | 396.1 GB | 2x MI325X (256 GB) Vultr |
$4.00 | 2x MI355X $5.18 · 3x MI300X $5.55 |
| 8-bit | 165.0 GB | 198.0 GB | 1x MI325X (256 GB) Vultr |
$2.00 | 1x MI355X $2.59 · 2x MI300X $3.70 |
| 4-bit | 82.5 GB | 99.0 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x MI325X $2.00 · 1x MI355X $2.59 |
Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.
Model Card
By Tenhkspark, published under mit, revision 655e3a5ed2f5.
Versions below for what a v2 would have to clear. A derivative of the official nvidia/GLM-5.3-Flash-NVFP4 checkpoint in which the layers NVIDIA's release left in BF16 — the attention linears (KDA fused inproj/out projections, MLA q/kv/o, indexer wqb), the shared experts (gate/up/down), and lmhead — are quantized to W4A16 NVFP4 (weight-only 4-bit, group size 16). Routers, norms, embeddings, the vision tower, and the MTP layer (layers.45) stay BF16; the routed experts keep the stock W4A4 NVFP4 quantization. enjoying the incomplete This checkpoint does not boot on the stock vLLM image. It needs the kda-quant and mla-quant source overlays from the companion repository (plus glm5next-mtp-bf16…
Read Tenhkspark's full model card
GLM-5.3-Flash-NVFP4-Wabi — route h W4A16 dense-side quantization
English · 日本語 · 简体中文 · 한국어
Current tag: v1 — the only version published so far. See
Versions below for what a v2 would have to clear.
A derivative of the official nvidia/GLM-5.3-Flash-NVFP4 checkpoint in
which the layers NVIDIA's release left in BF16 — the attention linears
(KDA fused in_proj/out projections, MLA q/kv/o, indexer wq_b), the
shared experts (gate/up/down), and lm_head — are quantized to
W4A16 NVFP4 (weight-only 4-bit, group size 16). Routers, norms,
embeddings, the vision tower, and the MTP layer (layers.45) stay BF16;
the routed experts keep the stock W4A4 NVFP4 quantization. enjoying the
incomplete
Read this before you download
This checkpoint does not boot on the stock vLLM image. It needs the
kda-quant and mla-quant source overlays from the companion
repository (plus glm5next-mtp-bf16 when you serve with speculation),
bind-mounted into the serving container as documented there. Served as a
drop-in nvidia/GLM-5.3-Flash-NVFP4 replacement it fails to load.
Those overlays are my own local patches for my own two-node machine. I have not proposed them upstream and they are not on anyone's upstream roadmap, so please treat them as something you patch in yourself, not as a feature arriving in a future vLLM release.
Purpose: single-stream decode speed on 2 NVIDIA DGX Spark nodes (GB10,
TP=2 over Ray, NCCL over RDMA). On my fixed ruler — 64 Japanese prose
prompts, max_tokens=512, temperature=0, thinking off, C=1 — this
checkpoint plus the model's own MTP draft at K=2 reaches 35.09 tok/s
on node pair 1, expert parallel on, NCCL over RDMA. The same checkpoint,
pair, K and EP over TCP sockets reads 24.01 tok/s on the same ruler.
Every other number, with its pair / transport / prompt count / K / EP,
is in the table below; I do not quote a ratio between rows whose
conditions differ.
Several improvement ideas are still on my list, but the model has reached a level I find usable, so I am releasing it now. It is a work in progress: please take it and move it toward your own idea of complete. I will keep working and publish updates as results come in, and I would be glad to see your versions on Hugging Face and GitHub too.
About the name
Wabi (侘び) is the Japanese sense of accepting what is imperfect or plain and finding richness in it; this release is a work in progress that I publish as it is, improvements included when they are measured.
Base checkpoint
nvidia/GLM-5.3-Flash-NVFP4at revision09b04e5e74bca08ca8549fc736d4cdd8624bfde3(pinned).- Upstream base model:
zai-org/GLM-5.3-Flash(MIT).
How it was made
Produced by requant/requant.py --target h in the companion repository:
https://github.com/tenhkspark/glm53-flash-nvfp4-2node. The script reads
the official checkpoint read-only, rewrites the target tensors into fresh
safetensors shards, and rewrites config.json / hf_quant_config.json /
model.safetensors.index.json for the modelopt MIXED_PRECISION loader.
Non-target tensors are copied through unchanged.
The MTP draft checkpoint
The --speculative-config in the serving command below points at
/checkpoint-mtp, a separate BF16 draft directory. It is not part of
this repository — you build it, and the companion repository builds it
for you in STEP 8 (build-mtp-draft) of scripts/agent-run.sh, which
runs:
python3 requant/build-mtp-draft.py \
--target /path/to/GLM-5.3-Flash-NVFP4 \
--out /path/to/checkpoint-mtp
The MTP layer is layers.45, which the NVIDIA release already stores in
BF16 and which this repository carries through unchanged. The script
copies those tensors as-is, adds the two keys the vLLM draft does not
share with the target (embed_tokens and shared_head.head, sourced
from embed_tokens and lm_head), and writes a flattened text-only
config.json with the quantization section dropped. Point --target at
the stock nvidia/GLM-5.3-Flash-NVFP4 directory rather than at this one:
route h quantizes lm_head to NVFP4 and the draft has to be BF16. The
output is roughly 16 GiB and each node needs its own copy.
Quality delta (gate v2, measured)
I evaluated this checkpoint against a stock baseline capture using the same serving stack on 2026-09-16:
| criterion | threshold | stock | route h | result |
|---|---|---|---|---|
| degenerate outputs over the 64-prompt ruler | 0 | 0 | 0 | PASS |
| perplexity ratio on the inline Japanese probe | <= 1.10 | 1.0 | 1.051 | PASS |
| eval-200 live accuracy (150 items, the 0/50 tool floor excluded) | no more than 0.02 below stock | 0.447 | 0.453 (+0.0067) | PASS |
| TTFT on the same probe text | within 1.2x of stock | 2.380 s | 2.436 s (x1.02) | PASS |
Perplexity. I measure it on eight short Japanese sentences (249
characters in total) written for this test and defined inline in
requant/verify.py; they are not drawn from any public corpus and I
claim no training-set exclusion. Each sentence goes to the served
endpoint as a prefill-only request (/v1/completions with echo=true,
max_tokens=1, prompt_logprobs=1, temperature 0); the
log-probabilities of the actual prompt tokens are accumulated across all
eight sentences, and the mean negative log-likelihood over that single
pooled token stream is exponentiated into one number. Stock and
candidate are measured by the identical function against the same served
model: route h scored 12.543 against 11.938 for stock, a ratio of 1.051.
The sample is small, so this checks for gross degradation, not for
parity. The TTFT gate reuses the same eight sentences, repeated twelve
times, as its probe input.
eval-200. A fixed set of 200 Japanese single-turn prompts shipped in
the companion repository (bench/eval-200.jsonl): 50 each of reason,
trap, tool and longread, mixed easy/medium/hard. Every item is answered
greedily on the served endpoint (temperature 0, max_tokens 384,
thinking skipped via an empty assistant continuation) and graded by a
deterministic rule in requant/verify.py: whitespace-insensitive exact
match when the item's rubric demands the answer alone, substring
containment otherwise; trap items must carry a refusal and no digits,
tool items must name the required functions in order plus the expected
final value. The tool column is 0/50 on every checkpoint by construction
-- the prompts never name a callable function and no tool schema is sent
-- so the gate scores the remaining 150 live items and requires the
candidate to stay within 0.02 of stock with zero request errors (0.447
stock -> 0.453 route h). The raw totals 67/200 -> 68/200 are reported
for transparency only; they include that permanent zero column and are
not the pass/fail criterion.
Accuracy by category (50 items each):
| kind | stock | route h |
|---|---|---|
| reason | 32/50 | 27/50 |
| trap | 13/50 | 16/50 |
| tool | 0/50 | 0/50 |
| longread | 22/50 | 25/50 |
The category moves are not independent: the trap grader rewards refusal, so a more hedging model scores higher there and lower on reason. A net +1 total can be produced by a directional degradation.
The perplexity ratio of 1.051 is a 5.1% increase over the stock model: inside the declared gate threshold, but an increase rather than strict parity. Eval-200 moved by +0.0067 on the 150 live items the gate scores. A 150-item set resolves accuracy to roughly +-0.08 at 95% confidence (paired, assuming ~20% of items flip), so the +0.0067 I measured is indistinguishable from zero — and so would be a true regression of 0.05. The 0.02 gate threshold is finer than what this set can resolve: the +-0.08 half-width is 4x the threshold. Treating that half-width as a normal sampling error (sigma = 0.08 / 1.96, about 0.041) and the gate's accuracy condition as 'observed difference at least -0.02', a checkpoint genuinely 0.06 worse than stock still clears that condition about 16% of the time, and one genuinely 0.08 worse about 7%. Those 0.06 and 0.08 are percentage-point differences in accuracy, not relative drops, and the two probabilities are for the accuracy condition alone, not for the composite gate. Reading this gate as 'no regression' is wrong; it only rules out a large one.
Every gate and eval-200 row above is Japanese, single-turn and thinking off. The one evaluation that would cover long context, parallel requests and agentic tool use — the 13-item end-to-end serve suite — did run, on 2026-09-16, on both node pairs, but 4 of its 13 items returned nothing in either run: agentic tool success, agentic final answer, integrated task en 108K and integrated task ja 108K. So agentic tool use and 108K long context are still unmeasured on this checkpoint — not because the suite is waiting to run, but because the items that cover them produced no result when it did. The two runs are also not a like-for-like route h vs stock comparison: pair 1 ran with no speculation and pair 2 ran with speculative decoding at 78.0% acceptance, so even the items that did return cannot be read as a difference between the two checkpoints.
What this gate does not measure. Every number above is Japanese, single-turn, thinking off, greedy, under 2k tokens of context and one request at a time. I have no measurement of this checkpoint on English or any other language, on code correctness, on multi-turn conversations, on tool calling (the tool column is a structural zero), on instruction following, on long-context quality — the needle probe in "Long context, measured" scores 40/40 up to 194,544 tokens, but that is retrieval, and every quality probe above fits in about 2k — on safety behaviour, or with thinking enabled — which is how this model family is normally used. The requant rewrites the dense linears and lm_head, so those are exactly the places a regression could hide from these four probes. If you depend on any of them, measure it yourself before adopting this checkpoint; requant/verify.py takes a different prompt set with one flag.
For reference, the base release's own BF16-vs-NVFP4 card numbers (quoted from the base card at the pinned revision): GPQA Diamond 0.9217 -> 0.9211, SciCode 0.5621 -> 0.5769, MMMU Pro 0.7688 -> 0.7630, AA-LCR 0.7100 -> 0.7106, IFBench 0.6130 -> 0.6054, Terminal-Bench 2.1 0.8258 -> 0.8315. Those card figures come from standard public benchmarks under NVIDIA's own harness and are not comparable with the eval-200 numbers above, which are scored by this repository's own strict grader on Japanese prompts. Nothing here places my checkpoint on the same axis as the base card's benchmarks.
Speed (measured, with conditions)
Ruler: 64 Japanese prose prompts, temperature=0, max_tokens=512,
thinking skipped via an empty assistant continuation. C=1 streams the
prompts sequentially; tok/s = completion tokens / whole-window wall time
including TTFT. Hardware: 2x DGX Spark (GB10, 128 GB unified memory each),
TP=2 over Ray, RoCE link (ConnectX-7 class) between the nodes.
Serve flags in the repo's scripts: --max-model-len 204800,
--max-num-seqs 20, --gpu-memory-utilization 0.85,
--enable-prefix-caching, FP8 KV cache and
RAY_memory_usage_threshold=0.99 on both nodes. The
rows in the table ran the measurement rig instead — almost all of them at
--max-model-len 16384 with --max-num-seqs 20 and
--gpu-memory-utilization 0.86 and --enable-expert-parallel, except
the 19.05 stock + MTP row which ran with expert parallel off (ep =
off in the table), the pair-2 stock baseline at
--max-model-len 131072 with 32 slots, and the pair-2 stock RDMA row at
16384 with 32 slots.
Image: vllm/vllm-openai:glm53-flash-arm64-cu130 (+ derived RDMA image).
The shipped window is longer than the rig one on purpose. Re-measured on 2026-09-17 at the shipped window, with expert parallel on as everywhere else in this table, the ruler reads 34.78 tok/s — TPOT median 28.1 ms, TTFT median 0.313 s, 0 of the 64 requests failed — against 35.09 at 16384 / 20 / 0.86. That is a 0.9% difference across a 12.5x longer window, inside the run-to-run spread of about 2% on identical configurations.
The serve scripts in the recipe repo do not enable expert parallel,
while every row in this table was measured with it on. Run exactly as
they ship, the same ruler reads 37.33 tok/s (36.95 on a repeat); adding
the two EP flags back drops it to 35.05, reproducing the 34.78 row to
within 0.8%. EP is also hungrier on this hardware, so the scripts keep
it off. One operational note that matters more than either number: GB10
is unified memory, the engine sizes its budget from whatever was free
when it profiled, and the same script has reached READY with anywhere
from 5470 to 10270 MiB of host memory left. About 4.7 GiB of that is
spent after READY, the first time each larger prefill shape arrives, so
a boot that starts too low is killed later by one long prompt — and DGX
OS ships no OOM daemon, which makes that a hung node rather than a
killed process. The recipe repo ships serve/check-headroom.sh to gate
this at boot; run it before you send the first request. The longer window does not cost
concurrency either: 20 sequence slots do come up at 204800 (909 s to
READY, then the same 0-failure ruler), which they did not at 307200.
Utilization is the one flag that moved down rather than up — 0.89 has
been refused at boot on this pair, and at 0.88 the head node ran with
about 2.1% of host RAM free — close enough to exhaustion that it does
not matter what reaps the worker first — so the scripts ship 0.85 and
leave 8.6%. The requant is part of why the window fits: at 0.88 the
stock checkpoint tops out at 156672 tokens (vLLM prints that ceiling
when it refuses to start), while route h boots at 204800.
| checkpoint | pair | transport | K | ep | tok/s | TPOT med ms | weighted TPOT ms | TTFT med s | accept | quality gate | TTFT gate |
|---|---|---|---|---|---|---|---|---|---|---|---|
| stock | 2 | sockets | - | on | 10.75 † | 95.3 † | 0.361 † | - | ref | — | |
| stock + MTP | 2 | sockets | 2 | on | 17.71 | 55.5 | 55.73 | 0.447 | 0.6128 | ref | — |
| stock + MTP | 1 | sockets | 2 | off | 19.05 | 51.9 | 51.77 | 0.430 | 0.6175 | ref | — |
| stock | 2 | NCCL/IB (RDMA) | - | on | 14.32 † | 69.3 † | 69.29 | 0.349 † | - | ref | — |
| stock | 1 | NCCL/IB (RDMA) | - | on | 14.39 | 69.0 | 0.293 | - | ref | — | |
| stock + MTP | 2 | NCCL/IB (RDMA) | 2 | on | 23.93 | 41.0 | 41.17 | 0.366 | 0.6137 | ref | — |
| route h | 1 | sockets | - | on | 18.98 | 51.5 | 52.12 | 0.330 | - | PASS (PPL x1.051) | verified on sockets |
| route h | 1 | NCCL/IB (RDMA) | - | on | 27.15 | 35.7 | 36.4 | 0.269 | - | PASS (PPL x1.051) | verified on sockets; not re-run under RDMA |
| route h + MTP | 1 | sockets | 2 | on | 24.01 | 40.7 | 40.96 | 0.389 | 0.6221 | PASS (PPL x1.051) | verified on sockets |
| route h + MTP | 1 | NCCL/IB (RDMA) | 2 | on | 35.09 | 27.9 | 27.92 | 0.320 | 0.6221 | PASS (PPL x1.051) | verified on sockets; not re-run under RDMA |
| route g + MTP | 2 | NCCL/IB (RDMA) | 2 | on | 29.46 | 33.3 | 33.35 | 0.340 | 0.6182 | PASS (PPL x0.999) | verified on sockets; not re-run under RDMA |
| route h + MTP | 1 | NCCL/IB (RDMA) | 1 | on | 34.28 | 28.6 | 0.304 | 0.7979 | PASS (PPL x1.051) | verified on sockets; not re-run under RDMA | |
| route h + MTP | 1 | NCCL/IB (RDMA) | 3 | on | 31.98 | 30.8 | 0.331 | 0.483 | PASS (PPL x1.051) | verified on sockets; not re-run under RDMA | |
| route h + MTP | 2 | NCCL/IB (RDMA) | 2 | on | 34.47 | 28.2 | 28.43 | 0.313 | 0.6221 | PASS (PPL x1.051) | verified on sockets; not re-run under RDMA |
| route h + MTP | 1 | NCCL/IB (RDMA) | 2 | on | 34.48 | 28.3 | 28.4 | 0.317 | 0.6193 | PASS (PPL x1.051) | verified on sockets; not re-run under RDMA |
| route h + MTP | 1 | sockets | 2 | on | 23.75 | 41.2 | 41.4 | 0.394 | 0.6213 | PASS (PPL x1.051) | verified on sockets |
| route h + MTP | 1 | NCCL/IB (RDMA) | 2 | on | 35.12 | 27.9 | 27.9 | 0.313 | 0.6193 | PASS (PPL x1.051) | verified on sockets; not re-run under RDMA |
| route h + MTP | 1 | sockets | 2 | on | 24.35 | 40.4 | 40.4 | 0.39 | 0.6174 | PASS (PPL x1.051) | verified on sockets |
† these rows ran C=1 over the first 8 prompts of the ruler, not all 64.
The ep column records --enable-expert-parallel state per row — the
19.05 stock + MTP row ran with it off. The gate is split per row:
quality gate covers the checkpoint-level legs (degenerate / PPL /
eval-200); TTFT gate is transport-dependent and was probed once on
the route-h sockets serving, so RDMA rows read "verified on sockets;
not re-run under RDMA" rather than a four-condition PASS on that
transport. The weighted TPOT ms column is recorded directly in five
shipped JSONs (the four release re-runs plus the route-h no-draft RDMA
row); for nine more rows I computed the same quantity myself from each
run's per-prompt records in the pre-sanitisation factory log (not part
of this repository) and added the result to the row's shipped
results/logs/ entry. The 2026-09-13 stock baseline has no per-prompt
records anywhere I could find, so that row's cell stays empty rather
than filled with its own plain-mean figure, which is a different
statistic. Per-row measurement excerpts ship in the repo under
results/logs/ (see the source_log column of results/results.tsv).
K sweep, what is and is not settled: on route h + RDMA, pair 1, K=1
measured 34.28 tok/s and K=3 measured 31.98, both already in the table
above; K=2 remains the best of the values measured, not a proven
optimum. The re-measurement question is answered: the same config on the second
node pair read 34.47 tok/s, and an interleaved TCP/RDMA re-run on a
fresh, non-overlapping 64-prompt set with prefix caching disabled read
34.48 and 35.12 tok/s over RDMA and 23.75 and 24.35 over TCP — within
~2% of the original 35.09 on the RDMA side. The headline run: 32407
completion tokens over 923.43 s of wall, 0 failed requests. C=32
aggregate on stock was 95.08 tok/s (sockets) and 109.03 tok/s (RDMA);
both of those passes ran with 32 sequence slots, which the shipped
--max-num-seqs 20 cannot reproduce, so read them as rig numbers rather
than as what the released scripts do. No C=32 row exists for route h yet.
Long context, measured
The probe is the companion repo's bench/longctx.py: Japanese filler
grown to a target length, ten facts planted at head, middle and tail token depths, one
question each, temperature=0, graded by exact code match rather than
by a judge. The set is bench/longctx-probe.jsonl, and every document
is the shared prefix of its ten questions, so one long prefill is paid
per length. One pass on the shipped configuration —
--max-model-len 204800, --max-num-seqs 20,
--gpu-memory-utilization 0.85, --enable-prefix-caching, route h with
MTP at K=2 over RDMA:
| prompt tokens | needles found | first token, cold | first token, cache warm | prefill |
|---|---|---|---|---|
| 16,345 | 10/10 | 10.02 s | 4.27 s | 1630.6 tok/s |
| 65,545 | 10/10 | 39.95 s | 2.91 s | 1640.8 tok/s |
| 130,990 | 10/10 | 79.86 s | 3.15 s | 1640.2 tok/s |
| 194,544 | 10/10 | 120.04 s | 4.74 s | 1620.6 tok/s |
40 of 40, no depth weaker than another: head 12/12, middle 16/16, tail
12/12. One reading from a different configuration, kept out of the table
on purpose: at --max-model-len 307200 with utilization at 0.88 — not
what the scripts ship — a single 204,767-token document answered 10/10.
It needs a window the released scripts do not open, so it is not a row
above.
This is retrieval, not quality. It says the model still finds a planted fact at 194,544 tokens; it says nothing about whether its prose or its reasoning hold up there.
Three limits come with the long window.
- A prompt the size of the window does not fit. The probe's top
stage aims at 190k rather than 200k for a measured reason: a
204,754-token document plus
max_tokens=64is 204,818 against a 204,800 limit, and the server returnsHTTP 400. The engine survives — the next request is answered normally — but that document cannot be asked about. The usable ceiling is the window minus the generation budget minus the chat template. - The first token on a long prompt is slow. Prefill holds between
1620 and 1641 tok/s across all four lengths, so the 194,544-token
prompt takes 120.04 s before the first token appears. Nothing is
stuck; that is prefill.
--enable-prefix-cachingis in the shipped flags for this reason: pin one long document as the prefix and vary only the question, and every request after the first is an order of magnitude faster — at 130,990 tokens, 79.86 s cold against 3.15 s warm. - Past about 259,000 tokens nothing gets through on this hardware.
Not with a longer
--max-model-len, not with a lower--gpu-memory-utilization, not with a smaller--max-num-batched-tokens: host memory runs out and the node's OOM reaper takes the vLLM worker. Thepeak activationfigure vLLM prints at boot is measured on an 8192-token dummy run, and the scratch space the sparse-attention indexer wants during a long prefill sits outside the--gpu-memory-utilizationbudget entirely. Dropping utilization from 0.88 to 0.85 moved the death from 62 s into the prefill to 155 s; it did not make the prompt fit.
RAY_memory_usage_threshold=0.99 is exported on both nodes by the
repo's serve/start-head.sh and serve/start-worker.sh, and it is not
optional. Unified memory means --gpu-memory-utilization is taken out
of host RAM, so Ray's own OOM monitor sees the node above its default
0.95 and kills the largest actor it can find, which is the vLLM TP0
worker. What you see is EngineDeadError — at boot, or in the middle of
a request — with nothing wrong in the vLLM log; the kill is in the
raylet log. The two scripts bring up two separate raylets, each with its
own monitor, so the variable has to be exported on both.
Serving
The served API has no authentication, and it listens on every
interface. The scripts below pass --host 0.0.0.0, so anything that
can reach port 8000 on either node can send requests. Keep this on a
network you trust and do not expose it to a public one; access control
(firewall rules or the equivalent for your site) is yours to arrange,
and if the endpoint has to be reachable more widely, put authentication
and TLS in front of it. Ray needs separate treatment: its dashboard and
inter-node ports have to be isolated on their own, because
authentication in front of the API does not protect Ray. A long enough
prompt can also exhaust host memory on this hardware (see "Long
context, measured"), so an endpoint left open is worse here than the
usual.
2 DGX Spark nodes, TP=2. Summary of the repository's pipeline:
# on each node: pull the official image, then build the RDMA derivative
# (installs Ubuntu questing rdma-core over the noble base; self-verifies
# the MLX5_1.25 symbols the NCCL NET/IB plugin needs)
docker pull vllm/vllm-openai:glm53-flash-arm64-cu130
docker build -t vllm/vllm-openai:glm53-flash-arm64-cu130-nccl-ib \
- < docker/Dockerfile.nccl-ib
# serve.env filled per node, then:
serve/start-head.sh # Ray head + vllm serve
serve/start-worker.sh # joins the Ray cluster
The headline speed is a draft number — the + MTP rows above. If
you bring the server up from the companion repository's
serve/serve.env.example, set MTP_DIR there: the line is commented
out by default, so an untouched file starts the server with no draft.
Without the draft, the 64-prompt ruler reads 28.28 tok/s against 34.99
tok/s with the draft at K=2 — the no-draft figure sits alongside the
27.15 route-h no-draft row above. The draft directory is the one built
above with requant/build-mtp-draft.py.
The effective vllm serve invocation (inside the derived image, with the
kda-quant / mla-quant / mtp-bf16 overlays bind-mounted, and
/checkpoint-mtp built as described above):
vllm serve /checkpoint \
--served-model-name GLM-5.3-Flash-NVFP4-Wabi \
--host 0.0.0.0 --port 8000 \
--tensor-parallel-size 2 --data-parallel-size 1 \
--distributed-executor-backend ray \
--reasoning-parser glm45 \
--kernel-config '{"enable_flashinfer_autotune": false}' \
--kv-cache-dtype fp8 \
--model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 32}' \
--max-num-batched-tokens 8192 --enable-chunked-prefill \
--max-num-seqs 20 --max-model-len 204800 \
--gpu-memory-utilization 0.85 \
--enable-prefix-caching \
--limit-mm-per-prompt '{"image":0,"video":0}' \
--speculative-config '{"method":"mtp","model":"/checkpoint-mtp","num_speculative_tokens":2}'
NCCL_IB=0 in the repo's serve config falls back to the stock image over
sockets. Do not raise num_speculative_tokens past 2 for this checkpoint
at these memory settings (128 GB unified): every deeper K that was
actually measured came out slower. On route h, K=3 measured 31.98 tok/s
against K=2's 35.09; on the stock checkpoint K=4 gave 15.2 and K=5 gave
13.1 tok/s, below its own K=2 row — pair 2, sockets, expert parallel on,
64 prompts. Those two stock arms refused to start at the default
CUDA-graph capture sizes and booted only once the capture sizes were
narrowed to the ones the measurement needs. On route h, K=4 did boot
(READY 918 s) but the arm was stopped by the operator before it
measured, so no route-h number exists at K=4, and K=5 was never run on
route h at all. This K=2 comparator (17.71 tok/s) is the stock row
already in the table above; K=4 and K=5 stay here rather than in that
table because they are boot/speed notes on the stock checkpoint, not on
route h.
What is left for a v2
This is a list of what I know is unfinished, not a roadmap. I am not predicting how much any of it is worth; I will publish a number when I have measured one.
Speed, still open
- My own NVFP4 MoE kernel. It has never actually run in a served request: three attempts hooked a class the GB10 build does not instantiate, then the wrong class, then a host-side call that threw and silently disabled the overlay. Until it runs with a proof-of-life line in the engine log, there is nothing to measure.
- All-reduce. Cross-node peer wait was 20.8 ms of an ~87 ms decode step in the one step attribution I ran (eager, stock, single stream). RDMA cut the waiting a lot; what remains has not been re-attributed under RDMA, and I have not tried to reduce the number of collectives itself.
- A trained draft model. The checkpoint's own MTP head accepts 0.62 of the tokens it proposes. A draft trained on this model's own outputs could accept more, which shortens every cycle. My first attempt is data-starved — 594 training records, about 301k tokens against a ~3M-token recipe target, holdout per-slot hit 0.03-0.08 — and needs roughly ten times the data before it is worth measuring.
- Speculation depth. K=1 (34.28) and K=2 (35.09) are not separated by this experiment's run-to-run spread, and K=3 is slower. A better draft would change where that optimum sits.
Quality, not yet measured
- English and other languages. Every probe here is Japanese.
- Code correctness. Code prompts appear in the speed table and nowhere in the quality table.
- Multi-turn conversations, tool calling, instruction following, long-context quality — the needle probe reaches 194,544 tokens at 40/40, but that is retrieval and nothing in the quality table ran on a prompt longer than about 2k — safety behaviour, and the model with thinking enabled, which is how this family is normally used.
- Perplexity on a larger corpus, with a confidence interval. The current probe is eight sentences and the code does not keep the per-token values, so the ratio has no interval attached to it.
- A paired test on eval-200. The grader discards per-item results, so the same 150 items cannot be tested as pairs, which is what they are.
What is still missing from the reproduction path, and the list of things I have already measured as not working (FP8 dense linears, expert parallelism off, K of 3 or more, forced eager mode), are in the companion repository: its clean-room section and its failure table.
Versions
This is the v1 drop (git tag v1): the route-h checkpoint and the
measurements above. If you only need the weights, v1 is what you want.
A v2 would appear on this same repo only if a faster configuration — a
trained drafter, or a better K from a holdout sweep — clears the same
quality gate; nothing of it is published yet.
Config expectations
The requantization rewrites the quantization metadata for the modelopt
MIXED_PRECISION loader:
config.json:quantization_config.quant_algo=MIXED_PRECISION;quantization_config.producer.requant=requant.py (weight-only NVFP4 quantization);producer.requant_target=h;quantized_layersholds 369 modules, each withquant_algo+group_size= 16 — 318 atW4A16_NVFP4(what this pass converted) and 51 left at the stockNVFP4(the 42 routed expert blocks plus the 9 dense-MLP projections of layers 0-2).ignorehas 45 entries: the 42 per-layermlp.gaterouters,embed_tokens,model.visual*, andmodel.language_model.layers.45*(the MTP layer stays BF16 and must remain excluded).hf_quant_config.json:quantization.quant_algo=MIXED_PRECISION,kv_cache_quant_algo=FP8,group_size= 16, the same 369-entryquantized_layersand the matching 45-entryexclude_modules.model.safetensors.index.json:weight_maprepointed at the 18 regenerated shards and covering 148,659 tensors;metadata.total_size= 193,304,829,584.
The repo's requant/verify.py config <dir> statically gates all of this
(missing NVFP4 global scales show up as KeyError ...weight_scale_2 at
load — the gate catches them first).
Files in this repo
The weights ship as 18 safetensors shards, model-00001-of-00018.safetensors
through model-00018-of-00018.safetensors. Most of them are about
10.3 GiB; the set is 193,324,988,184 bytes on disk, holding 148,659
tensors.
| file | size | notes |
|---|---|---|
| the safetensors shards | see above | regenerated by requant.py; non-target tensors byte-copied from the base |
model.safetensors.index.json |
regenerated | weight_map + total_size for the new shards |
config.json |
regenerated | MIXED_PRECISION + producer.requant_target=h |
hf_quant_config.json |
regenerated | as above |
chat_template.jinja |
10,644 B | byte-identical to base @ 09b04e5e |
generation_config.json |
215 B | byte-identical |
processor_config.json |
909 B | byte-identical |
tokenizer.json |
20,217,442 B | byte-identical |
tokenizer_config.json |
761 B | byte-identical |
.quant_summary.txt |
616,642 B | carried from the base release — describes NVIDIA's stock quantization, not the route-h pass |
LICENSE |
1,070 B | MIT, upstream text (see below) |
NOTICE |
short | attribution for the base release and for this modification |
README.md |
this card | — |
.gitattributes |
short | written by the Hub for LFS tracking |
License and attribution
MIT. LICENSE carries the upstream MIT text (Copyright (c) 2026 Z.AI Co., Ltd) and NOTICE records the attribution chain. Model by Z.AI (zai-org/GLM-5.3-Flash); base NVFP4
quantization by NVIDIA with Model Optimizer v0.47.0; the route-h quantization
is the modification published here. Ready for commercial and
non-commercial use under the same terms as the base release.
Contributors
The design decisions, the acceptance calls and every measurement reported on this card are mine. The implementation was carried out with the AI seats below, each working to a brief I wrote and under my review.
- tenhkspark (the maintainer) — the one who worried, watched, and said go.
- Claude Fable 5.1 (Anthropic) — direction, experiment design, review and acceptance.
- Claude Opus 5 (Anthropic) — pre-release review.
- Astra (OpenAI) — adversarial review of the measurements and the plan.
- Devin SWE-2 (Cognition) — implementation, diagnostics, running the experiment queue, repository drafting.
- GLM-5.3 (Z.ai) — implementation seat.
- GLM-5.3-Flash (Z.ai) — implementation seat and summarisation.
Built and measured by the author of the companion repository
(glm53-flash-nvfp4-2node). Development tooling credit: Grok, Claude
Code, Codex.
Configuration
- Architecture
- Glm5NextForConditionalGeneration
- Context length (tokens)
- 1,048,576
- Layers
- 45
- Hidden size
- 4,096
- Feed-forward size
- 12,288
- Attention heads
- 64
- Key/value heads
- 64
- Head dimension
- 0
- Vocabulary size
- 154,880
- Routed experts
- 288
- Experts active per token
- 8
- Model type
- glm5_next
- Quantization
- modelopt
Identity and Version
- Repository
- tenhkspark/GLM-5.3-Flash-NVFP4-Wabi
- Publisher
- Tenhkspark
- Task
- Not stated by the source
- Modality
- Other
- Library
- Model Optimizer
- Parameters
- 165B parameters
- Languages
- Not stated by the source
- Revision
- 655e3a5ed2f52632adf87fcb1b4dda9ead7ae52b
- First published
- 2026-09-16
- Last updated
- 2026-09-18
Files and Weights
36 files, 193.4 GB in total. The weights are 18 files totalling 193.3 GB in safetensors.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| model-00001-of-00018.safetensors | Weights | 8.0 GB | 7fe1cc8391a4 |
| model-00002-of-00018.safetensors | Weights | 8.9 GB | 4e5d39d63ddd |
| model-00003-of-00018.safetensors | Weights | 8.9 GB | 68768e2527f1 |
| model-00004-of-00018.safetensors | Weights | 12.0 GB | 03b6742aa472 |
| model-00005-of-00018.safetensors | Weights | 11.1 GB | 6013c6f5e1bf |
| model-00006-of-00018.safetensors | Weights | 11.1 GB | 7c1d44cb8e14 |
| model-00007-of-00018.safetensors | Weights | 11.1 GB | bc26617fc680 |
| model-00008-of-00018.safetensors | Weights | 11.1 GB | 53f9b395ff22 |
| model-00009-of-00018.safetensors | Weights | 11.1 GB | c03d8ce9a4b9 |
| model-00010-of-00018.safetensors | Weights | 11.1 GB | 9b66df6eacfe |
| model-00011-of-00018.safetensors | Weights | 11.1 GB | d0391d8af9f0 |
| model-00012-of-00018.safetensors | Weights | 11.1 GB | ebeb7af44d10 |
| model-00013-of-00018.safetensors | Weights | 11.1 GB | 84654a5adb73 |
| model-00014-of-00018.safetensors | Weights | 11.1 GB | 7c2e1db4267f |
| model-00015-of-00018.safetensors | Weights | 11.1 GB | 73fef9387688 |
| model-00016-of-00018.safetensors | Weights | 11.2 GB | 60da0154a34c |
| model-00017-of-00018.safetensors | Weights | 11.1 GB | b9ce0048f58a |
| model-00018-of-00018.safetensors | Weights | 11.1 GB | 50d79969bd77 |
| config.json | Configuration | 58.8 KB | — |
| generation_config.json | Configuration | 215 B | — |
| hf_quant_config.json | Configuration | 51.9 KB | — |
| model.safetensors.index.json | Configuration | 16.4 MB | 3ce8dfe3216d |
| processor_config.json | Configuration | 909 B | — |
| LICENSE | Documentation | 1.1 KB | — |
| NOTICE | Documentation | 692 B | — |
| README.ja.md | Documentation | 39.6 KB | — |
| README.ko.md | Documentation | 38.1 KB | — |
| README.md | Documentation | 32.9 KB | — |
| README.zh.md | Documentation | 33.3 KB | — |
| SHA256SUMS | Other | 3.1 KB | — |
| chat_template.jinja | Other | 10.6 KB | — |
| files.tsv | Other | 1.2 KB | — |
| .gitattributes | Repository | 1.6 KB | — |
| .quant_summary.txt | Repository | 616.6 KB | — |
| tokenizer.json | Tokenizer | 20.2 MB | 19e773648cb4 |
| tokenizer_config.json | Tokenizer | 761 B | — |
License and Download
- License
- mit
- Access
- Open weights, no gate
- Download size
- 193.3 GB
Released by Tenhkspark through its official repository on Hugging Face. Read the license.
Built From
- Derived from nvidia/GLM-5.3-Flash-NVFP4
- Quantized from nvidia/GLM-5.3-Flash-NVFP4
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 193.3 GB |
| 16-bit | 330.1 GB |
| 8-bit | 165.0 GB |
| 4-bit | 82.5 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About GLM-5.3-Flash-NVFP4-Wabi
How much GPU memory does GLM-5.3-Flash-NVFP4-Wabi need?
About 396.1 GB at 16-bit and 99 GB at 4-bit: the weights (165B parameters) plus a working margin. A long context needs more.
What is the cheapest GPU to run GLM-5.3-Flash-NVFP4-Wabi on?
At 16-bit, 2x MI325X from $4.00 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.
Can I use GLM-5.3-Flash-NVFP4-Wabi commercially?
Yes. GLM-5.3-Flash-NVFP4-Wabi is released under MIT License. The MIT License is a short permissive license. It permits commercial use, modification and redistribution, provided the copyright notice and permission notice are included.
What is GLM-5.3-Flash-NVFP4-Wabi's context length?
1,048,576 tokens, from the maximum position embeddings in its published configuration.