SAVRN
Search Contact SAVRN

Open-weight model

GLM-5.3-Flash-NVFP4-Wabi

by Tenhkspark tenhkspark/GLM-5.3-Flash-NVFP4-Wabi

Versions below for what a v2 would have to clear. A derivative of the official nvidia/GLM-5.3-Flash-NVFP4 checkpoint in which the layers NVIDIA's release left in BF16 — the attention linears (KDA fused inproj/out projections, MLA q/kv/o, indexer wqb), the…

Parameters165B
Context1,048,576
Weights193.3 GB
Licensemit
AccessOpen weights
Monthly Downloads47

Runs On

What it takes to serve GLM-5.3-Flash-NVFP4-Wabi (165B parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

PrecisionWeightsMemory neededCheapest setupPer hourAlso fits
16-bit 330.1 GB 396.1 GB 2x MI325X (256 GB)
Vultr
$4.00 2x MI355X $5.18 · 3x MI300X $5.55
8-bit 165.0 GB 198.0 GB 1x MI325X (256 GB)
Vultr
$2.00 1x MI355X $2.59 · 2x MI300X $3.70
4-bit 82.5 GB 99.0 GB 1x MI300X (192 GB)
Vultr
$1.85 1x MI325X $2.00 · 1x MI355X $2.59

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.

Model Card

By Tenhkspark, published under mit, revision 655e3a5ed2f5.

Versions below for what a v2 would have to clear. A derivative of the official nvidia/GLM-5.3-Flash-NVFP4 checkpoint in which the layers NVIDIA's release left in BF16 — the attention linears (KDA fused inproj/out projections, MLA q/kv/o, indexer wqb), the shared experts (gate/up/down), and lmhead — are quantized to W4A16 NVFP4 (weight-only 4-bit, group size 16). Routers, norms, embeddings, the vision tower, and the MTP layer (layers.45) stay BF16; the routed experts keep the stock W4A4 NVFP4 quantization. enjoying the incomplete This checkpoint does not boot on the stock vLLM image. It needs the kda-quant and mla-quant source overlays from the companion repository (plus glm5next-mtp-bf16…

Read Tenhkspark's full model card

GLM-5.3-Flash-NVFP4-Wabi — route h W4A16 dense-side quantization

English · 日本語 · 简体中文 · 한국어

Current tag: v1 — the only version published so far. See Versions below for what a v2 would have to clear.

A derivative of the official nvidia/GLM-5.3-Flash-NVFP4 checkpoint in which the layers NVIDIA's release left in BF16 — the attention linears (KDA fused in_proj/out projections, MLA q/kv/o, indexer wq_b), the shared experts (gate/up/down), and lm_head — are quantized to W4A16 NVFP4 (weight-only 4-bit, group size 16). Routers, norms, embeddings, the vision tower, and the MTP layer (layers.45) stay BF16; the routed experts keep the stock W4A4 NVFP4 quantization. enjoying the incomplete

Read this before you download

This checkpoint does not boot on the stock vLLM image. It needs the kda-quant and mla-quant source overlays from the companion repository (plus glm5next-mtp-bf16 when you serve with speculation), bind-mounted into the serving container as documented there. Served as a drop-in nvidia/GLM-5.3-Flash-NVFP4 replacement it fails to load.

Those overlays are my own local patches for my own two-node machine. I have not proposed them upstream and they are not on anyone's upstream roadmap, so please treat them as something you patch in yourself, not as a feature arriving in a future vLLM release.

Purpose: single-stream decode speed on 2 NVIDIA DGX Spark nodes (GB10, TP=2 over Ray, NCCL over RDMA). On my fixed ruler — 64 Japanese prose prompts, max_tokens=512, temperature=0, thinking off, C=1 — this checkpoint plus the model's own MTP draft at K=2 reaches 35.09 tok/s on node pair 1, expert parallel on, NCCL over RDMA. The same checkpoint, pair, K and EP over TCP sockets reads 24.01 tok/s on the same ruler. Every other number, with its pair / transport / prompt count / K / EP, is in the table below; I do not quote a ratio between rows whose conditions differ.

Several improvement ideas are still on my list, but the model has reached a level I find usable, so I am releasing it now. It is a work in progress: please take it and move it toward your own idea of complete. I will keep working and publish updates as results come in, and I would be glad to see your versions on Hugging Face and GitHub too.

About the name

Wabi (侘び) is the Japanese sense of accepting what is imperfect or plain and finding richness in it; this release is a work in progress that I publish as it is, improvements included when they are measured.

Base checkpoint

  • nvidia/GLM-5.3-Flash-NVFP4 at revision 09b04e5e74bca08ca8549fc736d4cdd8624bfde3 (pinned).
  • Upstream base model: zai-org/GLM-5.3-Flash (MIT).

How it was made

Produced by requant/requant.py --target h in the companion repository: https://github.com/tenhkspark/glm53-flash-nvfp4-2node. The script reads the official checkpoint read-only, rewrites the target tensors into fresh safetensors shards, and rewrites config.json / hf_quant_config.json / model.safetensors.index.json for the modelopt MIXED_PRECISION loader. Non-target tensors are copied through unchanged.

The MTP draft checkpoint

The --speculative-config in the serving command below points at /checkpoint-mtp, a separate BF16 draft directory. It is not part of this repository — you build it, and the companion repository builds it for you in STEP 8 (build-mtp-draft) of scripts/agent-run.sh, which runs:

python3 requant/build-mtp-draft.py \
  --target /path/to/GLM-5.3-Flash-NVFP4 \
  --out /path/to/checkpoint-mtp

The MTP layer is layers.45, which the NVIDIA release already stores in BF16 and which this repository carries through unchanged. The script copies those tensors as-is, adds the two keys the vLLM draft does not share with the target (embed_tokens and shared_head.head, sourced from embed_tokens and lm_head), and writes a flattened text-only config.json with the quantization section dropped. Point --target at the stock nvidia/GLM-5.3-Flash-NVFP4 directory rather than at this one: route h quantizes lm_head to NVFP4 and the draft has to be BF16. The output is roughly 16 GiB and each node needs its own copy.

Quality delta (gate v2, measured)

I evaluated this checkpoint against a stock baseline capture using the same serving stack on 2026-09-16:

criterion threshold stock route h result
degenerate outputs over the 64-prompt ruler 0 0 0 PASS
perplexity ratio on the inline Japanese probe <= 1.10 1.0 1.051 PASS
eval-200 live accuracy (150 items, the 0/50 tool floor excluded) no more than 0.02 below stock 0.447 0.453 (+0.0067) PASS
TTFT on the same probe text within 1.2x of stock 2.380 s 2.436 s (x1.02) PASS

Perplexity. I measure it on eight short Japanese sentences (249 characters in total) written for this test and defined inline in requant/verify.py; they are not drawn from any public corpus and I claim no training-set exclusion. Each sentence goes to the served endpoint as a prefill-only request (/v1/completions with echo=true, max_tokens=1, prompt_logprobs=1, temperature 0); the log-probabilities of the actual prompt tokens are accumulated across all eight sentences, and the mean negative log-likelihood over that single pooled token stream is exponentiated into one number. Stock and candidate are measured by the identical function against the same served model: route h scored 12.543 against 11.938 for stock, a ratio of 1.051. The sample is small, so this checks for gross degradation, not for parity. The TTFT gate reuses the same eight sentences, repeated twelve times, as its probe input.

eval-200. A fixed set of 200 Japanese single-turn prompts shipped in the companion repository (bench/eval-200.jsonl): 50 each of reason, trap, tool and longread, mixed easy/medium/hard. Every item is answered greedily on the served endpoint (temperature 0, max_tokens 384, thinking skipped via an empty assistant continuation) and graded by a deterministic rule in requant/verify.py: whitespace-insensitive exact match when the item's rubric demands the answer alone, substring containment otherwise; trap items must carry a refusal and no digits, tool items must name the required functions in order plus the expected final value. The tool column is 0/50 on every checkpoint by construction -- the prompts never name a callable function and no tool schema is sent -- so the gate scores the remaining 150 live items and requires the candidate to stay within 0.02 of stock with zero request errors (0.447 stock -> 0.453 route h). The raw totals 67/200 -> 68/200 are reported for transparency only; they include that permanent zero column and are not the pass/fail criterion.

Accuracy by category (50 items each):

kind stock route h
reason 32/50 27/50
trap 13/50 16/50
tool 0/50 0/50
longread 22/50 25/50

The category moves are not independent: the trap grader rewards refusal, so a more hedging model scores higher there and lower on reason. A net +1 total can be produced by a directional degradation.

The perplexity ratio of 1.051 is a 5.1% increase over the stock model: inside the declared gate threshold, but an increase rather than strict parity. Eval-200 moved by +0.0067 on the 150 live items the gate scores. A 150-item set resolves accuracy to roughly +-0.08 at 95% confidence (paired, assuming ~20% of items flip), so the +0.0067 I measured is indistinguishable from zero — and so would be a true regression of 0.05. The 0.02 gate threshold is finer than what this set can resolve: the +-0.08 half-width is 4x the threshold. Treating that half-width as a normal sampling error (sigma = 0.08 / 1.96, about 0.041) and the gate's accuracy condition as 'observed difference at least -0.02', a checkpoint genuinely 0.06 worse than stock still clears that condition about 16% of the time, and one genuinely 0.08 worse about 7%. Those 0.06 and 0.08 are percentage-point differences in accuracy, not relative drops, and the two probabilities are for the accuracy condition alone, not for the composite gate. Reading this gate as 'no regression' is wrong; it only rules out a large one.

Every gate and eval-200 row above is Japanese, single-turn and thinking off. The one evaluation that would cover long context, parallel requests and agentic tool use — the 13-item end-to-end serve suite — did run, on 2026-09-16, on both node pairs, but 4 of its 13 items returned nothing in either run: agentic tool success, agentic final answer, integrated task en 108K and integrated task ja 108K. So agentic tool use and 108K long context are still unmeasured on this checkpoint — not because the suite is waiting to run, but because the items that cover them produced no result when it did. The two runs are also not a like-for-like route h vs stock comparison: pair 1 ran with no speculation and pair 2 ran with speculative decoding at 78.0% acceptance, so even the items that did return cannot be read as a difference between the two checkpoints.

What this gate does not measure. Every number above is Japanese, single-turn, thinking off, greedy, under 2k tokens of context and one request at a time. I have no measurement of this checkpoint on English or any other language, on code correctness, on multi-turn conversations, on tool calling (the tool column is a structural zero), on instruction following, on long-context quality — the needle probe in "Long context, measured" scores 40/40 up to 194,544 tokens, but that is retrieval, and every quality probe above fits in about 2k — on safety behaviour, or with thinking enabled — which is how this model family is normally used. The requant rewrites the dense linears and lm_head, so those are exactly the places a regression could hide from these four probes. If you depend on any of them, measure it yourself before adopting this checkpoint; requant/verify.py takes a different prompt set with one flag.

For reference, the base release's own BF16-vs-NVFP4 card numbers (quoted from the base card at the pinned revision): GPQA Diamond 0.9217 -> 0.9211, SciCode 0.5621 -> 0.5769, MMMU Pro 0.7688 -> 0.7630, AA-LCR 0.7100 -> 0.7106, IFBench 0.6130 -> 0.6054, Terminal-Bench 2.1 0.8258 -> 0.8315. Those card figures come from standard public benchmarks under NVIDIA's own harness and are not comparable with the eval-200 numbers above, which are scored by this repository's own strict grader on Japanese prompts. Nothing here places my checkpoint on the same axis as the base card's benchmarks.

Speed (measured, with conditions)

Ruler: 64 Japanese prose prompts, temperature=0, max_tokens=512, thinking skipped via an empty assistant continuation. C=1 streams the prompts sequentially; tok/s = completion tokens / whole-window wall time including TTFT. Hardware: 2x DGX Spark (GB10, 128 GB unified memory each), TP=2 over Ray, RoCE link (ConnectX-7 class) between the nodes. Serve flags in the repo's scripts: --max-model-len 204800, --max-num-seqs 20, --gpu-memory-utilization 0.85, --enable-prefix-caching, FP8 KV cache and RAY_memory_usage_threshold=0.99 on both nodes. The rows in the table ran the measurement rig instead — almost all of them at --max-model-len 16384 with --max-num-seqs 20 and --gpu-memory-utilization 0.86 and --enable-expert-parallel, except the 19.05 stock + MTP row which ran with expert parallel off (ep = off in the table), the pair-2 stock baseline at --max-model-len 131072 with 32 slots, and the pair-2 stock RDMA row at 16384 with 32 slots. Image: vllm/vllm-openai:glm53-flash-arm64-cu130 (+ derived RDMA image).

The shipped window is longer than the rig one on purpose. Re-measured on 2026-09-17 at the shipped window, with expert parallel on as everywhere else in this table, the ruler reads 34.78 tok/s — TPOT median 28.1 ms, TTFT median 0.313 s, 0 of the 64 requests failed — against 35.09 at 16384 / 20 / 0.86. That is a 0.9% difference across a 12.5x longer window, inside the run-to-run spread of about 2% on identical configurations.

The serve scripts in the recipe repo do not enable expert parallel, while every row in this table was measured with it on. Run exactly as they ship, the same ruler reads 37.33 tok/s (36.95 on a repeat); adding the two EP flags back drops it to 35.05, reproducing the 34.78 row to within 0.8%. EP is also hungrier on this hardware, so the scripts keep it off. One operational note that matters more than either number: GB10 is unified memory, the engine sizes its budget from whatever was free when it profiled, and the same script has reached READY with anywhere from 5470 to 10270 MiB of host memory left. About 4.7 GiB of that is spent after READY, the first time each larger prefill shape arrives, so a boot that starts too low is killed later by one long prompt — and DGX OS ships no OOM daemon, which makes that a hung node rather than a killed process. The recipe repo ships serve/check-headroom.sh to gate this at boot; run it before you send the first request. The longer window does not cost concurrency either: 20 sequence slots do come up at 204800 (909 s to READY, then the same 0-failure ruler), which they did not at 307200. Utilization is the one flag that moved down rather than up — 0.89 has been refused at boot on this pair, and at 0.88 the head node ran with about 2.1% of host RAM free — close enough to exhaustion that it does not matter what reaps the worker first — so the scripts ship 0.85 and leave 8.6%. The requant is part of why the window fits: at 0.88 the stock checkpoint tops out at 156672 tokens (vLLM prints that ceiling when it refuses to start), while route h boots at 204800.

checkpoint pair transport K ep tok/s TPOT med ms weighted TPOT ms TTFT med s accept quality gate TTFT gate
stock 2 sockets - on 10.75 † 95.3 † 0.361 † - ref
stock + MTP 2 sockets 2 on 17.71 55.5 55.73 0.447 0.6128 ref
stock + MTP 1 sockets 2 off 19.05 51.9 51.77 0.430 0.6175 ref
stock 2 NCCL/IB (RDMA) - on 14.32 † 69.3 † 69.29 0.349 † - ref
stock 1 NCCL/IB (RDMA) - on 14.39 69.0 0.293 - ref
stock + MTP 2 NCCL/IB (RDMA) 2 on 23.93 41.0 41.17 0.366 0.6137 ref
route h 1 sockets - on 18.98 51.5 52.12 0.330 - PASS (PPL x1.051) verified on sockets
route h 1 NCCL/IB (RDMA) - on 27.15 35.7 36.4 0.269 - PASS (PPL x1.051) verified on sockets; not re-run under RDMA
route h + MTP 1 sockets 2 on 24.01 40.7 40.96 0.389 0.6221 PASS (PPL x1.051) verified on sockets
route h + MTP 1 NCCL/IB (RDMA) 2 on 35.09 27.9 27.92 0.320 0.6221 PASS (PPL x1.051) verified on sockets; not re-run under RDMA
route g + MTP 2 NCCL/IB (RDMA) 2 on 29.46 33.3 33.35 0.340 0.6182 PASS (PPL x0.999) verified on sockets; not re-run under RDMA
route h + MTP 1 NCCL/IB (RDMA) 1 on 34.28 28.6 0.304 0.7979 PASS (PPL x1.051) verified on sockets; not re-run under RDMA
route h + MTP 1 NCCL/IB (RDMA) 3 on 31.98 30.8 0.331 0.483 PASS (PPL x1.051) verified on sockets; not re-run under RDMA
route h + MTP 2 NCCL/IB (RDMA) 2 on 34.47 28.2 28.43 0.313 0.6221 PASS (PPL x1.051) verified on sockets; not re-run under RDMA
route h + MTP 1 NCCL/IB (RDMA) 2 on 34.48 28.3 28.4 0.317 0.6193 PASS (PPL x1.051) verified on sockets; not re-run under RDMA
route h + MTP 1 sockets 2 on 23.75 41.2 41.4 0.394 0.6213 PASS (PPL x1.051) verified on sockets
route h + MTP 1 NCCL/IB (RDMA) 2 on 35.12 27.9 27.9 0.313 0.6193 PASS (PPL x1.051) verified on sockets; not re-run under RDMA
route h + MTP 1 sockets 2 on 24.35 40.4 40.4 0.39 0.6174 PASS (PPL x1.051) verified on sockets

† these rows ran C=1 over the first 8 prompts of the ruler, not all 64.

The ep column records --enable-expert-parallel state per row — the 19.05 stock + MTP row ran with it off. The gate is split per row: quality gate covers the checkpoint-level legs (degenerate / PPL / eval-200); TTFT gate is transport-dependent and was probed once on the route-h sockets serving, so RDMA rows read "verified on sockets; not re-run under RDMA" rather than a four-condition PASS on that transport. The weighted TPOT ms column is recorded directly in five shipped JSONs (the four release re-runs plus the route-h no-draft RDMA row); for nine more rows I computed the same quantity myself from each run's per-prompt records in the pre-sanitisation factory log (not part of this repository) and added the result to the row's shipped results/logs/ entry. The 2026-09-13 stock baseline has no per-prompt records anywhere I could find, so that row's cell stays empty rather than filled with its own plain-mean figure, which is a different statistic. Per-row measurement excerpts ship in the repo under results/logs/ (see the source_log column of results/results.tsv).

K sweep, what is and is not settled: on route h + RDMA, pair 1, K=1 measured 34.28 tok/s and K=3 measured 31.98, both already in the table above; K=2 remains the best of the values measured, not a proven optimum. The re-measurement question is answered: the same config on the second node pair read 34.47 tok/s, and an interleaved TCP/RDMA re-run on a fresh, non-overlapping 64-prompt set with prefix caching disabled read 34.48 and 35.12 tok/s over RDMA and 23.75 and 24.35 over TCP — within ~2% of the original 35.09 on the RDMA side. The headline run: 32407 completion tokens over 923.43 s of wall, 0 failed requests. C=32 aggregate on stock was 95.08 tok/s (sockets) and 109.03 tok/s (RDMA); both of those passes ran with 32 sequence slots, which the shipped --max-num-seqs 20 cannot reproduce, so read them as rig numbers rather than as what the released scripts do. No C=32 row exists for route h yet.

Long context, measured

The probe is the companion repo's bench/longctx.py: Japanese filler grown to a target length, ten facts planted at head, middle and tail token depths, one question each, temperature=0, graded by exact code match rather than by a judge. The set is bench/longctx-probe.jsonl, and every document is the shared prefix of its ten questions, so one long prefill is paid per length. One pass on the shipped configuration — --max-model-len 204800, --max-num-seqs 20, --gpu-memory-utilization 0.85, --enable-prefix-caching, route h with MTP at K=2 over RDMA:

prompt tokens needles found first token, cold first token, cache warm prefill
16,345 10/10 10.02 s 4.27 s 1630.6 tok/s
65,545 10/10 39.95 s 2.91 s 1640.8 tok/s
130,990 10/10 79.86 s 3.15 s 1640.2 tok/s
194,544 10/10 120.04 s 4.74 s 1620.6 tok/s

40 of 40, no depth weaker than another: head 12/12, middle 16/16, tail 12/12. One reading from a different configuration, kept out of the table on purpose: at --max-model-len 307200 with utilization at 0.88 — not what the scripts ship — a single 204,767-token document answered 10/10. It needs a window the released scripts do not open, so it is not a row above.

This is retrieval, not quality. It says the model still finds a planted fact at 194,544 tokens; it says nothing about whether its prose or its reasoning hold up there.

Three limits come with the long window.

  • A prompt the size of the window does not fit. The probe's top stage aims at 190k rather than 200k for a measured reason: a 204,754-token document plus max_tokens=64 is 204,818 against a 204,800 limit, and the server returns HTTP 400. The engine survives — the next request is answered normally — but that document cannot be asked about. The usable ceiling is the window minus the generation budget minus the chat template.
  • The first token on a long prompt is slow. Prefill holds between 1620 and 1641 tok/s across all four lengths, so the 194,544-token prompt takes 120.04 s before the first token appears. Nothing is stuck; that is prefill. --enable-prefix-caching is in the shipped flags for this reason: pin one long document as the prefix and vary only the question, and every request after the first is an order of magnitude faster — at 130,990 tokens, 79.86 s cold against 3.15 s warm.
  • Past about 259,000 tokens nothing gets through on this hardware. Not with a longer --max-model-len, not with a lower --gpu-memory-utilization, not with a smaller --max-num-batched-tokens: host memory runs out and the node's OOM reaper takes the vLLM worker. The peak activation figure vLLM prints at boot is measured on an 8192-token dummy run, and the scratch space the sparse-attention indexer wants during a long prefill sits outside the --gpu-memory-utilization budget entirely. Dropping utilization from 0.88 to 0.85 moved the death from 62 s into the prefill to 155 s; it did not make the prompt fit.

RAY_memory_usage_threshold=0.99 is exported on both nodes by the repo's serve/start-head.sh and serve/start-worker.sh, and it is not optional. Unified memory means --gpu-memory-utilization is taken out of host RAM, so Ray's own OOM monitor sees the node above its default 0.95 and kills the largest actor it can find, which is the vLLM TP0 worker. What you see is EngineDeadError — at boot, or in the middle of a request — with nothing wrong in the vLLM log; the kill is in the raylet log. The two scripts bring up two separate raylets, each with its own monitor, so the variable has to be exported on both.

Serving

The served API has no authentication, and it listens on every interface. The scripts below pass --host 0.0.0.0, so anything that can reach port 8000 on either node can send requests. Keep this on a network you trust and do not expose it to a public one; access control (firewall rules or the equivalent for your site) is yours to arrange, and if the endpoint has to be reachable more widely, put authentication and TLS in front of it. Ray needs separate treatment: its dashboard and inter-node ports have to be isolated on their own, because authentication in front of the API does not protect Ray. A long enough prompt can also exhaust host memory on this hardware (see "Long context, measured"), so an endpoint left open is worse here than the usual.

2 DGX Spark nodes, TP=2. Summary of the repository's pipeline:

# on each node: pull the official image, then build the RDMA derivative
# (installs Ubuntu questing rdma-core over the noble base; self-verifies
# the MLX5_1.25 symbols the NCCL NET/IB plugin needs)
docker pull vllm/vllm-openai:glm53-flash-arm64-cu130
docker build -t vllm/vllm-openai:glm53-flash-arm64-cu130-nccl-ib \
    - < docker/Dockerfile.nccl-ib

# serve.env filled per node, then:
serve/start-head.sh     # Ray head + vllm serve
serve/start-worker.sh   # joins the Ray cluster

The headline speed is a draft number — the + MTP rows above. If you bring the server up from the companion repository's serve/serve.env.example, set MTP_DIR there: the line is commented out by default, so an untouched file starts the server with no draft. Without the draft, the 64-prompt ruler reads 28.28 tok/s against 34.99 tok/s with the draft at K=2 — the no-draft figure sits alongside the 27.15 route-h no-draft row above. The draft directory is the one built above with requant/build-mtp-draft.py.

The effective vllm serve invocation (inside the derived image, with the kda-quant / mla-quant / mtp-bf16 overlays bind-mounted, and /checkpoint-mtp built as described above):

vllm serve /checkpoint \
  --served-model-name GLM-5.3-Flash-NVFP4-Wabi \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 2 --data-parallel-size 1 \
  --distributed-executor-backend ray \
  --reasoning-parser glm45 \
  --kernel-config '{"enable_flashinfer_autotune": false}' \
  --kv-cache-dtype fp8 \
  --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 32}' \
  --max-num-batched-tokens 8192 --enable-chunked-prefill \
  --max-num-seqs 20 --max-model-len 204800 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching \
  --limit-mm-per-prompt '{"image":0,"video":0}' \
  --speculative-config '{"method":"mtp","model":"/checkpoint-mtp","num_speculative_tokens":2}'

NCCL_IB=0 in the repo's serve config falls back to the stock image over sockets. Do not raise num_speculative_tokens past 2 for this checkpoint at these memory settings (128 GB unified): every deeper K that was actually measured came out slower. On route h, K=3 measured 31.98 tok/s against K=2's 35.09; on the stock checkpoint K=4 gave 15.2 and K=5 gave 13.1 tok/s, below its own K=2 row — pair 2, sockets, expert parallel on, 64 prompts. Those two stock arms refused to start at the default CUDA-graph capture sizes and booted only once the capture sizes were narrowed to the ones the measurement needs. On route h, K=4 did boot (READY 918 s) but the arm was stopped by the operator before it measured, so no route-h number exists at K=4, and K=5 was never run on route h at all. This K=2 comparator (17.71 tok/s) is the stock row already in the table above; K=4 and K=5 stay here rather than in that table because they are boot/speed notes on the stock checkpoint, not on route h.

What is left for a v2

This is a list of what I know is unfinished, not a roadmap. I am not predicting how much any of it is worth; I will publish a number when I have measured one.

Speed, still open

  • My own NVFP4 MoE kernel. It has never actually run in a served request: three attempts hooked a class the GB10 build does not instantiate, then the wrong class, then a host-side call that threw and silently disabled the overlay. Until it runs with a proof-of-life line in the engine log, there is nothing to measure.
  • All-reduce. Cross-node peer wait was 20.8 ms of an ~87 ms decode step in the one step attribution I ran (eager, stock, single stream). RDMA cut the waiting a lot; what remains has not been re-attributed under RDMA, and I have not tried to reduce the number of collectives itself.
  • A trained draft model. The checkpoint's own MTP head accepts 0.62 of the tokens it proposes. A draft trained on this model's own outputs could accept more, which shortens every cycle. My first attempt is data-starved — 594 training records, about 301k tokens against a ~3M-token recipe target, holdout per-slot hit 0.03-0.08 — and needs roughly ten times the data before it is worth measuring.
  • Speculation depth. K=1 (34.28) and K=2 (35.09) are not separated by this experiment's run-to-run spread, and K=3 is slower. A better draft would change where that optimum sits.

Quality, not yet measured

  • English and other languages. Every probe here is Japanese.
  • Code correctness. Code prompts appear in the speed table and nowhere in the quality table.
  • Multi-turn conversations, tool calling, instruction following, long-context quality — the needle probe reaches 194,544 tokens at 40/40, but that is retrieval and nothing in the quality table ran on a prompt longer than about 2k — safety behaviour, and the model with thinking enabled, which is how this family is normally used.
  • Perplexity on a larger corpus, with a confidence interval. The current probe is eight sentences and the code does not keep the per-token values, so the ratio has no interval attached to it.
  • A paired test on eval-200. The grader discards per-item results, so the same 150 items cannot be tested as pairs, which is what they are.

What is still missing from the reproduction path, and the list of things I have already measured as not working (FP8 dense linears, expert parallelism off, K of 3 or more, forced eager mode), are in the companion repository: its clean-room section and its failure table.

Versions

This is the v1 drop (git tag v1): the route-h checkpoint and the measurements above. If you only need the weights, v1 is what you want. A v2 would appear on this same repo only if a faster configuration — a trained drafter, or a better K from a holdout sweep — clears the same quality gate; nothing of it is published yet.

Config expectations

The requantization rewrites the quantization metadata for the modelopt MIXED_PRECISION loader:

  • config.json: quantization_config.quant_algo = MIXED_PRECISION; quantization_config.producer.requant = requant.py (weight-only NVFP4 quantization); producer.requant_target = h; quantized_layers holds 369 modules, each with quant_algo + group_size = 16 — 318 at W4A16_NVFP4 (what this pass converted) and 51 left at the stock NVFP4 (the 42 routed expert blocks plus the 9 dense-MLP projections of layers 0-2). ignore has 45 entries: the 42 per-layer mlp.gate routers, embed_tokens, model.visual*, and model.language_model.layers.45* (the MTP layer stays BF16 and must remain excluded).
  • hf_quant_config.json: quantization.quant_algo = MIXED_PRECISION, kv_cache_quant_algo = FP8, group_size = 16, the same 369-entry quantized_layers and the matching 45-entry exclude_modules.
  • model.safetensors.index.json: weight_map repointed at the 18 regenerated shards and covering 148,659 tensors; metadata.total_size = 193,304,829,584.

The repo's requant/verify.py config <dir> statically gates all of this (missing NVFP4 global scales show up as KeyError ...weight_scale_2 at load — the gate catches them first).

Files in this repo

The weights ship as 18 safetensors shards, model-00001-of-00018.safetensors through model-00018-of-00018.safetensors. Most of them are about 10.3 GiB; the set is 193,324,988,184 bytes on disk, holding 148,659 tensors.

file size notes
the safetensors shards see above regenerated by requant.py; non-target tensors byte-copied from the base
model.safetensors.index.json regenerated weight_map + total_size for the new shards
config.json regenerated MIXED_PRECISION + producer.requant_target=h
hf_quant_config.json regenerated as above
chat_template.jinja 10,644 B byte-identical to base @ 09b04e5e
generation_config.json 215 B byte-identical
processor_config.json 909 B byte-identical
tokenizer.json 20,217,442 B byte-identical
tokenizer_config.json 761 B byte-identical
.quant_summary.txt 616,642 B carried from the base release — describes NVIDIA's stock quantization, not the route-h pass
LICENSE 1,070 B MIT, upstream text (see below)
NOTICE short attribution for the base release and for this modification
README.md this card
.gitattributes short written by the Hub for LFS tracking

License and attribution

MIT. LICENSE carries the upstream MIT text (Copyright (c) 2026 Z.AI Co., Ltd) and NOTICE records the attribution chain. Model by Z.AI (zai-org/GLM-5.3-Flash); base NVFP4 quantization by NVIDIA with Model Optimizer v0.47.0; the route-h quantization is the modification published here. Ready for commercial and non-commercial use under the same terms as the base release.

Contributors

The design decisions, the acceptance calls and every measurement reported on this card are mine. The implementation was carried out with the AI seats below, each working to a brief I wrote and under my review.

  • tenhkspark (the maintainer) — the one who worried, watched, and said go.
  • Claude Fable 5.1 (Anthropic) — direction, experiment design, review and acceptance.
  • Claude Opus 5 (Anthropic) — pre-release review.
  • Astra (OpenAI) — adversarial review of the measurements and the plan.
  • Devin SWE-2 (Cognition) — implementation, diagnostics, running the experiment queue, repository drafting.
  • GLM-5.3 (Z.ai) — implementation seat.
  • GLM-5.3-Flash (Z.ai) — implementation seat and summarisation.

Built and measured by the author of the companion repository (glm53-flash-nvfp4-2node). Development tooling credit: Grok, Claude Code, Codex.

Configuration

Architecture
Glm5NextForConditionalGeneration
Context length (tokens)
1,048,576
Layers
45
Hidden size
4,096
Feed-forward size
12,288
Attention heads
64
Key/value heads
64
Head dimension
0
Vocabulary size
154,880
Routed experts
288
Experts active per token
8
Model type
glm5_next
Quantization
modelopt

Identity and Version

Repository
tenhkspark/GLM-5.3-Flash-NVFP4-Wabi
Publisher
Tenhkspark
Task
Not stated by the source
Modality
Other
Library
Model Optimizer
Parameters
165B parameters
Languages
Not stated by the source
Revision
655e3a5ed2f52632adf87fcb1b4dda9ead7ae52b
First published
2026-09-16
Last updated
2026-09-18

Files and Weights

36 files, 193.4 GB in total. The weights are 18 files totalling 193.3 GB in safetensors.

Weights18 files · 193.3 GB
Configuration5 files · 16.5 MB
Tokenizer2 files · 20.2 MB
Documentation6 files · 145.7 KB
Other3 files · 14.9 KB
Repository2 files · 618.3 KB
Every file
FileTypeSizeSHA-256
model-00001-of-00018.safetensorsWeights8.0 GB 7fe1cc8391a4
model-00002-of-00018.safetensorsWeights8.9 GB 4e5d39d63ddd
model-00003-of-00018.safetensorsWeights8.9 GB 68768e2527f1
model-00004-of-00018.safetensorsWeights12.0 GB 03b6742aa472
model-00005-of-00018.safetensorsWeights11.1 GB 6013c6f5e1bf
model-00006-of-00018.safetensorsWeights11.1 GB 7c1d44cb8e14
model-00007-of-00018.safetensorsWeights11.1 GB bc26617fc680
model-00008-of-00018.safetensorsWeights11.1 GB 53f9b395ff22
model-00009-of-00018.safetensorsWeights11.1 GB c03d8ce9a4b9
model-00010-of-00018.safetensorsWeights11.1 GB 9b66df6eacfe
model-00011-of-00018.safetensorsWeights11.1 GB d0391d8af9f0
model-00012-of-00018.safetensorsWeights11.1 GB ebeb7af44d10
model-00013-of-00018.safetensorsWeights11.1 GB 84654a5adb73
model-00014-of-00018.safetensorsWeights11.1 GB 7c2e1db4267f
model-00015-of-00018.safetensorsWeights11.1 GB 73fef9387688
model-00016-of-00018.safetensorsWeights11.2 GB 60da0154a34c
model-00017-of-00018.safetensorsWeights11.1 GB b9ce0048f58a
model-00018-of-00018.safetensorsWeights11.1 GB 50d79969bd77
config.jsonConfiguration58.8 KB
generation_config.jsonConfiguration215 B
hf_quant_config.jsonConfiguration51.9 KB
model.safetensors.index.jsonConfiguration16.4 MB 3ce8dfe3216d
processor_config.jsonConfiguration909 B
LICENSEDocumentation1.1 KB
NOTICEDocumentation692 B
README.ja.mdDocumentation39.6 KB
README.ko.mdDocumentation38.1 KB
README.mdDocumentation32.9 KB
README.zh.mdDocumentation33.3 KB
SHA256SUMSOther3.1 KB
chat_template.jinjaOther10.6 KB
files.tsvOther1.2 KB
.gitattributesRepository1.6 KB
.quant_summary.txtRepository616.6 KB
tokenizer.jsonTokenizer20.2 MB 19e773648cb4
tokenizer_config.jsonTokenizer761 B

License and Download

License
mit
Access
Open weights, no gate
Download size
193.3 GB
Download from Tenhkspark

Released by Tenhkspark through its official repository on Hugging Face. Read the license.

Built From

  • Derived from nvidia/GLM-5.3-Flash-NVFP4
  • Quantized from nvidia/GLM-5.3-Flash-NVFP4

Memory Requirements

PrecisionWeights in memory
As published193.3 GB
16-bit330.1 GB
8-bit165.0 GB
4-bit82.5 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About GLM-5.3-Flash-NVFP4-Wabi

How much GPU memory does GLM-5.3-Flash-NVFP4-Wabi need?

About 396.1 GB at 16-bit and 99 GB at 4-bit: the weights (165B parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run GLM-5.3-Flash-NVFP4-Wabi on?

At 16-bit, 2x MI325X from $4.00 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use GLM-5.3-Flash-NVFP4-Wabi commercially?

Yes. GLM-5.3-Flash-NVFP4-Wabi is released under MIT License. The MIT License is a short permissive license. It permits commercial use, modification and redistribution, provided the copyright notice and permission notice are included.

What is GLM-5.3-Flash-NVFP4-Wabi's context length?

1,048,576 tokens, from the maximum position embeddings in its published configuration.