GLM5.3-Flash-E192-DGX-Spark

autotrust/GLM5.3-Flash-E192-DGX-Spark is a compact build of zai-org/GLM-5.3-Flash for desktop Blackwell systems like NVIDIA DGX Spark. It's an unofficial derivative.

It keeps 192 of the 288 routed experts in each layer by Neural Architecture Search (NAS), uses NVFP4 for the experts and still activates 18 B parameters per token. The weights take 124 GiB, the smallest in the family: two DGX Sparks connected by ConnectX-7 get about 62 GiB of weights per node and the most KV room of any build, and a single 180 GB Blackwell GPU (B200/GB200) runs it with 32 GiB of KV cache. The KV cache uses FP8.

The repo includes a ready-to-use NVFP4 MTP draft in mtp-nvfp4/ (6.95 GB) for speculative decoding. On a single B200 it gives 1.80× single-stream decode speed, and output quality is unchanged.

The model keeps the full 154,880-token vocabulary and has the vision tower intact.

Benchmarks

Measured on one B200 with vLLM. Sampling follows the base model's recipe (temperature=1.0, top_p=0.95); HumanEval uses greedy decoding. reasoning_effort is the GLM-5.3-Flash chat-template thinking budget (low / high / max). The token budget is 65,536 (16,384 for MMMU).

Scoring is strict: a response that runs out of tokens before giving a final answer counts as wrong. All numbers are single runs. MoE decoding in vLLM isn't bit-deterministic, so treat ±2–3 points as noise. E224 and E256 numbers come from the same harness on the same machine.

About the GPQA-Diamond number: it is a quick self-test at reasoning_effort=low with a 65,536-token budget, run to compare builds against each other, not the model's best score. At low the model thinks briefly and some answers are cut off before they finish. At the base model's recommended reasoning_effort=max with a 163,840-token budget, this family scores 90.9 % on GPQA-Diamond (measured on E224).

Tool use: BFCL v4 (function calling, AST match)

BFCL was run with bfcl-eval v4 against the local OpenAI-compatible endpoint (--tool-call-parser glm47, template-default effort, --num-threads 32).

Throughput and MTP speculative decoding (single B200)

reasoning_effort=low, 1,024 output tokens, short prompts, 8 requests per level, served from this repository as uploaded (--max-model-len 66560 --gpu-memory-utilization 0.93):

Text, Chinese, tool-calling and image smoke tests pass with MTP on. Speculative decoding is lossless: the target model verifies every drafted token, so the draft can only change speed, not output quality. With MTP on, the KV pool on one B200 drops from 2.6 M to 1.65 M tokens (25 concurrent requests at 66 K), so turn it off for very high-concurrency batch serving.

The mtp-nvfp4/ draft

• It is the original GLM-5.3-Flash MTP layer (one nextn layer, all 288 routed experts). Only the storage precision of its MoE weights changed.
• It is exported with NVIDIA ModelOpt (NVFP4QTensor.quantize), the same format as the E224 repo's mtp-nvfp4/: the 288 routed experts and the shared expert are NVFP4 (e2m1 weights packed two per byte, E4M3 block scales over 16-element groups). Attention, the DSA indexer, the router, eh_proj, norms, embeddings and shared_head stay BF16.
• One file, 6.95 GB, vs 17.4 GB for the BF16 draft.
• The same draft works with E192, E224 and E256: the MTP layer doesn't depend on how many experts the main model keeps.
• To regenerate it from the BF16 MTP layer (the mtp/ folder of E224), run scripts/export_mtp_nvfp4.py (needs nvidia-modelopt):

# expects <dir>/mtp (BF16 draft), writes <dir>/mtp-nvfp4
python3 scripts/export_mtp_nvfp4.py <dir>

Fixed 2026-10-10. The first upload of mtp-nvfp4/ (two shards) kept the shared expert in BF16, but its quantization manifest only excluded it under a module name that vLLM 0.31 doesn't use. On vLLM 0.31 the draft then failed to load with a shape assertion. It is now replaced by the ModelOpt export above. If you downloaded the old two-shard version, re-download mtp-nvfp4/.

Deployment

Requirements

• vLLM with GLM-5.3-Flash support. Validated on the ZJY0516/vllm@glm-release branch (vllm-project/vllm#53906) at commit 7e2d791 plus 8f8cc41 ("Make GLM-5.3 kpool metadata graph-safe"). Without that fix, full CUDA graphs can crash under concurrency.
• For MTP: apply scripts/vllm_glm5next_standalone_mtp_draft.patch to vllm/models/glm5next/nvidia/mtp.py. The draft is a standalone directory with its own config (288 experts, num_nextn_predict_layers=1), and the patch makes vLLM build the MTP layer from the draft's config instead of the target's.
• transformers >= 5.16.1
• Set VLLM_USE_DEEP_GEMM=0.

Single B200 / GB200 (validated)

export VLLM_USE_DEEP_GEMM=0
vllm serve autotrust/GLM5.3-Flash-E192-DGX-Spark \
  --served-model-name glm53-flash-e192 \
  --max-model-len 66560 --max-num-seqs 128 \
  --gpu-memory-utilization 0.93 \
  --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45

This loads 123.7 GiB of weights and leaves 31.8 GiB of FP8 KV cache: 2.6 M tokens, or 39 concurrent requests at 66 K. For reasoning_effort=max, use --max-model-len 172032.

With the NVFP4 MTP draft (interactive use), add:

  --speculative-config '{"method": "mtp", "model": "<local-path-to-this-repo>/mtp-nvfp4", "num_speculative_tokens": 2}'

Draft and target together load 127.9 GiB and leave a 1.65 M-token KV pool at 66 K context.

2× DGX Spark

The E224 two-Spark runbook (network setup, NCCL/CX7, swap, launch scripts) applies unchanged. Replace the model path and served name. At TP=2 this model puts about 62 GiB of weights on each node (E224: 71 GiB), so the same --gpu-memory-utilization leaves roughly 9 GiB more KV cache per node than E224. That's enough for reasoning_effort=max at 160 K+ context. The NVFP4 MTP draft adds ~3.5 GiB per node and is the recommended draft on Spark.

We haven't run this model on DGX Spark hardware. The per-node figure above is estimated from the measured weight footprint. Reports from Spark owners are very welcome in the Community tab.

Request format

• Thinking is always on and comes back in the reasoning / reasoning_content field.
• reasoning_effort: low, high or max (the default). Pass it as the top-level OpenAI field or via chat_template_kwargs.
• Recommended sampling: temperature=1.0, top_p=0.95. Greedy decoding can make long thinking loop on hard prompts.
• Use low for chat, Q&A, tool calls, MCQ and vision. Use max with max_tokens ≥ 131,072 for competition math and GPQA-level science.
• Tools: --tool-call-parser glm47 --enable-auto-tool-choice returns structured tool_calls.
• Images and videos use standard OpenAI multi-part content (image_url / video_url).

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(
    model="glm53-flash-e192",
    messages=[{"role": "user", "content": "用三句话解释量子纠缠。"}],
    temperature=1.0, top_p=0.95, max_tokens=16384,
    extra_body={"reasoning_effort": "low"},
)
print(r.choices[0].message.content)

Evaluation protocol

Limitations

• This is an unofficial derivative, not produced or endorsed by Z.ai or NVIDIA.
• Chinese knowledge (C-Eval 78.3 %) and vision (MMMU 70.7 %) are below E224 and E256. Prefer E256 for Chinese and vision workloads.
• It doesn't fit a single DGX Spark. The 128 GB unified memory is smaller than the 124 GiB of weights plus runtime overhead. You need two Sparks or a ≥180 GB GPU.
• MTP serving needs the vLLM patch in scripts/.
• Vision was validated on MMMU only.
• Like the base model, it can produce inaccurate, biased or unsafe content. Evaluate it for your use case before deploying.

License: MIT (inherited from the base model).