PAW-125B-FLASH-NEXT-X3.1

If this model is useful to you, support the work by following @nawariokra on X.

Qwen/Qwen3.8-Flash-Next (125B MoE, 6B active) on one 24 GB GPU plus CPU, with the full 262,144-token context. Two GGUF shards: 47.5 GiB of weights and a 48.3 GiB n-gram table (95.8 GiB in all). It runs on our own engine TRUSS.

Stock llama.cpp cannot read this file. It needs TRUSS. See Requirements and SERVING.md.

Demos

Example scenes made with this model: web pages it wrote, screen-recorded in a browser.

Watermelon jelly (WebGPU)

Rocket launch scene

Paris nocturne (voxel garden with fireworks)

What sits where

Flash-Next carries a 51.2-billion-parameter n-gram embedding table. It is most of the file, and it is not loaded into GPU or RAM: it stays on disk and only the few rows each token needs are read.

Of the 42.0 GiB of routed experts (24,576 experts), about 6.4 GB are the hottest experts held on the GPU all the time. The rest of the GPU's 13.5 GiB expert space is a cache of recently used experts, and the remainder of the bank is in RAM.

Speed

Measured on one RTX 3090 (PCIe 4.0 x8, power limit 250 W), Threadripper 3960X (24 cores), 125 GB RAM, NVMe, at the full 262,144-token context, serving real agent sessions (OpenAI-compatible server, MTP head with 3 drafts, about 2.7 tokens per step). The machine is shared with other users, so decode varies by about 10% from run to run.

Decode speed falls as the context grows: peak is with a short prompt, the 57-60 figure is with 150K tokens of context.

Quality

KL divergence against the BF16 base model

Lower is better; 0 means the same next-token probabilities as BF16. Measured by running the real BF16 model and each quantized version side by side over the same text: 64 chunks of 2,048 tokens of code/chat and 64 chunks of WikiText-2 (plain English), scoring the second half of each chunk (65,472 tokens per corpus). In every row only the 24,576 routed experts come from the quantized file; everything else stays BF16, so the numbers compare the expert quantization alone.

Benchmark

* union over up to four attempts per question, so not a single-pass score.

Requirements

• Engine: TRUSS, commit 0b584c8 or newer. Stock llama.cpp cannot read this file.
• GPU: one 24 GB card (tested on an RTX 3090). CPU: AVX2, 24 cores recommended. RAM: 64 GB.
• Disk: 97 GB on NVMe. The n-gram table is read from disk while decoding; a spinning or USB disk will be slow.

Files

The two flashnext-x31 shards are one model in llama.cpp's split-GGUF layout: put them in the same directory and pass the -00001-of-00002 file to --model; TRUSS finds the second shard by name. They are byte-identical to the former single file (every tensor hash-checked); a bench on the split reproduces the single file's token counts and speed. The MTP head is optional but gives about 2.7 tokens per step instead of 1. Start commands are in SERVING.md.

Limitations

• TRUSS only, on one 24 GB GPU plus CPU. Two-GPU and smaller-GPU setups have not been tried.
• Benchmarks are not in yet (see Quality).
• Output at temperature 0 can differ slightly between runs, because which experts run on the CPU or the GPU depends on timing.
• Long-context (256k) generation quality has not been measured.

Integrity

7ee7f7bf2f96ffb62a6fe9ec5e09084606f31b8d184a67f3bced374264cfb52e  flashnext-x31-00001-of-00002.gguf
28262010f5207b6798189e414e637f4d5ac351e47b971dcd8f62c042873f2139  flashnext-x31-00002-of-00002.gguf
21de8ab563f562b52245e05e0a9af30609bfe3814a7d9c5b6179a78a9331736b  flashnext-mtp-x3k3.gguf

License

Apache 2.0, inherited from the base model.

Credits and attribution

• ExLlamaV3 by turboderp: its EXL3 format (a QTIP-style trellis quantizer) is the basis of the PAW X3 / X3.1 codec, and v1.5.1 is the encoder used here.
• Strata by Niko1221: the serving shape TRUSS's one-GPU path is built on (hot experts on the GPU, the rest on the CPU, speculative decoding). TRUSS reimplements it independently.
• TRUSS: our own from-scratch inference engine for this model.
• Base model Qwen3.8-Flash-Next by Qwen.