# MoE Expert Grading
**Qwen3.6-35B-A3B · every expert gets the bit width it earns**
[](#read-this-first)
[](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)
[](#-pick-one)
[](https://github.com/wuliaott/moe-expert-grading)
**English** · [简体中文](https://huggingface.co/E-stick/Qwen3.6-35B-A3B-MoEGraded/blob/main/README_zh.md)
Read this first
[!WARNING]
This is experimental. The only thing I measured is perplexity — one
corpus, one machine, no task benchmarks. I also never sat down and compared
outputs against the unquantized model line by line. So "the numbers look fine"
is all I can honestly claim; it is not a promise that this is good. Only
tested on Qwen3.6-35B-A3B, and speculative decoding is untested.
[!IMPORTANT]
You need a patched llama.cpp. These files carry per-tier expert tensors
(*_hot / *_warm / *_cold) and moe_expert_groups metadata that upstream
has no idea about. A stock build gives you failed to load model.
Fork is here, with a
prebuilt Windows binary on its release page.
The idea
A MoE model doesn't activate all of its experts equally often. Some get called
constantly, some barely at all — yet a normal quantization gives every expert
the same bit width anyway. So the routed experts get quantized by how often
they actually fire in real tasks: high frequency keeps more bits, low frequency
gets squeezed.
Pick one
|
Balanced |
Compact |
| File |
Qwen3.6-35B-A3B-MoEGraded-Balanced.gguf |
Qwen3.6-35B-A3B-MoEGraded-Compact.gguf |
| Size |
17.26 GiB |
15.59 GiB |
| PPL ↓ |
1.5030 |
1.5096 |
| Hot tier |
Q4_K gate/up, Q5_K down |
Q4_K everywhere |
| Warm tier |
Q3_K + Q4_K down |
Q3_K everywhere |
| Cold tier |
Q3_K |
Q2_K gate/up, Q3_K down |
| Grab it when |
you have the 17 GB |
you don't |
Noise on this corpus runs ±0.029, so honestly these two measure the same.
The difference is what the middle and cold tiers get to keep.
The numbers
14 chunks × 512 tokens, -t 20, same machine throughout:
| model |
size |
PPL ↓ |
vs F16 |
| F16 reference |
67 GB |
1.5053 |
— |
| Balanced |
17.26 GiB |
1.5030 |
−0.002 |
| Compact |
15.59 GiB |
1.5096 |
+0.004 |
| APEX I-Compact (upstream) |
16.10 GiB |
1.5125 |
+0.007 |
| Uniform Q3_K, no grading |
15.82 GiB |
1.5180 |
+0.013 |
| Upstream Q4_K_S |
20 GB |
1.5167 |
+0.011 |
| Upstream Q3_K_S |
15 GB |
1.5404 |
+0.035 |
Balanced dipping under F16 isn't a real win — 0.002 is well inside the noise.
The part worth looking at is the lower half: both files beat APEX I-Compact,
and Compact does it from a file 0.51 GiB smaller.
How it works
1 · Measure the frequency. Run an importance matrix over 5,120 calibration
tokens to count how often each routed expert gets activated.
2 · Sort into three groups. Top 20% hot (51 experts), next 30% warm (77),
the rest cold (128).
3 · Quantize each separately. Every group gets its own tensors and its own
type, gate/up/down independently. Router, norms, attention, embeddings and
shared experts stay high precision either way.
4 · Reassemble at load time. The patched loader sends every mul_mat_id
through a tier lookup table and a mask — one pass covers all three groups, and
the router still sees all 256 experts.
None of this would work if MoE were dense. Because only 8 experts fire per
token, a cold expert that's slightly off barely gets touched — and that slack is
what pays for the hot tier's extra bits.
▶ Run it
Prebuilt — Windows x64, CPU + Vulkan, tested on an RX 5700 XT:
⬇ llama-fork-vulkan-graded-win-x64.zip
Or build it from source:
git clone --depth 1 --branch 0c1e570 https://github.com/ggml-org/llama.cpp
cd llama.cpp
patch -p1 < graded-moe-0c1e570.patch
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
build/bin/llama-cli -m Qwen3.6-35B-A3B-MoEGraded-Balanced.gguf \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.5 \
-p "Write a haiku about quantization."
Sampling follows the official Qwen3.6 recommendation. Don't use greedy
decoding — it hits Qwen3.6's repetition loop.
Reproducing from scratch
The patch, `quant_recipe.py` and the tiering scripts are in the
[GitHub repo](https://github.com/wuliaott/moe-expert-grading). `README-TECH.md`
there has the details and the pitfalls.
python3 quant_recipe.py --list
python3 quant_recipe.py --recipe v8 --target-mib 17673 --run --ppl
`--target-mib` makes the script assert the predicted size against what actually
lands on disk, so a recipe change can't silently slip through.
Built with an AI coding agent from a written design. The measurements are
real; nobody outside the project has reviewed the code.
Weights Apache-2.0, following Qwen3.6-35B-A3B · fork and scripts MIT.
