oQ6e mixed-precision MLX quantization of Qwen/Qwen3.6-35B-A3B,
made with oMLX oQe (v0.7.1.dev1). This is the bf16 build.
The multi-token-prediction (MTP) head is kept and the vision tower is included.
Variants
The same family at every bit width and dtype (this repo in bold):
| Variant |
dtype |
Size |
KL vs bf16 ↓ |
Top-1 agreement ↑ |
Perplexity |
| oQ4e |
bf16 |
21.6 GB |
0.0479 |
90.7% |
9.84 |
| oQ4e |
fp16 |
22.5 GB |
0.0467 |
90.7% |
9.84 |
| oQ5e |
bf16 |
26.0 GB |
0.0221 |
93.7% |
9.68 |
| oQ5e |
fp16 |
26.9 GB |
0.0211 |
93.8% |
9.69 |
| oQ6e |
bf16 |
30.3 GB |
0.0138 |
95.2% |
9.65 |
| oQ6e |
fp16 |
31.2 GB |
0.0132 |
95.2% |
9.64 |
| oQ8e |
bf16 |
38.6 GB |
0.0106 |
95.8% |
9.64 |
| oQ8e |
fp16 |
39.5 GB |
0.0101 |
95.7% |
9.63 |
| original |
bf16 |
72 GB |
0 |
100% |
9.63 |
Which dtype? bf16 (no suffix) is the default. fp16 builds store the language model's
non-quantized weights and the quantization scales as float16 instead of bfloat16. They are marginally closer to the original at every level. oMLX notes
that float16 can give faster prefill on M1/M2, which lack native bfloat16. They are about
0.9 GB larger because the vision tower is kept in float32; vision quality was not measured.
How these were measured: mean KL divergence of each quant against the original bf16 model,
over 48 windows x 2048 tokens of English prose, Python code, Japanese and Korean text. Less than
0.3% of this text (60-character chunks) appears in oQe's calibration data. KL is computed on the
reference's top-256 tokens plus one bucket for the remainder, so it slightly underestimates the
full KL; compare numbers from this method only with each other. These are fidelity measurements,
not task benchmarks, and they apply to this 35B-A3B model only. On this model KL is higher at every bit width than on the dense Qwen3.6-27B (for example 0.0106 versus 0.0021 at oQ8e) while perplexity stays close to the original; compare KL only between quants of the same model.
About oQe
oQe sets a base bit width per level and raises sensitive layers above it (for example, oQ4e
uses a 4-bit base with some projections at 5-8 bits), and chooses each weight group's scale and
bias using an importance matrix from calibration data. These builds include oMLX's weighted
least-squares refit of group scales and biases
(jundot/omlx#4385). Compared with earlier oQe builds of this model (pre-oMLX 0.7.1 builds), these builds lower KL by 5-14% depending on level (for example, oQ4e 0.0560 to 0.0479). At oQ5e, oQ6e and oQ8e the per-layer bit allocation is identical to the earlier builds, so the difference there comes from how the weights and scales were fitted, not from which layers got more bits.
- Group size 64, affine quantization
- Calibration: oMLX built-in
oqe_code_multilingual, 128 samples x 512 tokens
- MTP head: preserved (1 layer); its linear layers are quantized with the rest
- Vision tower: not quantized; weights unchanged apart from the patch-embed conv, which is
transposed to MLX's channels-last layout
Chat template
chat_template.jinja is froggeric's v22.5
(froggeric/Qwen-Fixed-Chat-Templates,
Apache-2.0), not Qwen's official template. Qwen's original is included as
chat_template.jinja.official.
For plain chat, with or without a system prompt, the two produce identical prompts. They differ in:
- Reasoning history: earlier turns' reasoning stays in the prompt instead of being dropped.
- Tool calling: the tool-instruction text is rewritten, and JSON-string tool arguments (as sent
by OpenAI-style clients) render instead of raising an error.
- System messages: several leading system messages are merged instead of erroring.
Tool-calling quality with the modified template has not been evaluated. To use Qwen's template,
replace chat_template.jinja with chat_template.jinja.official.
Credits
Base model: Qwen team. Quantizer: oMLX project. Chat template: froggeric.
