Gemma-4-26B-A4B-it · QAT · abliterated · MLX 4-bit
An MLX (Apple silicon) build of the abliterated Gemma-4-26B-A4B-it MoE
(128 experts, top-8, ~4B active). It uses Google's QAT int4 lattice weights
wherever they are exact. The build targets a Mac mini M4 with 24 GB serving
2 concurrent requests at 64k context. It runs through oMLX
fully resident at a 21.5 GB Metal limit. At a 20 GB limit, MoE expert offload streams non-resident experts from the NVMe SSD on demand.
- Size: 16.0 GB (text-only; the vision tower is not included)
- Quality vs the exact bf16 model: KL 0.053, top-1 agreement 92.4%. This is the
same as the measured noise floor between two exact bf16 implementations (≈0.05 / ≈91%).
- Refusal behaviour is identical to the bf16 abliterated model (XSTest safe prompts: 0.8%).
Name note: the repo name says "dwq". During the project we trained a DWQ variant
(scale/bias distillation: KL 0.100, top-1 88.2%). Keeping Google's exact QAT lattice for
the experts beat it clearly, so this card describes the lattice build that is published here.
How it was built
| Tensor group |
Format |
Why |
| Routed experts (gate/up/down), q/k/v, dense MLP gate/up, embeddings |
4-bit affine gs32 |
Google QAT q4_0 values, an exact (lossless) re-encoding of the QAT int4 lattice |
self_attn.o_proj, dense mlp.down_proj (all 30 layers) |
8-bit affine gs64 |
The abliteration edit moved these tensors off the lattice |
| Router, norms |
bf16 |
Unquantized |
The source is Google's gemma-4-26B-A4B-it-qat-q4_0-unquantized, plus the 60 abliterated tensors of
huihui-ai's QAT abliteration. All other 953 tensors are bit-identical (verified).
Quality (vs exact bf16 QAT-abliterated model)
| Metric |
This model |
Exact bf16 teacher |
| KL (T=1), 32 held-out batches, top-1024 logits |
0.053 |
noise floor ≈0.05* |
| Top-1 agreement |
92.4% |
≈91%* |
| Long-context KL @1k / 16k / 32k |
0.119 / 0.091 / 0.077 |
— |
| ARC-Challenge (acc_norm, 200) |
0.425 |
0.425 |
| HellaSwag (acc_norm, 200) |
0.555 |
0.545 |
| WinoGrande (acc, 200) |
0.630 |
0.615 |
| GSM8K (5-shot, flexible, 200) |
0.945 |
0.950 |
| Over-refusal, XSTest 250 safe prompts |
0.8% |
0.8% |
* Two faithful bf16 implementations of this MoE disagree by KL ≈0.046–0.054 and top-1 ≈91%,
because expert-routing near-ties flip under tiny numeric differences. Differences below that floor are noise.
Speed on Mac mini M4 (24 GB), oMLX, 2 concurrent × 64k
Setup:
- The model is fully resident, with no expert offload.
- Metal wired limit 21.5 GB (sudo sysctl iogpu.wired_limit_mb=22016; resets on reboot). This leaves macOS about 2.5 GB.
- oMLX memory guard 22 GB, TurboQuant 4-bit KV, 64k window, max 2 concurrent requests.
| Load |
Time to first token |
Decode |
| 1 user, 1k prompt |
3.5 s |
31.5 tok/s |
| 1 user, 8k prompt |
22 s |
29.3 tok/s |
| 1 user, 32k prompt |
102 s |
23.1 tok/s |
| 2 users × 64k prompts (cold, concurrent) |
264 s / 550 s (prefills run one after another) |
~8–10 tok/s each on follow-ups |
| Follow-up turn on a cached 64k conversation |
8–11 s |
~8 tok/s each (2 users) |
Validated on a fresh server and again after a 20-minute 2-user soak (1.56M prompt tokens): 0 errors. Peak memory was 20–21 GB. Swap grew 0.5–0.8 GB during the 2×64k tests, which is tight.
Lower memory limit (20 GB): SSD expert offload. oMLX MoE expert offload streams non-resident experts from NVMe; it is exact, so quality is unchanged. With offload on decoder layers 10–29 at 85% residency (OMLX_MOE_OFFLOAD_LAYERS=10-29, a local oMLX patch), the reference machine measured 8 / 25 / 115 s to first token and 25.5 / 24 / 19.7 tok/s decode, and 2×64k still worked.
Usage
pip install mlx-lm
mlx_lm.generate --model quocbao747/gemma-4-26B-A4B-it-qat-abliterated-mlx-4bit-dwq \
--prompt "Hello" --max-tokens 200
oMLX model_settings.json entry used for the numbers above:
{"max_context_window": 65536, "turboquant_kv_enabled": true, "turboquant_kv_bits": 4,
"enable_thinking": false, "is_pinned": true}
# 20 GB limit variant: add "moe_expert_offload_enabled": true, "moe_expert_offload_resident_fraction": 0.85
# and env OMLX_MOE_OFFLOAD_LAYERS=10-29 (local oMLX patch)
Limitations
- This model is abliterated: refusal behaviour was removed. You are responsible for how you use it.
- On a 24 GB Mac, oMLX's prefill memory guard ratchets its transient estimate upward during a long session.
After heavy concurrent use, a brand-new cold 64k prompt can be refused. Restarting oMLX when it is idle
resets this. The reference deployment does it automatically with a small watcher: restart after 90 s idle
once 150k+ prompt tokens have been served. Restart takes 6 s, and the SSD KV cache survives.
Validated sequence:
1. 2×64k on a fresh server
2. a 20-minute 2-user soak (1.38M prompt tokens)
3. an idle refresh
4. 4 cold 65k needle prompts, all served, single retrieval 4/4
5. 2×64k again: 0 errors, peak 20 GB, no swap
- Two-hop retrieval at 64k is about 70% (8/11), and matches the DWQ build on identical haystacks, so it is a
base-model limit. Single-needle retrieval at 64k is 100%.
- Text only.
Credits
Google (Gemma 4 and QAT weights), huihui-ai (abliteration), Apple MLX / mlx-lm, oMLX.
