The balanced Qwen3.8-27B quant for AMD Strix Halo (Ryzen AI Max+ 395 / gfx1151). Measured, not marketed: one 15 GB file that serves 128K of context on the iGPU — full attention, no offload flags, speculative decoding built in.
Brand: PaoAI · community quant, not affiliated with Alibaba or the Qwen team.
Updated 2026-09-17 — engine update, weights unchanged. Re-measured on the Strix-Halo-tuned Vulkan fork
(LaurentZuijdwijk/llama.cpp, commit 11bfe8a): prefill is +11 % to +33 % across 8K→128K at matching filled
context, chain quality is unchanged (three runs, 100/100/100 · task 100), and the 128K window holds with integrity at
every point. The model file itself did not change — same sha256, same 15.01 GB. See Update log.
Prefill is ahead at every depth — that is the solid result of this update.
What is this? (30 seconds)
Qwen3.8-27B is a hybrid: 48 cheap linear-attention layers and only 16 full-attention layers that actually re-read your context. We compressed it organ by organ — precision where it matters (attention K/V, the MTP draft head, every norm), 4-bit FP4 where it doesn't. One 15 GB file, no offload flags, speculative decoding built in, intact from 8K to 128K of filled context.
The recipe (what's inside)
| Model part |
What it does |
Format |
| Full attention Q/K/V/O (16 layers) |
reads your prompt, tracks context |
Q4_0_ROCMFP4_STRIX (~4.5 bpw) |
| Linear attention (48 gated-delta layers) |
the fast path — constant-time state |
ROCmFP4 (qkv, type 100) |
| FFN gate/up + down (dense) |
the knowledge (biggest block: 9.2 GB) |
ROCmFP4 (~4.5 bpw, imatrix) |
| Draft head (blk.64) |
MTP speculative guessing |
Q8_0 + FP32 (kept sharp) |
| Token embeddings |
vocab lookup |
Q6_K (~6.6 bpw) |
| Norms, gates, glue |
keep signals correct |
FP32 (never touched) |
Measured tensor sizes: FFN gate/up 6.16 GB · FFN down 3.08 GB · linear-attn qkv 1.42 GB · GDN state ops 0.81 GB · token embeddings 1.04 GB · output 0.68 GB · full-attn 0.62 GB · draft head 0.10 GB · norms <0.01 GB → 15.01 GB total.
Chain-test results (N=3, graded by execution)
We run a fixed coding exam where tasks are graded by execution, not opinion. Three identical runs per engine; we report medians.
| Task (what it measures) |
Published engine |
Vulkan fork |
| Hunt — read lying code: trace it + find the planted bug |
100 · 100 · 100 |
100 · 100 · 100 |
| Automaton — implement novel rules + exact state trace |
100 · 100 · 100 |
100 · 100 · 100 |
| Ring buffer — multi-threaded code under stress |
100 · 100 · 100 |
100 · 100 · 100 |
| Task median |
100 |
100 |
A same-session reference run of the published engine also scored 100/100/100 · task 100. Zero protocol events on either engine — quality is a straight tie.
Speed (decode, tokens/second, per phase)
| Phase |
Published |
Fork |
Ours, same session |
Fork vs ours |
| Spec (planning) |
19.2 |
20.4 |
19.3 |
+6 % |
| Hunt (analysis) |
21.6 |
26.3 |
24.7 |
+6 % |
| Automaton (code writing) |
23.2 |
29.8 |
26.1 |
+14 % |
| Ring buffer (code writing) |
20.7 |
22.9 |
19.8 |
+16 % |
| Publish (long output) |
23.2 |
27.5 |
25.2 |
+9 % |
The published column is from September. The defensible comparison is the last column — same session, same flags, engine the only variable: +6 % to +16 %.
Does quantization hurt? (perplexity + KL test — measured 2026-09-17, published 2026-09-18)
Plain words: the chain test above can only tell you if the model still passes. This test asks a different question: did compressing the weights change what the model actually predicts? We ran the industry-standard check — llama.cpp's own llama-perplexity tool — comparing this 15 GB FP4 file against a full-precision F16 copy of the same model (built by us from Qwen's official BF16 release, sha-verified shards), reading the same ~295,000 tokens of standard Wikipedia text through both at 8K windows.
| Metric |
What it means |
Result |
| Perplexity delta |
how much worse the quant predicts text vs its own F16 parent (lower = better) |
−1.0 % (6.587 vs 6.654 — statistically identical) |
| Mean KL divergence |
average "extra surprise" per token vs the parent — 0 = identical |
0.086 (median 0.029) |
| Same top token |
how often both models pick the exact same next word |
88.8 % of positions |
Plain read: on standard text, the FP4 cut is statistically indistinguishable from the full-precision model — no measurable quality loss. The small difference that exists is concentrated in a tail of rare positions (~1 % of tokens shift noticeably), and the chain test above shows those shifts didn't cost task success. Not lossless in the mathematical sense — lossy in a way we cannot measure on standard text.
Reproducible: llama.cpp llama-perplexity, engine commit 11bfe8a, wikitext-2-raw, -c 8192, 36 chunks (~295K tokens). Reference F16 converted by us from Qwen's official BF16 release (sha-verified shards). Full data in our results file.
Context decay — what happens when the window fills (8K → 128K)
| Depth |
Verified fill |
Fork prefill |
Fork decode |
MTP acc |
Prev prefill |
Prev decode |
Prefill Δ |
391 |
| 8K |
8,172 |
306.0 |
31.1 |
0.70 |
229.9 |
21.9 |
+33 % |
Yes |
| 16K |
16,365 |
281.4 |
28.5 |
0.75 |
211.7 |
22.4 |
+33 % |
Yes |
| 32K |
32,784 |
243.7 |
28.6 |
0.83 |
186.9 |
20.5 |
+30 % |
Yes |
| 64K |
65,505 |
173.1 |
21.6 |
0.77 |
150.1 |
17.5 |
+15 % |
Yes |
| 96K |
98,304 |
138.5 |
14.0 |
0.58 |
125.0 |
16.5 |
+11 % |
Yes |
| 128K |
131,067 |
113.8 |
14.0 |
0.66 |
102.9 |
10.6 |
+11 % |
Yes |
| 192K |
196,587 |
84.0 |
10.7 |
0.70 |
— |
— |
beyond card |
Every fill is the number the engine itself reported (prompt eval time = … / N tokens), not a target. Flags identical on both engines, a fresh prompt at every point. 192K is a bonus row (single run, integrity ); no claim is made beyond 192K.**
Plain read: prefill is ahead at every depth (+11 % to +33 %), and nothing breaks at depth — every point answered the integrity question correctly. Decode is not a clean win: better from 8K–64K, worse at 96K (14.0 vs 16.5 — reproduced on a repeat run), level at 128K.
Facts
| Metric |
Result |
| File size |
15.01 GB (13.98 GiB), single file |
| sha256 |
3727594804afef8140165e36c0919c70a98b43751a3153d43dd10385c1f100fd |
| Load |
full GPU (GTT ≈ 51.6 GB at 64K ctx), no special flags needed |
| Context |
standard serve -c 65536; measured 8K→128K; 192K reached (integrity ) |
| Served with |
MTP speculative decoding (n-max 4), flash attention, KV q8_0 |
| State of the art here |
100/100/100/100 chain median (3 runs, both engines) · 6/6 depth points integrity-pass (8K–128K) · 192K bonus pass |
| Weights |
unchanged by this update — same sha256 as the original publish |
Requirements — which llama.cpp build
This model uses two things stock llama.cpp doesn't have: the qwen3_5 architecture (hybrid linear + full attention + MTP) and the Q4_0_ROCMFP4_STRIX quant type (type 105, gfx1151-native).
| Build |
Error you'll see |
| stock llama.cpp |
unknown model architecture: 'qwen3_5' |
| llama.cpp without the ROCmFP4 types |
invalid ggml type 105. should be in [0, 43) |
How to run (measured, locked)
llama-server -m Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED.gguf \
--host 0.0.0.0 --port 8091 -ngl 99 -c 65536 \
--spec-type draft-mtp --spec-draft-n-max 4 \
-fa on --cache-type-k q8_0 --cache-type-v q8_0 --reasoning-budget 1500
Same flags for both engines. Raise -c for long context (up to 262144). Do not add -b/-ub 2048, and do not enable adaptive drafting — both were measured and rejected (see Known limits).
Thinking budget — --reasoning-budget caps how long the model thinks before answering:
| Value |
Behaviour |
-1 (or omit) |
unrestricted thinking, best quality, slowest |
1500 |
what every number on this card was measured with |
4000 |
noticeably deeper reasoning on hard tasks |
12000 |
deep work — long waits before the answer starts |
Known limits (honest)
- Depth costs speed: prefill 306 → 114 t/s and decode 31 → 14 t/s from 8K → 128K of filled context. The 16 full-attention layers re-read history for every token; the 48 linear layers stay cheap. Linear, no cliff — but plan for it at 100K+.
- Decode is not a clean win on the new engine: better at 8K–64K, worse at 96K (14.0 vs 16.5), level at 128K.
- Caveat on the comparison: the "prev" numbers were measured under an unrecorded power state; the fork's are at the AC performance profile — so the deltas blend engine + profile. The clean single-variable figure is +18–27 % prefill (4-arm test, identical fills, same-session reference).
- Not adopted, and why: adaptive drafting (1 formatting failure in 3 chain runs) · wide batch
-b 2048 -ub 2048 (no gain, and it lost the GPU device at depth).
- Vulkan only: every number here is from the gfx1151 Vulkan backend on a Strix Halo iGPU. CPU builds and other GPUs are not comparable.
- Long thinking: a bigger
--reasoning-budget means noticeably longer waits before the answer starts.
- Don't crank context blindly: a single 15 GB file, but GTT grows with context (≈51.6 GB at 64K). Raise
-c in steps on a 128 GB box; don't merge shards.
Serving features explained (plain words)
| Flag |
What it does |
Cost |
--spec-type draft-mtp --spec-draft-n-max 4 |
the model's own draft head guesses 4 tokens ahead, the model verifies them in one pass — roughly 2× decode |
needs the Q8_0 draft head we kept |
-fa on |
flash attention — faster prefill and decode |
none |
--cache-type-k/v q8_0 |
half the context memory |
negligible |
--reasoning-budget 1500 |
caps thinking so answers always arrive |
complex tasks may need more |
prompt caching (default) |
follow-up turns prefill in milliseconds |
none |
Draft acceptance on the depth-sweep probes measured 0.58–0.83 on the fork — lower than the 0.86–0.90 on the original card. The sweep prompts are repetitive filler, which a draft head predicts poorly, so acceptance there is not comparable across tests. On the chain (real work) the fork decoded faster than the previous engine.
Update log
| Date |
What changed |
| 2026-09-18 |
Added quant-loss measurement (perplexity + KL vs own F16 parent): no measurable loss. Card text only — weights untouched. |
| 2026-09-17 |
Numbers re-measured on the STRIX Vulkan fork (11bfe8a). Prefill +11–33 % at matching fills; quality unchanged (3 runs 100/100/100); 128K holds with integrity; 192K bonus row. Weights untouched — same sha256, same 15.01 GB. |
| 2026-09-16 |
Model published: 15.01 GB single file, MTP draft head, sweep measured on the ROCmFPX engine. |
Earlier engine numbers are not repeated here — they were measured on a different build; the tables above are the current measurement. If you reproduce a number, quote the engine commit with it.
Fleet recipes
Serving configs and our benchmark battery: paoai-strix-recipes (GitHub).
Credits & license
- Qwen team (Alibaba) — Qwen3.8-27B base model (Apache-2.0)
- ggml-org / llama.cpp contributors — the engine, the quant formats, the Vulkan backend
- LaurentZuijdwijk — the Strix-Halo-tuned Vulkan fork measured on this card
- nathanw1014 — ROCmFP4 types, qwen3.5/MTP work, Strix Halo gfx1151 performance
- kingjones30 / charlie12345 — ROCmFPX format & recipes
- vcruz305 — converter branch work (qwen3_5 support path)
- unsloth — imatrix methodology reference
- deepu105 — same-box Strix Halo proof and serving-config intel
- antirez — GGUF layout reference
- PaoAI — the STRIX recipe (per-tensor cut/keep table), imatrix-guided builds, sweep benchmarking, chain test
Apache-2.0, same as the base model. Not affiliated with Alibaba or the Qwen team.
