Ornith-1.5-35B-A3B-ROCmFP4-GGUF

Ornith-1.5-35B-A3B ROCmFP4

ROCmFP4 quantization of Ornith 1.5 35B-A3B for AMD Strix Halo (gfx1151) and RDNA 3.5 GPUs, produced with the Q4_0_ROCMFP4_STRIX_LEAN preset (97.0% of parameters in ROCmFP4, attention K/V in the dual-scale ROCmFP4 variant, token_embd at Q5_K, all norms at F32, and the nextn MTP eh_proj at q8_0).

Known issue: output depends on the previous request. With a server reused across requests, this model's output is not reproducible — the same prompt can return different text depending on what ran before it, and can collapse into degenerate repetition (分割线, mesh mesh mesh, 恒 恒 恒).

The root cause is most likely #29092 — Gated Delta Net recurrent state not being fully cleared between requests on a reused slot. The weights are fine; this is a runtime state bug, and it is not specific to this quant.

Recommended setting: --spec-type draft-mtp --spec-mtp-strict-qwen --spec-draft-n-max 2 --spec-draft-p-min 0.6. Counter-intuitively this reduces the problem rather than causing it — see Known Issues for the measurements. An earlier revision of this card told you to disable MTP; that advice was wrong and has been retracted.

Revision 2026-08-28: rebuilt from the ornith-ai aug-24 MTP refresh. The retrained MTP head gives ~100 tok/s effective decode on Strix Halo at the settings above. Note: the new MTP head requires this base revision; do not graft it onto older quants.

Quantization Details

Both checksums above are the content hashes recorded in this repo's git-LFS pointers, which is what huggingface_hub verifies on download. If SHA256SUMS ever disagrees with the table above, trust the table above and the LFS OID — see the note below.

Note 2026-09-27: the SHA256SUMS file in this repo was stale, carrying the checksum from an earlier build (b42fb74c…). It has been corrected to match the shipped weights. If you verified against SHA256SUMS and got a mismatch, your download was fine. The checksum in the table above was always correct.

Verifying this card

The factual claims on this card are machine-checked against the shipped files by verify_card.py, which runs in CI on every push. It reads the GGUF header over HTTP range requests and the repo's LFS OIDs, so it needs neither the weights nor a GPU.

# check the card as published on the Hub
python3 verify_card.py --repo julianmb/Ornith-1.5-35B-A3B-ROCmFP4-GGUF

# check a local card and/or a local copy of the weights
python3 verify_card.py --repo julianmb/Ornith-1.5-35B-A3B-ROCmFP4-GGUF \
    --card ./README.md --gguf ./Ornith-1.5-35B-A3B-ROCmFP4.gguf

It verifies: both SHA256s (card vs LFS OID vs SHA256SUMS), both file sizes, the parameter count, the bits-per-weight figure, block_count, and that SHA256SUMS lists only files that exist. Exit code is non-zero on any mismatch, so it fails a build rather than shipping a wrong number.

This exists because the card drifted from the artifact several times — a stale SHA256SUMS, a wrong file size, a wrong size-percentage, and a quant description that did not match the actual tensor types. Please run it before publishing a new revision.

Tensor composition (read from the GGUF metadata)

Note for anyone re-deriving these: earlier revisions of this card described the preset as "FP16 embedding/norm preservation". That was inaccurate — the norms are F32 and the token embedding is Q5_K, not FP16. Only the MTP eh_proj is q8_0; the three MTP norm vectors are F32.

Measured Performance (AMD Ryzen AI Max+ 395, Radeon 8060S, Mesa RADV Wave64)

Bare-decode comparison, no speculative decoding on either side:

ROCmFP4 is ~7.3–7.6% faster and 12.3% smaller than the base repo's Q4_K_M (21,713,463,040 bytes vs 19,052,438,944 bytes; equivalently 4.29 vs 4.89 bits per weight).

Note that the fastest row is not the most usable one. The n4 setting is ~5% quicker than n2 but roughly triples the number of distinct outputs. Treat throughput and stability as separate axes here.

The "reproducible?" column was measured by chandlerma on Strix Halo / Vulkan / RADV, on this quant, by issuing the same request repeatedly and counting distinct outputs. Details and caveats in Known Issues.

Corrected 2026-09-27: this line previously said 16.7% smaller, which is not supported by the actual files. The measured figure is 12.3%, cross-checked two ways — raw bytes and bits-per-weight, which agree. The 16.7% figure could not be reproduced against the Q4_K_M in the base repo under any assumption about whether that file carries an MTP head, since an MTP head adds only ~9 MB at q8_0. Treat the original number as a transcription error.

Draft acceptance rate is not a health signal for speculative decoding: when the drafter and the target are affected by the same state leak, they are blinded in the same way and acceptance stays high while the output varies. Judge MTP settings by output reproducibility, not acceptance.

Serving

HaloFPX (Text + Vision)

halofpx pull downloads and verifies both the ROCmFP4 weights and BF16 vision projector:

halofpx pull ornith-1.5-35b
halofpx serve
halofpx load ornith-1.5-35b

halofpx list reports model-weight and vision-projector readiness separately.

Direct llama-server Vision

llama-server \
  -m Ornith-1.5-35B-A3B-ROCmFP4.gguf \
  -mm mmproj-Ornith-1.5-35B-BF16.gguf \
  -ngl 999 -c 131072 -fa on -np 1 \
  --spec-type draft-mtp --spec-mtp-strict-qwen \
  --spec-draft-n-max 2 --spec-draft-p-min 0.6

Full 256K Max Context (Strix Halo 128GB UMA):

llama-server \
  -m Ornith-1.5-35B-A3B-ROCmFP4.gguf \
  -ngl 999 -c 262144 -fa on -np 1 \
  --spec-type draft-mtp --spec-mtp-strict-qwen \
  --spec-draft-n-max 2 --spec-draft-p-min 0.6

Multi-Slot / Conservative Memory (131K Context):

llama-server \
  -m Ornith-1.5-35B-A3B-ROCmFP4.gguf \
  -ngl 999 -c 131072 -fa on -np 1 \
  --spec-type draft-mtp --spec-mtp-strict-qwen \
  --spec-draft-n-max 2 --spec-draft-p-min 0.6

Note on flags: - Keep MTP on, at n-max 2, with --spec-mtp-strict-qwen. Disabling MTP makes reproducibility worse, not better. An earlier revision of this card said the opposite; see the retraction below. - --spec-mtp-strict-qwen is a fork-local flag. It exists in ROCmFPX (PR #54) and not in upstream ggml-org/llama.cpp. You need a ROCmFP4-capable build regardless, since the ROCmFP4 tensor types do not exist upstream — see Requirements. - ROCmFP4 in ROCmFPX is built as a Vulkan backend plugin, off by default. If you enable it, pass -dev ROCmFPXVulkan0 (not Vulkan0) and set ROCMFPX_PLUGIN_PATH. If your build has ROCmFP4 compiled into the normal Vulkan0 backend instead, use -dev Vulkan0. Check --list-devices for what your build exposes. - --mmap / --no-mmap were removed in favour of --load-mode — see PR #26934. The default --load-mode auto memory-maps unless the device reports it cannot. - -np 1 is deliberate. The known state leak is per reused slot, and concurrent slots make the cross-talk harder to reason about.

Requirements

This quant will not load in stock ggml-org/llama.cpp, LM Studio, or Ollama. The q4_0_rocmfp4 tensor types (ggml ids 100 and 101) do not exist upstream. You need a ROCmFP4-capable build:

• ROCmFPX/ROCmFPX — current, and the source of --spec-mtp-strict-qwen
• charlie12345/rocmfp4-llama (branch mtp-rocmfp4-strix) — the original ROCmFP4 work

Building the Vulkan path in ROCmFPX:

cmake -B build -DGGML_VULKAN=ON -DROCMFPX_VULKAN_PLUGIN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server

Then export ROCMFPX_PLUGIN_PATH=<build>/bin and select the device listed by --list-devices.

Known Issues

Retraction: an earlier revision of this card gave the wrong advice

Between 2026-09-27 and this revision, this card said "Do not enable --spec-type draft-mtp", pointed users at bare decode, and offered a no-MTP variant as the workaround. That was wrong. It was based on the hypothesis that MTP itself caused the problem. It does not. Disabling MTP makes output reproducibility worse.

I got it wrong because I took a repro from llama.cpp#26425 at face value — in that report, disabling --spec-type made output byte-identical across repeated calls. On this quant on Strix Halo the opposite is observed. The difference is that #26425's clean control was measured on a fresh sequence of requests, whereas the failure here is a per-slot state leak that a short control does not expose.

If you followed the earlier advice, go back to the recommended settings in Serving.

The actual issue: per-slot state leak (#29092)

Symptom. The same prompt returns different text depending on which request ran before it. With enough requests it collapses into degenerate repetition — 分割线, mesh mesh mesh, 恒 恒 恒, jijijijiji, op1 op1 op1.

This is not a quant defect. It has been reported on Q4_K_M and other quants of this architecture too.

Upstream: ggml-org/llama.cpp#29092 — "HIP/ROCm — fused Gated Delta Net op carries recurrent state across requests on a reused server slot (qwen35 / qwen35moe); earlier prompts' text is emitted verbatim in later completions". Recurrent state on a Gated Delta Net layer cannot be partially rolled back the way a KV cache can, so a reused slot does not fully return to a clean state.

There is a related open issue, #26425, whose MTP symptom overlaps. Which of the two is the dominant cause here is not settled.

Measured mitigation (Strix Halo / Vulkan / RADV, this quant)

By chandlerma, counting distinct outputs for a repeated identical request:

MTP is a mitigation here, not the cause. --spec-mtp-strict-qwen performs a bounded recurrent rollback of GDN state during draft verification, which partially suppresses the leak. That is why n-max 2 converges better than either n-max 4 or MTP-off, and it is consistent with the flag's own description of its mechanism.

The 2 remaining variants were semantically equivalent — same answer, slightly different wording. So n-max 2 is stable for practical purposes, not perfectly deterministic.

Caveats: measured on one machine, one prompt, -np 1, and Vulkan rather than HIP. n-max 2 vs n-max 4 was not tested across many prompts, and the -ngl 998 and MMVQ arms were not combined with --spec-mtp-strict-qwen. Treat it as the best available starting point, not a guarantee.

If you need strict reproducibility

• Use -np 1 and restart the server between unrelated sessions. Much of the effect is a reused-slot artefact.
• Set cache_prompt: false for the requests you care about.
• There is no flag that makes this architecture fully deterministic until #29092 is fixed. Low temperature helps but does not remove the cross-request dependence.

Related open upstream issues

All four are open as of 2026-09-27. These can produce similar-looking symptoms and are worth ruling out if disabling MTP does not resolve it for you:

KV cache types

turbo3 / turbo4 are not valid -ctk/-ctv values on current llama.cpp. The allowed set is f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1, and the default is f16.

If you want to trade memory for speed, -ctk q8_0 -ctv q8_0 is the safe step down — community testing on Qwen3.5 found q8_0 indistinguishable from f16 in both directions, with real degradation only below that. Going to q4_0/q4_1 is a known cause of degraded long-horizon coherence, so avoid it if you are seeing repetition.

The default for both is f16, so passing nothing is fine.

Disabling thinking / reasoning

This is a hybrid reasoning model. Its chat template honours enable_thinking, and llama.cpp exposes it as --reasoning on|off|auto (default auto, which detects from the template). With --reasoning off the template emits a pre-closed empty <think> block, so no thinking tokens are generated.

Two separate flags are easy to confuse:

• --reasoning off — controls whether the template starts a thinking block (generation-side).
• --reasoning-format — controls how thought tags in the response are parsed and returned (none, deepseek, deepseek-legacy; default auto). Use --reasoning-format none if you want any thought text left inline in message.content, or deepseek to get it in message.reasoning_content.

--reasoning is a server startup flag and is not settable per request. Note that enable_thinking passed via --chat-template-kwargs is deprecated and ignored on current builds.

If you see raw reasoning-style text in the output despite --reasoning off, that is a different problem from the degeneration above — check that --jinja is on (it is by default on current builds) and that your client is not injecting its own <think> tags.