Qwen3.8-Flash-Next-ROCmFPX

ROCmFPX quantized GGUF of Qwen/Qwen3.8-Flash-Next — the official base model, locally requantized to 7 ROCmFPX tiers (plus vision mmproj).

• Base model: Qwen3.8 Flash Next (official)
• Architecture: qwen4exp — MoE, 512 experts (10 active per token), 48 transformer blocks, hidden 2560, total ≈176.9B params (model_n_params measured from GGUF/bench JSON)
• Quantization: 7 tiers — Q4_0_ROCMFP4 (FAST / FAST_COHERENT / STRIX), Q6_0_ROCMFPX (base / AGENT), Q8_0_ROCMFPX (base / AGENT)
• Vision: bundled mmproj (CLIP +qwen3vl_merger vision tower)

Important: Format Notice

ROCmFPX is a proprietary quantization format (a llama.cpp-rocm extension; this batch uses GGML file types 103/104/105/110/111/114/115) — upstream llama.cpp cannot load these files. Load them with llama.cpp-rocm, a llama.cpp fork that supports all upstream GGUF formats plus ROCmFPX natively.

Run directly with llama-server (OpenAI-compatible API), or pool two Strix Halo machines with ggml-rpc-server + --rpc (see Usage below).

Files

Quantization Tiers

Quantization Benchmarks

Main table — RPC pooling, 2× Strix Halo (256 GB)

Measured with the llama.cpp-rocm RPC build (build-rpc/bin/llama-bench, sha256 66dd122b0078fb2b504a568b779cdae8315c5349573a74559263a34561b5f67f), two AMD Strix Halo (gfx1151, 128 GB unified memory each) machines pooled over ggml-rpc-server (bond0 40G) into a 256 GB virtual machine: A = client (ms-s1-master 10.0.10.10), B = RPC server (ms-s1-node 10.0.10.11:50052). Command per tier: --rpc 10.0.10.11:50052 -ngl 99 --split-mode layer --tensor-split <per-tier> -p 256,2048 -n 256 -r 5 -o json -lm <per-tier>; each pass runs 3 tests (pp256 / pp2048 / tg256) × 5 shots. Caliber: r5 drop-first — mean of the last 4 shots (first shot dropped as warm-up).

• --tensor-split 3/1 = A holds 25% of layers / B holds 75%; Q6/Q6A (smaller embeddings) used 6/1 and 8/1 with -lm dio; the five -lm none tiers (FAST/FC/STRIX/Q8/Q8A) use -lm none (see Usage).
• r3 (3-shot reference, drop-first of 3): FAST 527.84 / 566.07 / 28.23; FC 526.26 / 555.85 / 28.06; STRIX 528.61 / 566.00 / 28.32; Q6 317.00 / 362.36 / 22.74; Q6A 364.82 / 419.17 / 21.93; Q8 464.03 / 533.08 / 20.30; Q8A 450.40 / 526.71 / 20.03.
• Sample jitter (tg256 stddev/mean) < 1% across all valid passes.

Reference only — do not compare across calibers

Calibers are not directly comparable: the RPC-pooled runs (256 GB, build-rpc engine 66dd122b…), the single-machine mmap runs (build engine d372178f…), and the old auto-caliber runs use different engines and run modes; numbers are archived side by side for reference only — do not compare across rows. The RPC-pooled caliber for Q4_0_ROCMFP4_FAST is now in the main table; the old auto-caliber row (high variance) remains for reference.

Usage

① Fork load (single machine, small tiers)

# llama.cpp-rocm: llama-server, OpenAI-compatible API
llama-server \
  -m Qwen3.8-Flash-Next_Q4_0_ROCMFP4_FAST.gguf \
  -ngl 99 -fa on --port 8080

# Vision: append --mmproj mmproj-Qwen3.8-Flash-Next_BF16.gguf

• OpenAI-compatible API at http://127.0.0.1:8080/v1.
• Single-machine full-resident: -ngl 99 needs ~full model in unified memory; smaller tiers (Q4_FAST 87.79 GiB) fit one 128 GB Strix Halo. Larger tiers require either -lm mmap -ngl 16 (pageable, slower) or two-machine RPC pooling below.
• Vision is enabled by --mmproj <file> (there is no separate vision switch).
• Context: up to 256K (model qwen4exp.context_length = 262144).

② Two-machine RPC pooling (all 7 tiers, the benchmarked method)

# ---- B machine (RPC server, e.g. 10.0.10.11) ----
# escape the session cgroup to avoid the memlock ceiling, then bind port 50052
systemd-run --user --no-block -p LimitMEMLOCK=infinity /bin/bash -c \
  'cd ~/source/llama.cpp-rocm/build/bin && \
   LD_LIBRARY_PATH=$PWD exec ./ggml-rpc-server --host 10.0.10.11 --port 50052'

# ---- A machine (client; model file must be LOCAL to A) ----
# canonical run for the large-embedding tiers (FAST / FC / STRIX / Q8 / Q8A):
systemd-run --user --scope -p MemoryMax=102400M -p MemorySwapMax=0 -- bash -c '
  LD_LIBRARY_PATH=~/source/llama.cpp-rocm/build-rpc/bin \
  ~/source/llama.cpp-rocm/build-rpc/bin/llama-bench \
    -m Qwen3.8-Flash-Next_Q4_0_ROCMFP4_STRIX.gguf \
    --rpc 10.0.10.11:50052 -ngl 99 --split-mode layer --tensor-split 3/1 \
    -p 256,2048 -n 256 -r 5 -o json -lm none'

# Q6 / Q6A (smaller embeddings) run with dio + a lighter A share:
#   Q6_0_ROCMFPX:        --tensor-split 6/1 -lm dio
#   Q6_0_ROCMFPX_AGENT:  --tensor-split 8/1 -lm dio

Key parameters and why:

• -lm none：-lm dio builds an equal-size (~50 GiB) intermediate buffer for the large embedding tensors (read_raw → read_aligned_chunk), which OOMs the client alongside the pinned host buffer; -lm mmap deadlocks with RPC (known). -lm none reads straight into the destination (read_raw_unsafe) — no intermediate buffer, no page-cache deadlock.
• --tensor-split 3/1：A keeps 25% of layers, B holds 75%. The qwen4exp embedding (token_embd + per_layer_token_embd + output ≈ 51 GB) can only be pinned on the client A's local CPU, so A runs "light on layers" and B absorbs the rest.
• --split-mode layer：tensor is not implemented for qwen4exp; row reports "RPC0 does not support split buffers".
• -ngl 99：all layers offloaded; the RPC device sits in device[0].
• Memory guard：MemoryMax=102400M + MemorySwapMax=0 caps the A-side cgroup (A peak ≈81 GiB for the largest tiers); B holds up to ~93 GiB GTT.
• LimitMEMLOCK=infinity on the server side escapes the cgroup memlock ceiling.
• Restart ggml-rpc-server before each model change to clear GTT; the model file must exist locally on the client (A).

License & Attribution

• Base model: Qwen3.8 Flash Next (official) — qwen-community-1.0 (from GGUF metadata general.license.name)
• mmproj (vision tower): CLIP + qwen3vl_merger of the base model — license follows the base model
• This quantized version: Apache-2.0

Disclaimer

This is a quantized version of the official model, provided "as is", without warranty of any kind. ROCmFPX numbers depend on the exact engine build and hardware; the benchmark figures above are measured values under the stated environment, not end-to-end business throughput. Use at your own risk.

Qwen3.8-Flash-Next-ROCmFPX

基于 Qwen/Qwen3.8-Flash-Next 的 ROCmFPX 量化 GGUF——官方基座，本地 requantize 为 7 档 ROCmFPX（另附视觉 mmproj）。

• 基座模型：Qwen3.8 Flash Next（官方）
• 架构：qwen4exp——MoE，512 专家（每 token 激活 10 个），48 层 transformer，hidden 2560，总参数约 176.9B（GGUF/bench JSON 实测 model_n_params）
• 量化：7 档——Q4_0_ROCMFP4（FAST / FAST_COHERENT / STRIX）、Q6_0_ROCMFPX（base / AGENT）、Q8_0_ROCMFPX（base / AGENT）
• 视觉： 附带 mmproj（CLIP +qwen3vl_merger 视觉塔）

重要：格式说明

ROCmFPX 是专有量化格式（llama.cpp-rocm 扩展；本批档位 GGML file_type 为 103/104/105/110/111/114/115）——上游 llama.cpp 无法加载这些文件。请使用 llama.cpp-rocm 引擎加载（llama.cpp fork，全量兼容上游 GGUF 并原生支持 ROCmFPX）。

可直接用 llama-server（OpenAI 兼容 API）加载，或用 ggml-rpc-server + --rpc 做双机拼池（见「使用」）。

文件

量化档位

量化基准测试

主表 — RPC 拼池，2× Strix Halo（256G）

测试环境：AMD Strix Halo（gfx1151，单机 128G 统一内存）×2，经 ggml-rpc-server（bond0 40G 互联）拼池为 256G 虚机；A=客户端（ms-s1-master 10.0.10.10）、B=RPC server（ms-s1-node 10.0.10.11:50052）。引擎＝llama.cpp-rocm RPC 构建（build-rpc/bin/llama-bench，sha256 66dd122b0078fb2b504a568b779cdae8315c5349573a74559263a34561b5f67f）。每档命令：--rpc 10.0.10.11:50052 -ngl 99 --split-mode layer --tensor-split <per-tier> -p 256,2048 -n 256 -r 5 -o json -lm <per-tier>；每 pass 3 测试（pp256/pp2048/tg256）× 5 枪。口径＝r5 剔首枪（5 枪剔首枪取后 4 枪均值）。

• --tensor-split 3/1＝A 25% 层 / B 75% 层；Q6/Q6A（嵌入较小）分别用 6/1、8/1 且 -lm dio；五个 -lm none 档（FAST/FC/STRIX/Q8/Q8A）用 -lm none（见「使用」）。
• r3（3 枪参考口径，剔首枪）：FAST 527.84 / 566.07 / 28.23；FC 526.26 / 555.85 / 28.06；STRIX 528.61 / 566.00 / 28.32；Q6 317.00 / 362.36 / 22.74；Q6A 364.82 / 419.17 / 21.93；Q8 464.03 / 533.08 / 20.30；Q8A 450.40 / 526.71 / 20.03。
• 各有效 pass 样本抖动（tg256 stddev/均值）均 <1%。

仅作对照注记 — 跨口径不可直接比较

Calibers are not directly comparable: the RPC-pooled runs (256 GB, build-rpc engine 66dd122b…), the single-machine mmap runs (build engine d372178f…), and the old auto-caliber runs use different engines and run modes; numbers are archived side by side for reference only — do not compare across rows. The RPC-pooled caliber for Q4_0_ROCMFP4_FAST is now in the main table; the old auto-caliber row (high variance) remains for reference.

使用

① fork 加载（单机，小档）

# llama.cpp-rocm：llama-server，OpenAI 兼容 API
llama-server \
  -m Qwen3.8-Flash-Next_Q4_0_ROCMFP4_FAST.gguf \
  -ngl 99 -fa on --port 8080

# 视觉：追加 --mmproj mmproj-Qwen3.8-Flash-Next_BF16.gguf

• OpenAI 兼容 API：http://127.0.0.1:8080/v1
• 单机全驻留 -ngl 99 需模型接近整机统一内存；小档（Q4_FAST 87.79 GiB）单台 128G Strix Halo 可容。大档需 -lm mmap -ngl 16（可换页、较慢）或下文双机 RPC 拼池。
• 视觉由 --mmproj <文件> 启用（无独立视觉开关）
• 上下文：最高 256K（模型 qwen4exp.context_length = 262144）

② 双机 RPC 拼池（7 档通用，即基准测试跑法）

# ---- B 机（RPC server，例 10.0.10.11）----
# 脱离会话 cgroup 防 memlock 上限，绑定 50052 端口
systemd-run --user --no-block -p LimitMEMLOCK=infinity /bin/bash -c \
  'cd ~/source/llama.cpp-rocm/build/bin && \
   LD_LIBRARY_PATH=$PWD exec ./ggml-rpc-server --host 10.0.10.11 --port 50052'

# ---- A 机（客户端；模型文件必须本地在 A）----
# 大嵌入档（FAST / FC / STRIX / Q8 / Q8A）标准跑法：
systemd-run --user --scope -p MemoryMax=102400M -p MemorySwapMax=0 -- bash -c '
  LD_LIBRARY_PATH=~/source/llama.cpp-rocm/build-rpc/bin \
  ~/source/llama.cpp-rocm/build-rpc/bin/llama-bench \
    -m Qwen3.8-Flash-Next_Q4_0_ROCMFP4_STRIX.gguf \
    --rpc 10.0.10.11:50052 -ngl 99 --split-mode layer --tensor-split 3/1 \
    -p 256,2048 -n 256 -r 5 -o json -lm none'

# Q6 / Q6A（嵌入较小）用 dio + 更轻的 A 侧份额：
#   Q6_0_ROCMFPX:        --tensor-split 6/1 -lm dio
#   Q6_0_ROCMFPX_AGENT:  --tensor-split 8/1 -lm dio

关键参数与理由：

• -lm none：-lm dio 会对大体量嵌入做一块等量（约 50 GiB）中间缓冲（read_raw → read_aligned_chunk），叠加 pinned 宿主缓冲致客户端 OOM；-lm mmap 与 RPC 死锁（已知）。-lm none 走 read_raw_unsafe 直读入目的地——无中间缓冲、无页缓存死锁。
• --tensor-split 3/1：A 留 25% 层、B 扛 75% 层。qwen4exp 嵌入（token_embd + per_layer_token_embd + output ≈ 51 GB）只能钉在客户端 A 本地 CPU，故 A「轻层」、B 兜底。
• --split-mode layer：tensor 对 qwen4exp 未实现；row 报「RPC0 does not support split buffers」。
• -ngl 99：全层卸载；RPC 设备在 device[0]。
• 内存护栏：MemoryMax=102400M + MemorySwapMax=0 限制 A 侧 cgroup（最大档 A 峰值 ≈81 GiB）；B 侧最高约 93 GiB GTT。
• LimitMEMLOCK=infinity：服务端脱离 cgroup 防 memlock 上限。
• 换模型前重启 ggml-rpc-server 清 GTT；模型文件必须本地在客户端（A）。

许可证与归属

• 基座模型：Qwen3.8 Flash Next（官方）— qwen-community-1.0（GGUF 元数据 general.license.name）
• mmproj（视觉塔）：基座 CLIP + qwen3vl_merger — 许可随基座
• 本量化版：Apache-2.0

免责声明

本仓库为官方原版模型的量化版本，按「原样」提供，不附带任何保证。ROCmFPX 数值依赖具体引擎构建与硬件；上表为给定环境下的实测值，非业务端到端吞吐。使用者自行承担全部责任。