llama.cpp-openPangu-2.0-Flash

openPangu-2.0-Flash support for llama.cpp v0.7.0

This repository contains a patch that adds support for openPangu-2.0-Flash (92B MoE, ~6B active parameters, 512K context) to the latest llama.cpp release v0.7.0 (commit 620d2225cc436ab0b001e3200e590e3b286ef9ab, 2026-10-15).

The support was ported from mrexodia/llama.cpp-openPangu-2.0-Flash (which tracks an older llama.cpp base) and rebased on top of v0.7.0.

Files:

• llama.cpp-openpangu-v0.7.0.diff — the full patch (43 files, +3290 / −367)
• patch.sh — downloads the pinned llama.cpp commit, applies the patch and prints build/run instructions
• LICENSE — GPL-3.0 (this project) with the references/licenses of the upstream projects it is built on

中文說明

本 repo 是將 openPangu-2.0-Flash（92B MoE、約 6B 激活參數、512K 上下文） 的 llama.cpp 支援，從 mrexodia 的 fork 移植到最新的 llama.cpp v0.7.0 上。

用法：

1. 執行 ./patch.sh（會自動下載 v0.7.0 原始碼並套用 patch）
2. cd llama.cpp-620d2225cc436ab0b001e3200e590e3b286ef9ab
3. cmake -B build -S . -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
4. 下載 GGUF 模型（見下方連結）後即可執行 llama-cli / llama-server

What is included

• The openpangu-v2 architecture: absorbed MLA attention, DSA sparse attention (lightning-indexer top-2048 selection on the global layers), per-layer sliding-window attention via a hybrid iSWA cache, 4-stream manifold hyper-connections (mHC), MoME causal convolutions with recurrent cross-batch state, and 128 learned cache-resident attention sinks per layer
• Fused CPU/CUDA kernels for the mHC mixing chain (GGML_OP_SINKHORN, GGML_OP_HC_MIX), with automatic fallback to the unfused ops on backends without the kernels
• Gather-based sparse attention at decode: DSA layers attend only the top-2048 selected tokens (plus sinks), with a fused indexer-scoring kernel (GGML_OP_DSA_SCORE)
• Fused DSA indexer scoring at prefill (tiled kernel that skips the multi-GB scoring intermediate) plus a radix-select top-k for GGML_OP_TOP_K
• Multi-token-prediction self-speculative decoding (--spec-type draft-mtp, with the split MTP draft GGUF)
• Chat support: tool calling and thinking reasoning parsing for the openPangu template
• HF → GGUF conversion (conversion/pangu.py) and a base/MTP split tool (conversion/split_pangu_mtp.py)

Ported and adapted to v0.7.0 APIs, plus fixes on top of the original fork:

• adapted to the v0.7.0 common chat parser API (build_chat_peg_parser), per-layer n_ff_exp(), and the new build_attn_mha(..., n_kv_max, ...) signature
• CPU AMX fix: the AMX weight packer only implements 2D layouts; batched (3D/4D) quantized weights — e.g. the MLA wk_b/wv_b tensors — were corrupted when routed through the AMX extra buffer, producing gibberish output. The AMX backend now rejects batched tensors and they run on the plain CPU path
• the absorbed-MLA wk_b/wv_b inputs are kept contiguous so they are also correct with weight repacking on other backends

Requirements

• A C/C++ build environment (CMake ≥ 3.14, C++17 compiler)
• No GPU required (CPU-only works); CUDA/ Metal/ Vulkan also supported
• Get the GGUF models at mrexodia/openPangu-2.0-Flash-GGUF:
• openPangu-2.0-Flash-base-Q3_K_M.gguf — 40.7 GiB
• openPangu-2.0-Flash-base-Q4_K_M.gguf — 51.5 GiB
• openPangu-2.0-Flash-base-Q8_0.gguf — 90.8 GiB
• openPangu-2.0-Flash-base-BF16.gguf — 170.7 GiB
• openPangu-2.0-Flash-mtp-Q8_0.gguf — 9.2 GiB (optional MTP head)

Quick start

./patch.sh
cd llama.cpp-620d2225cc436ab0b001e3200e590e3b286ef9ab
cmake -B build -S . -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-cli llama-server

CUDA build:

cmake -B build -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON

Run

./build/bin/llama-cli -m openPangu-2.0-Flash-base-Q3_K_M.gguf -c 65536 --jinja -t $(nproc)

./build/bin/llama-server -m openPangu-2.0-Flash-base-Q3_K_M.gguf \
    -c 65536 --jinja --host 0.0.0.0 --port 8080

Context scales up to -c 524288; the compressed MLA KV cache needs only ~12 GB at the full 512K.

Multi-token prediction (self-speculative decoding)

./build/bin/llama-cli -m openPangu-2.0-Flash-base-Q3_K_M.gguf \
    --spec-draft-model openPangu-2.0-Flash-mtp-Q8_0.gguf \
    --spec-type draft-mtp -c 65536 --jinja

Notes:

• The MTP draft adds ~10 GB of RAM usage on top of the base model
• On CPUs with AMX, weight repacking (--repack, enabled by default) costs ~16 GiB of extra RAM; if the combined footprint does not fit, run with -nr (no repack)
• On CPU-only setups the draft overhead can make generation slower than plain decoding — measure on your hardware (llama-bench does not support draft models; use llama-cli and compare the reported t/s)

Test environment（測試環境）

All performance numbers below were measured on a real machine:

• CPU: Intel Xeon Platinum 8559C (Emerald Rapids, supports AMX); the test container is capped at 16 vCPUs (host has 2 sockets × 48 cores × 2 SMT)
• RAM: 2.0 TiB host; the test environment enforces a 64 GiB per-process memory cap, which is why all testing used the 40.7 GiB Q3_K_M quant
• GPU: none — CPU-only inference, 16 threads (-t 16)
• OS/toolchain: Linux, gcc 14.2.0, CMake 3.31.6
• Model: openPangu-2.0-Flash-base-Q3_K_M.gguf (40.72 GiB, 91.6 B params)

Measured resident memory of llama-cli with this model (-c 512, -n 1):

Tests

• test-backend-ops -b CPU -o SINKHORN -o HC_MIX -o DSA_SCORE -o TOP_K: 534/534 tests pass
• test-chat, test-chat-template, test-chat-peg-parser, test-chat-auto-parser: all pass

Comparison with the original fork

Measured against the original mrexodia/llama.cpp-openPangu-2.0-Flash (HEAD 76c92350b, llama.cpp base 6f4f53f2b from 2026-06-29), same machine, same model, same flags.

Compatibility on this machine (AMX CPU)

The original fork cannot be used with default settings on a CPU with AMX. Its llama-bench has no option to disable repacking at all — the original numbers below required a local source edit. This port fixes the underlying AMX packing bug (the AMX packer silently corrupted batched/3D weights).

llama-bench, Q3_K_M, -t 16 -r 3 (2026-10-16)

Both builds with weight repacking disabled (-nr, the only common runnable configuration on this machine):

This port with the default AMX weight repacking (original fork cannot run this configuration):

Reading these numbers:

• Prefill is ~1.5× faster than the original fork, mostly from the v0.7.0 base (3.5 months of upstream CPU optimizations). Generation is on par.
• On this CPU, AMX weight repacking is performance-neutral and costs ~16 GiB of extra RAM, so -nr is usually the better choice.
• The host is shared and measurements vary between runs (see the ±).

Real inference (llama-cli end-to-end)

Actual generation with a real prompt (3,271 tokens of English text), llama-cli -c 8192 -n 128 -t 16 --no-warmup -st, two runs each (prompt eval / generation speed as reported by llama-cli):

• real prompt processing is ~+40 % faster than the original fork (+29 % in the default AMX repack configuration)
• generation with a ~3K-token context is ~6 t/s in all configurations (the DSA indexer adds per-token work over the cached context, so this is lower than the tg128 llama-bench numbers on a fresh 128-token run)
• run-to-run variance on this host is small for these end-to-end runs

MTP (self-speculative decoding)

Same prompt, llama-cli -n 128 -nr:

On CPU the MTP draft head is a net loss in both forks; it is included for completeness and may pay off on GPUs where the draft forward pass is cheap.

CPU optimization experiments（CPU 優化實驗記錄）

Upstream CPU-speedup candidates (open pull requests of llama.cpp) are benchmarked one by one on the machine described above. Because this shared host is noisy, each candidate is measured with an alternating A/B protocol: two binaries built from the same source tree (baseline vs baseline + candidate patch) are run interleaved in the same session (llama-bench -t 16 -r 3 -p 512 -n 128, default settings, 3–8 rounds each); the table reports the medians over all rounds.

Only candidates that win beyond the run-to-run noise are merged into the patch ( 有幫助); candidates without a real gain are recorded here with their data and arenotmerged ( 沒幫助).

Round-by-round for #27024 (candidate vs baseline, t/s):

The candidate won 4 of 5 pairwise rounds on both metrics; the single losing round (r5) is a transient quiet window on the host (both absolute values spike upward). Median deltas stay positive (+3.4 % / +2.7 %), so the change is merged. All values t/s; measured 2026-10-16.

Round-by-round for #22525 (candidate vs baseline, t/s), default settings. Note: Q3_K_M does contain 134 Q4_K tensors (the MoE expert weights), so the Q4_K GEMM path this PR optimizes is actually exercised:

The candidate won 6 of 8 pairwise rounds on prefill (1 loss, 1 tie within 0.5 %); decode is neutral (4 W / 3 L / 1 T). The PR only changes the AVX-512 interleaving/staging of the Q4_K GEMM (the prefill path), so the measured profile matches the expectation. Merged. Measured 2026-10-16.

Round-by-round for #19706 (candidate vs baseline, t/s), default settings:

Verdict 沒幫助 (no measurable gain). Medians differ by +0.3 % (prefill) and +0.1 % (decode), with a 4 W / 3 L / 1 T split on both — pure noise. The base Q3_K_M model contains only one single Q6_K tensor, so a faster Q6_K kernel has almost no weight to act on. The PR is not merged; the data is recorded here for the record. Measured 2026-10-16.

Round-by-round for #27590 (candidate vs baseline, t/s), -nr mode. The Q5_K/Q6_K dots this PR changes are used on the plain CPU path selected by -nr; with default repacking those weights go through the repack gemv/gemm kernels instead, and default mode indeed measured neutral (pp512 −1.7 % / tg128 +1.0 %, within noise):

Verdict 有幫助 (-nr 模式, +2.9 % prefill / +2.5 % decode medians, 5 of 7 rounds won on both). In this model the PR affects Q5_K (2 tensors, the dense ffn_down of layers 0–1) and Q6_K (1 tensor — output.weight, the 388 M-param head that is read on every token). One caveat: the upstream PR is closed but never merged (only an automated AI-content flag, no review), so it received extra local validation before merging:

• perplexity on a fixed corpus: 1.0354 → 1.0354 (identical)
• test-backend-ops -b CPU -o MUL_MAT -p 'type=q5_K|type=q6_K' passes

Merged. Measured 2026-10-16.

Round-by-round for #18698 (candidate vs baseline, t/s), default settings. This PR repacks the quantized weights in parallel with OpenMP and pins each repack thread NUMA-local. (Ported by hand: the year-old PR conflicts with the current tree and depends on the never-merged PR #18239.)

Verdict 沒幫助 (slightly slower, not merged). Medians: prefill −2.1 % (2 W / 6 L) and decode −2.5 % (3 W / 3 L / 2 T). Side effect worth recording: the parallel repack cuts model load wall time (load + tiny bench, warm page cache) from 24 s (2/2 runs) to 10 s (2/2 runs) — but the inference phase itself gains nothing, so per the merge rule (real tok/s gains) the candidate is not merged. Correctness: perplexity 1.0361 → 1.0361 (identical), i.e. the parallel repack produces identical weights; the regression is presumably a NUMA page-placement effect, not a data error. Measured 2026-10-16.

License

This project (the patch set, patch.sh and documentation) is licensed under GPL-3.0 — see LICENSE. The patch is a derivative of llama.cpp; the MIT-licensed portions remain under their original licenses, with the references listed in the LICENSE header and repeated here:

• llama.cpp — MIT — © ggml-org contributors (base commit 620d2225cc436ab0b001e3200e590e3b286ef9ab)
• mrexodia/llama.cpp-openPangu-2.0-Flash — MIT — original openPangu-2.0-Flash port
• upstream PR #27024 by @jinzihao — MIT (merged)
• upstream PR #22525 by @HyeongiJeon — MIT (merged)
• upstream PR #27590 by @SongXiaoXi — MIT (merged; closed upstream unmerged, verified locally)
• ggml — MIT (bundled with llama.cpp)