Kolibri-1-APEX-GGUF

GGUF quantizations of Aleph-Alpha/Kolibri-1 — Aleph Alpha's German/English reasoning model: a 78 B-total / 3.46 B-active mixture-of-experts trained from scratch, with tool calling, an explicit reasoning mode and a 262 K native context. Apache-2.0.

Because only \~3.5 B parameters are active per token, it decodes like a small model while carrying the knowledge capacity of a large one: \~51 tokens/s on a DGX Spark (GB10) at 4.6 bpw.

Requires a forked llama.cpp — upstream CANNOT load this model

Stock llama.cpp will fail to load these files at the architecture check. As of 2026-10-07 upstream has no support for the kolibri1 architecture (0 hits at origin/master b86d2f075).

You need this fork and branch:

brywil/llama.cpp — branch kolibri1

bash git clone -b kolibri1 https://github.com/brywil/llama.cpp llama.cpp-kolibri cd llama.cpp-kolibri cmake -B build -DGGML_CUDA=ON # drop -DGGML_CUDA=ON for a CPU-only build cmake --build build -j --config Release

Verified at commit 68fa28b56, two commits on top of upstream 22397c31a. They add the architecture (converter, tensor loading, compute graph), one new MoE gating mode, and accept kolibri1 as a tokenizer pre-type name; nothing else is changed. This fork is maintained as the home for Kolibri support — it is not a staging branch for an upstream PR.

Every llama-server / llama-cli invocation on this page means the binary from that build tree.

Architecture

arch             kolibri1        78.1 B params, vocab 128,000
layers           50
FFN MoE          384 experts, top-6, +1 shared (ungated); expert ffn 512
routing          select top-6 on logits + per-expert bias, weight by sigmoid(logits), no renorm
attention        48 heads / 4 KV, head_dim 128, per-head q/k RMSNorm
                 4 sliding-window layers (window 513, RoPE) : 1 full-attention layer (no RoPE)
norms            sandwich: pre- and post-norm around both attention and FFN
context          262,144 native

Two properties matter for running it:

• KV cache is cheap. Only the 10 full-attention layers grow with context — \~20 KiB per token at f16 (10 × 4 × 128 ÷ 256), so the full 262,144-token context costs about 5 GiB. The 40 sliding-window layers hold a small constant-size cache. Decode speed stays nearly flat as context grows.
• Every K-quant is legal. The hidden size (2560) and expert width (512) are both divisible by 256, so no tensor is silently substituted with a block-32 type — the files contain exactly the types listed below.

Files

Upload in progress — the files appear here as they finish, largest-value first.

KLD and top-1 are measured against the BF16 model's own logits (mean KL divergence per token, and how often the quant's top token matches BF16's), on 20 × 512 tokens each of WikiText-2 test and a code-heavy corpus. Lower KLD / higher top-1 is closer to the original.

Which one to take

• uniform-Q4_K (41.89 GiB) — the default. Closest to BF16 on every measure; 42/42 on the agentic suites below. Fits a 64 GB machine with the full 262 K context.
• uniform-IQ3_S (32.55 GiB) — the \~3.5 bpw pick. Fits 48 GB with plenty of context.
• APEX-i-mini (25.30 GiB) — for \~32 GB of memory, with a moderate context.
• APEX-2.4bpw (22.70 GiB) — the smallest, and it also cleared every agentic coding task (42/42, 253/253) while decoding faster (\~57 t/s). Prose suffers first (81% top-1 on WikiText), but code holds up far better (91.6% top-1) — true at every size here. For coding on a tight budget, this is the one.

APEX-i-quality and APEX-i-compact use the edge-heavy APEX band recipe at roughly the same sizes as the two uniform files. They measured slightly behind them on mean KLD and top-1, inside the noise at \~4.4 bpw (see Limitations 2), and they keep a somewhat better code p99. APEX-i-quality is also agentic-verified: 42/42 at both effort high and low on a 16 GB GPU (below). It is \~2 GB smaller than uniform-Q4_K, which matters when experts spill to system RAM.

What is in each file

The expensive-to-get-wrong tensors are protected in every file, and apart from one exception (below) only the routed experts change between files:

The exception is APEX-2.4bpw, which to reach its size also drops 32 of the 40 sliding-window layers' attention projections to Q3_K (8 stay at Q6_K; the full-attention layers stay at Q8_0).

Routed experts (96.7% of the parameters):

Every file's tensor-type histogram was read back from the shipped bytes and checked against its recipe.

The importance matrix was computed on a bilingual, code-inclusive corpus (332 × 512 tokens: \~40% English prose, \~20% German, \~40% code). Experts that calibration never routes to (996 of 19,200, 60% of them in the first three layers) are given their layer's mean importance rather than a flat default.

Agentic coding — 42/42 tasks, 253/253 test cases, at 4.6, 4.4 and 2.5 bpw

The model drives a real tool-use loop (OpenCode: read, write and run files) and is graded by a fresh pytest it never sees, at the model card's sampling (temp 1.0, top-p 0.97, top-k 128), reasoning effort high, on a DGX Spark (GB10). Two files, identical settings:

Zero failures and zero truncated responses on both. Each task ran in a single uninterrupted context — the agent's history was never summarised or compacted — with a 262,144-token window; the largest context any request actually used was 58,529 tokens.

Half the bits cost nothing measurable here. The 2.5 bpw file scored identically and spent about the same number of tokens (one run each — read that as a tie, not a saving); it finished sooner because it decodes faster.

Reasoning effort low is worth trying. At --reasoning-effort low, uniform-Q4_K also scored 42/42 (253/253) with 303,509 output tokens and 2.1 h; the saving comes mostly on easy tasks.

On a 16 GB GPU: APEX-i-quality, effort high and low

The same suites, on an RTX 4060 Ti 16 GB with the experts mostly in system RAM (2× Xeon E5-2670 v2, 125 GB):

Zero failures and zero truncations again. On this box, effort low spent fewer output tokens than high on 32 of 42 tasks (median ratio 0.80; two-sided sign test p = 0.0009), one run each. Token totals are not comparable to the GB10 table above: different hardware and different file.

A full score is the entry bar, not the whole result — these suites are built to saturate for a competent coder. What separates models that all pass is how many tokens they spend, and that needs repeated runs to compare; the token column above is one run.

Running it

These commands need the FORKED binary — stockllama.cpp cannot load these files

If you scrolled straight here to copy a command: kolibri1 has no support in upstream llama.cpp. Build the fork first:

bash git clone -b kolibri1 https://github.com/brywil/llama.cpp llama.cpp-kolibri cd llama.cpp-kolibri && cmake -B build -DGGML_CUDA=ON && cmake --build build -j

Details and the verified commit are at the top of this card.

./llama.cpp-kolibri/build/bin/llama-server \
  -m Kolibri-1-uniform-Q4_K.gguf \
  --ctx-size 131072 --parallel 1 -ngl 999 \
  --jinja --reasoning-format deepseek --reasoning on --reasoning-effort high \
  --temp 1.0 --top-p 0.97 --top-k 128

• Always pass --ctx-size. The model's metadata declares its 262,144-token native context and llama.cpp otherwise allocates all of it. At \~20 KiB/token that is only \~5 GiB here, but set it deliberately.
• It is a reasoning model. The chat template emits <think>…</think>; with --reasoning-format deepseek the reasoning arrives in reasoning_content and content stays clean. Thinking can be long, so give it a generous output budget — a response cut off mid-thought arrives with empty content.
• Tool calling works through llama-server's OpenAI-compatible endpoint with --jinja (the embedded template); the agentic results above used exactly this path.
• Use the sampling above — it is the original model card's recommendation.

Speed on a DGX Spark (GB10, unified memory), single stream, measured over each full agentic run:

On a 16 GB GPU, with the experts in system RAM

Measured on an RTX 4060 Ti 16 GB (PCIe Gen3 x8) with 2× Xeon E5-2670 v2 (AVX, no AVX2) and 125 GB of RAM, using APEX-i-quality, on a \~5 K-token code prompt (median of 7 runs):

taskset -c 0-9,20-29 ./llama.cpp-kolibri/build/bin/llama-server \
  -m Kolibri-1-APEX-i-quality.gguf --parallel 1 \
  -ngl 99 --n-cpu-moe 42 -c 131072 -ctk q8_0 -ctv q8_0 -fa on \
  -ub 2048 -b 2048 -t 20 -tb 20 --numa isolate --load-mode none \
  --jinja --reasoning-format deepseek --reasoning on --reasoning-effort high \
  --temp 1.0 --top-p 0.97 --top-k 128

• Use --load-mode none. This llama.cpp has no --no-mmap, and its default (auto: mmap, plus lazy reads of large tensors) cost 45% of prompt speed here.
• Generation is CPU-bound. It barely moves across --n-cpu-moe 42–44, so take the value that leaves \~1.5 GB of VRAM free. f16 KV adds nothing and costs an expert layer, and 10 threads match 20.
• Prompt processing costs \~5–8 s per batch, because each batch streams the CPU-resident experts to the GPU. Larger -ub means fewer batches.
• taskset / --numa isolate pin to the GPU's NUMA node. Adjust the CPU list for your machine, or drop it on a single socket.
• uniform-Q4_K is \~2 GB larger. Expect it to need --n-cpu-moe about 2 higher at the same context.

Limitations

1. Agentic results are for uniform-Q4_K, APEX-i-quality and APEX-2.4bpw. The other three files were checked for coherent German and English generation and a small executed coding test, and measured by KLD / top-1, but were not run through the agentic suites. Given that the smallest file scored 42/42, the in-between sizes are very likely fine — but that is an inference, not a measurement.
2. The KLD figures rest on 10,240 tokens per corpus. Mean KLD carries roughly ±0.005 at that size, so the gap between uniform-Q4_K and APEX-i-quality is under 2σ; the gap at \~3.5 bpw is clearer.
3. The allocation is structural, not measured. The APEX files band experts by depth; no per-tensor sensitivity map exists for this architecture yet. That is the most likely route to a better file below 3 bpw.
4. Perplexity is not quoted as a quality figure. BF16 scores 27.96 on raw WikiText-2 — normal for a reasoning chat model reading unformatted text, and not comparable to other models' numbers. The KLD and top-1 columns, measured against this model's own BF16, are the meaningful ones.

Provenance

• Converted from the original BF16 safetensors with the fork's converter. The port was validated layer by layer against an independent fp32 PyTorch implementation written from Aleph Alpha's own vLLM model code: identical expert selection in every layer sampled through layer 16, full-layer outputs agreeing to \~1e-3 at every layer, and the same top-9 next-token ranking.
• Quantized with quantkit (dynamic_apex.py recipes, the memory-bounded banded imatrix generator serialized_gen_kolibri.py) and the fork's llama-quantize. The imatrix was cross-checked against stock llama-imatrix (dense tensors correlate 0.9996–1.0000, routed experts median 0.995–0.999).
• KLD reference logits from the BF16 GGUF; WikiText-2 test split, 1,290,590 bytes, sha256 173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08.
• Licensed Apache-2.0, as the original. All credit for the model to Aleph Alpha.