K2-Horizon-3.7B-Q4_0-ARM-GGUF

K2-Horizon-3.7B, pure Q4_0 GGUF for ARM phones

What it is: IFM/K2-Horizon-3.7B as a single llama.cpp file in which every weight matrix, including the 250,624-token embedding table and output head, is Q4_0 (llama-quantize --pure), made from IFM's BF16 GGUF. The official GGUF repo has no Q4_0 file. On ARM CPUs with the i8mm or dotprod instructions llama.cpp runs Q4_0 faster than the K-quants: on a Snapdragon phone this file reads prompts 1.4-1.7x as fast as IFM's Q4_K_M, but generates only 1.04-1.11x as fast, and it is 9 % smaller.

Read this first: the accuracy trade-off. For this model, pure Q4_0 costs a lot of accuracy for a modest speed gain. Against IFM's Q8_0 it deviates about twice as much as IFM's Q4_K_M: mean KL divergence 0.149 vs 0.074, a different most likely next token 20.5 % vs 13.7 % of the time, perplexity +14.9 % vs +8.1 %. Use it only if prompt latency matters more to you than answer quality. Otherwise use IFM's Q4_K_M, or their Q5_0 / Q6_K, which IFM's own benchmarks show much closer to the original.

I tested it as an on-device chat model for an Android voice assistant (llama.cpp, CPU only).

The file name follows the naming of IFM's GGUF repo (K2-Horizon-4B-*.gguf).

Base model, credit and licence

• Model: K2-Horizon-3.7B by IFM, the small dense member of the K2-Horizon family (3.7B core parameters plus a large vocabulary; 5.06B parameters in the file). All credit for the model goes to IFM; please read their model card for intended use, evaluations and safety notes.
• Source file: K2-Horizon-4B-BF16.gguf from IFM/K2-Horizon-3.7B-GGUF at commit 78a0635e8f19b880e468a2fc82923ee6c8a469e4 (sha256 372ecf99…43b1, checked).
• Licence: Apache-2.0, as the base model (no extra use policy is listed on IFM's cards). LICENSE is the Apache-2.0 text; NOTICE lists what was changed. Only the weight format changed: tokenizer, chat template and all other metadata are identical to IFM's files (only general.file_type differs). This repo is not affiliated with IFM.
• Needs a llama.cpp build with K2-Horizon support (IFM: release b11454 or newer).

@misc{k2horizon2026,
  title  = {Introducing K2 Horizon: Frontier Performance, Radically Open},
  author = {{IFM Team}},
  year   = {2026},
  url    = {https://ifm.ai/blog/k2/},
}

Why Q4_0 is faster on ARM (repacking)

When llama.cpp's CPU backend loads a model it can repack the weights: it rearranges them into an interleaved layout so that the matrix-multiply instructions of current ARM cores (SMMLA from the i8mm extension, SDOT from dotprod) process several rows at once. For Q4_0 on an i8mm core it uses 4x8 interleaved blocks; on this phone repacking alone makes Q4_0 prompt processing about 2.3x faster (measured on a 2B model with the same build).

Recent llama.cpp builds (this one included) repack Q4_K and Q6_K too, so IFM's Q4_K_M is repacked as well, but it still trails pure Q4_0 on prompt speed: Q4_0 is the simplest 4-bit block format (32 weights and one scale), while Q4_K_M mixes Q4_K super-blocks (packed 6-bit sub-scales and minimums) with Q6_K tensors. For generating tokens the gain is small: every generated token reads about 2.5-2.8 GB of weights (all of them except the embedding table), and on this phone that keeps generation near 8-9.5 t/s whatever the 4-bit format.

--pure matters for this model: the default Q4_0 recipe keeps the output head (0.64B parameters) in Q6_K. I made that variant too ("Q4_0 + Q6_K head", 3.02 GB) and measured it below. It is more accurate than pure Q4_0 (mean KLD 0.118 vs 0.149) but generated 8-14 % slower in the same session, which is about the speed of IFM's Q4_K_M, and Q4_K_M is more accurate still (0.074). So I did not publish it: for speed use this file; for accuracy use IFM's Q4_K_M or larger.

The cost: the repacked copy lives in ordinary app memory (see Memory), and Q4_0 is clearly less accurate than Q4_K_M (about twice the KL divergence, see Accuracy).

Speed on the phone

Nothing Phone (4a) Pro, Snapdragon SM7750 (1x Cortex-A720 at 2.8 GHz, 4x A720 at 2.4 GHz, 3x A520 at 1.84 GHz), 12 GB RAM. llama.cpp b11671 (commit ba3c05f), CPU backend built on the phone with GCC 15.2 (-DGGML_NATIVE=ON, i8mm + dotprod + bf16 detected) and without OpenMP (-DGGML_OPENMP=OFF). llama-bench, 3 repetitions, flash attention on, KV cache q8_0, --poll 100, --cpu-strict 1.

Back-to-back session, 2026-10-16 (phone charging and warm: the thermal governor capped the big cores at 1.9 GHz, 2.2 GHz in the first 5-core Q4_0 run; about 0.6-0.8 core of other load), run in the order Q4_0, Q4_0 + Q6_K head, Q4_K_M:

A second back-to-back pair on 2026-10-18 (5 cores, phone on low battery and warm) gave Q4_0 52.2 ± 9.1 / 8.14 ± 1.87 t/s and Q4_K_M 37.3 ± 0.2 / 7.83 ± 0.45 t/s (1.40x / 1.04x; the Q4_0 run was noisy). Generation appears limited by memory bandwidth on this phone (about 25 GB of weights read per second), so the 4-bit format changes it little.

Other measurements with this file (same build and phone):

In llama-server (4 big cores, system prompt cached) a short answer starts after a median 0.66 s (first full sentence 2.7 s) with generation around 7.4 t/s; the model loads in 7.6 s.

Accuracy

Next-token distribution vs a high-precision reference

llama-perplexity --kl-divergence on the wikitext-2-raw test split (the file llama.cpp's scripts/get-wikitext-2.sh downloads), first 27 chunks of 512 tokens (6,885 scored tokens; 27 chunks keep each run under 18 minutes on the phone). Reference: IFM's Q8_0 GGUF from the same commit (K2-Horizon-4B-Q8_0.gguf, sha256 8203991019c3…7d3), used because the 10 GB BF16 file was too large for the phone run; IFM's own table shows Q8_0 within 1.8 points of BF16 on average.

(Mean KLD: the average KL divergence between the reference's and the quantized model's next-token distributions; 0 means identical. "Same top token": how often both pick the same most likely next token. Uncertainties are llama.cpp's standard errors. All four runs used --no-repack so that they fit next to the phone's other apps; repacking changes the memory layout of the weights, not their quantized values.)

Reading: this is a real accuracy cost. Pure Q4_0 is about twice as far from the reference as IFM's Q4_K_M (mean KLD 0.149 vs 0.074), picks a different most likely next token 20.5 % of the time (Q4_K_M: 13.7 %), and raises perplexity by 14.9 % (Q4_K_M: 8.1 %). The loss is larger than for a 2B Qwen model quantized the same way (mean KLD 0.086), so treat this file as the fast option, not the accurate one. Keeping the 250k-token output head in Q6_K (llama-quantize's default recipe for Q4_0) recovers part of it (0.118, 83.6 % same top token). The reference is itself Q8_0, so all of these numbers slightly understate the distance to the original BF16 weights.

Assistant answers

This file is the one I tested in a private 68-prompt eval of the voice-assistant use case: 40 everyday prompts and 28 harder maths, reasoning and instruction-following prompts with written reference answers, thinking off, the chat settings below, one sample per prompt. Automatic answer checks (the expected answer appears, or the exact requested format is kept) passed on 13 of 16 checkable everyday prompts and 13 of 20 checkable hard prompts. With thinking off it made arithmetic slips (for example 15 % of 80 answered as 120).

A Q4_0 vs Q4_K_M comparison on this eval was not run, so the accuracy cost of Q4_0 for this model is known from the KL-divergence numbers above and from IFM's own quantization table below. (With 20 checkable prompts and sampling at temperature 1.0, a difference of one or two answers would be within run-to-run noise anyway.)

IFM's published quantization results (context)

IFM's GGUF card reports these averages over IFEval, GSM8K, MBPP, MMLU-Pro, GPQA-Diamond, BBH and AIME 26 (their numbers, not re-measured here):

IFM did not publish a Q4_0. This Q4_0 is further from the reference than Q4_K_M in the KL-divergence test above, so on these benchmarks (thinking at high effort, up to 64k tokens) it would likely score at or below IFM's Q4_K_M row; that was not measured.

Memory

Repacking puts a rearranged copy of the weights in anonymous (non-reclaimable) memory; the memory-mapped GGUF pages are only page cache that the OS can drop. Measured in llama-server right after load: about 5.0-5.4 GB RSS = 3.0 GB anonymous (repacked weights and buffers) + 2.0-2.3 GB file pages; llama-bench peaks were 4.9-5.2 GB for this file, 5.5 GB for the Q6_K-head variant and 5.8 GB for Q4_K_M. Plan for about 5.5 GB free RAM; on a 12 GB phone with normal apps open that is often not available. With --no-repack the weights stay file-backed (about 4.3 GB free is enough) but prompt speed drops.

How it was made

# source: IFM/K2-Horizon-3.7B-GGUF @ 78a0635e8f19b880e468a2fc82923ee6c8a469e4, K2-Horizon-4B-BF16.gguf
llama-quantize --pure K2-Horizon-4B-BF16.gguf K2-Horizon-4B-Q4_0.gguf Q4_0 4      # llama.cpp b11671 (ba3c05f)
# the "Q4_0 + Q6_K head" comparison file: the same command without --pure

Accuracy commands (see eval/ for the full logs):

llama-perplexity -m K2-Horizon-4B-Q8_0.gguf -f wikitext-2-raw/wiki.test.raw -c 512 -b 512 --chunks 27 -t 4 \
  --no-repack --kl-divergence-base k2-q8_0.kld
for m in K2-Horizon-4B-Q4_0 K2-Horizon-4B-Q4_K_M K2-Horizon-4B-Q4_0-mix; do
  llama-perplexity -m $m.gguf --kl-divergence-base k2-q8_0.kld --kl-divergence -c 512 -b 512 -t 4 --no-repack
done

Recommended llama.cpp settings (ARM phone)

llama-server -m K2-Horizon-4B-Q4_0.gguf --jinja -c 4096 \
  -t 5 -C 0xF8 --cpu-strict 1 --poll 100 -fa on -ctk q8_0 -ctv q8_0

• Threads on the big cores only. -C 0xF8 -t 5 uses cpus 3-7 (four A720 cores + the prime core of the SM7750), the fastest generation measured; -C 0x78 -t 4 leaves the prime core for the app's UI and audio at a cost of about 8 %. Use the mask of your phone's big cores; never include the little cores.
• Build without OpenMP (-DGGML_OPENMP=OFF): ggml's own thread pool with polling was 2.3x faster at generation than an OpenMP build of the same commit (measured with a 2B model).
• Keep repacking on (the default), -fa on, KV cache q8_0 (+8 to +17 % generation in two comparisons here).
• The GPU (OpenCL, Adreno) did not help generation on this phone and took 58 s to load.

Chat settings that worked for a short-answer voice assistant

• Thinking off for fast replies: chat_template_kwargs: {"enable_thinking": false} (the template then renders an empty think block). IFM's recommended setting for accuracy is the opposite (reasoning_effort: "high", at least 32k output tokens); with thinking off the model starts answering in under a second but makes more arithmetic slips.
• Sampler: IFM's temperature 1.0, top_p 0.95.
• System prompt: short: a private on-phone assistant, one to three sentences, no live facts. A stricter "voice" prompt (no markdown, numbers spelled out) removed markdown but made arithmetic worse in my tuning set, so I kept the short one.
• Strip any <ifm|think> / </ifm|think> tags from the visible answer: a few leaked through with thinking off.

Limitations

• Less accurate than Q4_K_M: see the tables. If you have the RAM and speed headroom, IFM's Q5_0 / Q5_K_M / Q6_K files are much closer to the original (their own benchmarks above).
• It needs about 5.5 GB of free RAM with repacking on.
• Generation stays at about 8-9.5 t/s on this phone whatever the 4-bit format; Q4_0 mainly buys prompt speed here.
• Speeds are for one phone and one llama.cpp build. x86 CPUs also repack Q4_0 (AVX2) but were not measured.
• Accuracy here was measured on English wikitext and a small private English eval only.
• All limitations listed by IFM apply (it may produce inaccurate or unsafe content; not tuned for moderation).

Files and checksums