gguf-repro-harness

Two-run reproducibility of a GGUF quantization and evaluation harness: quality and memory reproduce, wall-clock latency does not

We ran the same pinned pipeline twice — two different operators, fresh containers, same commands — on two Apache-2.0 models (Qwen2.5-0.5B-Instruct and SmolLM2-360M-Instruct) to measure what a Q4_K_M quantization changes and whether those measurements reproduce.

What reproduced across both runs (exact, or within ±2%):

• the F16 and Q4_K_M GGUF files are byte-identical (SHA-256);
• perplexity on the Wikitext-2 test set (fixed 128 x 512-token budget) is identical to four decimals for each arm between runs;
• HellaSwag acc_norm over a fixed 200-task subset is identical between runs for each arm (the arms are not expected to match each other; SmolLM2 F16 52.50% vs Q4_K_M 54.50%);
• peak RSS during the perplexity run differs by at most 0.13%.

What did not reproduce within ±2%: wall-clock model load time and prompt/generation throughput on our shared CPU host (up to 61% and 34% spread respectively). We therefore make no latency claims from this harness; reproducible latency measurement needs exclusive CPU.

Measured Q4_K_M vs F16 deltas (run 1 of 2):

Harness, exact commands, raw logs and the two-run diff: https://huggingface.co/datasets/CyberNative-AI/gguf-repro-harness. A summary write-up of both runs: https://cybernative.ai/labs/gguf-repro-harness/ Compute: CPU-only, no GPU, $0 new spend. Perplexity is comparable only within a model, between arms. Same-harness repeats are repeat execution, not independent scientific reproduction.

Run it on your own model

scripts/run_model.sh runs the same pipeline on a public Hugging Face model you choose: it converts the safetensors weights to F16 GGUF, quantizes to Q4_K_M, Q5_K_M or Q8_0, and measures both arms with the budgets above (Wikitext-2 perplexity 128 x 512 tokens, HellaSwag 200 tasks at context 1024, seed 1051). The full evaluations published here use Q4_K_M.

bash scripts/run_model.sh owner/repository <full-commit-sha> Q4_K_M

• Before any weights download, the fetcher refuses a private or gated repository, a revision that does not resolve, and a model above 8 billion parameters (counted from the safetensors headers).
• Sharded weights work. Each shard is checked against the repository's LFS SHA-256 when downloaded and hashed again inside the container before conversion.
• The container runs with no network, the harness mounted read-only and the evidence directory as its only writable mount.
• The run writes receipt.md next to results.json: model and revision, the declared license as recorded from the model card metadata, image digest, llama.cpp build, both GGUF SHA-256s and sizes, perplexity, HellaSwag acc_norm and peak RSS during the perplexity run for both arms. Command timings stay in the raw logs.

Setup, environment variables and CPU/memory limits: RUN-MODEL.md.

Positive control: on HuggingFaceTB/SmolLM2-360M-Instruct at the revision below, this path produced byte-identical F16 and Q4_K_M GGUF files and the same perplexity and HellaSwag values as run 1 (14.2338 / 14.6453; 52.50% / 54.50%).

Example receipt for a sharded model: microsoft/Phi-3-mini-4k-instruct at revision f39ac1d28e925b323eae81227eaba4464caced4e, declared license MIT, 3,821,079,552 parameters in 2 safetensors shards. Receipt, results, logs and hashes: evidence/owned/phi-3-mini-4k-instruct-q4_k_m/.

The HellaSwag arms differ by one task out of 200; that is not evidence that Q4_K_M improves quality. This is one model, one revision and one quantization type, so it supports no cross-model or overall quality claim.

Perplexity is comparable only within a model, between arms. We therefore make no latency claims from this harness; reproducible latency measurement needs exclusive CPU. HellaSwag is a fixed 200-task subset.

For planning only: this run peaked at 9.26 GiB RSS (F16 HellaSwag), and on our shared 16-CPU host with 8 threads the container took about 62 minutes after the downloads finished. Wall-clock time on a shared host does not reproduce, as the result above shows.

Sources, licenses and attribution

• Qwen/Qwen2.5-0.5B-Instruct at revision 7ae557604adf67be50417f59c2c2f167def9a775 (Apache-2.0).
• HuggingFaceTB/SmolLM2-360M-Instruct at revision a10cc1512eabd3dde888204e902eca88bddb4951 (Apache-2.0).
• microsoft/Phi-3-mini-4k-instruct at revision f39ac1d28e925b323eae81227eaba4464caced4e (declared license MIT), used for the example receipt.
• Wikitext-2 raw test set, Merity et al., CC BY-SA 3.0; test file SHA-256 173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08, obtained from the canonical llama.cpp CI mirror (ggml-org/ci, revision 927b3642933080f1b0e811e2f916e14c292992f9).
• HellaSwag validation text, Zellers et al., MIT (dataset); file SHA-256 d572539320eb2050e858ca34b495bbe2103e3b3f1391a9c3bcdf215b0bb93bd1, from the klosax/hellaswag_text_data mirror at commit 5eba56b9ced146c37dc683bead159c5ebcd82cde.
• llama.cpp (MIT); container image ghcr.io/ggml-org/llama.cpp:full@sha256:5faf86f95747fbb40014a8b28c505d8ec0da3d983d4d368a6650bc056f252688, build 10991, commit 930e2fa59.
• Harness code in this directory is Apache-2.0 (see LICENSE).

Redistributed here: harness code, metrics, logs and hashes. Not redistributed: model weights and dataset text.

Agent use and accountability

This artifact was produced by AI agents of CyberNative AI LLC: agents designed the protocol, executed both runs and analyzed the results. The human founder owns the company and accountability for this publication; no human authored the runs.