Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-Final-Experimental-GGUF

Experimental GGUF quantization blends —

This repository contains GGUF quantizations of Hermes3.6-35B-A3B-Uncensored-Genesis-Final-Q8_K_P produced with a custom, measurement-driven requantizer developed as a personal study of quantization formats for MoE models. Unlike a standard one-format quantization, each file here uses a per-tensor blend of formats (Q4_K, Q6_K, Q8_0, NVFP4), selected by measuring the element-wise quantization error of every tensor individually and applying a sensitivity policy derived from controlled perplexity experiments.

All files were requantized directly from the Q8_0 source checkpoint. No F16/BF16 intermediate was produced; source tensors are dequantized in memory only.

(Now working on imatrix improvement and further checkpoint shrinking down, later will add MTP versions and vision mmproj

UPD1: Q4_Blend2 was calibrated on diverse imatrix UPD2: All blends are calibrated on diverse imatrix )

Files

Results

Perplexity: llama-perplexity, wikitext-2-raw (wiki.test.raw), n_ctx = 512.Throughput: llama-bench, single RTX 5090 (32 GB), llama.cpp CUDA build, all models measured back-to-back in one session (run-to-run variance is ~5%). Absolute perplexity values are meaningful only as in-table comparisons. |Variant |Size |PPL |pp512 (t/s)|tg128 (t/s)| |--------------------------|--------|-----|-----------|-----------| |Q8_K_P (source) |40.6 GiB|7.009|— |— | |Q4_K_M (reference recipe) |19.7 GiB|7.078|8316 |281 | |NVFP4 (tuned) |28.4 GiB|7.265|8855 |250 | |Q4_blend2 |19.5 GiB|7.050|8566 |247 | |Q4_blend4 |19.1 GiB|7.058|8137 |258 | |Q4_blend5 |18.9 GiB|7.058|8123 |258 |

Summary: blend2 achieves the best perplexity of any file in this repository while being smaller than the reference Q4_K_M; blend5 is the smallest file while retaining a clear perplexity advantage over Q4_K_M at comparable generation speed.

Key findings

Measured on this model class (MoE with hybrid SSM layers), the following sensitivity pattern was established experimentally, tensor class by tensor class:

MoE expert tensors tolerate Q4_K well. Converting the full expert set(~17 GiB) costs ~0.04 PPL — the cheapest quality-per-byte trade in the model.
Always-on tensors are far more sensitive. Attention-class tensors at 4-bit cost roughly 15x more perplexity per byte of saved error than expert tensors; Q6_K is the appropriate target for them.
output.weight and token embeddings convert to Q6_K at no measurable cost (PPL delta below measurement noise).
Norms, routers and SSM state parameters were kept at F16/F32 and are not recommended for conversion.
At equal bitrate, the Q4_K format reaches roughly half the element-wise error of NVFP4 on these weights. An imatrix-weighted scale search closes most of the gap between naive and optimized NVFP4, but not the format-level gap.

Method (outline)

An importance matrix (imatrix) is computed directly from the Q8_0 source.
Each candidate tensor is dequantized in memory and measured: a per-block weighted scale search is fitted, the result is re-encoded through the real(fp16/6-bit/int8) on-disk representations, and the true round-trip error is recorded.
A tensor is converted only if its measured error passes a gate; codecs and tensor groups are assigned according to the sensitivity table.
All codecs were validated by decoding tensors from files produced by the llama.cpp reference quantizer and verifying the error against the Q8_0 source.

Usage

blend2, blend4 and blend5 use only standard GGUF types (Q4_K, Q6_K, Q8_0, F16,F32) and load with any current llama.cpp build.

The NVFP4 file requires a llama.cpp build with GGML_TYPE_NVFP4 support; native FP4 acceleration applies on NVIDIA Blackwell GPUs.

llama-cli -m blend5.gguf -p "Your prompt" -ngl 99

Notes and limitations

All quantizations here derive from an already-quantized Q8_0 checkpoint rather than original BF16 weights. This is close to lossless in practice, but files requantized from BF16 by the original authors may be marginally better.
Perplexity was measured at n_ctx = 512; long-context behavior was not evaluated as of now.
The benchmark table reflects one hardware platform and one llama.cpp build; absolute throughput numbers will differ elsewhere.
All blends are calibrated with diverse imatrix (n_ctx = 6144, chunks = 3500). Feel free to experiment with it.