GLM-4.7-Flash-Hauhau-Aggressive-W6A8

Native compressed-tensors W6A8 checkpoint published by Xananthium. Credit and upstream terms: HauhauCS/GLM-4.7-Flash-Uncensored-HauhauCS-Aggressive. The source model card is preserved in README_ORIGINAL.md. Quantization provenance and retained high-precision components are recorded in conversion-receipt.json.

Six bits on a 3090? Hell yes. >:)

I wanted W6A8, so that is what we built: six-bit stored weights, dynamic eight-bit activations, and vLLM doing the serving. Humming packs the INT6 values across INT32 word boundaries and unpacks them for the 3090's supported INT8 matrix math. The cards do not need a native six-bit Tensor Core instruction. The extra unpacking and scale work is part of the cost, and the speed tests include it.

The six-bit weights use symmetric groups of 128. Embeddings, routers, norms, vision, and designated sensitive components keep higher precision. BF16 remains the surrounding model dtype; eligible quantized linears actually use INT8 activations, with silent activation fallback disabled. TurboQuant Q4 is a separate KV-cache choice when the architecture supports it.

Accuracy matters too. A smaller checkpoint gets us memory back, but the model still has to follow the damn instructions, call tools, and get the security answers right. That is why the results include the knowledge quiz, security-policy checks, long-context recall, real Claude Code tool use, and concurrency. Failed checks stay in the report.

The measured numbers for this checkpoint are below. A six-bit speed gain over an eight-bit baseline is unmeasured unless a matched comparison report is attached.

Measured on two RTX 3090 24 GiB cards, vLLM 0.31.0

Reports in results/ contain the exact per-model profile, budgets, formatting metrics, failed checks and client integration outcomes. Counting successful test programs does not mean all accuracy checks passed. Token throughput includes reasoning tokens; it is not visible-answer throughput. The policy assessment is synthetic, and lengthy reasoning can exhaust the output budget. Eight independent long prefixes were replayed; this was not a fresh eight-turn conversation. CyberMetric is a small public knowledge quiz that may overlap training data, and does not establish practical security competence or a broad model ranking. Questions and private reasoning traces are not redistributed.

vLLM loading

hf download Xananthium/GLM-4.7-Flash-Hauhau-Aggressive-W6A8 --local-dir ./checkpoint
pip install --no-deps ./checkpoint/runtime/humming-compile-compat
LD_LIBRARY_PATH="$VIRTUAL_ENV"/lib/python3.12/site-packages/nvidia/cu13/lib \
LOCAL_HUMMING_COMPILE_COMPAT=1 \
VLLM_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"int8","group_size":0,"allow_fallback":false}' \
vllm serve ./checkpoint --quantization humming --linear-backend humming --moe-backend humming --tensor-parallel-size 2 --dtype bfloat16 --trust-remote-code --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 --kv-cache-dtype bfloat16 --max-model-len 147456

The loading recipe requires the architecture and quantization support in vLLM 0.31.0. Consult results/serving-profile.json for tested sampling and memory settings. Model-side INT8 activations apply only to eligible quantized linears; embeddings, routers, norms and designated sensitive components retain higher precision. KV cache precision is a separate setting. GLM MLA does not use TurboQuant in this version. Retained MTP weights do not imply speculative decoding was tested.