Activation-aware asymmetric INT4 weight-only quantization of Mapika/decider-4b, revision 99d7fd5bfcc4d11ba37239dafe0ed653803819d3. This is a typed decision model: read probabilities over specified options in one forward pass. It is not trained for conversational text generation.
Recipe
llm-compressor 0.12.0.1, Transformers 5.10.1, PyTorch 2.12.0+cu130. AWQ scaling followed by QuantizationModifier, scheme W4A16_ASYM, group size 128, asymmetric integer weights, BF16 activations. save_compressed=True produces physical weight_packed tensors in compressed-tensors format.
All Linear layers are targeted except lm_head, linear_attn.in_proj_a, linear_attn.in_proj_b, and MTP (if present). Embeddings, norms, convolution and sensitive delta-net gates stay unquantized. Native Qwen3.5 hybrid attention AWQ mappings are used. No vision tower is present.
The source ties input embeddings and the LM head. The serializer's bit-identical duplicate lm_head.weight was omitted while preserving tie_word_embeddings: true, as in the source checkpoint. The final file was reloaded and its probabilities checked again after this storage-only deduplication.
Calibration uses 256 individual Choice/yes-no prompts from the author's public custom_questions.jsonl, seed 42 for source-state split and seed 43 for prompt sampling, max sequence length 2048. Prompts follow the canonical plain Context / Question / Options / Answer: ( layout. calibration.json contains the exact prompts. quantize.py and recipe.yaml record the procedure.
The exact source data file is included (SHA256 f8a1430b7bc405d4ecfb3454c981e4cd70a21cc4e5b75868020629c9d4589659). eval_inputs.json and baseline.json preserve the test inputs and BF16 reference probabilities. No source states are shared by calibration and validation.
Validation
Safetensors: 8.412 GB → 3.134 GB, 2.68× smaller, 200 packed weight tensors. Sizes are decimal GB, weight files only.
64 prompts from separate source states, excluded from calibration, were evaluated with Transformers before quantization and after reloading the saved checkpoint. Source type-specific temperatures were preserved.
| Metric |
Result |
| BF16 accuracy against teacher labels |
100.00% |
| AWQ accuracy against teacher labels |
100.00% |
| AWQ/BF16 decision agreement |
100.00% |
| Mean absolute option probability difference |
0.017986 |
This small smoke check is not a full benchmark or a probability calibration study. Teacher labels are not guaranteed ground truth. validation.json includes measured results and an example. Original temperatures are preserved, not refitted for INT4. Quantization may alter both decisions and confidence.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "Ar4ikov/decider-4b-AWQ-ASYM-W4A16"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto").eval()
prompt = ("Context:\nMy card was charged twice for the same purchase.\n\n"
"Question: Which department should handle this?\nOptions:\n"
"(A) billing\n(B) technical support\n(C) sales\nAnswer: (")
enc = tok(prompt, return_tensors="pt").to(model.device)
letters = [tok.encode(x, add_special_tokens=False)[0] for x in "ABC"]
with torch.inference_mode():
logits = model(**enc).logits[0, -1, letters].float()
print(dict(zip(["billing", "technical support", "sales"],
(logits / 1.110).softmax(-1).tolist())))
Transformers can decompress packed weights for this path, so this smoke test does not demonstrate INT4 runtime memory savings. Exact runtime VRAM depends on kernel support, KV cache, activations and context length; it was not inferred from Transformers loading memory.
The original decider/ inference helpers, decider_config.json and prompt files are included. Refer to the original model card for typed multi-question APIs, training, licensing and model limitations. Original benchmark scores have not been remeasured for this checkpoint.
For a native packed runtime, load this repository in vLLM (compressed-tensors is auto-detected):
vllm serve Ar4ikov/decider-4b-AWQ-ASYM-W4A16 --dtype bfloat16 --max-model-len 8192
Use the canonical decision prompt and restrict scoring to the supplied option-letter tokens. Ordinary chat responses do not implement decider's typed decision API.
To reproduce in a downloaded repository folder, install the versions above plus datasets and accelerate, set DECIDER_QUANT_ROOT to the absolute folder path, run prepare.py, then quantize.py. This downloads the pinned BF16 source into base/ and writes the compressed checkpoint to out/. GPU calibration needs enough free memory for BF16 weights and working buffers.
Native packed inference was also tested with vLLM 0.29.0 on RTX 3090, Marlin weight kernels, PyTorch sampler (VLLM_USE_FLASHINFER_SAMPLER=0), eager mode, context limit 2048, 64 constrained one-token decisions. Accuracy: 100.00%; agreement with the BF16 Transformers reference: 100.00%. The weight loader reported 3.05 GiB of GPU memory for model loading, excluding KV cache and other runtime buffers. See vllm_validation.json.
