d3-W4A16-G128-AWQ

d3-W4A16-G128-AWQ is an unofficial INT4 weight-only (W4A16) AWQ quantization of d3, the 27B multimodal foundation decision model of Decision 3.0. It takes the same inputs and questions as d3 and returns a probability for every answer, without generating text, with 16.5 GB of weights instead of 52 GB.

Other quantizations of d3: d3-FP8 · d3-W4A16-G128 · d3-NVFP4 · d3-NVFP4-AWQ.

Highlights

• Keeps 97.76% of d3's answers on the 14,031 test requests of tasksource-jev-typed-decisions, with accuracy 83.09% against BF16's 83.25%.
• 16.9 GB of GPU memory after loading instead of 52.3 GB, and 17.5 GB at the peak of the 20 longest of 1,000 text requests (up to 5,928 characters of state) on an MI300X.
• Saves memory, not time: each int4 layer unpacks its weights to BF16 on every call, which is slower than BF16 on an MI300X (60.2 ms against 45.1 ms). With the vLLM Semantic Router max_speed profile on NVIDIA GPUs, inputs of up to 256 tokens take a fused int4 kernel instead.
• Reads images and videos as d3 does: the vision tower is unchanged, and d3's example image and video questions get the same answers as in BF16. Not measured here: the Decision Index suites and the image and video benchmarks.

Quickstart

pip install -r requirements.txt "compressed-tensors>=0.17" triton   # transformers 5.17.0, as d3
hf download subin/d3-W4A16-G128-AWQ --local-dir d3-W4A16-G128-AWQ

import json
import sys

sys.path.insert(0, "d3-W4A16-G128-AWQ")
from d3q.load import load_d3

model = load_d3("d3-W4A16-G128-AWQ", device="cuda:0")
result = model.system_one(
    state="The order arrived damaged yesterday. The customer has a receipt and asks for a replacement today.",
    questions={
        "route": {
            "type": "choice",
            "instructions": "Which team should handle this request?",
            "criteria": {
                "returns": "Refunds, replacements and damaged deliveries",
                "billing": "Payments, invoices and charges",
                "technical": "Product setup and faults"
            }
        },
        "receipt": {"type": "noul", "instructions": "Does the customer have a receipt?"}
    },
)
print(json.dumps(result["answers"], indent=2))

load_d3 is d3's own runtime (d3_runtime.py) with the backbone loader swapped: transformers would otherwise hand the runtime layers it cannot multiply, so d3q/ loads the checkpoint with its weights compressed and replaces each quantized layer with an int4 layer (weights kept packed and unpacked to BF16 by one Triton kernel per call). Prompting, noncausal full attention, the FP32 readout, images and videos are d3's own. AutoModel.from_pretrained(..., trust_remote_code=True) alone does not run these weights. Tested on an AMD Instinct MI300X with PyTorch 2.12, Transformers 5.17.0, compressed-tensors 0.17.0 and Triton 3.7.1: on 100 test rows it gave 99 of 100 of BF16 d3's top answers, with 16.2 GB of weights.

Run with vLLM Semantic Router

Support for FP8 and int4 d3 packages is in vllm-project/semantic-router#5066. With that model runtime:

vllm-srun serve subin/d3-W4A16-G128-AWQ --revision 3af0ab6d3c2c351562d2a940cb1df5698e64ee7b --device cuda:0 --port 8100

curl -s localhost:8100/v1/decisions -H 'content-type: application/json' -d '{
  "state": "The order arrived damaged yesterday. The customer has a receipt and asks for a replacement today.",
  "questions": {
    "route": {"type": "choice", "instructions": "Which team should handle this request?",
              "choices": [{"key": "returns", "description": "Refunds, replacements and damaged deliveries"},
                          {"key": "billing", "description": "Payments, invoices and charges"},
                          {"key": "technical", "description": "Product setup and faults"}]},
    "receipt": {"type": "noul", "instructions": "Does the customer have a receipt?"}
  }
}'

The package keeps d3's package format, with a MODEL_MANIFEST.json that hashes every file of this checkpoint, so the runtime verifies it as it verifies d3. --revision is needed because it is not a built-in model. Without the model runtime, use the Quickstart above.

Evaluation

Text: tasksource-jev-typed-decisions, test split

Every usable row of the test split of tasksource/tasksource-jev-typed-decisions at revision 28c39125 (state of at most 6,000 characters, no repeated options): 14,031 requests of one question each, answered by the vLLM Semantic Router model runtime on one AMD Instinct MI300X, exact profile. Agreement compares the top answer of every question with d3 in BF16 on the same requests (choice and score: highest-probability option; yes / no: p ≥ 0.5). The full record is decision3-quantized.md.

The Jev Decision Index suites, the public vision suite and Perception Test were not run on this checkpoint.

Speed and memory

_{One AMD Instinct MI300X, vllm/vllm-openai-rocm image (PyTorch 2.12.0, Triton 3.7.1, FLA 0.5.2), one request at a time. Graphs replaying: the first 500 of 1,000 rows of the train split, each answered a third time so that its shape's graph replays, with the Decision 3.0 graph capture of #4999. No graphs: those 1,000 rows on the runtime's main branch.}

How it was made

• Calibration: 128 rows of the first train shard of tasksource/tasksource-jev-typed-decisions at revision 28c39125 (state of at most 6,000 characters), rendered with d3's own prompt template. Text only, from the train split: none of the 128 states appears in the test split evaluated above.
• AWQ, layer by layer: for each group of projections that read one input (q/k/v_proj; in_proj_qkv/z with in_proj_a/b; gate/up_proj; down_proj), a per-input-channel scale x^a / w^(1-a) is searched over a in [0, 1) to minimise the squared error of the decoder layer's output with those projections quantized, and folded into what produces their input (the 1 + w RMSNorms, or up_proj's rows), so the BF16 model is unchanged by it. o_proj and linear_attn.out_proj have no linear producer to fold into and take no scale. Each projection except q_proj / k_proj then searches its clipping range per row and group over 10 steps, from its largest weight down to half of it.
• Quantized: the text decoder's mlp.{gate,up,down}_proj, self_attn.{q,k,v,o}_proj and linear_attn.{in_proj_qkv,in_proj_z,out_proj} (400 modules). The vision tower, embeddings, linear_attn.in_proj_{a,b}, conv1d, norms and the FP32 readout (readout.safetensors) keep their published values, except the norms and in_proj_{a,b} that the scales were folded into.
• Tool: awq.py in bench/d3-quant-eval, on one AMD Instinct MI300X (PyTorch 2.12.0, Transformers 5.17.0), in 12 minutes.

License

Apache-2.0, as d3 (LICENSE, NOTICE). Unofficial: not a release of the vllm-sr organization.