Decision-1-NF4

A pre-quantized NF4 checkpoint in the Worthify Decision-1 family. Download 7.64 GB of weight shards and load the saved NF4 tensors directly. In three short usage checks on an RTX 3090, peak PyTorch-allocated inference memory was 7.26 GiB.

We converted Decision-1 at cab55818cfb6f4163f2b2597bed31cceb6ba2030 with bitsandbytes NF4 and double quantization. This release adds no training. Decision-1 is a single-seed research preview derived from Gemma 4 12B; its original model card describes the training and BF16 evaluation.

Give the model a text state, a question, and 2–16 described options. Read the next-token scores for the supplied option letters to select an option ID. Those scores are conditional on the menu and uncalibrated. The model cannot select an omitted option.

Load the NF4 checkpoint

Use the versions in requirements.txt. We verified one visible CUDA GPU with PyTorch 2.10.0+cu128, Transformers 5.17.0, Accelerate 1.12.0, and bitsandbytes 0.50.2.

pip install torch==2.10.0 transformers==5.17.0 accelerate==1.12.0 \
  bitsandbytes==0.50.2 huggingface-hub==1.31.0 safetensors==0.8.0

Run this Python example with CUDA_VISIBLE_DEVICES=0. The code resolves an immutable repository revision before loading:

import torch
from huggingface_hub import HfApi
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "Worthify/Decision-1-NF4"
REVISION = HfApi().model_info(MODEL_ID).sha

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    revision=REVISION,
    dtype=torch.bfloat16,
    device_map={"": "cuda:0"},
    trust_remote_code=False,
).eval()
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, revision=REVISION)

model.config.use_cache = False
# Required FP32 final-softcap policy; forward hooks do not survive serialization.
model.lm_head.register_forward_hook(lambda _module, _inputs, output: output.float())

The saved quantization_config enables NF4 on reload. Keep the FP32 LM-head hook and disabled cache when scoring decisions. numeric-policy.json records this policy. For the decision prompt and categorical readout, see the Decision-1 source and the authored input rows.

Measured size and reload checks

We hashed the source shards against the pinned public checkpoint, saved tensors with native text-model names, and checked the exported file hashes. A separate process reloaded the NF4 checkpoint and reproduced all three NF4 option-logit vectors exactly. The export receipt, source weight hashes, and reload receipt retain the evidence.

The memory figure covers three short single-row inference calls. It excludes some CUDA runtime and driver overhead and does not establish a minimum GPU capacity or memory use at longer context lengths.

Observed difference from BF16

We scored the same three authored evidence examples in BF16 on an A100 and NF4 on an RTX 3090, using fresh single-row scoring, disabled caching, and the FP32 final softcap in both runs.

NF4 changed the advisory-only example to supported. The paired comparison includes the option logits; BF16 rows, NF4 export rows, and NF4 reload rows retain the complete scores.

These are three usage checks on different GPUs. They do not estimate task accuracy or reproduce Decision-1's BF16 benchmark results. Measure this quantized variant on your own held-out decisions before relying on its selections. Inputs longer than the source model's 2,048-token training prompt cap have not been validated.

Provenance and attribution

The source is Worthify's Decision-1 checkpoint at the immutable revision above, originally trained from google/gemma-4-12B-it at 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7. This repository retains its original model card, Apache-2.0 license, training-data attribution, and original BF16 numeric policy. The original policy describes the source checkpoint; the NF4 loading policy is in numeric-policy.json.

The Gemma-derived weights carry Apache-2.0 terms with applicable attribution. Source datasets retain their separate terms. Quantization adds no dataset or new training examples.