CLM-v0.1-8B-MLX-8bit

This is the 8bit variant. Also available: czl/CLM-v0.1-8B-MLX-4bit, czl/CLM-v0.1-8B-MLX-6bit, and the bf16 reference.

CLM-v0.1-8B — MLX encoder (8bit)

The encoder half of Contrastive-LM/CLM-v0.1-8B, quantised for MLX on Apple Silicon.

What is in this repo

mlx_lm is generate-only — it has no embeddings entrypoint, and mlx_lm.server routes just /v1/completions, /v1/chat/completions, /v1/models and /health. So clm_mlx supplies the pooling pass over mlx_lm internals: Qwen3Model.__call__ already returns self.norm(h), the post-final-RMSNorm hidden states, which is what vLLM's pooling runner returns in last-token mode.

How to use

# From the Hub, directly (mlx-lm loads MLX repos natively):
mlx_lm.load("czl/CLM-v0.1-8B-MLX-8bit")

# Or serve it for CLM:
python -m clm_mlx.server --model <dir> --port 8092
clm-serve --emb-url http://127.0.0.1:8092/v1/embeddings

Or in process:

from clm_mlx import Encoder
vecs, tokens = Encoder("<dir>", max_tokens=2048).embed(["What causes tides on Earth?"])

Evaluation

Agreement against a bf16 MLX reference of the same encoder in the same runtime, over 23,926 scored System One questions, through the real head stack (argmax(scale · cos), scale = 100.0 — cosine error is amplified 100×).

Pass line. top-1 >= 1.0000 — the measured bf16-vs-bf16 noise floor of this corpus in this runtime — and top-1 (decisive) >= 0.995, where decisive means the reference's own top-1 led by more than 1 nat. A third condition applies: planner accuracy Δ, the change in accuracy against the T-Rex physics planner's label. A variant that holds every decisive decision but still loses measurable accuracy is marked usable rather than recommended.

Against the numbers published in the parent model card

Those probabilities came from vLLM bf16. This runtime reproduces the arg-max on both cases and agrees with an independent llama.cpp bf16 build to a cosine of 0.998680 minimum and 0.999958 mean over all 4,448 texts, so the residual difference is cross-runtime bf16 numerics rather than a pooling or tokenisation error. Tokenisation is byte-identical to AutoTokenizer, with no BOS and no EOS. All quantisation comparisons here are therefore against the same-runtime bf16 reference.

Quantization details

Tool: mlx-lm 0.31.3, one mlx_lm.convert pass per width:

mlx_lm.convert --hf-path Qwen/Qwen3-8B --mlx-path <dir> -q --q-bits N --q-group-size 128

• affine mode, per-group bf16 scale and bias.
• group_size = 32 — and this is the one number that mattered most in this whole exercise, and it is the opposite of both defaults. mlx-lm's own default is 64; Qwen's mlx-community Qwen3-8B-{4,6,8}bit repos use 128. Both are worse here.

MLX affine stores a bf16 scale and a bf16 bias per group, so it spends 32 bits per group on quantisation metadata. That makes the group size a first-class quality knob rather than a rounding detail: gs=128 is 4x coarser than ggml's 32-weight blocks and stores its scales at bf16's 8-bit significand against ggml's fp16 11-bit. Every 0.25 bits/weight spent on finer groups bought back more accuracy than it cost, monotonically:

At group 32 the decisive bucket is perfect at every width and essentially all remaining disagreement sits below 0.10 nats, i.e. in genuine near-ties. The 6-bit gain is inside the noise of the metric and should be read as "no measurable cost", not as an improvement.

The published artifacts are the group 32 ones. The group 64 and 128 models are not shipped; they exist here only as the measurements above.

• --q-bits 6 is a supported affine width, not a silent no-op.

Deliberately not used: mlx_lm.awq and mlx_lm.dwq. DWQ distils 16-bit down to 6- and 8-bit and upstream warns it "often doesn't work well" at those widths; AWQ is statistically indistinguishable from naive round-to-nearest on embedding encoders. See the GGUF repo's quantisation notes for the measurements behind that choice.

Limitations

• The projection heads are encoder-locked to Qwen3-8B last-token-pooled 4096-d embeddings. The pooling and the head checkpoint must both match.
• This is a ranker, not a generator. lm_head is never evaluated under pooling.
• The head stack amplifies cosine error 100×. A 0.001 cosine error is 0.10 nats. Read the confident-decision column, not the mean cosine.
• Right-padding and the last token. clm_mlx.Encoder reads each row's own last real index, never h[:, -1, :]; on a right-padded batch the latter returns the pad token's hidden state for every short row. Attention is causal, so trailing pads cannot change a real token — but only if the right element is indexed.
• truncate_prompt_tokens is honoured server-side. vLLM truncates to fit; llama.cpp answers ERROR_TYPE_EXCEED_CONTEXT_SIZE and 400s instead. CLM sends the field on every request, so ignoring it would break the client on long inputs.
• Probabilities are relative to the candidate set, and quantisation moves the logits by 100× the cosine error — see above.
• Upstream asymmetry, reproduced here: CLM's training recipe keeps the last tokens of a state while the serving path truncates from the head.

License

Apache-2.0, inherited from Qwen/Qwen3-8B and Contrastive-LM/CLM-v0.1-8B. This repo is not gated.

Citation

@misc{kwok2026contrastivelanguagemodels,
  title={Contrastive Language Models: A System One Model for Fast and Generalizable Decision-Making},
  author={Jacky Kwok and Hangoo Kang and Tarun Suresh and Jon Saad-Falcon and Marco Pavone and Christopher Ré and Azalia Mirhoseini},
  year={2026},
  note={Notion Blog},
  url={https://contrastive-lm.notion.site}
}