Nick Chapman

INT4 weight-only GPTQ quantization of Qwen/Qwen3.5-4B for memory-constrained single-GPU serving. Also the verifier for the (DFlash2 speculative decoding, τ 3.8 at the serving point / 4.1 greedy vs native MTP 2.9 / 3.0 on the same ladder). - W4A16 (INT4 weights, BF16 activations), group 128, act-order, compressed-tensors (pack-quantized) — vLLM Marlin kernels - Calibrated with llm-compressor dataset-native proportions from (~2.6M effective tokens). The exact rows are hash-pinned in — that pin is the citable calibration generation. (A -cascade suffix on local paths in the lab repo refers to the same generation; this repo carries only the current one and supersedes the initial pre-cascade…

ONNX conversion of jinaai/jina-code-embeddings-0.5b (Qwen2.5-Coder-0.5B finetune, 494M params), laid out like weights under onnx/, tokenizer + config at root, transformers.js compatible. Dynamic int8 quantization was tried (ORT quantizedynamic, QInt8, per-channel + reduce-range, with and without quantpreprocess shape inference) and scored drift ≈ 0.145 — two orders of magnitude over the 1e-3 bar. This GQA-decoder + RMSNorm + last-token-pooling architecture is far more quant-sensitive than BERT-style encoders where int8 dynamic works. Per the sentence-transformers team's guidance, the recommended efficiency path for embeddings is embedding quantization (quantize output vectors to int8/binary…

DFlash2 speculative drafter for Qwen/Qwen3.5-4B. selected at the serving temperature. Beats the model's native MTP by ~+1 τ at every temperature in every measured config, and drafts against the bf16 base as well as against the W4A16-GPTQ quant it was trained on. run-to-run spread (t=1.0 is heavy-tailed: 0.34 observed on identical inputs; t=0.7 spread ~0.1). Numbers above are reported with n and ranges, never as single draws. Ranking is temperature-dependent — see Selection. All six epochs' checkpoints were ranked at three temperatures with repeats. At the serving point (t=0.7 — the same sampling config the training corpus was generated with, temp 0.7 / topp 0.95 / topk 20), τ rises…

The frozen calibration, probe, and ladder sets underpinning every published number in the warp-3090-quant-lab repo (W4A16 GPTQ quants + DFlash2 speculative drafters for Qwen 3.5-family models). These files are deliberately frozen: the headline metrics are defined on these exact rows — - calib-cascade-v1.jsonl (999 rows) — the champion GPTQ calibration set (weightedKL 0.1087 @ 4B, 0.0889 @ 9B, 0.1264 @ 27B trace to it) - probe-v4.jsonl (378 rows) — the fidelity ruler; the weightedKL metric - probe-guard.jsonl (76 rows) — OOD guard (norobots + Belebele, 9 languages) - ladder-20.jsonl (20 prompts) — the acceptance ladder; the τ metric (shipped drafter τ 3.79 at the serving point / native MTP…