Ling-3.1-flash-VQ-GGUF

Experimental VQ8 conversion of the official text-only Ling-3.1-flash checkpoint, pinned to a4ccd6b45c002ddb236f9cb77d52cc427482bf19.

This release uses a custom ling31_vq GGUF family. It requires a runtime that implements FORMAT.md; ordinary llama.cpp, Ollama and Transformers loaders do not implement this format. The ds4-dfm-rs work includes artifact readers and numerical reference tools. A complete GB10 serving backend is a subsequent development step.

Selected recipe: ling31-vq8-mix8r13-60r12-mtp13-q4h-v1. Full conversion and final held-out evaluation are in progress. This initial publication contains the model card, format specification and original model assets; GGUF weights will be uploaded after conversion and validation. The precision policy below is the selected release policy; final file hashes and quality results remain to be recorded before release validation is declared complete.

Model and precision

The original model has 72 trunk layers: 63 KDA and nine gated MLA blocks. Layers 0–3 use dense FFNs; layers 4–71 have 512 routed experts and select eight per token. Hidden width is 4096, expert intermediate width is 1280, and the vocabulary contains 157184 tokens. Layer 72 is a separate MLA/MoE MTP block. There is no vision encoder, Engram, PLE, mHC or sparse attention indexer in this checkpoint.

Routed expert weights use vectors of eight consecutive input-axis weights, one independently trained E4M3 codebook shared by all expert projections in each layer, and FP16 row RMS gains. The selected mixed allocation and its screening evidence are recorded in RECIPE.md and recipe.json. Trunk layers 4, 5, 6, 9, 12, 43, 57 and 58 use K8192; the other 60 routed trunk layers use K4096. MTP layer 72 uses a separate K8192 codebook. K4096 uses 12-bit indices (1.5 index bits/weight); K8192 uses 13-bit indices (1.625 index bits/weight). Gains, codebooks and container overhead are additional. The 13th bit is a separate bit plane. Each K4096 codebook was trained separately; it is not a truncated K8192 codebook.

The conservative nonexpert policy quantizes 504 large projections to Q4_K and uses a separately evaluated Q4_K output head. Embeddings remain BF16. Router, bias, norms, KDA f/g, b-projection, convolution, recurrence controls, MLA kv_b and head gates retain their original mixed precision. MTP nonexpert weights retain their original precision. No zchain, GR/RB correction, LoRA, QAT, pruning or distillation is applied. All quantized weights originate directly from the pinned original checkpoint.

Files and optional MTP

release.json is the loading entrypoint. These are independent GGUF parts, not standard numbered GGUF splits; do not concatenate them.

The optional MTP experts use their own K8192 codebook and FP16 row gains. MTP is an approximately 8.17B-parameter block, not merely an extra output head. With MTP disabled, select the trunk feature before mapping or uploading parts; there must be no MTP GPU weights, repacking, cache or workspace allocation. The selected MTP parts contain no duplicate original expert weights.

tensor-manifest.json maps each of the 214025 original tensors exactly once to its preserved or quantized output and lists the generated codebooks. files.json and SHA256SUMS identify the released file contents. Configuration, tokenizer, chat template and special-token files are unchanged original assets. See SOURCE.md for provenance and licensing. The unchanged config.json describes the original FP8 checkpoint; the release recipe and tensor manifest define the stored GGUF precisions.

Numerical and quality scope

The numerical reference separates original-checkpoint inference (A) from inference using the exact stored quantized bytes (B). Original FP8 values use their multiplicative F32 block scales, then the declared BF16 weight boundary. VQ and Q4_K stored values are independently reconstructed before the same boundary. Router and recurrence controls follow the documented F32 contract; SwiGLU clamps follow the pinned serving implementations, including the distinct shared/routed layer lists and unclamped MTP layer.

The final report will record NLL/PPL, KL, top-1 agreement, sampled generations, normal EOS versus cap exhaustion, semantic review, and separate MTP block comparison. Screening measurements are not a final model-quality certificate. FP8 activation execution, gain-after-dot kernels and direct Q4_K serving arithmetic are outside this reference comparison. BF16 reconstruction in the quality reference does not establish serving speed or dequantization cost.

The original EOS is 156895 (<|role_end|>); padding is 156892. The original thinking and tool-call tokens and chat template are preserved. Greedy diagnostics are distinct from official sampling with temperature 1.0, top-p 0.95 and top-k 20.

Memory and context

The selected serialized layout is expected to occupy 118214943744 bytes (110.096246 GiB) across all 71 GGUF parts, including MTP. The trunk accounts for 116415118208 bytes and optional MTP adds 1799825536 bytes. These are exact layout calculations; the final assembly receipt will confirm written sizes and hashes. Tokenizer, documentation and audit manifests are additional files.

The final memory report will use exact selected part sizes, batch-one KDA states and BF16 latent MLA caches. Runtime budgets reserve 4 GiB for OS/driver, 3 GiB for shared workspace and 1 GiB for loading. These are assumptions, not measured GB10 peaks; resident repacking, duplicate CPU weights and persistent BF16 expansion require additional memory. MTP draft/verify workspace must fit the shared allowance. All active compressed weights are counted as resident. Ordinary layer or expert offloading would need separately measured transfer and compute costs; this budget includes no offload savings.

The unchanged configuration specifies max_position_embeddings=262144 and rope_scaling=null. The upstream model card's 1M-context statement does not establish a supported context for this conversion. Neither storage-budget scenarios nor short numerical evaluations qualify long-context serving. GB10 kernels, actual peak residency, throughput, long-running serving and MTP speculative decoding remain to be implemented or validated on the target.