Local Inference Lab

Model · Image and text to text

Qwen3.8-27B-NVFP4-QAD

Local Inference Lab

WORK IN PROGRESS A mixed NVFP4/MXFP8 quantization-aware distillation of Qwen3.8-27B, trained for one epoch. The student learns from the original BF16 teacher while its MLP weights are quantized in the forward pass. Distillation updates the MLP weights and text normalization weights to account for quantization error. This is a trained distillation checkpoint, not a post-training conversion of the original weights. Attention/GDN projections and the LM head were frozen in their MXFP8 representations during distillation. Packed NVFP4 and MXFP8 weights reconstruct to the same BF16 weight values used by the student during training. The tokenizer, chat template, generation configuration and…

Model

lil-catalog

Local Inference Lab

Launch manifests for the lil launcher. Each entry is a directory holding one lil.yaml that names the repository holding the weights, optionally pins the commit the local inference lab qualified, and states the serving policy the launcher cannot read from the checkpoint itself. Checkpoint facts come from the weight repository at the resolved commit. Entries of kind: draft describe speculative-decoding drafts and are not served directly. The entry directory name is the launch name: The schema is documented in the launcher repository.