Qwen3.8-35B-A3B-Distill-GGUF by Nguyen Quoc Vinh: Open Model

Model Card

By Nguyen Quoc Vinh, published under apache-2.0, revision dd3197c74980.

Community GGUF quantizations of empero-ai/Qwen3.8-35B-A3B-Distill. If this GGUF made your day easier, a coffee would make mine. I build and test these releases myself. Your coffee helps keep me going. Thank you for supporting this work. This release is based on the upstream Qwen3.8-35B-A3B-Distill model card. The model is a text-generation MoE model with 35B total parameters and approximately 3B active parameters per token, using 256 experts with 8 routed experts. It uses hybrid attention components and has a native context length of 262,144 tokens in the upstream configuration. The GGUF files are intended for local inference with llama.cpp and compatible runtimes. The upstream model card…

Read Nguyen Quoc Vinh's full model card

Community GGUF quantizations of empero-ai/Qwen3.8-35B-A3B-Distill.

If this GGUF made your day easier, a coffee would make mine.
Send a coffee
I build and test these releases myself. Your coffee helps keep me going.
Thank you for supporting this work.

About Qwen3.8-35B-A3B-Distill

This release is based on the upstream Qwen3.8-35B-A3B-Distill model card. The model is a text-generation MoE model with 35B total parameters and approximately 3B active parameters per token, using 256 experts with 8 routed experts. It uses hybrid attention components and has a native context length of 262,144 tokens in the upstream configuration.

The GGUF files are intended for local inference with llama.cpp and compatible runtimes. The upstream model card and runtime support should be consulted before using long-context, tool-calling, or other task-specific features.

No training or fine-tuning was performed for this release. These are community GGUF quantizations, not an official Qwen3.8 release or endorsement by the upstream authors.

Files

File	Approx. size	Notes
`Qwen3.8-35B-A3B-Distill-Q8_0.gguf`	35.22 GiB	Highest published precision in this package
`Qwen3.8-35B-A3B-Distill-Q6_K.gguf`	27.20 GiB	High-fidelity option
`Qwen3.8-35B-A3B-Distill-Q5_K_M.gguf`	23.60 GiB	High-fidelity / memory-balanced option
`Qwen3.8-35B-A3B-Distill-Q4_K_M.gguf`	20.22 GiB	Balanced default candidate
`Qwen3.8-35B-A3B-Distill-Q3_K_M.gguf`	15.98 GiB	Experimental lower-bit option
`Qwen3.8-35B-A3B-Distill-Q2_K.gguf`	12.34 GiB	Experimental lower-bit option

Fidelity measurements

The measurements below compare each published GGUF with the locked BF16 GGUF source. They are next-token fidelity measurements, not a general capability benchmark. Values are averaged over WikiText-2 valid and test, using 2 chunks per split, context length 2048, and the same llama.cpp runtime. Lower KLD, ΔPPL, and RMS Δp indicate closer next-token behavior to BF16; higher Top-1 agreement is better.

File	Mean KLD ↓	Top-1 vs BF16 ↑	ΔPPL	RMS Δp
`Qwen3.8-35B-A3B-Distill-Q8_0.gguf`	0.002454	97.972%	-0.068%	1.396%
`Qwen3.8-35B-A3B-Distill-Q6_K.gguf`	0.004134	97.166%	-0.226%	1.793%
`Qwen3.8-35B-A3B-Distill-Q5_K_M.gguf`	0.008311	95.748%	+0.205%	2.660%
`Qwen3.8-35B-A3B-Distill-Q4_K_M.gguf`	0.017029	94.135%	+1.299%	4.032%
`Qwen3.8-35B-A3B-Distill-Q3_K_M.gguf`	0.052876	90.274%	+4.728%	6.933%
`Qwen3.8-35B-A3B-Distill-Q2_K.gguf`	0.125190	84.213%	+12.617%	10.822%

For a general memory/quality balance, Q4_K_M is the practical starting point from this measurement set. Q3_K_M and Q2_K should be treated as experimental choices where memory limits justify the measured fidelity trade-off. Quality summaries and exact reproduction details are in reproducibility/quality-summary.tsv and reproducibility/manifest.md.

Quick start

From a llama.cpp checkout, with reproducibility/chat_template.jinja downloaded from this repository:

./llama-cli \
  -m Qwen3.8-35B-A3B-Distill-Q4_K_M.gguf \
  --chat-template-file reproducibility/chat_template.jinja \
  --jinja \
  --reasoning off \
  --cpu-moe \
  -p 'Answer briefly in English: What is GGUF and why is it useful for running language models locally?' \
  -n 128 -c 4096 -ngl 8

Adjust -ngl for the available GPU memory. The validation profile for this release used 8 GPU layers, CPU MoE, and llama.cpp's qwen35moe support on an NVIDIA A10M.

Reproducibility and validation

SHA256SUMS.txt contains checksums for the public files.
reproducibility/manifest.md records the locked BF16 source, calibration data, imatrix, runtime, commands, and validation profile.
RELEASE_MANIFEST.json records the machine-readable source and artifact ledger.
reproducibility/quality-summary.tsv contains the machine-readable fidelity table.
reproducibility/runtime-summary.tsv contains the optional Q4_K_M llama-bench result.
reproducibility/chat_template.jinja is the upstream chat template used by the smoke test.

Raw conversion, quantization, smoke-test, fidelity, and benchmark logs are kept locally under reports/ and are not part of this public package.

License and attribution

The upstream model is released under the Apache-2.0 license. See LICENSE and the upstream model card for the original model's terms and attribution.

These are community GGUF quantizations, not an official Qwen3.8 release or endorsement.

Identity and Version

Repository: ngquocvinh/Qwen3.8-35B-A3B-Distill-GGUF
Publisher: Nguyen Quoc Vinh
Task: Text generation
Modality: Text
Library: llama.cpp
Parameters: Not stated by the source
Languages: en
Revision: dd3197c749809d7f75a5668349603024ee43fef9
First published: 2026-09-17
Last updated: 2026-09-21

Files and Weights

27 files, 144.7 GB in total. The weights are 7 files totalling 144.7 GB in gguf.

Weights7 files · 144.7 GB

Configuration3 files · 9.1 KB

Documentation5 files · 23.8 KB

Other11 files · 25.1 KB

Repository1 file · 2.0 KB

Every file

File	Type	Size	SHA-256
Qwen3.8-35B-A3B-Distill-Q2_K.gguf	Weights	13.2 GB	52215d1b6a15
Qwen3.8-35B-A3B-Distill-Q3_K_M.gguf	Weights	17.2 GB	9e282d752ac2
Qwen3.8-35B-A3B-Distill-Q4_K_M.gguf	Weights	21.7 GB	75959a987ca5
Qwen3.8-35B-A3B-Distill-Q5_K_M.gguf	Weights	25.3 GB	72e33fd30407
Qwen3.8-35B-A3B-Distill-Q6_K.gguf	Weights	29.2 GB	62acdff57b1e
Qwen3.8-35B-A3B-Distill-Q8_0.gguf	Weights	37.8 GB	7d986d310e68
reproducibility/qwen3.8-35b-a3b-distill.imatrix.gguf	Weights	192.2 MB	e644adcaf7e8
RELEASE_MANIFEST.json	Configuration	4.0 KB	—
reproducibility/summarize_bench.py	Configuration	2.4 KB	—
reproducibility/summarize_fidelity.py	Configuration	2.7 KB	—
LICENSE	Documentation	11.3 KB	—
README.md	Documentation	5.6 KB	—
reproducibility/calibration-manifest.md	Documentation	2.1 KB	—
reproducibility/manifest.md	Documentation	4.1 KB	—
reproducibility/source-manifest.md	Documentation	766 B	—
SHA256SUMS.txt	Other	2.5 KB	—
reproducibility/benchmark_gpu0.sh	Other	1.8 KB	—
reproducibility/chat_template.jinja	Other	7.8 KB	—
reproducibility/quality-summary.tsv	Other	404 B	—
reproducibility/quality_eval_gpu0.sh	Other	2.5 KB	—
reproducibility/quantize_ladder.sh	Other	1.5 KB	—
reproducibility/qwen3.8-35b-a3b-distill_en_zh_code_tool.txt	Other	5.6 KB	—
reproducibility/runtime-summary.tsv	Other	183 B	—
reproducibility/runtime-version.txt	Other	634 B	—
reproducibility/smoke-summary.tsv	Other	378 B	—
reproducibility/smoke_test_gpu0.sh	Other	1.8 KB	—
.gitattributes	Repository	2.0 KB	—

License and Download

License: apache-2.0
Access: Open weights, no gate
Download size: 144.7 GB

Download from Nguyen Quoc Vinh

Released by Nguyen Quoc Vinh through its official repository on Hugging Face. Read the license.

Built From

Derived from empero-ai/Qwen3.8-35B-A3B-Distill
Quantized from empero-ai/Qwen3.8-35B-A3B-Distill

Memory Requirements

Precision	Weights in memory
As published	144.7 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About Qwen3.8-35B-A3B-Distill-GGUF

Can I use Qwen3.8-35B-A3B-Distill-GGUF commercially?

Yes. Qwen3.8-35B-A3B-Distill-GGUF is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

Similar Models

Model · Text generation

opt-125m

AI at Meta

OPT was first introduced in Open Pre-trained Transformer Language Models and first released in metaseq's repository on May 3rd 2022 by Meta AI. Disclaimer: The team releasing OPT wrote an official model card, which is available in Appendix D of the paper. Content from this model card has been written by the Hugging Face team. To quote the first two paragraphs of the official paper OPT was predominantly pretrained with English text, but a small amount of non-English data is still present within the training corpus via CommonCrawl. The model was pretrained using a causal language modeling (CLM) objective. OPT belongs to the same family of decoder-only models like GPT-3. As such, it was…

Open weights other 2,048 tokens transformers

View model

Model · Text generation

Ornith-1.5-9B-GGUF

Ornith

Chirp Chirp! We are introducing Ornith-1.5, a major step toward building foundation models through end-to-end self-improvement. Ornith-1.5 extends Ornith-1.0 (which was developed on top of Qwen3.5 and Gemma4 with additional continued pretraining, mid-training, and post-training) by expanding the self-improvement loop from scaffold and rollout optimization to jointly optimizing task generation, scaffold construction, and solution rollouts. Rather than relying on a fixed set of human-curated tasks and manually designed harnesses, Ornith-1.5 continuously generates new training tasks, discovers effective strategies for solving them, and improves the policy through reinforcement learning. For…

Open weights mit transformers

View model

Model · Text generation

Ornith-1.5-35B-A3B-GGUF

Ornith

Chirp Chirp! We are introducing Ornith-1.5, a major step toward building foundation models through end-to-end self-improvement. Ornith-1.5 extends Ornith-1.0 (which was developed on top of Qwen3.5 and Gemma4 with additional continued pretraining, mid-training, and post-training) by expanding the self-improvement loop from scaffold and rollout optimization to jointly optimizing task generation, scaffold construction, and solution rollouts. Rather than relying on a fixed set of human-curated tasks and manually designed harnesses, Ornith-1.5 continuously generates new training tasks, discovers effective strategies for solving them, and improves the policy through reinforcement learning. For…

Open weights mit transformers

View model

Model · Text generation

Ternary-Bonsai-2-27B-gguf

Prism ML

Full 27B-class reasoning in ternary transformer weights, for llama.cpp (CUDA, Metal, CPU) - \~5.9 GB language model (down from \~54 GB FP16) — full 27B-class reasoning on a standard laptop or a single GPU - 98.2% of FP16 intelligence retained: 84.78 average across 14 thinking-mode benchmarks — far above the conventional IQ2XXS build (72.59) at about 82% of its footprint, and within 0.4 points of UD-Q4KXL at three times the footprint - Retains thinking, reasoning, and agentic behavior deep in the sub-4-bit regime, where conventional low-bit representations collapse: math within half a point of full precision (96.57), coding level with the baseline (89.42), agentic tool calling at 74.92…

Open weights apache-2.0 llama.cpp

View model

Model · Text generation

Qwen3.8-27B-Uncensored-GGUF

Jonathan Coletti

Uncensored Qwen3.8-27B, published as GGUF quantizations with the multi token prediction (MTP) head retained and verified. Refusal behaviour has been substantially reduced, not eliminated. See Measured behaviour for the numbers. Capabilities, training data, and architecture are otherwise unchanged. - Refusal directions removed with Heretic, which co minimizes refusal count against KL divergence from the base model. No handwritten refusal removal code, no finetuning, no additional training data. - Abliteration runs at bf16 (no 4 bit quantization). the resulting LoRA is merged into the bf16 base, so the published weights are not a quantized round trip. - mtp. tensors are copied verbatim from…

Open weights apache-2.0 llama.cpp

View model

Model · Text generation

Ornith-1.0-9B-GGUF

Ornith

Aloha! Today, we are releasing Ornith-1.0, a self-improving family of open-source models for agentic coding. This model card documents Ornith-1.0-9B, the most lightweight member of the Ornith family, designed for efficient single-GPU deployment. Ornith-1.0-9B is a dense ~9B model (≈19 GB in bf16), so it serves comfortably on a single 80GB GPU. The recipes below stand up an OpenAI-compatible server; add --tensor-parallel-size / --tp if you want to shard across more GPUs. For a quick local test (or to script offline generation), load the model directly with Transformers. Make sure you have a recent release installed — see the Transformers installation guide; Ornith-1.0-9B requires…

Open weights mit transformers

View model