An unofficial community EXL3 quantization of ukisai/Swift-1.5-Qwen3.8-27b, UkisAI's reasoning-efficient derivative of Qwen3.8-27B, made with exllamav3's per-tensor bit-allocation ("SC") pipeline with our own calibration, YAQA Hessians, a 5-bit lmhead, a 6-bit vision tower, a 4-bit MTP head and INT8 token embeddings. It is served with vLLM (continuous batching) through the OrcaSAQ2 plugin and runs a 215k-token context with vision and MTP speculative decoding on a single 24 GB RTX 3090, with a KL divergence of 0.009 against BF16 (≈5× lower than our W4A16 GPTQ version). The 3.70bpw branch trades a little fidelity for a 226k context. Highlights (one RTX 3090 24 GB, measured on 2026-10-14…
Open weights
other
7.9B parameters
262,144 tokens
exllamav3
An EXL3 quantization of ornith-ai/Ornith-1.5-9B made with exllamav3's per-tensor bit-allocation ("SC") pipeline with our own calibration, in two sizes: - main — 6.00 bpw, for 16 GB GPUs (e.g. RTX 5060 Ti 16 GB / 4060 Ti 16 GB): the full 262,144-token context with vision, MTP speculative decoding and 2 concurrent sessions, KL divergence 0.0019 against BF16. - 4.50bpw — 4.50 bpw, for 12 GB GPUs (e.g. RTX 3060 12 GB): 198,000 tokens with vision or 220,000 text-only, KLD 0.0095. Both are served with vLLM (continuous batching) through the OrcaSAQ2 plugin and a small plugin of ours (included), with INT8 token embeddings. Highlights (one RTX 3090 24 GB, measured on 2026-10-14; details in Family…
Open weights
mit
4.4B parameters
262,144 tokens
exllamav3
An EXL3 quantization of Qwen/Qwen3.8-27B made with exllamav3's per-tensor bit-allocation ("SC") pipeline with our own calibration, a 5-bit lmhead, a 6-bit vision tower, a 4-bit MTP head and INT8 token embeddings. It is served with vLLM (continuous batching) through the OrcaSAQ2 plugin and runs a 215k-token context with vision and MTP speculative decoding on a single 24 GB RTX 3090, with a KL divergence of 0.0073 against BF16 (≈5× lower than our W4A16 GPTQ version, and lower than turboderp's 4.00 bpw at a smaller size). The 3.70bpw branch trades a little fidelity (KLD 0.0089) for a 226k context. Highlights (one RTX 3090 24 GB, measured on 2026-10-14; details in Family comparison): - KLD…
Open weights
apache-2.0
7.9B parameters
262,144 tokens
exllamav3
An EXL3 quantization of ornith-ai/Ornith-1.5-35B-A3B made with exllamav3's per-tensor bit-allocation ("SC") pipeline, with a 6-bit lmhead, 6-bit vision tower, 4-bit MTP head and INT8 token embeddings. Ornith's agent-oriented 35B-A3B MoE (Terminal-Bench 2.1 67.8 per its card). Of the three MoEs it is the best agent in our tests (tool calling 30/30 in every mode, short reasoning) and the hardest to quantize, so this repo also has 4.50 and 5.00 bpw branches: full 262k on GPUs with more than 24 GB, and less context on a 24 GB card (see Quick start). It is served with vLLM (continuous batching) through the OrcaSAQ2 EXL3 plugin plus our plugin qwen38exl3vision (0.3.2), which adds EXL3 support for…
Open weights
mit
9.7B parameters
262,144 tokens
exllamav3
An EXL3 quantization of Jackrong/Qwopus3.8-27B-Flash-V2 made with exllamav3's per-tensor bit-allocation ("SC") pipeline with our own calibration, a 5-bit lmhead, a 6-bit vision tower, a 4-bit MTP head and INT8 token embeddings. It is served with vLLM (continuous batching) through the OrcaSAQ2 plugin, with the base Qwen3.8 chat template for reliable tool calls. KL divergence against BF16: 0.0060 (≈6× lower than our W4A16 GPTQ version). It runs a 215k-token context with vision and MTP speculative decoding on a single 24 GB RTX 3090; the 3.70bpw branch (since 2026-10-14) runs 226k with vision and 4 sessions (see Variants). Highlights (one RTX 3090 24 GB, measured on 2026-10-14; details in…
Open weights
apache-2.0
7.9B parameters
262,144 tokens
exllamav3
An EXL3 quantization of Jackrong/Qwopus3.6-35B-A3B-Coder made with exllamav3's per-tensor bit-allocation ("SC") pipeline, with a 6-bit lmhead, 6-bit vision tower, 4-bit MTP head and INT8 token embeddings. Jackrong's thinking-off coding/agent fine-tune of Qwen3.6-35B-A3B (3B active). It is the easiest of the three MoEs to quantize: 3.75 bpw is already below 0.01 KLD. It is served with vLLM (continuous batching) through the OrcaSAQ2 EXL3 plugin plus our plugin qwen38exl3vision (0.3.2), which adds EXL3 support for the MoE experts (fused exllamav3 kernel). On a single RTX 3090 (24 GB) the main revision runs the full 262,144-token context with vision, MTP and 2 concurrent sessions with ≈1 GB of…
Open weights
apache-2.0
9.7B parameters
262,144 tokens
exllamav3
An EXL3 quantization of bytkim/Qwen3.8-27B-pi (bytkim's agent/coding fine-tune of Qwen3.8-27B for the Pi harness), made with exllamav3's per-tensor bit-allocation ("SC") pipeline calibrated on a trace generated by the pi model itself, with a 5-bit lmhead, a 6-bit vision tower, a 4-bit MTP head and INT8 token embeddings. It is served with vLLM (continuous batching) through the OrcaSAQ2 plugin and runs a 215k-token context with vision and MTP speculative decoding on a single 24 GB RTX 3090, with a KL divergence of 0.0074 against BF16 (0.0070 on turboderp's public trace). The 3.70bpw branch trades a little fidelity (KLD 0.0093) for a 226k context. Same recipe, plugins and serving command as…
Open weights
apache-2.0
7.9B parameters
262,144 tokens
exllamav3
An EXL3 quantization of Qwen/Qwen3.6-35B-A3B made with exllamav3's per-tensor bit-allocation ("SC") pipeline, with a 6-bit lmhead, 6-bit vision tower, 4-bit MTP head and INT8 token embeddings. Qwen's 35B-A3B MoE (3B active parameters, 256 experts, hybrid GatedDeltaNet attention, vision and MTP). We did not find another EXL3 of it that keeps vision and MTP and runs under vLLM. It is served with vLLM (continuous batching) through the OrcaSAQ2 EXL3 plugin plus our plugin qwen38exl3vision (0.3.2), which adds EXL3 support for the MoE experts (fused exllamav3 kernel). On a single RTX 3090 (24 GB) the main revision runs the full 262,144-token context with vision, MTP and 2 concurrent sessions with…
Open weights
apache-2.0
9.7B parameters
262,144 tokens
exllamav3
An EXL3 quantization of TokenRhythm/NeoHorse-1-9B (an agent-oriented finetune of Qwen3.5-9B) made with exllamav3, in two sizes: - main — 6.00 bpw, for 16 GB GPUs (e.g. RTX 5060 Ti 16 GB / 4060 Ti 16 GB): the full 262,144-token context with vision, MTP speculative decoding and 2 concurrent sessions, KL divergence 0.0015 against BF16. - 4.50bpw — 4.50 bpw, for 12 GB GPUs (e.g. RTX 3060 12 GB): 176,000 tokens with vision, KLD 0.0058. Both are served with vLLM (continuous batching) through the OrcaSAQ2 plugin and a small plugin of ours (included), with INT8 token embeddings. Why this model: we evaluate small models as a Linux sysadmin agent driven by opencode, used in Spanish. Among the 9B…
Open weights
apache-2.0
4.4B parameters
262,144 tokens
exllamav3
EXL3 quantization (2.25 bits per weight on average) of UkisAI's Swift 1.5 Qwen3.8-Flash-Next (MoE, 125B total / 6B active, plus a 51B n-gram "PLE" table, MTP head and vision), made to serve the whole model on one 48 GB GPU with vLLM (continuous batching): 262,144 tokens per session without YaRN, 4 concurrent sessions sharing a 420k-token KV cache, vision and MTP. Validated on a real NVIDIA RTX PRO 5000 Blackwell (48 GB). All Swift 1.5 Qwen3.8-Flash-Next (125B-A6B) variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number): Download size: 66.7 GB (40.5 GB of EXL3 weights + 26.2 GB EXL3 n-gram table). vLLM users can skip the n-gram table…
Open weights
other
20.2B parameters
262,144 tokens
exllamav3
EXL3 quantization (2.25 bits per weight on average) of Qwen3.8-Flash-Next (MoE, 125B total / 6B active, plus a 51B n-gram "PLE" table, MTP head and vision), made to serve the whole model on one 48 GB GPU with vLLM (continuous batching): 262,144 tokens per session without YaRN, 4 concurrent sessions sharing a 420k-token KV cache, vision and MTP. Same layout and memory footprint as our Swift 1.5 Flash-Next 2.25 bpw, which was validated on a real NVIDIA RTX PRO 5000 Blackwell (48 GB). All Qwen3.8-Flash-Next (125B-A6B) variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number): Download size: 66.7 GB (40.5 GB of EXL3 weights + 26.2 GB EXL3…
Open weights
other
20.2B parameters
262,144 tokens
exllamav3
EXL3 quantization (3.75 bits per weight on average) of UkisAI's Swift 1.5 Qwen3.8-Flash-Next (MoE, 125B total / 6B active, plus a 51B n-gram "PLE" table, MTP head and vision), made to serve the whole model on one 96 GB GPU with vLLM (continuous batching): 262,144 tokens per session without YaRN, 4 concurrent sessions in a 1.1M-token KV cache with 15 GB still free, vision and MTP. Validated on a real NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB, power-limited to 300 W). It is ≈3.3× closer to the BF16 model than our 2.25 bpw build (KLD 0.013 vs 0.044). All Swift 1.5 Qwen3.8-Flash-Next (125B-A6B) variants we publish, measured on 2026-10-14 under the same conditions (see Family…
Open weights
other
31.8B parameters
262,144 tokens
exllamav3
