Qwen3.8-27B-EXL3-3.85bpw-24GB

Qwen3.8-27B — EXL3 3.85 bpw, for 24 GB GPUs (vision + MTP, served with vLLM)

Install from this card, not from generic snippets. This is an EXL3 checkpoint: vLLM loads it only together with exllamav3 and our plugins. Follow Requirements & install and Quick start; a plain pip install vllm, Transformers or SGLang cannot load it.

An EXL3 quantization of Qwen/Qwen3.8-27B made with exllamav3's per-tensor bit-allocation ("SC") pipeline with our own calibration, a 5-bit lm_head, a 6-bit vision tower, a 4-bit MTP head and INT8 token embeddings. It is served with vLLM (continuous batching) through the OrcaSAQ2 plugin and runs a 215k-token context with vision and MTP speculative decoding on a single 24 GB RTX 3090, with a KL divergence of 0.0073 against BF16 (≈5× lower than our W4A16 GPTQ version, and lower than turboderp's 4.00 bpw at a smaller size). The 3.70bpw branch trades a little fidelity (KLD 0.0089) for a 226k context.

Tested setup: RTX 3090 (24 GB, Ampere, power limit 350 W), vLLM 0.31.0 + exllamav3 1.5.1 + the OrcaSAQ2 plugin + our qwen38_exl3_vision 0.3.1 plugin, weights downloaded from this repo, the Quick start configuration (2026-10-14). Only this setup has been tested.

Highlights (one RTX 3090 24 GB, measured on 2026-10-14; details in Family comparison): - KLD 0.0073 against BF16 on our evaluation set (0.0067 on turboderp's public trace). - 215,000 with vision on one 24 GB GPU with MTP (3.03 tokens per step); needle 8/8 at 197,991. - Tool calls with the real opencode request 60/60, opencode agent tasks 29/30 (10 sysadmin tasks × 3 runs), images 15/15. - Reasoning 48/48 at xhigh with 22,474 reasoning tokens in total. - Decode 87.9 / 137.4 / 149.5 tok/s with 1 / 2 / 8 requests; 300 MiB of VRAM really free at peak.

Our focus. We quantize and evaluate models for one use case: a local Linux system-administration agent (opencode with tool calling) for a university IT department, used in Spanish. Our calibration traces include real sysadmin/agent conversations in Spanish, and our quality checks (tool calls with the real opencode request, agent tasks with real command execution, long-context retrieval, long answers in Spanish) reflect that use. KL divergence is also reported on turboderp's public trace, so it can be compared and reproduced. Numbers for other domains or languages may differ.

What is in this repository

Which one to use

All Qwen3.8-27B variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number):

Variants / revisions

Unlike our Swift EXL3 quantizations, these were made without YAQA (on Swift, YAQA changed the KLD by 0–3 %).

Files and reproducibility

• model-*.safetensors, config.json (quantization_config: exl3, 3.85 bits, head 5, vision 6, MTP 4), quantization_config.json (per-tensor allocation).
• plugin/qwen38_exl3_vision/: our vLLM plugin (vision tensors and padding). scripts/embed_int8.py: re-encodes embed_tokens to INT8 per row.
• The calibration and evaluation sets are internal and not published; the public KLD figures use turboderp's qbench_prompts_gen.json, so they can be reproduced with exllamav3's qbench against the BF16 model.

Results

Family comparison (Qwen3.8-27B, same conditions)

Measured on 2026-10-14 for every variant in the same way: weights downloaded from Hugging Face, one RTX 3090 24 GB (power limit 350 W, no other use), vLLM 0.31.0 (GPTQ: + the embedding patch / EXL3: + exllamav3 1.5.1 + plugins), each variant with its own Quick start configuration. KLD and perplexity come from exllamav3's qbench (see How to read it).

How to read it: - KLD / perplexity: next-token KL divergence against the BF16 model with exllamav3's qbench (lower is better). Internal: our sysadmin/agent evaluation set (not published, separate from the calibration data); public: turboderp's qbench_prompts_gen.json, reproducible by anyone. GPTQ checkpoints were evaluated as a BF16 copy holding the dequantized weights. These two columns come from qbench, not from this campaign. - Needle: 8 facts spread through one prompt at about 92 % of the served context; all 8 must come back. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, at the model's default sampling (generation_config.json); a try counts when it calls a valid tool with valid arguments and the expected command. Agent: 10 sysadmin tasks run with opencode run and real command execution, 3 runs, at the recommended temperature; with sampling on, the same checkpoint varies by about ±2 of 30 between runs, so other measurements in this card may differ by that much. Reasoning: 6 checkable problems × 8 at xhigh; total reasoning tokens over the 48. Vision: our 4-image test. - Prefill / decode: prefill from the needle prompt; decode is aggregate throughput with 1 / 2 / 8 simultaneous requests (temperature 0; with more requests than Sessions, the extra ones wait). MTP tokens per step: vLLM's speculative-decoding metrics over the whole run. - VRAM really free: lowest memory.free while serving (nvidia-smi every 0.5 s, from the end of start-up to the end of the run); we require ≥100 MiB. - Older sections below keep their own date and setup; use this table to compare variants.

How many sessions fit with each Quick start configuration (KV size in tokens as reported by vLLM at start-up; a session uses the KV of its own context, and vLLM never runs more than max sessions at once):

Few users or many users. The Quick start configuration is tuned for 1–4 simultaneous users (fastest single request). For more users, the many users option below (measured the same day, same GPU) gives more total throughput; each request gets slower when many run at once. Same model, same quality: only the serving settings change.

Benchmarks and validated configurations (single RTX 3090 24 GB, 350 W)

Earlier measurements, before the controlled comparison of 2026-10-14: other dates, vLLM versions or settings, as stated in each table. Use Family comparison above to compare variants.

Service configuration (quick start above)

Images (EXL3 rows, vision enabled): 4/4 tasks, 18/18 checks with our 2026-10-02 battery (failed-service screenshot, chart, dense firewall ruleset, and a terminal where the port is held by another process while the journal shows a dead PID as a trap). This battery replaces the older 3-image one (8/9), whose first image wrongly required that dead PID; the PID is still checked in a follow-up turn.

Reasoning and agent tasks

6 problems with checkable answers (arithmetic, word problem, counting, logic, subnetting, systemd) × 8 tries, temperature 0.6, total reasoning tokens over 48 requests; agent = 4 sysadmin tasks × 2 with opencode run (xhigh, temperature 0.3, our setting until 2026-10-13, real tool execution, reasoning counted by a proxy):

The EXL3 versions reason 12–17 % less at xhigh on these problems; on the short agent tasks all reason little and the differences (−24 % to +11 %) are within the run-to-run spread. The level (low / medium / xhigh) does not regulate the amount of reasoning much in either. For a model that reasons markedly less, see the Swift 1.5 family.

Agent turns and 8 sessions (--max-num-seqs 8, same KV pool)

Agent turn = the real opencode request (≈13.5k → 14.7k tokens over 10 turns) with prefix caching. For comparison, the W4A16 sibling needs 2.7 / 3.6 s per turn and reaches 472 tok/s with 8 sessions, but only up to a 200k pool and without vision.

MTP acceptance

On the mixed workload of the family table: 3.03 tokens per step for main, per-position acceptance 0.86 / 0.66 / 0.52. 3.70bpw has no 8-session run yet; on the validation run (needle, decode, images, tools) it gave 2.96 tokens per step, against 2.93 for main on the same kind of run.

Sampling and repetition loops

Measured on 2026-10-14 with the same software and GPU as Family comparison, on our EXL3 3.85 build of Qwen3.8-27B (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated 8-grams. Serving: the Quick start with a 32k context and 8 sessions.

From temperature 0.7 up, loops are rare (at most 1 of 30). This repo's default (generation_config.json) is 1.0.

Other models, same test and setup (loops / 30):

Requirements & install (vLLM 0.31 + exllamav3 1.5.1 + plugins)

EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point: - Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers through exllamav3's kernels, and INT8 / packed embed_tokens and lm_head for qwen3_5 models (no vLLM patch needed). - qwen38_exl3_vision 0.4.0 (ours; 0.4.0 adds multi-GPU: tensor parallelism for the dense models, -tp 2, e.g. Swift 27B on 2 GPUs with the same log-likelihood (+0.04 %) and +43 % speed for one request, and expert parallelism for the MoE ones, -tp 2 --enable-expert-parallel; 0.3.2 freed the host-RAM copy of the checkpoint, ~14 GB less for a 27B; in plugin/qwen38_exl3_vision/, the same version in all our EXL3 repos): the EXL3 vision tower (drops the separate attn.{q,k,v}_proj tensors and pads input dimensions to multiples of 128 where needed), EXL3 MoE experts with exllamav3's fused kernel, and the Qwen3.8-Flash-Next model.

Use exllamav3 1.5.1 (with 1.6.x the KLD is the same, but tool calls without reasoning got worse in our tests) and Python 3.12 (the exllamav3 wheels are cp312). flashinfer compiles some kernels on first start with the CUDA toolkit that comes with the vLLM wheel; if your system nvcc does not match, point CUDA_HOME at a CUDA 13.0 toolkit.

uv venv --python 3.12 venv
uv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto        # torch 2.13.0
W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whl
curl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"
git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)
hf download JRamirez-UAB/Qwen3.8-27B-EXL3-3.85bpw-24GB --include "plugin/*" --local-dir .
uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_vision
venv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start (single RTX 3090, 24 GB)

main (3.85 bpw), vision + MTP 3, 215k context, 2 concurrent sequences, 1 image per request (≤1 MP) — our validated configuration:

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0
vllm serve JRamirez-UAB/Qwen3.8-27B-EXL3-3.85bpw-24GB \
  --served-model-name qwen3.8-27b \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --mm-processor-kwargs '{"max_pixels":1003520}' \
  --max-model-len 215000 \
  --kv-cache-dtype fp8_e4m3 \
  --kv-cache-memory-bytes 8420000000 \
  --max-num-batched-tokens 1024 \
  --max-num-seqs 2 \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

3.70bpw branch, 226k context: the same command with --revision 3.70bpw --max-model-len 226000 --kv-cache-memory-bytes 9000000000 --max-num-seqs 4 (4 sessions; validated in Family comparison).

8 concurrent sessions: add --max-num-seqs 8 and keep the same KV cache (the pool is shared: 8 × ≈23k tokens at once, or one long session). Validated on main with 149 MiB really free at peak (not yet run on 3.70bpw).

For many users (measured on 2026-10-14, same GPU): the same command with --max-num-seqs 8 — 8 sessions of up to 215,000 tokens, MTP 3. Aggregate decode 296 tok/s with 8 requests (Quick start: 150 with 8); a single request runs at 84 tok/s (88 with the Quick start); 8 of 8 simultaneous sessions returned all 8 needle facts; 138 MiB of VRAM really free. Rule for CUDA graphs: they must cover sessions × (1 + MTP) tokens per step.

Using it with opencode (how we tested)

This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). Tool calling was measured with the real opencode request (its long system prompt and 11 tools, captured and replayed against vLLM) and with opencode run agent tasks; context, speed, memory and image tests were run directly against the vLLM API.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-vllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "local vLLM",
      "options": { "baseURL": "http://127.0.0.1:8000/v1" },
      "models": {
        "qwen3.8-27b": {
          "name": "Qwen3.8-27B EXL3 (local)",
          "tool_call": true,
          "temperature": true,
          "attachment": true,
          "modalities": { "input": ["text", "image"], "output": ["text"] },
          "limit": { "context": 215000, "output": 22000 },
          "variants": {
            "none":   { "chat_template_kwargs": { "enable_thinking": false } },
            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },
            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },
            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }
          }
        }
      }
    }
  },
  "agent": {
    "build": { "model": "local-vllm/qwen3.8-27b", "variant": "xhigh", "temperature": 1.0 },
    "plan":  { "model": "local-vllm/qwen3.8-27b", "variant": "xhigh", "temperature": 1.0 }
  }
}

• Temperature 1.0 (since 2026-10-13; it was 0.3): our first agent tests of this family used 0.3, but at 0.3 the model falls into repetition loops on long answers (see Sampling and repetition loops), and in our agent tests (10 real sysadmin tasks × 3 runs with opencode run) 1.0 scored the same as 0.3 within one task: 29/30 at 1.0 and 30/30 at 0.3 for Qwen3.8-27B, -pi and Swift 1.5 27B; Qwopus3.8, served with its chat_template-xml-tools.jinja, 28/30 at 1.0 and 26/30 at 0.3.

• limit.context = --max-model-len (226000 for the 3.70bpw branch); with output: 22000 opencode compacts the history at ≈context − 22k tokens of input.

• Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model (checked by capturing the requests).
• Without attachment / modalities opencode will not send images; drop them if you serve with --language-model-only.
• Start opencode inside a project directory (it takes a git snapshot before each step); for opencode run from scripts, close stdin (`