Qwen3.8-27B-pi-EXL3-3.85bpw-24GB

Qwen3.8-27B-pi — EXL3 3.85 bpw, for 24 GB GPUs (vision + MTP, served with vLLM)

Install from this card, not from generic snippets. This is an EXL3 checkpoint: vLLM loads it only together with exllamav3 and our plugins. Follow Requirements & install and Quick start; a plain pip install vllm, Transformers or SGLang cannot load it.

An EXL3 quantization of bytkim/Qwen3.8-27B-pi (bytkim's agent/coding fine-tune of Qwen3.8-27B for the Pi harness), made with exllamav3's per-tensor bit-allocation ("SC") pipeline calibrated on a trace generated by the pi model itself, with a 5-bit lm_head, a 6-bit vision tower, a 4-bit MTP head and INT8 token embeddings. It is served with vLLM (continuous batching) through the OrcaSAQ2 plugin and runs a 215k-token context with vision and MTP speculative decoding on a single 24 GB RTX 3090, with a KL divergence of 0.0074 against BF16 (0.0070 on turboderp's public trace). The 3.70bpw branch trades a little fidelity (KLD 0.0093) for a 226k context.

Same recipe, plugins and serving command as our Qwen3.8-27B EXL3; this card lists what was measured on this model.

Tested setup: RTX 3090 (24 GB, Ampere, power limit 350 W), vLLM 0.31.0 + exllamav3 1.5.1 + the OrcaSAQ2 plugin + our qwen38_exl3_vision 0.3.1 plugin, weights downloaded from this repo, the Quick start configuration (2026-10-14). Only this setup has been tested.

License: Apache-2.0, as the pi model and Qwen3.8-27B (see LICENSE and NOTICE). This is a community quantization, not an official release of bytkim or the Qwen team.

Highlights (one RTX 3090 24 GB, measured on 2026-10-14; details in Family comparison): - KLD 0.0074 against BF16 on our evaluation set (0.0070 on turboderp's public trace). - 215,000 with vision on one 24 GB GPU with MTP (3.11 tokens per step); needle 8/8 at 197,991. - Tool calls with the real opencode request 60/60, opencode agent tasks 29/30 (10 sysadmin tasks × 3 runs), images 15/15. - Reasoning 48/48 at xhigh with 18,535 reasoning tokens in total. - Decode 81.1 / 134.0 / 145.4 tok/s with 1 / 2 / 8 requests; 320 MiB of VRAM really free at peak.

Our focus. We quantize and evaluate models for one use case: a local Linux system-administration agent (opencode with tool calling) for a university IT department, used in Spanish. Our calibration traces include real sysadmin/agent conversations in Spanish, and our quality checks (tool calls with the real opencode request, agent tasks with real command execution, long-context retrieval, long answers in Spanish) reflect that use. KL divergence is also reported on turboderp's public trace, so it can be compared and reproduced. Numbers for other domains or languages may differ.

What is in this repository

Which one to use

All Qwen3.8-27B-pi variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number):

Variants / revisions

Files and reproducibility

• model-*.safetensors, config.json (quantization_config: exl3, 3.85 bits, head 5, vision 6, MTP 4), quantization_config.json (per-tensor allocation).
• plugin/qwen38_exl3_vision/: our vLLM plugin (vision tensors and padding; 0.2.0). scripts/embed_int8.py: re-encodes embed_tokens to INT8 per row.
• LICENSE (Apache-2.0) and NOTICE (the pi model's notice, plus this quantization).

Results

Family comparison (Qwen3.8-27B-pi, same conditions)

Measured on 2026-10-14 for every variant in the same way: weights downloaded from Hugging Face, one RTX 3090 24 GB (power limit 350 W, no other use), vLLM 0.31.0 (GPTQ: + the embedding patch / EXL3: + exllamav3 1.5.1 + plugins), each variant with its own Quick start configuration. KLD and perplexity come from exllamav3's qbench (see How to read it).

How to read it: - KLD / perplexity: next-token KL divergence against the BF16 model with exllamav3's qbench (lower is better). Internal: our sysadmin/agent evaluation set (not published, separate from the calibration data); public: turboderp's qbench_prompts_gen.json, reproducible by anyone. GPTQ checkpoints were evaluated as a BF16 copy holding the dequantized weights. These two columns come from qbench, not from this campaign. - Needle: 8 facts spread through one prompt at about 92 % of the served context; all 8 must come back. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, at the model's default sampling (generation_config.json); a try counts when it calls a valid tool with valid arguments and the expected command. Agent: 10 sysadmin tasks run with opencode run and real command execution, 3 runs, at the recommended temperature; with sampling on, the same checkpoint varies by about ±2 of 30 between runs, so other measurements in this card may differ by that much. Reasoning: 6 checkable problems × 8 at xhigh; total reasoning tokens over the 48. Vision: our 4-image test. - Prefill / decode: prefill from the needle prompt; decode is aggregate throughput with 1 / 2 / 8 simultaneous requests (temperature 0; with more requests than Sessions, the extra ones wait). MTP tokens per step: vLLM's speculative-decoding metrics over the whole run. - VRAM really free: lowest memory.free while serving (nvidia-smi every 0.5 s, from the end of start-up to the end of the run); we require ≥100 MiB. ¹ While loading the weights, free memory dipped below 100 MiB for a moment (it started and served normally); on a GPU that also drives a display, leave a little more room. - Older sections below keep their own date and setup; use this table to compare variants.

How many sessions fit with each Quick start configuration (KV size in tokens as reported by vLLM at start-up; a session uses the KV of its own context, and vLLM never runs more than max sessions at once):

Few users or many users. The Quick start configuration is tuned for 1–4 simultaneous users (fastest single request). For more users, the many users option below (measured the same day, same GPU) gives more total throughput; each request gets slower when many run at once. Same model, same quality: only the serving settings change.

Benchmarks (single RTX 3090 24 GB, 300 W)

Earlier measurements, before the controlled comparison of 2026-10-14: other dates, vLLM versions or settings, as stated in each table. Use Family comparison above to compare variants.

For comparison, the base Qwen3.8-27B EXL3 3.85 reasoned 23,221 tokens at xhigh on the same problems (pi: 17,070, −26 %), with the same context and memory.

Sampling and repetition loops

Measured on 2026-10-14 with the same software and GPU as Family comparison, on our EXL3 3.85 build of Qwen3.8-27B-pi (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated 8-grams. Serving: the Quick start with a 32k context and 8 sessions.

From temperature 0.9 up, loops are rare (at most 1 of 30). This repo's default (generation_config.json) is 1.0.

Other models, same test and setup (loops / 30):

Requirements & install (vLLM 0.31 + exllamav3 1.5.1 + plugins)

EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point: - Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers through exllamav3's kernels, and INT8 / packed embed_tokens and lm_head for qwen3_5 models (no vLLM patch needed). - qwen38_exl3_vision 0.4.0 (ours; 0.4.0 adds multi-GPU: tensor parallelism for the dense models, -tp 2, e.g. Swift 27B on 2 GPUs with the same log-likelihood (+0.04 %) and +43 % speed for one request, and expert parallelism for the MoE ones, -tp 2 --enable-expert-parallel; 0.3.2 freed the host-RAM copy of the checkpoint, ~14 GB less for a 27B; in plugin/qwen38_exl3_vision/, the same version in all our EXL3 repos): the EXL3 vision tower (drops the separate attn.{q,k,v}_proj tensors and pads input dimensions to multiples of 128 where needed), EXL3 MoE experts with exllamav3's fused kernel, and the Qwen3.8-Flash-Next model.

Use exllamav3 1.5.1 (with 1.6.x the KLD is the same, but tool calls without reasoning got worse in our tests) and Python 3.12 (the exllamav3 wheels are cp312). flashinfer compiles some kernels on first start with the CUDA toolkit that comes with the vLLM wheel; if your system nvcc does not match, point CUDA_HOME at a CUDA 13.0 toolkit.

uv venv --python 3.12 venv
uv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto        # torch 2.13.0
W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whl
curl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"
git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)
hf download JRamirez-UAB/Qwen3.8-27B-pi-EXL3-3.85bpw-24GB --include "plugin/*" --local-dir .
uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_vision
venv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start (single RTX 3090, 24 GB)

main (3.85 bpw), vision + MTP 3, 215k context, 2 concurrent sequences, 1 image per request (≤1 MP):

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0
vllm serve JRamirez-UAB/Qwen3.8-27B-pi-EXL3-3.85bpw-24GB \
  --served-model-name qwen3.8-27b-pi \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --mm-processor-kwargs '{"max_pixels":1003520}' \
  --max-model-len 215000 \
  --kv-cache-dtype fp8_e4m3 \
  --kv-cache-memory-bytes 8420000000 \
  --max-num-batched-tokens 1024 \
  --max-num-seqs 2 \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

3.70bpw branch, 226k context: the same command with --revision 3.70bpw --max-model-len 226000 --kv-cache-memory-bytes 9000000000 --max-num-seqs 4 (4 sessions; validated in Family comparison).

For many users (measured on 2026-10-14, same GPU): the same command with --max-num-seqs 8 — 8 sessions of up to 215,000 tokens, MTP 3. Aggregate decode 296 tok/s with 8 requests (Quick start: 145 with 8); a single request runs at 80 tok/s (81 with the Quick start); 8 of 8 simultaneous sessions returned all 8 needle facts; 178 MiB of VRAM really free. Rule for CUDA graphs: they must cover sessions × (1 + MTP) tokens per step.

Using it with opencode (how we tested)

This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). Tool calling was measured with the real opencode request (its long system prompt and 11 tools, captured and replayed against vLLM) and with opencode run agent tasks; context, speed, memory and image tests were run directly against the vLLM API.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-vllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "local vLLM",
      "options": { "baseURL": "http://127.0.0.1:8000/v1" },
      "models": {
        "qwen3.8-27b-pi": {
          "name": "Qwen3.8-27B-pi EXL3 (local)",
          "tool_call": true,
          "temperature": true,
          "attachment": true,
          "modalities": { "input": ["text", "image"], "output": ["text"] },
          "limit": { "context": 215000, "output": 22000 },
          "variants": {
            "none":   { "chat_template_kwargs": { "enable_thinking": false } },
            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },
            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },
            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }
          }
        }
      }
    }
  },
  "agent": {
    "build": { "model": "local-vllm/qwen3.8-27b-pi", "variant": "xhigh", "temperature": 1.0 },
    "plan":  { "model": "local-vllm/qwen3.8-27b-pi", "variant": "xhigh", "temperature": 1.0 }
  }
}

• Temperature 1.0 (since 2026-10-13; it was 0.3): our first agent tests of this family used 0.3, but at 0.3 the model falls into repetition loops on long answers (see Sampling and repetition loops), and in our agent tests (10 real sysadmin tasks × 3 runs with opencode run) 1.0 scored the same as 0.3 within one task: 29/30 at 1.0 and 30/30 at 0.3 for Qwen3.8-27B, -pi and Swift 1.5 27B; Qwopus3.8, served with its chat_template-xml-tools.jinja, 28/30 at 1.0 and 26/30 at 0.3.

• limit.context = --max-model-len (226000 for the 3.70bpw branch); with output: 22000 opencode compacts the history at ≈context − 22k tokens of input.

• Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model (checked by capturing the requests).
• Without attachment / modalities opencode will not send images; drop them if you serve with --language-model-only.
• Start opencode inside a project directory (it takes a git snapshot before each step); for opencode run from scripts, close stdin (`