Qwopus3.8-27B-Flash-V2-EXL3-3.85bpw-24GB

Qwopus3.8-27B-Flash-V2 — EXL3 3.85 bpw, for 24 GB GPUs (vision + MTP, served with vLLM)

Install from this card, not from generic snippets. This is an EXL3 checkpoint: vLLM loads it only together with exllamav3 and our plugins. Follow Requirements & install and Quick start; a plain pip install vllm, Transformers or SGLang cannot load it.

An EXL3 quantization of Jackrong/Qwopus3.8-27B-Flash-V2 made with exllamav3's per-tensor bit-allocation ("SC") pipeline with our own calibration, a 5-bit lm_head, a 6-bit vision tower, a 4-bit MTP head and INT8 token embeddings. It is served with vLLM (continuous batching) through the OrcaSAQ2 plugin, with the base Qwen3.8 chat template for reliable tool calls. KL divergence against BF16: 0.0060 (≈6× lower than our W4A16 GPTQ version). It runs a 215k-token context with vision and MTP speculative decoding on a single 24 GB RTX 3090; the 3.70bpw branch (since 2026-10-14) runs 226k with vision and 4 sessions (see Variants).

Tested setup: RTX 3090 (24 GB, Ampere, power limit 350 W), vLLM 0.31.0 + exllamav3 1.5.1 + the OrcaSAQ2 plugin + our qwen38_exl3_vision 0.3.1 plugin, weights downloaded from this repo, the Quick start configuration (2026-10-14). Only this setup has been tested.

Highlights (one RTX 3090 24 GB, measured on 2026-10-14; details in Family comparison): - KLD 0.0060 against BF16 on our evaluation set (0.0048 on turboderp's public trace). - 215,000 with vision on one 24 GB GPU with MTP (3.40 tokens per step); needle 8/8 at 197,991. - Tool calls with the real opencode request 54/60, opencode agent tasks 27/30 (10 sysadmin tasks × 3 runs), images 15/15. - Reasoning 48/48 at xhigh with 23,460 reasoning tokens in total. - Decode 86.9 / 140.5 / 145.3 tok/s with 1 / 2 / 8 requests; 320 MiB of VRAM really free at peak.

Our focus. We quantize and evaluate models for one use case: a local Linux system-administration agent (opencode with tool calling) for a university IT department, used in Spanish. Our calibration traces include real sysadmin/agent conversations in Spanish, and our quality checks (tool calls with the real opencode request, agent tasks with real command execution, long-context retrieval, long answers in Spanish) reflect that use. KL divergence is also reported on turboderp's public trace, so it can be compared and reproduced. Numbers for other domains or languages may differ.

What is in this repository

Which one to use

All Qwopus3.8-27B-Flash-V2 variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number):

Variants / revisions

Made without YAQA (on our Swift EXL3 quantizations YAQA changed the KLD by 0–3 %).

The 3.70bpw branch (added 2026-10-14). We first held it back: in our opencode agent tasks it fell into repetition loops in 3 of 8 runs, issuing the same command 33–70 times. Those runs used temperature 0.3, our setting at the time, which is the range where this family loops (see Sampling and repetition loops). Re-tested on 2026-10-14 at temperature 1.0 (this repo's generation_config.json) with the 3.70bpw Quick start below on an RTX 3090: 0 of 30 loops on long answers (3 of 30 at 0.6), tool calls with the real opencode request 30/30, agent tasks 27/30 (10 tasks × 3 runs; the 3 misses were the same file written as errors.txt instead of the requested errores.txt, not loops), 119 MiB really free at peak. So the loops came from the low temperature, not from the quantization. Earlier results for this branch: tools 59/60 · 60/60, images 18/18.

Files and reproducibility

• model-*.safetensors, config.json (quantization_config: exl3, 3.85 bits, head 5, vision 6, MTP 4), quantization_config.json (per-tensor allocation).
• chat_template-xml-tools.jinja: the base Qwen3.8 chat template (XML tool calls), recommended for tool calling (Apache-2.0, Qwen).
• plugin/qwen38_exl3_vision/: our vLLM plugin (vision tensors and padding). scripts/embed_int8.py: re-encodes embed_tokens to INT8 per row.
• The calibration and evaluation sets are internal and not published; the public KLD figures use turboderp's qbench_prompts_gen.json, so they can be reproduced with exllamav3's qbench against the BF16 model.

Results

Family comparison (Qwopus3.8-27B-Flash-V2, same conditions)

Measured on 2026-10-14 for every variant in the same way: weights downloaded from Hugging Face, one RTX 3090 24 GB (power limit 350 W, no other use), vLLM 0.31.0 (GPTQ: + the embedding patch / EXL3: + exllamav3 1.5.1 + plugins), each variant with its own Quick start configuration. KLD and perplexity come from exllamav3's qbench (see How to read it).

How to read it: - KLD / perplexity: next-token KL divergence against the BF16 model with exllamav3's qbench (lower is better). Internal: our sysadmin/agent evaluation set (not published, separate from the calibration data); public: turboderp's qbench_prompts_gen.json, reproducible by anyone. GPTQ checkpoints were evaluated as a BF16 copy holding the dequantized weights. These two columns come from qbench, not from this campaign. - Needle: 8 facts spread through one prompt at about 92 % of the served context; all 8 must come back. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, at the model's default sampling (generation_config.json); a try counts when it calls a valid tool with valid arguments and the expected command. Agent: 10 sysadmin tasks run with opencode run and real command execution, 3 runs, at the recommended temperature; with sampling on, the same checkpoint varies by about ±2 of 30 between runs, so other measurements in this card may differ by that much. Reasoning: 6 checkable problems × 8 at xhigh; total reasoning tokens over the 48. Vision: our 4-image test. - Prefill / decode: prefill from the needle prompt; decode is aggregate throughput with 1 / 2 / 8 simultaneous requests (temperature 0; with more requests than Sessions, the extra ones wait). MTP tokens per step: vLLM's speculative-decoding metrics over the whole run. - VRAM really free: lowest memory.free while serving (nvidia-smi every 0.5 s, from the end of start-up to the end of the run); we require ≥100 MiB. - Older sections below keep their own date and setup; use this table to compare variants.

How many sessions fit with each Quick start configuration (KV size in tokens as reported by vLLM at start-up; a session uses the KV of its own context, and vLLM never runs more than max sessions at once):

Few users or many users. The Quick start configuration is tuned for 1–4 simultaneous users (fastest single request). For more users, the many users option below (measured the same day, same GPU) gives more total throughput; each request gets slower when many run at once. Same model, same quality: only the serving settings change.

Benchmarks and validated configurations (single RTX 3090 24 GB, 350 W)

Earlier measurements, before the controlled comparison of 2026-10-14: other dates, vLLM versions or settings, as stated in each table. Use Family comparison above to compare variants.

Quality: KL divergence against BF16 (2026-10-02)

Service configuration (quick start above)

Images: failed-service screenshot, chart, dense firewall ruleset, and a terminal where the port is held by another process while the journal shows a dead PID as a trap (the model must take the right PID from the image and free the port). This battery replaces an older 3-image one (8/9), whose first image wrongly required that dead PID; the PID is still checked in a follow-up turn.

Reasoning and agent tasks

6 problems with checkable answers (arithmetic, word problem, counting, logic, subnetting, systemd) × 8 tries, temperature 0.6, total reasoning tokens over 48 requests; agent = 4 sysadmin tasks × 2 with opencode run (xhigh, temperature 0.3, real tool execution, reasoning counted by a proxy):

The only agent failures (one of two runs here, both runs with W4A16) were on the log-summary task. The level (low / medium / xhigh) does not regulate the amount of reasoning.

Agent turns and 8 sessions (--max-num-seqs 8, same KV pool)

Agent turn = the real opencode request (≈13.5k → 14.4k tokens over 10 turns) with prefix caching. For comparison, the W4A16 sibling needs 2.6 / 3.3 s per turn and reaches 460 tok/s with 8 sessions, but only up to a 200k pool and without vision.

MTP acceptance

On the mixed workload of the family table: 3.03 tokens per step, per-position acceptance 0.85 / 0.67 / 0.52.

Sampling and repetition loops

Measured on 2026-10-14 with the same software and GPU as Family comparison, on our EXL3 3.85 build of Qwopus3.8-27B-Flash-V2 (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated 8-grams. Serving: the Quick start with a 32k context and 8 sessions.

From temperature 0.8 up, loops are rare (at most 1 of 30). This repo's default (generation_config.json) is 1.0.

Other models, same test and setup (loops / 30):

Requirements & install (vLLM 0.31 + exllamav3 1.5.1 + plugins)

EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point: - Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers through exllamav3's kernels, and INT8 / packed embed_tokens and lm_head for qwen3_5 models (no vLLM patch needed). - qwen38_exl3_vision 0.4.0 (ours; 0.4.0 adds multi-GPU: tensor parallelism for the dense models, -tp 2, e.g. Swift 27B on 2 GPUs with the same log-likelihood (+0.04 %) and +43 % speed for one request, and expert parallelism for the MoE ones, -tp 2 --enable-expert-parallel; 0.3.2 freed the host-RAM copy of the checkpoint, ~14 GB less for a 27B; in plugin/qwen38_exl3_vision/, the same version in all our EXL3 repos): the EXL3 vision tower (drops the separate attn.{q,k,v}_proj tensors and pads input dimensions to multiples of 128 where needed), EXL3 MoE experts with exllamav3's fused kernel, and the Qwen3.8-Flash-Next model.

Use exllamav3 1.5.1 (with 1.6.x the KLD is the same, but tool calls without reasoning got worse in our tests) and Python 3.12 (the exllamav3 wheels are cp312). flashinfer compiles some kernels on first start with the CUDA toolkit that comes with the vLLM wheel; if your system nvcc does not match, point CUDA_HOME at a CUDA 13.0 toolkit.

uv venv --python 3.12 venv
uv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto        # torch 2.13.0
W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whl
curl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"
git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)
hf download JRamirez-UAB/Qwopus3.8-27B-Flash-V2-EXL3-3.85bpw-24GB --include "plugin/*" --local-dir .
uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_vision
venv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start (single RTX 3090, 24 GB)

main (3.85 bpw), vision + MTP 3, 215k context, 2 concurrent sequences, 1 image per request (≤1 MP) — our validated configuration:

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0
vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-EXL3-3.85bpw-24GB \
  --served-model-name qwopus3.8-27b \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --mm-processor-kwargs '{"max_pixels":1003520}' \
  --max-model-len 215000 \
  --kv-cache-dtype fp8_e4m3 \
  --kv-cache-memory-bytes 8420000000 \
  --max-num-batched-tokens 1024 \
  --max-num-seqs 2 \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --chat-template chat_template-xml-tools.jinja

3.70bpw, vision + MTP 3, 226k context, 4 concurrent sequences (validated on 2026-10-14 with 119 MiB really free at peak): same command with --revision 3.70bpw --max-model-len 226000 --kv-cache-memory-bytes 9000000000 --max-num-seqs 4.

8 concurrent sessions: add --max-num-seqs 8 and keep the same KV cache (the pool is shared: 8 × ≈23k tokens at once, or one long session). Validated with 145 MiB really free at peak.

For many users (measured on 2026-10-14, same GPU): the same command with --max-num-seqs 8 — 8 sessions of up to 215,000 tokens, MTP 3. Aggregate decode 314 tok/s with 8 requests (Quick start: 145 with 8); a single request runs at 88 tok/s (87 with the Quick start); 8 of 8 simultaneous sessions returned all 8 needle facts; 138 MiB of VRAM really free. Rule for CUDA graphs: they must cover sessions × (1 + MTP) tokens per step.

Using it with opencode (how we tested)

This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). Tool calling is measured with the real opencode request (its long system prompt and 11 tools, captured and replayed against vLLM) and with opencode run agent tasks; context, speed, memory and image tests are run directly against the vLLM API.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-vllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "local vLLM",
      "options": { "baseURL": "http://127.0.0.1:8000/v1" },
      "models": {
        "qwopus3.8-27b": {
          "name": "Qwopus3.8-27B-Flash-V2 EXL3 (local)",
          "tool_call": true,
          "temperature": true,
          "attachment": true,
          "modalities": { "input": ["text", "image"], "output": ["text"] },
          "limit": { "context": 215000, "output": 22000 },
          "variants": {
            "none":   { "chat_template_kwargs": { "enable_thinking": false } },
            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },
            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },
            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }
          }
        }
      }
    }
  },
  "agent": {
    "build": { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 1.0 },
    "plan":  { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 1.0 }
  }
}

• Temperature 1.0 (since 2026-10-13; it was 0.3): our first agent tests of this family used 0.3, but at 0.3 the model falls into repetition loops on long answers (see Sampling and repetition loops), and in our agent tests (10 real sysadmin tasks × 3 runs with opencode run) 1.0 scored the same as 0.3 within one task.

• limit.context = --max-model-len; with output: 22000 opencode compacts the history at ≈context − 22k tokens of input.

• Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model (checked by capturing the requests).
• Tool calling needs --chat-template chat_template-xml-tools.jinja and --tool-call-parser qwen3_coder; with the original template and hermes opencode receives empty answers.
• Without attachment / modalities opencode will not send images; drop them if you serve with --language-model-only.
• Start opencode inside a project directory (it takes a git snapshot before each step); for opencode run from scripts, close stdin (`