Qwopus3.6-35B-A3B-Coder-EXL3-4.00bpw-24GB

Qwopus3.6-35B-A3B-Coder — EXL3 MoE for vLLM, 262k context with vision and MTP on a 24 GB GPU

Install from this card, not from generic snippets. This is an EXL3 checkpoint: vLLM loads it only together with exllamav3 and our plugins. Follow Requirements & install and Quick start; a plain pip install vllm, Transformers or SGLang cannot load it.

An EXL3 quantization of Jackrong/Qwopus3.6-35B-A3B-Coder made with exllamav3's per-tensor bit-allocation ("SC") pipeline, with a 6-bit lm_head, 6-bit vision tower, 4-bit MTP head and INT8 token embeddings. Jackrong's thinking-off coding/agent fine-tune of Qwen3.6-35B-A3B (3B active). It is the easiest of the three MoEs to quantize: 3.75 bpw is already below 0.01 KLD.

It is served with vLLM (continuous batching) through the OrcaSAQ2 EXL3 plugin plus our plugin qwen38_exl3_vision (0.3.2), which adds EXL3 support for the MoE experts (fused exllamav3 kernel). On a single RTX 3090 (24 GB) the main revision runs the full 262,144-token context with vision, MTP and 2 concurrent sessions with ≈1 GB of VRAM to spare.

Tested setup: RTX 3090 (24 GB, Ampere, power limit 350 W), vLLM 0.31.0 + exllamav3 1.5.1 + the OrcaSAQ2 plugin + our qwen38_exl3_vision 0.3.1 plugin, weights downloaded from this repo, the Quick start configuration (2026-10-14). Only this setup has been tested.

Our focus. We quantize and evaluate models for one use case: a local Linux system-administration agent (opencode with tool calling) for a university IT department, used in Spanish. Our calibration traces include real sysadmin/agent conversations in Spanish, and our quality checks (tool calls with the real opencode request, agent tasks with real command execution, long-context retrieval, long answers in Spanish) reflect that use. KL divergence is also reported on turboderp's public trace, so it can be compared and reproduced. Numbers for other domains or languages may differ.

What is in this repository

Which one to use

All Qwopus3.6-35B-A3B-Coder variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number):

Download a branch with hf download JRamirez-UAB/Qwopus3.6-35B-A3B-Coder-EXL3-4.00bpw-24GB --revision <branch> --local-dir <dir>.

Results

Family comparison (Qwopus3.6-35B-A3B-Coder, same conditions)

Measured on 2026-10-14 for every variant in the same way: weights downloaded from Hugging Face, one RTX 3090 24 GB (power limit 350 W, no other use), vLLM 0.31.0 (EXL3: + exllamav3 1.5.1 + plugins), each variant with its own Quick start configuration. KLD and perplexity come from exllamav3's qbench (see How to read it).

How to read it: - KLD / perplexity: next-token KL divergence against the BF16 model with exllamav3's qbench (lower is better). Internal: our sysadmin/agent evaluation set (not published, separate from the calibration data); public: turboderp's qbench_prompts_gen.json, reproducible by anyone. GPTQ checkpoints were evaluated as a BF16 copy holding the dequantized weights. These two columns come from qbench, not from this campaign. - Needle: 8 facts spread through one prompt at about 92 % of the served context; all 8 must come back. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, at the model's default sampling (generation_config.json); a try counts when it calls a valid tool with valid arguments and the expected command. Agent: 10 sysadmin tasks run with opencode run and real command execution, 3 runs, at the recommended temperature; with sampling on, the same checkpoint varies by about ±2 of 30 between runs, so other measurements in this card may differ by that much. Reasoning: 6 checkable problems × 8 at xhigh; total reasoning tokens over the 48. Vision: our 4-image test. - Prefill / decode: prefill from the needle prompt; decode is aggregate throughput with 1 / 2 / 8 simultaneous requests (temperature 0; with more requests than Sessions, the extra ones wait). MTP tokens per step: vLLM's speculative-decoding metrics over the whole run. - VRAM really free: lowest memory.free while serving (nvidia-smi every 0.5 s, from the end of start-up to the end of the run); we require ≥100 MiB. - Older sections below keep their own date and setup; use this table to compare variants.

How many sessions fit with each Quick start configuration (KV size in tokens as reported by vLLM at start-up; a session uses the KV of its own context, and vLLM never runs more than max sessions at once):

Few users or many users. The Quick start configuration is tuned for 1–2 simultaneous users (fastest single request). For more users, the many users option below (measured the same day, same GPU) gives more total throughput; each request gets slower when many run at once. Same model, same quality: only the serving settings change.

Earlier measurements

Earlier measurements, before the controlled comparison of 2026-10-14: other dates, vLLM versions or settings, as stated in each table. Use Family comparison above to compare variants.

How to read it: - KLD: mean KL divergence of the next-token distribution against the BF16 model, measured with exllamav3's qbench on an internal sysadmin/agent evaluation set (not published) and on turboderp's public trace (qbench_prompts_gen.json). Lower is better. - Needle: 8 facts spread through one prompt, all 8 returned; 2 concurrent sessions of 116k tokens also returned 8/8 each in every row. - Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 10 tries; A = xhigh at the model's default temperature, B = xhigh at T=0.3, C = thinking off at T=0.3. Reasoning: total reasoning tokens over 6 checkable problems × 8 tries (T=0.6), and how many were correct. - Agent (opencode run, 4 sysadmin tasks × 2 with real tool execution): 6/8 (both). - Configuration of every row: --max-model-len 262144, fp8 KV cache, MTP 1, 2 sequences, prefix caching, 1 image per request (≤1 MP). RTX 3090 rows: KV 3.5e9 B (287,575-token pool).

Sampling and repetition loops

Measured on 2026-10-14 with the same software and GPU as Family comparison, on our EXL3 4.00 build of Qwopus3.6-35B-A3B-Coder (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated 8-grams. Serving: the Quick start with a 32k context and 8 sessions.

Loops do not go away at any temperature we tested: the fewest is 4 of 30 at 1.0. Use 1.0, keep a sensible output limit (max_tokens) and expect some long answers to repeat themselves. This repo's default (generation_config.json) is 1.0.

Other models, same test and setup (loops / 30):

Requirements & install (vLLM 0.31 + exllamav3 1.5.1 + plugins)

EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point: - Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers through exllamav3's kernels, and INT8 / packed embed_tokens and lm_head for qwen3_5 models (no vLLM patch needed). - qwen38_exl3_vision 0.4.0 (ours; 0.4.0 adds multi-GPU: tensor parallelism for the dense models, -tp 2, e.g. Swift 27B on 2 GPUs with the same log-likelihood (+0.04 %) and +43 % speed for one request, and expert parallelism for the MoE ones, -tp 2 --enable-expert-parallel; 0.3.2 freed the host-RAM copy of the checkpoint, ~14 GB less for a 27B; in plugin/qwen38_exl3_vision/, the same version in all our EXL3 repos): the EXL3 vision tower (drops the separate attn.{q,k,v}_proj tensors and pads input dimensions to multiples of 128 where needed), EXL3 MoE experts with exllamav3's fused kernel, and the Qwen3.8-Flash-Next model.

Use exllamav3 1.5.1 (with 1.6.x the KLD is the same, but tool calls without reasoning got worse in our tests) and Python 3.12 (the exllamav3 wheels are cp312). flashinfer compiles some kernels on first start with the CUDA toolkit that comes with the vLLM wheel; if your system nvcc does not match, point CUDA_HOME at a CUDA 13.0 toolkit.

uv venv --python 3.12 venv
uv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto        # torch 2.13.0
W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whl
curl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"
git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)
hf download JRamirez-UAB/Qwopus3.6-35B-A3B-Coder-EXL3-4.00bpw-24GB --include "plugin/*" --local-dir .
uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_vision
venv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start (single RTX 3090, 24 GB)

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_DISABLE_SHARED_EXPERTS_STREAM=1
vllm serve JRamirez-UAB/Qwopus3.6-35B-A3B-Coder-EXL3-4.00bpw-24GB \
  --served-model-name qwopus3.6-35b-a3b-coder \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --mm-processor-kwargs '{"max_pixels":1003520}' \
  --max-model-len 262144 \
  --kv-cache-dtype fp8_e4m3 \
  --kv-cache-memory-bytes 3500000000 \
  --max-num-batched-tokens 1024 \
  --max-num-seqs 2 \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
  --compilation-config '{"max_cudagraph_capture_size":16}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

• 3.75bpw branch: add --revision 3.75bpw (same flags; validated as in the table).

On 48 GB GPUs raise --kv-cache-memory-bytes (10e9 B gives an 821k-token pool: room for 8 sessions; add --max-num-seqs 8). For more sessions, raise max_cudagraph_capture_size with them (see Important notes: ≥ max-num-seqs × 2 with MTP 1).

For many users (measured on 2026-10-14, same GPU): the same command with --max-num-seqs 32 --compilation-config '{"max_cudagraph_capture_size":64}' — 32 sessions of up to 262,144 tokens, MTP 1, CUDA graphs up to 64 tokens. Aggregate decode 681 tok/s with 32 requests (Quick start: 208 with 8); a single request runs at 120 tok/s (123 with the Quick start); 16 of 16 simultaneous sessions returned all 8 needle facts; 794 MiB of VRAM really free. Rule for CUDA graphs: they must cover sessions × (1 + MTP) tokens per step.

Using it with opencode

How we use it as a coding / sysadmin agent from opencode (opencode 1.18.x). limit.context must match --max-model-len of your vllm serve command (262144 in the Quick start above).

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-vllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "local vLLM",
      "options": { "baseURL": "http://127.0.0.1:8000/v1" },
      "models": {
        "qwopus3-6-35b-a3b-coder-exl3-4-00bpw-24g": {
          "name": "Qwopus3.6-35B-A3B-Coder-EXL3-4.00bpw-24GB (local)",
          "tool_call": true,
          "temperature": true,
          "attachment": true,
          "modalities": { "input": ["text", "image"], "output": ["text"] },
          "limit": { "context": 262144, "output": 22000 }
        }
      }
    }
  },
  "agent": {
    "build": { "model": "local-vllm/qwopus3-6-35b-a3b-coder-exl3-4-00bpw-24g", "temperature": 1.0 },
    "plan":  { "model": "local-vllm/qwopus3-6-35b-a3b-coder-exl3-4-00bpw-24g", "temperature": 1.0 }
  }
}

• Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model and vLLM falls back to its defaults.
• Temperature 1.0 is this repo's generation_config.json default.
• With output: 22000 opencode compacts the history at ≈context − 22k tokens of input.
• Start opencode inside a project directory (it takes a git snapshot before each step); for opencode run from scripts, close stdin (`