Ornith-1.5-9B-EXL3-16GB-12GB

Ornith-1.5-9B — EXL3 for 16 GB and 12 GB GPUs (vision + MTP, served with vLLM)

Install from this card, not from generic snippets. This is an EXL3 checkpoint: vLLM loads it only together with exllamav3 and our plugins. Follow Requirements & install and Quick start; a plain pip install vllm, Transformers or SGLang cannot load it.

An EXL3 quantization of ornith-ai/Ornith-1.5-9B made with exllamav3's per-tensor bit-allocation ("SC") pipeline with our own calibration, in two sizes:

• main — 6.00 bpw, for 16 GB GPUs (e.g. RTX 5060 Ti 16 GB / 4060 Ti 16 GB): the full 262,144-token context with vision, MTP speculative decoding and 2 concurrent sessions, KL divergence 0.0019 against BF16.
• 4.50bpw — 4.50 bpw, for 12 GB GPUs (e.g. RTX 3060 12 GB): 198,000 tokens with vision or 220,000 text-only, KLD 0.0095.

Both are served with vLLM (continuous batching) through the OrcaSAQ2 plugin and a small plugin of ours (included), with INT8 token embeddings.

Fix 2026-10-09 — sampling parameters (please update): until 2026-10-09 this repo, like the original ornith-ai/Ornith-1.5-9B, had no generation_config.json. vLLM then samples from the whole vocabulary (temperature 1.0, top_p 1.0, no top_k) whenever the client sends no sampling parameters (e.g. opencode without "temperature": true), and the model drifts into stray Chinese text and broken output in long answers. The repo now ships generation_config.json with top_p 0.95 / top_k 20 (Ornith's values for coding) and temperature 0.8 (Ornith recommends 0.6; 0.8 since 2026-10-14, see Sampling and repetition loops), which vLLM applies automatically. If you downloaded before, fetch it again (hf download JRamirez-UAB/Ornith-1.5-9B-EXL3-16GB-12GB generation_config.json, also with --revision 4.50bpw) or add --override-generation-config '{"temperature":0.8,"top_p":0.95,"top_k":20,"min_p":0.0,"repetition_penalty":1.0}' to vllm serve. Measured on the long opencode request: stray CJK text in 2 of 4 answers without it, in 0 of 8 with it.

Our focus. We quantize and evaluate models for one use case: a local Linux system-administration agent (opencode with tool calling) for a university IT department, used in Spanish. Our calibration traces include real sysadmin/agent conversations in Spanish, and our quality checks (tool calls with the real opencode request, agent tasks with real command execution, long-context retrieval, long answers in Spanish) reflect that use. KL divergence is also reported on turboderp's public trace, so it can be compared and reproduced. Numbers for other domains or languages may differ.

Re-evaluation (2026-10-11): with the sampling configured, Ornith-1.5-9B works well as a sysadmin agent. Our earlier note said it did not; most of that came from running it without sampling parameters. With this repo's generation_config.json (temperature 0.6 at the time of this test, top_p 0.95, top_k 20), on the same tests as the other 9B models we evaluate (main 6.00 bpw, vLLM, RTX PRO 6000 at 300 W): - Agent tasks (10 real sysadmin tasks with opencode run, real command execution): 9/10, and 23/30 over three more runs (NeoHorse-1-9B: 21/30 on the same runs). Over 60 tasks (20 × 3): 45/60 as written, 55/60 with file names in backticks (NeoHorse-1-9B: 50/60 and 53/60). Tool calls with the real opencode request: 30/30. - Stray Chinese text in long Spanish answers (10 long answers each): no sampling parameters 7/10; with the sampling above 1/10 (3 characters); with the sampling plus a system line "Responde siempre en castellano" 0/10. In English: 3/10 without parameters, 0/10 with them. Again on 2026-10-14 (10 long Spanish + 10 English answers per run, no sampling parameters): 0 of 20 at 0.6 and at 0.7; at 0.8, 1 of 40 (2 stray characters, no drift into Chinese). For non-English use, add a language line to your system prompt or AGENTS.md: unlike other 9B models we tested, Ornith needs it. - Weak spot: code that runs. In our small browser-game test only 0-1 of 4 games ran (NeoHorse-1-9B: 2-3 of 4). Prefer a larger model for coding. - It reasons briefly: ≈25-30k reasoning tokens for 48 checkable problems at xhigh (Qwen3.5-9B: ≈146k). The quantization itself is very close to the original (KLD 0.0019 at 6.00 bpw, 0.0095 at 4.50 bpw).

Recommended sampling (Ornith's model card; for coding we use temperature 0.8 instead of 0.6): - Coding / agents: temperature 0.8, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0 (the default in generation_config.json since 2026-10-14; 0.6, Ornith's value, from 2026-10-09: 2 of 30 long answers looped at 0.6 and 0 of 30 at 0.8, see Sampling and repetition loops). - General tasks: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0. - opencode: declare "temperature": true on the model and set the agent temperature to 0.8; otherwise opencode sends no sampling parameters.

Tested setup: RTX 3090 (24 GB, Ampere, power limit 350 W), vLLM 0.31.0 + exllamav3 1.5.1 + the OrcaSAQ2 plugin + our qwen38_exl3_vision 0.3.1 plugin, weights downloaded from this repo, the Quick start configuration (2026-10-14). Only this setup has been tested.

Highlights (one RTX 3090 24 GB, measured on 2026-10-14; details in Family comparison): - KLD 0.0019 against BF16 on our evaluation set (0.0017 on turboderp's public trace). - 262,144 with vision on one 24 GB GPU with MTP (2.88 tokens per step); needle 8/8 at 241,437. - Tool calls with the real opencode request 60/60, opencode agent tasks 23/30 (10 sysadmin tasks × 3 runs), images 14/15. - Reasoning 45/48 at xhigh with 27,067 reasoning tokens in total. - Decode 136.5 / 240.6 / 252.3 tok/s with 1 / 2 / 8 requests; 15,232 MiB of VRAM used at peak.

What is in this repository

Which one to use

All Ornith-1.5-9B variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number):

Variants / revisions

For comparison, a uniform 6.00 bpw allocation measured 0.0023 / 0.0021: the per-tensor recipe is ≈18 % closer to BF16 at the same size. We also made 3.50 and 3.00 bpw versions for 8 GB GPUs (KLD 0.032 and 0.065; text only, 46k / 63k tokens of context, peak ≈7,400 MiB used, tools 15/15, reasoning 46–47/48); they are an ongoing project and not published here.

Files and reproducibility

• model-*.safetensors, config.json (quantization_config: exl3), quantization_config.json (per-tensor allocation); SHA256SUMS.
• plugin/qwen38_exl3_vision/: our vLLM plugin. scripts/embed_int8.py: re-encodes embed_tokens to INT8 per row.
• The calibration and evaluation sets are internal and not published; the public KLD figures use turboderp's qbench_prompts_gen.json, so they can be reproduced with exllamav3's qbench against the BF16 model.
• NOTICE: license and attribution of the original model.

Results

Family comparison (Ornith-1.5-9B, same conditions)

Measured on 2026-10-14 for every variant in the same way: weights downloaded from Hugging Face, one RTX 3090 24 GB (power limit 350 W, no other use), vLLM 0.31.0 (EXL3: + exllamav3 1.5.1 + plugins), each variant with its own Quick start configuration. KLD and perplexity come from exllamav3's qbench (see How to read it).

How to read it: - KLD / perplexity: next-token KL divergence against the BF16 model with exllamav3's qbench (lower is better). Internal: our sysadmin/agent evaluation set (not published, separate from the calibration data); public: turboderp's qbench_prompts_gen.json, reproducible by anyone. GPTQ checkpoints were evaluated as a BF16 copy holding the dequantized weights. These two columns come from qbench, not from this campaign. - Needle: 8 facts spread through one prompt at about 92 % of the served context; all 8 must come back. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, at the model's default sampling (generation_config.json); a try counts when it calls a valid tool with valid arguments and the expected command. Agent: 10 sysadmin tasks run with opencode run and real command execution, 3 runs, at the recommended temperature; with sampling on, the same checkpoint varies by about ±2 of 30 between runs, so other measurements in this card may differ by that much. Reasoning: 6 checkable problems × 8 at xhigh; total reasoning tokens over the 48. Vision: our 4-image test. - Prefill / decode: prefill from the needle prompt; decode is aggregate throughput with 1 / 2 / 8 simultaneous requests (temperature 0; with more requests than Sessions, the extra ones wait). MTP tokens per step: vLLM's speculative-decoding metrics over the whole run. - VRAM really free: lowest memory.free while serving (nvidia-smi every 0.5 s, from the end of start-up to the end of the run); we require ≥100 MiB. - Older sections below keep their own date and setup; use this table to compare variants.

How many sessions fit with each Quick start configuration (KV size in tokens as reported by vLLM at start-up; a session uses the KV of its own context, and vLLM never runs more than max sessions at once):

Few users or many users. The Quick start configuration is tuned for 1–2 simultaneous users (fastest single request). For more users, the many users option below (measured the same day, same GPU) gives more total throughput; each request gets slower when many run at once. Same model, same quality: only the serving settings change.

Benchmarks and validated configurations (RTX 3090, 350 W, within the target memory budget)

Earlier measurements, before the controlled comparison of 2026-10-14: other dates, vLLM versions or settings, as stated in each table. Use Family comparison above to compare variants.

Images: failed-service screenshot, chart, dense firewall ruleset, and a terminal where a port is held by another process while the journal shows a dead PID as a trap. The missed check, when there is one, is always the same: after freeing the port the model restarts the service without checking the port again first.

Reasoning and agent tasks

6 problems with checkable answers (arithmetic, word problem, counting, logic, subnetting, systemd) × 8 tries, temperature 0.6, total reasoning tokens over 48 requests; agent = 4 sysadmin tasks × 2 with opencode run (xhigh, temperature 0.3, our setting until 2026-10-13, real tool execution):

For reference, on the same 48 problems Qwen3.5-9B used ≈146k reasoning tokens at xhigh (39/48, 9 answers cut off) and a 27B Qwen3.8 ≈18–23k.

Agent turns and 8 sessions (--max-num-seqs 8, same KV pool)

Sampling and repetition loops

Measured on 2026-10-14 with the same software and GPU as Family comparison, on our EXL3 6.00 (16 GB) build of Ornith-1.5-9B (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated 8-grams. Serving: the Quick start with a 32k context and 8 sessions.

From temperature 0.6 up, loops are rare (at most 1 of 30). This repo's default (generation_config.json) is 0.8.

Other models, same test and setup (loops / 30):

Requirements & install (vLLM 0.31 + exllamav3 1.5.1 + plugins)

EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point: - Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers through exllamav3's kernels, and INT8 / packed embed_tokens and lm_head for qwen3_5 models (no vLLM patch needed). - qwen38_exl3_vision 0.4.0 (ours; 0.4.0 adds multi-GPU: tensor parallelism for the dense models, -tp 2, e.g. Swift 27B on 2 GPUs with the same log-likelihood (+0.04 %) and +43 % speed for one request, and expert parallelism for the MoE ones, -tp 2 --enable-expert-parallel; 0.3.2 freed the host-RAM copy of the checkpoint, ~14 GB less for a 27B; in plugin/qwen38_exl3_vision/, the same version in all our EXL3 repos): the EXL3 vision tower (drops the separate attn.{q,k,v}_proj tensors and pads input dimensions to multiples of 128 where needed), EXL3 MoE experts with exllamav3's fused kernel, and the Qwen3.8-Flash-Next model.

Use exllamav3 1.5.1 (with 1.6.x the KLD is the same, but tool calls without reasoning got worse in our tests) and Python 3.12 (the exllamav3 wheels are cp312). flashinfer compiles some kernels on first start with the CUDA toolkit that comes with the vLLM wheel; if your system nvcc does not match, point CUDA_HOME at a CUDA 13.0 toolkit.

uv venv --python 3.12 venv
uv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto        # torch 2.13.0
W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whl
curl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"
git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)
hf download JRamirez-UAB/Ornith-1.5-9B-EXL3-16GB-12GB --include "plugin/*" --local-dir .
uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_vision
venv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start

16 GB GPU — main (6.00 bpw), vision + MTP 3, full 262,144 context, 2 concurrent sequences, 1 image per request (≤1 MP):

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0
vllm serve JRamirez-UAB/Ornith-1.5-9B-EXL3-16GB-12GB \
  --served-model-name ornith-1.5-9b \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --mm-processor-kwargs '{"max_pixels":1003520}' \
  --max-model-len 262144 \
  --kv-cache-dtype fp8_e4m3 \
  --kv-cache-memory-bytes 5250000000 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 2 \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

12 GB GPU — 4.50bpw: the same command with --revision 4.50bpw and - with vision: --max-model-len 198000 --kv-cache-memory-bytes 4000000000 --max-num-batched-tokens 2048 (peak 11,500 MiB used; 4096 goes 22 MiB over a 12 GB budget); - text only (drop --limit-mm-per-prompt / --mm-processor-kwargs, add --language-model-only): --max-model-len 220000 --kv-cache-memory-bytes 4430000000 --max-num-batched-tokens 1024 (peak 11,406 MiB used).

Both leave ≈250–300 MiB under the ≈11,700 MiB a 12 GB card offers. A first measurement at 210,000 / 235,000 tokens peaked at 11,684 / 11,706 MiB: it works, but with no margin at all, so we recommend the values above.

8 concurrent sessions: add --max-num-seqs 8 and keep the same KV cache (the pool is shared: 8 × ≈30k tokens at once, or one long session). Validated on 16 GB (peak ≈14,180 MiB used) and on 12 GB with vision at 198,000 (peak ≈11,530 MiB used) — both measured with --max-num-batched-tokens 1024: use 1024 for 8 sessions.

For many users (measured on 2026-10-14, same GPU): the same command with --max-num-seqs 8 — 8 sessions of up to 262,144 tokens, MTP 3. Aggregate decode 579 tok/s with 8 requests (Quick start: 252 with 8); a single request runs at 131 tok/s (136 with the Quick start); 6 of 8 simultaneous sessions returned all 8 needle facts; 15,442 MiB of VRAM used at peak. Rule for CUDA graphs: they must cover sessions × (1 + MTP) tokens per step.

Using it with opencode (how we tested)

This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). Tool calling was measured with the real opencode request (captured and replayed against vLLM) and with opencode run agent tasks; context, speed, memory and image tests were run against the vLLM API.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-vllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "local vLLM",
      "options": { "baseURL": "http://127.0.0.1:8000/v1" },
      "models": {
        "ornith-1.5-9b": {
          "name": "Ornith 1.5 9B EXL3 (local)",
          "tool_call": true,
          "temperature": true,
          "attachment": true,
          "modalities": { "input": ["text", "image"], "output": ["text"] },
          "limit": { "context": 262144, "output": 32000 },
          "variants": {
            "none":   { "chat_template_kwargs": { "enable_thinking": false } },
            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },
            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },
            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }
          }
        }
      }
    }
  },
  "agent": {
    "build": { "model": "local-vllm/ornith-1.5-9b", "variant": "xhigh", "temperature": 0.8 },
    "plan":  { "model": "local-vllm/ornith-1.5-9b", "variant": "xhigh", "temperature": 0.8 }
  }
}

• limit.context = --max-model-len (use the 12 GB value with the 4.50bpw branch); opencode compacts the history at ≈context − output tokens of input.
• Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model. Drop attachment / modalities for text-only serving.
• Start opencode inside a project directory; for opencode run from scripts, close stdin (`