NeoHorse-1-9B-EXL3-16GB-12GB

NeoHorse-1-9B — EXL3 for 16 GB and 12 GB GPUs (vision + MTP, served with vLLM)

Install from this card, not from generic snippets. This is an EXL3 checkpoint: vLLM loads it only together with exllamav3 and our plugins. Follow Requirements & install and Quick start; a plain pip install vllm, Transformers or SGLang cannot load it.

An EXL3 quantization of TokenRhythm/NeoHorse-1-9B (an agent-oriented finetune of Qwen3.5-9B) made with exllamav3, in two sizes:

• main — 6.00 bpw, for 16 GB GPUs (e.g. RTX 5060 Ti 16 GB / 4060 Ti 16 GB): the full 262,144-token context with vision, MTP speculative decoding and 2 concurrent sessions, KL divergence 0.0015 against BF16.
• 4.50bpw — 4.50 bpw, for 12 GB GPUs (e.g. RTX 3060 12 GB): 176,000 tokens with vision, KLD 0.0058.

Both are served with vLLM (continuous batching) through the OrcaSAQ2 plugin and a small plugin of ours (included), with INT8 token embeddings.

Why this model: we evaluate small models as a Linux sysadmin agent driven by opencode, used in Spanish. Among the 9B models we measured with the same tests (Ornith-1.5-9B, AdQWENistrator-9B, Mellum2.1, Qwen3.5-9B), NeoHorse and Ornith-1.5-9B are the two good agents: over 60 real sysadmin tasks with opencode run (20 tasks × 3 runs), NeoHorse 50/60 as written and 53/60 with file names in backticks (Ornith: 45/60 and 55/60). NeoHorse is the stronger coder (browser games: 2-3 of 4 run, Ornith 0-1) and the more robust one without backticks; Ornith thinks ≈4× less. 30/30 tool calls with the real opencode request, even without reasoning.

Our focus. We quantize and evaluate models for one use case: a local Linux system-administration agent (opencode with tool calling) for a university IT department, used in Spanish. Our calibration traces include real sysadmin/agent conversations in Spanish, and our quality checks (tool calls with the real opencode request, agent tasks with real command execution, long-context retrieval, long answers in Spanish) reflect that use. KL divergence is also reported on turboderp's public trace, so it can be compared and reproduced. Numbers for other domains or languages may differ.

Vision and MTP are grafted. The original NeoHorse-1-9B release has no vision tower and no MTP head. We added both, unchanged, from Qwen/Qwen3.5-9B (same architecture). Images work in our tests (14/15 checks) and the MTP head is accepted 55-66 % of the time (≈2.7-3 tokens per step), but they were trained for Qwen3.5-9B, not for NeoHorse.

Sampling matters (please keep generation_config.json). The original repository has no generation_config.json. Without it, vLLM samples from the whole vocabulary (temperature 1.0, no top_k) whenever the client sends no sampling parameters (e.g. opencode without "temperature": true), and in long answers the model drifts into stray Chinese text: 6 of 10 long Spanish answers, 2 of 10 English ones. This repo ships generation_config.json with temperature 0.8 (0.6 until 2026-10-13), top_p 0.95, top_k 20 (applied automatically by vLLM): 0 of 10 in both languages at 0.6, and 0 of 10 in both at 0.8 (checked again on 2026-10-13). For non-English use, a system line such as "Responde siempre en castellano" does not change the CJK rate, but it made answers ≈35 % shorter.

Tested hardware: all numbers were measured on an RTX 3090 (24 GB, Ampere, 300 W) with the memory budget of the target card: the KV cache is fixed with --kv-cache-memory-bytes so that the peak memory used stays under what a 16 GB / 12 GB card offers (≈15,700 / ≈11,700 MiB usable after the driver's reservation). They have not been run on an actual 16 GB or 12 GB card yet — reports welcome (see Feedback).

What is in this repository

Which one to use

All NeoHorse-1-9B variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number):

Results

Family comparison (NeoHorse-1-9B, same conditions)

Measured on 2026-10-14 for every variant in the same way: weights downloaded from Hugging Face, one RTX 3090 24 GB (power limit 350 W, no other use), vLLM 0.31.0 (EXL3: + exllamav3 1.5.1 + plugins), each variant with its own Quick start configuration. KLD and perplexity come from exllamav3's qbench (see How to read it).

How to read it: - KLD / perplexity: next-token KL divergence against the BF16 model with exllamav3's qbench (lower is better). Internal: our sysadmin/agent evaluation set (not published, separate from the calibration data); public: turboderp's qbench_prompts_gen.json, reproducible by anyone. GPTQ checkpoints were evaluated as a BF16 copy holding the dequantized weights. These two columns come from qbench, not from this campaign. - Needle: 8 facts spread through one prompt at about 92 % of the served context; all 8 must come back. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, at the model's default sampling (generation_config.json); a try counts when it calls a valid tool with valid arguments and the expected command. Agent: 10 sysadmin tasks run with opencode run and real command execution, 3 runs, at the recommended temperature; with sampling on, the same checkpoint varies by about ±2 of 30 between runs, so other measurements in this card may differ by that much. Reasoning: 6 checkable problems × 8 at xhigh; total reasoning tokens over the 48. Vision: our 4-image test. - Prefill / decode: prefill from the needle prompt; decode is aggregate throughput with 1 / 2 / 8 simultaneous requests (temperature 0; with more requests than Sessions, the extra ones wait). MTP tokens per step: vLLM's speculative-decoding metrics over the whole run. - VRAM really free: lowest memory.free while serving (nvidia-smi every 0.5 s, from the end of start-up to the end of the run); we require ≥100 MiB. - Older sections below keep their own date and setup; use this table to compare variants.

How many sessions fit with each Quick start configuration (KV size in tokens as reported by vLLM at start-up; a session uses the KV of its own context, and vLLM never runs more than max sessions at once):

Few users or many users. The Quick start configuration is tuned for 1–2 simultaneous users (fastest single request). For more users, the many users option below (measured the same day, same GPU) gives more total throughput; each request gets slower when many run at once. Same model, same quality: only the serving settings change.

Earlier measurements

Earlier measurements (RTX 3090 within the target memory budget), before the controlled comparison of 2026-10-14: other dates, vLLM versions or settings, as stated in each table. Use Family comparison above to compare variants.

† Measured with the same 4.50bpw weights at 198,000 tokens / KV 4.0e9 (peak 11,899 MiB, slightly over a 12 GB budget); the 176,000 configuration above was then validated for memory, needle, sessions, images and tools.

For reference, on the same tests AdQWENistrator-9B (same size, also quantized by us) scored 5/10 and 5/10 on the 10 agent tasks; Qwen3.5-9B base needed ≈146k reasoning tokens at xhigh for 38-39/48.

Sampling and repetition loops

Measured on 2026-10-14 with the same software and GPU as Family comparison, on our EXL3 6.00 (16 GB) build of NeoHorse-1-9B (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated 8-grams. Serving: the Quick start with a 32k context and 8 sessions.

From temperature 0.6 up, loops are rare (at most 1 of 30). This repo's default (generation_config.json) is 0.8.

Other models, same test and setup (loops / 30):

Requirements & install (vLLM 0.31 + exllamav3 1.5.1 + plugins)

EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point: - Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers through exllamav3's kernels, and INT8 / packed embed_tokens and lm_head for qwen3_5 models (no vLLM patch needed). - qwen38_exl3_vision 0.4.0 (ours; 0.4.0 adds multi-GPU: tensor parallelism for the dense models, -tp 2, e.g. Swift 27B on 2 GPUs with the same log-likelihood (+0.04 %) and +43 % speed for one request, and expert parallelism for the MoE ones, -tp 2 --enable-expert-parallel; 0.3.2 freed the host-RAM copy of the checkpoint, ~14 GB less for a 27B; in plugin/qwen38_exl3_vision/, the same version in all our EXL3 repos): the EXL3 vision tower (drops the separate attn.{q,k,v}_proj tensors and pads input dimensions to multiples of 128 where needed), EXL3 MoE experts with exllamav3's fused kernel, and the Qwen3.8-Flash-Next model.

Use exllamav3 1.5.1 (with 1.6.x the KLD is the same, but tool calls without reasoning got worse in our tests) and Python 3.12 (the exllamav3 wheels are cp312). flashinfer compiles some kernels on first start with the CUDA toolkit that comes with the vLLM wheel; if your system nvcc does not match, point CUDA_HOME at a CUDA 13.0 toolkit.

uv venv --python 3.12 venv
uv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto        # torch 2.13.0
W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whl
curl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"
git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)
hf download JRamirez-UAB/NeoHorse-1-9B-EXL3-16GB-12GB --include "plugin/*" --local-dir .
uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_vision
venv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start

16 GB GPU — main:

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0
hf download JRamirez-UAB/NeoHorse-1-9B-EXL3-16GB-12GB --local-dir NeoHorse-1-9B-EXL3
vllm serve ./NeoHorse-1-9B-EXL3 \
  --served-model-name neohorse-1-9b \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --mm-processor-kwargs '{"max_pixels":1003520}' \
  --max-model-len 262144 \
  --kv-cache-dtype fp8_e4m3 \
  --kv-cache-memory-bytes 5250000000 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 2 \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

12 GB GPU — 4.50bpw: download with --revision 4.50bpw and use --max-model-len 176000 --kv-cache-memory-bytes 3600000000 --max-num-batched-tokens 4096 (peak 11,462 MiB used). 198,000 tokens with 4.0e9 B of KV also runs, but peaked at 11,899 MiB: only if your card leaves that much free.

Prompt processing (updated 2026-10-11): --max-num-batched-tokens 8192 (16 GB) / 4096 (12 GB) reads long prompts ≈22–27 % faster than the 1024 we used before (124k-token prompt: 2,236 vs 1,764 tok/s on the RTX 3090), and still fits: peak 15,260 / 11,462 MiB used, validated with the needle test (8/8 at 233k / 157k), 2 concurrent sessions, vision and tool calls. If you run something else on the same GPU, go back to 1024 (≈1 GB lower peak on 16 GB).

For many users (measured on 2026-10-14, same GPU): the same command with --max-num-seqs 8 — 8 sessions of up to 262,144 tokens, MTP 3. Aggregate decode 574 tok/s with 8 requests (Quick start: 253 with 8); a single request runs at 136 tok/s (139 with the Quick start); 8 of 8 simultaneous sessions returned all 8 needle facts; 15,386 MiB of VRAM used at peak. Rule for CUDA graphs: they must cover sessions × (1 + MTP) tokens per step.

Using it with opencode (how we tested)

Tool calling was measured with the real opencode request (captured and replayed against vLLM) and with opencode run agent tasks in a sandbox with deterministic checks.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-vllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "local vLLM",
      "options": { "baseURL": "http://127.0.0.1:8000/v1" },
      "models": {
        "neohorse-1-9b": {
          "name": "NeoHorse 1 9B EXL3 (local)",
          "tool_call": true,
          "temperature": true,
          "attachment": true,
          "modalities": { "input": ["text", "image"], "output": ["text"] },
          "limit": { "context": 262144, "output": 32000 },
          "variants": {
            "none":   { "chat_template_kwargs": { "enable_thinking": false } },
            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },
            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },
            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }
          }
        }
      }
    }
  },
  "agent": {
    "build": { "model": "local-vllm/neohorse-1-9b", "variant": "xhigh", "temperature": 0.8 },
    "plan":  { "model": "local-vllm/neohorse-1-9b", "variant": "xhigh", "temperature": 0.8 }
  }
}

• limit.context = --max-model-len (176000 with the 4.50bpw branch). Declare "temperature": true, otherwise opencode sends no temperature.
• For Spanish (or another language), add the language line to your AGENTS.md or agent instructions.

Important notes

• Keep generation_config.json (see the note at the top), or pass --override-generation-config '{"temperature":0.8,"top_p":0.95,"top_k":20}'.
• Memory. The driver reserves ≈450 MiB that vLLM cannot use: read the real headroom with nvidia-smi --query-gpu=memory.used,memory.free --format=csv. Size the KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization. With MTP, KV capacity grows in block-sized steps: check the GPU KV cache size line and keep --max-model-len below it.
• Leave the GPU to the LLM. On a 12 GB card the margin is a couple of hundred MiB: a desktop session or a browser on the same GPU will not fit.
• Hugging Face page warnings: the "Model size" shown by Hugging Face counts packed storage elements (EXL3 packs several weights per int16), not parameters: this is the 9B model. The Configuration Parsing Warning about quantization_config.bits is expected (for EXL3 it is an average).
• Use --tool-call-parser qwen3_coder and --reasoning-parser qwen3. reasoning_effort accepts xhigh, medium and low, but in our measurements it does not change how much the model thinks (see Known weak spots).

Quantization recipe

Source: TokenRhythm/NeoHorse-1-9B revision ba5b6e40. Tokenizer and chat template unchanged (Qwen3.5 XML tool calls, qwen3_coder).

Integrity and validation

What was checked for this checkpoint, how, and where the numbers are (measured on 2026-10-14; the method and every figure are in Family comparison).

Artifact integrity - Every weight file is checked against our local copy with SHA-256 before publication (SHA256SUMS in this repo). - The measurements used the weights downloaded from this repo at commit 941a379aa41c; later commits change only metadata, documentation or the plugin, never the .safetensors files. - vLLM loads every tensor listed in model.safetensors.index.json on start-up (a missing tensor or a wrong shape would stop it): checked from a fresh download. - After the metadata update of 2026-10-14 (quantization_config.json, plugin 0.3.1, embed_int8.py): downloaded again from this repo and re-tested: loads; vision 14/15 needle 64k 8/8; tool calls 15/15; engine errors 0.

Quantization - Per-tensor bit allocation is recorded by exllamav3 in quantization_config.json; its tensor_storage entry for embed_tokens describes the INT8 qweight + BF16 scales actually stored (updated on 2026-10-14; the plugin only reads the .trellis entries). - The INT8 embedding conversion (scripts/embed_int8.py) is reproducible: re-running it on the source checkpoint gives the published qweight and scales bit for bit (checked on Ornith-1.5-9B 6.00 bpw); its reported error now uses the stored BF16 scales (0.00806 vs 0.00803 with FP32 scales). - Fidelity: KL divergence against the BF16 NeoHorse-1-9B with exllamav3's qbench on two traces (ours and turboderp's public one).

Chat template - The published template renders correctly in 40 edge cases (empty and very long system prompt, no tools, two tool calls in one message, nested JSON and multi-line arguments, images with tools, two images, earlier reasoning, multi-turn with several tool results) × the reasoning settings (enable_thinking: false, reasoning_effort low / xhigh): all pass (prueba_plantillas.py). With Transformers directly, pass tool-call arguments as dicts (vLLM converts JSON strings itself).

Runtime - Load, long-context needle, vision, tool calls with the real opencode request, 10 agent tasks × 3 runs, concurrency and VRAM: see Family comparison; engine errors during the run: 0.

Scope: results apply to the revision, software stack (vLLM 0.31.0 + exllamav3 1.5.1 + plugins), hardware and settings stated above; other setups are not tested.

Limitations

• Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: review them before running anything as root or with sudo, and give the agent a non-privileged account and a test machine where you can.
• Measured on an RTX 3090 within the memory of the target cards, not yet on actual 16 GB / 12 GB GPUs (expect lower speeds on them).
• The grafted vision tower and MTP head come from Qwen3.5-9B, not from NeoHorse's training.
• Quality was evaluated with our own sysadmin-oriented tests (KL divergence, long-context retrieval, tool calls with the real opencode request, agent tasks, synthetic screenshots, small coding tasks), not with standard benchmarks. The original model card reports NeoHorse's benchmarks.

Known weak spots (measured 2026-10-11, same tests as Ornith-1.5-9B)

These come from the model, not from the quantization (KLD 0.0015 at 6.00 bpw); listed so you know what to watch: - File names in non-English prompts. It sometimes "translates" the file name it was asked for: errors.txt instead of errores.txt, processes.csv instead of procesos.csv (the content is right). Measured over 340 agent tasks in Spanish: 5 % of tasks; a line in AGENTS.md asking for exact names barely changes it (4 %). With the file names and paths written in backticks in the request (we did it automatically for every user message): 0 of 360 tasks, and +6 points of agent success. With the same tasks in English: 0 of 40 — it is a non-English problem. Same behaviour in Ornith-1.5-9B. The model is not translating: it renames output files whose Spanish name is a system concept (processes, users, errors) to the conventional English name; names unrelated to the system (e.g. tortuga.txt) are kept (34/34). Backticks fix it and did not hurt when the user describes a system file in words or misspells its path (70/70). - Invented command options and fields. With less common CLIs (Slurm sacctmgr/scontrol, nvidia-smi --query-*, systemd timer syntax) it may guess field names, subcommands or syntax that do not exist (scontrol show qos, --query-processes, OnCalendar=02:30 daily) and retry them; check its commands. This is a knowledge limit of a 9B model, the same in Spanish and English; a generic "read --help first" instruction did not fix it. Documenting the tools your agent uses (field names, correct syntax) in its instructions is the practical workaround. - Long reasoning. On our 6 reasoning problems × 3 it used ≈32,000–38,000 reasoning tokens per run (median ≈800–1,600 per answer), about 4× Ornith-1.5-9B, with 14–16/18 correct; reasoning_effort low/medium/xhigh made no difference. It also missed a systemd unit question that Ornith gets right. - Agent tasks: 7/10 (16 GB) and 9/10 (12 GB) on the RTX 3090, 21/30 over three more runs, 26/30 on ten additional control tasks.

Feedback welcome

Field reports are very welcome in the Community tab: GPU, driver, vLLM version, context and KV settings, whether it worked or not, and what failed.

Credits & license

Apache License 2.0, as the original NeoHorse-1-9B and Qwen3.5-9B. See NOTICE for what was changed and the attribution. Unofficial release by JRamirez-UAB; not affiliated with TokenRhythm, Alibaba Cloud or the Qwen team.