Qwen3.8-27B-Clef-LoRA-GGUF

Qwen3.8-27B + Cloudflare Clef on one GPU

One resident copy of Qwen3.8-27B serves two paths from a single llama.cpp server on a 24 GB GPU:

• Decide -- Cloudflare Clef: typed questions over a state, a probability for every option of every question in one forward pass, Jev/SystemOne POST /v1/systemone.
• Write -- plain Qwen3.8-27B chat with MTP speculative decoding, POST /v1/chat/completions.

The trick: Clef's backbone is Qwen3.8-27B plus a merged LoRA on layers 40-63 only (layers 0-39, embeddings, lm_head and the vision tower are byte-identical to the base). This repo holds that LoRA, recovered from the release, so it rides on the same weights you chat with. llama.cpp keeps it loaded at scale 0; a decision switches it to scale 1 for that request only.

Running both paths together on a 24 GB card is the point: Clef as its own model (including llama.cpp's native clef architecture) is a second ~15 GB model that does not fit beside a chat model.

Files

Base model GGUF: trevest/Qwen3.8-27B-Tetris-GGUF (4.41 bpw, 14.03 GiB). The LoRA follows the current convert_hf_to_gguf.py Gated-DeltaNet V-head layout; the Tetris file matches it (828/866 tensors byte-identical to a fresh convert, the rest are ssm_a at 1e-7 relative). Any GGUF converted from Qwen/Qwen3.8-27B revision 1d4bf0f2 with a current converter works.

How it works

client ──POST /v1/systemone──> clefhead (CPU) ──POST /embeddings {lora: [{id:0, scale:1}],
                                    │             encoding: f16_base64}──> llama-server ─┐
                                    │<── final hidden state of every prompt token ───────┤ one copy of
                                    └─ Clef joint head -> P(option) per question         │ Qwen3.8-27B
client ──POST /v1/chat/completions─────────────────────────────────────> llama-server ──┘ (LoRA scale 0,
                                                                                            MTP drafting)

Clef reads every prompt token's final (post-norm) hidden state, which llama-server --embeddings --pooling none exports. Stock llama.cpp needs a small patch for this setup:

1. /embeddings takes a per-request lora (stock only parses it for completions), returns hidden states as one base64 f16 blob (encoding: "f16_base64") instead of a JSON float per value, and last_only.
2. MTP next to --embeddings. The MTP draft context inherited embeddings mode (a pooling layer its graph cannot feed), the server capped outputs at one per sequence in embeddings mode (MTP verification needs 1 + n_draft), and the draft inherited the large ubatch with its compute buffer (OOM). The patch turns embeddings off for the draft, caps its ubatch at 512 and keeps the speculative output budget. Clef hidden states are bit-identical with and without MTP.

Run it

1. Build the patched server (CUDA shown; Metal below) (or use the prebuilt image ghcr.io/trevestforvor/llamacpp-glimmer:cf-ad1de39-p2, CUDA 13.0, sm_120 / Blackwell only):

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout ad1de39e0708e3ced9c71bb3c82d93a2c046a73f
git apply /path/to/llama.cpp-patch/0001-embeddings-lora-f16.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 && cmake --build build --target llama-server -j

2. Download the weights

hf download trevest/Qwen3.8-27B-Tetris-GGUF Qwen3.8-27B-Tetris-4.4bpw.gguf --local-dir models
hf download unsloth/Qwen3.8-27B-GGUF mmproj-F16.gguf --local-dir models          # optional, chat vision
hf download trevest/Qwen3.8-27B-Clef-LoRA-GGUF --local-dir clef

3. Start the engine (the exact flags measured below):

llama-server -m models/Qwen3.8-27B-Tetris-4.4bpw.gguf --mmproj models/mmproj-F16.gguf \
  --lora-scaled clef/clef-lora-f16.gguf:0 \
  --embeddings --pooling none -ub 4096 -b 4096 \
  -c 49152 -np 1 -ngl 999 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --jinja \
  --spec-type draft-mtp --spec-draft-n-max 3 --host 0.0.0.0 --port 8080

Use --lora-scaled <file>:0, not --lora-init-without-apply: the latter still applies scale 1 to requests that send no lora field, so plain chat would silently run through Clef's adapter. -ub must cover the longest decision prompt, because llama.cpp never splits an embeddings request.

4. Start the head

docker run --rm -p 8095:8095 -v $PWD:/data \
  -e ENGINE_URL=http://host.docker.internal:8080 -e HF_ROOT=/data -e SENTINEL=/data/clef/tokenizer.json \
  -e MODEL_FILE=Qwen3.8-27B-Tetris-4.4bpw.gguf -e MAX_TOKENS=4096 \
  ghcr.io/trevestforvor/llamacpp-glimmer:clefhead-1.0.1
# or: pip install torch numpy safetensors tokenizers gguf && python clefhead/server.py (same env vars)

HF_ROOT is searched recursively for tokenizer.json, the head files and the base GGUF (lm_head rows for option tokens are dequantized from it on demand). SENTINEL is any file that exists once the downloads are complete.

5. Use both paths

curl -s localhost:8095/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "clef", "state": "Our checkout started returning errors and orders are blocked.",
  "questions": {
    "team":    {"type": "choice", "instructions": "Who should handle it?",
                "criteria": {"billing": "Payments", "technical": "Bugs or outages"}},
    "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},
    "outage":  {"type": "noul", "instructions": "Is a service down?"}}}'

curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages": [{"role": "user", "content": "Hello"}]}'

node examples/one_model_demo.mjs --decide http://127.0.0.1:8095 --chat http://127.0.0.1:8080

Measured (RTX 5090 Laptop GPU, 24 GB, CUDA 13.1 driver, 2026-10-02..04)

Fidelity - Hidden states, base BF16 + this LoRA vs Clef BF16: 0.6% median relative difference, cosine >= 0.9995. The ~3% per-tensor SVD residual is bf16 rounding of Clef's merged weights, not missing rank. - Answers, Tetris 4.4 bpw + LoRA through this stack vs Cloudflare's own BF16 pipeline: 16/16 argmax-equal, per-question max |dp| median 0.012, worst 0.087.

Quality (BTZSC classification, 500 per dataset; paired bootstrap, * = 95% CI excludes 0)

JevBench public: easy 48/48, standard 71/72, hard 81/111 (Jev 1.13: hard 80/111).

Speed and memory (-c 49152 -ub 4096, MTP n-max 3) - A decision costs one full prefill of its prompt at ~1,100 tok/s with the LoRA on: ~0.3-0.6 s for a few-hundred-token ticket, ~2.3 s for Banking77's 2,360-token prompt. The head adds ~30 ms on CPU. - Chat: 57-95 tok/s with MTP (~40 without). - VRAM: 22,960 MiB including the vision projector.

Apple Silicon (Metal) -- verified

The same patch and flags run on a Mac with llama.cpp's Metal backend. Verified 2026-10-04 on an M1 Ultra (64 GB) from a clean checkout of this repo: examples/one_model_demo.mjs passes 24/24 with the same answers as on CUDA.

cmake -B build -DGGML_METAL=ON && cmake --build build --target llama-server -j
# step 3 above, unchanged (-ngl 999 offloads to Metal); clefhead runs on CPU via python as in step 4

A 32 GB Mac should fit it at a smaller context (macOS gives the GPU ~70-75% of RAM). Decisions are bound by prefill speed, so newer M-series chips will be faster.

Limitations

• Text and JSON states only on /v1/systemone (Clef also takes images; this stack does not yet).
• One request = one ubatch: 4096 tokens at -ub 4096; clefhead answers 413 above that instead of silently truncating the state (Cloudflare's encode_record truncates).
• /embeddings never reuses the prompt cache, and Clef's prompt puts the state before the questions, so there is no shared prefix to cache: every decision is a full prefill.
• The prebuilt images target sm_120 only. The patch is against ad1de39e0; newer llama.cpp also has native Clef (#29831, its own decision-only architecture) and /v1/systemone (#29818). Use those if you want Clef alone; use this setup to keep Clef and chat on one copy of the weights.

Packaged for Olares

The same stack ships as the llamacppqwen3827bdecide app (engine + clefhead + gateway): source.

License

Apache-2.0, following Cloudflare Clef and Qwen3.8-27B. The LoRA is a derivative of the Clef release weights; the head, tokenizer and clefhead/joint_schema_model.py are Cloudflare's, unmodified (see clefhead/NOTICE). The llama.cpp patch is offered under llama.cpp's MIT license.