Qwen3.8-27B-RANA-abliterated-FP8

FP8 version of preemware/Qwen3.8-27B-RANA-abliterated, a refusal-ablated Qwen/Qwen3.8-27B. It uses exactly the same FP8 format as Qwen's official Qwen3.8-27B-FP8, at about half the size of the BF16 build, and scores within noise of it on every capability task tested.

This is a safety-alignment-removed research model. Read Intended use and Limitations before using it. Method, full evaluation and release gates are on the BF16 card.

At a glance

FP8 compared with the BF16 build it was converted from, on the same prompts and settings. Capability differences are paired per question (± standard error); none is larger than about 2 standard errors.

¹ Mean over seeds 1–3. Answers are shown as gpt-oss-safeguard-20b / Mistral-Small-24B judge; p = exact McNemar on the same prompts, pooled over seeds. ² After re-judging disputed labels (BF16: 2, FP8: 3, all from the gpt-oss judge), the procedure described in the BF16 card's G1 disclosure.

In short: the FP8 build keeps the BF16 build's capability (it passes the same capability gate vs base: mean |Δ| 0.58, worst task −1.0 pp on GSM8K) and behaves the same on refusal prompts over 3 seeds and two judges. The one borderline gap (held-out, −2.2 pp under gpt-oss, p = 0.05) does not appear under the second judge (+0.5 pp, p = 0.88).

Quick start

Tested with vLLM 0.30.0 on an RTX PRO 6000 (Blackwell, 96 GB):

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve preemware/Qwen3.8-27B-RANA-abliterated-FP8 \
  --max-model-len 20480 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --limit-mm-per-prompt '{"image": 1, "video": 0}' \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

• vLLM picks up the FP8 format from config.json; no extra quantization flag is needed.
• Thinking is on by default. Recommended sampling: T = 1.0, top_p = 0.95, top_k = 20.
• --max-model-len 20480 matches the evaluation (16k-token budget). Long technical requests can need 20–50k tokens of reasoning; raise --max-model-len and max_tokens for those (73,728 tested, native context 262k).
• VLLM_USE_FLASHINFER_SAMPLER=0 was needed on Blackwell GPUs.
• Vision input, multi-turn tool calling and MTP speculative decoding all work (tested on these shards).

Intended use

• Research only: interpretability, red-teaming, and robustness evaluation of refusal behaviour.
• Not for public or end-user deployment without a separate moderation layer. The model's own refusals have been largely removed, so any safety filtering has to happen outside it.
• You are responsible for complying with applicable law, the Apache-2.0 license inherited from Qwen, and the terms of any platform where outputs are used.

How it was made

The FP8 conversion is applied to the BF16 build's weights. Nothing else about the model changes.

• Format: identical to Qwen's official FP8 checkpoint. The same 407 linear weights are stored as FP8 (e4m3) in 128 × 128 blocks, each with one BF16 scale (weight_scale_inv); activations are quantized dynamically. Embeddings, lm_head, norms, gates, the whole vision tower and the other non-converted tensors stay in BF16. The tensor names, dtypes and shapes match the official checkpoint exactly (1,606 tensors).
• BF16 tensors: every tensor left in BF16 is byte-identical to the BF16 build.
• Same arithmetic as Qwen: on the tensors abliteration did not touch, this conversion reproduces Qwen's own FP8 file: 100% of block scales and 99.91% of FP8 values are bit-identical (the rest differ by rounding at the last step).
• Rounding error: dequantized FP8 weights differ from the BF16 ones by 2.65% (relative Frobenius norm), uniformly across all 407 tensors, including the 131 edited by abliteration.

Results

Capability

Same harness as the BF16 card: vLLM, thinking on, 16k-token generation budget, identical prompts and seeds. Change versus base in percentage points ± standard error (paired on the same questions).

MMLU-Pro uses 50 questions per subject (700 total); all other sets are full size. HumanEval uses a custom chat-format harness and is not comparable with lm-eval's raw-completion HumanEval.

Refusals (seeds 1–3, two judges)

200 HarmBench "standard" behaviours plus 240 held-out prompts (StrongREJECT, JailbreakBench, CategoricalHarmfulQA), judged by openai/gpt-oss-safeguard-20b and mistralai/Mistral-Small-24B-Instruct-2501. "Budget hits" = the response reached the 16k-token generation limit before finishing (no judge involved).

Hard / soft = mean count per seed, gpt-oss judge, raw labels. Paired over 3 seeds (FP8-only vs BF16-only, exact McNemar): answers on HarmBench 38 vs 43 (p = 0.66) under gpt-oss and 87 vs 80 (p = 0.64) under Mistral; held-out 22 vs 38 (p = 0.05) and 93 vs 90 (p = 0.88). Budget hits on HarmBench 27 vs 33 (p = 0.52); held-out 16 vs 12 (p = 0.57).

What the budget hits are

Mostly long reasoning on technical requests, not repetition. The 59 FP8 seed-1 prompts that hit the budget (41 HarmBench, 18 held-out) were re-run with the same seed and sampling:

Finished responses used up to 49k tokens. With these prompts scored at 64k, seed-1 answer rates rise from 76.0% to 92.0% (HarmBench) and from 86.2% to 93.3% (held-out), and budget hits fall from 13.4% to 0.5% of all 440 prompts. presence_penalty, Qwen's setting against repetition, barely helps, which fits: the text is not repeating. The 16k-budget answer rates above are therefore lower bounds. Qwen recommends allowing up to 262k tokens for reasoning; set --max-model-len and max_tokens well above 16k if you need complete answers to long technical requests.

Functional smoke tests (these shards)

• Vision: reads the code word and shape from a synthetic image: pass.
• Tool calling: 3-turn call → result → second call with a new argument: pass.
• MTP speculative decoding: works; mean acceptance length 2.18 averaged over the whole evaluation workload. The BF16 card's 2.45 was measured on one short generation, so the two numbers are not directly comparable.

Limitations

• Answer rates depend on the judge. The two judges differ by 21–36 pp on the same responses; compare models only under the same judge.
• Hard-refusal count relies on re-judging. The gpt-oss judge gave 3 raw HarmBench hard refusals (all seed 1, all 16k budget hits). Re-judged with the auditor reminder, the same procedure as the BF16 card's G1 disclosure, they became 2 answers and 1 degenerate. The Mistral judge gave 0 on all seeds.
• Long reasoning on technical requests. About 20% of HarmBench and 7% of held-out responses need more than 16k tokens; see What the budget hits are.
• KL from base was not measured for FP8. Capability deltas above are the quality evidence.
• Everything listed under Limitations on the BF16 card applies here too: judge-measured refusal rates, a 700-question MMLU-Pro subset, and a custom HumanEval harness.

File layout

• *.safetensors, model.safetensors.index.json, config.json (with quantization_config), tokenizer/processor files, chat_template.jinja, LICENSE: the model.
• results/: every number on this card. fp8_vs_bf16.json (paired FP8 − BF16 capability deltas and the seed-1 refusal run), refusal_3seed.json (seeds 1–3, both judges, re-judging, paired tests), budget_rerun.json (the 64k / presence-penalty re-run), capability.{md,json} (scores vs base for base, orca, BF16, FP8), verify_fp8.json (format and byte-level checks), smoke_fp8.json, humaneval_fp8.json.

Credits

• Qwen team: base model Qwen/Qwen3.8-27B and the FP8 format of Qwen/Qwen3.8-27B-FP8.
• Arditi et al., 2024: "Refusal in Language Models Is Mediated by a Single Direction".
• Jim Lai (grimjim): prior work on norm-preserving abliteration.
• Benchmarks/datasets: HarmBench, StrongREJECT, JailbreakBench, CategoricalHarmfulQA; lm-evaluation-harness; vLLM.