qwen3.8-27b-buddhist by Wulezhi: Open-Weight Model

Model Card

By Wulezhi, published under apache-2.0, revision 452514495463.

基于 Qwen3.8-27B（qwen35 架构，稠密 27B，48 线性注意力 + 16 全注意力，带视觉模块）的佛学辞典微调模型，使用佛光大辞典与中华佛教大百科全书的语料做 35,841 条中文佛教 SFT 微调，各目录的 quantizationconfig 不同（quantmethod 各异），加载命令需对应：未量化的层主要是三类（各版本 config.json 的 ignore 字段可查）： 1. 线性注意力（Gated DeltaNet / linearattn）投影：nvfp4a16 明确忽略了 240 条、autoawq 忽略 inproja / inprojb —— 量化工具链对线性注意力算子支持不完整，硬量化容易掉点； 2. 视觉塔 visual（约 110 个模块）：对精度敏感，且纯文本部署可用 --language-model-only 直接跳过； 3. lmhead 与各 norm / bias / embedding：量化收益小、风险大。 - ① SM75 没有 FP8 硬件能力（FP8 张量核从 Ada / SM89 才有）。所以在 2080Ti 上 FP8 权重只能走社区 fork 的模拟路径（Marlin fallback + 实时反量化），实际行为是"权重 FP8 + 计算 BF16"（weight-only），激活不参与量化 —— 这也是为什么单独做了一份 block 128×128 的 FP8（面向通用 vLLM 的 per-channel 那份在 SM75 上加载不了）。 - ② 覆盖率低 ≠ 残次品：BF16…

Read Wulezhi's full model card

Qwen3.8-27B-Buddhist（佛学辞典 SFT 微调版）

基于 Qwen3.8-27B（qwen3_5 架构，稠密 27B，48 线性注意力 + 16 全注意力，带视觉模块）的佛学辞典微调模型，使用 佛光大辞典 与 中华佛教大百科全书 的语料做 35,841 条中文佛教 SFT 微调，目的是让模型解释和理解佛学术语、名相、义理——服务于佛学学习与查阅场景。

架构更正说明：本模型为稠密模型（Qwen3_5ForConditionalGeneration，非 MoE）。早期文档误标为 "MoE / 激活 3B"，与事实不符，已订正。

模型卡

项	值
架构	`qwen3_5` 稠密 27B（hidden 5120，64 层：48 linear_attention + 16 full_attention，GQA 24/4 头，vocab 248320，bf16）
视觉	内置 vision encoder（`Qwen3_5ForConditionalGeneration`，multimodal），可选搭配 mmproj
基座	Qwen3.8-27B（Unsloth 4-bit NF4 版）
微调方法	Unsloth QLoRA，r=8, alpha=8
语料	佛光大辞典 + 中华佛教大百科全书，35,841 条（佛教 SFT：科判注解、义理问答、经文释义）
用途	佛学术语/名相/义理的解释与理解

已上传量化版本（加载框架）

本仓库提供的量化权重按推理框架分为两类：

1. GGUF —— llama.cpp / Ollama 加载

文件	大小	说明
`qwen3.8-27b-Buddhist-16bit.Q4_K_M.gguf`	~16 GB	主力推荐，llama.cpp / Ollama 直接加载
`qwen3.8-27b-Buddhist-16bit.Q6_K.gguf`	~21 GB	高精度档，显存充足时选
`qwen3.8-27b-Buddhist-16bit.BF16-mmproj.gguf`	~0.9 GB	视觉模块，图像理解时搭配 `--mmproj` 使用

GGUF 由 save_pretrained_gguf(..., save_method="forced_merged_4bit") 导出，即 "4-bit NF4 基座 + LoRA 权重" 合并后的量化结果，并非 16-bit 全量微调。

2. safetensors 量化 —— vLLM 加载

各目录的 quantization_config 不同（quant_method 各异），加载命令需对应：

目录	量化	`quant_method`	vLLM 加载
`qwen38-autoawq-w4a16/`	W4A16 AutoAWQ（gemm, group 128）	`awq`	`vllm serve . --quantization awq`
`qwen38-awq-w4a16/`	W4A16 原生 AWQ（gptqmodel 导出，gemm / group 128；与上一行是两条不同产线）	`awq`	`vllm serve . --quantization awq`
`qwen38-gptq-w4a16/`	W4A16 GPTQ（group 128，MTP 权重含在分片内）	`gptq`	`vllm serve . --quantization gptq_marlin`
`qwen38-nvfp4a16/`	W4A16 NVFP4（llm-compressor 出品，容器 `compressed-tensors` / `nvfp4-pack-quantized`，data-free 量化）	`compressed-tensors`	`vllm serve . --quantization compressed-tensors`
`qwen38-fp8-perchannel/`	W8A8 FP8（llm-compressor 出品，容器 `compressed-tensors`，weight per-channel；面向通用 vLLM）	`compressed-tensors`	`vllm serve . --quantization compressed-tensors`
`qwen38-fp8-block-mtp/`	W8A8 FP8（block 128×128 缩放、`activation=dynamic`，vLLM 原生 `fp8` 风格；MTP 头同样已量化；面向 2080Ti fork）	`fp8`	`vllm serve . --quantization fp8`

qwen38-nvfp4a16/ 实测结论：已在双 2080Ti（Turing / SM75，44GB）上用社区 vLLM-2080Ti fork 实测可用 ——compressed-tensors 的 nvfp4 变体在该 fork 里有对应 kernel（社区 README 亦实测过 ~220 tok/s），与下条注记里"int4 pack-quantized 不可用"是两个变体，别混。

两份 FP8 的区别 —— 按 runtime 选，不是好坏之分 - qwen38-fp8-perchannel/：llm-compressor 标准产物（compressed-tensors 容器，weight per-channel 缩放）→ 给通用 vLLM / SM89+ 等原生支持 FP8 的环境用。 - qwen38-fp8-block-mtp/：block 128×128 缩放（weight_block_size=[128,128]、激活 dynamic）、config 为 vLLM 原生 fp8 风格，MTP 头也已量化 → 给 2080Ti（Turing / SM75）用：社区 vLLM-2080Ti fork 为 SM75 做了 FP8 模拟路径（Marlin fallback + 实时反量化），已在双 2080Ti 实测可用。 - 选版口诀：2080Ti 双卡 → block 128×128 版（qwen38-fp8-block-mtp/）；通用 vLLM → per-channel。Turing 没有 FP8 硬件，走的是软件模拟路径，所以 per-channel 那份在 SM75 上加载不了，才另出一份 block 版。

# 示例：起 vLLM 在线服务（路径换成实际下载位置）
vllm serve ./qwen38-gptq-w4a16 --quantization gptq_marlin \
  --served-model-name qwen3.8-27b-buddhist --max-model-len 65536

目标硬件门槛（重要）：在 Turing / SM75（2080Ti 等）上，compressed-tensors 容器的 int4 pack-quantized（W4A16 / W8A8）没有可用 kernel —— 症状极具欺骗性：加载正常、/health 200、HTTP 200，但输出退化为复读（内容为空 / 只重复输入词）。本仓库曾上传过两个此类版本（当时同名的 qwen38-awq-w4a16——即 compressed-tensors 那版——以及 qwen38-int8-w8a8），在 2080Ti 上不可用，均已删除。现在表里的qwen38-awq-w4a16/ 与那条注记无关：它是后来重新做的原生 AWQ（quant_method=awq），可直接用。选版请认 quant_method：awq / gptq（vLLM 原生格式）在 SM75 上实测可用；目录名里写着 awq/gptq 不代表真是那个格式。

这些 safetensors 量化版面向 vLLM 在线服务场景（视觉模块 visual 已排除在量化之外，保留 fp）； llama.cpp 侧请用上面的 GGUF，两者不要混用。

3. 量化范围说明（为什么大量层保持 BF16）

先看这条，再判断"量化全不全"：本仓库的量化版是针对 2080Ti（Turing / SM75，44GB 双卡）部署做的，量化范围是刻意收敛的结果。

实测覆盖率（量化模块数 / 线性层总数 ≈ 937）：

目录	量化模块	保持 BF16 的层	覆盖率
`qwen38-autoawq-w4a16/`	400	537	~43%
`qwen38-awq-w4a16/`	400	537	~43%
`qwen38-gptq-w4a16/`	400	537	~43%
`qwen38-nvfp4a16/`	256	681	~27%
`qwen38-fp8-perchannel/`	496	441	~53%
`qwen38-fp8-block-mtp/`	407	530	~43%（其中 MTP 头 7 个模块也已量化）

未量化的层主要是三类（各版本 config.json 的 ignore 字段可查）：

线性注意力（Gated DeltaNet / linear_attn）投影：nvfp4a16 明确忽略了 240 条、autoawq 忽略 in_proj_a / in_proj_b —— 量化工具链对线性注意力算子支持不完整，硬量化容易掉点；
视觉塔 visual（约 110 个模块）：对精度敏感，且纯文本部署可用 --language-model-only 直接跳过；
lm_head 与各 norm / bias / embedding：量化收益小、风险大。

两个"为什么"要说清：

① SM75 没有 FP8 硬件能力（FP8 张量核从 Ada / SM89 才有）。所以在 2080Ti 上 FP8 权重只能走社区 fork 的模拟路径（Marlin fallback + 实时反量化），实际行为是"权重 FP8 + 计算 BF16"（weight-only），激活不参与量化 —— 这也是为什么单独做了一份 block 128×128 的 FP8（面向通用 vLLM 的 per-channel 那份在 SM75 上加载不了）。
② 覆盖率低 ≠ 残次品：BF16 层越多，显存占用越高、数值越稳；这套组合是"2080Ti 双卡 44GB 装得下 + 跑得出可用速度"之间的折中。想追求全量化 / 更省显存，请在 SM89+ 上用 qwen38-fp8-perchannel/，或自行重量化。

4. 适用硬件与运行时（哪些是 2080Ti 专用、哪些通用）

依据：llm-compressor / vLLM 官方 scheme 表的 minimum compute capability，以及社区 fork weicj/vLLM-2080Ti-Definitive README 中的 validated 组合表（其 FP8 与 NVFP4 条目正是在 2×2080Ti 上实测的）。

目录	格式	官方 vLLM 最低算力	2080Ti（SM75）	定位
`qwen38-autoawq-w4a16/`	AutoAWQ W4A16	7.5（Turing）	Yes原生支持	通用（≥SM75 都能跑）
`qwen38-awq-w4a16/`	原生 AWQ W4A16	7.5	原生	通用
`qwen38-gptq-w4a16/`	原生 GPTQ W4A16	7.5	原生	通用
`qwen38-fp8-perchannel/`	W8A8-FP8（per-channel，compressed-tensors）	8.9（Ada / Lovelace+）	加载不了	通用（面向 SM89+ 原生 FP8 环境）
`qwen38-fp8-block-mtp/`	FP8（block 128×128，vLLM 原生风格，MTP 头已量化）	官方 8.9	Yes靠 fork 模拟路径	2080Ti fork 专用（SM89+ 上同样可用）
`qwen38-nvfp4a16/`	NVFP4（compressed-tensors）	10.0（Blackwell）	Yes靠 fork 的 nvfp4 kernel	2080Ti fork 专用（官方要求 Blackwell）

一句话记法：AWQ / GPTQ（W4A16）是通用格式，2080Ti 官方原生支持；FP8 与 NVFP4 在 2080Ti 上完全依赖社区 fork 的模拟路径（官方 vLLM 分别要求 SM8.9 与 SM10.0）——换句话，"2080Ti 专用"的恰恰是那两个官方不支持这代卡的格式。

LoRA 适配器（复现 / 继续训练）

文件	说明
`adapter_model.safetensors` + `adapter_config.json`	LoRA r=8, alpha=8。若在 transformers 侧加载，请搭配 4-bit 基座 `unsloth/Qwen3.8-27B-unsloth-bnb-4bit`，不要配 16-bit 基座。日常推理直接用上面的量化版即可。

训练参数

基座：Qwen3.8-27B（qwen3_5 稠密架构）
方法：Unsloth QLoRA，r=8, alpha=8，基座 4-bit NF4
数据：佛光大辞典 + 中华佛教大百科全书，35,841 条（佛教 SFT，Plan A 清洗：0 空 think 块 / 0 重复 / 0 system 污染）
训练：3 epoch，13,443 步，final loss 1.591（2.245 → 1.591，健康收敛）
硬件：RTX 3090 24GB
导出：GGUF（forced_merged_4bit）→ Q4_K_M / Q6_K；safetensors 量化 → AutoAWQ（awq）、GPTQ（gptqmodel，vLLM 原生格式）

推理示例（llama.cpp）

# Q4_K_M（推荐，需 ~19GB 显存）
llama-server -m qwen3.8-27b-Buddhist-16bit.Q4_K_M.gguf -c 32768 -ngl 99

# 带视觉模块
llama-server -m qwen3.8-27b-Buddhist-16bit.Q4_K_M.gguf \
  --mmproj qwen3.8-27b-Buddhist-16bit.BF16-mmproj.gguf -c 32768 -ngl 99

显存需求参考：

硬件	Q4_K_M	Q6_K
RTX 3060 12GB	需 CPU offload（实测 ~19-23 t/s）	No
RTX 3090 24GB	全 GPU	全 GPU
双 2080Ti 44GB	全 GPU	全 GPU

许可

基于 Qwen3.8-27B（Apache-2.0）微调，权重与数据均以 Apache-2.0 发布。

Identity and Version

Repository: wulezhi/qwen3.8-27b-buddhist
Publisher: Wulezhi
Task: Text generation
Modality: Text
Library: peft
Parameters: Not stated by the source
Languages: sft, awq
Revision: 452514495463bfbbd3e630ff7d2548ed874beed8
First published: 2026-09-25
Last updated: 2026-10-08

Files and Weights

78 files, 189.8 GB in total. The weights are 28 files totalling 189.7 GB in gguf, safetensors.

Weights28 files · 189.7 GB

Configuration28 files · 1.2 MB

Tokenizer12 files · 119.9 MB

Documentation1 file · 11.1 KB

Other8 files · 91.2 KB

Repository1 file · 2.6 KB

Every file

File	Type	Size	SHA-256
gguf/Q4/qwen3.8-27b-Buddhist-16bit.Q4_K_M.gguf	Weights	16.8 GB	630a1543730c
gguf/Q6/qwen3.8-27b-Buddhist-16bit.Q6_K.gguf	Weights	22.4 GB	c31b1e657a90
gguf/qwen3.8-27b-Buddhist-16bit.BF16-mmproj.gguf	Weights	931.1 MB	6b684d29c3bc
lora/adapter_model.safetensors	Weights	233.6 MB	6747fc1f3fb7
qwen38-autoawq-w4a16/model-00001-of-00004.safetensors	Weights	5.0 GB	95c88989433a
qwen38-autoawq-w4a16/model-00002-of-00004.safetensors	Weights	5.0 GB	f111cf80c6a4
qwen38-autoawq-w4a16/model-00003-of-00004.safetensors	Weights	5.0 GB	c94a92d2864b
qwen38-autoawq-w4a16/model-00004-of-00004.safetensors	Weights	3.8 GB	625301c9c9f3
qwen38-autoawq-w4a16/model_mtp.safetensors	Weights	849.4 MB	1d8268aa85ac
qwen38-awq-w4a16/model-00001-of-00005.safetensors	Weights	3.5 GB	42f6f9784da6
qwen38-awq-w4a16/model-00002-of-00005.safetensors	Weights	4.3 GB	77bfae2f4a02
qwen38-awq-w4a16/model-00003-of-00005.safetensors	Weights	4.3 GB	fc53173abc44
qwen38-awq-w4a16/model-00004-of-00005.safetensors	Weights	4.3 GB	aa6a4cf16991
qwen38-awq-w4a16/model-00005-of-00005.safetensors	Weights	3.2 GB	2820e7b5a878
qwen38-fp8-block-mtp/model-00001-of-00002.safetensors	Weights	20.0 GB	1b064e23c388
qwen38-fp8-block-mtp/model-00002-of-00002.safetensors	Weights	10.4 GB	70ff65d84e7a
qwen38-fp8-block-mtp/model_mtp.safetensors	Weights	477.2 MB	bd29616f430e
qwen38-fp8-perchannel/model-00001-of-00002.safetensors	Weights	19.9 GB	a8469b346670
qwen38-fp8-perchannel/model-00002-of-00002.safetensors	Weights	10.4 GB	eafead3448b5
qwen38-fp8-perchannel/model_mtp.safetensors	Weights	849.4 MB	1d8268aa85ac
qwen38-gptq-w4a16/model-00001-of-00005.safetensors	Weights	3.5 GB	42f6f9784da6
qwen38-gptq-w4a16/model-00002-of-00005.safetensors	Weights	4.3 GB	16b5a7c54e0f
qwen38-gptq-w4a16/model-00003-of-00005.safetensors	Weights	4.3 GB	5256dfc385ca
qwen38-gptq-w4a16/model-00004-of-00005.safetensors	Weights	4.3 GB	7edb00d4065a
qwen38-gptq-w4a16/model-00005-of-00005.safetensors	Weights	3.2 GB	31f6140f48c8
qwen38-nvfp4a16/model-00001-of-00002.safetensors	Weights	20.0 GB	1c32085678ae
qwen38-nvfp4a16/model-00002-of-00002.safetensors	Weights	7.7 GB	6af72fcde0c9
qwen38-nvfp4a16/model_mtp.safetensors	Weights	849.4 MB	1d8268aa85ac
lora/adapter_config.json	Configuration	1.7 KB	—
qwen38-autoawq-w4a16/config.json	Configuration	3.9 KB	—
qwen38-autoawq-w4a16/generation_config.json	Configuration	214 B	—
qwen38-autoawq-w4a16/model.safetensors.index.json	Configuration	190.9 KB	—
qwen38-awq-w4a16/config.json	Configuration	5.3 KB	—
qwen38-awq-w4a16/generation_config.json	Configuration	202 B	—
qwen38-awq-w4a16/model.safetensors.index.json	Configuration	191.0 KB	—
qwen38-awq-w4a16/processor_config.json	Configuration	1.2 KB	—
qwen38-awq-w4a16/quantize_config.json	Configuration	1.2 KB	—
qwen38-fp8-block-mtp/config.json	Configuration	15.0 KB	—
qwen38-fp8-block-mtp/generation_config.json	Configuration	214 B	—
qwen38-fp8-block-mtp/model.safetensors.index.json	Configuration	155.8 KB	—
qwen38-fp8-block-mtp/processor_config.json	Configuration	1.2 KB	—
qwen38-fp8-perchannel/config.json	Configuration	9.8 KB	—
qwen38-fp8-perchannel/generation_config.json	Configuration	214 B	—
qwen38-fp8-perchannel/model.safetensors.index.json	Configuration	164.1 KB	—
qwen38-fp8-perchannel/processor_config.json	Configuration	1.2 KB	—
qwen38-fp8-perchannel/recipe.yaml	Configuration	261 B	—
qwen38-gptq-w4a16/config.json	Configuration	5.3 KB	—
qwen38-gptq-w4a16/generation_config.json	Configuration	202 B	—
qwen38-gptq-w4a16/model.safetensors.index.json	Configuration	229.8 KB	—
qwen38-gptq-w4a16/processor_config.json	Configuration	1.2 KB	—
qwen38-gptq-w4a16/quantize_config.json	Configuration	1.3 KB	—
qwen38-nvfp4a16/config.json	Configuration	24.3 KB	—
qwen38-nvfp4a16/generation_config.json	Configuration	214 B	—
qwen38-nvfp4a16/model.safetensors.index.json	Configuration	167.4 KB	—
qwen38-nvfp4a16/processor_config.json	Configuration	1.2 KB	—
qwen38-nvfp4a16/recipe.yaml	Configuration	251 B	—
README.md	Documentation	11.1 KB	—
qwen38-autoawq-w4a16/chat_template.jinja	Other	10.0 KB	—
qwen38-awq-w4a16/chat_template.jinja	Other	9.0 KB	—
qwen38-awq-w4a16/quant_log.csv	Other	17.2 KB	—
qwen38-fp8-block-mtp/chat_template.jinja	Other	9.0 KB	—
qwen38-fp8-perchannel/chat_template.jinja	Other	9.0 KB	—
qwen38-gptq-w4a16/chat_template.jinja	Other	9.0 KB	—
qwen38-gptq-w4a16/quant_log.csv	Other	19.2 KB	—
qwen38-nvfp4a16/chat_template.jinja	Other	9.0 KB	—
.gitattributes	Repository	2.6 KB	—
qwen38-autoawq-w4a16/tokenizer.json	Tokenizer	20.0 MB	06b9509352d2
qwen38-autoawq-w4a16/tokenizer_config.json	Tokenizer	1.2 KB	—
qwen38-awq-w4a16/tokenizer.json	Tokenizer	20.0 MB	06b9509352d2
qwen38-awq-w4a16/tokenizer_config.json	Tokenizer	1.1 KB	—
qwen38-fp8-block-mtp/tokenizer.json	Tokenizer	20.0 MB	06b9509352d2
qwen38-fp8-block-mtp/tokenizer_config.json	Tokenizer	1.2 KB	—
qwen38-fp8-perchannel/tokenizer.json	Tokenizer	20.0 MB	06b9509352d2
qwen38-fp8-perchannel/tokenizer_config.json	Tokenizer	1.2 KB	—
qwen38-gptq-w4a16/tokenizer.json	Tokenizer	20.0 MB	06b9509352d2
qwen38-gptq-w4a16/tokenizer_config.json	Tokenizer	1.1 KB	—
qwen38-nvfp4a16/tokenizer.json	Tokenizer	20.0 MB	06b9509352d2
qwen38-nvfp4a16/tokenizer_config.json	Tokenizer	1.2 KB	—

License and Download

License: apache-2.0
Access: Open weights, no gate
Download size: 189.7 GB

Download from Wulezhi

Released by Wulezhi through its official repository on Hugging Face. Read the license.

Built From

Adapter of unsloth/Qwen3.8-27B-unsloth-bnb-4bit
Derived from unsloth/Qwen3.8-27B-unsloth-bnb-4bit

Memory Requirements

Precision	Weights in memory
As published	189.7 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About qwen3.8-27b-buddhist

Can I use qwen3.8-27b-buddhist commercially?

Yes. qwen3.8-27b-buddhist is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

Similar Models

Model · Text generation

opt-125m

AI at Meta

OPT was first introduced in Open Pre-trained Transformer Language Models and first released in metaseq's repository on May 3rd 2022 by Meta AI. Disclaimer: The team releasing OPT wrote an official model card, which is available in Appendix D of the paper. Content from this model card has been written by the Hugging Face team. To quote the first two paragraphs of the official paper OPT was predominantly pretrained with English text, but a small amount of non-English data is still present within the training corpus via CommonCrawl. The model was pretrained using a causal language modeling (CLM) objective. OPT belongs to the same family of decoder-only models like GPT-3. As such, it was…

Open weights other 2,048 tokens transformers

View model

Model · Text generation

Ornith-1.5-9B-GGUF

Ornith

Chirp Chirp! We are introducing Ornith-1.5, a major step toward building foundation models through end-to-end self-improvement. Ornith-1.5 extends Ornith-1.0 (which was developed on top of Qwen3.5 and Gemma4 with additional continued pretraining, mid-training, and post-training) by expanding the self-improvement loop from scaffold and rollout optimization to jointly optimizing task generation, scaffold construction, and solution rollouts. Rather than relying on a fixed set of human-curated tasks and manually designed harnesses, Ornith-1.5 continuously generates new training tasks, discovers effective strategies for solving them, and improves the policy through reinforcement learning. For…

Open weights mit transformers

View model

Model · Text generation

Ornith-1.5-35B-A3B-GGUF

Ornith

Chirp Chirp! We are introducing Ornith-1.5, a major step toward building foundation models through end-to-end self-improvement. Ornith-1.5 extends Ornith-1.0 (which was developed on top of Qwen3.5 and Gemma4 with additional continued pretraining, mid-training, and post-training) by expanding the self-improvement loop from scaffold and rollout optimization to jointly optimizing task generation, scaffold construction, and solution rollouts. Rather than relying on a fixed set of human-curated tasks and manually designed harnesses, Ornith-1.5 continuously generates new training tasks, discovers effective strategies for solving them, and improves the policy through reinforcement learning. For…

Open weights mit transformers

View model

Model · Text generation

Ternary-Bonsai-2-27B-gguf

Prism ML

Full 27B-class reasoning in ternary transformer weights, for llama.cpp (CUDA, Metal, CPU) - \~5.9 GB language model (down from \~54 GB FP16) — full 27B-class reasoning on a standard laptop or a single GPU - 98.2% of FP16 intelligence retained: 84.78 average across 14 thinking-mode benchmarks — far above the conventional IQ2XXS build (72.59) at about 82% of its footprint, and within 0.4 points of UD-Q4KXL at three times the footprint - Retains thinking, reasoning, and agentic behavior deep in the sub-4-bit regime, where conventional low-bit representations collapse: math within half a point of full precision (96.57), coding level with the baseline (89.42), agentic tool calling at 74.92…

Open weights apache-2.0 llama.cpp

View model

Model · Text generation

Qwen3.8-27B-Uncensored-GGUF

Jonathan Coletti

Uncensored Qwen3.8-27B, published as GGUF quantizations with the multi token prediction (MTP) head retained and verified. Refusal behaviour has been substantially reduced, not eliminated. See Measured behaviour for the numbers. Capabilities, training data, and architecture are otherwise unchanged. - Refusal directions removed with Heretic, which co minimizes refusal count against KL divergence from the base model. No handwritten refusal removal code, no finetuning, no additional training data. - Abliteration runs at bf16 (no 4 bit quantization). the resulting LoRA is merged into the bf16 base, so the published weights are not a quantized round trip. - mtp. tensors are copied verbatim from…

Open weights apache-2.0 llama.cpp

View model

Model · Text generation

Ornith-1.0-9B-GGUF

Ornith

Aloha! Today, we are releasing Ornith-1.0, a self-improving family of open-source models for agentic coding. This model card documents Ornith-1.0-9B, the most lightweight member of the Ornith family, designed for efficient single-GPU deployment. Ornith-1.0-9B is a dense ~9B model (≈19 GB in bf16), so it serves comfortably on a single 80GB GPU. The recipes below stand up an OpenAI-compatible server; add --tensor-parallel-size / --tp if you want to shard across more GPUs. For a quick local test (or to script offline generation), load the model directly with Transformers. Make sure you have a recent release installed — see the Transformers installation guide; Ornith-1.0-9B requires…

Open weights mit transformers

View model