Open-Weight Models

Model · Text generation

qwen3.8-flash-next-40b-prune-research

DavidB

One upstream MoE, six generations of masks, one bit-exact surgery, a rebuilt factual memory, and a three-stage post-training (SFT → SimPO → RLVR) — with every error I made and every idea I falsified along the way. This repository is the complete technical record of the project: the activation-instrumentation artifacts, the mask-generation methodology, the surgery code, the PLE (n-gram memory) rebuild, the trainers, and the full lab notebook. The champion weights live in sibling repos What I set out to do: take Qwen3.8-Flash-Next — an ~180B-class MoE (~130B of core network + 51B of n-gram memory table + 4B MTP) — and cut it by 75% of its experts (512 → 128) while conserving the maximum…

Model · Text generation

qwen3.8-flash-next-radiance-q5-w5dn64

Gostar

A checkpoint-specific calibrated quantization of Qwen/Qwen3.8-Flash-Next for the Radiance inference engine. This repository contains a single self-contained.rad inference container. It is not a Transformers, safetensors, llama.cpp, or vLLM checkpoint. Use a Radiance build with the Qwen3.8-Flash-Next model support and w4nl/w5nl expert kernels. - All 48 calibrated trunk layers use GPTQ with three coordinate-descent sweeps. - Routed expert down projections use non-linear w5nl (5-bit) codebooks, group 64, E4M3 scales, and FWHT128. - Routed expert gate/up projections use w4nl (4-bit) with the same transform and group structure. - Ten selected high-energy routed experts remain BF16. - Attention…

Qwen3.8-Flash-Next (48 hybrid layers: gated delta-net, sparse attention with an indexer, a 512-expert top-10 MoE at BF4, plus a PLE n-gram layer) served through vLLM and the Tenstorrent vLLM plugin on 4x p150 (a QuietBox or four p150 cards) or on a QuietBox 2 (2x p300c). One interactive user at a time; contexts of 32k, 64k or 128k tokens; thinking off by default. Updated 2026-09-26: prefill in 2048-row slabs by default (1,990 prompt tokens/s), decode 29.1 tokens/s greedy / 27.4 sampled at batch 1 (fork commit ce5b0f8eb6eb). Runs on 4x p150 (a QuietBox or four p150 cards; profiles c32k, c64k, c128k, c32k-quietbox) and on a QuietBox 2 (2x p300c; the default c32k or c32k-quietbox2) — see the…

Full Qwen3.5-9B model from the September 22, 2026 Harvey notes-only training runs. This revision is epoch 2, step 1550 of a two-epoch run, job 1016041. The final checkpoint is on main and final; epoch 1 is on epoch1. Each saved checkpoint is also available by its checkpoint- reference. - 100,000,076 note labels/epoch before causal shift; 99,993,882 scored labels/epoch after shift. - 223,681 notes, packed into 6,200 rows of 16,384 tokens with seed 731. - 199,987,764 scored supervised labels seen through this checkpoint. - No supervised trajectory examples. These are the notes-only controls, separate from notes + trajectory mixture models. - Eight GH200 GPUs, effective batch 8, learning rate…

Full Qwen3.5-9B model from the September 22, 2026 Harvey notes-only training runs. This revision is epoch 2, step 1550 of a two-epoch run, job 1016164. The final checkpoint is on main and final; epoch 1 is on epoch1. Each saved checkpoint is also available by its checkpoint- reference. - 100,000,076 note labels/epoch before causal shift; 99,993,882 scored labels/epoch after shift. - 223,681 notes, packed into 6,200 rows of 16,384 tokens with seed 731. - 199,987,764 scored supervised labels seen through this checkpoint. - No supervised trajectory examples. These are the notes-only controls, separate from notes + trajectory mixture models. - Eight GH200 GPUs, effective batch 8, learning rate…

Full Qwen3.5-9B model from the September 22, 2026 Harvey notes-only training runs. This revision is epoch 2, step 466 of a two-epoch run, job 1016040. The final checkpoint is on main and final; epoch 1 is on epoch1. Each saved checkpoint is also available by its checkpoint- reference. - 29,999,869 note labels/epoch before causal shift; 29,998,010 scored labels/epoch after shift. - 68,799 notes, packed into 1,864 rows of 16,384 tokens with seed 731. - 59,996,020 scored supervised labels seen through this checkpoint. - No supervised trajectory examples. These are the notes-only controls, separate from notes + trajectory mixture models. - Eight GH200 GPUs, effective batch 8, learning rate…

wm-internalization v4 checkpoint — condition kl-mix30m, save final. Full-finetune of Qwen/Qwen3.5-9B on the Calderwood & Harkness synthetic law-firm corpus (world-internalization study, v4 lineage: 9B student, ~50k think-on seed pool). Grafted back into the hub composite layout (Qwen35ForConditionalGeneration) — servable with vLLM out of the box.

EPQ（Expert Paging Quantization）是以 llama.cpp 為基礎的 CPU 推論路徑，讓 22.1 GB 的 Qwen3.6-35B-A3B-UD-Q4KM.gguf 在路由專家權重無法全部放入 RAM 時仍可執行。 GGUF 權重保留在儲存裝置，只有當前選中的專家會分頁載入容量受限的 RAM arena， MoE 專家計算則由 CPU 執行。本 repo 收錄程式碼與量測結果，不包含模型權重。 中文描述名：Qwen3-35B-A3B 專家分頁 4-bit 量化 CPU 推理版。 EPQ 的名稱刻意描述實際機制：MoE expert paging、一般 GGUF 權重量化，以及 CPU 低記憶體推理。本專案不是論文中的 SDQ（Sparse Decomposed Quantization）： 沒有 2:4 結構化稀疏硬體跳零計算，也不宣稱使用 GPU 稀疏加速。sdq/ Python package、sdq C++ 檔案與 SDQ 除錯環境變數是歷史 implementation identifiers， 仍保留以避免破壞既有驗證工具；對外產品名稱與執行入口已改為 EPQ。 需求：Linux x86-64、C++ 工具鏈、CMake、Git、Python 3、足以存放 22.1 GB GGUF 與建置檔的磁碟空間，以及可提供實用解碼速度的本機高速 SSD/NVMe。已測試路徑使用 llama.cpp commit 836d57176dc699a726c55418e4f96b8ca628e1bf。 在其他位置執行時，此腳本會取得本 repo…

A dictionarylearning-format BatchTopK sparse autoencoder on the post-layer-42 residual stream of Qwen/Qwen3.6-27B (d = 5120), with F = 2,097,152 (2^21, 409.6x) features and k = 64. Companion to the 131,072-feature SAE at ceselder/qwen36-27b-sae-l42 (same layer, same recipe, 16x smaller dictionary). BatchTopK (Bussmann et al.), dictionarylearning-style trainer run feature-parallel over 8 GPUs (each rank owns 262,144 features; global top-k via all-gather, all-reduced reconstruction, aux-k loss alpha 1/32 with 40,960 aux features, learned activation threshold with EMA beta 0.999). lr 1.77e-5, warm-up 1,000 steps, decay from step 195,312, batch 4,096 tokens x 244,140 steps = 1.0e9 tokens…

Uncensored build of Qwen/Qwen3.8-27B, produced with Apostate using the diode path. The diode repurposes one MLP neuron per layer into a gated refusal subtractor: it removes the residual refusal direction only when a benign-calibrated detector fires above threshold, so benign inputs keep the original weights. The result is a plain checkpoint: no runtime hook, adapter, finetune, or router. Delivery and KL are measured separately by apostate test, not during the bake; diodereport.json records the edit settings. This is a standard Transformers checkpoint. This model is uncensored and will answer harmful and dangerous requests. You are responsible for how you use it.

This is a Hugging Face safetensors export of the completed Step 8 full-parameter GRPO checkpoint. The language-model tensors are converted from the Miles/Megatron training checkpoint. The visual and MTP tensors, tokenizer, configuration, and processor assets come from Qwen/Qwen3.8-27B at revision The run trained on 1,586 SWE-rebench v2 Filtered tasks using independent mini-swe-agent rollouts. It used eight NVIDIA B300 GPUs (TP=2, CP=4), 32 prompt groups with eight rollouts each per optimizer step, global batch size 256, and learning rate 2e-6. This is the no-branch GRPO baseline checkpoint, not a branching-policy checkpoint. Source training-checkpoint completion marker SHA-256…

This repository preserves the completed Step 8 checkpoint from a full-parameter GRPO run using a 1,586-task SWE-rebench v2 Filtered corpus. It is a Miles/Megatron distributed checkpoint, including model and optimizer state, metadata, and the dataset cursor. It is intended for research recovery and continued training. The model weights for direct inference are published separately as a Hugging Face safetensors export. The archived run used eight NVIDIA B300 GPUs with tensor parallelism 2 and context parallelism 4. Each step requested 32 prompt groups with eight rollouts per group (global batch size 256); the learning rate was 2e-6. Training used mini-swe-agent, independent sandboxes and…

Model · Text generation

qwen38-40b-prune

DavidB

One upstream MoE, six generations of masks, one bit-exact surgery, a rebuilt factual memory, and a three-stage post-training (SFT → SimPO → RLVR) — with every error I made and every idea I falsified along the way. This repository is the complete technical record of the project: the activation-instrumentation artifacts, the mask-generation methodology, the surgery code, the PLE (n-gram memory) rebuild, the trainers, and the full lab notebook. The champion weights live in sibling repos What I set out to do: take Qwen3.8-Flash-Next — an ~180B-class MoE (~130B of core network + 51B of n-gram memory table + 4B MTP) — and cut it by 75% of its experts (512 → 128) while conserving the maximum…

5-layer DSpark MoE drafter for Qwen3.8-27B with 256 routed experts (width 512, top-10) and one shared expert (2048): Kan Wu's kan/moe-3-qwen38 SpecForge recipe with half the experts, trained from scratch for 3 epochs on RadixArk/Qwen3.8-27B-Regen-Mixture-v1 (global batch 256, lr 5e-4 constant) on one 4x GB300 node. Each finished epoch is a complete servable SGLang export in its own folder: Training curves: https://wandb.ai/milestraining/specforge-5layer-moe-0.5%2Aexperts/runs/qwen38-dspark-moe-256exp-3ep-gb300 Recipe and config: branch exp/qwen38-dspark-moe-256exp-3ep-gb300 of https://github.com/yushengsu-thu/SpecForge

2-layer MoE DSpark draft models for Qwen3.8-27B: an ablation of Kan Wu's 5-layer Qwen3.8-27B DSpark MoE drafter (SpecForge branch kan/moe-3-qwen38) with the draft reduced from 5 to 2 decoder layers and nothing else changed (512 routed experts x 512 wide, top-10 softmax routing, sigmoid-gated shared expert of width 2048, router centering; 8.48B total / about 1.08B active parameters). One servable export per training epoch, each in its own folder (config.json, model.safetensors, tokenizer and a README with the training metrics of that checkpoint): Serving needs an SGLang build with the Qwen3MoeDSparkModel draft class (upstream SGLang 0.5.18 only ships the dense Qwen3DSparkModel). Download one…

Placeholder. The servable export of the 2-layer Qwen3.8-27B DSpark MoE drafter after epoch 1 will be uploaded here automatically when that checkpoint is written. Live training metrics: https://wandb.ai/milestraining/specforge-2%20layer%20moe Code and recipes: https://github.com/yushengsu-thu/SpecForge/tree/exp/qwen3.8-27b-dspark-moe-2layer

Placeholder. The servable export of the 2-layer Qwen3.8-27B DSpark MoE drafter after epoch 2 will be uploaded here automatically when that checkpoint is written. Live training metrics: https://wandb.ai/milestraining/specforge-2%20layer%20moe Code and recipes: https://github.com/yushengsu-thu/SpecForge/tree/exp/qwen3.8-27b-dspark-moe-2layer