qwen36-27b-sae2m-l42

Qwen3.6-27B layer-42 BatchTopK SAE — 2,097,152 features (k = 64)

A dictionary_learning-format BatchTopK sparse autoencoder on the post-layer-42 residual stream of Qwen/Qwen3.6-27B (d = 5120), with F = 2,097,152 (2^21, 409.6x) features and k = 64. Companion to the 131,072-feature SAE at ceselder/qwen36-27b-sae-l42 (same layer, same recipe, 16x smaller dictionary).

Files

Training

• BatchTopK (Bussmann et al.), dictionary_learning-style trainer run feature-parallel over 8 GPUs (each rank owns 262,144 features; global top-k via all-gather, all-reduced reconstruction, aux-k loss alpha 1/32 with 40,960 aux features, learned activation threshold with EMA beta 0.999). lr 1.77e-5, warm-up 1,000 steps, decay from step 195,312, batch 4,096 tokens x 244,140 steps = 1.0e9 tokens streamed from openbmb/Ultra-FineWeb (en, contexts of 512, documents from index 100,000 on), activations scaled to unit mean norm (norm factor 83.37, folded back into the weights), bf16 matrices.
• Caveat, please read: the training-time streamer sharded the dataset AFTER applying the document skip, so all 8 ranks streamed the SAME documents — the run saw ~125M unique tokens, each 8 times, rather than 1B unique tokens. The bug was found after training and fixed before the max-activation scan (maxacts_top5.pt is over the true 1B distinct tokens). Held-out reconstruction is nevertheless on par with the 131k SAE (below); a retrain on 1B distinct tokens would likely improve it further.

Held-out metrics (verify.json; 20,004,864 fresh Ultra-FineWeb tokens, single stream)

For comparison the 131k SAE scores EV 0.693-0.704 at L0 64 with 0.017% dead features.

Fire-rate percentiles over the 1B-token scan (fraction of tokens): p1 2.0e-6, p50 5.0e-6, p95 6.5e-5, p99 4.5e-4. Peak activation percentiles: p1 3.8, p50 6.3, p95 17.8, p99 25.3.

Loading

import torch
sd = torch.load("saes_Qwen_Qwen3.6-27B_batch_top_k/resid_post_layer_42/trainer_0/ae.pt", map_location="cpu")  # 43 GB
W_enc, b_enc, W_dec, b_dec, thr = sd["encoder.weight"], sd["encoder.bias"], sd["decoder.weight"], sd["b_dec"], float(sd["threshold"])
# encode: pre = (h - b_dec) @ W_enc.T + b_enc ; acts = pre * (pre > thr)     (BatchTopK inference uses the learned threshold, not top-k)
# decode: h_hat = acts @ W_dec.T + b_dec

h is the raw post-layer-42 residual (model.layers[42] output) of Qwen3.6-27B; the norm factor is already folded in.

Provenance

Trained 2026-09-11 (Modal, 8 x B200, ~8 h) for the MAEMM activation-to-text inverter project (MATS, Neel Nanda stream). Max-act windows: docs_used 747,623, tokens_seen 1,000,013,824. Related datasets: ceselder/maemm-27b-heldout.