De Schamphelaere

A dictionarylearning-format BatchTopK sparse autoencoder on the post-layer-42 residual stream of Qwen/Qwen3.6-27B (d = 5120), with F = 2,097,152 (2^21, 409.6x) features and k = 64. Companion to the 131,072-feature SAE at ceselder/qwen36-27b-sae-l42 (same layer, same recipe, 16x smaller dictionary). BatchTopK (Bussmann et al.), dictionarylearning-style trainer run feature-parallel over 8 GPUs (each rank owns 262,144 features; global top-k via all-gather, all-reduced reconstruction, aux-k loss alpha 1/32 with 40,960 aux features, learned activation threshold with EMA beta 0.999). lr 1.77e-5, warm-up 1,000 steps, decay from step 195,312, batch 4,096 tokens x 244,140 steps = 1.0e9 tokens…

The workspace Jacobian lens (Jacobian Lens paper; lenses from camilablank/workspace-lenses) is the averaged Jacobian Econtext[∂hpenult / ∂hℓ] of the penultimate residual (last block omitted) w.r.t. the layer-ℓ residual, read out with the model's finalnorm + WU. This repo composes J[42] with layer-42's MLP weights to give the averaged Jacobian to/from the MLP — i.e. the MLP state read through the workspace lens. d = 5120, dff = 17408, model = Qwen/Qwen3.6-27B, penultimate block = 62. - The read-out is an exact averaged Jacobian: it reads the current-token concept/operation the MLP contributes (e.g. bug → removing, antibiotics → healing, capital → geographical). It is a concept readout, not a…