Open-weight model
2026.TA.gemma2_2b_gemmascope_transcoders_instruct_ft_L0-24-25
by Siddharth M. Bhatia siddharthmb/2026.TA.gemma2_2b_gemmascope_transcoders_instruct_ft_L0-24-25
2026.TA.gemma2_2b_gemmascope_transcoders_instruct_ft_L0-24-25 is an open-weight model from Siddharth M. Bhatia. Its published files total 906.9 MB.
The setup. GemmaScope transcoders are sparse, interpretable modules that approximate each MLP layer of google/gemma-2-2b: for a layer, transcoder(x) ≈ MLPbase(x) where x is the post-ln2 hidden state, and only a small number of features fire (low L0 = active…
Model Card
The setup. GemmaScope transcoders are sparse, interpretable modules that approximate each MLP layer of google/gemma-2-2b: for a layer, transcoder(x) ≈ MLPbase(x) where x is the post-ln2 hidden state, and only a small number of features fire (low L0 = active features/token). They are the building block circuit-tracer uses to turn a forward pass into an interpretable feature graph. Crucially, they were trained to imitate the base MLP on the base model's hidden states. The problem this fixes. When you circuit-trace the base↔instruct gap, you feed these transcoders the instruction-tuned model's (google/gemma-2-2b-it) hidden states instead — a distribution shift. We measured that this degrades…
Excerpt from the card by Siddharth M. Bhatia.
Identity and Version
- Repository
- siddharthmb/2026.TA.gemma2_2b_gemmascope_transcoders_instruct_ft_L0-24-25
- Publisher
- Siddharth M. Bhatia
- Task
- Not stated by the source
- Modality
- Other
- Library
- Not stated by the source
- Parameters
- Not stated by the source
- Languages
- Not stated by the source
- Revision
- 0734d7e1df8bfc02c297abff1ab39746202bc646
- First published
- 2026-07-15
- Last updated
- 2026-10-11
Files and Weights
12 files, 906.9 MB in total. The weights are 3 files totalling 906.4 MB in safetensors.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| finetuned_layer_0.safetensors | Weights | 302.1 MB | b8adc636e65b |
| finetuned_layer_24.safetensors | Weights | 302.1 MB | e4998387ba21 |
| finetuned_layer_25.safetensors | Weights | 302.1 MB | 3db1edb59e9d |
| finetune_report.json | Configuration | 2.4 KB | — |
| gemmascope_config.json | Configuration | 3.4 KB | — |
| README.md | Documentation | 8.8 KB | — |
| summary.md | Documentation | 783 B | — |
| figures/transcoder_finetune_before_after.png | Other | 44.2 KB | — |
| figures/transcoder_finetune_l0_before_after.png | Other | 47.3 KB | — |
| figures/transcoder_finetune_sparsity_comparison.png | Other | 104.4 KB | bb144689ebb9 |
| figures/transcoder_input_shift_overview.png | Other | 297.0 KB | 964ceb256ea6 |
| .gitattributes | Repository | 1.7 KB | — |
License and Download
- License
- Not stated by the source
- Access
- Open weights, no gate
- Download size
- 906.4 MB
Released by Siddharth M. Bhatia through its official repository on Hugging Face.
Built From
- Derived from google/gemma-2-2b
- Trained on (disclosed) science-of-finetuning/fineweb-1m-sample
- Trained on (disclosed) siddharthmb/2026.transcoder-adapters.lmsys-chat-1m-splits
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 906.4 MB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.