Siddharth M. Bhatia

Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper). Load it with the repo's model classes; the adapter branch is part of the checkpoint, and settranscodersenabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings): Chat with it, or steer one feature by its cantor id (--steer CANTORID:STRENGTH): Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.buildgraphs --adaptercheckpoint…

Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper). Load it with the repo's model classes; the adapter branch is part of the checkpoint, and settranscodersenabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings): Chat with it, or steer one feature by its cantor id (--steer CANTORID:STRENGTH): Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.buildgraphs --adaptercheckpoint…

Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper). Load it with the repo's model classes; the adapter branch is part of the checkpoint, and settranscodersenabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings): Chat with it, or steer one feature by its cantor id (--steer CANTORID:STRENGTH): Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.buildgraphs --adaptercheckpoint…

Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper). Load it with the repo's model classes; the adapter branch is part of the checkpoint, and settranscodersenabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings): Chat with it, or steer one feature by its cantor id (--steer CANTORID:STRENGTH): Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.buildgraphs --adaptercheckpoint…

Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper). Load it with the repo's model classes; the adapter branch is part of the checkpoint, and settranscodersenabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings): Chat with it, or steer one feature by its cantor id (--steer CANTORID:STRENGTH): Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.buildgraphs --adaptercheckpoint…

Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper). Load it with the repo's model classes; the adapter branch is part of the checkpoint, and settranscodersenabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings): Chat with it, or steer one feature by its cantor id (--steer CANTORID:STRENGTH): Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.buildgraphs --adaptercheckpoint…

Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper). Load it with the repo's model classes; the adapter branch is part of the checkpoint, and settranscodersenabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings): Chat with it, or steer one feature by its cantor id (--steer CANTORID:STRENGTH): Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.buildgraphs --adaptercheckpoint…

Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper). Load it with the repo's model classes; the adapter branch is part of the checkpoint, and settranscodersenabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings): Chat with it, or steer one feature by its cantor id (--steer CANTORID:STRENGTH): Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.buildgraphs --adaptercheckpoint…

Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper). Load it with the repo's model classes; the adapter branch is part of the checkpoint, and settranscodersenabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings): Chat with it, or steer one feature by its cantor id (--steer CANTORID:STRENGTH): Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.buildgraphs --adaptercheckpoint…

Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper). Load it with the repo's model classes; the adapter branch is part of the checkpoint, and settranscodersenabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings): Chat with it, or steer one feature by its cantor id (--steer CANTORID:STRENGTH): Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.buildgraphs --adaptercheckpoint…

The setup. GemmaScope transcoders are sparse, interpretable modules that approximate each MLP layer of google/gemma-2-2b: for a layer, transcoder(x) ≈ MLPbase(x) where x is the post-ln2 hidden state, and only a small number of features fire (low L0 = active features/token). They are the building block circuit-tracer uses to turn a forward pass into an interpretable feature graph. Crucially, they were trained to imitate the base MLP on the base model's hidden states. The problem this fixes. When you circuit-trace the base↔instruct gap, you feed these transcoders the instruction-tuned model's (google/gemma-2-2b-it) hidden states instead — a distribution shift. We measured that this degrades…