# 2026.TA.gemma2_2b_huge…lr2e-04_bs8_sl by Siddharth M. Bhatia Source: https://savrn.com/models/2026-ta-gemma2-2b-huge-tc16384-decb-l1w0-0003-norm-sch-tarbb-lb2-0-ln1-0-dr500000-lr2e-04 Markdown alternate of the page above; the site index is https://savrn.com/llms.txt --- ## Runs On What it takes to serve 2026.TA.gemma2_2b_huge_tc16384_decb_l1w0.0003_norm_sch_tarbb_lb2.0_ln1.0_dr500000_lr2e-04_bs8_sl (4.6B parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them. | Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits | | --- | --- | --- | --- | --- | --- | | 16-bit | 9.2 GB | 11.0 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | | 8-bit | 4.6 GB | 5.5 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | | 4-bit | 2.3 GB | 2.7 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the [SAVRN Index](https://savrn.com/ai-index/pricing/gpus), read Oct 12, 2026. [2026.TA.gemma2_2b_huge_tc16384_decb_l1w0.0003_norm_sch_tarbb_lb2.0_ln1.0_dr500000_lr2e-04_bs8_sl on every accelerator the SAVRN Index prices, at every precision](https://savrn.com/models/2026-ta-gemma2-2b-huge-tc16384-decb-l1w0-0003-norm-sch-tarbb-lb2-0-ln1-0-dr500000-lr2e-04/gpus) ## Model Card Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper). Load it with the repo's model classes; the adapter branch is part of the checkpoint, and settranscodersenabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings): Chat with it, or steer one feature by its cantor id (--steer CANTORID:STRENGTH): Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.buildgraphs --adaptercheckpoint… Excerpt from the card by Siddharth M. Bhatia. ## Configuration Architecture Gemma2ForCausalLMWithTranscoder Context length (tokens) 8,192 Layers 26 Hidden size 2,304 Feed-forward size 9,216 Attention heads 8 Key/value heads 4 Head dimension 256 Vocabulary size 256,000 Sliding window (tokens) 4,096 RoPE base 10000 Model type gemma2 ## Identity and Version Repository siddharthmb/2026.TA.gemma2_2b_huge_tc16384_decb_l1w0.0003_norm_sch_tarbb_lb2.0_ln1.0_dr500000_lr2e-04_bs8_sl Publisher Siddharth M. Bhatia Task Not stated by the source Modality Other Library transformers Parameters 4.6B parameters Languages Not stated by the source Revision a2fa85aabd58d58ca95124957dd68d329ebad249 First published 2026-07-01 Last updated 2026-10-11 ## Files and Weights 13 files, 9.2 GB in total. The weights are 2 files totalling 9.2 GB in safetensors. Weights2 files · 9.2 GB Configuration5 files · 37.1 KB Tokenizer3 files · 38.7 MB Documentation1 file · 5.6 KB Other1 file · 591 B Repository1 file · 1.6 KB Every file | File | Type | Size | SHA-256 | | --- | --- | --- | --- | | model-00001-of-00002.safetensors | Weights | 5.0 GB | 8cf0e891f5fa | | model-00002-of-00002.safetensors | Weights | 4.2 GB | a630e6a069a9 | | config.json | Configuration | 1.5 KB | — | | generation_config.json | Configuration | 187 B | — | | model.safetensors.index.json | Configuration | 33.0 KB | — | | special_tokens_map.json | Configuration | 636 B | — | | train_config.yaml | Configuration | 1.8 KB | — | | README.md | Documentation | 5.6 KB | — | | chat_template.jinja | Other | 591 B | — | | .gitattributes | Repository | 1.6 KB | — | | tokenizer.json | Tokenizer | 34.4 MB | 5f7eee611703 | | tokenizer.model | Tokenizer | 4.2 MB | 61a7b147390c | | tokenizer_config.json | Tokenizer | 46.4 KB | — | ## License and Download License Not stated by the source Access Open weights, no gate Download size 9.2 GB [Download from Siddharth M. Bhatia](https://huggingface.co/siddharthmb/2026.TA.gemma2_2b_huge_tc16384_decb_l1w0.0003_norm_sch_tarbb_lb2.0_ln1.0_dr500000_lr2e-04_bs8_sl) Released by Siddharth M. Bhatia through its official repository on Hugging Face. ## Built From - Derived from google/gemma-2-2b - Described by [arXiv:2602.20904](https://savrn.com/papers/transcoder-adapters-for-reasoning-model-diffing) - Trained on (disclosed) science-of-finetuning/fineweb-1m-sample - Trained on (disclosed) siddharthmb/2026.transcoder-adapters.lmsys-chat-1m-splits ## Memory Requirements | Precision | Weights in memory | | --- | --- | | As published | 9.2 GB | | 16-bit | 9.2 GB | | 8-bit | 4.6 GB | | 4-bit | 2.3 GB | Weights only, from the published parameter count; the key-value cache and runtime add to this. ## Questions About 2026.TA.gemma2_2b_huge_tc16384_decb_l1w0.0003_norm_sch_tarbb_lb2.0_ln1.0_dr500000_lr2e-04_bs8_sl ### How much GPU memory does 2026.TA.gemma2_2b_huge_tc16384_decb_l1w0.0003_norm_sch_tarbb_lb2.0_ln1.0_dr500000_lr2e-04_bs8_sl need? About 11 GB at 16-bit and 2.7 GB at 4-bit: the weights (4.6B parameters) plus a working margin. A long context needs more. ### What is the cheapest GPU to run 2026.TA.gemma2_2b_huge_tc16384_decb_l1w0.0003_norm_sch_tarbb_lb2.0_ln1.0_dr500000_lr2e-04_bs8_sl on? At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists. ### What is 2026.TA.gemma2_2b_huge_tc16384_decb_l1w0.0003_norm_sch_tarbb_lb2.0_ln1.0_dr500000_lr2e-04_bs8_sl's context length? 8,192 tokens, from the maximum position embeddings in its published configuration. ## Siddharth M. Bhatia [All models and datasets](https://savrn.com/model-publishers/siddharthmb) ## Versions - [a2fa85aabd58](https://savrn.com/models/2026-ta-gemma2-2b-huge-tc16384-decb-l1w0-0003-norm-sch-tarbb-lb2-0-ln1-0-dr500000-lr2e-04/versions/a2fa85aabd58) · current 2026-10-11 ## Explore More - [Model comparisons](https://savrn.com/models/comparisons) - [The model directory](https://savrn.com/models) - [Open model prices by host](https://savrn.com/ai-index/pricing/open-models) ## Source - Repository metadata, read 2026-10-11. - [Hugging Face record](https://huggingface.co/siddharthmb/2026.TA.gemma2_2b_huge_tc16384_decb_l1w0.0003_norm_sch_tarbb_lb2.0_ln1.0_dr500000_lr2e-04_bs8_sl) - [How the hub is built](https://savrn.com/model-hub/methodology)