# Transcoder Adapters for Reasoning-Model Diffing
Source: https://savrn.com/papers/transcoder-adapters-for-reasoning-model-diffing
Markdown alternate of the page above; the site index is https://savrn.com/llms.txt

---

## Abstract

While reasoning models are increasingly ubiquitous, the effects of reasoning training on a model's internal mechanisms remain poorly understood. In this work, we introduce transcoder adapters, a technique for learning an interpretable approximation of the difference in MLP computation before and after fine-tuning. We apply transcoder adapters to characterize the differences between Qwen2.5-Math-7B and its reasoning-distilled variant, DeepSeek-R1-Distill-Qwen-7B. Learned adapters are faithful to the target model's internal computation and next-token predictions. When evaluated on reasoning benchmarks, adapters match the reasoning model's response lengths and typically recover 50-90% of the accuracy gains from reasoning fine-tuning. Adapter features are sparsely activating and interpretable. When examining adapter features, we find that only ~8% have activating examples directly related to reasoning behaviors. We deeply study one such behavior -- the production of hesitation tokens (e.g., "wait"). Using attribution graphs, we trace hesitation to only ~2.4% of adapter features (5.6k total) performing one of two functions. These features are necessary and sufficient for producing hesitation tokens; removing them reduces response length, often without affecting accuracy. Overall, our results provide insight into reasoning training and suggest transcoder adapters may be useful for studying fine-tuning more broadly.

[Full paper on arXiv](https://arxiv.org/abs/2602.20904) · [Code](https://github.com/https://github.com/nathanhu0/transcoder-adapters)

## Details

arXiv identifier

2602.20904

Published

2026-02-24

Authors

Nathan Hu, Jake Ward, Thomas Icard, Christopher Potts

## Open Models Built on This Paper

Every model in the SAVRN Model Hub whose card cites this paper, most downloaded first, with what it takes to run each one.

| Model | Task | Size | License | Monthly downloads | Cheapest setup at 16-bit |
| --- | --- | --- | --- | --- | --- |
| [2026.TA.gemma2_2b_huge_tc16384_decb_l1w0.0003_norm_sch_tarbb_lb2.0_ln1.0_dr500000_lr2e-04_bs8_sl](https://savrn.com/models/2026-ta-gemma2-2b-huge-tc16384-decb-l1w0-0003-norm-sch-tarbb-lb2-0-ln1-0-dr500000-lr2e-04) Siddharth M. Bhatia | — | 4.6B | — | 41 | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [2026.TA.gemma2_2b_mixed_truncate_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_sl14802576](https://savrn.com/models/2026-ta-gemma2-2b-mixed-truncate-tc8192-decb-l1w0-001-tarbb-lb2-0-ln1-dr20000-lr8e-04-sl14) Siddharth M. Bhatia | — | 3.6B | — | 7 | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [2026.TA.gemma2_2b_mixed_filter_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_sl14802575](https://savrn.com/models/2026-ta-gemma2-2b-mixed-filter-tc8192-decb-l1w0-001-tarbb-lb2-0-ln1-dr20000-lr8e-04-sl1480) Siddharth M. Bhatia | — | 3.6B | — | 7 | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793861](https://savrn.com/models/2026-ta-gemma2-2b-tc8192-decb-l1w0-001-tarbb-lb2-0-ln1-dr20000-lr8e-04-bs4-sl14793861) Siddharth M. Bhatia | — | 3.6B | — | 6 | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [2026.TA.gemma2_2b_mixed_filter_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_sl14797890](https://savrn.com/models/2026-ta-gemma2-2b-mixed-filter-tc8192-decb-l1w0-001-tarbb-lb2-0-ln1-dr20000-lr8e-04-sl1479) Siddharth M. Bhatia | — | 3.6B | — | 6 | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793862](https://savrn.com/models/2026-ta-gemma2-2b-tc8192-decb-l1w0-001-tarbb-lb2-0-ln1-dr20000-lr8e-04-bs4-sl14793862) Siddharth M. Bhatia | — | 3.6B | — | 5 | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793602](https://savrn.com/models/2026-ta-gemma2-2b-tc8192-decb-l1w0-001-tarbb-lb2-0-ln1-dr20000-lr8e-04-bs4-sl14793602) Siddharth M. Bhatia | — | 3.6B | — | 5 | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [2026.TA.gemma2_2b_mixed_truncate_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_sl14797891](https://savrn.com/models/2026-ta-gemma2-2b-mixed-truncate-tc8192-decb-l1w0-001-tarbb-lb2-0-ln1-dr20000-lr8e-04-sl14-2) Siddharth M. Bhatia | — | 3.6B | — | 5 | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [2026.TA.gemma2_2b_chat_truncate_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_sl14802574](https://savrn.com/models/2026-ta-gemma2-2b-chat-truncate-tc8192-decb-l1w0-001-tarbb-lb2-0-ln1-dr20000-lr8e-04-sl148) Siddharth M. Bhatia | — | 3.6B | — | 5 | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [2026.TA.gemma2_2b_chat_filter_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_sl14802573](https://savrn.com/models/2026-ta-gemma2-2b-chat-filter-tc8192-decb-l1w0-001-tarbb-lb2-0-ln1-dr20000-lr8e-04-sl14802) Siddharth M. Bhatia | — | 3.6B | — | 5 | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |

## Explore More

- [All research papers](https://savrn.com/papers)
- [The model directory](https://savrn.com/models)
- [Datasets](https://savrn.com/datasets)

## Source

- arXiv metadata, read 2026-10-11.
- [How the hub is built](https://savrn.com/model-hub/methodology)