# Disentangling Geometry, Performance, and Training in
Source: https://savrn.com/papers/disentangling-geometry-performance-and-training-in-language-models
Markdown alternate of the page above; the site index is https://savrn.com/llms.txt

---

## Abstract

Geometric properties of Transformer weights, particularly the unembedding matrix, have been widely useful in language model interpretability research. Yet, their utility for estimating downstream performance remains unclear. In this work, we systematically investigate the relationship between model performance and the unembedding matrix geometry, particularly its effective rank. Our experiments, involving a suite of 108 OLMo-style language models trained under controlled variation, reveal several key findings. While the best-performing models often exhibit a high effective rank, this trend is not universal across tasks and training setups. Contrary to prior work, we find that low effective rank does not cause late-stage performance degradation in small models, but instead co-occurs with it; we find adversarial cases where low-rank models do not exhibit saturation. Moreover, we show that effective rank is strongly influenced by pre-training hyperparameters, such as batch size and weight decay, which in-turn affect the model's performance. Lastly, extending our analysis to other geometric metrics and final-layer representation, we find that these metrics are largely aligned, but none can reliably predict downstream performance. Overall, our findings suggest that the model's geometry, as captured by existing metrics, primarily reflects training choices rather than performance.

[Full paper on arXiv](https://arxiv.org/abs/2602.20433)

## Details

arXiv identifier

2602.20433

Published

2026-06-21

Authors

Atharva Kulkarni, Jacob Mitchell Springer, Arjun Subramonian, Swabha Swayamdipta

## Open Models Built on This Paper

Every model in the SAVRN Model Hub whose card cites this paper, most downloaded first, with what it takes to run each one.

| Model | Task | Size | License | Monthly downloads | Cheapest setup at 16-bit |
| --- | --- | --- | --- | --- | --- |
| [GeoLadder-40M-8B-bs32-lr0.0065-wd0.1-decay0.1](https://savrn.com/models/geoladder-40m-8b-bs32-lr0-0065-wd0-1-decay0-1) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-8B-bs32-lr0.0065-wd0.1-decay0.01](https://savrn.com/models/geoladder-40m-8b-bs32-lr0-0065-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-8B-bs32-lr0.0065-wd0.1-decay0.0](https://savrn.com/models/geoladder-40m-8b-bs32-lr0-0065-wd0-1-decay0-0) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-8B-bs256-lr0.0065-wd0.1-decay0.01](https://savrn.com/models/geoladder-40m-8b-bs256-lr0-0065-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-8B-bs128-lr0.0065-wd0.1-decay0.01](https://savrn.com/models/geoladder-40m-8b-bs128-lr0-0065-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-8B-bs32-lr0.0065-wd0.05-decay0.01](https://savrn.com/models/geoladder-40m-8b-bs32-lr0-0065-wd0-05-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-8B-bs32-lr0.0065-wd0.01-decay0.01](https://savrn.com/models/geoladder-40m-8b-bs32-lr0-0065-wd0-01-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-8B-bs32-lr0.00065-wd0.1-decay0.01](https://savrn.com/models/geoladder-40m-8b-bs32-lr0-00065-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-64B-bs256-lr0.0065-wd0.1-decay0.01](https://savrn.com/models/geoladder-40m-64b-bs256-lr0-0065-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-32B-bs64-lr0.0065-wd0.1-decay0.01](https://savrn.com/models/geoladder-40m-32b-bs64-lr0-0065-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-32B-bs512-lr0.0065-wd0.1-decay0.01](https://savrn.com/models/geoladder-40m-32b-bs512-lr0-0065-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-32B-bs32-lr0.0065-wd0.1-decay0.01](https://savrn.com/models/geoladder-40m-32b-bs32-lr0-0065-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-32B-bs256-lr0.0065-wd0.1-decay0.01](https://savrn.com/models/geoladder-40m-32b-bs256-lr0-0065-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-32B-bs128-lr0.0065-wd0.1-decay0.01](https://savrn.com/models/geoladder-40m-32b-bs128-lr0-0065-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-40M-128B-bs512-lr0.0065-wd0.1-decay0.01](https://savrn.com/models/geoladder-40m-128b-bs512-lr0-0065-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-28M-8B-bs32-lr0.074-wd0.1-decay0.01](https://savrn.com/models/geoladder-28m-8b-bs32-lr0-074-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-28M-8B-bs64-lr0.0074-wd0.1-decay0.01](https://savrn.com/models/geoladder-28m-8b-bs64-lr0-0074-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-28M-8B-bs512-lr0.0074-wd0.1-decay0.01](https://savrn.com/models/geoladder-28m-8b-bs512-lr0-0074-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-28M-8B-bs32-lr0.0074-wd0.1-decay0.1](https://savrn.com/models/geoladder-28m-8b-bs32-lr0-0074-wd0-1-decay0-1) Dill Lab at USC | — | — | — | — | — |
| [GeoLadder-28M-8B-bs32-lr0.0074-wd0.1-decay0.01](https://savrn.com/models/geoladder-28m-8b-bs32-lr0-0074-wd0-1-decay0-01) Dill Lab at USC | — | — | — | — | — |

## Explore More

- [All research papers](https://savrn.com/papers)
- [The model directory](https://savrn.com/models)
- [Datasets](https://savrn.com/datasets)

## Source

- arXiv metadata, read 2026-10-13.
- [How the hub is built](https://savrn.com/model-hub/methodology)