Open-weight model
GeoLadder-40M-128B-bs512-lr0.0065-wd0.1-decay0.01
by Dill Lab at USC dill-lab/GeoLadder-40M-128B-bs512-lr0.0065-wd0.1-decay0.01
GeoLadder-40M-128B-bs512-lr0.0065-wd0.1-decay0.01 is an open-weight model from Dill Lab at USC. Its published files total 787.2 MB.
OLMo-style model trained on the deduplicated Pile, one of the 108 models released with (ICML 2026).
Model Card
OLMo-style model trained on the deduplicated Pile, one of the 108 models released with (ICML 2026). Disentangling Geometry, Performance, and Training in Language Models Atharva Kulkarni, Jacob Mitchell Springer, Arjun Subramonian, Swabha Swayamdipta ICML 2026 (PMLR 306) · Paper · The paper asks whether the geometry of a language model's unembedding matrix, in particular its effective rank, can estimate downstream performance. Using a suite of 108 OLMo-style models (4M-75M parameters) trained under controlled variations of batch size, weight decay, learning rate, token budget and final LR, it finds that the relationship is inconsistent: high effective rank often accompanies the best models…
Excerpt from the card by Dill Lab at USC.
Configuration
- Architecture
- OLMoForCausalLM
- Vocabulary size
- 50,280
- RoPE base
- 10,000
- Model type
- hf_olmo
Identity and Version
- Repository
- dill-lab/GeoLadder-40M-128B-bs512-lr0.0065-wd0.1-decay0.01
- Publisher
- Dill Lab at USC
- Task
- Not stated by the source
- Modality
- Other
- Library
- Not stated by the source
- Parameters
- Not stated by the source
- Languages
- Not stated by the source
- Revision
- 38a4e41831efabe547c0d41423743501781559f8
- First published
- 2026-10-12
- Last updated
- 2026-10-13
Files and Weights
14 files, 787.2 MB in total. The weights are 1 file totalling 269.0 MB in bin.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| pytorch_model.bin | Weights | 269.0 MB | 12bb7befb14b |
| config.json | Configuration | 1.3 KB | — |
| config.yaml | Configuration | 16.3 KB | — |
| README.md | Documentation | 3.2 KB | — |
| data-indices/rank0.tsv.gz | Other | 33.2 MB | 8c74785851c1 |
| data-indices/rank1.tsv.gz | Other | 33.2 MB | 36018127111d |
| data-indices/rank2.tsv.gz | Other | 33.2 MB | 73d247395246 |
| data-indices/rank3.tsv.gz | Other | 33.2 MB | 89fb368801e8 |
| data-indices/rank4.tsv.gz | Other | 33.2 MB | 5d99e74ac96f |
| data-indices/rank5.tsv.gz | Other | 33.2 MB | 5fdbd3f321b5 |
| data-indices/rank6.tsv.gz | Other | 33.2 MB | cd013356bcdc |
| data-indices/rank7.tsv.gz | Other | 33.2 MB | 754506955955 |
| train_data/global_indices.npy | Other | 252.7 MB | b2da778b51ed |
| .gitattributes | Repository | 1.5 KB | — |
License and Download
- License
- Not stated by the source
- Access
- Open weights, no gate
- Download size
- 269.0 MB
Released by Dill Lab at USC through its official repository on Hugging Face.
Built From
- Described by arXiv:2602.20433
- Trained on (disclosed) EleutherAI/the_pile_deduplicated
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 269.0 MB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.