A 97.6M-parameter language model trained from scratch, then fine-tuned for short, polite, everyday English conversation. It is a small, open research and learning model: you can read every line of its training code, run it on a laptop CPU, and see exactly where a model of this size is good and where it fails. - HellaSwag 33.6% (accnorm). That is above GPT-2 small (~30%) and below SmolLM2-135M (43.1%), which saw about 250x more training text. - The custom PyTorch code is included. The model does not use transformers; see How to use. This is QuickTalk run 8. "LMLM97M1" is its published name. 1. Learning and teaching how LLMs work. It is a complete, small, from-scratch GPT, with its training…
pythia-70m-deduped is an open-weight model for text generation from EleutherAI, released under Apache License 2.0. It has 96M parameters and a 2,048-token context. At 16-bit it needs about 0.2 GB of GPU memory, which fits on 1x MI300X from $1.85 an hour, at the lowest prices in the SAVRN Index. It draws 1.4M downloads a month.
The Pythia Scaling Suite is a collection of models developed to facilitate interpretability research (see paper). It contains two sets of eight models of sizes 70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, and 12B.
Parameters96M
Context2,048
Weights332.1 MB
Licenseapache-2.0
AccessOpen weights
Monthly Downloads1.4M
Runs On
What it takes to serve pythia-70m-deduped (96M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.
| Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits |
|---|---|---|---|---|---|
| 16-bit | 0.2 GB | 0.2 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 8-bit | 0.1 GB | 0.1 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 4-bit | 0.0 GB | 0.1 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Oct 13, 2026.
pythia-70m-deduped on every accelerator the SAVRN Index prices, at every precision
SAVRN's Notes on pythia-70m-deduped
EleutherAI built the Pythia suite for interpretability research, and pythia-70m-deduped is its smallest member: 6 layers, hidden size 512, 96M parameters, trained on the deduplicated Pile in the same order as the other seven sizes up to 12B, plus 154 intermediate checkpoints. The 16-bit run needs 0.2 GB of memory. On the cheapest Index setup, one MI300X with 192 GB at $1.85 per hour, you could hold hundreds of copies at once, which is the point: many checkpoints resident together to watch a behavior emerge through training.
Apache 2.0 permits commercial use, modification and redistribution, keeps the notices with the code, and carries a patent grant. Check the dates: released February 13, 2023, last updated July 9, 2023, nothing since. The 2,048-token context is a research window, not a document window. The training set is EleutherAI/the_pile_deduplicated, documented in arXiv:2101.00027, and the weights ship in safetensors and pytorch.
Model Card
By EleutherAI, published under apache-2.0, revision e93a9faa9c77.
The Pythia Scaling Suite is a collection of models developed to facilitate interpretability research (see paper). It contains two sets of eight models of sizes 70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, and 12B. For each size, there are two models: one trained on the Pile, and one trained on the Pile after the dataset has been globally deduplicated. All 8 model sizes are trained on the exact same data, in the exact same order. We also provide 154 intermediate checkpoints per model, hosted on Hugging Face as branches.
The Pythia model suite was designed to promote scientific research on large language models, especially interpretability research. Despite not centering downstream performance as a design goal, we find the models match or exceed the performance of similar and same-sized models, such as those in the OPT and GPT-Neo suites.
Configuration
- Architecture
- GPTNeoXForCausalLM
- Context length (tokens)
- 2,048
- Layers
- 6
- Hidden size
- 512
- Feed-forward size
- 2,048
- Attention heads
- 8
- Vocabulary size
- 50,304
- Stored precision
- float16
- Model type
- gpt_neox
Identity and Version
- Repository
- EleutherAI/pythia-70m-deduped
- Publisher
- EleutherAI
- Task
- Text generation
- Modality
- Text
- Library
- transformers
- Parameters
- 96M parameters
- Languages
- en
- Revision
- e93a9faa9c77e5d09219f6c868bfc7a1bd65593c
- First published
- 2023-02-13
- Last updated
- 2023-07-09
Files and Weights
8 files, 334.2 MB in total. The weights are 2 files totalling 332.1 MB in bin, safetensors.
Weights2 files · 332.1 MB
Configuration2 files · 666 B
Tokenizer2 files · 2.1 MB
Documentation1 file · 13.7 KB
Repository1 file · 1.5 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| model.safetensors | Weights | 166.0 MB | 3da388330e45 |
| pytorch_model.bin | Weights | 166.0 MB | a03394baafe5 |
| config.json | Configuration | 567 B | — |
| special_tokens_map.json | Configuration | 99 B | — |
| README.md | Documentation | 13.7 KB | — |
| .gitattributes | Repository | 1.5 KB | — |
| tokenizer.json | Tokenizer | 2.1 MB | — |
| tokenizer_config.json | Tokenizer | 396 B | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 332.1 MB
Download from EleutherAI
Released by EleutherAI through its official repository on Hugging Face. Read the license.
Built From
- Described by arXiv:2101.00027
- Described by arXiv:2201.07311
- Described by arXiv:2304.01373
- Trained on (disclosed) EleutherAI/the_pile_deduplicated
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 332.1 MB |
| 16-bit | 0.2 GB |
| 8-bit | 0.1 GB |
| 4-bit | 0.0 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About pythia-70m-deduped
How much GPU memory does pythia-70m-deduped need?
About 0.2 GB at 16-bit and 0.1 GB at 4-bit: the weights (96M parameters) plus a working margin. A long context needs more.
What is the cheapest GPU to run pythia-70m-deduped on?
At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.
Can I use pythia-70m-deduped commercially?
Yes. pythia-70m-deduped is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.
What is pythia-70m-deduped's context length?
2,048 tokens, from the maximum position embeddings in its published configuration.
Similar Models
A 75M parameter decoder-only language model built entirely from scratch in PyTorch — no HuggingFace model classes, no nanoGPT wrapping. Every component (tokenizer, architecture, data pipeline, training loop, SFT) was written from scratch with Claude (Anthropic's AI assistant). Pretraining - ~17.8B tokens of English web text - AdamW (β₁=0.9, β₂=0.95), weight decay 0.1 SFT Fine-tuning - 100K examples from OpenHermes-2.5 - ChatML format with loss masking on user/system tokens Evaluated with log-likelihood scoring (no few-shot): Comparable to GPT-2 (117M) at 0.64× the parameter count. This is a small research model, built to learn how language models work from the ground up. It is not suitable…
Myosotis-1-base is the first flagship release from us, introducing a 100-million parameter recurrent language model built on the FWKV architecture. Myosotis-1 is engineered to never truly forget—using a mathematically clamped exponential decay that guarantees an infinite effective context window while maintaining blazing-fast inference on consumer hardware. Instead of pairwise attention, Myosotis uses a fixed-size state vector updated via a gated linear recurrence: $$ St = S{t-1} \odot W + kt \odot vt $$ - \\( W = \text{clamp}(\sigma(w), 0.1) \\) is a learned, constant-per-channel decay. - By clamping the minimum decay to 0.1, the model guarantees that past information decays exponentially…
Fine tuning pre-trained language models for text generation. Pretrained model on Chinese language using a GPT2 for Large Language Head Model objective. transferlearning from DavidLanz/uuufinetunetaipower and fine-tuning with medical dataset for the GPT-2 architecture. You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
macbert4csc-base-chinese evaluate SIGHAN2015 test data： 由于训练使用的数据使用了SIGHAN2015的训练集（复现paper），在SIGHAN2015的测试集上达到SOTA水平。 模型结构，魔改于softmaskedbert： 本项目开源在中文文本纠错项目：pycorrector，可支持macbert4csc模型，通过如下命令调用： 当然，你也可使用transformers调用： SIGHAN+Wang271K中文纠错数据集，数据格式： 如果需要训练macbert4csc，请参考https://github.com/shibing624/pycorrector/tree/master/pycorrector/macbert MacBERT is an improved BERT with novel MLM as correction pre-training task, which mitigates the discrepancy of pre-training and fine-tuning. Here is an example of our pre-training task. Except for the new pre-training task, we also incorporate the following techniques. Note that our MacBERT can be directly replaced with the original BERT as there is no…
DistilGPT2 (short for Distilled-GPT2) is an English-language model pre-trained with the supervision of the smallest version of Generative Pre-trained Transformer 2 (GPT-2). Like GPT-2, DistilGPT2 can be used to generate text. Users of this model card should also consider information about the design, training, and limitations of GPT-2. CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes. As the developers of GPT-2 (OpenAI) note in their model card, “language models like GPT-2 reflect the biases inherent to the systems they were trained on.” Significant research has explored bias and…