# pythia-70m-deduped by EleutherAI: Open-Weight Model Source: https://savrn.com/models/pythia-70m-deduped Markdown alternate of the page above; the site index is https://savrn.com/llms.txt --- ## Runs On What it takes to serve pythia-70m-deduped (96M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them. | Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits | | --- | --- | --- | --- | --- | --- | | 16-bit | 0.2 GB | 0.2 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | | 8-bit | 0.1 GB | 0.1 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | | 4-bit | 0.0 GB | 0.1 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the [SAVRN Index](https://savrn.com/ai-index/pricing/gpus), read Oct 22, 2026. [pythia-70m-deduped on every accelerator the SAVRN Index prices, at every precision](https://savrn.com/models/pythia-70m-deduped/gpus) ## SAVRN's Notes on pythia-70m-deduped EleutherAI built the Pythia suite for interpretability research, and pythia-70m-deduped is its smallest member: 6 layers, hidden size 512, 96M parameters, trained on the deduplicated Pile in the same order as the other seven sizes up to 12B, plus 154 intermediate checkpoints. The 16-bit run needs 0.2 GB of memory. On the cheapest Index setup, one MI300X with 192 GB at $1.85 per hour, you could hold hundreds of copies at once, which is the point: many checkpoints resident together to watch a behavior emerge through training. Apache 2.0 permits commercial use, modification and redistribution, keeps the notices with the code, and carries a patent grant. Check the dates: released February 13, 2023, last updated July 9, 2023, nothing since. The 2,048-token context is a research window, not a document window. The training set is EleutherAI/the_pile_deduplicated, documented in arXiv:2101.00027, and the weights ship in safetensors and pytorch. SAVRN Research, 2026-09-18 ## Model Card By EleutherAI, published under apache-2.0, revision e93a9faa9c77. The Pythia Scaling Suite is a collection of models developed to facilitate interpretability research [(see paper)](https://savrn.com/papers/pythia-a-suite-for-analyzing-large-language-models-across-training-and-scaling). It contains two sets of eight models of sizes 70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, and 12B. For each size, there are two models: one trained on the Pile, and one trained on the Pile after the dataset has been globally deduplicated. All 8 model sizes are trained on the exact same data, in the exact same order. We also provide 154 intermediate checkpoints per model, hosted on Hugging Face as branches. The Pythia model suite was designed to promote scientific research on large language models, especially interpretability research. Despite not centering downstream performance as a design goal, we find the models match or exceed the performance of similar and same-sized models, such as those in the OPT and GPT-Neo suites. [Read the full model card (1,373 words)](https://savrn.com/models/pythia-70m-deduped/card) ## Configuration Architecture GPTNeoXForCausalLM Context length (tokens) 2,048 Layers 6 Hidden size 512 Feed-forward size 2,048 Attention heads 8 Vocabulary size 50,304 Stored precision float16 Model type gpt_neox ## Identity and Version Repository EleutherAI/pythia-70m-deduped Publisher EleutherAI Task Text generation Modality Text Library transformers Parameters 96M parameters Languages en Revision e93a9faa9c77e5d09219f6c868bfc7a1bd65593c First published 2023-02-13 Last updated 2023-07-09 ## Files and Weights 8 files, 334.2 MB in total. The weights are 2 files totalling 332.1 MB in bin, safetensors. Weights2 files · 332.1 MB Configuration2 files · 666 B Tokenizer2 files · 2.1 MB Documentation1 file · 13.7 KB Repository1 file · 1.5 KB Every file | File | Type | Size | SHA-256 | | --- | --- | --- | --- | | model.safetensors | Weights | 166.0 MB | 3da388330e45 | | pytorch_model.bin | Weights | 166.0 MB | a03394baafe5 | | config.json | Configuration | 567 B | — | | special_tokens_map.json | Configuration | 99 B | — | | README.md | Documentation | 13.7 KB | — | | .gitattributes | Repository | 1.5 KB | — | | tokenizer.json | Tokenizer | 2.1 MB | — | | tokenizer_config.json | Tokenizer | 396 B | — | ## License and Download License apache-2.0 Access Open weights, no gate Download size 332.1 MB [Download from EleutherAI](https://huggingface.co/EleutherAI/pythia-70m-deduped) Released by EleutherAI through its official repository on Hugging Face. [Read the license](https://www.apache.org/licenses/LICENSE-2.0). ## Built From - Described by [arXiv:2101.00027](https://savrn.com/papers/the-pile-an-800gb-dataset-of-diverse-text-for-language-modeling) - Described by [arXiv:2201.07311](https://savrn.com/papers/datasheet-for-the-pile) - Described by [arXiv:2304.01373](https://savrn.com/papers/pythia-a-suite-for-analyzing-large-language-models-across-training-and-scaling) - Trained on (disclosed) EleutherAI/the_pile_deduplicated ## Memory Requirements | Precision | Weights in memory | | --- | --- | | As published | 332.1 MB | | 16-bit | 0.2 GB | | 8-bit | 0.1 GB | | 4-bit | 0.0 GB | Weights only, from the published parameter count; the key-value cache and runtime add to this. ## Questions About pythia-70m-deduped ### How much GPU memory does pythia-70m-deduped need? About 0.2 GB at 16-bit and 0.1 GB at 4-bit: the weights (96M parameters) plus a working margin. A long context needs more. ### What is the cheapest GPU to run pythia-70m-deduped on? At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists. ### Can I use pythia-70m-deduped commercially? Yes. pythia-70m-deduped is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors. ### What is pythia-70m-deduped's context length? 2,048 tokens, from the maximum position embeddings in its published configuration. ## Similar Models Model · Text generation ### [LMLM_97M_1](https://savrn.com/models/lmlm-97m-1) [Sr Aivante](https://savrn.com/model-publishers/sraivante) A 97.6M-parameter language model trained from scratch, then fine-tuned for short, polite, everyday English conversation. It is a small, open research and learning model: you can read every line of its training code, run it on a laptop CPU, and see exactly where a model of this size is good and where it fails. - HellaSwag 33.6% (accnorm). That is above GPT-2 small (~30%) and below SmolLM2-135M (43.1%), which saw about 250x more training text. - The custom PyTorch code is included. The model does not use transformers; see How to use. This is QuickTalk run 8. "LMLM97M1" is its published name. 1. Learning and teaching how LLMs work. It is a complete, small, from-scratch GPT, with its training… Open weights apache-2.0 98M parameters pytorch [View model](https://savrn.com/models/lmlm-97m-1) Model · Text generation ### [tyrian-75m](https://savrn.com/models/tyrian-75m) [Phil McCanham](https://savrn.com/model-publishers/redptam) A 75M parameter decoder-only language model built entirely from scratch in PyTorch — no HuggingFace model classes, no nanoGPT wrapping. Every component (tokenizer, architecture, data pipeline, training loop, SFT) was written from scratch with Claude (Anthropic's AI assistant). Pretraining - ~17.8B tokens of English web text - AdamW (β₁=0.9, β₂=0.95), weight decay 0.1 SFT Fine-tuning - 100K examples from OpenHermes-2.5 - ChatML format with loss masking on user/system tokens Evaluated with log-likelihood scoring (no few-shot): Comparable to GPT-2 (117M) at 0.64× the parameter count. This is a small research model, built to learn how language models work from the ground up. It is not suitable… Open weights mit 99M parameters transformers [View model](https://savrn.com/models/tyrian-75m) Model · Text generation ### [Myosotis-1-base](https://savrn.com/models/myosotis-1-base) [FWKV Project](https://savrn.com/model-publishers/fwkv) Myosotis-1-base is the first flagship release from us, introducing a 100-million parameter recurrent language model built on the FWKV architecture. Myosotis-1 is engineered to never truly forget—using a mathematically clamped exponential decay that guarantees an infinite effective context window while maintaining blazing-fast inference on consumer hardware. Instead of pairwise attention, Myosotis uses a fixed-size state vector updated via a gated linear recurrence: $$ St = S{t-1} \odot W + kt \odot vt $$ - \\( W = \text{clamp}(\sigma(w), 0.1) \\) is a learned, constant-per-channel decay. - By clamping the minimum decay to 0.1, the model guarantees that past information decays exponentially… Open weights apache-2.0 102M parameters transformers [View model](https://savrn.com/models/myosotis-1-base) Model · Text generation ### [uuu_fine_tune_gpt2](https://savrn.com/models/uuu-fine-tune-gpt2) [David Lanz](https://savrn.com/model-publishers/davidlanz) Fine tuning pre-trained language models for text generation. Pretrained model on Chinese language using a GPT2 for Large Language Head Model objective. transferlearning from DavidLanz/uuufinetunetaipower and fine-tuning with medical dataset for the GPT-2 architecture. You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we Open weights gpl 102M parameters transformers [View model](https://savrn.com/models/uuu-fine-tune-gpt2) Model · Text generation ### [macbert4csc-base-chinese](https://savrn.com/models/macbert4csc-base-chinese) [Ming Xu (徐明)](https://savrn.com/model-publishers/shibing624) macbert4csc-base-chinese evaluate SIGHAN2015 test data： 由于训练使用的数据使用了SIGHAN2015的训练集（复现paper），在SIGHAN2015的测试集上达到SOTA水平。 模型结构，魔改于softmaskedbert： 本项目开源在中文文本纠错项目：pycorrector，可支持macbert4csc模型，通过如下命令调用： 当然，你也可使用transformers调用： SIGHAN+Wang271K中文纠错数据集，数据格式： 如果需要训练macbert4csc，请参考https://github.com/shibing624/pycorrector/tree/master/pycorrector/macbert MacBERT is an improved BERT with novel MLM as correction pre-training task, which mitigates the discrepancy of pre-training and fine-tuning. Here is an example of our pre-training task. Except for the new pre-training task, we also incorporate the following techniques. Note that our MacBERT can be directly replaced with the original BERT as there is no… Open weights apache-2.0 102M parameters 512 tokens transformers [View model](https://savrn.com/models/macbert4csc-base-chinese) Model · Text generation ### [distilgpt2](https://savrn.com/models/distilgpt2) [DistilBERT community](https://savrn.com/model-publishers/distilbert) DistilGPT2 (short for Distilled-GPT2) is an English-language model pre-trained with the supervision of the smallest version of Generative Pre-trained Transformer 2 (GPT-2). Like GPT-2, DistilGPT2 can be used to generate text. Users of this model card should also consider information about the design, training, and limitations of GPT-2. CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes. As the developers of GPT-2 (OpenAI) note in their model card, “language models like GPT-2 reflect the biases inherent to the systems they were trained on.” Significant research has explored bias and… Open weights apache-2.0 88M parameters transformers [View model](https://savrn.com/models/distilgpt2) ## EleutherAI [All models and datasets](https://savrn.com/model-publishers/eleutherai) ## Versions - [e93a9faa9c77](https://savrn.com/models/pythia-70m-deduped/versions/e93a9faa9c77) · current 2026-10-22 ## Explore More - [All text generation models](https://savrn.com/models/tasks/text-generation) - [All models under apache-2.0](https://savrn.com/models/licenses/apache-2-0) - [Model comparisons](https://savrn.com/models/comparisons) - [The model directory](https://savrn.com/models) - [Open model prices by host](https://savrn.com/ai-index/pricing/open-models) ## Source - Repository metadata, read 2026-10-19. - [EleutherAI website](https://www.eleuther.ai/) - [Hugging Face record](https://huggingface.co/EleutherAI/pythia-70m-deduped) - [How the hub is built](https://savrn.com/model-hub/methodology)