T5Gemma-2-270M-OWTdistilled is an open-weight model from Zekai Zhang, released under Gemma Terms of Use. Its published files total 905.2 MB.
A text encoder for continuous diffusion language models (DLMs), distilled from the encoder of T5Gemma-2-270M. It gives the SoTA latent embeddings for continuous DLMs. From the paper Scaling and Distilling Text Embeddings for Better Diffusibility.
Model Card
A text encoder for continuous diffusion language models (DLMs), distilled from the encoder of T5Gemma-2-270M. It gives the SoTA latent embeddings for continuous DLMs. From the paper Scaling and Distilling Text Embeddings for Better Diffusibility. A continuous diffusion language model generates text embeddings and decodes them to words. T5Gemma-2 provides a strong embedding space for this, but it keeps the words that could fill the same position far apart, and a generated embedding often lands between them and decodes problematically. The student is trained by minimizing the KL divergence between the probabilities that the same frozen T5Gemma-2 decoder predicts from the student's embeddings…
Excerpt from the card by Zekai Zhang, licensed gemma.
Identity and Version
- Repository
- la0ka1/T5Gemma-2-270M-OWTdistilled
- Publisher
- Zekai Zhang
- Task
- Not stated by the source
- Modality
- Other
- Library
- pytorch
- Parameters
- Not stated by the source
- Languages
- en
- Revision
- 0feafcffd5f31646080d8193d143456ec78fcb26
- First published
- 2026-09-30
- Last updated
- 2026-10-01
Files and Weights
8 files, 905.2 MB in total. The weights are 1 file totalling 871.8 MB in pt.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| T5Gemma-2-270M-OWTdistilled.pt | Weights | 871.8 MB | 255098992298 |
| special_tokens_map.json | Configuration | 662 B | — |
| NOTICE | Documentation | 97 B | — |
| README.md | Documentation | 4.1 KB | — |
| SHA256SUMS | Other | 97 B | — |
| .gitattributes | Repository | 1.6 KB | — |
| tokenizer.json | Tokenizer | 33.4 MB | 4c76318660cc |
| tokenizer_config.json | Tokenizer | 769 B | — |
License and Download
- License
- gemma
- Access
- Open weights, no gate
- Download size
- 871.8 MB
Released by Zekai Zhang through its official repository on Hugging Face.
Built From
- Derived from google/t5gemma-2-270m-270m
- Trained on (disclosed) Skylion007/openwebtext
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 871.8 MB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About T5Gemma-2-270M-OWTdistilled
Can I use T5Gemma-2-270M-OWTdistilled commercially?
Yes, with conditions. T5Gemma-2-270M-OWTdistilled is released under Gemma Terms of Use. Gemma models are released under Google's Gemma Terms of Use, which permit commercial use and redistribution subject to the Gemma Prohibited Use Policy, whose restrictions must be passed on to anyone the model is distributed to.