ELF-M, a continuous diffusion language model, trained on the embeddings of T5Gemma-2-270M-OWTdistilled, distilled from T5Gemma-2. From the paper Scaling and Distilling Text Embeddings for Better Diffusibility. ELF (Embedded Language Flow) is a continuous diffusion language model that generates a sequence of text embeddings and decodes them to tokens. This model is the unchanged ELF-M trained on OpenWebText-1024 (sequences of 1024 tokens) for 5 epochs at a batch of 512, on the frozen embeddings of the distilled student T5Gemma-2-270M-OWTdistilled. Sampling uses the SDE sampler with self-conditioning guidance; --sc sets the guidance scale and --nfe the number of steps. With the code of the…
Open weights
gemma
pytorch
ELF-B, a continuous diffusion language model, trained on the embeddings of T5Gemma-2-270M-OWTdistilled, distilled from T5Gemma-2. From the paper Scaling and Distilling Text Embeddings for Better Diffusibility. ELF (Embedded Language Flow) is a continuous diffusion language model that generates a sequence of text embeddings and decodes them to tokens. This model is the unchanged ELF-B trained on OpenWebText-1024 (sequences of 1024 tokens) for 5 epochs at a batch of 512, on the frozen embeddings of the distilled student T5Gemma-2-270M-OWTdistilled. Sampling uses the SDE sampler with self-conditioning guidance; --sc sets the guidance scale and --nfe the number of steps. With the code of the…
Open weights
gemma
pytorch
ELF-M, a continuous diffusion language model, trained on the embeddings of T5Gemma-2-270M. From the paper Scaling and Distilling Text Embeddings for Better Diffusibility. ELF (Embedded Language Flow) is a continuous diffusion language model that generates a sequence of text embeddings and decodes them to tokens. This model is the unchanged ELF-M trained on OpenWebText-1024 (sequences of 1024 tokens) for 5 epochs at a batch of 512, on the frozen embeddings of the T5Gemma-2 encoder. Sampling uses the SDE sampler with self-conditioning guidance; --sc sets the guidance scale and --nfe the number of steps. With the code of the paper; the checkpoint and the tokenizer are downloaded from this…
Open weights
gemma
pytorch
A text encoder for continuous diffusion language models (DLMs), distilled from the encoder of T5Gemma-2-270M. It gives the SoTA latent embeddings for continuous DLMs. From the paper Scaling and Distilling Text Embeddings for Better Diffusibility. A continuous diffusion language model generates text embeddings and decodes them to words. T5Gemma-2 provides a strong embedding space for this, but it keeps the words that could fill the same position far apart, and a generated embedding often lands between them and decodes problematically. The student is trained by minimizing the KL divergence between the probabilities that the same frozen T5Gemma-2 decoder predicts from the student's embeddings…
Open weights
gemma
pytorch
ELF-B, a continuous diffusion language model, trained on the embeddings of T5Gemma-2-270M. From the paper Scaling and Distilling Text Embeddings for Better Diffusibility. ELF (Embedded Language Flow) is a continuous diffusion language model that generates a sequence of text embeddings and decodes them to tokens. This model is the unchanged ELF-B trained on OpenWebText-1024 (sequences of 1024 tokens) for 5 epochs at a batch of 512, on the frozen embeddings of the T5Gemma-2 encoder. Sampling uses the SDE sampler with self-conditioning guidance; --sc sets the guidance scale and --nfe the number of steps. With the code of the paper; the checkpoint and the tokenizer are downloaded from this…
Open weights
gemma
pytorch
