This repository hosts intermediate training checkpoints (step 170,000) for two ~1B-active-parameter masked diffusion language models (MDMs) built on an OLMoE-style Mixture-of-Experts transformer backbone. The two checkpoints share the same architecture and data but differ in the schedule used to sample the diffusion timestep t during training. Code: https://github.com/chen-hao-chao/olmoe-mdm (fork of AI2's OLMo) Monte-Carlo ELBO objective rather than next-token autoregression. The model has bidirectional attention (bidirectional: true) and is trained to denoise randomly masked token subsets. position embeddings, RMSNorm, and SwiGLU activations. - dmodel: 2240 - nlayers: 18 - nheads: 35…