ermine-megatron is an open-weight model from Chen-Hao Chao, released under MIT License. Its published files total 35.1 GB.
Megatron-LM checkpoints for Masked Diffusion Models (MDM) with Mixture-of-Experts (MoE) at various scales. Each.tar file contains a torchdist Megatron-LM distributed checkpoint.
Model Card
By Chen-Hao Chao, published under mit, revision 02286fd868c6.
Megatron-LM checkpoints for Masked Diffusion Models (MDM) with Mixture-of-Experts (MoE) at various scales. Each.tar file contains a torchdist Megatron-LM distributed checkpoint.
Read Chen-Hao Chao's full model card
Ermine ? MDM MoE Scale Checkpoints
Megatron-LM checkpoints for Masked Diffusion Models (MDM) with Mixture-of-Experts (MoE) at various scales.
Model Variants
| Checkpoint | Layers | Hidden | Heads | Experts | Train Iters |
|---|---|---|---|---|---|
| mdm_moe_param_14M | 6 | 448 | 7 | 8 (top-2) | 75000 |
| mdm_moe_param_25M | 8 | 512 | 8 | 8 (top-2) | 40000 |
| mdm_moe_param_36M | 9 | 576 | 9 | 8 (top-2) | 45000 |
| mdm_moe_param_48M | 10 | 640 | 10 | 8 (top-2) | 60000 |
| mdm_moe_param_63M | 13 | 640 | 10 | 8 (top-2) | 85000 |
| mdm_moe_param_77M | 16 | 640 | 10 | 8 (top-2) | 125000 |
| mdm_moe_param_106M | 15 | 768 | 12 | 8 (top-2) | 90000 |
| mdm_moe_param_153M | 16 | 896 | 16 | 8 (top-2) | 105000 |
| mdm_moe_param_199M | 16 | 1024 | 16 | 8 (top-2) | 80000 |
| mdm_moe_param_249M | 20 | 1024 | 16 | 8 (top-2) | 65000 |
Architecture
- Sequence length: 2048
- Tokenizer: GPT-2 BPE
- Positional embedding: RoPE
- Normalization: RMSNorm
- Activation: SwiGLU
- MoE dispatcher: all-to-all
Checkpoint Format
Each .tar file contains a torch_dist Megatron-LM distributed checkpoint.
Identity and Version
- Repository
- chen-hao-chao/ermine-megatron
- Publisher
- Chen-Hao Chao
- Task
- Not stated by the source
- Modality
- Other
- Library
- Not stated by the source
- Parameters
- Not stated by the source
- Languages
- Not stated by the source
- Revision
- 02286fd868c68465b7625f67693c93d43ff2b107
- First published
- 2026-10-05
- Last updated
- 2026-10-05
Files and Weights
12 files, 35.1 GB in total.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| README.md | Documentation | 1.2 KB | — |
| mdm_moe_param_106M_model_only_ermine.tar | Other | 3.2 GB | d8c28ee77ae4 |
| mdm_moe_param_14M_model_only_ermine.tar | Other | 875.3 MB | 13b2528884ec |
| mdm_moe_param_153M_model_only_ermine.tar | Other | 6.6 GB | 373efc8ac64e |
| mdm_moe_param_199M_model_only_ermine.tar | Other | 5.6 GB | 84081e989ab6 |
| mdm_moe_param_249M_model_only_ermine.tar | Other | 8.5 GB | 85996cfec40b |
| mdm_moe_param_25M_model_only_ermine.tar | Other | 1.0 GB | ca1605c688d3 |
| mdm_moe_param_36M_model_only_ermine.tar | Other | 1.3 GB | 2223ddd76ba3 |
| mdm_moe_param_48M_model_only_ermine.tar | Other | 2.5 GB | 3650ccb49f7e |
| mdm_moe_param_63M_model_only_ermine.tar | Other | 2.5 GB | 0e9b34ea3610 |
| mdm_moe_param_77M_model_only_ermine.tar | Other | 3.0 GB | dc58dc74450a |
| .gitattributes | Repository | 1.5 KB | — |
License and Download
- License
- mit
- Access
- Open weights, no gate
Released by Chen-Hao Chao through its official repository on Hugging Face. Read the license.
Questions About ermine-megatron
Can I use ermine-megatron commercially?
Yes. ermine-megatron is released under MIT License. The MIT License is a short permissive license. It permits commercial use, modification and redistribution, provided the copyright notice and permission notice are included.