One dense giant. Sixty-four specialist minds. Morpho-180B takes the
battle-tested open dense foundation and grows it into a routed collective:
every token is served by the two domain experts that know it best, plus a
shared generalist that never sleeps. Train each expert to convergence. Leave
none behind.
Why Morpho
- 180B total / ~78B active per request — giant knowledge, agile inference.
- 64 domain experts, zero overlap — every training example maps to exactly one expert. Auditable by construction.
- Fully auditable data pipeline — raw → tagged → tokenized corpora, code included, in
Morpho-72B-MoE-Data.
- Resume-first engineering — every expert checkpoint (weights + loss + step) is versioned in
Morpho-72B-MoE-Checkpoints. Any box can pick up exactly where the last one died.
- 128K context via RoPE YaRN for long documents, codebases, and books.
Architecture
| Component |
Spec |
| Base |
72B dense backbone, 4-bit NF4 QLoRA (frozen) |
| Layers |
80 (40 dense even + 40 MoE odd) |
| Hidden |
8192 · 64 Q heads / 8 KV heads (GQA) · SwiGLU · RMSNorm |
| Router |
Learned, top-2 of 8 per MoE layer + 1 shared expert |
| Adapters |
LoRA rank-128 / alpha-256 on q,k,v,o,gate,up,down (~1B trainable params) |
| Context |
128K via RoPE YaRN |
| Total / active |
~180B / ~78B per request |
The 64 experts
Eight sectors — Finance, Law, Medicine, Tech, Science, Business, Arts, Humanities — each with 8 specialists. Full roster with per-expert data counts is logged at every training run; example counts range from ~10K to ~35K per expert, 200K+ examples total, each mapped to exactly one expert.
Data refinery — every byte auditable
Build pipeline
flowchart LR
A["1 · Base<br/>72B dense"] --> B["2 · Data<br/>823K routed"]
B --> C["3 · MoE structure<br/>64 routed experts"]
C --> D["4 · SFT<br/>62/64 converged"]
D --> E["5 · DPO<br/>blocked"]
E --> F["6 · Merge<br/>gated"]
F --> G["7 · Verify<br/>gated"]
G --> H["8 · GGUF<br/>gated"]
H --> I["9 · Distill<br/>gated"]
style D fill:#ff2d78,stroke:#00f0ff,color:#fff
Roadmap status — what's done, what's moving
| Stage |
Status |
Detail |
| 1 · Base (72B dense, 4-bit QLoRA) |
done |
frozen backbone, all experts train on it |
| 2 · Data (822,745 ex, v5 taxonomy, code+manifests) |
done |
auditable in Morpho-72B-MoE-Data |
| 3 · MoE structure (64 routed LoRA, top-2+shared, 128K YaRN) |
done |
design locked, see Architecture |
| 4 · SFT (per-expert to convergence) |
done —64/64 |
all experts converged to full patience |
| 5 · DPO (preference alignment) |
in progress —17/31 |
02/04/05/06/09–15/17/18/20–23 DONE on Morpho-72B-MoE-DPO; final 14 training on fourth 97GB GPU box with AUTO_SKIP; 33 thin experts keep SFT-only |
| 6 · Merge (unified MoE) |
ready |
scripts ready; 64/64 SFT complete; pending torch env; weights → this repo |
| 7 · Verify (benchmarks + audit) |
gated on merge |
no scores published before merge — policy |
| 8 · GGUF (quantized release) |
gated |
quants → this repo for download |
| 9 · Distill (compact students) |
gated |
students → this repo for download |
Right now: SFT 64/64 complete. DPO 17/31 — final 14 training on the GPU box, then merge → verify → GGUF → distill.
Convergence gallery — every expert trains to its floor
