2025801013-anlp-assignment2

Mixture-of-Experts FFN ablations — Advanced NLP Assignment 2

Ten decoder-only translation models: five feed-forward variants, each trained at two token budgets. Everything except the feed-forward block is identical across variants, so any measured difference is attributable to the FFN alone.

Trained on belumind/en-vi-ja-curated-500k-triplets for Vietnamese→English and Japanese→English translation.

Variants

A dense 2-layer MLP of hidden width h costs 2·d·h per layer. Variants 2–4 split that same budget across four experts (h/4 each), so total parameters match the dense baseline. Variant 5 instead sizes experts at h/2 so that active parameters match the baseline, giving twice the total.

Each exists at two budgets: <name> (30M tokens) and <name>-100m (100M tokens). The token budget is identical within each group — that equality is what makes the comparison valid.

Architecture

6 layers, d_model 384, 6 heads, d_ff 1536, context 256, RMSNorm pre-norm, RoPE, tied embeddings, 16k byte-level BPE shared across all three languages.

Results

Test perplexity and BLEU (sacrebleu, signature nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.6.0):

Limitations

These are coursework models and translation quality is low.

• At 100M tokens the models sit at ~6 tokens per parameter, well under the Chinchilla-optimal ~20. They are undertrained by design, to fit ten runs into the available compute.
• Roughly 24% of generations from the best model degenerate into repetition. The models translate short formulaic inputs reasonably and collapse on harder ones.
• BLEU in the 2–7 range is far below production systems (25–40). The numbers support relative comparison between variants, which is what the ablation needs, but should not be read as translation quality.

Usage

import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="samvaran/2025801013-anlp-assignment2",
    filename="part1/moe_active_matched-100m/model.pt",
)
state = torch.load(path, weights_only=False)
print(state["model_config"])   # architecture
print(state["tokens_seen"])    # training budget actually consumed

Each checkpoint stores model_state, model_config, train_config and tokens_seen. The model class and tokenizer live in the code repository; assets/tokenizer.json there is required to encode or decode.