gpt2-traning

GPT-2 trained on C4 — Lab 4, AY

This is the lab4-yoru-AY-482206 checkpoint, trained from scratch on English C4. It was selected by the lowest development loss among nine experimental recipes. The independent seed repeat and reserved audit evaluation were still pending when this checkpoint was published. Reported scores are local evaluation proxies, not an official online-judge result.

Model and training

• Stock Hugging Face GPT2LMHeadModel: 12 layers, 12 attention heads, hidden width 768, context length 1,024, vocabulary 50,257, tied input/output embeddings.
• 124,439,808 unique parameters, commonly described as GPT-2 small. The lab uses the historical 117M model-family label.
• GPT-2 tokenizer; documents packed with EOS separators. No pretrained model weights were used.
• Two NVIDIA H200 GPUs, BF16 training, FP32 saved parameters, 30-minute allocation.
• 1,631 optimizer updates; global batch 524,288 input tokens; 855,113,728 input tokens processed.
• Muon for hidden matrices: learning rate 0.0012, momentum 0.95, Nesterov, five Newton–Schulz steps, AdamW RMS matching.
• Auxiliary AdamW: learning rate 0.0006, betas (0.9, 0.95).
• Weight decay 0.1, no decay on normalization/bias parameters, gradient clipping 1.0, dropout 0, random seed 42.
• Five percent warmup followed by cosine decay to zero; torch.compile with the default mode. Training used PyTorch 2.14.0 and Transformers 5.17.0.
• C4 revision: 1588ec454efa1a09f29cd18ddd04fe05fc8653a2.

Evaluation

English C4 validation documents were shuffled with seed 42, packed into 1,024-token blocks, and scored using 1,023 shifted prediction targets per block.

Development packing retains complete groups of 64 blocks for distributed evaluation. These splits overlap and are not independent tests. The lab's official evaluation uses 100,000 validation documents and may differ in packing and scoring. The larger reserved audit split is not included in these results.

Load the checkpoint

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "yorukot/gpt2-traning"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

Inference and evaluation use stock Transformers; the training optimizer is not required. This is a base language model for an educational experiment, without instruction tuning. Its web training data can produce inaccurate or biased text.

Training run on W&B.