MMLU-Pro

SAVRN's Notes on MMLU-Pro

MMLU-Pro is 12,102 questions in 4.2 MB under MIT terms, but the split layout tells you what it is: 12,032 rows of test and only 70 rows of validation. This is an evaluation set, published by TIGER-Lab as a harder, more option-dense successor to MMLU, and the near-empty validation split leaves almost no room for prompt development or calibration without touching the test data itself.

MIT licensing removes friction for commercial eval pipelines, and at 4.2 MB it costs nothing to mirror internally. The discipline that matters is contamination: this is a headline benchmark with a fixed public test split, so any model that has seen it in training will overstate capability, and reviewers increasingly check for exactly that. The page was last updated May 2026 — verify the revision when comparing scores against published results.

SAVRN Research