mmlu-prox-lite-evals

MMLU-ProX-Lite Evaluation Outputs

Raw outputs from evaluating recent LLMs on li-lab/MMLU-ProX-Lite (revision e82aafb) with lm-evaluation-harness 0.4.13, in 10 languages (en, zh, ja, ko, pt, de, id, ru, ar, sw).

The purpose is within-model diagnosis: for each model, which languages (relative to its own English accuracy on the same parallel questions) and which subjects it is weak in. It is not a leaderboard, and models are not meant to be compared with each other: reasoning effort differs by model (the developer's default for most of them; see the GitHub README).

Status: complete. All 10 models are uploaded.

Code, protocol, model settings and summary tables: github.com/3ae3ae/mmlu-prox-lite-evals.

Files

One folder per model, in the layout lm-evaluation-harness writes:

• results_*.json: accuracy per language and subject, plus the full run configuration (model arguments, generation settings, harness version).
• samples_mmlu_prox_lite_<language>_<subject>_*.jsonl: one row per question with the prompt, the raw response, the extracted answer (filtered_resps; [invalid] when no answer could be extracted), and the score (exact_match).

Each model is also available as a dataset config (split test) in the viewer.

Token usage

token-usage/token_usage.csv (config token-usage) has one row per model, language and question: prompt, output (completion_tokens) and reasoning tokens, finish reason, and the provider that served the answer. Join it to the samples on model (the folder name), language and question_id_src (doc.question_id_src in the samples).

lm-evaluation-harness did not log these. They were recovered afterwards from the providers' own request records (Neuralwatt's usage API, LLM Gateway's request logs), which carry no text and were tied to questions by request order and prompt token count. Keep in mind:

• reasoning_tokens_source: reported by the API, or estimated for the Neuralwatt models as output − answer − a fixed overhead (within ±1 token on checked requests).
• match_method: exact prompt tokens (Neuralwatt and GPT models, checked against requests of known identity) or anchored order (Claude and Gemini: about 9 in 10 rows exact, the rest swapped with a neighbouring question of the same language). Per-language totals hold; single rows of the anchored models less so.
• Token columns are empty where no record was matched or the record had no counts, e.g. the 68 blocked Swahili math questions of the GPT models.
• Counts use each model's own tokenizer: compare languages within a model, not counts across models.

Method, checks and summary tables: the GitHub README, section "Token usage".

Notes

• Rows include questions from MMLU-ProX (Xuan et al., 2025), redistributed under the MIT License stated on its dataset card. MMLU-ProX is built on MMLU-Pro (MIT).
• Model responses were generated by the models named in each folder and are shared for evaluation analysis.
• Responses contain only the final message content. The models' reasoning, which some APIs return in a separate field, was not logged by lm-evaluation-harness and is not included; its token count is in token-usage.
• In gpt-6-luna and gpt-6.1-sol, all 68 Swahili math responses are empty: Azure's content filter blocked those prompts. They are scored as wrong.

Limitations

• Each model ran once, at temperature 1.0. Scores are single samples, and reasoning budgets differ across models.
• Subjects are unbalanced (per language: 19 history … 68 math questions), so per-subject scores in one language are noisy.
• In the sample files, a response that hit the output-token limit cannot be told apart from a wrong answer; token-usage has its finish reason (length).
• Some open-weight models are served quantized (see the GitHub README); their results describe the served checkpoint.

Citation

If you use these outputs, please cite this dataset and MMLU-ProX:

@misc{3ae3ae2026mmluproxlite,
  author       = {3ae3ae},
  title        = {{MMLU-ProX-Lite}: Where Do Recent Lightweight {LLMs} Break Down?},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/3ae3ae3ae/mmlu-prox-lite-evals}},
  note         = {Code and analysis: \url{https://github.com/3ae3ae/mmlu-prox-lite-evals}}
}

@article{xuan2025mmluprox,
  title={Mmlu-prox: A multilingual benchmark for advanced large language model evaluation},
  author={Weihao Xuan and Rui Yang and Heli Qi and Qingcheng Zeng and Yunze Xiao and Aosong Feng and Dairui Liu and Yun Xing and Junjue Wang and Fan Gao and Jinghui Lu and Yuang Jiang and Huitao Li and Xin Li and Kunyu Yu and Ruihai Dong and Shangding Gu and Yuekang Li and Xiaofei Xie and Felix Juefei-Xu and Foutse Khomh and Osamu Yoshie and Qingyu Chen and Douglas Teodoro and Nan Liu and Randy Goebel and Lei Ma and Edison Marrese-Taylor and Shijian Lu and Yusuke Iwasawa and Yutaka Matsuo and Irene Li},
  journal={arXiv preprint arXiv:2503.10497},
  year={2025}
}