The Math-Aware Retrieval benchmark from MathNet
(paper): given an Olympiad problem, retrieve a
mathematically equivalent reformulation of it from a large corpus of hard
negatives. Each equivalent is provided at three difficulty tiers — easy,
medium, hard — that increasingly disguise the surface form while preserving the
underlying problem, so retrieval gets sharply harder even though the answer is the same.
- 15,000 queries, one gold-equivalent target per tier.
- 3 tiers (
easy, medium, hard), over a shared 117,088-doc corpus (equivalents at all
tiers + LLM near-miss hard negatives + distractors).
- Standard BEIR / MTEB layout.
Quick start
from datasets import load_dataset
tier = "hard" # one of "easy", "medium", "hard"
corpus = load_dataset("ShadenA/MathNet-Retrieve", f"{tier}-corpus", split="test") # {_id, text}
queries = load_dataset("ShadenA/MathNet-Retrieve", f"{tier}-queries", split="test") # {_id, text}
qrels = load_dataset("ShadenA/MathNet-Retrieve", f"{tier}-qrels", split="test") # {query-id, corpus-id, score}
All three configs share the same corpus and queries; they differ only in which
reformulation is the gold positive. Each query has exactly one positive, and the
query's own equivalents at the other tiers act as negatives.
Tiers
| Tier |
Queries |
Corpus |
What changes |
| easy |
15,000 |
117,088 |
light paraphrase (renamed vars, reordered clauses) |
| medium |
15,000 |
117,088 |
moderate restructuring (notation, framing) |
| hard |
15,000 |
117,088 |
heavy disguise; minimal lexical overlap |
Results
Recall@k (%) on the shared corpus; bold = best.
| Model |
Easy R@1 |
Easy R@5 |
Medium R@1 |
Medium R@5 |
Hard R@1 |
Hard R@5 |
gemini-embedding-001 |
11.4 |
90.7 |
2.5 |
77.1 |
0.0 |
9.3 |
qwen3-embedding-4B |
14.8 |
83.5 |
2.9 |
71.6 |
0.0 |
8.4 |
text-embedding-3-large |
6.8 |
78.1 |
1.6 |
68.3 |
0.0 |
2.6 |
all-mpnet-base-v2 |
6.8 |
82.5 |
1.3 |
51.0 |
0.0 |
1.2 |
multi-qa-mpnet-base-dot-v1 |
5.1 |
80.6 |
1.1 |
47.3 |
0.0 |
0.8 |
text-embedding-ada-002 |
3.3 |
66.1 |
1.1 |
55.4 |
0.0 |
0.7 |
text-embedding-3-small |
3.6 |
59.8 |
1.4 |
51.5 |
0.0 |
0.5 |
Every model degrades sharply from easy to hard: even the strongest solve easy but
collapse on hard (Recall@1 ≈ 0; hard Recall@10 tops out near 54%). Retrieving
equivalent problems remains unreliable once the surface form is disguised.
Caveats
- Not contamination-clean — source problems may appear in pretraining data; prefer relative comparisons.
- Equivalents and hard negatives are LLM-generated and machine-verified; a small fraction may be imperfect.
License & citation
Released under CC BY 4.0, consistent with ShadenA/MathNet.
@inproceedings{alshammari2026mathnet,
title = {MathNet: A Global Multimodal Benchmark for Mathematical Reasoning and Retrieval},
author = {Alshammari, Shaden and Wen, Kevin and Zainal, Abrar and Hamilton, Mark and
Safaei, Navid and Albarakati, Sultan and Freeman, William T. and Torralba, Antonio},
booktitle = {International Conference on Learning Representations},
year = {2026},
url = {https://mathnet.mit.edu}
}
Project site · Paper · Main dataset