A long-context retrieval and question-answering benchmark with 1,143,371 documents and 10,000 questions, including 50 questions about 0G.
Dataset Card
A long-context retrieval and question-answering benchmark with 1,143,371 documents and 10,000 questions, including 50 questions about 0G. It extends the ms100M bank from MSA-RAG-BENCHMARKS with 20 million additional text tokens and 655 new questions grounded in the added documents. 120M is the benchmark's nominal size. The actual corpus contains 125,708,601 raw tokens under the MSA-4B tokenizer: 105,708,601 original tokens plus exactly 20,000,000 added tokens. Counts exclude prompts, wrappers, and special tokens. This is a derived benchmark, not an official Microsoft MS MARCO release. The original documents occupy positions 0:961686; the original questions occupy positions 0:9345. Their…
Excerpt from the card by 0G.AI, licensed other.
Structure
questions 10,000 rows
| Split | Rows | Size |
|---|---|---|
| test | 10,000 | 6.6 MB |
documents 1,143,371 rows
| Split | Rows | Size |
|---|---|---|
| corpus | 1,143,371 | 668.3 MB |
Details
- Repository
- 0G-AI/MS-MARCO-0G-120M
- Publisher
- 0G.AI
- Task category
- Question answering
- Tags
- retrieval-augmented-generation, long-context, msmarco
- Size category
- 1M<n<10M
- Languages
- en
- Revision
- a8030a0b6544692decaeb9c30b89c246d5b1cdbe
- Last updated
- 2026-09-18
Files
19 files, 1.7 GB in total.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| documents.jsonl | Data | 724.5 MB | 21a0c05748a2 |
| documents.parquet | Data | 311.5 MB | 565c4d80b505 |
| manifest.json | Data | 3.3 KB | — |
| metadata/documents.jsonl | Data | 107.3 MB | 393a1d129c6e |
| metadata/questions.jsonl | Data | 193.4 KB | — |
| questions.jsonl | Data | 7.3 MB | — |
| questions.parquet | Data | 2.9 MB | b9e6c8e57ee7 |
| sidecars/new_evidence_sets.jsonl | Data | 371.8 KB | — |
| sidecars/qrels.jsonl | Data | 1.6 MB | — |
| sidecars/reference_positions.jsonl | Data | 598.2 KB | — |
| validation.json | Data | 1.0 KB | — |
| README.md | Documentation | 12.4 KB | — |
| TERMS.md | Documentation | 2.0 KB | — |
| checksums.sha256 | Other | 1.5 KB | — |
| data/ms_120M/mdata_msmarco_120M.pkl | Other | 547.7 MB | 7387df420ff9 |
| data/ms_120M/qdata_msmarco_120M.pkl | Other | 6.4 MB | 420d4ca1473b |
| verify.py | Other | 10.6 KB | — |
| .gitattributes | Repository | 2.6 KB | — |
| metadata/tokenizer.json | Tokenizer | 1.1 KB | — |
License and Download
- License
- other
- Access
- No access gate
Released by 0G.AI through its official repository on Hugging Face. Read the license.