The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on.
Dataset Card
By Google Research Datasets, published under cc-by-4.0, revision 4bb6404fdc6c.
Dataset Card for Mostly Basic Python Problems (mbpp)
Table of Contents
- Dataset Card for Mostly Basic Python Problems (mbpp)
- Table of Contents
- Dataset Description
- Dataset Summary
- Supported Tasks and Leaderboards
- Languages
- Dataset Structure
- Data Instances
- Data Fields
- Data Splits
- Dataset Creation
- Curation Rationale
- Source Data
- Initial Data Collection and Normalization
- Who are the source language producers?
- Annotations
- Annotation process
- Who are the annotators?
- Personal and Sensitive Information
- Considerations for Using the Data
- Social Impact of Dataset
- Discussion of Biases
- Other Known Limitations
- Additional Information
- Dataset Curators
- Licensing Information
- Citation Information
- Contributions
Dataset Description
- Repository: https://github.com/google-research/google-research/tree/master/mbpp
- Paper: Program Synthesis with Large Language Models
Dataset Summary
The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. As described in the paper, a subset of the data has been hand-verified by us.
Structure
full 974 rows
| Split | Rows | Size |
|---|---|---|
| train | 374 | 176.9 KB |
| test | 500 | 244.2 KB |
| validation | 90 | 42.4 KB |
| prompt | 10 | 4.6 KB |
sanitized 427 rows
| Split | Rows | Size |
|---|---|---|
| train | 120 | 63.5 KB |
| test | 257 | 132.8 KB |
| validation | 43 | 20.1 KB |
| prompt | 7 | 3.4 KB |
Details
- Repository
- google-research-datasets/mbpp
- Publisher
- Google Research Datasets
- Task category
- Text to text
- Tags
- code-generation
- Size category
- n<1K
- Languages
- en
- Revision
- 4bb6404fdc6cacfda99d4ac4205087b89d32030c
- Last updated
- 2024-01-04
SAVRN's Notes on mbpp
MBPP is 1,401 crowd-sourced entry-level Python problems in 361.7 KB, 10 files, 498.6k monthly downloads, cc-by-4.0, from Google Research under the google-research-datasets account, task category text-to-text, tagged code-generation. Two configs: full at 974 rows (374 train, 500 test, 90 validation) and sanitized at 427 rows (120, 257, 43). The sanitized split exists because the original had noise; prefer it for reporting.
This is an eval for code models, and a contaminated one, since every code LLM has been benchmarked against it for years. Do not train on it unless you enjoy invalidating your own numbers, and at 1,401 rows it is useless as training data anyway. If you need basic Python problems, generate synthetically and keep MBPP as a held-out check.
Files
10 files, 361.7 KB in total.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| full/prompt-00000-of-00001.parquet | Data | 7.9 KB | a053e4bb85ce |
| full/test-00000-of-00001.parquet | Data | 115.8 KB | 566fd53060ff |
| full/train-00000-of-00001.parquet | Data | 87.2 KB | 09d125ca31ed |
| full/validation-00000-of-00001.parquet | Data | 25.1 KB | 3f0ec0609874 |
| sanitized/prompt-00000-of-00001.parquet | Data | 6.7 KB | 73c623309b7b |
| sanitized/test-00000-of-00001.parquet | Data | 60.9 KB | e9e9efa2c0d5 |
| sanitized/train-00000-of-00001.parquet | Data | 33.9 KB | d95f8ad6d2ff |
| sanitized/validation-00000-of-00001.parquet | Data | 14.0 KB | 27e065fcab3c |
| README.md | Documentation | 9.1 KB | — |
| .gitattributes | Repository | 1.2 KB | — |
License and Download
- License
- cc-by-4.0
- Access
- No access gate
Released by Google Research Datasets through its official repository on Hugging Face. Read the license.