The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on.
Dataset Card
By Google Research Datasets, published under cc-by-4.0, revision 4bb6404fdc6c.
Dataset Card for Mostly Basic Python Problems (mbpp)
Table of Contents
- Dataset Card for Mostly Basic Python Problems (mbpp)
- Table of Contents
- Dataset Description
- Dataset Summary
- Supported Tasks and Leaderboards
- Languages
- Dataset Structure
- Data Instances
- Data Fields
- Data Splits
- Dataset Creation
- Curation Rationale
- Source Data
- Initial Data Collection and Normalization
- Who are the source language producers?
- Annotations
- Annotation process
- Who are the annotators?
- Personal and Sensitive Information
- Considerations for Using the Data
- Social Impact of Dataset
- Discussion of Biases
- Other Known Limitations
- Additional Information
- Dataset Curators
- Licensing Information
- Citation Information
- Contributions
Dataset Description
- Repository: https://github.com/google-research/google-research/tree/master/mbpp
- Paper: Program Synthesis with Large Language Models
Dataset Summary
The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. As described in the paper, a subset of the data has been hand-verified by us.
Structure
full 974 rows
| Split | Rows | Size |
|---|---|---|
| train | 374 | 176.9 KB |
| test | 500 | 244.2 KB |
| validation | 90 | 42.4 KB |
| prompt | 10 | 4.6 KB |
sanitized 427 rows
| Split | Rows | Size |
|---|---|---|
| train | 120 | 63.5 KB |
| test | 257 | 132.8 KB |
| validation | 43 | 20.1 KB |
| prompt | 7 | 3.4 KB |
Details
- Repository
- google-research-datasets/mbpp
- Publisher
- Google Research Datasets
- Task category
- Text to text
- Tags
- code-generation
- Size category
- n<1K
- Languages
- en
- Revision
- 4bb6404fdc6cacfda99d4ac4205087b89d32030c
- Last updated
- 2024-01-04
Files
10 files, 361.7 KB in total.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| full/prompt-00000-of-00001.parquet | Data | 7.9 KB | a053e4bb85ce |
| full/test-00000-of-00001.parquet | Data | 115.8 KB | 566fd53060ff |
| full/train-00000-of-00001.parquet | Data | 87.2 KB | 09d125ca31ed |
| full/validation-00000-of-00001.parquet | Data | 25.1 KB | 3f0ec0609874 |
| sanitized/prompt-00000-of-00001.parquet | Data | 6.7 KB | 73c623309b7b |
| sanitized/test-00000-of-00001.parquet | Data | 60.9 KB | e9e9efa2c0d5 |
| sanitized/train-00000-of-00001.parquet | Data | 33.9 KB | d95f8ad6d2ff |
| sanitized/validation-00000-of-00001.parquet | Data | 14.0 KB | 27e065fcab3c |
| README.md | Documentation | 9.1 KB | — |
| .gitattributes | Repository | 1.2 KB | — |
License and Download
- License
- cc-by-4.0
- Access
- No access gate
Released by Google Research Datasets through its official repository on Hugging Face. Read the license.