The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. As described in the paper, a subset of the data has been hand-verified by us. Released here as part of Program Synthesis with Large Language Models, Austin et. al., 2021. This dataset is used to evaluate code generations. English - Python code - sourcefile: unknown - text/prompt: description of programming task - code: solution for programming task - testsetupcode/testimports: necessary code…
SAVRN Model Hub · Datasets by Task
Text to Text Datasets
2 open-weight text to text datasets in the SAVRN Model Hub, with OpenAI and Google Research Datasets publishing the most.
2Datasets
2Publishers
2Licenses
Most Downloaded
| Dataset | Publisher | License | Monthly downloads |
|---|---|---|---|
| mbpp | Google Research Datasets | cc-by-4.0 | 498.6k |
| openai_humaneval | OpenAI | mit | 273k |
Licenses
| License | Datasets | Commercial use |
|---|---|---|
| mit | 1 | Yes |
| cc-by-4.0 | 1 | Yes |
Who Publishes Them
| Publisher | Datasets |
|---|---|
| OpenAI | 1 |
| Google Research Datasets | 1 |
All 2 Datasets
The HumanEval dataset released by OpenAI includes 164 programming problems with a function sig- nature, docstring, body, and several unit tests. They were handwritten to ensure not to be included in the training set of code generation models. The programming problems are written in Python and contain English natural text in comments and docstrings. An example of a dataset instance: - taskid: identifier for the data sample - prompt: input for the model containing function header and docstrings - canonicalsolution: solution for the problem in the prompt - test: contains function to test generated code for correctness - entrypoint: entry point for test The dataset only consists of a test split…