ProgramBench-Tests

ProgramBench Generated Tests

This dataset contains the AI-generated behavioral test suites used to evaluate model solutions in ProgramBench.

ProgramBench is a benchmark that evaluates whether language models can rebuild programs from scratch. Given only a compiled binary and its documentation, AI agents must architect and implement a complete codebase that reproduces the original program's behavior. These test suites are used to assess whether a candidate solution is behaviorally equivalent to the original program.

• Website
• Paper
• GitHub
• Leaderboard

Dataset Overview

The dataset covers 200 task instances spanning open-source repositories in 6 programming languages:

Task difficulties: 27 easy, 120 medium, 18 hard, 35 unrated.

Dataset Structure

Each task instance is a folder named <owner>__<repo>.<short_commit> containing:

<task_name>/
├── tests/
│   ├── <hash1>.tar.gz
│   ├── <hash2>.tar.gz
│   └── ...
├── ATTRIBUTION.md
└── LICENSE

• tests/ — Compressed archives, each containing a pytest-based test suite generated by an AI model against the original program's compiled executable. Multiple archives may exist per task (from different generation runs).
• ATTRIBUTION.md — Attribution information including a link to the original repository at the pinned commit and the license identifier.
• LICENSE — The full license text reproduced from the original repository.

Test Generation

The test suites were generated by AI models with an agent scaffold. Each model was given access to: - The compiled executable of the original program - The original repository's source code and any existing tests (as reference)

The model was then instructed to generate behavioral tests in a pytest harness that exercise the executable's functionality. The generated tests may be inspired by or substantially similar to existing tests in the original repository.

Attribution and Licensing

Each task folder includes: - ATTRIBUTION.md — A link to the original repository at the pinned commit and the SPDX license identifier. - LICENSE — The full license text reproduced from the original repository.

The test files themselves were AI-generated. For repositories under copyleft licenses (GPL, LGPL, MPL), the attribution notes that original license terms apply to any portions that may constitute derivative works.

The original repositories span a variety of open-source licenses:

Contact

For questions, license concerns, or takedown requests, please open an issue on the GitHub repository or contact the maintainers at [email protected] and [email protected].

Citation

If you use this dataset in your research, please cite:

@preprint{yang2026programbench,
  title={ProgramBench: Can Language Models Rebuild Programs From Scratch?},
  author={John Yang and Kilian Lieret and Jeffrey Ma and Parth Thakkar and Dmitrii Pedchenko and Sten Sootla and Emily McMilin and Pengcheng Yin and Rui Hou and Gabriel Synnaeve and Diyi Yang and Ofir Press},
  year={2026},
}