SWE-bench Lite is subset of SWE-bench, a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 300 test Issue-Pull Request pairs from 11 popular Python.
Dataset Card
SWE-bench Lite is subset of SWE-bench, a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 300 test Issue-Pull Request pairs from 11 popular Python. Evaluation is performed by unit test verification using post-PR behavior as the reference solution. The dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues? This dataset only contains the problemstatement (i.e. issue text) and the basecommit which can represents the state of the codebase before the issue has been resolved. If you want to run inference using the "Oracle" or BM25 retrieval settings mentioned in the paper, consider the following datasets.…
Excerpt from the card by Princeton NLP group.
Structure
default 323 rows
| Split | Rows | Size |
|---|---|---|
| dev | 23 | 232.2 KB |
| test | 300 | 3.5 MB |
Details
- Repository
- princeton-nlp/SWE-bench_Lite
- Publisher
- Princeton NLP group
- Task category
- Not stated by the source
- Tags
- Not stated by the source
- Size category
- Not stated by the source
- Languages
- Not stated by the source
- Revision
- 6ec7bb89b9342f664a54a6e0a6ea6501d3437cc2
- Last updated
- 2025-03-03
Files
4 files, 1.2 MB in total.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| data/dev-00000-of-00001.parquet | Data | 119.6 KB | 8312f3218380 |
| data/test-00000-of-00001.parquet | Data | 1.1 MB | 7a21f37b8bc1 |
| README.md | Documentation | 3.7 KB | — |
| .gitattributes | Repository | 2.3 KB | — |
License and Download
- License
- Not stated by the source
- Access
- No access gate
Released by Princeton NLP group through its official repository on Hugging Face.