# SWE-bench_Verified by Princeton NLP group: Dataset
Source: https://savrn.com/datasets/swe-bench-verified
Markdown alternate of the page above; the site index is https://savrn.com/llms.txt

---

## Dataset Card

SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality. SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. See this post for more details on the human-validation process. The dataset collects 500 test Issue-Pull Request pairs from popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution. The original SWE-bench dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Want to run inference now? This dataset only contains the problemstatement (i.e. issue text) and the basecommit which…

Excerpt from the card by Princeton NLP group.

## Structure

### default 500 rows

| Split | Rows | Size |
| --- | --- | --- |
| test | 500 | 7.8 MB |

repostringinstance_idstringbase_commitstringpatchstringtest_patchstringproblem_statementstringhints_textstringcreated_atstringversionstringFAIL_TO_PASSstringPASS_TO_PASSstringenvironment_setup_commitstringdifficultystring

## Details

Repository

princeton-nlp/SWE-bench_Verified

Publisher

Princeton NLP group

Task category

Not stated by the source

Tags

Not stated by the source

Size category

Not stated by the source

Languages

Not stated by the source

Revision

c104f840cc67f8b6eec6f759ebc8b2693d585d4a

Last updated

2025-02-18

## SAVRN's Notes on SWE-bench_Verified

SWE-bench Verified is the human-validated 500-sample subset of SWE-bench, the Princeton NLP benchmark where models resolve real GitHub issues and unit tests verify the fix. It is 2.1 MB — 500 rows, three files, 288.3k monthly downloads, last updated 2025-02-18 — and deliberately minimal: each record carries the problem statement and base commit, nothing more. Evaluation infrastructure lives outside the dataset.

This is an eval, full stop, and one the industry cites constantly. Training on it — or on its 500 issue texts — does not build capability, it just games the most-quoted coding-agent number. The hub record states no license, another reason to keep it out of training pipelines. Use it to score agents, and pair it with the full SWE-bench set if you need volume; at 2.1 MB, running it is free, leaking it is expensive.

SAVRN Research

## Files

3 files, 2.1 MB in total.

Data1 file · 2.1 MB

Documentation1 file · 3.3 KB

Repository1 file · 2.4 KB

Every file

| File | Type | Size | SHA-256 |
| --- | --- | --- | --- |
| data/test-00000-of-00001.parquet | Data | 2.1 MB | a45b1fe4e2f0 |
| README.md | Documentation | 3.3 KB | — |
| .gitattributes | Repository | 2.4 KB | — |

## License and Download

License

Not stated by the source

Access

No access gate

[Download from Princeton NLP group](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified)

Released by Princeton NLP group through its official repository on Hugging Face.

## Models Trained on This Dataset

- Trained on (disclosed)[Kronumos-Kairos-v2](https://savrn.com/models/kronumos-kairos-v2)
- Trained on (disclosed)[NadevA23_Kronumos-Kairos-v2-GGUF](https://savrn.com/models/nadeva23-kronumos-kairos-v2-gguf)

## Princeton NLP group

[All models and datasets](https://savrn.com/model-publishers/princeton-nlp)

## Explore More

- [The dataset directory](https://savrn.com/datasets)
- [The model directory](https://savrn.com/models)
- [Research papers](https://savrn.com/papers)

## Source

- Repository metadata, read 2026-10-19.
- [Hugging Face record](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified)
- [How the hub is built](https://savrn.com/model-hub/methodology)