SWE-bench-Live

A brand-new, continuously updated SWE-bench-like dataset powered by an automated curation pipeline.

For the official data release page, please see microsoft/SWE-bench-Live.

Dataset Summary

SWE-bench-Live is a live benchmark for issue resolving, designed to evaluate an AI system’s ability to complete real-world software engineering tasks. Thanks to our automated dataset curation pipeline, we plan to update SWE-bench-Live on a monthly basis to provide the community with up-to-date task instances and support rigorous and contamination-free evaluation.

News

• 9/17/2025: Dataset updated! We’ve finalized the update process for SWE-bench-Live: Each month, we will add 50 newly verified, high-quality issues to the dataset. The lite and verified splits will remain frozen, ensuring fair leaderboard comparisons and keeping evaluation costs manageable. To access the latest issues, please refer to the full split!
• 07/19/2025: We've employed a LLM filter to automatically filter full dataset to create SWE-bench-Live Verified. The initial Verified subset contains 500 instances from 2024-07 to 2025-04.
• 06/30/2025: We’ve updated the dataset — it now includes a total of 1,565 task instances across 164 repositories!
• 05/21/2025: The initial release of SWE-bench-Live includes 1,319 latest (created after 2024) task instances, each paired with an instance-level Docker image for test execution, covering 93 repositories.

SWE-bench-Live Curation Pipeline

Dataset Structure

Evaluation Protocol

[!NOTE] SWE-bench-Live evaluation strictly follows the original SWE-bench protocol:

1. During a rollout, the agent may access only the problem_statement field of the Hugging Face dataset and the docker image of the task instance. It must not access any other fields, such as hint, FAIL_TO_PASS, or test_patch. The test_patch must not be applied to the repository before or during the rollout. The agent must perform a single rollout based solely on the problem_statement on the docker container started from the image of the task instance.
2. Prompts, skills, and workflow instructions provided to the agent must not contain solutions specific to any task instance. They may contain only general instructions for the entire benchmark or, at most, for a specific repository. The agent must not use results from the ground-truth evaluation script to refine its solution.

Compliant prompts should follow the SWE-agent prompt and the OpenHands prompt, which contain only the problem statement and general workflow instructions.

When submitting results to our submissions repository, you must include your agent's raw rollout trajectories so that the maintainers can verify compliance with the SWE-bench protocol. A trajectory consists of the complete sequence of inputs to and outputs from your agent across all rollout rounds for a given task instance, including the initial prompt provided to the agent. Please follow this SWE-agent compliant trajectory example when submitting your result. If your organization's policies prohibit sharing the complete set of trajectories, you must provide at least some representative samples for verification. There is a checklist when submitting a PR to help you check whether you meet the protocol requirements again.