A brand-new, continuously updated SWE-bench-like dataset powered by an automated curation pipeline.
- The evaluation script to evaluate the prediction patches of your agent on our public huggingface datasets: _SWE-bench-Live/SWE-bench-Live (Python)_, _SWE-bench-Live/MultiLang_ and _SWE-bench-Live/Windows_.
- The task-creation source code for you to create your customized SWE tasks for large-scale agentic RFT/RL, each paired with an executable docker sandbox.
News
- 21/08/2026: Today, SWE-bench-Live/MultiLang has grown to 1077 task instances, covering 431 repositories and 8 languages, with each language split containing more than 100 instances! SWE-bench-Live/Windows has grown to 66 instances covering 48 repos and 9 languages. For this update we take an improved strategy suggested by RepoLaunch's users to build&test the repos: only one commit for each repo is selected for RepoLaunch to build&test; after RepoLaunch completes that commit, other commits of the same repo in the dataset is git-checked out directly from the built image, and the RepoLaunch extracted commands and parsers are re-used. This method achieves >=98% success with 82% savings on LM API cost and 78% savings on Docker image storage space when creating execution environments for 856 GitHub issues from 93 repos. See Development.md.
- 08/03/2026: SWE-bench-Live/Windows has been released along with the leaderboard, evaluating LLM's ability to resolve Windows-specific implementation and take actions in powershell. Newest paper on the multi-language and multi-os SWE task sets is available at RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms.
- 10/01/2026: SWE-bench-Live/Multi-Language with the leaderboard has been released. Merged into main. Supported languages: C/C++, C#, Java, TS/JS, Go, Rust. For old source code SWE-bench-Live/SWE-bench-Live (Python-only, the NIPS paper version), refer to python-only branch.
- 09/17/2025: Dataset updated (through 08/2025)! We’ve finalized the update process for huggingface dataset SWE-bench-Live/SWE-bench-Live (Python tasks): Each month, we will add 50 newly verified, high-quality issues to the dataset test split. The
liteandverifiedsplits will remain frozen, ensuring fair leaderboard comparisons and keeping evaluation costs manageable. To access all the latest issues, please refer to thefullsplit!
🚀 Set Up
``bash
Python >= 3.10
pip install -e .
`
[!NOTE]
Though this eval script has ensured backward compatibility with SWE-bench-Live/SWE-bench-Live (Python-only, the NIPS paper version), which uses swebench library for evaluation, if you want to evaluate on SWE-bench-Live/SWE-bench-Live (Python), for fair comparison we still recommend you to go to our old Python-only branch and follow the old evaluation method. The below eval script is more suitable for our new datasets SWE-bench-Live/MultiLang and SWE-bench-Live/Windows.
Test your installation by running:
`bash
python -m evaluation.evaluation \
--dataset SWE-bench-Live/MultiLang \
--instance_ids rsyslog__rsyslog-6047 \
--platform linux \
--patch_dir gold \
--output_dir logs/test \
--workers 1 \
--overwrite 1
`
🚥 Evaluation
[!NOTE]
Several users have raised questions about the evaluation protocol, so we would like to clarify that SWE-bench-Live evaluation strictly follows the original
SWE-bench protocol:
1. During a rollout, the agent may access only the
problem_statement field of the Hugging Face dataset and the docker image of the task instance. It must not access any other fields, such as hint, FAIL_TO_PASS, or test_patch. The test_patch must not be applied to the repository before or during the rollout. The agent must perform a single rollout based solely on the problem_statement on the docker container started from the image of the task instance.
2. Prompts, skills, and workflow instructions provided to the agent must not contain solutions specific to any task instance. They may contain only general instructions for the entire benchmark or, at most, for a specific repository. The agent must not use results from the ground-truth evaluation script to refine its solution.
Compliant prompts should follow the SWE-agent prompt and the OpenHands prompt, which contain only the problem statement and general workflow instructions.
When submitting results to our submissions repository, you must include your agent's raw rollout trajectories so that the maintainers can verify compliance with the SWE-bench protocol. A trajectory consists of the complete sequence of inputs to and outputs from your agent across all rollout rounds for a given task instance, including the initial prompt provided to the agent. Please follow this SWE-agent compliant trajectory example when submitting your result. If your organization's policies prohibit sharing the complete set of trajectories, you must provide at least some representative samples for verification. There is a checklist when submitting a PR to help you check whether you meet the protocol requirements again.
Guide on running your model/agent on SWE-bench-Live: GO TO evaluation/README.md
⬆️ Submit your results
Thank you for your interest in submitting the success rate of your agent/model to SWE-bench-Live! We coordinate results submission via Pull Requests, see SWE-bench-Live/submissions for instructions.
🐳 Development
If you would like to create your own SWE task instances with executable sandboxes, please follow Development.md.
Dataset Curation
In SWE-bench-Live, we propose an automated pipeline for curating SWE-bench-like dataset.
SWE-bench-Live Curation Pipeline
RepoLaunch
We addresses the bottleneck of setting up execution environments by automating the process through an LLM-based agentic tool – RepoLaunch. It can deliver a testable containerized environment for any given GitHub repository, thereby enabling test-based evaluation in SWE-bench-Live.
Collaboration
We welcome external collaborators to help us create more SWE tasks each month, and improve the curation and RepoLaunch source code. Please feel free to raise issues, open discussions and contribute pull requests to this repository and to the RepoLaunch repository to help us improve.
📚 Citation
If you refer to the SWE task creation pipeline of SWE-bench-Live, or SWE-bench-Live/SWE-bench-Live (Python only tasks), please cite
`bibtex
@article{zhang2025swebenchgoeslive,
title={SWE-bench Goes Live!},
author={Linghao Zhang and Shilin He and Chaoyun Zhang and Yu Kang and Bowen Li and Chengxing Xie and Junhao Wang and Maoquan Wang and Yufan Huang and Shengyu Fu and Elsie Nallipogu and Qingwei Lin and Yingnong Dang and Saravan Rajmohan and Dongmei Zhang},
journal={Advances in Neural Information Processing Systems},
volume={38},
year={2025}
}
`
If you refer to the automated build and test tool _RepoLaunch_, SWE benchmarking/training/RFT/RL environment build, SWE-bench-Live/Multi-Language or SWE-bench-Live/Windows, please cite
`bibtex
@article{li2026repolaunch,
title={RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms},
author={Kenan Li and Rongzhi Li and Linghao Zhang and Qirui Jin and Liao Zhu and Xiaosong Huang and Geng Zhang and Yikai Zhang and Shilin He and Chengxing Xie and Xin Zhang and Zijian Jin and Bowen Li and Chaoyun Zhang and Yu Kang and Yufan Huang and Elsie Nallipogu and Saravan Rajmohan and Qingwei Lin and Dongmei Zhang},
journal={arXiv preprint arXiv:2603.05026},
year={2026}
}
``