preprint · arXiv (Cornell University) · 2023

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Jimenez, Carlos E., Yang, John, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan

Cited by 2,096Open access

Abstract

Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models. To this end, we introduce SWE-bench, an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks. Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere $1.96$% of the issues. Advances on SWE-bench represent steps towards LMs that are more practical, intelligent, and autonomous.

Cite this paper

APA 7

Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv (Cornell University). https://doi.org/10.48550/arxiv.2310.06770

MLA 9

Jimenez, Carlos E., et al. “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” arXiv (Cornell University), 2023, https://doi.org/10.48550/arxiv.2310.06770.

Chicago

Jimenez, Carlos E., John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2310.06770.

Harvard

Jimenez, C.E. et al. (2023) ‘SWE-bench: Can Language Models Resolve Real-World GitHub Issues?’, arXiv (Cornell University). Available at: https://doi.org/10.48550/arxiv.2310.06770.

BibTeX

@misc{jimenez2023bench,
  title = {{SWE-bench: Can Language Models Resolve Real-World GitHub Issues?}},
  author = {Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik},
  journal = {arXiv (Cornell University)},
  year = {2023},
  doi = {10.48550/arxiv.2310.06770}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.