CoChat
Sign Up
PreprintCited by 7Software Engineering ResearchSoftware Testing and Debugging TechniquesSoftware Engineering Techniques and Practices

SWE-Bench+: Enhanced Coding Benchmark for LLMs

Open access
Aleithan, Reem, Xue, Haoran, Mohajer, Mohammad Mahdi, Nnorom, Elijah, Uddin, Gias, Wang, Song·2024·arXiv
DOI 10.48550/arxiv.2410.06992arXiv 2410.06992
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) in Software Engineering (SE) can offer assistance for coding. To facilitate a rigorous evaluation of LLMs in practical coding contexts, Carlos et al. introduced the SWE-bench dataset, which comprises 2,294 real-world GitHub issues and their corresponding pull requests, collected from 12 widely used Python repositories. Several impressive LLM-based toolkits recently are developed and evaluated on this dataset. However, a systematic evaluation of the quality of SWE-bench remains missing. In this paper, we addressed this gap by presenting an empirical analysis of the SWE-bench dataset. We conducted a manual screening of instances where SWEAgent + GPT-4 successfully resolved issues by comparing the model-generated patches with the actual pull requests. SWE-Agent+GPT-4 was at the top of SWE-bench leaderboard during the time of our study. Our analysis reveals some critical issues with the SWE-bench dataset: 1) 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments. We refer to as solution leakage problem. 2) 31.08% of the passed patches are suspicious patches due to weak test cases, i.e., the tests were not adequate to verify the correctness of a patch. When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. We also observed that the same data quality issues also exist in the two variants of SWE-bench, i.e., SWE-bench Lite and SWE-Bench Verified. In addition, over 94% of the issues were created before LLM's knowledge cutoff dates, posing potential data leakage issues.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.06992

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 7

Research works printed but not linked: 64

  1. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, "Swe-bench: Can language models resolve real-world github issues?" 2024. [Online]. Available: https://arxiv.org/abs/2310.06770 · arXiv
  2. D. Chen, S. Lin, M. Zeng, D. Zan, J.-G. Wang, A. Cheshkov, J. Sun, H. Yu, G. Dong, A. Aliev, J. Wang, X. Cheng, G. Liang, Y. Ma, P. Bian, T. Xie, and Q. Wang, "Coder: Issue resolving with multi-agent and task graphs," 2024. [Online]. Available: https://arxiv.org/abs/2406.01304 · arXiv
  3. K. Zhang, W. Yao, Z. Liu, Y. Feng, Z. Liu, R. Murthy, T. Lan, L. Li, R. Lou, J. Xu, B. Pang, Y. Zhou, S. Heinecke, S. Savarese, H. Wang, and C. Xiong, "Diversity empowers intelligence: Integrating expertise of software engineering agents," 2024. [Online]. Available: https://arxiv.org/abs/2408.07060 · arXiv
  4. C. S. Xia, Y. Deng, S. Dunn, and L. Zhang, "Agentless: Demystifying llm-based software engineering agents," 2024. [Online]. Available: https://arxiv.org/abs/2407.01489 · arXiv
  5. J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, "Swe-agent: Agent-computer interfaces enable automated software engineering," 2024. [Online]. Available: https://arxiv.org/abs/2405.15793 · arXiv
  6. Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, "Autocoderover: Autonomous program improvement," 2024. [Online]. Available: https://arxiv.org/abs/2404.05427 · arXiv
  7. R. L. Rosa, C. Hulse, and B. Liu, "Can github issues be solved with tree of thoughts?" 2024. [Online]. Available: https://arxiv.org/abs/2405.13057 · arXiv
  8. D. Zan, Z. Huang, A. Yu, S. Lin, Y. Shi, W. Liu, D. Chen, Z. Qi, H. Yu, L. Yu, D. Ran, M. Zeng, B. Shen, P. Bian, G. Liang, B. Guan, P. Huang, T. Xie, Y. Wang, and Q. Wang, "Swe-bench-java: A github issue resolving benchmark for java," 2024. [Online]. Available: https://arxiv.org/abs/2408.14354 · arXiv
  9. J. Liu, C. S. Xia, Y. Wang, and L. Zhang, "Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation," Advances in Neural Information Processing Systems, vol. 36, pp. 21 558-21 572, 2023.
  10. J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, "A survey on large language models for code generation," 2024. [Online]. Available: https://arxiv.org/abs/2406.00515 · arXiv
  11. X. Li and T. Döhmen, "Towards efficient data wrangling with llms using code generation," in Proceedings of the Eighth Workshop on Data Management for End-to-End Machine Learning, ser. DEEM '24. New York, NY, USA: Association for Computing Machinery, 2024, p. 62-66. [Online]. Available: https://doi.org/10.1145/3650203.3663334 · DOI
  12. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, "Evaluating large language models trained on code," 2021. [Online]. Available: https://arxiv.org/abs/2107.03374 · arXiv
  13. H. Luo, J. Wu, J. Liu, and M. F. Antwi-Afari, "Large language model-based code generation for the control of construction assembly robots: A hierarchical generation approach," Developments in the Built Environment, vol. 19, p. 100488, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2666165924001698
  14. X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y. Chen, J. Feng, C. Sha, X. Peng, and Y. Lou, "Evaluating large language models in class-level code generation," in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE '24. New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3597503.3639219 · DOI
  15. Q. Zhang, T. Zhang, J. Zhai, C. Fang, B. Yu, W. Sun, and Z. Chen, "A critical review of large language model on software engineering: An example from chatgpt and automated program repair," 2024. [Online]. Available: https://arxiv.org/abs/2310.08879 · arXiv
  16. B. Yang, H. Tian, W. Pian, H. Yu, H. Wang, J. Klein, T. F. Bissyandé, and S. Jin, "Cref: An llm-based conversational software repair framework for programming tutors," in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2024. New York, NY, USA: Association for Computing Machinery, 2024, p. 882-894. [Online]. Available: https://doi.org/10.1145/3650212.3680328 · DOI · Paper page
  17. D. de Fitero-Dominguez, E. Garcia-Lopez, A. Garcia-Cabot, and J.-J. Martinez-Herraiz, "Enhanced automated code vulnerability repair using large language models," Engineering Applications of Artificial Intelligence, vol. 138, p. 109291, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0952197624014490
  18. K. Alrashedy and A. Binjahlan, "Language models are better bug detector through code-pair classification," 2024. [Online]. Available: https://arxiv.org/abs/2311.07957 · arXiv
  19. S. B. Hossain, N. Jiang, Q. Zhou, X. Li, W.-H. Chiang, Y. Lyu, H. Nguyen, and O. Tripp, "A deep dive into large language models for automated bug localization and repair," Proc. ACM Softw. Eng., vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3660773 · DOI
  20. A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, "Large language models for software engineering: Survey and open problems," 2023. [Online]. Available: https://arxiv.org/abs/2310.03533 · arXiv
  21. X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, "Large language models for software engineering: A systematic literature review," 2024. [Online]. Available: https://arxiv.org/abs/2308.10620 · arXiv
  22. D. Zan, B. Chen, F. Zhang, D. Lu, B. Wu, B. Guan, Y. Wang, and J.-G. Lou, "Large language models meet nl2code: A survey," 2023. [Online]. Available: https://arxiv.org/abs/2212.09420 · arXiv
  23. Z. Zheng, K. Ning, Y. Wang, J. Zhang, D. Zheng, M. Ye, and J. Chen, "A survey of large language models for code: Evolution, benchmarking, and future trends," 2024. [Online]. Available: https://arxiv.org/abs/2311.10372 · arXiv
  24. Y. Wang, H. Le, S. Joty, and S. C. Hoi, "Codet5+: Open code large language models for code understanding and generation," arXiv preprint arXiv:2305.07922, 2023. · arXiv
  25. B. Shen, J. Zhang, T. Chen, D. Zan, B. Geng, A. Fu, M. Zeng, A. Yu, J. Ji, J. Zhao, Y. Guo, and Q. Wang, "Pangu-coder2: Boosting large language models for code with ranking feedback," 2023. [Online]. Available: https://arxiv.org/abs/2307.14936 · arXiv
  26. F. F. Xu, V. J. Hellendoorn, R. Singh, P. Maniatis, and A. Terzis, "A systematic evaluation of large language models of code," arXiv preprint arXiv:2202.13169, 2022. · arXiv
  27. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021. · arXiv
  28. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, "A conversational paradigm for program synthesis," arXiv preprint arXiv:2203.13474, 2022. · arXiv
  29. B. Roziere, G. Izacard, S. Withnall, V. Dinh, C. Lovett, K. Simonyan, Y. Belinkov, S. Edunov, V. Karpukhin, A. Parikh et al., "Code llama: Open foundation models for code," 2023. [Online]. Available: https://arxiv.org/abs/2308.12950 · arXiv
  30. W. Weimer, S. Forrest, C. Le Goues, and T. Nguyen, "Automatically finding patches using genetic programming," in Proceedings of the 31st International Conference on Software Engineering. IEEE Computer Society, 2009, pp. 364-374.
  31. J. Jiang, Y. Xiong, H. Zhang, Q. Gao, and X. Chen, "Shaping program repair space with existing patches and similar code," in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2018, pp. 298-309.
  32. J. Xuan, M. Martinez, F. Demarco, M. Clement, S. L. Marcote, T. Durieux, D. Le Berre, and M. Monperrus, "Nopol: Automatic repair of conditional statement bugs in java programs," IEEE Transactions on Software Engineering, vol. 43, no. 1, pp. 34-55, 2017.
  33. H. Liu, J. Chen, Y. Xiong, D. Wang, and L. Zhang, "Tbar: revisiting template-based automated program repair," in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2019, pp. 31-42.
  34. Z. Chen, S. J. Kommrusch, M. Tufano, L.-N. Pouchet, D. Poshyvanyk, and M. Monperrus, "Sequencer: Sequence-to-sequence learning for end-to-end program repair," in IEEE Transactions on Software Engineering. IEEE, 2019.
  35. T. Lutellier, W. Yu, L. Tan, and V. Le, "Coconut: Combining context-aware neural translation models using ensemble for program repair," in Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2020, pp. 101-114.
  36. Q. Zhu, Z. Sun, Y.-a. Xiao, W. Zhang, K. Yuan, Y. Xiong, and L. Zhang, "A syntax-guided edit decoder for neural program repair," in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2021, pp. 341-353.
  37. C. S. Xia, Y. Wei, and L. Zhang, "Automated program repair in the era of large pre-trained language models," in Proceedings of the 45th International Conference on Software Engineering, ser. ICSE '23. IEEE Press, 2023, p. 1482-1494. [Online]. Available: https://doi.org/10.1109/ICSE48619.2023.00129 · DOI · Paper page
  38. C. S. Xia and L. Zhang, "Less training, more repairing please: Revisiting automated program repair via zero-shot learning," 2022. [Online]. Available: https://arxiv.org/abs/2207.08281 · arXiv
  39. I. Bouzenia, P. Devanbu, and M. Pradel, "Repairagent: An autonomous, llm-based agent for program repair," arXiv preprint arXiv:2403.17134, 2024. · arXiv
  40. F. Li, J. Jiang, J. Sun, and H. Zhang, "Hybrid automated program repair by combining large language models and program analysis," arXiv preprint arXiv:2406.00992, 2024. · arXiv
  41. P. Wang, S. Jain, E. Zelikman, E. Shi, J. Wu, P. Liang, and T. B. Hashimoto, "Swe-agent: Autonomously fixing software bugs using large language models," arXiv preprint arXiv:2305.13214, 2023. [Online]. Available: https://arxiv.org/abs/2305.13214 · arXiv
  42. Z. Yu, X. Lu, F. Wang, and Y. Xue, "Autocoderover: Revisiting program repair from a retriever-generator perspective," arXiv preprint arXiv:2403.19827, 2024. [Online]. Available: https://arxiv.org/abs/2403.19827 · arXiv
  43. R. Bavishi, H. Yoshida, M. Sridharan, S. Chandra, M. Kulkarni, and M. R. Prasad, "Phoenix: Automated data-driven synthesis of repairs for static analysis violations," in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2019, pp. 613-624.
  44. R. van Tonder and C. L. Goues, "Static automated program repair for heap properties," in Proceedings of the 40th International Conference on Software Engineering (ICSE). ACM, 2018, pp. 151-162.
  45. K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, "Avatar: Fixing semantic bugs with fix patterns of static analysis violations," in 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2019, pp. 1-12.
  46. D. Marcilio, R. Bonifácio, E. Monteiro, E. Canedo, W. Luz, and G. Pinto, "Are static analysis violations really fixed? a closer look at realistic usage of sonarqube," in 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, 2019, pp. 209-219.
  47. A. Carvalho, W. Luz, D. Marcílio, R. Bonifácio, G. Pinto, and E. D. Canedo, "C-3pr: A bot for fixing static analysis violations via pull requests," in 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2020, pp. 161-171.
  48. C. Calcagno and D. Distefano, "Moving fast with software verification," in NASA Formal Methods Symposium. Springer, 2015, pp. 3-11.
  49. M. M. Mohajer, R. Aleithan, N. S. Harzevili, M. Wei, A. B. Belle, H. V. Pham, and S. Wang, "Skipanalyzer: An embodied agent for code analysis with large language models," CoRR, 2023.
  50. Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang, "Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt," 2023. [Online]. Available: https://arxiv.org/abs/2304.02014 · arXiv
  51. Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, "Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models," 2023. [Online]. Available: https://arxiv.org/abs/2212.14834 · arXiv
  52. M. Allamanis, P. Chanthirasegaran, M. Brockschmidt, O. Polozov, and D. Tarlow, "Self-supervised bug detection and repair," in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 25 638-25 650.
  53. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, "Codebert: A pre-trained model for programming and natural languages," in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 1536-1547.
  54. L. Shi, F. Mu, X. Chen, S. Wang, J. Wang, Y. Yang, G. Li, X. Xia, and Q. Wang, "Are we building on the rock? on the importance of data preprocessing for code summarization," BIBLIOGRAPHY in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 107-119.
  55. Z. Sun, L. Li, Y. Liu, X. Du, and L. Li, "On the importance of building high-quality training datasets for neural code search," CoRR, vol. abs/2202.06649, 2022. [Online]. Available: https://arxiv.org/abs/2202.06649 · arXiv
  56. D. Gros, H. Sezhiyan, P. Devanbu, and Z. Yu, "Code to comment "translation": Data, metrics, baselining & evaluation," in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2020, pp. 746-757.
  57. E. Shi, Y. Wang, L. Du, J. Chen, S. Han, H. Zhang, D. Zhang, and H. Sun, "Neural code summarization: How far are we?" arXiv preprint arXiv:2107.07112, 2021. · arXiv
  58. Y. Huang, N. Jia, J. Shu, X. Hu, X. Chen, and Q. Zhou, "Does your code need comment?" Software: Practice and Experience, vol. 50, no. 3, pp. 227-245, 2020.
  59. S. Roy, M. Allamanis, and C. Sutton, "On the reliability of automatic evaluation in code summarization," in Proceedings of the 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 1147-1158.
  60. A. Mahmud, S. Roy, and M. M. Rahman, "Code summarization: How far have we come?" in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 1077-1089.
  61. W. Xu, S. Wang, X. Zhang, L. Ou, X. Wang, and B. Sun, "How good is the software produced by github copilot? an exploratory study," in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 2022, pp. 1213-1225.
  62. W. Tao, Y. Zhou, Y. Wang, W. Zhang, H. Zhang, and Y. Cheng, "Magis: Llm-based multi-agent framework for github issue resolution," 2024. [Online]. Available: https://arxiv.org/abs/2403.17927 · arXiv
  63. Y. Zheng, X. Mu, Y. Wang, H. Zhang, H. Zhao, Y. Gao, X. Zou, X. Jin, V. Sriku- mar, K.-W. Chang et al., "Code evaluation with enhanced test suites," arXiv preprint arXiv:2305.01210, 2023. · arXiv
  64. J. Liu, C. S. Xia, Y. Wang, and L. Zhang, "Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation," Advances in Neural Information Processing Systems, vol. 36, 2024.

Authors
Aleithan, R., Xue, H., Mohajer, M. M., Nnorom, E., Uddin, G., & Wang, S.
Affiliations
York University (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2410.06992
arXiv
2410.06992
OpenAlex
W4403363594
Cited by
7
References
0
Citations by year
2026: 3 · 2025: 4
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • York University Digital Library (York University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
SWE-bench, patch evaluation, test adequacy, data leakage, LLM-based code generation, GitHub issues, SWE-agent, GPT-4, Python repositories

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review