Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
Open accessAbstract
The task of issue resolving is to modify a codebase to generate a patch that addresses a given issue. However, existing benchmarks, such as SWE-bench, focus almost exclusively on Python, making them insufficient for evaluating Large Language Models (LLMs) across diverse software ecosystems. To address this, we introduce a multilingual issue-resolving benchmark, called Multi-SWE-bench, covering Java, TypeScript, JavaScript, Go, Rust, C, and C++. It includes a total of 1,632 high-quality instances, which were carefully annotated from 2,456 candidates by 68 expert annotators, ensuring that the benchmark can provide an accurate and reliable evaluation. Based on Multi-SWE-bench, we evaluate a series of state-of-the-art models using three representative methods (Agentless, SWE-agent, and OpenHands) and present a comprehensive analysis with key empirical insights. In addition, we launch a Multi-SWE-RL open-source community, aimed at building large-scale reinforcement learning (RL) training datasets for issue-resolving tasks. As an initial contribution, we release a set of 4,723 well-structured instances spanning seven programming languages, laying a solid foundation for RL research in this domain. More importantly, we open-source our entire data production pipeline, along with detailed tutorials, encouraging the open-source community to continuously contribute and expand the dataset. We envision our Multi-SWE-bench and the ever-growing Multi-SWE-RL community as catalysts for advancing RL toward its full potential, bringing us one step closer to the dawn of AGI.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior benchmarks emphasize isolated issue resolution or narrow subskills more than multistep evolution across commits and longer planning horizons.
“Prior benchmarks therefore emphasize isolated issue resolution or narrow subskills more than multistep evolution across commits and longer planning horizons.”
From SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios (2025) · cites this paper as Zan et al., 2025
The paper contrasts benchmarks that isolate code subtasks with SWE-bench’s broader repository-level issue-resolution tasks (passage labeled “Docker container”; page unavailable). Its Introduction describes SWE-bench as generating a patch for an issue, which supports the isolated-issue part of the claim. Our copy does not compare prior benchmarks with multistep evolution across commits or longer planning horizons. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 3
- B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y. Tian, M. Tan, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, et al. Multi-lingual evaluation of code generation models. In ICLR, 2023.
- M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code, 2021a. URL https: //arxiv.org/abs/2107.03374.
- J. Yang, C. E. Jimenez, A. L. Zhang, K. Lieret, J. Yang, X. Wu, O. Press, N. Muennighoff, G. Synnaeve, K. R. Narasimhan, D. Yang, S. I. Wang, and O. Press. SWE-bench multimodal: Do ai systems generalize to visual software domains? In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=riTiq3i21b.
Cited by
Sorted by citations
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Deng, X., Da, J., Pan, E., He, Y. Y., Ide, C., Garg, K., Lauffer, N., Park, A., Pasari, N., Rane, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.16941 · Cited by 4 · see citing works
“Building on this foundation, Zan et al. [24] extended the approach to multiple programming languages with Multi-SWE-bench, covering Java, TypeScript, JavaScript, Go, Rust, C, and C++ with 1,632 expert-curated instances.”
Government risk communication and response networks: Insights from typhoon news in China
Wang, L.-H., & Jiang, S.-Y. (2025) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2025.105587 · Cited by 3 · see citing works
Context as a Tool: Context Management for Long-Horizon SWE-Agents
Liu, S., Yang, J., Jiang, B., Li, Y., Guo, J., Liu, X., & Dai, B. (2025) · arXiv · doi:10.48550/arxiv.2512.22087 · Cited by 1 · see citing works
Self-citationSWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
Le, T., Thai, M. V. T., Manh, D. N., Nhat, H. P., & Bui, N. D. Q. (2025) · arXiv · doi:10.48550/arxiv.2512.18470
“Several extensions have since emerged to broaden coverage: Multi-SWE-bench (Zan et al., 2025) spans multiple programming languages including Java, TypeScript, Go, Rust, C, and C++, while Multimodal SWE-bench (Yang et al., 2024b) incorporates JavaScript and UI screenshots to test cross-modal…”
Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
Xia, C. S., Wang, Z., Yang, Y., Wei, Y., & Zhang, L. (2025) · arXiv · doi:10.48550/arxiv.2511.13646 · Cited by 2 · see citing works
“Since SWE-bench mostly focused on Python projects, researchers have proposed a number of SWE-bench-style benchmarks for projects in multiple languages for more comprehensive agent evaluation, including SWE-PolyBench [28], SWE-bench Multilingual [18], and Multi-SWE-bench [42].”
Self-citationMiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement
Core Team, Qiao, Z., Hua, Z., Ou, Z., Yue, Z., Jiang, Z., Huang, Z., Chen, Z., Zheng, Z., Xu, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2610.11959
Self-citationSWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
Desai, R., Hu, J., Cabezas, J., Harsola, N., Shukla, P., Chaim, R. B., Assadi, A. E., Kamath, O. M., Faldu, F., Hebbar, P., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.07682
“SWE-Bench [36] and SWE-Bench Verified [55] established repository-level patch generation from real GitHub issues, and Multi-SWE-bench [81] extends this setting across programming languages.”
Token Reduction Is Not Cost Reduction
Weinberger, S., & Hozez, A. (2026) · arXiv · doi:10.48550/arxiv.2607.12161 · Cited by 1 · see citing works
Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
Li, H., He, R., Mang, Q., Zhang, Q., Mao, H., Chen, X., Zhou, H., Zhang, H., Cheung, A., Gonzalez, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.02230
When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation
Yang, H., Yu, Q., & Desell, T. (2026) · arXiv · doi:10.48550/arxiv.2607.10569
PR-Aware Automated Unit Test Generation: Challenges and Opportunities
Haratian, V., Akar, A., Çakar, B., & Tüzün, E. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00055
PlayCoder: Making LLM-Generated GUI Code Playable
Peng, Z., Tao, W., Yin, X., Ying, C., Luo, Y., & Guo, Y. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808097 · Cited by 2 · see citing works
GREPO: A Benchmark for Graph Neural Networks on Repository-Level Bug Localization
Wang, J., Chen, L., Wang, X., Kang, S., Yang, H., Zheng, D., & Zhang, M. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817529
Self-citationCao, Y., Keung, J., Wang, Y., Sun, Y., & Mao, Z. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00062
Zhang, L., He, S., Zhang, C., Kang, Y., Li, B., Xie, C., Wang, J., Wang, M., Huang, Y., Fu, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.23419
Self-citation
- Authors
- Zan, D., Huang, Z., Liu, W., Chen, H., Zhang, L., Xin, S., Chen, L., Liu, Q., Zhong, X., Li, A., Liu, S., Xiao, Y., Chen, L., Zhang, Y., Su, J., Liu, T., Long, R., Shen, K., & Xiang, L.
- Affiliations
- ByteDance (China) (CN)
- Chinese Academy of Sciences (CN)
- National University of Singapore (SG)
- Shenzhen University (CN)
- Shandong University (CN)
- Alibaba Group (China) (CN)
- Tencent (China) (CN)
- Fudan University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- Pages
- 70112-70144
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2504.02605
- OpenAlex
- W4410313682
- Cited by
- 5
- References
- 0
- Citations by year
- 2026: 4 · 2025: 1
- FWCI
- 5.72
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Shandong University
- Shenzhen Technology University
- Peking University
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- software bug fixing, benchmark, multilingual programming languages, large language models, SWE-agent, OpenHands, reinforcement learning, code generation, annotated dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
