Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 2

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios

Chen, Zhi, Ma, Wei, Jiang, Lingxiao·2026·Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering
DOI 10.1145/3744916.3773140
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3744916.3773140

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 68

  1. 10.1145/3650212.3680384 · DOI · Paper page
  2. Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:https://arXiv.org/abs/2303.08774 (2023). · arXiv
  3. 10.1145/3551349.3559555 · DOI · Paper page
  4. Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:https://arXiv.org/abs/2108.07732 (2021). · arXiv
  5. Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024. Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning.
  6. Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde De Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:https://arXiv.org/abs/2107.03374 (2021). · arXiv
  7. Zhi Chen and Lingxiao Jiang. 2024. Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios. In 32nd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER 2025). arXiv preprint arXiv:https://arXiv.org/abs/2410.12468 · arXiv
  8. 10.1145/3691620.3695021 · DOI
  9. Zimin Chen Steve Kommrusch Michele Tufano Louis-Noël Pouchet Denys Poshyvanyk and Martin Monperrus. 2019. Sequencer: Sequence-to-sequence learning for end-to-end program repair. IEEE Transactions on Software Engineering 47 9 (2019) 1943–1959.
  10. 10.1145/3597503.3639184 · DOI
  11. Israel Cohen Yiteng Huang Jingdong Chen Jacob Benesty Jacob Benesty Jingdong Chen Yiteng Huang and Israel Cohen. 2009. Pearson correlation coefficient. Noise reduction in speech processing (2009) 1–4. · DOI · Paper page
  12. Luís Cruz Xavier Franch Gutierrez and Silverio Martínez-Fernández. 2024. Innovating for Tomorrow: The Convergence of Software Engineering and Green AI. ACM Transactions on Software Engineering and Methodology (2024).
  13. Yanjun Fu Ethan Baker Yu Ding and Yizheng Chen. 2024. Constrained decoding for secure code generation. arXiv preprint arXiv:https://arXiv.org/abs/2405.00218 (2024). · arXiv
  14. Deipali Vikram Gore Mahima Binoj Sayali Borate Ritu Devnani and Sakshi Gopale. 2023. Syntax Error Detection and Correction in Python Code using ML.Grenze International Journal of Engineering & Technology (GIJET) 9 2 (2023).
  15. Aaron Grattafiori Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Alex Vaughan et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:https://arXiv.org/abs/2407.21783 (2024). · arXiv
  16. Daya Guo Dejian Yang Haowei Zhang Junxiao Song Ruoyu Zhang Runxin Xu Qihao Zhu Shirong Ma Peiyi Wang Xiao Bi et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:https://arXiv.org/abs/2501.12948 (2025). · arXiv
  17. 10.1109/SPW63631.2024.00014 · DOI
  18. Junda He Christoph Treude and David Lo. 2024. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review Vision and the Road Ahead. ACM Transactions on Software Engineering and Methodology (2024).
  19. SU Hongjin, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan O Arik. [n. d.]. Learn-by-interact: A Data-Centric Framework For Self-Adaptive Agents in Realistic Environments. In The Thirteenth International Conference on Learning Representations.
  20. 10.1109/ICSE.2019.00047 · DOI
  21. 10.1109/ICSE43902.2021.00107 · DOI · Paper page
  22. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66
  23. Haolin Jin Linghan Huang Haipeng Cai Jun Yan Bo Li and Huaming Chen. 2024. From llms to llm-based agents for software engineering: A survey of current challenges and future. arXiv preprint arXiv:https://arXiv.org/abs/2408.02479 (2024). · arXiv
  24. 10.1145/3597503.3640325 · doi:10.1145/3597503.3640325
  25. 10.1145/3510003.3510153 · DOI
  26. Jiaolong Kong Xiaofei Xie and Shangqing Liu. 2025. Demystifying Memorization in LLM-Based Program Repair via a General Hypothesis Testing Framework. Proceedings of the ACM on Software Engineering 2 FSE (2025) 2712–2734. · DOI
  27. 10.52202/079017-1837 · DOI
  28. 10.1145/3377812.3382172 · doi:10.1145/3377812.3382172
  29. 10.1145/3377811.3380345 · DOI · Paper page
  30. Jiawei Liu Chunqiu Steven Xia Yuyao Wang and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2023) 21558–21572. · DOI · Paper page
  31. Tianyang Liu, Canwen Xu, and Julian McAuley. 2024. RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=pPjZIOuQuF
  32. Yizhou Liu Pengfei Gao Xinchen Wang Jie Liu Yexuan Shi Zhao Zhang and Chao Peng. 2024. Marscode agent: Ai-native automated bug fixing. arXiv preprint arXiv:https://arXiv.org/abs/2409.00899 (2024). · arXiv
  33. Yue Liu Thanh Le-Cong Ratnadira Widyasari Chakkrit Tantithamthavorn Li Li Xuan-Bach D Le and David Lo. 2024. Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology 33 5 (2024) 1–26. · DOI · Paper page
  34. Zhijie Liu Yutian Tang Xiapu Luo Yuming Zhou and Liang Feng Zhang. 2024. No need to lift a finger anymore? assessing the quality of code generation by chatgpt. IEEE Transactions on Software Engineering (2024).
  35. 10.1109/ISSRE59848.2023.00026 · DOI
  36. Yingwei Ma Rongyu Cao Yongchang Cao Yue Zhang Jue Chen Yibo Liu Yuchen Liu Binhua Li Fei Huang and Yongbin Li. 2024. Lingma swe-gpt: An open development-process-centric language model for automated software improvement. arXiv preprint arXiv:https://arXiv.org/abs/2411.00622 (2024). · arXiv
  37. Yingwei Ma Qingping Yang Rongyu Cao Binhua Li Fei Huang and Yongbin Li. 2024. How to understand whole software repository?arXiv preprint arXiv:https://arXiv.org/abs/2406.01422 (2024). · arXiv
  38. 10.1109/SANER60148.2024.00051 · DOI
  39. Tina Marjanov Ivan Pashchenko and Fabio Massacci. 2022. Machine learning for source code vulnerability detection: What works and what isn’t there yet. IEEE Security & Privacy 20 5 (2022) 60–76. · DOI
  40. 10.1145/3460319.3464797 · DOI
  41. 10.1145/3524842.3528470 · DOI · Paper page
  42. 10.1145/3691620.3695545 · DOI
  43. 10.1109/SP46214.2022.9833571 · DOI · Paper page
  44. 10.1145/3597503.3608132 · DOI
  45. 10.1145/3643991.3645076 · DOI
  46. Brian Randell and Jie Xu. 2025. Looking Back on Recovery Blocks and Conversations. IEEE Transactions on Software Engineering (2025). · doi:10.1109/tse.2025.3533973
  47. Haifeng Ruan Yuntong Zhang and Abhik Roychoudhury. 2024. Specrover: Code intent extraction via llms. arXiv preprint arXiv:https://arXiv.org/abs/2408.02232 (2024). · arXiv
  48. 10.1109/SANER.2018.8330219 · DOI
  49. Jieke Shi Zhou Yang and David Lo. 2024. Efficient and green large language models for software engineering: Vision and the road ahead. ACM Transactions on Software Engineering and Methodology (2024).
  50. 10.1145/3643991.3645071 · DOI
  51. Hongjin Su Ruoxi Sun Jinsung Yoon Pengcheng Yin Tao Yu and Sercan Ö Arık. 2025. Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments. arXiv preprint arXiv:https://arXiv.org/abs/2501.10893 (2025). · arXiv
  52. Gemini Team Rohan Anil Sebastian Borgeaud Jean-Baptiste Alayrac Jiahui Yu Radu Soricut Johan Schalkwyk Andrew M Dai Anja Hauth Katie Millican et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:https://arXiv.org/abs/2312.11805 (2023). · arXiv
  53. Valerio Terragni Annie Vella Partha Roop and Kelly Blincoe. 2025. The Future of AI-Driven Software Engineering. ACM Transactions on Software Engineering and Methodology (2025). · DOI · Paper page
  54. 10.1109/MSR59073.2023.00084 · DOI · Paper page
  55. Carmine Vassallo Sebastiano Panichella Fabio Palomba Sebastian Proksch Harald C Gall and Andy Zaidman. 2020. How developers engage with static analysis tools in different contexts. Empirical Software Engineering 25 (2020) 1419–1457. · DOI · Paper page
  56. 10.1145/3551349.3560437 · doi:10.1145/3551349.3560437
  57. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2024. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representations.
  58. Yue Wang Weishi Wang Shafiq Joty and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:https://arXiv.org/abs/2109.00859 (2021). · DOI · Paper page
  59. 10.1145/3377811.3380426 · DOI
  60. Chunqiu Steven Xia Yinlin Deng Soren Dunn and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:https://arXiv.org/abs/2407.01489 (2024). · arXiv
  61. 10.1145/3650212.3680323 · DOI · Paper page
  62. John Yang Carlos E. Jimenez Alexander Wettig Kilian Lieret Shunyu Yao Karthik Narasimhan and Ofir Press. 2024. SWE-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024) 50528–50652. · DOI · Paper page
  63. 10.1145/3597503.3639074 · DOI
  64. 10.1145/3510003.3510127 · doi:10.1145/3510003.3510127
  65. Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu Yonghao Zhuang Zi Lin Zhuohan Li Dacheng Li Eric Xing et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023) 46595–46623. · doi:10.52202/075280-2020 · Paper page
  66. 10.1609/aaai.v38i19.30185 · DOI
  67. Mingchen Zhuge Changsheng Zhao Dylan R. Ashley Wenyi Wang Dmitrii Khizbullin Yunyang Xiong Zechun Liu Ernie Chang Raghuraman Krishnamoorthi Yuandong Tian Yangyang Shi Vikas Chandra and Jürgen Schmidhuber. 2025. Agent-as-a-Judge: Evaluating Agents with Agents. https://openreview.net/forum?id=DeVm3YUnpj
  68. Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. 2025. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=YrycTjllL0

Metadata

Authors
Chen, Z., Ma, W., & Jiang, L.
Venue
Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering
Publisher
ACM
Published
2026-04-12
Pages
1532-1544
DOI
10.1145/3744916.3773140
Cited by
2
References
5
Licence
creativecommons.org/licenses/by/4.0/legalcode (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios. https://app.cochat.ai/papers/10.1145/3744916.3773140
The paper
Chen, Z., Ma, W., & Jiang, L. (2026). Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios. Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering. https://doi.org/10.1145/3744916.3773140

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.