CoChat
Sign Up
Preprint

The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering

Li, Hao, Zhang, Haoxiang, Hassan, Ahmed E.·2025·arXiv
DOI 10.48550/arxiv.2507.15003arXiv 2507.15003
Open this paper in CoChat — free account

Abstract

The future of software engineering--SE 3.0--is unfolding with the rise of AI teammates: autonomous, goal-driven systems collaborating with human developers. Among these, autonomous coding agents are especially transformative, now actively initiating, reviewing, and evolving code at scale. This paper introduces AIDev, the first large-scale dataset capturing how such agents operate in the wild. Spanning over 456,000 pull requests by five leading agents--OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code--across 61,000 repositories and 47,000 developers, AIDev provides an unprecedented empirical foundation for studying autonomous teammates in software development. Unlike prior work that has largely theorized the rise of AI-native software engineering, AIDev offers structured, open data to support research in benchmarking, agent readiness, optimization, collaboration modeling, and AI governance. The dataset includes rich metadata on PRs, authorship, review timelines, code changes, and integration outcomes--enabling exploration beyond synthetic benchmarks like SWE-bench. For instance, although agents often outperform humans in speed, their PRs are accepted less frequently, revealing a trust and utility gap. Furthermore, while agents accelerate code submission--one developer submitted as many PRs in three days as they had in three years--these are structurally simpler (via code complexity metrics). We envision AIDev as a living resource: extensible, analyzable, and ready for the SE and AI communities. Grounding SE 3.0 in real-world evidence, AIDev enables a new generation of research into AI-native workflows and supports building the next wave of symbiotic human-AI collaboration. The dataset is publicly available at https://github.com/SAILResearch/AI_Teammates_in_SE3. > AI Agent, Agentic AI, Coding Agent, Agentic Coding, Software Engineering Agent

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2507.15003

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The AIDev dataset contains 932,791 Agentic PRs.

    “The dataset contains 932,791 Agentic PRs.”

    From AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub (2026) · cites this paper as [34]

    The Dataset Overview (Section 3.2, pp. 6–7) states that the full AIDev dataset contains 456,535 pull requests, not 932,791. Table 3 confirms the 456,535 total. (LLM, conf=0.94)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Existing AI-centric datasets primarily focus on high-level metadata such as acceptance rates and discussion dynamics and do not provide explicit or reproducible labels for textual merge conflicts.

    “While these datasets offer valuable insights into how AI-generated contributions are created and reviewed, they primarily focus on high-level metadata such as acceptance rates and discussion dynamics. They do not provide explicit or reproducible labels for textual merge conflicts, limiting our ability to systematically quantify the integration friction introduced by AI-generated changes.”

    From AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub (2026) · cites this paper as [34]

    The Dataset Schema (p. 7) lists PR outcomes, review comments, timelines, and code patches, and the analyses examine acceptance rates and review dynamics. It does not explicitly state that AIDev or other AI-centric datasets lack reproducible labels for textual merge conflicts, or that this limits systematic measurement of integration friction; the discussion of resolving merge conflicts in Section 4.2 (p. 14) is a proposed research direction. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • The 'SE 3.0' framing describes development as orchestration.

    “the 'SE 3.0' framing of development as orchestration [92]”

    From Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems (2026) · cites this paper as [92]

  • Software Engineering 3.0 is characterized by agents acting as AI teammates rather than tools.

    “Li et al. [18] characterize this as 'Software Engineering 3.0,' where agents act as AI teammates rather than tools”

    From OpenClaw AI Agents as Informal Learners at Moltbook: Characterizing an Emergent Learning Community at Scale (2026) · cites this paper as [18]

  • AIDev provides a large-scale collection of contributions from multiple AI coding agents.

    “AIDev [34] provides a large-scale collection of contributions from multiple AI coding agents.”

    From AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub (2026) · cites this paper as [34]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Research works printed but not linked: 32

  1. [3][3] Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2025. Can LLMs Replace Manual Annotation of Software Engineering Artifacts?. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR) . 526-538. doi:10.1109/MSR66628.2025.00086 - · DOI
  2. [4][4] Leonhard Applis, Yuntong Zhang, Shanchao Liang, Nan Jiang, Lin Tan, and Abhik Roychoudhury. 2025. Unified Software Engineering agent as AI Software Engineer. arXiv:2506.14683 [cs.SE] https://arxiv.org/abs/2506.14683 - · arXiv
  3. [5][5] Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2025. To Code or Not To Code? Exploring Impact of Code in Pre-training. In The Thirteenth International Conference on Learning Representations . https://openreview.net/forum?id=zSfeN1uAcx -
  4. [6][6] Aaditya Bhatia, Gustavo A. Oliva, Gopi Krishnan Rajbahadur, Haoxiang Zhang, Yihao Chen, Zhilong Chen, Arthur Leung, Dayi Lin, Boyuan Chen, and Ahmed E. Hassan. 2025. SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation. (Jul 2025). arXiv:2507.09108 [cs.SE] doi:10.48550/arXiv.2507.09108 - · DOI
  5. [7][7] Islem Bouzenia, Prem Devanbu, and Michael Pradel. 2024. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair. 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) (2024), 2188-2200. -
  6. [8][8] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair . In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) . IEEE Computer Society, Los Alamitos, CA, USA, 2188-2200. doi:10.1109/ ICSE55347.2025.00157 -
  7. [9][9] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. arXiv:1606.01540 [cs.LG] https://arxiv.org/abs/1606.01540 - · arXiv
  8. [10][10] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code . arXiv:2107.03374 [cs.SE] https://arxiv.org/abs/2107.03374 - · arXiv
  9. [11][11] Runxiang Cheng, Michele Tufano, Jürgen Cito, José Cambronero, Pat Rondon, Renyao Wei, Aaron Sun, and Satish Chandra. 2025. Agentic Bug Reproduction for Effective Automated Program Repair at Google. (2025). arXiv:2502.01821 [cs.SE] https://arxiv.org/abs/2502.01821 - · arXiv
  10. [12][12] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132 [cs.AI] https://arxiv.org/abs/2403.04132 - · arXiv
  11. [13][13] Umut Cihan, Vahid Haratian, Arda İçöz, Mert Kaan Gül, Ömercan Devran, Emircan Furkan Bayendur, Baykal Mehmet Uçar, and Eray Tüzün. 2024. Automated Code Review In Practice. arXiv:2412.18531 [cs.SE] https://arxiv.org/abs/2412.18531 - · arXiv
  12. [14][14] Dekun Dai, MingWei Liu, Anji Li, Jialun Cao, Yanlin Wang, et al. 2025. FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks. arXiv preprint arXiv:2504.06939 (2025). - · arXiv
  13. [15][15] Giuseppe Desolda, Andrea Esposito, Francesco Greco, Cesare Tucci, Paolo Buono, and Antonio Piccinno. 2025. Understanding User Mental Models in AI-Driven Code Completion Tools: Insights from an Elicitation Study. arXiv:2502.02194 [cs.HC] https://arxiv.org/abs/2502.02194 - · arXiv
  14. [16][16] Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023. Self-Collaboration Code Generation via ChatGPT. ACM Transactions on Software Engineering and Methodology 33 (2023), 1-38. -
  15. [17][17] Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler. 2021. The SPACE of Developer Productivity: There's more to it than you think. Queue 19, 1 (March 2021), 20-48. doi:10.1145/3454122.3454124 - · DOI · Paper page
  16. [18][18] Ahmed E. Hassan, Dayi Lin, Gopi Krishnan Rajbahadur, Keheliya Gallaba, Filipe Roseiro Cogo, Boyuan Chen, Haoxiang Zhang, Kishanthan Thangarajah, Gustavo Oliva, Jiahuei (Justina) Lin, et al. 2024. Rethinking Software Engineering in the Era of Foundation Models: A Curated Catalogue of Challenges in the Development of Trustworthy FMware. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE) . 294-305. -
  17. [19][19] Ahmed E. Hassan, Gustavo A. Oliva, Dayi Lin, Boyuan Chen, and Zhen Ming Jiang. 2024. Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmap . arXiv:2410.06107 [cs.SE] https://arxiv.org/abs/2410.06107 - · arXiv
  18. [20][20] Junda He, Christoph Treude, and David Lo. 2025. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead. ACM Trans. Softw. Eng. Methodol. 34, 5, Article 124 (May 2025), 30 pages. doi:10.1145/3712003 - · DOI · Paper page
  19. [21][21] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In The Twelfth International Conference on Learning Representations . https://openreview.net/forum?id= VTF8yNQM66 -
  20. [22][22] Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam. 2025. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason. arXiv:2506.12286 [cs.AI] https://arxiv.org/abs/2506.12286 - · arXiv
  21. [23][23] Shanchao Liang, Yiran Hu, Nan Jiang, and Lin Tan. 2024. Can Language Models Replace Programmers? REPOCOD Says 'Not Yet'. arXiv preprint arXiv:2410.21647 (2024). - · arXiv
  22. [24][24] Feng Lin, Dong Jae Kim, and Tse-Hsun Chen. 2024. SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents. 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) (2024), 1527-1539. -
  23. [25][25] Jeffrey D. Long, Du Feng, and Norman Cliff. 2003. Ordinal Analysis of Behavioral Data. In Handbook of Psychology , Irving B. Weiner (Ed.). John Wiley & Sons, Inc., Hoboken, NJ, USA, Chapter 25, 635-661. doi:10.1002/0471264385.wei0225 - · DOI
  24. [26][26] H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. Annals of Mathematical Statistics 18 (1947), 50-60. -
  25. [27][27] Dimitrios-Nikitas Nastos, Themistoklis Diamantopoulos, Davide Tosi, Martina Tropeano, and Andreas L. Symeonidis. 2025. Towards an Interpretable Analysis for Estimating the Resolution Time of Software Issues. arXiv:2505.01108 [cs.SE] https://arxiv.org/abs/2505.01108 - · arXiv
  26. [28][28] Ketai Qiu, Niccolò Puccinelli, Matteo Ciniselli, and Luca Di Grazia. 2025. From Today's Code to Tomorrow's Symphony: The AI Transformation of Developer's Routine by 2030. ACM Trans. Softw. Eng. Methodol. 34, 5, Article 121 (May 2025), 17 pages. doi:10.1145/3709353 - · DOI
  27. [29][29] Peter C. Rigby, Seth Rogers, Sadruddin Saleem, Parth Suresh, Daniel Suskin, Patrick Riggs, Chandra Maddila, Nachiappan Nagappan, and Audris Mockus. 2025. Improving Code Reviewer Recommendation: Accuracy, Latency, Workload, and Bystanders. ACM Trans. Softw. Eng. Methodol. (May 2025). doi:10.1145/3736405 Just Accepted. - · DOI
  28. [31][31] Abhik Roychoudhury, Corina Pasareanu, Michael Pradel, and Baishakhi Ray. 2025. Agentic AI Software Engineers: Programming with Trust. arXiv:2502.13767 [cs.SE] https://arxiv.org/abs/2502.13767 - · arXiv
  29. [33][33] Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning. In The Thirteenth International Conference on Learning Representations . https://openreview.net/forum?id=4FWAwZtd2n -
  30. [34][34] John Yang, Carlos E. Jimenez, Alex L. Zhang, et al. 2024. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? arXiv preprint arXiv:2410.03859 (2024). - · arXiv
  31. [35][35] Shaoyi Yang. 2025. Chain of Draft for Software Engineering: Challenges in Applying Concise Reasoning to Code Tasks. arXiv:2506.10987 [cs.SE] https://arxiv.org/abs/2506.10987 - · arXiv
  32. [36][36] Xin Zhou, Martin Weyssow, Ratnadira Widyasari, Ting Zhang, Junda He, Yunbo Lyu, Jianming Chang, Beiqi Zhang, Dan Huang, and David Lo. 2025. LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks. arXiv:2502.06215 [cs.SE] https://arxiv.org/abs/2502.06215 · arXiv

Authors
Li, H., Zhang, H., & Hassan, A. E.
Venue
arXiv
Publisher
arXiv
Published
2025
DOI
10.48550/arxiv.2507.15003
arXiv
2507.15003
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review