CoChat
Sign Up
PreprintCited by 5Software Engineering Techniques and PracticesSoftware Engineering ResearchArtificial Intelligence Applications

Walking the Tightrope of LLMs for Software Development: A Practitioners' Perspective

Open access
Ferino, Samuel, Hoda, Rashina, Grundy, John, Treude, Christoph·2025·arXiv
DOI 10.48550/arxiv.2511.06428arXiv 2511.06428
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Large Language Models emerged with the potential of provoking a revolution in software development (e.g., automating processes, workforce transformation). Although studies have started to investigate the perceived impact of LLMs for software development, there is a need for empirical studies to comprehend how to balance forward and backward effects of using LLMs. Objective: We investigated how LLMs impact software development and how to manage the impact from a software developer's perspective. Method: We conducted 22 interviews with software practitioners across 3 rounds of data collection and analysis, between October (2024) and September (2025). We employed Socio-Technical Grounded Theory for Data Analysis (STGT4DA) to rigorously analyse interview participants' responses. Results: We identified the benefits (e.g., maintain developer flow, improve developer mental models, and foster entrepreneurship) and challenges (e.g., damage to developers' reputation) of using LLMs at individual, team, organisation, and society levels; as well as actionable guidances into how mitigate these challenges. Conclusion: Critically, we present the trade-offs that software practitioners, teams, and organisations face in working with LLMs. Our findings are particularly useful for software team leaders and IT managers to assess the viability of LLMs within their specific context.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2511.06428

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Research envisions a change in the role of a software developer to focus more on code verification [40], [41].

    “This is particularly important since research envisions a change in the role of a software developer to focus more on code verification [40], [41].”

    From Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code (2026) · cites this paper as [40]

    The paper’s recommendations section (“3. https://h2o.ai”; page unavailable) calls for human review of LLM-generated code before it is merged, supporting the code-verification aspect. Its Implications for Research section also describes developers overseeing autonomous agents. Neither passage says that developers’ roles will shift to focus more on code verification specifically. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 49

  1. I. Ozkaya, "Application of large language models to software engineering tasks: Opportunities, risks, and implications," IEEE Software, vol. 40, no. 3, pp. 4-8, 2023.
  2. T. Teubner et al., "Welcome to the era of chatgpt et al. the prospects of large language models," Business & Information Systems Engineering, vol. 65, no. 2, pp. 95-101, 2023.
  3. Y. K. Dwivedi et al., "Opinion paper:"so what if chatgpt wrote it?" multidisciplinary perspectives on opportunities, challenges and implica- tions of generative conversational AI for research, practice and policy," International Journal of Information Management, vol. 71, p. 102642, 2023.
  4. A. M. Dakhel et al., "Github copilot AI pair programmer: Asset or liability?" Journal of Systems and Software, vol. 203, p. 111734, 2023.
  5. X. Zhou et al., "Exploring the problems, their causes and solutions of AI pair programming: A study on github and stack overflow," Journal of Systems and Software, p. 112204, 2024.
  6. X. Hou et al., "Large language models for software engineering: A systematic literature review," ACM Trans. Softw. Eng. Methodol., Sep. 2024. [Online]. Available: https://doi.org/10.1145/3695988 · DOI · Paper page
  7. J. Wang et al., "Software testing with large language models: Survey, landscape, and vision," IEEE Transactions on Software Engineering, vol. 50, no. 4, pp. 911-936, 2024.
  8. J. Di Rocco et al., "On the use of large language models in model-driven engineering," Software and Systems Modeling, pp. 1-26, 2025.
  9. H. Mayer et al., "Superagency in the workplace: Empowering people to unlock AI's full potential," McKinsey Digital, vol. 28, 2025.
  10. G. Burtch et al., "The consequences of generative AI for online knowledge communities," Scientific Reports, vol. 14, no. 1, p. 10413, 2024.
  11. S. Kabir, D. N. Udo-Imeh, B. Kou, and T. Zhang, "Is stack overflow obsolete? an empirical study of the characteristics of chatgpt answers to stack overflow questions," in Proceedings of the CHI Conference on Human Factors in Computing Systems, 2024, pp. 1-17.
  12. A. Ziegler et al., "Measuring github copilot's impact on productivity," Communications of the ACM, vol. 67, no. 3, pp. 54-63, 2024.
  13. Z. K. Cui, M. Demirer, S. Jaffe, L. Musolff, S. Peng, and T. Salz, "The effects of generative AI on high skilled work: Evidence from three field experiments with software developers," Available at SSRN 4945566, 2024.
  14. C. Ebert and P. Louridas, "Generative AI for software practitioners," IEEE Software, vol. 40, no. 4, pp. 30-38, 2023.
  15. D. Nam, A. Macvean, V. Hellendoorn, B. Vasilescu, and B. Myers, "Using an llm to help with code understanding," in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1-13.
  16. M. A. Kuhail et al., ""will i be replaced?" assessing chatgpt's effect on software development and programmer perceptions of AI tools," Science of Computer Programming, vol. 235, p. 103111, 2024.
  17. V. Krauß et al., ""create a fear of missing out"-chatgpt implements unsolicited deceptive designs in generated websites without warning," in Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 2025, pp. 1-20.
  18. H.-P. H. Lee et al., "The impact of generative AI on critical thinking: Self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers," in Proceedings of the CHI Conference on Human Factors in Computing Systems, 2025.
  19. X. Chen et al., "An empirical study on challenges for llm application de- velopers," ACM Transactions on Software Engineering and Methodology, 2025.
  20. J. Shi, Z. Yang, and D. Lo, "Efficient and green large language models for software engineering: Literature review, vision, and the road ahead," ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, pp. 1-22, 2025.
  21. A. Mohamed, M. Assi, and M. Guizani, "The impact of llm-assistants on software developer productivity: A systematic literature review," arXiv preprint arXiv:2507.03156, 2025. · arXiv
  22. S. Ferino, R. Hoda, J. Grundy, and C. Treude, "Novice developers' perspectives on adopting LLMs for software development: A systematic literature review," ACM Transactions on Software Engineering and Methodology, 2026.
  23. R. Hoda, "Socio-technical grounded theory for software engineering," IEEE Transactions on Software Engineering, vol. 48, no. 10, pp. 3808- 3832, 2022.
  24. C. Sindermann et al., "Assessing the attitude towards artificial intelligence: Introduction of a short measure in german, chinese, and english language," KI-Künstliche intelligenz, vol. 35, no. 1, pp. 109-118, 2021.
  25. R. Hoda, Qualitative Research with Socio-Technical Grounded Theory. Springer, 2024. [Online]. Available: https://link.springer.com/book/10. 1007/978-3-031-60533-8 · DOI
  26. A. Pant, R. Hoda, S. V. Spiegler, C. Tantithamthavorn, and B. Turhan, "Ethics in the age of AI: An analysis of AI practitioners' awareness and challenges," ACM Transactions on Software Engineering and Methodol- ogy, vol. 33, no. 3, pp. 1-35, 2024.
  27. K. Madampe et al., "The struggle is real! the agony of recruiting partic- ipants for empirical software engineering studies," in 2024 IEEE Sym- posium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 2024, pp. 417-422.
  28. A. Madugalla, T. Kanij, R. Hoda, D. Hidellaarachchi, A. Pant, S. Ferdousi, and J. Grundy, "Challenges, adaptations, and fringe benefits of conducting software engineering research with human participants during the covid- 19 pandemic," Empirical Software Engineering, vol. 29, no. 4, p. 86, 2024.
  29. H. Gunatilake, J. Grundy, R. Hoda, and I. Mueller, "Manifestations of empathy in software engineering: How, why, and when it matters," IEEE Transactions on Software Engineering, 2025.
  30. Z. Masood, R. Hoda, and K. Blincoe, "Real world scrum a grounded theory of variations in practice," IEEE Transactions on Software Engineering, vol. 48, no. 5, pp. 1579-1591, 2020.
  31. M. F. Argerich and M. Patiño-Martínez, "Measuring and improving the energy efficiency of large language models inference," IEEE Access, vol. 12, pp. 80 194-80 207, 2024.
  32. X. Sun, D. Ståhl, K. Sandahl, and C. Kessler, "Quality assurance of llm-generated code: Addressing non-functional quality characteristics," arXiv preprint arXiv:2511.10271, 2025. · arXiv
  33. M. McLuhan, "Laws of the media," ETC: A Review of General Semantics, pp. 173-179, 1977.
  34. N. S. Mathews and M. Nagappan, "Test-driven development and llm-based code generation," in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE '24. New York, NY, USA: Association for Computing Machinery, 2024, p. 1583-1594. [Online]. Available: https://doi.org/10.1145/3691620.3695527 · DOI
  35. M. Azanza, J. Pereira, A. Irastorza, and A. Galdos, "Can LLMs facilitate onboarding software developers? an ongoing industrial case study," in 2024 36th International Conference on Software Engineering Education and Training (CSEE&T). IEEE, 2024, pp. 1-6.
  36. N. Jegham, M. Abdelatti, L. Elmoubarki, and A. Hendawi, "How hungry is AI? benchmarking energy, water, and carbon footprint of llm inference," arXiv preprint arXiv:2505.09598, 2025. · arXiv
  37. A. Alami and N. Ernst, "Human and machine: How software engineers perceive and engage with AI-assisted code reviews compared to their peers," in 2025 IEEE/ACM 18th International Conference on Cooperative and Human Aspects of Software Engineering (CHASE). IEEE, 2025, pp. 63-74.
  38. E. Kalliamvakou et al., "What makes a great manager of software engineers?" IEEE Transactions on Software Engineering, vol. 45, no. 1, pp. 87-106, 2017.
  39. P. Naur, "Intuition in software development," in International Joint Conference on Theory and Practice of Software Development. Springer, 1985, pp. 60-79.
  40. V. Chen et al., "Understanding the role of human intuition on reliance in human-ai decision-making with explanations," Proceedings of the ACM on Human-computer Interaction, vol. 7, no. CSCW2, pp. 1-32, 2023.
  41. T. R. Tulili, A. Capiluppi, and A. Rastogi, "Burnout in software engineering: A systematic mapping study," Information and Software Technology, vol. 155, p. 107116, 2023.
  42. Z. Feng, S. Afroz, and A. Sarma, "From gains to strains: Modeling developer burnout with genai adoption," arXiv preprint arXiv:2510.07435, 2025. · arXiv
  43. L. Banh, F. Holldack, and G. Strobel, "Copiloting the future: How generative AI transforms software engineering," Information and Software Technology, vol. 183, p. 107751, 2025.
  44. J. T. Liang et al., "Prompts are programs too! understanding how developers build software containing prompts," Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 1591-1614, 2025.
  45. Z. S. Li et al., "Ai tool use and adoption in software development by individuals and organizations: a grounded theory study," arXiv preprint arXiv:2406.17325, 2024. · arXiv
  46. B. Glaser and A. Strauss, Discovery of grounded theory: Strategies for qualitative research. Routledge, 2017.
  47. M. Juliet and S. Corbin, Basics of qualitative research: Techniques and procedures for developing grounded theory. SAGE Publications, Incorporated, 2015.
  48. P. Lenberg et al., "Qualitative software engineering research: Reflections and guidelines," Journal of Software: Evolution and Process, vol. 36, no. 6, p. e2607, 2024.
  49. I. Korstjens and A. Moser, "Series: Practical guidance to qualitative research. part 4: Trustworthiness and publishing," European Journal of General Practice, vol. 24, no. 1, pp. 120-124, 2018.

Cited by

Sorted by citations

Authors
Ferino, S., Hoda, R., Grundy, J., & Treude, C.
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2511.06428
arXiv
2511.06428
OpenAlex
W7105020815
Cited by
5
References
2
Citations by year
2026: 5
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
large language models, software development, developer experience, developer reputation, software development teams, artificial intelligence in software development

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review