Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Large Language Models (LLMs) are frequently discussed in academia and the general public as support tools for virtually any use case that relies on the production of text, including software engineering. Currently, there is much debate, but little empirical evidence, regarding the practical usefulness of LLM-based tools such as ChatGPT for engineers in industry. We conduct an observational study of 24 professional software engineers who have been using ChatGPT over a period of one week in their jobs, and qualitatively analyse their dialogues with the chatbot as well as their overall experience (as captured by an exit survey). We find that rather than expecting ChatGPT to generate ready-to-use software artifacts (e.g., code), practitioners more often use ChatGPT to receive guidance on how to solve their tasks or learn about a topic in more abstract terms. We also propose a theoretical framework for how the (i) purpose of the interaction, (ii) internal factors (e.g., the user’s personality), and (iii) external factors (e.g., company policy) together shape the experience (in terms of perceived usefulness and trust). We envision that our framework can be used by future research to further the academic discussion on LLM usage by software engineering practitioners, and to serve as a reference point for the design of future empirical LLM research in this domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 1×
“We are aware of one study that explored the types of interactions between developers and a code-generating chatbot (Co-Pilot) by Barke et al. [21].”
BackgroundUse chat gpt to solve programming bugs
Surameery, N. M. S., & Shakor, M. Y. (2023) · International Journal of Information technology and Computer Engineering · doi:10.55529/ijitc.31.17.22 · Cited by 382 · cited 1×
“LLM-powered chatbots have been studied in more specific contexts and software-related activities, for example code generation [6], [7], [8], requirements analysis [9], test generation [10], and others [11], [12], [13].”
BackgroundAn empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313 · cited 1×
“LLM-powered chatbots have been studied in more specific contexts and software-related activities, for example code generation [6], [7], [8], requirements analysis [9], test generation [10], and others [11], [12], [13].”
BackgroundCodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306 · cited 1×
“LLM-powered chatbots have been studied in more specific contexts and software-related activities, for example code generation [6], [7], [8], requirements analysis [9], test generation [10], and others [11], [12], [13].”
de Vries, P., Midden, C., & Bouwhuis, D. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00039-9 · Cited by 264
Best humans still outperform artificial intelligence in a creative divergent thinking task
Koivisto, M., & Grassini, S. (2023) · Scientific Reports · doi:10.1038/s41598-023-40858-3 · Cited by 247
Towards Human-Bot Collaborative Software Architecting with ChatGPT
Ahmad, A., Waseem, M., Liang, P., Fahmideh, M., Aktar, M. S., & Mikkonen, T. (2023) · Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/3593434.3593468 · Cited by 168 · cited 1×
“Recent studies have been looking into the helpfulness of chatbots in terms of decision-making [17] and efficiency [18], which can vary depending on the context.”
Background
Cited by
Krauß, V., McGill, M., Kosch, T., Thiel, Y. M., Schön, D., & Gugenheimer, J. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713083 · Cited by 13
The Fast and Spurious: Developer Productivity with GenAI
Afroz, S., Feng, Z., Menezes, T., Kimura, K., Trinkenreich, B., Steinmacher, I., & Sarma, A. (2025) · arXiv · doi:10.48550/arxiv.2510.24265 · Cited by 11
On the Need to Rethink Trust in AI Assistants for Software Development: A Critical Review
Baltes, S., Speith, T., Chiteri, B., Mohsenimofidi, S., Chakraborty, S., & Buschek, D. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3659804 · Cited by 5
“GenAI tools, for example ChatGPT and GitHub Copilot, support software engineers in tasks such as understanding existing code, fixing bugs, or implementing new features [74], [147].”
From Disruptions to Discussions: How GenAI Impacts Human Interactions in Software Development
Salomon, M., Koshchenko, E., Sergeyuk, A., Holmes, R., Murphy, G. C., & Fritz, T. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3655626 · Cited by 1
“While numerous studies have explored how individual software developers use GenAI tools for their work and its impact (e.g., [3]–[8]), fewer have examined its role in supporting developers working together in a software development team (e.g., [9], [10]).”
Large Language Model Company Policies and Policy Implications in Software Organizations
Khojah, R., Mohamad, M., Erlenhov, L., de Oliveira Neto, F. G., & Leitner, P. (2026) · IEEE Software · doi:10.1109/ms.2025.3622039 · Cited by 1
Self-citationSame Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code
Khojah, R., Gomes de Oliveira Neto, F., Mohamad, M., Frattini, J., & Leitner, P. (2026) · Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3832783.3834423
Self-citation
Metadata
- Authors
- Khojah, R., Mohamad, M., Leitner, P., & de Oliveira Neto, F. G.
- Venue
- Proceedings of the ACM on Software Engineering
- Published
- 2024-07-12
- Volume
- 1
- Issue
- FSE
- Pages
- 1819-1840
- ISSN
- 2994-970X (ISSN-L)
- OpenAlex
- W4400582696
- Cited by
- 86
- References
- 15
- FWCI
- 32.91
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.