The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 9 of the 20 sources its claims cite were checked for support.
Human-AI Collaboration in Software Development: A Mixed-Methods Study of Developers' Use of GitHub Copilot and ChatGPT
Open accessAbstract
The integration of AI-powered coding assistants into software development presents both opportunities and challenges. These tools promise enhanced productivity and code quality while introducing significant concerns related to workflow integration, accuracy, reliability, and ethical considerations. This study investigates how software developers interact with Generative AI (GenAI) tools in a large-scale public sector organization. Using a mixed-methods approach, we analyze 13 interviews and a survey with 114 respondents to explore the impact of GitHub Copilot (IDE-integrated AI) and ChatGPT (conversational AI assistant) on software engineering workflows. The Human-AI Collaboration and Adaptation Framework (HACAF) is central to our analysis, which integrates insights from several technology acceptance models and theories. Our findings reveal that compatibility within existing workflows plays a crucial role in effectively utilizing GenAI tools. We explore individual, technological, and social factors that influence the adoption of these tools, highlighting the importance of organizational support and collective user patterns within teams. By studying both users and non-users, we provide insights into the factors driving GenAI adoption and the barriers preventing full integration. Our findings contribute to discussions on AI-enhanced IDEs, human-AI collaboration in coding, and the evolving role of LLM-based assistants in modern software development.
Overview
CoChat Summary
## TL;DR
This mixed-methods case study examines how developers in a large Norwegian public-sector organization used GitHub Copilot and ChatGPT, using 13 interviews and a survey of 114 respondents. The paper finds that adoption depends strongly on workflow compatibility, personal innovativeness, organizational support, and social/team norms, with clear differences between users and non-users.
## Key findings
- Among the 114 survey respondents, 67 were Copilot users and 47 were non-users; among the 13 interviewees, 4 were non-users and 1 had tried Copilot for a week and quit because it was “annoying.”
- Users reported higher averages on several outcomes than non-users, including job satisfaction (3.75 vs 3.60), flow while coding (4.20 vs 3.84), and ability to focus on satisfying work (3.77 vs 3.22); the paper reports that only the “focus on satisfying work” difference was significant, and that “over 70%” of users reported this advantage compared to “only half” of non-users.
- Users also reported lower dependence on colleagues: the paper says nearly half of users disagreed that they rely on colleagues for help and guidance, compared with just a fifth of non-users; elsewhere it states 45% of users explicitly said they do not depend on colleagues.
- The strongest qualitative themes were that GenAI is most attractive for repetitive/mundane tasks and when it fits existing workflows, while concerns about code quality, learning, ethics, and teamwork reduce adoption; the authors also state that adoption and dissemination are most successful when user patterns within teams are collective and similar.
## Methods
- The study uses a mixed-methods case study in “GovTech,” a large Norwegian public-sector organization responsible for social welfare and labor services, focusing on GitHub Copilot and ChatGPT.
- Data came from 13 semi-structured interviews conducted from November 7 to December 18, 2023, plus a survey with 114 respondents, of whom 81% were inhouse employees and 19% external consultants; 15% were women, 79% men, 1% “other,” and 5% declined to disclose gender.
- Interview data were coded with the Constant Comparison Method and a thematic analysis approach; the survey had 57 questions, used mostly 5-point Likert scales, was adapted from GitHub Research’s Copilot survey, and was analyzed predominantly in Python. The analysis was organized using Russo’s Human-AI Collaboration and Adaptation Framework (HACAF).
## Limitations
- The paper does not report effect sizes or p-values for the survey comparisons; it only says that most differences were not significant and that one finding was significant.
- The evidence is from a single organization and a pilot project with volunteers, so generalizability beyond this public-sector context is limited; this is evident from the study design, even though the paper does not explicitly frame it as a limitation.
- The survey items were adapted from a Copilot-focused instrument and generalized for users and non-users, which may limit comparability to the original survey; this is evident from the methods text, though not stated as a formal limitation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 4 Not valid · 12 Not checked · 75 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our investigation, guided by the Human-AI Collaboration and Adaptation Framework (HACAF) [13], highlights the complexities of integrating GenAI into professional workflows across individual, technological, and social dimensions.”
What the sources say
- Supportedcites [13] [13] Daniel Russo. 2024. Navigating the complexity of generative ai adoption in software engineering. ACMTransactions on Software Engineering and Methodology (2024). - source
Section 5 introduces HACAF as a framework for understanding GenAI adoption in software engineering. Section 7 describes its examination of technological perceptions, compatibility, social factors, and personal and environmental factors. The citation identifies the framework guiding the citing authors’ own investigation. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Since most studies of AI or GenAI adoption (e.g., Ziegler et al. [28]) centralize the experiences of users of such tools”
What the sources say
- Supportedcites [28] [28] Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming (MAPS 2022) . Association for Computing Machinery, New York, NY, USA, 21-29. source
The citation offers Ziegler et al. as an example of a study centered on tool users, not as evidence for the number of such studies. The Introduction (section 1) says the authors analyzed survey responses from 2,631 GitHub Copilot users to study their perceived productivity. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our interviews further revealed that work mode-whether the team was remote or co-located-affected GenAI tool usage patterns, similar to what was found in [19].”
What the sources say
- Supportedcites [19] [19] Viktoria Stray, Nils Brede Moe, Nivethika Ganeshan, and Simon Kobbenes. 2025. Generative AI and Developer Workflows: How GitHub Copilot and ChatGPT Influence Solo and Pair Programming. In Proceedings of the 58th Hawaii International Conference on System Sciences (HICSS) . - source
In §4.6 (Team dynamics), an interviewee says they may ask ChatGPT first when working from home because reaching colleagues takes longer, unlike the preference for asking colleagues when together. Section 4 (Results) also reports GenAI use in 29 of 31 physical pair-programming sessions versus one of two remote sessions. These passages support the cited similarity in usage patterns, without establishing a general causal effect. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All these five were very experienced software developers; one had worked in the profession for 13 years, two had worked for 17 years, and the last two had over 20 years' experience.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our analysis revealed themes and issues that corresponded to Russo's Human-AI Collaboration and Adaptation Framework (HACAF), and we hence present the results in accordance with the aspects drawn from the model (see Figure 1).”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
39 more claims
Create a free account to see all 99 claims and to verify the 4 references still open.
- Probably not a substantial claim. 38
- It may state two separate claims as one. 27
Citations
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483
User Acceptance of Information Technology: Toward A Unified View1
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003) · MIS Quarterly · doi:10.2307/30036540 · Cited by 45,035 · cited 1×
“…scope, the Unified Theory of Acceptance and Use of Technology (UTAUT), builds on TAM to explore users’ intentions to use specific technologies [24], and within a continuous software engineering environment, Barbala et al. [2] have investigated users’ ongoing experiences with rapid software…”
The Coding Manual for Qualitative Researchers
Lungu, M. (2022) · American Journal of Qualitative Research · doi:10.29333/ajqr/12085 · Cited by 10,826
Qualitative methods in empirical studies of software engineering
Seaman, C. B. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799955 · Cited by 1,493
The Essential Guide to Doing Your Research Project
Leary, Z. O., & Tennent, E. (2025) · SAGE Publications Ltd eBooks · doi:10.4135/9781036232665 · Cited by 1,353
Brynjolfsson, E., Li, D., & Raymond, L. (2023) · National Bureau of Economic Research · doi:10.3386/w31161 · Cited by 995
Adoption and use of AI tools: a research agenda grounded in UTAUT
Venkatesh, V. (2022) · Annals of Operations Research · doi:10.1007/s10479-020-03918-9 · Cited by 676
Proceedings of 48th Hawaii International Conference on System Sciences (HICSS)
Pöyry, E., Parvinen, P. M. T., & McFarland, R. (2015) · Cited by 643
Generative AI at WorkPreprint
Brynjolfsson, E., Li, D., & Raymond, L. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4426942 · Cited by 235
Investigating Explainability of Generative AI for Code through Scenario-based Design
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511119 · Cited by 219
Navigating the Complexity of Generative AI Adoption in Software Engineering
Russo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652154 · Cited by 217 · cited 4×
“Our investigation, guided by the Human-AI Collaboration and Adaptation Framework (HACAF) [13], highlights the complexities of integrating GenAI into professional workflows across individual, technological, and social dimensions.”
InfluentialProductivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200
The SPACE of Developer Productivity: There's more to it than you think.
Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021) · Queue · doi:10.1145/3454122.3454124 · Cited by 164
Bringing the Human Factor to Software Engineering
Capretz, L. F. (2014) · IEEE Software · doi:10.1109/ms.2014.30 · Cited by 113
Generative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71
ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks
Sridhara, G., G., R. H., & Mazumdar, S. (2023) · arXiv · doi:10.48550/arxiv.2305.16837 · Cited by 68
Copiloting the future: How generative AI transforms Software Engineering
Banh, L., Holldack, F., & Strobel, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107751 · Cited by 67
Ulfsnes, R., Moe, N. B., Stray, V., & Skarpen, M. (2024) · Generative AI for Effective Software Development · doi:10.1007/978-3-031-55642-5_10 · Cited by 62
Self-citationGenerative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
An Actionable Framework for Understanding and Improving Developer Experience
Greiler, M., Storey, M.-A., & Noda, A. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3175660 · Cited by 57
Diffusion of Innovations: An Overview
Rogers, E. M. (1981) · Computers and Medicine · doi:10.1007/978-1-4613-8674-2_9 · Cited by 52
Exploring human factors of the agile software tester
Stray, V. G., Florea, R., & Paruch, L. (2021) · Software Quality Journal · doi:10.1007/s11219-021-09561-2 · Cited by 36
Self-citationHuman-In-The-Loop Software Development Agents
Takerngsaksiri, W., Pasuksmit, J., Thongtanunam, P., Tantithamthavorn, C., Zhang, R., Jiang, F., Li, J., Cook, E., Chen, K., & Wu, M. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00036 · Cited by 36
Generative AI in Software Engineering Must Be Human-Centered: The Copenhagen Manifesto
Russo, D., Baltes, S., van Berkel, N., Avgeriou, P., Calefato, F., Cabrero-Daniel, B., Catolino, G., Cito, J., Ernst, N., Fritz, T., et al. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112115 · Cited by 34
Self-citationParikh, N. A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.04605 · Cited by 31
Parikh, N. A. (2023) · Advances in Business Strategy and Competitive Advantage · doi:10.4018/979-8-3693-1634-4.ch016 · Cited by 29
Human-In-the-Loop Software Development Agents
Takerngsaksiri, W., Pasuksmit, J., Thongtanunam, P., Tantithamthavorn, C., Zhang, R., Jiang, F., Li, J., Cook, E., Chen, K., & Wu, M. (2024) · arXiv · doi:10.48550/arxiv.2411.12924 · Cited by 24
The SPACE of developer productivity
Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021) · Communications of the ACM · doi:10.1145/3453928 · Cited by 21
(2025) · doi:10.24251/hicss.2025.883 · Cited by 14
Generative AI in the Workplace: Affective Affordances and Employee Flourishing
Barbala, A., Ulfsnes, R., Wivestad, V., et al. (2025) · Hawaii International Conference on System Sciences · doi:10.24251/hicss.2025.720 · Cited by 4
Self-citationIntelligent analysis for software data: research and applications
Tao, Z., Sun, X., Zheng, Z.-B., & Li, G. (2022) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2230000 · Cited by 3
Wivestad, V. T., Barbala, A., & Stray, V. G. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-72781-8_13 · Cited by 1
Self-citationA Case Study of Continuous Adoption in the Norwegian Public Sector
Barbala, A., Sporsem, T., & Stol, K.-J. (2024) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2024.248
Self-citation
Research works printed but not linked: 1
- Robert Yin. 2017. Case Study Research and Applications: Design and Methods (6 ed.). SAGE Publications.
Cited by
Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study
Stray, V., Brandtzæg, E. G., Wivestad, V., Barbala, A., & Moe, N. B. (2026) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2026.880 · Cited by 9
Self-citationCritiqueCrew: Orchestrating Multi-Perspective Conversational Design Critique
Chen, X., Zhou, J., Shu, Y., Wang, R., & Liu, Q. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791391 · Cited by 4
Duan, Y., Zhang, D., Jiang, Z., Xie, Z., Liu, Y., Xu, Y., Li, R., & Cui, D. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786872 · Cited by 1
Das, P., Reddy, Y. R., Potdar, S., & Warudkar, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36590-3_28
Liu, H. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-35043-5_15
Metadata
- Authors
- Stray, V., Barbala, A., & Wivestad, V. T.
- Affiliations
- University of Oslo (NO)
- SINTEF Digital (NO)
- Venue
- Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2025-06-23
- Pages
- 1325-1332
- OpenAlex
- W4412703929
- Semantic Scholar
- 58d8680bd7b9009af488bef393f63d117ca7fa61
- Cited by
- 19
- References
- 13
- Citations by year
- 2026: 19
- FWCI
- 10.42
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion, cc-by)
- Funders
- The Research Council of Norway (321477)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- human-AI collaboration, generative AI in software development, GitHub Copilot, ChatGPT, AI coding assistants, technology adoption, organizational support, public sector software development, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.