Human and Machine: How Software Engineers Perceive and Engage with AI-Assisted Code Reviews Compared to Their Peers
Open accessAbstract
The integration of artificial intelligence (AI) continues to increase and evolve, including in software engineering (SE). This integration involves processes traditionally entrusted to humans, such as coding. However, the impact on sociotechnical processes like code review remains underexplored. In this interview-based study (20 interviewees), we investigate how software engineers perceive and engage with Large Language Model (LLM)-assisted code reviews compared to human peerled reviews. In this inherently human-centric process, we aim to understand how software engineers navigate the introduction of AI into collaborative workflows. We found that engagement in code review is multi-dimensional, spanning cognitive, emotional, and behavioral dimensions. The introduction of LLM-assisted review impacts some of these attributes. For example, there is less need for emotional regulation and coping mechanisms when dealing with an LLM compared to peers. However, the cognitive load sometimes is higher in dealing with LLM-generated feedback due to its excessive details. Software engineers use a similar sense-making process to evaluate and adopt feedback suggestions from their peers and the LLM. However, the LLM feedback adoption is constrained by trust and lack of context in the review. Our findings contribute to a deeper understanding of how AI tools are impacting SE socio-technical processes and provide insights into the future of AI-human collaboration in SE practices.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most pull request datasets do not explicitly capture integration friction due to textual conflicts.
“As a result, integration friction due to textual conflicts is not explicitly captured in most pull request datasets.”
From AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub (2026) · cites this paper as [2]
Cites
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · see citing works
School Engagement: Potential of the Concept, State of the Evidence
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004) · Review of Educational Research · doi:10.3102/00346543074001059 · Cited by 12,581 · cited 1× · see citing works
“Behavioral: While emotional engagement involves the internal feelings and affective responses elicited by feedback, behavioral engagement refers to the observable actions of individuals in response to feedback as a social stimulus [25].”
Shute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 1× · see citing works
“For example, a prompt that lacks specificity or context may generate unrealistic and unrelatable feedback [49].”
Member Checking: A Tool to Enhance Trustworthiness or Merely a Nod to Validation?
Birt, L., Scott, S., Cavers, D., Campbell, C., & Walter, F. (2016) · Qualitative Health Research · doi:10.1177/1049732316654870 · Cited by 3,970 · see citing works
Why don't software developers use static analysis tools to find bugs?
Johnson, B., Song, Y., Murphy-Hill, E., & Bowdidge, R. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606613 · Cited by 568 · cited 1× · see citing works
“For example, Johnson et al. showed that reporting the factual outputs of the analysis is not sufficient for developers to adopt a static analysis tool [35].”
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 1× · see citing works
“The versatility and broad applicability of LLMs have the potential to fundamentally alter the landscape of software engineering by either reducing the reliance on human intervention or support in tasks that were previously considered too complex for automation [76], [82].”
Latent Predictor Networks for Code Generation
Ling, W., Blunsom, P., Grefenstette, E., Hermann, K. M., Kočiský, T., Wang, F., & Senior, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1057 · Cited by 307 · see citing works
Data Saturation: The Mysterious Step In Grounded Theory Method
Aldiabat, K., & Le Navenec, C.-L. (2018) · The Qualitative Report · doi:10.46743/2160-3715/2018.2994 · Cited by 222 · see citing works
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608128 · Cited by 203 · see citing works
On Designing Programming Error Messages for Novices: Readability and its Constituent Factors
Denny, P., Prather, J., Becker, B. A., Mooney, C., Homer, J., Albrecht, Z. C., & Powell, G. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445696 · Cited by 60 · see citing works
Human-AI collaboration in software engineering: lessons learned from a hands-on workshop
Hamza, M., Siemon, D., Akbar, M. A., & Rahman, T. (2024) · Proceedings of the 7th ACM/IEEE International Workshop on Software-intensive Business · doi:10.1145/3643690.3648236 · Cited by 38 · see citing works
Al Haque, E., Brown, C., LaToza, T. D., & Johnson, B. (2025) · Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3696630.3731665 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub
Ogenrwot, D., & Businge, J. (2026) · Proceedings of the 3rd ACM International Conference on AI-Powered Software · doi:10.1145/3805760.3814923
- Authors
- Alami, A., & Ernst, N.
- Affiliations
- University of Southern Denmark (DK)
- University of Victoria (CA)
- Venue
- 2025 IEEE/ACM 18th International Conference on Cooperative and Human Aspects of Software Engineering (CHASE)
- Publisher
- IEEE
- Published
- 2025-04-27
- Pages
- 63-74
- OpenAlex
- W4411233858
- Cited by
- 15
- References
- 13
- Citations by year
- 2026: 13 · 2025: 2
- FWCI
- 17.96
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- University of Southern Denmark Research Portal (University of Southern Denmark) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Aalborg University
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- AI-assisted code review, large language models, code review, human-AI collaboration, cognitive load, emotion regulation, sensemaking, socio-technical processes, collaborative workflows
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
