The DOI matches its Crossref record; 151 of the 151 references it cites were found in a registry or an index.
Trade-offs for Substituting a Human with an Agent in a Pair Programming Context: The Good, the Bad, and the Ugly
Abstract
Pair programming has a documented history of benefits, such as increased code quality, productivity, self-efficacy, knowledge transfer, and reduced gender gap. Research uncovered problems with pair programming related to scheduling, collocating, role imbalance, and power dynamics. We investigated the trade-offs of substituting a human with an agent to simultaneously provide benefits and alleviate obstacles in pair programming. We conducted gender-balanced studies with human-human pairs in a remote lab with 18 programmers and Wizard-of-Oz studies with 14 programmers, then analyzed results quantitatively and qualitatively. Our comparative analysis of the two studies showed no significant differences in productivity, code quality, and self-efficacy. Further, agents facilitated knowledge transfer; however, unlike humans, agents were unable to provide logical explanations or discussions. Human partners trusted and showed humility towards agents. Our results demonstrate that agents can act as effective pair programming partners and open the way towards new research on conversational agents for programming.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We envision that a conversational agent interface could be a natural fit for AI assistant for code or co-programming tools, as explored in recent work [42,43]”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [43]
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
WEST, C., & ZIMMERMAN, D. H. (1987) · Gender & Society · doi:10.1177/0891243287001002002 · Cited by 8,954
Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory
Wuetherick, B. (2010) · Canadian Journal of University Continuing Education · doi:10.21225/d5g01t · Cited by 8,311
Computer Self-Efficacy: Development of a Measure and Initial Test
Compeau, D. R., & Higgins, C. A. (1995) · MIS Quarterly · doi:10.2307/249688 · Cited by 6,553
Self-Determination Theory in Work Organizations: The State of a Science
Deci, E. L., Olafsen, A. H., & Ryan, R. M. (2017) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-032516-113108 · Cited by 3,199
Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817
Change by design : how design thinking transforms organizations and inspires innovation
Brown, T., & Kātz, B. M. (2009) · Cited by 2,588
Étude comparative de la distribution florale dans une portion des Alpes et du Jura
Jaccard, P. (1901) · E-Periodica · doi:10.5169/seals-266450 · Cited by 2,353
Amabile, T. M., & Pratt, M. G. (2016) · Research in Organizational Behavior · doi:10.1016/j.riob.2016.10.001 · Cited by 1,863 · cited 2×
“Adapting activity based on partner’s ability [97] Making rapport by greeting [84] and motivational dialogues [5, 31, 45, 53]”
Background
Research works printed but not linked: 33
- Michael Armstrong. 2012. Armstrong’s handbook of reward management practice: Improving performance through reward (12 ed.). Kogan Page Publishers, London.
- 10.4135/9781446221129.n6 · DOI · Paper page
- 10.1093/iwc/iwv046 · DOI · Paper page
- Ramón Burri. 2018. Improving user trust towards conversational chatbot interfaces with voice output. Master’s thesis. KTH School of Electrical Engineering and Computer Science (EECS).
- Lan Cao, Kannan Mohan, Peng Xu, and Balasubramaniam Ramesh. 2004. How Extreme Does Extreme Programming Have to Be? Adapting XP Practices to Large-Scale Projects. In Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS’04) - Track 3 - Volume 3(HICSS ’04). IEEE Computer Society, USA, 30083.3.
- Christopher P Cerasoli, Jessica M Nicklin, and Michael T Ford. 2014. Intrinsic motivation and extrinsic incentives jointly predict performance: A 40-year meta-analysis.Psychological bulletin 140, 4 (2014), 980.
- Pedro Costa and Luísa Ribas. 2019. AI becomes her: Discussing gender and artificial intelligence.Technoetic Arts: A Journal of Speculative Research 17, 1/2(2019), 171 – 193.
- Sally Jo Cunningham, Annika Hinze, and David M. Nichols. 2016. Supporting Gender-Neutral Digital Library Creation: A Case Study Using the GenderMag Toolkit. In Digital Libraries: Knowledge, Information, and Data in an Open Access Society, Atsuyuki Morishima, Andreas Rauber, and Chern Li Liew (Eds.). Springer International Publishing, Cham, 45–50.
- Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz studies—why and how. Knowledge-based systems 6, 4 (1993), 258–266.
- M. Day, M. R. Penumala, and J. Gonzalez-Sanchez. 2019. Annete: An Intelligent Tutoring Companion Embedded into the Eclipse IDE. In 2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI). IEEE, New York, US, 71–80.
- Berland Edelman and Inc. 2010. Creativity and education: Why it matters. Adobe. Retrieved September 18th, 2019 from http://www.adobe.com/aboutadobe/pressroom/pdfs/Adobe_Creativity_and_Education_Why_It_Matters_study.pdf
- S.T. Fiske E.H.P.P.S.T. Fiske and S.E. Taylor. 1991. Social Cognition. McGraw-Hill New York City USA. https://books.google.com/books?id=6Uq3QgAACAAJ
- H. Gallis, Erik Arisholm, and Tore Dybå. 2002. A Transition From Partner Programming to Pair Programming - an Industrial Case Study. In Workshop ”Pair Programming Installed” in 17th Annual ACM Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), Position paper. ACM, Seattle USA, –.
- H. Gallis, E. Arisholm, and T. Dyba. 2003. An initial framework for research on pair programming. In 2003 International Symposium on Empirical Software Engineering. ACM, NY, USA, 132–142.
- Dai Hasegawa, Justine Cassell, and Kenji Araki. 2010. The Role of Embodiment and Perspective in Direction-Giving Systems. In Proceedings of AAAI Fall Workshop on Dialog with Robots. AAAI PRESS, USA.
- 10.1177/0018720814547570PMID: · doi:10.1177/0018720814547570pmid
- D. James and S. Clarke. 1998. Women men and interruptions: A critical review. In D. Tannen (Ed.) Oxford studies in sociolinguistics. Gender and conversational interaction. Oxford University Press UK 231–280.
- 10.1016/j.jvlc.2017.04.005Special · doi:10.1016/j.jvlc.2017.04.005special
- S. Kaur Kuttal, K. Gerstner, and A. Bejarano. 2019. Remote Pair Programming in Online CS Education: Investigating through a Gender Lens. In 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, USA, 75–85.
- Danielle L. Jones and Scott D. Fleming. 2013. What use is a backseat driver? A qualitative investigation of pair programming. In Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC. IEEE, USA, 103–110.
- 10.1145/3195570.3195580 · DOI · Paper page
- Clayton Lewis. 1982. Using the ”thinking-aloud” method in cognitive interface design. IBM T.J. Watson Research Center, Yorktown Heights, N.Y.
- 10.1145/1145293 · doi:10.1145/1145293
- M. Minsky, R. Kurzweil, and S. Mann. 2013. The society of intelligent veillance. In 2013 IEEE International Symposium on Technology and Society (ISTAS): Social Implications of Wearable Computing and Augmediated Reality in Everyday Life. IEEE, USA, 13–17.
- Susanne van Mulken, Elisabeth André, and Jochen Müller. 1999. An Empirical Study on the Trustworthiness of Life-like Interface Agents. In Proceedings of the HCI International ’99 (the 8th International Conference on Human-Computer Interaction) on Human-Computer Interaction: Communication, Cooperation, and Application Design-Volume 2 - Volume 2. L. Erlbaum Associates Inc., Hillsdale, NJ, USA, 152–156.
- George Polya. 2004. How to solve it: A new aspect of mathematical method. Vol. 85. Princeton university press, NJ, USA.
- Lutz Prechelt, Ulrich Stärk, and Stephan Salinger. 2009. 7 Types of Cooperation Episodes in Side-by-Side Programming. In Proc. 21st Annual Workshop of the Psychology of Programming Interest Group (PPIG ’09). ACM, USA.
- Genaro Rebolledo-Mendez, Sara de Freitas, Jose Rafael Rojano-Caceres, and Alma Rosa García-Gaona. 2010. An Empirical Examination of the Relation Between Attention and Motivation in Computer-Based Education: a Modeling Approach. In Proceedings of the Twenty-Third International Florida Artificial Intelligence Research Society Conference, May 19-21, 2010, Daytona Beach, Florida, USA, Hans W. Guesgen and R. Charles Murray (Eds.). AAAI Press, USA, 74–79.
- T. Savage, M. Revelle, and D. Poshyvanyk. 2010. FLAT3: feature location and textual tracing tool. In 2010 ACM/IEEE 32nd International Conference on Software Engineering, Vol. 2. ACM, USA, 255–258.
- A. Seeger, J. Pfeiffer, and A. Heinzl. 2017. When Do We Need a Human? Anthropomorphic Design and Trustworthiness of Conversational Agents. In Proceedings of the SixteenthAnnual Pre-ICIS Workshop on HCI Research in MIS. ACM, USA, 1–5.
- R. Sharma, S. Gulia, and K. K. Biswas. 2014. Automated generation of activity and sequence diagrams from natural language requirements. In 9th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE). SCITEPRESS Digital Library, Portugal, 1–9.
- 10.5555/942801.943642 · doi:10.5555/942801.943642
- Laurie A. Williams. 2010. Pair Programming. John Wiley & Sons, USA. 311–322 pages.
Cited by
Ross, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584037 · Cited by 258
Investigating Explainability of Generative AI for Code through Scenario-based Design
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511119 · Cited by 219
“We envision that a conversational agent interface could be a natural fit for AI assistant for code or co-programming tools, as explored in recent work [42, 43] We also noticed that the explainability needs and user needs in general may differ for users with varied levels of programming skills.”
BackgroundJiang, E., Toh, E., Molina, A., Olson, K., Kayacik, C., Donsbach, A., Cai, C. J., & Terry, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501870 · Cited by 79
Better Together? An Evaluation of AI-Supported Code Translation
Weisz, J. D., Muller, M., Ross, S. I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., & Richards, J. T. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511157 · Cited by 69
Weisz, J. D., Kumar, S. V., Muller, M., Browne, K.-E., Goldberg, A., Heintze, K. E., & Bajpai, S. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706670 · Cited by 37
Metadata
- Authors
- Kuttal, S. K., Ong, B., Kwasny, K., & Robe, P.
- Affiliations
- University of Tulsa (US)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-20
- OpenAlex
- W3161416151
- Cited by
- 44
- References
- 151
- Citations by year
- 2026: 8 · 2025: 11 · 2024: 7 · 2023: 10 · 2022: 6 · 2021: 2
- FWCI
- 5.18
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- pair programming, chatbots, Wizard of Oz, code quality, developer productivity, self-efficacy, knowledge transfer, trust in AI, power relations, remote programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Trade-offs for Substituting a Human with an Agent in a Pair Programming Context: The Good, the Bad, and the Ugly. https://app.cochat.ai/papers/10.1145/3411764.3445659
- The paper
- Kuttal, S. K., Ong, B., Kwasny, K., & Robe, P. (2021). Trade-offs for Substituting a Human with an Agent in a Pair Programming Context: The Good, the Bad, and the Ugly. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445659
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.