Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 151 of the 151 references it cites were found in a registry or an index.

Trade-offs for Substituting a Human with an Agent in a Pair Programming Context: The Good, the Bad, and the Ugly

Kuttal, Sandeep Kaur, Ong, Bali, Kwasny, Kate, Robe, Peter · 2021 · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Abstract

Pair programming has a documented history of benefits, such as increased code quality, productivity, self-efficacy, knowledge transfer, and reduced gender gap. Research uncovered problems with pair programming related to scheduling, collocating, role imbalance, and power dynamics. We investigated the trade-offs of substituting a human with an agent to simultaneously provide benefits and alleviate obstacles in pair programming. We conducted gender-balanced studies with human-human pairs in a remote lab with 18 programmers and Wizard-of-Oz studies with 14 programmers, then analyzed results quantitatively and qualitatively. Our comparative analysis of the two studies showed no significant differences in productivity, code quality, and self-efficacy. Further, agents facilitated knowledge transfer; however, unlike humans, agents were unable to provide logical explanations or discussions. Human partners trusted and showed humility towards agents. Our results demonstrate that agents can act as effective pair programming partners and open the way towards new research on conversational agents for programming.

Overview

Integrity signals

Registry

Crossref · 10.1145/3411764.3445659

Cites

151 of 151 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 151 references

Research works printed but not linked: 33

  1. Michael Armstrong. 2012. Armstrong’s handbook of reward management practice: Improving performance through reward (12 ed.). Kogan Page Publishers, London.
  2. 10.4135/9781446221129.n6 · DOI · Paper page
  3. 10.1093/iwc/iwv046 · DOI · Paper page
  4. Ramón Burri. 2018. Improving user trust towards conversational chatbot interfaces with voice output. Master’s thesis. KTH School of Electrical Engineering and Computer Science (EECS).
  5. Lan Cao, Kannan Mohan, Peng Xu, and Balasubramaniam Ramesh. 2004. How Extreme Does Extreme Programming Have to Be? Adapting XP Practices to Large-Scale Projects. In Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS’04) - Track 3 - Volume 3(HICSS ’04). IEEE Computer Society, USA, 30083.3.
  6. Christopher P Cerasoli, Jessica M Nicklin, and Michael T Ford. 2014. Intrinsic motivation and extrinsic incentives jointly predict performance: A 40-year meta-analysis.Psychological bulletin 140, 4 (2014), 980.
  7. Pedro Costa and Luísa Ribas. 2019. AI becomes her: Discussing gender and artificial intelligence.Technoetic Arts: A Journal of Speculative Research 17, 1/2(2019), 171 – 193.
  8. Sally Jo Cunningham, Annika Hinze, and David M. Nichols. 2016. Supporting Gender-Neutral Digital Library Creation: A Case Study Using the GenderMag Toolkit. In Digital Libraries: Knowledge, Information, and Data in an Open Access Society, Atsuyuki Morishima, Andreas Rauber, and Chern Li Liew (Eds.). Springer International Publishing, Cham, 45–50.
  9. Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz studies—why and how. Knowledge-based systems 6, 4 (1993), 258–266.
  10. M. Day, M. R. Penumala, and J. Gonzalez-Sanchez. 2019. Annete: An Intelligent Tutoring Companion Embedded into the Eclipse IDE. In 2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI). IEEE, New York, US, 71–80.
  11. Berland Edelman and Inc. 2010. Creativity and education: Why it matters. Adobe. Retrieved September 18th, 2019 from http://www.adobe.com/aboutadobe/pressroom/pdfs/Adobe_Creativity_and_Education_Why_It_Matters_study.pdf
  12. S.T. Fiske E.H.P.P.S.T. Fiske and S.E. Taylor. 1991. Social Cognition. McGraw-Hill New York City USA. https://books.google.com/books?id=6Uq3QgAACAAJ
  13. H. Gallis, Erik Arisholm, and Tore Dybå. 2002. A Transition From Partner Programming to Pair Programming - an Industrial Case Study. In Workshop ”Pair Programming Installed” in 17th Annual ACM Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), Position paper. ACM, Seattle USA, –.
  14. H. Gallis, E. Arisholm, and T. Dyba. 2003. An initial framework for research on pair programming. In 2003 International Symposium on Empirical Software Engineering. ACM, NY, USA, 132–142.
  15. Dai Hasegawa, Justine Cassell, and Kenji Araki. 2010. The Role of Embodiment and Perspective in Direction-Giving Systems. In Proceedings of AAAI Fall Workshop on Dialog with Robots. AAAI PRESS, USA.
  16. 10.1177/0018720814547570PMID: · doi:10.1177/0018720814547570pmid
  17. D. James and S. Clarke. 1998. Women men and interruptions: A critical review. In D. Tannen (Ed.) Oxford studies in sociolinguistics. Gender and conversational interaction. Oxford University Press UK 231–280.
  18. 10.1016/j.jvlc.2017.04.005Special · doi:10.1016/j.jvlc.2017.04.005special
  19. S. Kaur Kuttal, K. Gerstner, and A. Bejarano. 2019. Remote Pair Programming in Online CS Education: Investigating through a Gender Lens. In 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, USA, 75–85.
  20. Danielle L. Jones and Scott D. Fleming. 2013. What use is a backseat driver? A qualitative investigation of pair programming. In Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC. IEEE, USA, 103–110.
  21. 10.1145/3195570.3195580 · DOI · Paper page
  22. Clayton Lewis. 1982. Using the ”thinking-aloud” method in cognitive interface design. IBM T.J. Watson Research Center, Yorktown Heights, N.Y.
  23. 10.1145/1145293 · doi:10.1145/1145293
  24. M. Minsky, R. Kurzweil, and S. Mann. 2013. The society of intelligent veillance. In 2013 IEEE International Symposium on Technology and Society (ISTAS): Social Implications of Wearable Computing and Augmediated Reality in Everyday Life. IEEE, USA, 13–17.
  25. Susanne van Mulken, Elisabeth André, and Jochen Müller. 1999. An Empirical Study on the Trustworthiness of Life-like Interface Agents. In Proceedings of the HCI International ’99 (the 8th International Conference on Human-Computer Interaction) on Human-Computer Interaction: Communication, Cooperation, and Application Design-Volume 2 - Volume 2. L. Erlbaum Associates Inc., Hillsdale, NJ, USA, 152–156.
  26. George Polya. 2004. How to solve it: A new aspect of mathematical method. Vol. 85. Princeton university press, NJ, USA.
  27. Lutz Prechelt, Ulrich Stärk, and Stephan Salinger. 2009. 7 Types of Cooperation Episodes in Side-by-Side Programming. In Proc. 21st Annual Workshop of the Psychology of Programming Interest Group (PPIG ’09). ACM, USA.
  28. Genaro Rebolledo-Mendez, Sara de Freitas, Jose Rafael Rojano-Caceres, and Alma Rosa García-Gaona. 2010. An Empirical Examination of the Relation Between Attention and Motivation in Computer-Based Education: a Modeling Approach. In Proceedings of the Twenty-Third International Florida Artificial Intelligence Research Society Conference, May 19-21, 2010, Daytona Beach, Florida, USA, Hans W. Guesgen and R. Charles Murray (Eds.). AAAI Press, USA, 74–79.
  29. T. Savage, M. Revelle, and D. Poshyvanyk. 2010. FLAT3: feature location and textual tracing tool. In 2010 ACM/IEEE 32nd International Conference on Software Engineering, Vol. 2. ACM, USA, 255–258.
  30. A. Seeger, J. Pfeiffer, and A. Heinzl. 2017. When Do We Need a Human? Anthropomorphic Design and Trustworthiness of Conversational Agents. In Proceedings of the SixteenthAnnual Pre-ICIS Workshop on HCI Research in MIS. ACM, USA, 1–5.
  31. R. Sharma, S. Gulia, and K. K. Biswas. 2014. Automated generation of activity and sequence diagrams from natural language requirements. In 9th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE). SCITEPRESS Digital Library, Portugal, 1–9.
  32. 10.5555/942801.943642 · doi:10.5555/942801.943642
  33. Laurie A. Williams. 2010. Pair Programming. John Wiley & Sons, USA. 311–322 pages.

Cited by

See all 44

Metadata

Authors
Kuttal, S. K., Ong, B., Kwasny, K., & Robe, P.
Affiliations
University of Tulsa (US)
Venue
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2021-05-06
Pages
1-20
OpenAlex
W3161416151
Cited by
44
References
151
Citations by year
2026: 8 · 2025: 11 · 2024: 7 · 2023: 10 · 2022: 6 · 2021: 2
FWCI
5.18
Type
conference paper
Language
en
Open access
closed
Locations
CHI Conference on Human Factors in Computing Systems (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
pair programming, chatbots, Wizard of Oz, code quality, developer productivity, self-efficacy, knowledge transfer, trust in AI, power relations, remote programming

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Trade-offs for Substituting a Human with an Agent in a Pair Programming Context: The Good, the Bad, and the Ugly. https://app.cochat.ai/papers/10.1145/3411764.3445659
The paper
Kuttal, S. K., Ong, B., Kwasny, K., & Robe, P. (2021). Trade-offs for Substituting a Human with an Agent in a Pair Programming Context: The Good, the Bad, and the Ugly. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445659

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.