The DOI matches its Crossref record; 151 of the 151 references it cites were found in a registry or an index.
Trade-offs for Substituting a Human with an Agent in a Pair Programming Context: The Good, the Bad, and the Ugly
Abstract
Pair programming has a documented history of benefits, such as increased code quality, productivity, self-efficacy, knowledge transfer, and reduced gender gap. Research uncovered problems with pair programming related to scheduling, collocating, role imbalance, and power dynamics. We investigated the trade-offs of substituting a human with an agent to simultaneously provide benefits and alleviate obstacles in pair programming. We conducted gender-balanced studies with human-human pairs in a remote lab with 18 programmers and Wizard-of-Oz studies with 14 programmers, then analyzed results quantitatively and qualitatively. Our comparative analysis of the two studies showed no significant differences in productivity, code quality, and self-efficacy. Further, agents facilitated knowledge transfer; however, unlike humans, agents were unable to provide logical explanations or discussions. Human partners trusted and showed humility towards agents. Our results demonstrate that agents can act as effective pair programming partners and open the way towards new research on conversational agents for programming.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We envision that a conversational agent interface could be a natural fit for AI assistant for code or co-programming tools, as explored in recent work [42,43]”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [43]
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
WEST, C., & ZIMMERMAN, D. H. (1987) · Gender & Society · doi:10.1177/0891243287001002002 · Cited by 8,954
Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory
Wuetherick, B. (2010) · Canadian Journal of University Continuing Education · doi:10.21225/d5g01t · Cited by 8,311
Computer Self-Efficacy: Development of a Measure and Initial Test
Compeau, D. R., & Higgins, C. A. (1995) · MIS Quarterly · doi:10.2307/249688 · Cited by 6,553
Self-Determination Theory in Work Organizations: The State of a Science
Deci, E. L., Olafsen, A. H., & Ryan, R. M. (2017) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-032516-113108 · Cited by 3,199
Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817
Change by design : how design thinking transforms organizations and inspires innovation
Brown, T., & Kātz, B. M. (2009) · Cited by 2,588
Étude comparative de la distribution florale dans une portion des Alpes et du Jura
Jaccard, P. (1901) · E-Periodica · doi:10.5169/seals-266450 · Cited by 2,353
Amabile, T. M., & Pratt, M. G. (2016) · Research in Organizational Behavior · doi:10.1016/j.riob.2016.10.001 · Cited by 1,863 · cited 2×
“Adapting activity based on partner’s ability [97] Making rapport by greeting [84] and motivational dialogues [5, 31, 45, 53]”
BackgroundPreliminary guidelines for empirical research in software engineering
Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., El Emam, K., & Rosenberg, J. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1027796 · Cited by 1,510
Qualitative methods in empirical studies of software engineering
Seaman, C. B. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799955 · Cited by 1,493
The Discovery of Grounded Theory: Strategies for Qualitative Research
Wakeford, J. (1969) · Sociology · doi:10.1177/003803856900300233 · Cited by 1,460
Strong Evidence for Gender Differences in Risk Taking
Charness, G., & Gneezy, U. (2012) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2011.06.007 · Cited by 1,357
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
Computer-Mediated Collaborative Learning: An Empirical Evaluation
Alavi, M. (1994) · MIS Quarterly · doi:10.2307/249763 · Cited by 1,148
Intelligence, creativity, and their educational implications
Guilford, J. P. (1968) · Medical Entomology and Zoology · Cited by 1,021
Guide to Advanced Empirical Software Engineering
(2008) · Springer eBooks · doi:10.1007/978-1-84800-044-5 · Cited by 1,003
Brown, T., & Katz, B. (2011) · Journal of Product Innovation Management · doi:10.1111/j.1540-5885.2011.00806.x · Cited by 937
Strengthening the case for pair programming
Williams, L., Kessler, R. R., Cunningham, W., & Jeffries, R. (2000) · IEEE Software · doi:10.1109/52.854064 · Cited by 780
Wizard of Oz Studies in HRI: A Systematic Review and New Reporting Guidelines
Riek, L. (2012) · Journal of Human-Robot Interaction · doi:10.5898/jhri.1.1.riek · Cited by 779
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670
The impact of animated interface agents: a review of empirical research
DEHN, D. M., & VAN MULKEN, S. (2000) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0325 · Cited by 567
Explorations in engagement for humans and robots
Sidner, C. L., Lee, C., Kidd, C. D., Lesh, N., & Rich, C. (2005) · Artificial Intelligence · doi:10.1016/j.artint.2005.03.005 · Cited by 558
Williams, L. A., & Kessler, R. R. (2007) · CERN Document Server (European Organization for Nuclear Research) · Cited by 550
Pronoun Use Reflects Standings in Social Hierarchies
Kacewicz, E., Pennebaker, J. W., Davis, M., Jeon, M., & Graesser, A. C. (2014) · Journal of Language and Social Psychology · doi:10.1177/0261927x13502654 · Cited by 541
If not now, when?: the effects of interruption at different moments within task execution
Adamczyk, P. D., & Bailey, B. P. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985727 · Cited by 531 · cited 1×
“” Interruptions are associated with interpersonal dominance [1, 120, 192], friendliness [123], engagement [59], and involvement in the interaction [151].”
BackgroundHuman–Agent Teaming for Multirobot Control: A Review of Human Factors Issues
Chen, J. Y. C., & Barnes, M. J. (2014) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2013.2293535 · Cited by 522
Relational agents: a model and implementation of building user trust
Bickmore, T., & Cassell, J. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365304 · Cited by 479
Ali, S., Briand, L. C., Hemmati, H., & Panesar-Walawege, R. K. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2009.52 · Cited by 452
The costs and benefits of pair programming
Cockburn, A., & Williams, L. A. (2001) · Cited by 425
Creating Rapport with Virtual Agents
Gratch, J., Wang, N., Gerten, J., Fast, E., & Duffy, R. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74997-4_12 · Cited by 419
Sproull, L., Subramani, M., Kiesler, S., Walker, J., & Waters, K. (1996) · Human-Computer Interaction · doi:10.1207/s15327051hci1102_1 · Cited by 399
Collective Intelligence and Group Performance
Woolley, A. W., Aggarwal, I., & Malone, T. W. (2015) · Current Directions in Psychological Science · doi:10.1177/0963721415599543 · Cited by 382
World class learners: educating creative and entrepreneurial students
(2013) · Choice Reviews Online · doi:10.5860/choice.50-5135 · Cited by 380
Evaluating and Informing the Design of Chatbots
Jain, M., Kumar, P., Kota, R., & Patel, S. N. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196735 · Cited by 376
The case for collaborative programming
Nosek, J. T. (1998) · Communications of the ACM · doi:10.1145/272287.272333 · Cited by 375
Fischer, C., Malycha, C. P., & Schafmann, E. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00137 · Cited by 369
Designing the whyline: a debugging interface for asking questions about program behavior
Ko, A. J., & Myers, B. A. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985712 · Cited by 361
The effects of pair-programming on performance in an introductory programming course
McDowell, C., Werner, L., Bullock, H., & Fernald, J. (2002) · Proceedings of the 33rd SIGCSE technical symposium on Computer science education · doi:10.1145/563340.563353 · Cited by 352
Peopleware: Productive Projects and Teams
JONES, L. (1990) · R&D Management · doi:10.1111/j.1467-9310.1990.tb00681.x · Cited by 349
Machine Heuristic: When We Trust Computers More than Humans with Our Personal Information
Sundar, S. S., & Kim, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300768 · Cited by 347
Debugging reinvented: asking and answering why and why not questions about program behavior
Ko, A. J., & Myers, B. A. (2008) · Proceedings of the 30th international conference on Software engineering · doi:10.1145/1368088.1368130 · Cited by 296
All I really need to know about pair programming I learned in kindergarten
Williams, L. A., & Kessler, R. R. (2000) · Communications of the ACM · doi:10.1145/332833.332848 · Cited by 293
Gender bias in artificial intelligence: the need for diversity and gender theory in machine learning
Leavy, S. (2018) · Proceedings of the 1st International Workshop on Gender Equality in Software Engineering · doi:10.1145/3195570.3195580 · Cited by 284
Duffy, M. C., & Azevedo, R. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.05.041 · Cited by 283
Perceptual user interfaces: multimodal interfaces that process what comes naturally
Oviatt, S., & Cohen, P. (2000) · Communications of the ACM · doi:10.1145/330534.330538 · Cited by 278
Intelligent Tutoring Systems: A Comprehensive Historical Survey with Recent Developments
Alkhatlan, A., & Kalita, J. K. (2019) · International Journal of Computer Applications · doi:10.5120/ijca2019918451 · Cited by 273
Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise
Arisholm, E., Gallis, H., Dyba, T., & Sjoberg, D. I. K. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.17 · Cited by 272
In Support of Pair Programming in the Introductory Computer Science Course
Williams, L., Wiebe, E., Yang, K., Ferzli, M., & Miller, C. (2002) · Computer Science Education · doi:10.1076/csed.12.3.197.8618 · Cited by 270
Yee, N., Bailenson, J. N., & Rickertsen, K. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240626 · Cited by 242
The Persona Effect: How Substantial Is It?
van Mulken, S., André, E., & Müller, J. (1998) · People and Computers XIII · doi:10.1007/978-1-4471-3605-7_4 · Cited by 235
Pair-programming helps female computer science students
Werner, L. L., Hanks, B., & McDowell, C. (2004) · Journal on Educational Resources in Computing · doi:10.1145/1060071.1060075 · Cited by 235
The effect of social presence and chatbot errors on trust
Toader, D.-C., Boca, G., Toader, R., Măcelaru, M., Toader, C., Ighian, D., & Rădulescu, A. T. (2019) · Sustainability · doi:10.3390/su12010256 · Cited by 233 · cited 2×
“The programmer could also choose the avatar’s gender [19, 35, 87, 165].”
BackgroundModeling and understanding students' off-task behavior in intelligent tutoring systems
Baker, R. S. J. D. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240785 · Cited by 230
Resilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns
Ashktorab, Z., Jain, M., Liao, Q. V., & Weisz, J. D. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300484 · Cited by 224
Feature location via information retrieval based filtering of a single scenario execution trace
Liu, D., Marcus, A., Poshyvanyk, D., & Rajlich, V. (2007) · Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering · doi:10.1145/1321631.1321667 · Cited by 214
The impact of pair programming on student performance, perception and persistence
McDowell, C., Werner, L., Bullock, H. E., & Fernald, J. L. (2003) · doi:10.5555/776816.776899 · Cited by 211
Design patterns for sociality in human-robot interaction
Kahn, P. H., Freier, N. G., Kanda, T., Ishiguro, H., Ruckert, J. H., Severson, R. L., & Kane, S. K. (2008) · Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction · doi:10.1145/1349822.1349836 · Cited by 174
Revisiting Bodies and Pleasures
Butler, J. (1999) · Theory Culture & Society · doi:10.1177/02632769922050520 · Cited by 172
Gender affordances of conversational agents
Brahnam, S., & Angeli, A. D. (2012) · Interacting with Computers · doi:10.1016/j.intcom.2012.05.001 · Cited by 171
SupportingTrust in Autonomous Driving
Häuslschmid, R., von Bülow, M., Pfleging, B., & Butz, A. (2017) · Proceedings of the 22nd International Conference on Intelligent User Interfaces · doi:10.1145/3025171.3025198 · Cited by 170
Handbook of Human Computer Interaction
(2021) · Springer eBooks · doi:10.1007/978-3-319-27648-9 · Cited by 166
Intrusive or co-operative? A cross-cultural study of interruption
Murata, K. (1994) · Journal of Pragmatics · doi:10.1016/0378-2166(94)90011-6 · Cited by 165
Retaining nearly one-third more majors with a trio of instructional best practices in CS1
Porter, L., & Simon, B. (2013) · Proceeding of the 44th ACM technical symposium on Computer science education · doi:10.1145/2445196.2445248 · Cited by 164
SWIM: synthesizing what i mean: code search and idiomatic snippet synthesis
Raghothaman, M., Wei, Y., & Hamadi, Y. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884808 · Cited by 163
The Social Dynamics of Pair Programming
Chong, J., & Hurlbutt, T. (2007) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2007.87 · Cited by 161
Face Value? Exploring the Effects of Embodiment for a Group Facilitation Agent
Shamekhi, A., Liao, Q., Wang, D., Bellamy, R., & Erickson, T. D. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173965 · Cited by 161
Personification of the Amazon Alexa: BFF or a Mindless Companion
Lopatovska, I., & Williams, H. (2018) · Proceedings of the 2018 Conference on Human Information Interaction&Retrieval - CHIIR '18 · doi:10.1145/3176349.3176868 · Cited by 160
Schyns, P. G., Petro, L. S., & Smith, M. L. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005625 · Cited by 153
Spotting working code examples
Keivanloo, I., Rilling, J., & Zou, Y. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568292 · Cited by 147
Gender-Inclusiveness Personas vs. Stereotyping: Can We Have it Both Ways?
Hill, C. G., Haag, M., Oleson, A., Mendez, C., Marsden, N., Sarma, A., & Burnett, M. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025609 · Cited by 144
Finding Gender-Inclusiveness Software Issues with GenderMag: A Field Investigation
Burnett, M., Peters, A. N., Hill, C., & Elarief, N. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858274 · Cited by 140
Are Two Heads Better than One? On the Effectiveness of Pair Programming
Dybå, T., Arisholm, E., Sjøberg, D. I. K., Hannay, J. E., & Shull, F. J. (2007) · IEEE Software · doi:10.1109/ms.2007.158 · Cited by 138
Weitz, K., Schiller, D., Schlagowski, R., Huber, T., & André, E. (2019) · Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents · doi:10.1145/3308532.3329441 · Cited by 136
Situated facial displays: towards social interaction
Takeuchi, A., & Naito, T. (1995) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/223904.223965 · Cited by 134
Larkin, J. H., & Chabay, R. W. (1992) · Medical Entomology and Zoology · Cited by 125
Improving the CS1 experience with pair programming
Nagappan, N., Williams, L. A., Ferzli, M., et al. (2003) · ACM SIGCSE Bulletin · doi:10.1145/792548.612006 · Cited by 121
Open source barriers to entry, revisited: a sociotechnical perspective
Mendez, C., Padala, H. S., Steine-Hanson, Z., Hilderbrand, C., Horvath, A., Hill, C., Simpson, L., Patil, N., Sarma, A., & Burnett, M. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180241 · Cited by 118
FarmChat: A Conversational Agent to Answer Farmer Queries
Jain, M., Kumar, P., Bhansali, I., Liao, Q. V., Truong, K., & Patel, S. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3287048 · Cited by 117
How to Solve Problems: Elements of a Theory of Problems and Problem Solving.
Rahman, N. A., & Wickelgren, W. A. (1975) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2345270 · Cited by 116
Creating Innovators: The Making of Young People Who Will Change the World
(2013) · Management Decision · doi:10.1108/00251741311301920 · Cited by 116
Teaching Creativity in Engineering
Schönwetter, D. J. (2004) · Cited by 116
F a C o Y: a code-to-code search engine
Kim, K., Kim, D., Bissyandé, T. F., Choi, E., Li, L., Klein, J., & Traon, Y. L. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180187 · Cited by 116
From Gender Biases to Gender-Inclusive Design: An Empirical Investigation
Vorvoreanu, M., Zhang, L., Huang, Y.-H., Hilderbrand, C., Steine-Hanson, Z., & Burnett, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300283 · Cited by 113
Exploring the Efficacy of Distributed Pair Programming
Baheti, P., Gehringer, E. F., & Stotts, D. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45672-4_20 · Cited by 112
The effects of "pair-pressure" and "pair-learning" on software engineering education
Williams, L. A., & Kessler, R. R. (2005) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/csee.2000.827023 · Cited by 112
Harnessing curiosity to increase correctness in end-user programming
Wilson, A., Burnett, M., Beckwith, L., Granatir, O., Casburn, L., Cook, C. R., Durham, M., & Rothermel, G. (2003) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642665 · Cited by 109
Exploring the Pair Programming Process: Characteristics of Effective Collaboration
Rodríguez, F. J., Price, K. M., & Boyer, K. E. (2017) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3017680.3017748 · Cited by 109
How Equity and Inequity Can Emerge in Pair Programming
Lewis, C. M., & Shah, N. (2015) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2787622.2787716 · Cited by 108
Ask-Elle: an Adaptable Programming Tutor for Haskell Giving Automated Feedback
Gerdes, A., Heeren, B. J., Jeuring, J. T., & van Binsbergen, L. T. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0080-x · Cited by 102
On how the brain decodes vocal cues about speaker confidence
Jiang, X., & Pell, M. D. (2015) · Cortex · doi:10.1016/j.cortex.2015.02.002 · Cited by 91
Definitions of engagement in human-agent interaction
Glas, N., & Pélachaud, C. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344688 · Cited by 91
Learning to rank code examples for code search engines
Niu, H., Keivanloo, I., & Zou, Y. (2016) · Empirical Software Engineering · doi:10.1007/s10664-015-9421-5 · Cited by 89
An initial framework for research on pair programming
Gallis, H., Arisholm, E., & Dyba, T. (2004) · International Symposium on Empirical Software Engineering · doi:10.1109/isese.2003.1237972 · Cited by 82
Motivation Diagnosis in Intelligent Tutoring Systems
de Vicente, Á., & Pain, H. (1998) · Lecture notes in computer science · doi:10.1007/3-540-68716-5_14 · Cited by 78
Impact of interruption style on end-user debugging
Robertson, T. J., Prabhakararao, S., Burnett, M., Cook, C. R., Ruthruff, J. R., Beckwith, L., & Phalgune, A. (2004) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985729 · Cited by 76
Karan, A., Rosenthal, R., & Robbins, M. L. (2018) · Journal of Social and Personal Relationships · doi:10.1177/0265407518795336 · Cited by 75
Eleven Guidelines for Implementing Pair Programming in the Classroom
Williams, L. A., McCrickard, D. S., Layman, L. M., & Hussein, K. (2008) · Agile Conference · doi:10.1109/agile.2008.12 · Cited by 73
Peer interaction effectively, yet infrequently, enables programmers to discover new tools
Murphy-Hill, E. R., & Murphy, G. C. (2011) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1958824.1958888 · Cited by 68
HCI and Usability for Medicine and Health Care
Holzinger, A. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-76805-0 · Cited by 67
Influence of Dyadic Interaction on Creative Functioning
Torrance, E. P. (1970) · Psychological Reports · doi:10.2466/pr0.1970.26.2.391 · Cited by 66
Production programming in the classroom
Allen, E., Cartwright, R., & Reis, C. (2003) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/611892.611940 · Cited by 66
On understanding compatibility of student pair programmers
Katira, N., Williams, L. A., Wiebe, E., et al. (2004) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/971300.971307 · Cited by 65
Interruption and Influence in Discussion Groups
Ng, S. H., Brooke, M., & Dunne, M. (1995) · Journal of Language and Social Psychology · doi:10.1177/0261927x950144003 · Cited by 58
Motivating the Learner: An Empirical Evaluation
Rebolledo‐Mendez, G., du Boulay, B., & Luckin, R. (2006) · Lecture notes in computer science · doi:10.1007/11774303_54 · Cited by 58
Empirical evaluation of distributed pair programming
Hanks, B. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.10.003 · Cited by 54
Cognitive walkthrough of a learning management system with gendered personas
Shekhar, A., & Marsden, N. (2018) · Conference on Gender & IT (GenderIT) · doi:10.1145/3196839.3196869 · Cited by 54
Thematic Analysis of Students' Reflections on Pair Programming in CS1
Celepkolu, M., & Boyer, K. E. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159516 · Cited by 53
The Impact of a Peer-Learning Agent Based on Pair Programming in a Programming Course
Han, K., Lee, E., & Lee, Y. (2009) · IEEE Transactions on Education · doi:10.1109/te.2009.2019121 · Cited by 52
Meiliana, M., Septian, I., Alianto, R. S., et al. (2017) · Procedia Computer Science · doi:10.1016/j.procs.2017.10.029 · Cited by 52
Women in engineering: Addressing the gender gap, exploring trust and our unconscious bias
Strachan, R., Peixoto, A., Emembolu, I., & Restivo, M. T. (2018) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon.2018.8363497 · Cited by 52
Ying, K. M., Pezzullo, L. G., Ahmed, M., Crompton, K., Blanchard, J. J., & Boyer, K. E. (2019) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3287324.3287380 · Cited by 49
Anterior Temporal Lobe Tracks the Formation of Prejudice
Spiers, H. J., Love, B. C., Pelley, M. E. L., Gibb, C. E., & Murphy, R. A. (2016) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01056 · Cited by 44
Encouraging student reflection and articulation using a learning companion
Goodman, B., Soller, A., Linton, F., Gaimari, R. D., & Mitre, T. (1998) · Cited by 42
Collaborative learning and anxiety: a phenomenographic study of collaborative learning activities
Falkner, K. E. K., Falkner, N., & Vivian, R. (2013) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2445196.2445268 · Cited by 40
Analyzing Work Productivity and Program Quality in Collaborative Programming
Duque, R., & Bravo, C. (2008) · International Conference on Software Engineering Advances · doi:10.1109/icsea.2008.82 · Cited by 39
On knowledge transfer skill in pair programming
Zieris, F., & Prechelt, L. (2014) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/2652524.2652529 · Cited by 35
Explaining program code: giving students the answer helps - but only just
Simon, & Snowdon, S. (2011) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2016911.2016931 · Cited by 34
Distributed Pair Programming: An Empirical Study
Hanks, B. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-27777-4_9 · Cited by 33
Psychology as a mother of invention
Landauer, T. K. (1986) · ACM SIGCHI Bulletin · doi:10.1145/1165387.275653 · Cited by 33
Promiscuous pairing and beginner's mind: embrace inexperience [agile programming]
Belshee, A. (2006) · doi:10.1109/adc.2005.37 · Cited by 30
Tsuei, M. (2017) · Interactive Learning Environments · doi:10.1080/10494820.2016.1276078 · Cited by 29
A principled evaluation for a principled idea garden
Jernigan, W., Horvath, A., Lee, M., Burnett, M., Cuilty, T., Kuttal, S., Peters, A., Kwan, I., Bahmani, F., & Ko, A. (2015) · 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2015.7357222 · Cited by 28
Self-citationWargnier, P., Carletti, G., Laurent-Corniquet, Y., Benveniste, S., Jouvelot, P., & Rigaud, A.-S. (2016) · 2016 IEEE International Conference on Serious Games and Applications for Health (SeGAH) · doi:10.1109/segah.2016.7586282 · Cited by 28 · cited 1×
“unique qualities of human-agent interactions [41], and study user interactions with conversational agents [17, 18, 173].”
BackgroundThe Importance of Producing Shared Code Through Pair Programming
Celepkolu, M., & Boyer, K. E. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159506 · Cited by 28
Observations of Pair Programming
Ruvalcaba, O., Werner, L., & Denner, J. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844558 · Cited by 27
Kuttal, S. K., Myers, J., Gurka, S., Magar, D., Piorkowski, D., & Bellamy, R. (2020) · 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc50065.2020.9127276 · Cited by 27
Self-citationAspect-oriented model-driven skeleton code generation: A graph-based transformation approach
Bennett, J., Cooper, K., & Dai, L. (2009) · Science of Computer Programming · doi:10.1016/j.scico.2009.05.005 · Cited by 26
A Large-Scale Experimental Study of Gender and Pair Composition in Pair Programming
Jarratt, L., Bowman, N. A., Culver, K. C., & Segre, A. M. (2019) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3304221.3319782 · Cited by 23
Evaluating Pair-Programming for Non-Computer Science Major Students
O'Donnell, C., Buckley, J., Mahdi, A. E., Nelson, J., & English, M. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677289 · Cited by 21
Explicit feedback from users attenuates memory biases in human-system dialogue
Knutsen, D., Bigot, L. L., & Ros, C. (2016) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2016.09.004 · Cited by 20
The Effect of Interruptions and Dyad Gender Combination on Perceptions of Interpersonal Dominance
Youngquist, J. (2009) · Communication Studies · doi:10.1080/10510970902834874 · Cited by 19
13th International Workshop on Program Comprehension (IWPC'05)
(2005) · doi:10.1109/ieeeconf10439.2005 · Cited by 19
Evaluating Gender Significance within a Pair Programming Context
Choi, K. S. (2013) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2013.209 · Cited by 16
The influence of reference acceptance and reuse on conversational memory traces.
Knutsen, D., & Bigot, L. L. (2014) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0000036 · Cited by 16
Motivation system and human model for intelligent tutoring
Matsubara, Y., & Nagamachi, M. (1996) · Lecture notes in computer science · doi:10.1007/3-540-61327-7_110 · Cited by 16
General principles for a Generalized Idea Garden
Jernigan, W., Horvath, A., Lee, M. J., Burnett, M., Cuilty, T., Kuttal, S. K., Peters, A. N., Kwan, I., Bahmani, F., Ko, A. J., et al. (2017) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2017.04.005 · Cited by 16
Self-citationAn Empirical Study on Factors related to Distributed Pair Programming
Tsompanoudi, D., Satratzemi, M., Xinogalos, S., & Karamitopoulos, L. (2019) · International Journal of Engineering Pedagogy (iJEP) · doi:10.3991/ijep.v9i2.9947 · Cited by 16
Hybrid Pair Programming - A Promising Alternative to Standard Pair Programming
Yuan, H., & Cao, Y. (2019) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3287324.3287352 · Cited by 15
Wizard of Oz Experiments and Companion Dialogues
Bradley, J., Benyon, D., Mival, O., & Webb, N. (2010) · Electronic workshops in computing · doi:10.14236/ewic/hci2010.16 · Cited by 14
Relationships between User Demographics and User Trust in an Autonomous Agent
Hillesheim, A. J., Rusnock, C. F., Bindewald, J. M., & Miller, M. E. (2017) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1541931213601560 · Cited by 14
Social effects of pair programming mitigate impact of bounded rationality
Li, Z., & Kraemer, E. (2014) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2538862.2538968 · Cited by 13
Explaining pair programming session dynamics from knowledge gaps
Zieris, F., & Prechelt, L. (2020) · doi:10.1145/3377811.3380925 · Cited by 13
Assessing the Impact of Virtual Human's Appearance on Users' Trust Levels
Zalake, M. S., Woodward, J., Kapoor, A., et al. (2018) · doi:10.1145/3267851.3267863 · Cited by 12
Automatic Generation of Test Cases for Agile using Natural Language Processing
Rane, P. P. (2017) · VTechWorks (Virginia Tech) · Cited by 11
Generating References in Naturalistic Face‐to‐Face and Phone‐Mediated Dialog Settings
Knutsen, D., Ros, C., & Bigot, L. L. (2016) · Topics in Cognitive Science · doi:10.1111/tops.12218 · Cited by 10
Nezu, C. M., Nezu, A. M., & Ricelli, S. E. (2015) · Sage eBooks · doi:10.4135/9781483385822.n4 · Cited by 10
Analyzing graph-based algorithms employed to generate test cases from finite state machines
Mariano, M. M., de Souza, É. F., Endo, A. T., et al. (2019) · IEEE Latin American Test Symposium (LATS) · doi:10.1109/latw.2019.8704603 · Cited by 10
Toward Data-Driven Example Feedback for Novice Programming.
Zhi, R., Marwan, S., Dong, Y., Lytle, N., Price, T., & Barnes, T. M. (2019) · Educational Data Mining · Cited by 10
Saltz, J., & Shamshurin, I. (2019) · International Journal of Higher Education and Sustainability · doi:10.1504/ijhes.2019.103360 · Cited by 7
Research works printed but not linked: 33
- Michael Armstrong. 2012. Armstrong’s handbook of reward management practice: Improving performance through reward (12 ed.). Kogan Page Publishers, London.
- 10.4135/9781446221129.n6 · DOI · Paper page
- 10.1093/iwc/iwv046 · DOI · Paper page
- Ramón Burri. 2018. Improving user trust towards conversational chatbot interfaces with voice output. Master’s thesis. KTH School of Electrical Engineering and Computer Science (EECS).
- Lan Cao, Kannan Mohan, Peng Xu, and Balasubramaniam Ramesh. 2004. How Extreme Does Extreme Programming Have to Be? Adapting XP Practices to Large-Scale Projects. In Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS’04) - Track 3 - Volume 3(HICSS ’04). IEEE Computer Society, USA, 30083.3.
- Christopher P Cerasoli, Jessica M Nicklin, and Michael T Ford. 2014. Intrinsic motivation and extrinsic incentives jointly predict performance: A 40-year meta-analysis.Psychological bulletin 140, 4 (2014), 980.
- Pedro Costa and Luísa Ribas. 2019. AI becomes her: Discussing gender and artificial intelligence.Technoetic Arts: A Journal of Speculative Research 17, 1/2(2019), 171 – 193.
- Sally Jo Cunningham, Annika Hinze, and David M. Nichols. 2016. Supporting Gender-Neutral Digital Library Creation: A Case Study Using the GenderMag Toolkit. In Digital Libraries: Knowledge, Information, and Data in an Open Access Society, Atsuyuki Morishima, Andreas Rauber, and Chern Li Liew (Eds.). Springer International Publishing, Cham, 45–50.
- Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz studies—why and how. Knowledge-based systems 6, 4 (1993), 258–266.
- M. Day, M. R. Penumala, and J. Gonzalez-Sanchez. 2019. Annete: An Intelligent Tutoring Companion Embedded into the Eclipse IDE. In 2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI). IEEE, New York, US, 71–80.
- Berland Edelman and Inc. 2010. Creativity and education: Why it matters. Adobe. Retrieved September 18th, 2019 from http://www.adobe.com/aboutadobe/pressroom/pdfs/Adobe_Creativity_and_Education_Why_It_Matters_study.pdf
- S.T. Fiske E.H.P.P.S.T. Fiske and S.E. Taylor. 1991. Social Cognition. McGraw-Hill New York City USA. https://books.google.com/books?id=6Uq3QgAACAAJ
- H. Gallis, Erik Arisholm, and Tore Dybå. 2002. A Transition From Partner Programming to Pair Programming - an Industrial Case Study. In Workshop ”Pair Programming Installed” in 17th Annual ACM Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), Position paper. ACM, Seattle USA, –.
- H. Gallis, E. Arisholm, and T. Dyba. 2003. An initial framework for research on pair programming. In 2003 International Symposium on Empirical Software Engineering. ACM, NY, USA, 132–142.
- Dai Hasegawa, Justine Cassell, and Kenji Araki. 2010. The Role of Embodiment and Perspective in Direction-Giving Systems. In Proceedings of AAAI Fall Workshop on Dialog with Robots. AAAI PRESS, USA.
- 10.1177/0018720814547570PMID: · doi:10.1177/0018720814547570pmid
- D. James and S. Clarke. 1998. Women men and interruptions: A critical review. In D. Tannen (Ed.) Oxford studies in sociolinguistics. Gender and conversational interaction. Oxford University Press UK 231–280.
- 10.1016/j.jvlc.2017.04.005Special · doi:10.1016/j.jvlc.2017.04.005special
- S. Kaur Kuttal, K. Gerstner, and A. Bejarano. 2019. Remote Pair Programming in Online CS Education: Investigating through a Gender Lens. In 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, USA, 75–85.
- Danielle L. Jones and Scott D. Fleming. 2013. What use is a backseat driver? A qualitative investigation of pair programming. In Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC. IEEE, USA, 103–110.
- 10.1145/3195570.3195580 · DOI · Paper page
- Clayton Lewis. 1982. Using the ”thinking-aloud” method in cognitive interface design. IBM T.J. Watson Research Center, Yorktown Heights, N.Y.
- 10.1145/1145293 · doi:10.1145/1145293
- M. Minsky, R. Kurzweil, and S. Mann. 2013. The society of intelligent veillance. In 2013 IEEE International Symposium on Technology and Society (ISTAS): Social Implications of Wearable Computing and Augmediated Reality in Everyday Life. IEEE, USA, 13–17.
- Susanne van Mulken, Elisabeth André, and Jochen Müller. 1999. An Empirical Study on the Trustworthiness of Life-like Interface Agents. In Proceedings of the HCI International ’99 (the 8th International Conference on Human-Computer Interaction) on Human-Computer Interaction: Communication, Cooperation, and Application Design-Volume 2 - Volume 2. L. Erlbaum Associates Inc., Hillsdale, NJ, USA, 152–156.
- George Polya. 2004. How to solve it: A new aspect of mathematical method. Vol. 85. Princeton university press, NJ, USA.
- Lutz Prechelt, Ulrich Stärk, and Stephan Salinger. 2009. 7 Types of Cooperation Episodes in Side-by-Side Programming. In Proc. 21st Annual Workshop of the Psychology of Programming Interest Group (PPIG ’09). ACM, USA.
- Genaro Rebolledo-Mendez, Sara de Freitas, Jose Rafael Rojano-Caceres, and Alma Rosa García-Gaona. 2010. An Empirical Examination of the Relation Between Attention and Motivation in Computer-Based Education: a Modeling Approach. In Proceedings of the Twenty-Third International Florida Artificial Intelligence Research Society Conference, May 19-21, 2010, Daytona Beach, Florida, USA, Hans W. Guesgen and R. Charles Murray (Eds.). AAAI Press, USA, 74–79.
- T. Savage, M. Revelle, and D. Poshyvanyk. 2010. FLAT3: feature location and textual tracing tool. In 2010 ACM/IEEE 32nd International Conference on Software Engineering, Vol. 2. ACM, USA, 255–258.
- A. Seeger, J. Pfeiffer, and A. Heinzl. 2017. When Do We Need a Human? Anthropomorphic Design and Trustworthiness of Conversational Agents. In Proceedings of the SixteenthAnnual Pre-ICIS Workshop on HCI Research in MIS. ACM, USA, 1–5.
- R. Sharma, S. Gulia, and K. K. Biswas. 2014. Automated generation of activity and sequence diagrams from natural language requirements. In 9th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE). SCITEPRESS Digital Library, Portugal, 1–9.
- 10.5555/942801.943642 · doi:10.5555/942801.943642
- Laurie A. Williams. 2010. Pair Programming. John Wiley & Sons, USA. 311–322 pages.
Cited by
Ross, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584037 · Cited by 258
Investigating Explainability of Generative AI for Code through Scenario-based Design
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511119 · Cited by 219
“We envision that a conversational agent interface could be a natural fit for AI assistant for code or co-programming tools, as explored in recent work [42, 43] We also noticed that the explainability needs and user needs in general may differ for users with varied levels of programming skills.”
BackgroundJiang, E., Toh, E., Molina, A., Olson, K., Kayacik, C., Donsbach, A., Cai, C. J., & Terry, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501870 · Cited by 79
Better Together? An Evaluation of AI-Supported Code Translation
Weisz, J. D., Muller, M., Ross, S. I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., & Richards, J. T. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511157 · Cited by 69
Weisz, J. D., Kumar, S. V., Muller, M., Browne, K.-E., Goldberg, A., Heintze, K. E., & Bajpai, S. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706670 · Cited by 37
Metadata
- Authors
- Kuttal, S. K., Ong, B., Kwasny, K., & Robe, P.
- Affiliations
- University of Tulsa (US)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-20
- OpenAlex
- W3161416151
- Cited by
- 44
- References
- 151
- Citations by year
- 2026: 8 · 2025: 11 · 2024: 7 · 2023: 10 · 2022: 6 · 2021: 2
- FWCI
- 5.18
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- pair programming, chatbots, Wizard of Oz, code quality, developer productivity, self-efficacy, knowledge transfer, trust in AI, power relations, remote programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.