Exploring the Learnability of Program Synthesizers by Novice Programmers
Open accessThe DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Abstract
Modern program synthesizers are increasingly delivering on their promise of lightening the burden of programming by automatically generating code, but little research has addressed how we can make such systems learnable to all. In this work, we ask: What aspects of program synthesizers contribute to and detract from their learnability by novice programmers? We conducted a thematic analysis of 22 observations of novice programmers, during which novices worked with existing program synthesizers, then participated in semi-structured interviews. Our findings shed light on how their specific points in the synthesizer design space affect these tools’ learnability by novice programmers, including the type of specification the synthesizer requires, the method of invoking synthesis and receiving feedback, and the size of the specification. We also describe common misconceptions about what constitutes meaningful progress and useful specifications for the synthesizers, as well as participants’ common behaviors and strategies for using these tools. From this analysis, we offer a set of design opportunities to inform the design of future program synthesizers that strive to be learnable by novice programmers. This work serves as a first step toward understanding how we can make program synthesizers more learnable by novices, which opens up the possibility of using program synthesizers in educational settings as well as developer tooling oriented toward novice programmers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We support and extend this (Sec.4.2.6), adding that this is characteristic of exploration mode, which indeed occurs more commonly in novices.”
From Grounded Copilot: How Programmers Interact with Code-Generating Models (2023) · cites this paper as Jayagopal et al. 2022
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning from television in young children
Barr, R., & Brito, N. H. (2012) · doi:10.1007/978-1- · Cited by 3,912
Costs of a predictible switch between simple cognitive tasks.
Rogers, R. D., & Monsell, S. (1995) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.124.2.207 · Cited by 3,170 · cited 1×
“These fndings resonate with broader research on the relatively high cognitive costs of task-switching [1, 32, 59] and the usability issues around modes [12, 57].”
BackgroundGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundPrinciples of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · cited 1×
“Some works offer guidelines for improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1×
“GitHub Copilot [22] is an editor plugin powered by OpenAI’s Codex model [9] that automatically synthesizes code on the fy based on the code and comments the user has already written.”
MethodLuger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176 · cited 1×
“Some works offer guidelines for improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundThe Humane Interface: New Directions for Designing Interactive Systems
Raskin, J. (2000) · Cited by 890
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · cited 1×
“improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundOracle-guided component-based program synthesis
Jha, S., Gulwani, S., Seshia, S. A., & Tiwari, A. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806833 · Cited by 510 · cited 1×
“Recent work in the programming languages community and a variety of other subfields has produced tremendous advances in program synthesis technology [3, 16, 23–25, 33, 35, 45, 53, 56, 61, 63].”
BackgroundIntelligibility and Accountability: Human Considerations in Context-Aware Systems
Bellotti, V., & Edwards, K. (2001) · Human–Computer Interaction · doi:10.1207/s15327051hci16234_05 · Cited by 471 · cited 1×
“improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundAutomating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · Proceedings of the 38th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages · doi:10.1145/1926385.1926423 · Cited by 446 · cited 4×
“Flash Fill [23] is a feature of Microsoft Excel that uses programming-by-example to automatically populate data cells based on existing patterns in other data cells.”
BackgroundMethodInfluentialExplaining Recommendations: Design and Evaluation
Tintarev, N., & Masthoff, J. (2015) · Recommender Systems Handbook · doi:10.1007/978-1-4899-7637-6_10 · Cited by 329 · cited 1×
“Participants’ lack of knowledge about what constitutes a good specifcation was exacerbated by the lack of informative feedback upon synthesis failure, which echoes fndings in the explainability of recommendation systems [67], the learnability of AI systems more broadly (Section 3.”
BackgroundA comparison of static, adaptive, and adaptable menus
Findlater, L., & McGrenere, J. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985704 · Cited by 316 · cited 1×
“improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundSteps to take before intelligent user interfaces become real
Höök, K. (2000) · Interacting with Computers · doi:10.1016/s0953-5438(99)00006-5 · Cited by 315 · cited 1×
“Some works offer guidelines for improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundNovice mistakes: are the folk wisdoms correct?
Spohrer, J. C., & Soloway, E. (1986) · Communications of the ACM · doi:10.1145/6138.6145 · Cited by 310 · cited 1×
“If decomposition is at fault, this echoes the difficulties novice programmers face with plan composition (putting together fragments of programming knowledge that accomplish particular kinds of tasks [64]) and task decomposition in programming more broadly [6, 18, 65].”
BackgroundHow might people interact with agents
Norman, D. A. (1994) · Communications of the ACM · doi:10.1145/176789.176796 · Cited by 287 · cited 1×
“improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundSchkufza, E., Sharma, R., & Aiken, A. (2013) · Proceedings of the eighteenth international conference on Architectural support for programming languages and operating systems · doi:10.1145/2451116.2451150 · Cited by 269
Jungloid mining: helping to navigate the API jungle
Mandelin, D., Xu, L., Bodík, R., & Kimelman, D. (2005) · Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation · doi:10.1145/1065010.1065018 · Cited by 261 · cited 1×
“Recent work in the programming languages community and a variety of other subfields has produced tremendous advances in program synthesis technology [3, 16, 23–25, 33, 35, 45, 53, 56, 61, 63].”
BackgroundAssessing demand for intelligibility in context-aware applications
Lim, B. Y., & Dey, A. K. (2009) · Proceedings of the 11th international conference on Ubiquitous computing · doi:10.1145/1620545.1620576 · Cited by 246 · cited 1×
“Some works offer guidelines for improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundPatterns for How Users Overcome Obstacles in Voice User Interfaces
Myers, C., Furqan, A., Nebolsky, J., Caro, K., & Zhu, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173580 · Cited by 213 · cited 1×
“improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundAttention and Performance 15: Conscious and Nonconscious Information Processing
Umiltà, C. A., & Moscovitch, M. (1994) · Cited by 203
Synthesis of loop-free programs
Gulwani, S., Jha, S., Tiwari, A., & Venkatesan, R. (2011) · Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/1993498.1993506 · Cited by 200
Program synthesis from polymorphic refinement types
Polikarpova, N., Kuraj, I., & Solar-Lezama, A. (2016) · Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2908080.2908093 · Cited by 193 · cited 2×
“matters, but seemingly not as much other dimensions—a surprising fnding in light of design guidelines and goals from the synthesis literature, which emphasize specifcation size [8, 23, 37, 43, 56].”
BackgroundSUGILITE: Creating Multimodal Smartphone Automation by Demonstration
Li, T. J.-J., Azaria, A., & Myers, B. A. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025483 · Cited by 184
Comprehension First: Evaluating a Novel Pedagogy and Tutoring System for Program Tracing in CS1
Nelson, G. L., Xie, B., & Ko, A. J. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106178 · Cited by 128 · cited 1×
“Codecademy-style [10] tutorial systems with built-in programming environments [51, 62] to inquisitive IDEs that prompt users with questions about program behavior during development [27].”
BackgroundProceedings of the 51st ACM Technical Symposium on Computer Science Education
(2020) · doi:10.1145/3328778 · Cited by 122
Yang, R., Shin, E., Newman, M. W., & Ackerman, M. S. (2015) · Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2750858.2804269 · Cited by 119
CodeHint: dynamic and interactive synthesis of code snippets
Galenson, J., Reames, P., Bodik, R., Hartmann, B., & Sen, K. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568250 · Cited by 118 · cited 1×
“Many works compare a novel synthesis tool against a manual programming condition [8, 15, 20, 30, 70].”
MethodWhitworth, B. (2005) · Behaviour and Information Technology · doi:10.1080/01449290512331333700 · Cited by 115 · cited 2×
“” However, these findings findings resonate with the polite software observation that ‘[impolite software] may help novices ... who may trade politeness for usefulness’ [72].”
BackgroundResultLi, T. J.-J., Radensky, M., Jia, J., Singarajah, K., Mitchell, T. M., & Myers, B. A. (2019) · Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3332165.3347899 · Cited by 111 · cited 1×
“Others focus exclusively on one novel synthesizer, with no control condition, and assess whether users can complete tasks with the tool at all [14, 41, 68].”
BackgroundSynthesis modulo recursive functions
Kneuss, E., Kuraj, I., Kuncak, V., & Suter, P. (2013) · Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications · doi:10.1145/2509136.2509555 · Cited by 109 · cited 1×
“Recent work in the programming languages community and a variety of other subfields has produced tremendous advances in program synthesis technology [3, 16, 23–25, 33, 35, 45, 53, 56, 61, 63].”
BackgroundDrosos, I., Barik, T., Guo, P. J., DeLine, R., & Gulwani, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376442 · Cited by 107 · cited 3×
“Synthesis approaches are now sufciently mature that we are seeing them adopted within HCI [8, 13, 26, 31, 40, 47, 66, 70] and integrated into mainstream products [19, 50, 60, 73].”
BackgroundUser Interaction Models for Disambiguation in Programming by Example
Mayer, M., Soares, G., Grechkin, M., Le, V., Marron, M., Polozov, O., Singh, R., Zorn, B., & Gulwani, S. (2015) · Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology · doi:10.1145/2807442.2807459 · Cited by 97 · cited 3×
“Synthesis approaches are now sufciently mature that we are seeing them adopted within HCI [8, 13, 26, 31, 40, 47, 66, 70] and integrated into mainstream products [19, 50, 60, 73].”
BackgroundMaking Sense of Sleep Sensors: How Sleep Sensing Technologies Support and Undermine Sleep Health
Ravichandran, R., Sien, S.-W., Patel, S. N., Kientz, J. A., & Pina, L. R. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025557 · Cited by 95 · cited 1×
“improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundRousillon: Scraping Distributed Hierarchical Web Data
Chasins, S. E., Mueller, M., & Bodik, R. (2018) · Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3242587.3242661 · Cited by 85 · cited 6×
“Although the early wave of program synthesis works from the programming languages community largely did not assess synthesizers’ learnability or usability, the more recent wave of synthesis-augmented interfaces within the HCI community has started to offer answers about how humans interact with synthesizers.”
BackgroundMethodInfluentialSelf-citationSynthesizing data structure transformations from input-output examples
Feser, J. K., Chaudhuri, S., & Dillig, I. (2015) · ACM SIGPLAN Notices · doi:10.1145/2813885.2737977 · Cited by 80
Jiang, E., Toh, E., Molina, A., Olson, K., Kayacik, C., Donsbach, A., Cai, C. J., & Terry, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501870 · Cited by 79
Mind your language: on novices' interactions with error messages
Marceau, G., Fisler, K., & Krishnamurthi, S. (2011) · Proceedings of the 10th SIGPLAN symposium on New ideas, new paradigms, and reflections on programming and software · doi:10.1145/2048237.2048241 · Cited by 77 · cited 2×
“The observational think-aloud design used in our work connects closely with prior studies [7, 39, 46, 71] that use observations of novice programmers to uncover patterns in a variety of processes, from novices’ plan composition strategies to how they track program state.”
BackgroundMethodFalx: Synthesis-Powered Visualization Authoring
Wang, C., Feng, Y., Bodík, R., Dillig, I., Cheung, A., & Ko, A. J. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445249 · Cited by 62 · cited 2×
“Many works compare a novel synthesis tool against a manual programming condition [8, 15, 20, 30, 70].”
BackgroundMethodScout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints
Swearngin, A., Wang, C., Oleson, A., Fogarty, J., & Ko, A. J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376593 · Cited by 58 · cited 1×
“Synthesis approaches are now sufficiently mature that we are seeing them adopted within HCI [8, 13, 26, 31, 40, 47, 66, 70] and integrated into mainstream products [19, 50, 60, 73].”
BackgroundInteractive Program Synthesis by Augmented Examples
Zhang, T., Lowmanstone, L., Wang, X., & Glassman, E. L. (2020) · Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3379337.3415900 · Cited by 57 · cited 3×
“Regae [76] is a standalone tool that implements programming-by-example synthesis for regular expressions (regexes).”
MethodResultProjection Boxes: On-the-fly Reconfigurable Visualization for Live Programming
Lerner, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376494 · Cited by 54 · cited 1×
“boxes [38] that dynamically show the values of variables at diferent points in the program.”
BackgroundA qualitative think-aloud study of novice programmers' code writing strategies
Whalley, J., & Kasto, N. (2014) · Proceedings of the 2014 conference on Innovation & technology in computer science education - ITiCSE '14 · doi:10.1145/2591708.2591762 · Cited by 52 · cited 2×
“The observational think-aloud design used in our work connects closely with prior studies [7, 39, 46, 71] that use observations of novice programmers to uncover patterns in a variety of processes, from novices’ plan composition strategies to how they track program state.”
BackgroundMethodThe importance of students' attention to program state: a case study of debugging behavior
Lewis, C. M. (2012) · Proceedings of the ninth annual international conference on International computing education research · doi:10.1145/2361276.2361301 · Cited by 50 · cited 2×
“The observational think-aloud design used in our work connects closely with prior studies [7, 39, 46, 71] that use observations of novice programmers to uncover patterns in a variety of processes, from novices’ plan composition strategies to how they track program state.”
BackgroundMethodSemi-Automated SVG Programming via Direct Manipulation
Hempel, B., & Chugh, R. (2016) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2984511.2984575 · Cited by 49 · cited 3×
“Synthesis approaches are now sufciently mature that we are seeing them adopted within HCI [8, 13, 26, 31, 40, 47, 66, 70] and integrated into mainstream products [19, 50, 60, 73].”
BackgroundWang, C., Feng, Y., Bodík, R., Cheung, A., & Dillig, I. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3371117 · Cited by 49 · cited 1×
“In fact, the performance of synthesis algorithms may degrade as the sizes of specifications grow large, even when parts of the larger specification are redundant and the specifications are satisfied by the same program [16, 69].”
BackgroundWhy Programming by Demonstration Systems Fail: Lessons Learned for Usable AI
Lau, T. (2009) · AI Magazine · doi:10.1609/aimag.v30i4.2262 · Cited by 48
On the fly synthesis of edit suggestions
Miltner, A., Gulwani, S., Le, V., Leung, A., Radhakrishna, A., Soares, G., Tiwari, A., & Udupa, A. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360569 · Cited by 47 · cited 3×
“The authors of Blue-Pencil did a feld study [48] that included follow-up interviews to collect qualitative feedback of their tool in an uncontrolled environment.”
BackgroundMethodBespoke: Interactively Synthesizing Custom GUIs from Command-Line Applications By Demonstration
Vaithilingam, P., & Guo, P. J. (2019) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3332165.3347944 · Cited by 46 · cited 1×
“Others focus exclusively on one novel synthesizer, with no control condition, and assess whether users can complete tasks with the tool at all [14, 41, 68].”
BackgroundType-and-example-directed program synthesis
Osera, P.-M., & Zdancewic, S. (2015) · ACM SIGPLAN Notices · doi:10.1145/2813885.2738007 · Cited by 45
Program sketching with live bidirectional evaluation
Lubin, J., Collins, N., Omar, C., & Chugh, R. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3408991 · Cited by 42 · cited 2×
“matters, but seemingly not as much other dimensions—a surprising fnding in light of design guidelines and goals from the synthesis literature, which emphasize specifcation size [8, 23, 37, 43, 56].”
BackgroundSelf-citationSmall-Step Live Programming by Example
Ferdowsifard, K., Ordookhanians, A., Peleg, H., Lerner, S., & Polikarpova, N. (2020) · Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3379337.3415869 · Cited by 42 · cited 19×
“Four of the six SnipPy participants attempted to write code manually to complete SnipPy-Reverse-2 instead of entering input-output examples to make a specification for the synthesizer.”
BackgroundMethodInfluentialProgramming not only by example
Peleg, H., Shoham, S., & Yahav, E. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180189 · Cited by 38 · cited 1×
“The first of these studies [55] compares how successful participants were using (i) a traditional programming-by-example synthesizer in which users provide input-output examples versus (ii) a variant of the synthesizer using a paradigm the authors dub “pro-gramming not only by example” that also…”
ResultCombinatorial sketching for finite programs
Solar-Lezama, A., Tancau, L., Bodik, R., Seshia, S., & Saraswat, V. (2006) · ACM SIGARCH Computer Architecture News · doi:10.1145/1168919.1168907 · Cited by 37
Interpretable Program Synthesis
Zhang, T., Chen, Z., Zhu, Y., Vaithilingam, P., Wang, X., & Glassman, E. L. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445646 · Cited by 36 · cited 1×
“…in the explainability of recommendation systems [67], the learnability of AI systems more broadly (Section 3.4), the notion of the user-synthesizer gap [15] that previous usability studies of synthesizers have explored [14, 15], and findings from follow-up work on the Regae synthesizer [75].”
BackgroundInfluentialInteractive synthesis of temporal specifications from examples and natural language
Gavran, I., Darulová, E., & Majumdar, R. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428269 · Cited by 34 · cited 1×
“For instance, tools may show partially completed programs at multiple stages during synthesis and request user feedback before completing more [4], or they may show users examples of program behaviors and ask if they are correct [21].”
BackgroundProgramming by manipulation for layout
Hottelier, T., Bodik, R., & Ryokai, K. (2014) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2642918.2647378 · Cited by 31 · cited 1×
“Many works compare a novel synthesis tool against a manual programming condition [8, 15, 20, 30, 70].”
MethodLudwig: an online programming tutoring and assessment system
Shaffer, S. C. (2005) · ACM SIGCSE Bulletin · doi:10.1145/1083431.1083464 · Cited by 26 · cited 1×
“Codecademy-style [10] tutorial systems with built-in programming environments [51, 62] to inquisitive IDEs that prompt users with questions about program behavior during development [27].”
BackgroundQualitative Analyses of Movements Between Task-level and Code-level Thinking of Novice Programmers
Castro, F. E. V., & Fisler, K. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366847 · Cited by 21
LooPy: interactive program synthesis with control structures
Ferdowsifard, K., Barke, S., Peleg, H., Lerner, S., & Polikarpova, N. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485530 · Cited by 18 · cited 2×
“…in the explainability of recommendation systems [67], the learnability of AI systems more broadly (Section 3.4), the notion of the user-synthesizer gap [15] that previous usability studies of synthesizers have explored [14, 15], and findings from follow-up work on the Regae synthesizer [75].”
BackgroundInfluentialAssuage: Assembly Synthesis Using A Guided Exploration
Hu, J., Vaithilingam, P., Chong, S., Seltzer, M., & Glassman, E. L. (2021) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3472749.3474740 · Cited by 17 · cited 3×
“Synthesis approaches are now sufciently mature that we are seeing them adopted within HCI [8, 13, 26, 31, 40, 47, 66, 70] and integrated into mainstream products [19, 50, 60, 73].”
BackgroundOn the Interplay Between Bottom-Up and Datatype-Driven Program Design
Castro, F. E. V., & Fisler, K. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844574 · Cited by 16 · cited 1×
“If decomposition is at fault, this echoes the difculties novice programmers face with plan composition (putting together fragments of programming knowledge that accomplish particular kinds of tasks [64]) and task decomposition in programming more broadly [6, 18, 65].”
BackgroundSometimes, Rainfall Accumulates: Talk-Alouds with Novice Functional Programmers
Fisler, K., & Castro, F. E. V. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106183 · Cited by 15 · cited 1×
“If decomposition is at fault, this echoes the difficulties novice programmers face with plan composition (putting together fragments of programming knowledge that accomplish particular kinds of tasks [64]) and task decomposition in programming more broadly [6, 18, 65].”
BackgroundAbstraction-Based Interaction Model for Synthesis
Peleg, H., Itzhaky, S., & Shoham, S. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-73721-8_18 · Cited by 10
Davies, S. P., & Castell, A. M. (1994) · Springer eBooks · doi:10.1007/978-3-662-03035-6_2 · Cited by 6 · cited 1×
“Participants completed tasks and thought aloud simultaneously, which could increase cognitive load relative to using tools outside a lab setting [11].”
MethodSoloway, E. M., & Ehrlich, K. (1986) · Elsevier eBooks · doi:10.1016/b978-0-934613-12-5.50042-2 · Cited by 4
Synthesizing Queries via Interactive SketchingPreprint
Bastani, O., Zhang, X., & Solar-Lezama, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.12659 · Cited by 4 · cited 1×
“For instance, tools may show partially completed programs at multiple stages during synthesis and request user feedback before completing more [4], or they may show users examples of program behaviors and ask if they are correct [21].”
Background
Research works printed but not linked: 4
- Matej Balog , Alexander Gaunt , Marc Brockschmidt , Sebastian Nowozin , and Daniel Tarlow . 2017 . DeepCoder: Learning to Write Programs . In Proceedings of ICLR’17 . Matej Balog, Alexander Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to Write Programs. In Proceedings of ICLR’17.
- Simon P. Davies and Adrian M. Castell. 1994. From Individuals to Groups Through Artifacts: The Changing Semantics of Design in Software Development. In User-Centred Requirements for Software Engineering Environments David J. Gilmore Russel L. Winder and Françoise Détienne (Eds.). https://doi.org/10.1007/978-3-662-03035-6_2 · DOI
- Austin Z. Henley , Julian Ball , Benjamin Klein , Aiden Rutter , and Dylan Lee . 2021. An Inquisitive Code Editor for Addressing Novice Programmers’ Misconceptions of Program Behavior. IEEE Press , 165–170. https://doi.org/10.1109/ICSE-SEET52601. 2021 .00026 10.1109/ICSE-SEET52601.2021.00026 Austin Z. Henley, Julian Ball, Benjamin Klein, Aiden Rutter, and Dylan Lee. 2021. An Inquisitive Code Editor for Addressing Novice Programmers’ Misconceptions of Program Behavior. IEEE Press, 165–170. https://doi.org/10.1109/ICSE-SEET52601.2021.00026 · doi:10.1109/icse-seet52601
- A. T. Jersild . 1927. Mental Set and Shift. Archives of Psychology 14, 89 ( 1927 ). A. T. Jersild. 1927. Mental Set and Shift. Archives of Psychology 14, 89 (1927).
Cited by
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
“[Jayagopal et al. 2022] study how undergraduate students learned to use six different synthesizers— Copilot among them—with different interaction modes.”
BackgroundStudying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366
“Finally, in a recent study to explore the learnability of program synthesizers, Jayagopal et al. [38] found that participants preferred program synthesis tools where users can also write code manually and liked the triggerless initiation and triggerless result communication mechanisms of Copilot.”
Kazemitabaar, M., Hou, X., Henley, A., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631806 · Cited by 130
“To our knowledge, only Jayagopal et al. [24] focused novice programmers’ experience with different program synthesizers including Copilot, but only for three tasks.”
BackgroundPrather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“observed novice programmers using ive program synthesis tools [50].”
Method
Metadata
- Authors
- Jayagopal, D., Lubin, J., & Chasins, S. E.
- Affiliations
- University of California, Berkeley (US)
- Venue
- Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology
- Publisher
- ACM
- Published
- 2022-10-28
- Pages
- 1-15
- OpenAlex
- W4307472488
- Semantic Scholar
- 99323bd786ee5be1e1aa589858e14e89630f207b
- Cited by
- 40
- References
- 62
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 15 · 2023: 9
- FWCI
- 11.09
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Symposium on User Interface Software and Technology (UIST) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1752814,CA-HDR 1936731,CA-HDR 2033558,FW-HTF 2129008)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Software › Spreadsheets and End-User Computing
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- program synthesis, learnability, novice programmers, user experience, specification design, feedback mechanisms, tool design, programming education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.