The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Exploring the Learnability of Program Synthesizers by Novice Programmers
Open accessAbstract
Modern program synthesizers are increasingly delivering on their promise of lightening the burden of programming by automatically generating code, but little research has addressed how we can make such systems learnable to all. In this work, we ask: What aspects of program synthesizers contribute to and detract from their learnability by novice programmers? We conducted a thematic analysis of 22 observations of novice programmers, during which novices worked with existing program synthesizers, then participated in semi-structured interviews. Our findings shed light on how their specific points in the synthesizer design space affect these tools’ learnability by novice programmers, including the type of specification the synthesizer requires, the method of invoking synthesis and receiving feedback, and the size of the specification. We also describe common misconceptions about what constitutes meaningful progress and useful specifications for the synthesizers, as well as participants’ common behaviors and strategies for using these tools. From this analysis, we offer a set of design opportunities to inform the design of future program synthesizers that strive to be learnable by novice programmers. This work serves as a first step toward understanding how we can make program synthesizers more learnable by novices, which opens up the possibility of using program synthesizers in educational settings as well as developer tooling oriented toward novice programmers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We support and extend this (Sec.4.2.6), adding that this is characteristic of exploration mode, which indeed occurs more commonly in novices.”
From Grounded Copilot: How Programmers Interact with Code-Generating Models (2023) · cites this paper as Jayagopal et al. 2022
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning from television in young children
Barr, R., & Brito, N. H. (2012) · doi:10.1007/978-1- · Cited by 3,912
Costs of a predictible switch between simple cognitive tasks.
Rogers, R. D., & Monsell, S. (1995) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.124.2.207 · Cited by 3,170 · cited 1×
“These fndings resonate with broader research on the relatively high cognitive costs of task-switching [1, 32, 59] and the usability issues around modes [12, 57].”
BackgroundGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundPrinciples of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · cited 1×
“Some works offer guidelines for improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1×
“GitHub Copilot [22] is an editor plugin powered by OpenAI’s Codex model [9] that automatically synthesizes code on the fy based on the code and comments the user has already written.”
MethodLuger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176 · cited 1×
“Some works offer guidelines for improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundThe Humane Interface: New Directions for Designing Interactive Systems
Raskin, J. (2000) · Cited by 890
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · cited 1×
“improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundOracle-guided component-based program synthesis
Jha, S., Gulwani, S., Seshia, S. A., & Tiwari, A. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806833 · Cited by 510 · cited 1×
“Recent work in the programming languages community and a variety of other subfields has produced tremendous advances in program synthesis technology [3, 16, 23–25, 33, 35, 45, 53, 56, 61, 63].”
BackgroundIntelligibility and Accountability: Human Considerations in Context-Aware Systems
Bellotti, V., & Edwards, K. (2001) · Human–Computer Interaction · doi:10.1207/s15327051hci16234_05 · Cited by 471 · cited 1×
“improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
Background
Research works printed but not linked: 4
- Matej Balog , Alexander Gaunt , Marc Brockschmidt , Sebastian Nowozin , and Daniel Tarlow . 2017 . DeepCoder: Learning to Write Programs . In Proceedings of ICLR’17 . Matej Balog, Alexander Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to Write Programs. In Proceedings of ICLR’17.
- Simon P. Davies and Adrian M. Castell. 1994. From Individuals to Groups Through Artifacts: The Changing Semantics of Design in Software Development. In User-Centred Requirements for Software Engineering Environments David J. Gilmore Russel L. Winder and Françoise Détienne (Eds.). https://doi.org/10.1007/978-3-662-03035-6_2 · DOI
- Austin Z. Henley , Julian Ball , Benjamin Klein , Aiden Rutter , and Dylan Lee . 2021. An Inquisitive Code Editor for Addressing Novice Programmers’ Misconceptions of Program Behavior. IEEE Press , 165–170. https://doi.org/10.1109/ICSE-SEET52601. 2021 .00026 10.1109/ICSE-SEET52601.2021.00026 Austin Z. Henley, Julian Ball, Benjamin Klein, Aiden Rutter, and Dylan Lee. 2021. An Inquisitive Code Editor for Addressing Novice Programmers’ Misconceptions of Program Behavior. IEEE Press, 165–170. https://doi.org/10.1109/ICSE-SEET52601.2021.00026 · doi:10.1109/icse-seet52601
- A. T. Jersild . 1927. Mental Set and Shift. Archives of Psychology 14, 89 ( 1927 ). A. T. Jersild. 1927. Mental Set and Shift. Archives of Psychology 14, 89 (1927).
Cited by
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
“[Jayagopal et al. 2022] study how undergraduate students learned to use six different synthesizers— Copilot among them—with different interaction modes.”
BackgroundStudying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366
“Finally, in a recent study to explore the learnability of program synthesizers, Jayagopal et al. [38] found that participants preferred program synthesis tools where users can also write code manually and liked the triggerless initiation and triggerless result communication mechanisms of Copilot.”
Kazemitabaar, M., Hou, X., Henley, A. Z., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631806 · Cited by 276
“To our knowledge, only Jayagopal et al. [24] focused novice programmers’ experience with different program synthesizers including Copilot, but only for three tasks.”
BackgroundPrather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“observed novice programmers using ive program synthesis tools [50].”
Method
Metadata
- Authors
- Jayagopal, D., Lubin, J., & Chasins, S. E.
- Affiliations
- University of California, Berkeley (US)
- Venue
- Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology
- Publisher
- ACM
- Published
- 2022-10-28
- Pages
- 1-15
- OpenAlex
- W4307472488
- Semantic Scholar
- 99323bd786ee5be1e1aa589858e14e89630f207b
- Cited by
- 40
- References
- 62
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 15 · 2023: 9
- FWCI
- 11.09
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Symposium on User Interface Software and Technology (UIST) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1752814,CA-HDR 1936731,CA-HDR 2033558,FW-HTF 2129008)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Software › Spreadsheets and End-User Computing
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- program synthesis, learnability, novice programmers, user experience, specification design, feedback mechanisms, tool design, programming education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Exploring the Learnability of Program Synthesizers by Novice Programmers. https://app.cochat.ai/papers/10.1145/3526113.3545659
- The paper
- Jayagopal, D., Lubin, J., & Chasins, S. E. (2022). Exploring the Learnability of Program Synthesizers by Novice Programmers. Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. https://doi.org/10.1145/3526113.3545659
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.