The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Exploring the Learnability of Program Synthesizers by Novice Programmers
Open accessAbstract
Modern program synthesizers are increasingly delivering on their promise of lightening the burden of programming by automatically generating code, but little research has addressed how we can make such systems learnable to all. In this work, we ask: What aspects of program synthesizers contribute to and detract from their learnability by novice programmers? We conducted a thematic analysis of 22 observations of novice programmers, during which novices worked with existing program synthesizers, then participated in semi-structured interviews. Our findings shed light on how their specific points in the synthesizer design space affect these tools’ learnability by novice programmers, including the type of specification the synthesizer requires, the method of invoking synthesis and receiving feedback, and the size of the specification. We also describe common misconceptions about what constitutes meaningful progress and useful specifications for the synthesizers, as well as participants’ common behaviors and strategies for using these tools. From this analysis, we offer a set of design opportunities to inform the design of future program synthesizers that strive to be learnable by novice programmers. This work serves as a first step toward understanding how we can make program synthesizers more learnable by novices, which opens up the possibility of using program synthesizers in educational settings as well as developer tooling oriented toward novice programmers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We support and extend this (Sec.4.2.6), adding that this is characteristic of exploration mode, which indeed occurs more commonly in novices.”
From Grounded Copilot: How Programmers Interact with Code-Generating Models (2023) · cites this paper as Jayagopal et al. 2022
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning from television in young children
Barr, R., & Brito, N. H. (2012) · doi:10.1007/978-1- · Cited by 3,912
Costs of a predictible switch between simple cognitive tasks.
Rogers, R. D., & Monsell, S. (1995) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.124.2.207 · Cited by 3,170 · cited 1×
“These fndings resonate with broader research on the relatively high cognitive costs of task-switching [1, 32, 59] and the usability issues around modes [12, 57].”
BackgroundGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundPrinciples of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · cited 1×
“Some works offer guidelines for improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1×
“GitHub Copilot [22] is an editor plugin powered by OpenAI’s Codex model [9] that automatically synthesizes code on the fy based on the code and comments the user has already written.”
MethodLuger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176 · cited 1×
“Some works offer guidelines for improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundThe Humane Interface: New Directions for Designing Interactive Systems
Raskin, J. (2000) · Cited by 890
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · cited 1×
“improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundOracle-guided component-based program synthesis
Jha, S., Gulwani, S., Seshia, S. A., & Tiwari, A. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806833 · Cited by 510 · cited 1×
“Recent work in the programming languages community and a variety of other subfields has produced tremendous advances in program synthesis technology [3, 16, 23–25, 33, 35, 45, 53, 56, 61, 63].”
BackgroundIntelligibility and Accountability: Human Considerations in Context-Aware Systems
Bellotti, V., & Edwards, K. (2001) · Human–Computer Interaction · doi:10.1207/s15327051hci16234_05 · Cited by 471 · cited 1×
“improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
Background
Research works printed but not linked: 4
- Matej Balog , Alexander Gaunt , Marc Brockschmidt , Sebastian Nowozin , and Daniel Tarlow . 2017 . DeepCoder: Learning to Write Programs . In Proceedings of ICLR’17 . Matej Balog, Alexander Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to Write Programs. In Proceedings of ICLR’17.
- Simon P. Davies and Adrian M. Castell. 1994. From Individuals to Groups Through Artifacts: The Changing Semantics of Design in Software Development. In User-Centred Requirements for Software Engineering Environments David J. Gilmore Russel L. Winder and Françoise Détienne (Eds.). https://doi.org/10.1007/978-3-662-03035-6_2 · DOI
- Austin Z. Henley , Julian Ball , Benjamin Klein , Aiden Rutter , and Dylan Lee . 2021. An Inquisitive Code Editor for Addressing Novice Programmers’ Misconceptions of Program Behavior. IEEE Press , 165–170. https://doi.org/10.1109/ICSE-SEET52601. 2021 .00026 10.1109/ICSE-SEET52601.2021.00026 Austin Z. Henley, Julian Ball, Benjamin Klein, Aiden Rutter, and Dylan Lee. 2021. An Inquisitive Code Editor for Addressing Novice Programmers’ Misconceptions of Program Behavior. IEEE Press, 165–170. https://doi.org/10.1109/ICSE-SEET52601.2021.00026 · doi:10.1109/icse-seet52601
- A. T. Jersild . 1927. Mental Set and Shift. Archives of Psychology 14, 89 ( 1927 ). A. T. Jersild. 1927. Mental Set and Shift. Archives of Psychology 14, 89 (1927).
Cited by
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
“[Jayagopal et al. 2022] study how undergraduate students learned to use six different synthesizers— Copilot among them—with different interaction modes.”
BackgroundStudying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366
“Finally, in a recent study to explore the learnability of program synthesizers, Jayagopal et al. [38] found that participants preferred program synthesis tools where users can also write code manually and liked the triggerless initiation and triggerless result communication mechanisms of Copilot.”
Kazemitabaar, M., Hou, X., Henley, A. Z., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631806 · Cited by 276
“To our knowledge, only Jayagopal et al. [24] focused novice programmers’ experience with different program synthesizers including Copilot, but only for three tasks.”
BackgroundPrather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“observed novice programmers using ive program synthesis tools [50].”
MethodCarbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models
Anthony, L. F. W., Kanding, B., & Selvan, R. (2020) · arXiv · doi:10.48550/arxiv.2007.03051 · Cited by 255
Lau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241
“analyzed the usability of 5 program synthesis tools (Copilot was one of them) and found that those which run in the background (without explicit user triggering) can bemore learnable for novices [49].”
BackgroundA Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2023) · arXiv · doi:10.48550/arxiv.2303.17125 · Cited by 206
“[28] also conducted usability studies to understand the learnability of five of these tools with novices.”
BackgroundA Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608128 · Cited by 203
“[30] also conducted usability studies to understand the learnability of five of these tools with novices.”
BackgroundOn the Design of AI-powered Code Assistants for Notebooks
McNutt, A. M., Wang, C., DeLine, R. A., & Drucker, S. M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580940 · Cited by 126
“[37] note that this is a factor for novices, but the presence of such concerns among our participants suggests it may be more universal among notebook users.”
BackgroundInfluentialLiu, M. X., Sarkar, A., Negreanu, C., Zorn, B., Williams, J., Toronto, N., & Gordon, A. D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580817 · Cited by 119
“[42], in particular, their analysis of Lau’s design guidelines for PBD systems [56] in the context of LLM-based code generation.”
BackgroundUsing AI-based coding assistants in practice: State of affairs, perceptions, and ways forward
Sergeyuk, A., Golubev, Y., Bryksin, T., & Ahmed, I. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2024.107610 · Cited by 101
“Moreover, with the evolution of AI-supported educational activities [49]– [51], the community of AI-aware developers will expand greatly in the near future.”
BackgroundA Survey of Vibe Coding with Large Language ModelsPreprint
Ge, Y., Mei, L., Duan, Z., Li, T., Zheng, Y., Wang, Y., Wang, L., Yao, J., Liu, T., Cai, Y., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.12399 · Cited by 97
Ironies of Generative AI: Understanding and Mitigating Productivity Loss in Human-AI Interaction
Simkute, A., Tankelevitch, L., Kewenig, V., Scott, A. E., Sellen, A., & Rintel, S. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2405782 · Cited by 89
“…disruptive to users’ workflows in the writing domain, as they “[force] writers to shift from thinking about their narrative or argument to thinking about instructions to the system” (Dang et al., 2023; see also Yuan et al., 2022), a finding echoed in the coding domain (Jayagopal et al., 2022).”
Yang, Z., Sun, Z., Yue, T. Z., Dévanbu, P., & Lo, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.07506 · Cited by 88
Liu, J., Li, S., & Dong, Q. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241242441 · Cited by 85
“Jayagopal et al. (2022) applied Github Copilot in undergraduate programming courses, allowing students to pair with GAI to complete programming tasks.”
Beyond Code Generation: LLM-supported Exploration of the Program Design Space
Zamfirescu-Pereira, J. D., Jun, E., Terry, M. A., Yang, Q., & Hartmann, B. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714154 · Cited by 79
Pan, Z., Xie, Z., Liu, T., & Xia, T. (2024) · Systems · doi:10.3390/systems12050176 · Cited by 78
StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code
Babe, H. M., Nguyen, S., Zi, Y., Guha, A., Feldman, M. Q., & Anderson, C. J. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.501 · Cited by 66
How Practitioners Expect Code Completion?
Wang, C., Hu, J., Gao, C., Jin, Y., Xie, T., Huang, H., Lei, Z., & Deng, Y. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616280 · Cited by 50
“The work [31] explored the relationship between the design of code generation tools and their ease of use (user learn-ability) and McNutt et al. [44] studied the design space and users’ expectations of code assistants in Jupyter Notebook.”
BackgroundNon-Expert Programmers in the Generative AI Future
Feldman, M. Q., & Anderson, C. J. (2024) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3663384.3663393 · Cited by 50
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 49
How Scientists Use Large Language Models to Program
O’Brien, G. E. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713668 · Cited by 42
“Beginners may also struggle to decompose programming problems into smaller tasks before prompting [40, 56], which is known to improve LLM performance on a variety of benchmarks [35, 77, 99] .”
How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study
Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641891 · Cited by 40
“As in other triggerless systems [53], participants received suggestions from the system without needing to take specific action.”
BackgroundNaik, A., Yin, J. R., Kamath, A., Ma, Q., Wu, T., Murray, C., Bogart, C., Sakr, M. F., & Rosé, C. P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64302-6_4 · Cited by 37
SQLucid: Grounding Natural Language Database Queries with Interactive Explanations
Tian, Y., Kummerfeld, J. K., Li, T. J.-J., & Zhang, T. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676368 · Cited by 37
“Task decomposition is a long-standing challenge in program synthesis and code generation [23, 31, 39, 71].”
Scholl, A., & Kiesler, N. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893442 · Cited by 35
“The student perspective on program synthesizers including Copilot was gathered by Jayagopal et al. [42], who focused on its learnability.”
Generating Programs Trivially: Student Use of Large Language Models
Prasad, S., Greenman, B., Nelson, T., & Krishnamurthi, S. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617921 · Cited by 35
Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPTPreprint
Scholl, A., Schiffner, D., & Kiesler, N. (2024) · arXiv (Cornell University) · doi:10.18420/delfi2024_05 · Cited by 32
Drosos, I., Sarkar, A., Xu, X., Negreanu, C., Rintel, S., & Tankelevitch, L. (2024) · Proceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work · doi:10.1145/3663384.3663389 · Cited by 30
Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming TasksPreprint
Etsenake, D., & Nagappan, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.01026 · Cited by 30
“(3) It could just be a preference to want to do some LLM interaction as well as manual work [35, 57].”
Interactions with Prompt Problems: A New Way to Teach Programming with Large Language ModelsPreprint
Prather, J. E., Denny, P. C., Leinonen, J., Smith, D. H., Reeves, B. N., MacNeil, S., Becker, B. A., Luxton-Reilly, A., Amarouche, T., & Kimmel, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.10759 · Cited by 26
Providing tailored reflection instructions in collaborative learning using large language models
Naik, A., Yin, J. R., Kamath, A., Ma, Q., Wu, T., Murray, R. C., Bogart, C., Sakr, M. F., & Rosé, C. P. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13548 · Cited by 25
In-IDE Human-AI Experience in the Era of Large Language Models; A Literature ReviewPreprint
Sergeyuk, A., Titov, S., & Izadi, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.10739 · Cited by 24
“[3, 6, 9, 14, 16, 19, 20, 23, 28, 29, 33, 34, 37, 38] focus on prototyping integration of AI assistants into existing programming environments, systematizing methodologies and proposing principles for designing AI-assisted programming.”
BackgroundMethodGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23
“[Jayagopal et al. 2022] study how undergraduate students learned to use six different synthesizers— Copilot among them—with different interaction modes.”
BackgroundPickering, M. A., Williams, H., Gan, A., He, W., Park, H., Vélez, F., Littman, M. L., & Ur, B. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713271 · Cited by 21
“Others [18, 19, 29, 61] primarily investigate user perceptions as they work collaboratively with programming LLMs, with Ross et al. [46] particularly focusing on how different interfaces affect user satisfaction.”
Basha, M., & Rodríguez-Pérez, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107804 · Cited by 19
Scratch Copilot Evaluation: Assessing AI-Assisted Creative Coding for FamiliesPreprint
Druga, S., & Otero, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.10417 · Cited by 17
“For example, novice coders often require more assistance with coding game ideas, whereas intermediate coders may benefit more from ideation and debugging support [17].”
BackgroundConversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities
Chopra, B., Singha, A., Fariha, A., Gulwani, S., Parnin, C., Tiwari, A., & Henley, A. Z. (2023) · arXiv · doi:10.48550/arxiv.2310.16164 · Cited by 16
How Scientists Use Large Language Models to Program
O'Brien, G. (2025) · arXiv · doi:10.48550/arxiv.2502.17348 · Cited by 14
“Beginners may also struggle to decompose programming problems into smaller tasks before prompting [40, 56], which is known to improve LLM performance on a variety of benchmarks [35, 77, 99] .”
Nigatu, H. H., Pickoff-White, L., Canny, J., & Chasins, S. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594093 · Cited by 13
“frustrations from participants, in line with previous research on users’ mental models of program synthesizers [25].”
Self-citationExplaining GitHub Actions Failures with Large Language Models: Challenges, Insights, and Limitations
Valenzuela-Toledo, P., Wu, C., Hernández, S., Boll, A., Machacek, R., Panichella, S., & Kehrer, T. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00037 · Cited by 11
PwR: Exploring the Role of Representations in Conversational ProgrammingPreprint
Pradyumna, Y. M., Ganesan, V., Arumugam, D. K., Gupta, M., Shadagopan, N., Dixit, T., Segal, S., Kumar, P., Jain, M., & Rajamani, S. K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.09495 · Cited by 9
“The recent emergence of these tools has piqued growing interest in the field of HCI and CSCW. Researchers have explored various facets of these AI-based programming assistants, including usability [16, 32], learnability [12, 28], and trustworthiness [4, 20].”
BackgroundInfluentialNovice Learners of Programming and Generative AI - Prior Knowledge Matters
Kiesler, N., Scholz, I., Albrecht, J., Stappert, F., & Wienkop, U. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699580 · Cited by 8
AI Friends: A Design Framework for AI-Powered Creative Programming for YouthPreprint
Druga, S., & Ko, A. J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.10412 · Cited by 8
Ironies of Generative AI: Understanding and mitigating productivity loss in human-AI interactions
Simkute, A., Tankelevitch, L., Kewenig, V., Scott, A. E., Sellen, A., & Rintel, S. (2024) · arXiv · doi:10.48550/arxiv.2402.11364 · Cited by 7
“…disruptive to users’ workflows in the writing domain, as they “[force] writers to shift from thinking about their narrative or argument to thinking about instructions to the system” (Dang et al., 2023; see also Yuan et al., 2022), a finding echoed in the coding domain (Jayagopal et al., 2022).”
Kobiella, C., Breidenstein, D., & Schmidt, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790757 · Cited by 8
“At the same time, additional studies have documented similar challenges among novice programmers [26, 29, 30, 56, 57].”
Procedural Adherence and Interpretability Through Neuro-Symbolic Generative AgentsPreprint
Rothkopf, R., Zeng, H. T., & Santolucito, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.16905 · Cited by 6
“One challenge to the wider adoption of our work, and thus a key threat to external validity, is the documented difficulty of temporal logic usability (Jayagopal, Lubin, and Chasins 2022; Rothkopf et al. 2023; De Francesco, Santone, and Vaglini 2007; Yamashita, Tsunoda, and Yokogawa 2017).”
BackgroundIRCHIVER: An Information-Centric Personal Web Archive for Revisiting Past Online Sensemaking Tasks
Huang, Z., & Huang, J. (2025) · Proceedings of the 2025 ACM SIGIR Conference on Human Information Interaction and Retrieval · doi:10.1145/3698204.3716459 · Cited by 3
“Such a large-scale deployment would also enable us to evaluate potential costs of irchiver, including information overload and memory usage, which are influenced by long-term, real-world usage.”
User Misconceptions of LLM-Based Conversational Programming AssistantsPreprint
O'Brien, G., Alves, A. P. S., Baltes, S., Liebel, G., & Kalinowski, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.25662 · Cited by 4
“Similarly, user studies with program synthesizers indicate a propensity for overconfidence, termed the "user-synthesizer gap" [14, 18].”
InfluentialEsnaola, L. M., Ramón, H. D., & Lanzarini, L. C. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10813-7 · Cited by 2
“This obser - vation resonates with emerging accounts of non-expert programmers and domain experts appropriating AI tools to automate routine tasks or prototype simple solutions (Feldman and Anderson 2024; Jayagopal et al. 2022 ).”
Gardella, N., Shelton, J., Graßl, I., & Riggs, S. L. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770015 · Cited by 2
“In addition to considering how AI can be used by educators [66], research has begun to explore how novice programmers use code-generating AI-driven Development Environments (AIDEs) such as generic chatbots and in-IDE assistants [38, 40, 54].”
Tidynote: Always-Clear Notebook Authoring
Huang, R. L., Hempel, B., Cao, Y., Hollan, J. D., Xia, H., & Lerner, S. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791919 · Cited by 2
Craft Ethics : Aiming for Virtue in Programming with Generative AI
Jönsson, M., & Tholander, J. (2024) · DiVA (Södertörn University) · Cited by 2
Unlocking the Full Potential of AI Chatbots: A Guide to Maximizing Your Digital Companions
Yu, C. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3664457 · Cited by 2
Pan, D. P., Luo, L., Zhu, J., Gao, Z., Tong, X., & Hui, P. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791410 · Cited by 1
“…shows LLMs can act as pedagogical agents, teachable tutees, feedback generators, and tools for automating grading and content creation [53, 55, 56, 61], offering scalable, personalized feedback to improve engagement and support code generation, debugging, and curriculum design [6, 22, 31, 49, 89].”
ScooPy: Enhancing Program Synthesis with Nested Example Specifications
Katz, T., & Peleg, H. (2025) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3759429.3762619 · Cited by 1
“Jayagopal et al. [19] showed that programmers favor synthesis tools that allow them to mix editing and synthesis in the same interface.”
Surbakti, F. P. S., Chabella, C., & Edison, H. (2025) · TEM Journal · doi:10.18421/tem144-36 · Cited by 1
“These results indicate that the developed information systems perform well, as exceeding the average task completion benchmark of 78% [19].”
Hutchison, J., Applebaum, I. T., Angelikas, K., Patel, K., Nguyen, P., Lázaro, A., Rucinski, N., Nabid, R. A., & MacNeil, S. (2026) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3803400.3809394 · Cited by 1
Gardella, N., Bolton, M. L., & Riggs, S. L. (2026) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208261467254 · Cited by 1
Neuro-Symbolic Program Synthesis for Data-Efficient Learning
Barke, S. (2024) · eScholarship (California Digital Library) · Cited by 1
The Role of Interactive Program Synthesis in the Future of Software Development
Shaout, A., Ditty, J., & Carr, K. (2025) · Cureus Journal of Computer Science. · doi:10.7759/s44389-025-05516-x
“This intent can be represented in various ways, each with its own advantages and disadvantages [57].”
Basha, M., Beschastnikh, I., Rodríguez-Pérez, G., & de Souza, C. R. B. (2025) · arXiv (Cornell University) · doi:10.17605/osf.io/tcfjr
Basha, M. (2025) · cIRcle (University of British Columbia) · doi:10.14288/1.0444836
Bui, D. M. (2025) · Aaltodoc (Aalto University)
Exploratory Phenomena in Program Synthesis
James, M. B. (2024) · eScholarship (California Digital Library)
Human-Centered Program Synthesis
Ferdowsifard, K. (2024) · eScholarship (California Digital Library)
CHATGPT-4 АРҚЫЛЫ JAVA КОДТАУ: ӘДІСТЕРІ МЕН МҮМКІНДІКТЕРІН ТАЛДАУ
Уандықова, М., Байтенова, Л., Востриков, К., Елеукулова, А., & Мухамеджанова, Г. (2025) · Вестник КазАТК · doi:10.52167/1609-1817-2025-139-4-141-159
Debugging into Existence with Program Synthesis
Frankel, G., Segal, S., & Peleg, H. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00021
“Their names, user-triggered and triggerless , are derived from the work of Jayagopal et al. [24].”
InfluentialMoreno-Alameda, D., Iglesias, J. B., Romero, A. D.-G., & Gimeno, A. S. (2026) · Springer eBooks · doi:10.1007/978-3-032-08213-8_9
Barniskyte, A. (2025) · Tampere University Institutional Repository (Tampere University)
"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv
“observed novice programmers using ive program synthesis tools [50].”
MethodLi, S., Liu, J., & Dong, Q. (2025) · Australasian Journal of Educational Technology · doi:10.14742/ajet.9932 · Cited by 32
Metadata
- Authors
- Jayagopal, D., Lubin, J., & Chasins, S. E.
- Affiliations
- University of California, Berkeley (US)
- Venue
- Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology
- Publisher
- ACM
- Published
- 2022-10-28
- Pages
- 1-15
- OpenAlex
- W4307472488
- Semantic Scholar
- 99323bd786ee5be1e1aa589858e14e89630f207b
- Cited by
- 40
- References
- 62
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 15 · 2023: 9
- FWCI
- 11.09
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Symposium on User Interface Software and Technology (UIST) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1752814,CA-HDR 1936731,CA-HDR 2033558,FW-HTF 2129008)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Software › Spreadsheets and End-User Computing
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- program synthesis, learnability, novice programmers, user experience, specification design, feedback mechanisms, tool design, programming education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.