CoChat

Exploring the Learnability of Program Synthesizers by Novice Programmers

Open access
Jayagopal, Dhanya, Lubin, Justin, Chasins, Sarah E. · 2022 · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology

The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.

Abstract

Modern program synthesizers are increasingly delivering on their promise of lightening the burden of programming by automatically generating code, but little research has addressed how we can make such systems learnable to all. In this work, we ask: What aspects of program synthesizers contribute to and detract from their learnability by novice programmers? We conducted a thematic analysis of 22 observations of novice programmers, during which novices worked with existing program synthesizers, then participated in semi-structured interviews. Our findings shed light on how their specific points in the synthesizer design space affect these tools’ learnability by novice programmers, including the type of specification the synthesizer requires, the method of invoking synthesis and receiving feedback, and the size of the specification. We also describe common misconceptions about what constitutes meaningful progress and useful specifications for the synthesizers, as well as participants’ common behaviors and strategies for using these tools. From this analysis, we offer a set of design opportunities to inform the design of future program synthesizers that strive to be learnable by novice programmers. This work serves as a first step toward understanding how we can make program synthesizers more learnable by novices, which opens up the possibility of using program synthesizers in educational settings as well as developer tooling oriented toward novice programmers.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3526113.3545659

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 4

  1. Matej Balog , Alexander Gaunt , Marc Brockschmidt , Sebastian Nowozin , and Daniel Tarlow . 2017 . DeepCoder: Learning to Write Programs . In Proceedings of ICLR’17 . Matej Balog, Alexander Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to Write Programs. In Proceedings of ICLR’17.
  2. Simon P. Davies and Adrian M. Castell. 1994. From Individuals to Groups Through Artifacts: The Changing Semantics of Design in Software Development. In User-Centred Requirements for Software Engineering Environments David J. Gilmore Russel L. Winder and Françoise Détienne (Eds.). https://doi.org/10.1007/978-3-662-03035-6_2 · DOI
  3. Austin Z. Henley , Julian Ball , Benjamin Klein , Aiden Rutter , and Dylan Lee . 2021. An Inquisitive Code Editor for Addressing Novice Programmers’ Misconceptions of Program Behavior. IEEE Press , 165–170. https://doi.org/10.1109/ICSE-SEET52601. 2021 .00026 10.1109/ICSE-SEET52601.2021.00026 Austin Z. Henley, Julian Ball, Benjamin Klein, Aiden Rutter, and Dylan Lee. 2021. An Inquisitive Code Editor for Addressing Novice Programmers’ Misconceptions of Program Behavior. IEEE Press, 165–170. https://doi.org/10.1109/ICSE-SEET52601.2021.00026 · doi:10.1109/icse-seet52601
  4. A. T. Jersild . 1927. Mental Set and Shift. Archives of Psychology 14, 89 ( 1927 ). A. T. Jersild. 1927. Mental Set and Shift. Archives of Psychology 14, 89 (1927).

Cited by

See all 73

Metadata

Authors
Jayagopal, D., Lubin, J., & Chasins, S. E.
Affiliations
University of California, Berkeley (US)
Venue
Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology
Publisher
ACM
Published
2022-10-28
Pages
1-15
OpenAlex
W4307472488
Cited by
40
References
62
Citations by year
2026: 6 · 2025: 10 · 2024: 15 · 2023: 9
FWCI
11.09
Type
conference paper
Language
en
Open access
Locations
ACM Symposium on User Interface Software and Technology (UIST) (publishedVersion, cc-by)
Funders
National Science Foundation (1752814,CA-HDR 1936731,CA-HDR 2033558,FW-HTF 2129008)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Software › Spreadsheets and End-User Computing
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
program synthesis, learnability, novice programmers, user experience, specification design, feedback mechanisms, tool design, programming education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.