Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models
The DOI matches its Crossref record; 22 of the 24 references it cites were found in a registry or an index.
Abstract
In this paper, we present a natural language code synthesis tool, GenLine, backed by 1) a large generative language model and 2) a set of task-specific prompts that create or change code. To understand the user experience of natural language code synthesis with these new types of models, we conducted a user study in which participants applied GenLine to two programming tasks. Our results indicate that while natural language code synthesis can sometimes provide a magical experience, participants still faced challenges. In particular, participants felt that they needed to learn the model’s “syntax,” despite their input being natural language. Participants also struggled to form an accurate mental model of the types of requests the model can reliably translate and developed a set of strategies to debug model input. From these findings, we discuss design implications for future natural language code synthesis tools built using large generative language models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In particular, the neural models introduce new tasks into a developer's workfow, such as writing AI prompts [17] and verifying AI suggestions [34], which can be lengthy.”
From Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming (2024) · cites this paper as [17]
In the citing paper
“Finally, we observe that there are three main ways participants craft prompts: 1) through writing a single line comment with natural language instructions, although the comment may resemble pseudo-code [17], an example: # impute missing values in X_train as average of column # where missinggn value is -1 2) through writing a docstring for the function:”
From Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming (2024) · cites this paper as [17]
In the citing paper
“Similar to our findings, developers in their study were willing to rewrite the natural language prompt to clarify their intent and expressed the need for a syntax to communicate with the model more clearly.”
From Grounded Copilot: How Programmers Interact with Code-Generating Models (2023) · cites this paper as Jiang et al. [2022]
In the citing paper
“This supports [12]'s findings on the implications of developers' syntax on an AI assistant for web applications”
From Do Users Write More Insecure Code with AI Assistants? (2023) · cites this paper as [12]
In the citing paper
“All of this can help them make informed decisions about when and how much to trust AI in their programming workflow, helping them build responsible trust [ 34 ].”
From "It would work for me too": How Online Communities Shape Software Developers' Trust in AI-Powered Code Generation Tools (2024) · cites this paper as [34]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The Discovery of Grounded Theory: Strategies for Qualitative Research.
Wagner, H. R., Glaser, B. G., & Strauss, A. L. (1968) · Social Forces · doi:10.2307/2575405 · Cited by 46,361
The vocabulary problem in human-system communication
Furnas, G. W., Landauer, T. K., Gomez, L. M., & Dumais, S. T. (1987) · Communications of the ACM · doi:10.1145/32206.32212 · Cited by 1,504 · cited 1×
“These types of suggestions may additionally aid novices who are new to the problem domain, and thus may not know how to solve a problem or how to express their goal in a way likely to produce a useful result (i.e., addressing the vocabulary problem [23]).”
BackgroundMariani, J. (2018) · HAL · Cited by 1,151
Hearst, M. A. (2009) · doi:10.1017/cbo9781139644082 · Cited by 797
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 3×
“For example, as part of its documentation or onboarding materials [17], a system could provide a variety of example requests that are likely to work well, for each prompt presented (e.g., for the “Generate HTML” prompt, the system could surface examples such as “make a blue button” or “make a text…”
BackgroundInfluentialSelf-citationAutomating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · Proceedings of the 38th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages · doi:10.1145/1926385.1926423 · Cited by 446 · cited 2×
“One of the known issues with inductive specifcation is the potential for ambiguity in interpreting the user’s input [10, 25, 42].”
BackgroundExample-centric programming: integrating web search into the development environment
Brandt, J., Dontcheva, M., Weskamp, M., & Klemmer, S. R. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753402 · Cited by 324 · cited 1×
“For example, Brandt et al. [14] showed how web search can be integrated with a development environment to ease the process of fnding and applying web-based examples to code.”
BackgroundNovice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models
Louie, R., Coenen, A., Huang, C. Z., Terry, M., & Cai, C. J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376739 · Cited by 271 · cited 2×
“For example, Cococo [33] integrates a generative music model into an end-user application.”
BackgroundSelf-citationNarechania, A., Srinivasan, A., & Stasko, J. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030378 · Cited by 208 · cited 1×
“Interpreting natural language specifcations (or intent), in particular, has been a long-standing goal in the research community, with prior work demonstrating systems that can transform natural language descriptions to spreadsheet macros [26], shell scripts [31, 32], SQL queries [36], and data visualizations [34] (amo…”
BackgroundTenney, I., Wexler, J., Bastings, J., Bolukbasi, T., Coenen, A., Gehrmann, S., Jiang, E., Pushkarna, M., Radebaugh, C., Reif, E., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.15 · Cited by 138 · cited 1×
“To help build a more robust understanding of the model and its capabilities, model attribution or interpretability techniques like those found in the Language Interpretability Tool [38] could be useful.”
MethodSelf-citationPerfection Not Required? Human-AI Partnerships in Code Translation
Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450656 · Cited by 129 · cited 2×
“Recognizing that errors are inevitable, the researchers note that actual adoption likely hinges on “how many errors are present and the nature of those errors” [40].”
BackgroundExploring exploratory programming
Beth Kery, M., & Myers, B. A. (2017) · 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2017.8103446 · Cited by 123 · cited 1×
“This ability to interactively and iteratively construct output enables interactions similar to live programming [22] or exploratory programming [12].”
BackgroundMulti-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs
Li, T. J.-J., Chen, J., Xia, H., Mitchell, T. M., & Myers, B. A. (2020) · Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3379337.3415820 · Cited by 98 · cited 1×
“Outside the realm of software engineering tools, other work examines how to recover from, and disambiguate within, conversational breakdowns (e.g., [30]).”
BackgroundNLyze: interactive programming by natural language for spreadsheet data analysis and manipulation
Gulwani, S., & Marron, M. (2014) · Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2588555.2612177 · Cited by 97 · cited 1×
“Interpreting natural language specifcations (or intent), in particular, has been a long-standing goal in the research community, with prior work demonstrating systems that can transform natural language descriptions to spreadsheet macros [26], shell scripts [31, 32], SQL queries [36], and data visualizations [34] (amo…”
BackgroundMulti-modal synthesis of regular expressions
Chen, Q., Wang, X., Ye, X., Durrett, G., & Dillig, I. (2020) · Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/3385412.3385988 · Cited by 79 · cited 1×
“Programming-by-example is also a form of inductive specifcation in which users defne a function by providing input-output examples (e.g., [19, 39, 42]).”
BackgroundTowards a theory of natural language interfaces to databases
Popescu, A.-M., Etzioni, O., & Kautz, H. (2003) · Proceedings of the 8th international conference on Intelligent user interfaces · doi:10.1145/604045.604120 · Cited by 63
Interactive Program Synthesis by Augmented Examples
Zhang, T., Lowmanstone, L., Wang, X., & Glassman, E. L. (2020) · Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3379337.3415900 · Cited by 57 · cited 4×
“…to help refne or discover the user’s true intent, such as generating additional examples for the user to evaluate in programming-by-example systems [42]; showing alternative, valid interpretations to the end-user for them to choose from [25]; or providing the specifcation with a combination of…”
BackgroundInfluentialKuttal, S. K., Ong, B., Kwasny, K., & Robe, P. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445659 · Cited by 44
Small-Step Live Programming by Example
Ferdowsifard, K., Ordookhanians, A., Peleg, H., Lerner, S., & Polikarpova, N. (2020) · Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3379337.3415869 · Cited by 42 · cited 1×
“This ability to interactively and iteratively construct output enables interactions similar to live programming [22] or exploratory programming [12].”
BackgroundSemantic programming by example with pre-trained models
Verbruggen, G., Le, V., & Gulwani, S. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485477 · Cited by 36 · cited 1×
“Programming-by-example is also a form of inductive specifcation in which users defne a function by providing input-output examples (e.g., [19, 39, 42]).”
BackgroundRahmani, K., Raza, M., Gulwani, S., Le, V., Morris, D., Radhakrishna, A., Soares, G., & Tiwari, A. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485535 · Cited by 33 · cited 1×
“…true intent, such as generating additional examples for the user to evaluate in programming-by-example systems [42]; showing alternative, valid interpretations to the end-user for them to choose from [25]; or providing the specifcation with a combination of natural language and examples [ 37].”
Background
Research works printed but not linked: 12
- Daniel Adiwardana Minh-Thang Luong David R. So Jamie Hall Noah Fiedel Romal Thoppilan Zi Yang Apoorv Kulshreshtha Gaurav Nemade Yifeng Lu and Quoc V. Le. 2020. Towards a Human-like Open-Domain Chatbot. arxiv:2001.09977 [cs.CL] Accessed: 2021-08-12. Daniel Adiwardana Minh-Thang Luong David R. So Jamie Hall Noah Fiedel Romal Thoppilan Zi Yang Apoorv Kulshreshtha Gaurav Nemade Yifeng Lu and Quoc V. Le. 2020. Towards a Human-like Open-Domain Chatbot. arxiv:2001.09977 [cs.CL] Accessed: 2021-08-12. · arXiv
- Miltiadis Allamanis , Earl T. Barr , Premkumar Devanbu , and Charles Sutton . 2018. A Survey of Machine Learning for Big Code and Naturalness. ACM Comput. Surv. 51, 4 , Article 81 (July 2018 ), 37 pages. https://doi.org/10.1145/3212695 Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles Sutton. 2018. A Survey of Machine Learning for Big Code and Naturalness. ACM Comput. Surv. 51, 4, Article 81 (July 2018), 37 pages. https://doi.org/10.1145/3212695 · DOI · Paper page
- Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le and Charles Sutton. 2021. Program Synthesis with Large Language Models. arxiv:2108.07732 [cs.PL] Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le and Charles Sutton. 2021. Program Synthesis with Large Language Models. arxiv:2108.07732 [cs.PL] · arXiv
- Rishi Bommasani Drew A. Hudson Ehsan Adeli Russ Altman Simran Arora Sydney von Arx Michael S. Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill Erik Brynjolfsson Shyamal Buch Dallas Card Rodrigo Castellon Niladri Chatterji Annie S. Chen Kathleen Creel Jared Quincy Davis Dorottya Demszky Chris Donahue Moussa Doumbouya Esin Durmus Stefano Ermon John Etchemendy Kawin Ethayarajh Li Fei-Fei Chelsea Finn Trevor Gale Lauren Gillespie Karan Goel Noah D. Goodman Shelby Grossman Neel Guha Tatsunori Hashimoto Peter Henderson John Hewitt Daniel E. Ho Jenny Hong Kyle Hsu Jing Huang Thomas Icard Saahil Jain Dan Jurafsky Pratyusha Kalluri Siddharth Karamcheti Geoff Keeling Fereshte Khani Omar Khattab Pang Wei Koh Mark S. Krass Ranjay Krishna Rohith Kuditipudi and etal2021. On the Opportunities and Risks of Foundation Models. CoRR abs/2108.07258(2021). arxiv:2108.07258https://arxiv.org/abs/2108.07258 Rishi Bommasani Drew A. Hudson Ehsan Adeli Russ Altman Simran Arora Sydney von Arx Michael S. Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill Erik Brynjolfsson Shyamal Buch Dallas Card Rodrigo Castellon Niladri Chatterji Annie S. Chen Kathleen Creel Jared Quincy Davis Dorottya Demszky Chris Donahue Moussa Doumbouya Esin Durmus Stefano Ermon John Etchemendy Kawin Ethayarajh Li Fei-Fei Chelsea Finn Trevor Gale Lauren Gillespie Karan Goel Noah D. Goodman Shelby Grossman Neel Guha Tatsunori Hashimoto Peter Henderson John Hewitt Daniel E. Ho Jenny Hong Kyle Hsu Jing Huang Thomas Icard Saahil Jain Dan Jurafsky Pratyusha Kalluri Siddharth Karamcheti Geoff Keeling Fereshte Khani Omar Khattab Pang Wei Koh Mark S. Krass Ranjay Krishna Rohith Kuditipudi and et al.2021. On the Opportunities and Risks of Foundation Models. CoRR abs/2108.07258(2021). arxiv:2108.07258https://arxiv.org/abs/2108.07258 · arXiv
- Tom Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared D Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal , Ariel Herbert-Voss , Gretchen Krueger , Tom Henighan , Rewon Child , Aditya Ramesh , Daniel Ziegler , Jeffrey Wu , Clemens Winter , Chris Hesse , Mark Chen , Eric Sigler , Mateusz Litwin , Scott Gray , Benjamin Chess , Jack Clark , Christopher Berner , Sam McCandlish , Alec Radford , Ilya Sutskever , and Dario Amodei . 2020. Language Models are Few-Shot Learners . In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates , Inc ., 1877 –1901. https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 1877–1901. https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
- Daniel Buschek Lukas Mecke Florian Lehmann and Hai Dang. 2021. Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems. arXiv preprint arXiv:2104.00358(2021). Daniel Buschek Lukas Mecke Florian Lehmann and Hai Dang. 2021. Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems. arXiv preprint arXiv:2104.00358(2021). · arXiv
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arxiv:2107.03374 [cs.LG] Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arxiv:2107.03374 [cs.LG] · arXiv
- Prem Devanbu Matthew Dwyer Sebastian Elbaum Michael Lowry Kevin Moran Denys Poshyvanyk Baishakhi Ray Rishabh Singh and Xiangyu Zhang. 2020. Deep Learning & Software Engineering: State of Research and Future Directions. arxiv:2009.08525 [cs.SE] Prem Devanbu Matthew Dwyer Sebastian Elbaum Michael Lowry Kevin Moran Denys Poshyvanyk Baishakhi Ray Rishabh Singh and Xiangyu Zhang. 2020. Deep Learning & Software Engineering: State of Research and Future Directions. arxiv:2009.08525 [cs.SE] · arXiv
- Cheng- Zhi Anna Huang , Ashish Vaswani , Jakob Uszkoreit , Ian Simon , Curtis Hawthorne , Noam Shazeer , Andrew M. Dai , Matthew D. Hoffman , Monica Dinculescu , and Douglas Eck . 2019 . Music Transformer. In International Conference on Learning Representations. https://openreview.net/forum?id=rJe4ShAcF7 Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. 2019. Music Transformer. In International Conference on Learning Representations. https://openreview.net/forum?id=rJe4ShAcF7
- Xi Victoria Lin . 2017. Program Synthesis from Natural Language Using Recurrent Neural Networks . http://victorialin.net/pubs/tellina_tr_2017.pdf Accessed: 2021-04-06. Xi Victoria Lin. 2017. Program Synthesis from Natural Language Using Recurrent Neural Networks. http://victorialin.net/pubs/tellina_tr_2017.pdf Accessed: 2021-04-06.
- Donald A. Norman . 2002. The Design of Everyday Things . Basic Books, Inc. , USA. Donald A. Norman. 2002. The Design of Everyday Things. Basic Books, Inc., USA.
- Frank F. Xu Bogdan Vasilescu and Graham Neubig. 2021. In-IDE Code Generation from Natural Language: Promise and Challenges. arxiv:2101.11149 [cs.SE] Frank F. Xu Bogdan Vasilescu and Graham Neubig. 2021. In-IDE Code Generation from Natural Language: Promise and Challenges. arxiv:2101.11149 [cs.SE] · arXiv
Cited by
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“Several papers also introduced code synthesis LLMs [49]– [52], based on large data sets with little pre-filtering of the training data.”
BackgroundLarge Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
“Several papers also introduced code synthesis LLMs [49]– [52], based on large data sets with little pre-filtering of the training data.”
BackgroundThe Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366
“Jiang et al. created GenLine [39] and conducted a study with 12 participants that worked on two web-programming tasks to explore the user experience of natural language programming with AI code generators.”
Metadata
- Authors
- Jiang, E., Toh, E., Molina, A., Olson, K., Kayacik, C., Donsbach, A., Cai, C. J., & Terry, M.
- Affiliations
- Google (United States) (US)
- Venue
- CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2022-04-29
- Pages
- 1-19
- OpenAlex
- W4225120919
- Semantic Scholar
- 7aa0ab836f198b9ba040a92534eb89f1d706b66d
- Cited by
- 79
- References
- 24
- Citations by year
- 2026: 7 · 2025: 25 · 2024: 31 · 2023: 15 · 2022: 1
- FWCI
- 16.10
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- natural language programming, large language models, prompt engineering, user experience, mental models, user strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.