The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development

Ross, Steven I.; Martinez, Fernando; Houde, Stephanie; Muller, Michael; Weisz, Justin D.

doi:10.1145/3581641.3584037

Cited by 258

The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.

The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development

Open access

Ross, Steven I., Martinez, Fernando, Houde, Stephanie, Muller, Michael, Weisz, Justin D.·2023·Proceedings of the 28th International Conference on Intelligent User Interfaces

DOI 10.1145/3581641.3584037

Open this paper in CoChat — free account

Abstract

Large language models (LLMs) have recently been applied in software engineering to perform tasks such as translating code between programming languages, generating code from natural language, and autocompleting code as it is being written. When used within development tools, these systems typically treat each model invocation independently from all previous invocations, and only a specific limited functionality is exposed within the user interface. This approach to user interaction misses an opportunity for users to more deeply engage with the model by having the context of their previous interactions, as well as the context of their code, inform the model’s responses. We developed a prototype system – the Programmer’s Assistant – in order to explore the utility of conversational interactions grounded in code, as well as software engineers’ receptiveness to the idea of conversing with, rather than invoking, a code-fluent LLM. Through an evaluation with 42 participants with varied levels of programming experience, we found that our system was capable of conducting extended, multi-turn discussions, and that it enabled additional knowledge and capabilities beyond code generation to emerge from the LLM. Despite skeptical initial expectations for conversational programming assistance, participants were impressed by the breadth of the assistant’s capabilities, the quality of its responses, and its potential for improving their productivity. Our work demonstrates the unique potential of conversational interactions with LLMs for co-creative processes like software development.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1145/3581641.3584037

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Machine Learning in Medicine
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Machines and Mindlessness: Social Responses to Computers
Nass, C., & Moon, Y. (2000) · Journal of Social Issues · doi:10.1111/0022-4537.00153 · Cited by 4,022 · cited 1×
“” The literature on Computers as Social Agents (CASA) helps us interpret this result as it demonstrates how computers are often treated like people [56, 67].”
Background
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 1×
“Many popular text-to-image models, such as DALL-E 2 [66] and Stable Diffusion [72], operate in a one-shot fashion, in which the user specifies a prompt, clicks a button, and gets results.”
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Chatbots: History, technology, and applications
Adamopoulou, E., & Moussiades, L. (2020) · Machine Learning with Applications · doi:10.1016/j.mlwa.2020.100006 · Cited by 1,090 · cited 1×
“Using natural language to interact with technology has had a long research history [2], starting in the 1960s with pattern-matching approaches like Eliza [104], and continuing to today with state-of-the-art large language modelbased conversational systems [107] such as Meena [3] and BlenderBot [84].”
Background

Show all 34 references

Research works printed but not linked: 68

10.1109/MS.2017.31 · DOI
Safinah Ali Nisha Elizabeth Devasia and Cynthia Breazeal. 2022. Escape! Bot: Social Robots as Creative Problem-Solving Partners. In Creativity and Cognition. 275–283. · DOI
10.1145/3212695 · DOI · Paper page
10.1109/VISSOFT.2013.6650527 · DOI
10.1145/257089.257151 · DOI
Amanda Askell Yuntao Bai Anna Chen Dawn Drain Deep Ganguli Tom Henighan Andy Jones Nicholas Joseph Ben Mann Nova DasSarma 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861(2021). · arXiv
10.1145/3209978.3210027 · DOI
Sallyann Bryant, Pablo Romero, and Benedict" du Boulay. 2006. The Collaborative Nature of Pair Programming. In Extreme Programming and Agile Processes in Software Engineering, Pekka Abrahamsson, Michele Marchesi, and Giancarlo Succi (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 53–64.
Gaetano Cascini Yukari Nagai Georgi V Georgiev Jader Zelaya Niccolò Becattini Jean-François Boujut Hernan Casakin Nathan Crilly Elies Dekoninck John Gero 2022. Perspectives on design creativity and innovation research: 10 years later. 30 pages. · DOI
Stephen Cass. 2022. Top Programming Languages 2022. IEEE Spectrum (23 Aug 2022). https://spectrum.ieee.org/top-programming-languages-2022
Cristina Catalan Aguirre Nuria Gonzalez Castro Carlos Delgado Kloos Carlos Alario-Hoyos and Pedro José Muñoz Merino. 2021. Conversational agent for supporting learners on a MOOC on programming with Java. (2021). · DOI
10.1145/966789.966803 · DOI
10.1109/APSEC.2005.22 · DOI
10.1007/978-3-642-39646-5_8 · DOI
10.1145/1066129.1066132 · DOI
Bobbie Eicher, Kathryn Cunningham, Sydni Peterson Marissa Gonzales, and Ashok Goel. 2017. Toward mutual theory of mind as a foundation for co-creation. In International Conference on Computational Creativity, Co-Creation Workshop.
Stephen M Fiore, Eduardo Salas, and Janis A Cannon-Bowers. 2001. Group dynamics and shared mental model development. How people evaluate others in organizations 234 (2001).
Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 1995. Design patterns: elements of reusable object-oriented software. Addison-Wesley.
Amelia Glaese Nat McAleese Maja Trębacz John Aslanides Vlad Firoiu Timo Ewalds Maribeth Rauh Laura Weidinger Martin Chadwick Phoebe Thacker Lucy Campbell-Gillingham Jonathan Uesato Po-Sen Huang Ramona Comanescu Fan Yang Abigail See Sumanth Dathathri Rory Greig Charlie Chen Doug Fritz Jaume Sanchez Elias Richard Green Soňa Mokrá Nicholas Fernando Boxi Wu Rachel Foley Susannah Young Iason Gabriel William Isaac John Mellor Demis Hassabis Koray Kavukcuoglu Lisa Anne Hendricks and Geoffrey Irving. 2022. Improving alignment of dialogue agents via targeted human judgements. https://arxiv.org/abs/2209.14375 · arXiv
10.14236/ewic/HCI2018.103 · DOI
10.1007/s10664-019-09730-9 · DOI · Paper page
Edwin L Hutchins, James D Hollan, and Donald A Norman. 1985. Direct manipulation interfaces. Human–computer interaction 1, 4 (1985), 311–338.
10.18653/v1/P16-1195 · DOI · Paper page
Andreas Jedlitschka and Markus Nick. 2003. Software Engineering Knowledge Repositories. Springer Berlin Heidelberg, Berlin, Heidelberg, 55–80.
10.1145/3411764.3445659 · DOI · Paper page
Lauramaria Laine. 2021. Exploring Advertising Creatives’ Attitudes Towards Human-AI Collaboration. (2021).
10.1002/hbe2.266 · DOI
10.1145/263552.263612 · DOI
John E Mathieu, Tonia S Heffner, Gerald F Goodwin, Eduardo Salas, and Janis A Cannon-Bowers. 2000. The influence of shared mental models on team process and performance.Journal of applied psychology 85, 2 (2000), 273.
Ekaterina A Moroz, Vladimir O Grizkevich, and Igor M Novozhilov. 2022. The Potential of Artificial Intelligence as a Method of Software Developer’s Productivity Improvement. In 2022 Conference of Russian Young Researchers in Electrical and Electronic Engineering (ElConRus). IEEE, 386–390.
Michael Muller, Stevean Ross, Stephanie Houde, Mayank Agarwal, Fernando Martinez, John Richards, Kartik Talamadupula, and Justin D Weisz. 2022. Drinking Chai with Your (AI) Programming Partner: A Design Fiction about Generative AI for Software Engineering. HAI-GEN Workshop at IUI 2022: 3rd Workshop on Human-AI Co-Creation with Generative Models (2022). https://hai-gen.github.io/2022/
10.1145/2662253.2662300 · doi:10.1145/2662253.2662300
10.1145/3126594.3126628 · DOI
Martin Nordio H Estler Carlo A Furia Bertrand Meyer 2011. Collaborative software development on the web. arXiv preprint arXiv:1105.0768(2011). · arXiv
Maxwell Nye, Anders Andreassen, Guy Gur-Ari, Henryk Witold Michalewski, Jacob Austin, David Bieber, David Martin Dohan, Aitor Lewkowycz, Maarten Paul Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show Your Work: Scratchpads for Intermediate Computation with Language Models. https://arxiv.org/abs/2112.00114. · arXiv
Peter Pirolli and Stuart Card. 1999. Information foraging.Psychological review 106, 4 (1999), 643.
10.1145/92755.92783 · doi:10.1145/92755.92783
Mawarny Md Rejab, James Noble, and George Allan. 2014. Distributing Expertise in Agile Software Development Projects. In 2014 Agile Conference. 33–36.
Jeba Rezwana and Mary Lou Maher. 2021. COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systems.. In ICCC. 444–448.
Charles H. Rich and Richard C. Waters. 1990. The Programmer’s Apprentice. Addison-Wesley Publishing Company, Reading, MA.
10.1145/3498326 · DOI
10.1145/964442.964518 · doi:10.1145/964442.964518
10.1145/964442.964536 · DOI
Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised Translation of Programming Languages. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 20601–20611.
Harvey Sacks. 1984. Notes on methodology. In Structures of Social Action: Studies in Conversation Analysis, John Heritageand J. Maxwell Atkinson (Eds.). Cambridge University Press, Cambridge, 2–27.
Nithya Sambasivan and Rajesh Veeraraghavan. 2022. The Deskilling of Domain Expertise in AI Development. In CHI Conference on Human Factors in Computing Systems. 1–14.
10.1145/3411764.3445544 · DOI
10.1177/1555343416682891 · DOI
10.1145/1180875.1180890 · DOI
10.17705/1thci.00131 · DOI · Paper page
Ben Shneiderman. 2022. Human-Centered AI. Oxford University Press. · DOI · Paper page
Kurt Shuster Jing Xu Mojtaba Komeili Da Ju Eric Michael Smith Stephen Roller Megan Ung Moya Chen Kushal Arora Joshua Lane 2022. BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188(2022). · arXiv
10.1145/3148330.3149407 · DOI
10.1109/MS.2012.61 · doi:10.1109/ms.2012.61
10.1145/2950290.2983989 · DOI
10.1145/3465074.3465080 · DOI
Michele Tufano Dawn Drain Alexey Svyatkovskiy Shao Kun Deng and Neel Sundaresan. 2020. Unit Test Case Generation with Transformers and Focal Context. arXiv preprint arXiv:2009.05617(2020). · arXiv
Severi Uusitalo, Anna Kantosalo, Antti Salovaara, Tapio Takala, and Christian Guckelsberger. 2022. Co-creative Product Design with Interactive Evolutionary Algorithms: A Practice-Based Reflection. In International Conference on Computational Intelligence in Music, Sound, Art and Design (Part of EvoStar). Springer, 292–307.
10.1145/3332165.3347944 · DOI
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Vol. 30. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
10.1145/3238147.3238206 · DOI · Paper page
10.1145/3489465 · DOI
10.1145/3025453.3025876 · DOI
Justin D Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021. Perfection Not Required? Human-AI Partnerships in Code Translation. In 26th International Conference on Intelligent User Interfaces. 402–412.
10.1145/3490099.3511157 · DOI · Paper page
Aditya Ankur Yadav, Ishan Garg, and Dr. Pratistha Mathur. 2019. PACT - Programming Assistant ChaTbot. In 2019 2nd International Conference on Intelligent Communication and Computational Techniques (ICCT). 131–136.
10.1145/3373017.3373028 · DOI
10.2196/11013 · DOI

Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
Large Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
A Survey on Large Language Models for Code Generation
Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3747588 · Cited by 251

See all 258

Authors: Ross, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D.
Venue: Proceedings of the 28th International Conference on Intelligent User Interfaces
Publisher: ACM
Published: 2023-03-27
Pages: 491-514
DOI: 10.1145/3581641.3584037
OpenAlex: W4321013654
Cited by: 258
References: 34
FWCI: 68.76
Type: conference paper
Language: en
Open access: hybrid
Licence: creativecommons.org/licenses/by-nc-nd/4.0/ (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page: CoChat. (2026). Citation check: The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development. https://app.cochat.ai/papers/10.1145/3581641.3584037
The paper: Ross, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023). The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development. Proceedings of the 28th International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3581641.3584037

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Self-check

Citations

Cites

Cited by

Metadata

Record

About these checks

How to cite

Data sources