Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 258

The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.

The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development

Open access
Ross, Steven I., Martinez, Fernando, Houde, Stephanie, Muller, Michael, Weisz, Justin D.·2023·Proceedings of the 28th International Conference on Intelligent User Interfaces
DOI 10.1145/3581641.3584037
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) have recently been applied in software engineering to perform tasks such as translating code between programming languages, generating code from natural language, and autocompleting code as it is being written. When used within development tools, these systems typically treat each model invocation independently from all previous invocations, and only a specific limited functionality is exposed within the user interface. This approach to user interaction misses an opportunity for users to more deeply engage with the model by having the context of their previous interactions, as well as the context of their code, inform the model’s responses. We developed a prototype system – the Programmer’s Assistant – in order to explore the utility of conversational interactions grounded in code, as well as software engineers’ receptiveness to the idea of conversing with, rather than invoking, a code-fluent LLM. Through an evaluation with 42 participants with varied levels of programming experience, we found that our system was capable of conducting extended, multi-turn discussions, and that it enabled additional knowledge and capabilities beyond code generation to emerge from the LLM. Despite skeptical initial expectations for conversational programming assistance, participants were impressed by the breadth of the assistant’s capabilities, the quality of its responses, and its potential for improving their productivity. Our work demonstrates the unique potential of conversational interactions with LLMs for co-creative processes like software development.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3581641.3584037

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 34 references

Research works printed but not linked: 68

  1. 10.1109/MS.2017.31 · DOI
  2. Safinah Ali Nisha Elizabeth Devasia and Cynthia Breazeal. 2022. Escape! Bot: Social Robots as Creative Problem-Solving Partners. In Creativity and Cognition. 275–283. · DOI
  3. 10.1145/3212695 · DOI · Paper page
  4. 10.1109/VISSOFT.2013.6650527 · DOI
  5. 10.1145/257089.257151 · DOI
  6. Amanda Askell Yuntao Bai Anna Chen Dawn Drain Deep Ganguli Tom Henighan Andy Jones Nicholas Joseph Ben Mann Nova DasSarma 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861(2021). · arXiv
  7. 10.1145/3209978.3210027 · DOI
  8. Sallyann Bryant, Pablo Romero, and Benedict" du Boulay. 2006. The Collaborative Nature of Pair Programming. In Extreme Programming and Agile Processes in Software Engineering, Pekka Abrahamsson, Michele Marchesi, and Giancarlo Succi (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 53–64.
  9. Gaetano Cascini Yukari Nagai Georgi V Georgiev Jader Zelaya Niccolò Becattini Jean-François Boujut Hernan Casakin Nathan Crilly Elies Dekoninck John Gero 2022. Perspectives on design creativity and innovation research: 10 years later. 30 pages. · DOI
  10. Stephen Cass. 2022. Top Programming Languages 2022. IEEE Spectrum (23 Aug 2022). https://spectrum.ieee.org/top-programming-languages-2022
  11. Cristina Catalan Aguirre Nuria Gonzalez Castro Carlos Delgado Kloos Carlos Alario-Hoyos and Pedro José Muñoz Merino. 2021. Conversational agent for supporting learners on a MOOC on programming with Java. (2021). · DOI
  12. 10.1145/966789.966803 · DOI
  13. 10.1109/APSEC.2005.22 · DOI
  14. 10.1007/978-3-642-39646-5_8 · DOI
  15. 10.1145/1066129.1066132 · DOI
  16. Bobbie Eicher, Kathryn Cunningham, Sydni Peterson Marissa Gonzales, and Ashok Goel. 2017. Toward mutual theory of mind as a foundation for co-creation. In International Conference on Computational Creativity, Co-Creation Workshop.
  17. Stephen M Fiore, Eduardo Salas, and Janis A Cannon-Bowers. 2001. Group dynamics and shared mental model development. How people evaluate others in organizations 234 (2001).
  18. Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 1995. Design patterns: elements of reusable object-oriented software. Addison-Wesley.
  19. Amelia Glaese Nat McAleese Maja Trębacz John Aslanides Vlad Firoiu Timo Ewalds Maribeth Rauh Laura Weidinger Martin Chadwick Phoebe Thacker Lucy Campbell-Gillingham Jonathan Uesato Po-Sen Huang Ramona Comanescu Fan Yang Abigail See Sumanth Dathathri Rory Greig Charlie Chen Doug Fritz Jaume Sanchez Elias Richard Green Soňa Mokrá Nicholas Fernando Boxi Wu Rachel Foley Susannah Young Iason Gabriel William Isaac John Mellor Demis Hassabis Koray Kavukcuoglu Lisa Anne Hendricks and Geoffrey Irving. 2022. Improving alignment of dialogue agents via targeted human judgements. https://arxiv.org/abs/2209.14375 · arXiv
  20. 10.14236/ewic/HCI2018.103 · DOI
  21. 10.1007/s10664-019-09730-9 · DOI · Paper page
  22. Edwin L Hutchins, James D Hollan, and Donald A Norman. 1985. Direct manipulation interfaces. Human–computer interaction 1, 4 (1985), 311–338.
  23. 10.18653/v1/P16-1195 · DOI · Paper page
  24. Andreas Jedlitschka and Markus Nick. 2003. Software Engineering Knowledge Repositories. Springer Berlin Heidelberg, Berlin, Heidelberg, 55–80.
  25. 10.1145/3411764.3445659 · DOI · Paper page
  26. Lauramaria Laine. 2021. Exploring Advertising Creatives’ Attitudes Towards Human-AI Collaboration. (2021).
  27. 10.1002/hbe2.266 · DOI
  28. 10.1145/263552.263612 · DOI
  29. John E Mathieu, Tonia S Heffner, Gerald F Goodwin, Eduardo Salas, and Janis A Cannon-Bowers. 2000. The influence of shared mental models on team process and performance.Journal of applied psychology 85, 2 (2000), 273.
  30. Ekaterina A Moroz, Vladimir O Grizkevich, and Igor M Novozhilov. 2022. The Potential of Artificial Intelligence as a Method of Software Developer’s Productivity Improvement. In 2022 Conference of Russian Young Researchers in Electrical and Electronic Engineering (ElConRus). IEEE, 386–390.
  31. Michael Muller, Stevean Ross, Stephanie Houde, Mayank Agarwal, Fernando Martinez, John Richards, Kartik Talamadupula, and Justin D Weisz. 2022. Drinking Chai with Your (AI) Programming Partner: A Design Fiction about Generative AI for Software Engineering. HAI-GEN Workshop at IUI 2022: 3rd Workshop on Human-AI Co-Creation with Generative Models (2022). https://hai-gen.github.io/2022/
  32. 10.1145/2662253.2662300 · doi:10.1145/2662253.2662300
  33. 10.1145/3126594.3126628 · DOI
  34. Martin Nordio H Estler Carlo A Furia Bertrand Meyer 2011. Collaborative software development on the web. arXiv preprint arXiv:1105.0768(2011). · arXiv
  35. Maxwell Nye, Anders Andreassen, Guy Gur-Ari, Henryk Witold Michalewski, Jacob Austin, David Bieber, David Martin Dohan, Aitor Lewkowycz, Maarten Paul Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show Your Work: Scratchpads for Intermediate Computation with Language Models. https://arxiv.org/abs/2112.00114. · arXiv
  36. Peter Pirolli and Stuart Card. 1999. Information foraging.Psychological review 106, 4 (1999), 643.
  37. 10.1145/92755.92783 · doi:10.1145/92755.92783
  38. Mawarny Md Rejab, James Noble, and George Allan. 2014. Distributing Expertise in Agile Software Development Projects. In 2014 Agile Conference. 33–36.
  39. Jeba Rezwana and Mary Lou Maher. 2021. COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systems.. In ICCC. 444–448.
  40. Charles H. Rich and Richard C. Waters. 1990. The Programmer’s Apprentice. Addison-Wesley Publishing Company, Reading, MA.
  41. 10.1145/3498326 · DOI
  42. 10.1145/964442.964518 · doi:10.1145/964442.964518
  43. 10.1145/964442.964536 · DOI
  44. Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised Translation of Programming Languages. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 20601–20611.
  45. Harvey Sacks. 1984. Notes on methodology. In Structures of Social Action: Studies in Conversation Analysis, John Heritageand J. Maxwell Atkinson (Eds.). Cambridge University Press, Cambridge, 2–27.
  46. Nithya Sambasivan and Rajesh Veeraraghavan. 2022. The Deskilling of Domain Expertise in AI Development. In CHI Conference on Human Factors in Computing Systems. 1–14.
  47. 10.1145/3411764.3445544 · DOI
  48. 10.1177/1555343416682891 · DOI
  49. 10.1145/1180875.1180890 · DOI
  50. 10.17705/1thci.00131 · DOI · Paper page
  51. Ben Shneiderman. 2022. Human-Centered AI. Oxford University Press. · DOI · Paper page
  52. Kurt Shuster Jing Xu Mojtaba Komeili Da Ju Eric Michael Smith Stephen Roller Megan Ung Moya Chen Kushal Arora Joshua Lane 2022. BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188(2022). · arXiv
  53. 10.1145/3148330.3149407 · DOI
  54. 10.1109/MS.2012.61 · doi:10.1109/ms.2012.61
  55. 10.1145/2950290.2983989 · DOI
  56. 10.1145/3465074.3465080 · DOI
  57. Michele Tufano Dawn Drain Alexey Svyatkovskiy Shao Kun Deng and Neel Sundaresan. 2020. Unit Test Case Generation with Transformers and Focal Context. arXiv preprint arXiv:2009.05617(2020). · arXiv
  58. Severi Uusitalo, Anna Kantosalo, Antti Salovaara, Tapio Takala, and Christian Guckelsberger. 2022. Co-creative Product Design with Interactive Evolutionary Algorithms: A Practice-Based Reflection. In International Conference on Computational Intelligence in Music, Sound, Art and Design (Part of EvoStar). Springer, 292–307.
  59. 10.1145/3332165.3347944 · DOI
  60. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Vol. 30. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  61. 10.1145/3238147.3238206 · DOI · Paper page
  62. 10.1145/3489465 · DOI
  63. 10.1145/3025453.3025876 · DOI
  64. Justin D Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021. Perfection Not Required? Human-AI Partnerships in Code Translation. In 26th International Conference on Intelligent User Interfaces. 402–412.
  65. 10.1145/3490099.3511157 · DOI · Paper page
  66. Aditya Ankur Yadav, Ishan Garg, and Dr. Pratistha Mathur. 2019. PACT - Programming Assistant ChaTbot. In 2019 2nd International Conference on Intelligent Communication and Computational Techniques (ICCT). 131–136.
  67. 10.1145/3373017.3373028 · DOI
  68. 10.2196/11013 · DOI

Cited by

Metadata

Authors
Ross, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D.
Venue
Proceedings of the 28th International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2023-03-27
Pages
491-514
DOI
10.1145/3581641.3584037
OpenAlex
W4321013654
Cited by
258
References
34
FWCI
68.76
Type
conference paper
Language
en
Open access
hybrid
Licence
creativecommons.org/licenses/by-nc-nd/4.0/ (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.