CoChat
Sign Up
PreprintCited by 27Software Engineering ResearchSoftware Engineering Techniques and PracticesUsability and User Interface Design

Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions

Open access
Vasconcelos, Helena, Bansal, Gagan, Fourney, Adam, Liao, Q. Vera, Vaughan, Jennifer Wortman·2023·arXiv
DOI 10.48550/arxiv.2302.07248arXiv 2302.07248
Open this paper in CoChat — free accountOpen PDF

Abstract

Large-scale generative models enabled the development of AI-powered code completion tools to assist programmers in writing code. However, much like other AI-powered tools, AI-powered code completions are not always accurate, potentially introducing bugs or even security vulnerabilities into code if not properly detected and corrected by a human programmer. One technique that has been proposed and implemented to help programmers identify potential errors is to highlight uncertain tokens. However, there have been no empirical studies exploring the effectiveness of this technique -- nor investigating the different and not-yet-agreed-upon notions of uncertainty in the context of generative models. We explore the question of whether conveying information about uncertainty enables programmers to more quickly and accurately produce code when collaborating with an AI-powered code completion tool, and if so, what measure of uncertainty best fits programmers' needs. Through a mixed-methods study with 30 programmers, we compare three conditions: providing the AI system's code completion alone, highlighting tokens with the lowest likelihood of being generated by the underlying generative model, and highlighting tokens with the highest predicted likelihood of being edited by a programmer. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits, and is subjectively preferred by study participants. In contrast, highlighting tokens according to their probability of being generated does not provide any benefit over the baseline with no highlighting. We further explore the design space of how to convey uncertainty in AI-powered code completion tools, and find that programmers prefer highlights that are granular, informative, interpretable, and not overwhelming.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2302.07248

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In a sample of undergraduate students with programming experience, working with Copilot on realistic programming tasks was perceived to be challenging (although participants still strongly preferred it).

    “In a sample of undergraduate students with programming experience, working with Copilot on realistic programming tasks was perceived to be challenging (although participants still strongly preferred it) [188].”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [188]

    The Study Design section says the 30 participants were employees of a US technology company, not undergraduate students. Section 3.3 says they worked with Codex-generated completions, and section 7.2 describes the tasks as timed LeetCode challenges that may not represent day-to-day programming. These passages contradict the claim’s description of the study; they do not establish that participants strongly preferred working with Copilot. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Vasconcelos et al., Spatharioti et al., and Zhou et al. found results supporting uncertainty expression as a promising technique to encourage appropriate reliance, particularly when uncertainty estimates are well calibrated.

    “These studies' results support uncertainty expression as a promising technique to encourage appropriate reliance, particularly when uncertainty estimates are well calibrated.”

    From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as Vasconcelos et al. [101]

    The Introduction reports that edit-model uncertainty highlighting led to faster task completion and more targeted edits, supporting its promise in AI code completion. Section 7.1 describes the edit model as a best-case, highly calibrated scenario. But Section 7.3 says it remains to be demonstrated whether those benefits increase oversight or reduce automation bias, so the study does not establish improved appropriate reliance. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Programmers were reluctant to deeply review and repair AI-generated code, preferring instead to re-write the entire code themselves.

    “Programmers were reluctant to deeply review and repair AI-generated code, preferring instead to re-write the entire code themselves [188].”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [188]

    In “Granularity of uncertainty highlighting” (page unavailable), one participant said they would probably discard and rewrite a completion if its score were low. That conditional remark does not establish that programmers generally preferred rewriting over reviewing and repairing generated code; Section 5.2 instead reports participants making targeted edits. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The research community has called for LLMs and LLM-infused applications to express the uncertainty of outputs to end users.

    “As one approach to reduce overreliance, the research community has called for LLMs and LLM-infused applications to express the uncertainty of outputs to end users [6, 56, 57, 73, 101, 120].”

    From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [101]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901.
  2. Carrie J Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. 2019. " Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making. Proceedings of the ACM on Human-computer Interaction 3, CSCW (2019), 1-24.
  3. Tuhin Chakrabarty, Vishakh Padmakumar, and He He. 2022. Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writing. arXiv preprint arXiv:2210.13669 (2022). · arXiv
  4. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning. PMLR, 1321-1330.
  5. Daniel D Johnson, Daniel Tarlow, and Christian Walder. 2023. RU-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intents. arXiv preprint arXiv:2303.00732 (2023). · arXiv
  6. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 (2022). · arXiv
  7. Adam Khakhar, Stephen Mell, and Osbert Bastani. 2023. PAC Prediction Sets for Large Language Models of Code. arXiv:2302.08703 [cs.LG] · arXiv
  8. Maia Kotelanski, Robert Gallo, Ashwin Nayak, and Thomas Savage. 2023. Methods to Estimate Large Language Model Confidence. arXiv:2312.03733 [cs.CL] · arXiv
  9. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664 (2023). · arXiv
  10. Vivian Lai, Han Liu, and Chenhao Tan. 2020. "Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (2020).
  11. Cynthia CS Liem, Markus Langer, Andrew Demetriou, Annemarie MF Hiemstra, Achmadnoer Sukma Wicaksana, Marise Ph Born, and Cornelius J König. 2018. Psychology Meets Machine Learning: Interdisciplinary Perspectives on Algorithmic Job Candidate Screening. In Explainable and Interpretable Models in Computer Vision and Machine Learning. Springer, 197-253.
  12. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334 (2022). · arXiv
  13. Genglin Liu, Xingyao Wang, Lifan Yuan, Yangyi Chen, and Hao Peng. 2023. Prudent Silence or Foolish Babble? Examining Large Language Models' Responses to the Unknown. arXiv:2311.09731 [cs.CL] · arXiv
  14. Ryan Louie, Andy Coenen, Cheng-Zhi Anna Huang, Michael Terry, and Carrie J. Cai. 2020. Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (2020).
  15. Scott M Lundberg, Bala Nair, Monica S Vavilala, Mayumi Horibe, Michael J Eisses, Trevor Adams, David E Liston, Daniel King-Wai Low, Shu-Fang Newman, Jerry Kim, et al. 2018. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature biomedical engineering 2, 10 (2018), 749-760.
  16. Marianna Martindale, Marine Carpuat, Kevin Duh, and Paul McNamee. 2019. Identifying fluently inadequate output in neural and statistical machine translation. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track. 233-243.
  17. Hussein Mozannar, Arvindmani Satyanarayan, and David A. Sontag. 2021. Teaching Humans When To Defer to a Classifier via Examplars. In AAAI.
  18. Raja Parasuraman and Dietrich H Manzey. 2010. Complacency and bias in human use of automation: An attentional integration. Human factors 52, 3 (2010), 381-410.
  19. Rohith Pudari and Neil A. Ernst. 2023. From Copilot to Pilot: Towards AI Supported Software Development. arXiv:2303.04142 [cs.SE] · arXiv
  20. Advait Sarkar, Andrew D Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with artificial intelligence? arXiv preprint arXiv:2208.06213 (2022). · arXiv
  21. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. arXiv:2206.05802 [cs.CL] · arXiv
  22. Vaishnavi Shrivastava, Percy Liang, and Ananya Kumar. 2023. Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimation. arXiv:2311.08877 [cs.CL] · arXiv
  23. Aniket Kumar Singh, Suman Devkota, Bishal Lamichhane, Uttam Dhakal, and Chandra Dhakal. 2023. The Confidence-Competence Gap in Large Language Models: A Cognitive Study. arXiv:2309.16145 [cs.CL] · arXiv
  24. Sree Harsha Tanneru, Chirag Agarwal, and Himabindu Lakkaraju. 2023. Quantifying Uncertainty in Natural Language Explanations of Large Language Models. arXiv:2311.03533 [cs.CL] · arXiv
  25. Anne Marthe Van der Bles, Sander Van Der Linden, Alexandra LJ Freeman, James Mitchell, Ana B Galvao, Lisa Zaval, and David J Spiegelhalter. 2019. Communicating uncertainty about facts, numbers and science. Royal Society open science 6, 5 (2019), 181870.
  26. Christopher D Wickens, Benjamin A Clegg, Alex Z Vieane, and Angelia L Sebok. 2015. Complacency and automation bias in the use of imperfect automation. Human factors 57, 5 (2015), 728-739.
  27. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework. arXiv:2308.08155 [cs.AI] · arXiv
  28. Ming Yin, Jennifer Wortman Vaughan, and Hanna M. Wallach. 2019. Understanding the Effect of Accuracy on Trust in Machine Learning Models. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (2019).
  29. Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023. Navigating the grey area: Expressions of overconfidence and uncertainty in language models. arXiv preprint arXiv:2302.13439 (2023). · arXiv

Cited by

Sorted by citations

Related

Authors
Vasconcelos, H., Bansal, G., Fourney, A., Liao, Q. V., & Vaughan, J. W.
Affiliations
  • Stanford University (US)
  • Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
1073-0516 (ISSN-L)
DOI
10.48550/arxiv.2302.07248
arXiv
2302.07248
OpenAlex
W4403860276
Cited by
23
References
34
Citations by year
2026: 4 · 2025: 10 · 2024: 8 · 2023: 1
FWCI
8.30
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • ACM Transactions on Computer-Human Interaction (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
generative models, human-AI collaboration, uncertainty visualization, user preferences

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review