When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Open accessAbstract
AI powered code-recommendation systems, such as Copilot and CodeWhisperer, provide code suggestions inside a programmer's environment (e.g., an IDE) with the aim of improving productivity. We pursue mechanisms for leveraging signals about programmers' acceptance and rejection of code suggestions to guide recommendations. We harness data drawn from interactions with GitHub Copilot, a system used by millions of programmers, to develop interventions that can save time for programmers. We introduce a utility-theoretic framework to drive decisions about suggestions to display versus withhold. The approach, conditional suggestion display from human feedback (CDHF), relies on a cascade of models that provide the likelihood that recommended code will be accepted. These likelihoods are used to selectively hide suggestions, reducing both latency and programmer verification time. Using data from 535 programmers, we perform a retrospective evaluation of CDHF and show that we can avoid displaying a significant fraction of suggestions that would have been rejected. We further demonstrate the importance of incorporating the programmer's latent unobserved state in decisions about when to display suggestions through an ablation study. Finally, we showcase how using suggestion acceptance as a reward signal for guiding the display of suggestions can lead to suggestions of reduced quality, indicating an unexpected pitfall.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Mozannar et al. do not consider that some completions, despite being rejected, may help guide the user in their thinking.
“and do not consider that some completions, despite being rejected, may help guide the user in their thinking.”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as Mozannar et al. [32]
Mozannar et al. propose a relatively complex ensemble model that can incur additional latency by filtering after a completion is generated.
“However, they propose a relatively complex ensemble model that can incur additional latency by filtering after a completion is generated”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as Mozannar et al. [32]
Mozannar et al.'s filter can depend on a completion being generated, which is guaranteed to incur additional latency and compute in practice.
“Additionally, their filter can depend on a completion being generated, which is guaranteed to incur additional latency and compute in practice.”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as Mozannar et al. [32]
Optimising the acceptance rate results in shorter and typically less useful suggestions.
“however, this is critiqued by Mozannar et al. [32], whom find that optimising the acceptance rate results in shorter and typically less useful suggestions.”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as Mozannar et al. [32]
Mozannar et al.'s filter can hide 53% of completions with the guarantee 91% would have been rejected by the user.
“They cite it can hide 53% of completions with the guarantee 91% would have been rejected by the user.”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as Mozannar et al. [32]
Related work shows promising results in personalising the invocation-filtering system.
“related work shows promising results in personalising the invocationfiltering system [5, 32].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [32]
Cites
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · cited 1× · see citing works
“…(e.g., suggestion length), previous features of the last five suggestions shown, the confidence reported by Copilot, an embedding using codeBERTa (Feng et al. 2020) of prompt and suggestion, presence of Python keywords (e.g., import , def try , etc.), and the output of the Tree-sitter Parser…”
Backgroundxgboost: Extreme Gradient Boosting
Chen, T., He, T., Benesty, M., Khotilovich, V., Tang, Y., Cho, H., Chen, K., Mitchell, R., Cano, I., Zhou, T., et al. (2014) · doi:10.32614/cran.package.xgboost · Cited by 2,735 · see citing works
Flow and the Foundations of Positive Psychology: The Collected Works of Mihaly Csikszentmihalyi
Csikszentmihalyi, M. (2014) · doi:10.1007/978-94-017-9088-8 · Cited by 1,653 · cited 1× · see citing works
“Second, requiring an explicit invocation can disrupt the natural flow of programming, breaking a state of flow achieved during intensive focus (Csikszentmihalyi and Larson 2014).”
BackgroundPrinciples of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · see citing works
Self-citationEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 2× · see citing works
“Code-recommendation systems are powered by large language models (LLMs) such as GPT that are trained on standard language modeling objectives using the Common Crawl data (Radford et al. 2019), and then fine-tuned on public code repositories (Chen et al. 2021).”
MethodCompetition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 1× · see citing works
“…performance of code recommendation models, but these typically assess how well the model can complete code in an offline setting without developer input rather than evaluating how well it assists programmers in situ (Ziegler et al. 2022; Li et al. 2022; Evtikhiev et al. 2022; Dakhel et al. 2022).”
BackgroundObtaining well calibrated probabilities using Bayesian binning
Pakdaman Naeini, M., Cooper, G., & Hauskrecht, M. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9602 · Cited by 991 · cited 1× · see citing works
“The model is also well calibrated: the expected calibration error is 0.10 (Naeini, Cooper, and Hauskrecht 2015).”
MethodInfluentialBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 1× · see citing works
“…to ask may lead to sub-optimal interactions because its success would rely on programmers having an accurate mental model of Copilot abilities (Sarkar et al. 2022) which can require long-term interactions with the model (Bansal et al. 2019) or training (Mozannar, Satyanarayan, and Sontag 2022).”
BackgroundSelf-citationTraining a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · cited 1× · see citing works
“…to ask may lead to sub-optimal interactions because its success would rely on programmers having an accurate mental model of Copilot abilities (Sarkar et al. 2022) which can require long-term interactions with the model (Bansal et al. 2019) or training (Mozannar, Satyanarayan, and Sontag 2022).”
BackgroundGitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417 · see citing works
Fine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413 · see citing works
(2001) · Cited by 341 · see citing works
Self-citationAttention-Sensitive AlertingPreprint
Horvitz, E., Jacobs, A., & Hovel, D. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.6707 · Cited by 317 · cited 2× · see citing works
“Designs requiring user initiative as well as those automatically displaying content can burden users with interruptions that decrease task performance (Bailey, Konstan, and Carlis 2001; Cutrell, Czerwinski, and Horvitz 2001).”
BackgroundSelf-citationDirect Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2305.18290 · Cited by 293 · see citing works
Learning and reasoning about interruption
Horvitz, E., & Apacible, J. (2003) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/958432.958440 · Cited by 275 · cited 1× · see citing works
“Designs requiring user initiative as well as those automatically displaying content can burden users with interruptions that decrease task performance (Bailey, Konstan, and Carlis 2001; Cutrell, Czerwinski, and Horvitz 2001).”
BackgroundSelf-citation(2001) · Cited by 269 · see citing works
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200 · cited 2× · see citing works
“…performance of code recommendation models, but these typically assess how well the model can complete code in an offline setting without developer input rather than evaluating how well it assists programmers in situ (Ziegler et al. 2022; Li et al. 2022; Evtikhiev et al. 2022; Dakhel et al. 2022).”
BackgroundInteractive Learning from Policy-Dependent Human Feedback
MacGlashan, J., Ho, M. K., Loftin, R., Peng, B., Wang, G., Roberts, D., Taylor, M. E., & Littman, M. L. (2017) · arXiv · doi:10.48550/arxiv.1701.06049 · Cited by 163 · cited 1× · see citing works
“Integrating human preferences when training machine learning models has long been studied in the literature (Knox and Stone 2008; MacGlashan et al. 2017).”
BackgroundOut of the BLEU: how should we assess quality of the Code Generation models?
Evtikhiev, M., Bogomolov, E., Sokolov, Y., & Bryksin, T. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111741 · Cited by 127 · cited 1× · see citing works
“…performance of code recommendation models, but these typically assess how well the model can complete code in an offline setting without developer input rather than evaluating how well it assists programmers in situ (Ziegler et al. 2022; Li et al. 2022; Evtikhiev et al. 2022; Dakhel et al. 2022).”
BackgroundTAMER: Training an Agent Manually via Evaluative Reinforcement
Bradley Knox, W., & Stone, P. (2008) · 2008 7th IEEE International Conference on Development and Learning · doi:10.1109/devlrn.2008.4640845 · Cited by 117 · cited 1× · see citing works
“Integrating human preferences when training machine learning models has long been studied in the literature (Knox and Stone 2008; MacGlashan et al. 2017).”
BackgroundReading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114 · cited 2× · see citing works
“This may be due to the high variance and un-observed confounders governing verification time.”
BackgroundMethodSelf-citationReading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641936 · Cited by 114 · see citing works
Self-citationConsistent Estimators for Learning to Defer to an Expert
Mozannar, H., & Sontag, D. (2020) · arXiv · doi:10.48550/arxiv.2006.01862 · Cited by 61 · cited 1× · see citing works
“Finally, the optimization problem in (5) is amenable to procedures inspired by learning to defer (Mozannar and Sontag 2020) that can outperform the post-hoc procedure proposed.”
Self-citationWhat is it like to program with artificial intelligence?
Sarkar, A., Gordon, A. D., Negreanu, C., Poelitz, C., Ragavan, S. S., & Zorn, B. (2022) · arXiv · doi:10.48550/arxiv.2208.06213 · Cited by 47 · see citing works
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · see citing works
When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28878 · Cited by 23 · see citing works
Self-citationLearning and reasoning about interruption
Horvitz, E., & Apacible, J. (2003) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/958439.958440 · Cited by 13 · see citing works
Self-citationTeaching Humans When to Defer to a Classifier via Exemplars
Mozannar, H., Satyanarayan, A., & Sontag, D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i5.20469 · Cited by 5 · cited 1× · see citing works
“…to ask may lead to sub-optimal interactions because its success would rely on programmers having an accurate mental model of Copilot abilities (Sarkar et al. 2022) which can require long-term interactions with the model (Bansal et al. 2019) or training (Mozannar, Satyanarayan, and Sontag 2022).”
BackgroundSelf-citationSun, Z., Du, X., Song, F., Wang, S., Ni, M., Li, L., & Lo, D. (2022) · arXiv · doi:10.48550/arxiv.2209.05948 · Cited by 3 · cited 5× · see citing works
“hide code suggestions was proposed in [Sun et al., 2022] (Quality Estimation Before Completion, QEBC).”
BackgroundMethodInfluential
Flagged first, then by citations
Cited by
Sorted by citations
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147 · see citing works
Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward
Sergeyuk, A., Golubev, Y., Bryksin, T., & Ahmed, I. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2024.107610 · Cited by 101 · see citing works
“…(raised in 3% of all responses) and Workflow disruption (raised in 2.1% of responses) should be addressed by companies who provide AI assistants for the market, considering their hands-on access to the systems and their users for more quick and meaningful research and innovation [45], [46].”
BackgroundInfluentialThe Rise of the AI Co-Pilot: Lessons for Design from Aviation and Beyond
Sellen, A. J., & Horvitz, E. (2024) · Communications of the ACM · doi:10.1145/3637865 · Cited by 43 · see citing works
“19 ˲ Support users in carrying out aspects of tasks that allow for self-expression, creativity, social judgment, and navigating complex situations that extend beyond AI capabilities.”
BackgroundSelf-citationLAN: Learning Adaptive Neighbors for Real-Time Insider Threat Detection
Cai, X., Wang, Y., Xu, S., Li, H., Zhang, Y., Liu, Z., & Yuan, X. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3488527 · Cited by 40 · see citing works
Need Help? Designing Proactive AI Assistants for Programming
Chen, V., Zhu, A., Zhao, S., Mozannar, H., Sontag, D. A., & Talwalkar, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714002 · Cited by 29 · see citing works
“While there has been less focus on investigating the use of chat assistants for programming, it is important to note that they complement assistance provided by code completions [28] and are the focus of this work.”
Self-citationWhen to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28878 · Cited by 23 · see citing works
Self-citationReading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114 · see citing works
“Moreover, the data collected in this work was labeled by the participants themselves rather than by the researchers interpreting their actions, allowing for more faithful intent and activity labeling and the data collected in our study can also be used to build predictive models as in [Mozannar et al., 2023].”
MethodInfluentialSelf-citationEverett, S. S., Bunning, B. J., Jain, P., Lopez, I., Agarwal, A., Desai, M., Gallo, R., Goh, E., Kadiyala, V. B., Kanjee, Z., et al. (2025) · doi:10.1101/2025.06.07.25329176 · Cited by 18 · see citing works
Self-citationHuman-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 26 · see citing works
“Despite the existence of automated checks and static analysis, most quality control remains a human responsibility (Mozannar et al., 2024a; Tang et al., 2024).”
InfluentialVindigni, G. (2024) · European Journal of Contemporary Education and E-Learning · doi:10.59324/ejceel.2024.2(4).10 · Cited by 15 · see citing works
“…with Csikszentmihalyi's Flow Theory (2014) and Nielsen's "10 Usability Heuristics" (1994), as they enhance user engagement and involvement through an improved interaction experience (Nielsen & Mack, 1994: 44–53; Vicente-Yagüe-Jara et al., 2023; Mozannar et al., 2024; Soares et al., 2022: 83– 89).”
Background"My productivity is boosted, but …" Demystifying Users’ Perception on AI Coding Assistants
Lyu, Y., Yang, Z., Shi, J., Chang, J., Liu, Y., & Lo, D. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00024 · Cited by 9 · see citing works
“Existing work on suggestion presentation [64], [65] can inform future research.”
A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
de Moor, A., van Deursen, A., & Izadi, M. (2024) · Proceedings of the 1st ACM International Conference on AI-Powered Software · doi:10.1145/3664646.3664760 · Cited by 7 · see citing works
A Workflow for Building Computationally Rational Models of Human Behavior
Chandramouli, S., Shi, D., Putkonen, A., Peuter, S. D., Zhang, S., Jokinen, J., Howes, A., & Oulasvirta, A. (2024) · Computational Brain & Behavior · doi:10.1007/s42113-024-00208-6 · Cited by 7 · see citing works
“By approximating a person’s mental state and processes, computational rationality models can make a highly valuable contribution to adaptive interactive systems that strategically intervene to assist the user; for discussion, see the work of Mozannar et al. (2023).”
Trust Dynamics in AI-Assisted Development: Definitions, Factors, and Implications
Sabouri, S., Eibl, P., Zhou, X.-Y., Ziyadi, M., Medvidović, N., Lindemann, L., & Chattopadhyay, S. (2025) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00199 · Cited by 17 · see citing works
“To enhance the trustworthiness of AI models, some have proposed selectively displaying results to users based on specific code characteristics [31].”
When Are Two Lists Better than One?: Benefits and Harms in Joint Decision-Making
Donahue, K., Gollapudi, S., & Kollias, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28866 · Cited by 4 · see citing works
Flows: Building Blocks of Reasoning and Collaborating AIPreprint
Josifoski, M., Klein, L., Peyrard, M., Baldwin, N., Li, Y., Geng, S., Schnitzler, J. P., Yao, Y., Wei, J., Paul, D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01285 · Cited by 14 · see citing works
Architecting open, accountable, and trustworthy AI-IDEs
Contreras, A., Guerra, E., & de Lara, J. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00608-x · Cited by 4 · see citing works
Penumajji, N. (2025) · IEEE International Conference on Electronics Information and Emergency Communication · doi:10.1109/iceiec65904.2025.11273141 · Cited by 4 · see citing works
Assessing GitHub Copilot in Solidity Development: Capabilities, Testing, and Bug Fixing
Baralla, G., Ibba, G., & Tonelli, R. (2024) · IEEE Access · doi:10.1109/access.2024.3486365 · Cited by 4 · see citing works
“Mozannar et al. [22] evaluate data drawn from interactions with GitHub Copilot, introducing a utility-theoretic framework to drive decisions about suggestions to display versus withhold.”
MethodTools for Implementing Multi-Agent Systems Based on Protocols
Chopra, A. K., Christie, S. H., V, & Singh, M. P. (2026) · Springer eBooks · doi:10.1007/978-3-032-01082-7_8 · Cited by 2 · see citing works
Awad, M. N. A., Ivanov, S. V., & Tikhonova, O. B. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00032 · Cited by 1 · see citing works
“Treating all editing contexts as equally appropriate for suggestion triggering is both computationally expensive and interactionally naive [2].”
Optimizing LLM Code Suggestions: Feedback-Driven Timing with Lightweight State Bounds
Awad, M. N. A., Ivanov, S., & Tikhonova, O. B. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00049
“Recent proposals such as [16] and [17] explore proactive suggestion timing, proposing conceptual frameworks for timing-aware interactions, but do not implement bounded, feedback-driven controllers in real-world systems.”
OptiMine: Scalable and Precise Code Optimization for Android Apps via LLM-Driven Semantic Analysis
Du, P., Yi, Q., Zhang, & Liang, H. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832215
“Recent advances in LLMs demonstrate impressive capabilities in code understanding and reasoning [26, 32, 33, 35], yet directly applying LLMs to optimization discovery raises concerns about hallucination, limited coverage, and the lack of program-semantic grounding required for exhaustive analysis.”
"Great! The Next Step Is...": In Pursuit of Proactive Assistance with Multimodal Foundation Models
Andrist, S., Bohus, D., Stiber, M., Bao, Y., Schoonbeek, T. J., & Horvitz, E. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831228
Self-citationThinking Inside the Box: Building User Interfaces with Question-Guided Inputs
Agarwal, S., Chen, H. W., Vogel, D., & Crisan, A. (2026) · Graphics Interface · doi:10.1145/3831423.3831467
A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
de Moor, A., van Deursen, A., & Izadi, M. (2024) · arXiv
- Authors
- Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E.
- Affiliations
- Massachusetts Institute of Technology (US)
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Volume
- 38
- Issue
- 9
- Pages
- 10137-10144
- ISSN
- 2159-5399 (ISSN-L)
- arXiv
- 2306.04930
- OpenAlex
- W4393153298
- Semantic Scholar
- 9beb5c2c17aa059b7c018a18eb88d183563e75f0
- Cited by
- 22
- References
- 29
- Citations by year
- 2026: 6 · 2025: 7 · 2024: 8 · 2023: 1
- FWCI
- 7.91
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- AI-assisted programming, code recommendation, human feedback, GitHub Copilot, developer productivity, reward signals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
