The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
IntelliCode compose: code generation using transformer
Open accessAbstract
In software development through integrated development environments (IDEs), code completion is one of the most widely used features. Nevertheless, majority of integrated development environments only support completion of methods and APIs, or arguments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Several methods have been proposed to incorporate repository context into the training of language models, including n-grams (Tu et al., 2014), LSTMs (Hellendoorn and Devanbu, 2017), and Transformers (Svyatkovskiy et al., 2020;Liu et al., 2022;Ding et al., 2022).”
From RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023) · cites this paper as Svyatkovskiy et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The IntelliCode Compose system uses the term CTR (Click Through Rate) for this and reports a value of 10% in online trials [11].”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“act syntax tree (AST) or concrete syntax tree (CST), and dataflow graph. A transformer block will typically consist of a multi-head self-attention, followed by a two-layer multi-layer perceptron (MLP) [17], optionally containing residual connections and layer normalization [23]. Recent neural architecture searches of t…”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 2×
“Several transformer models such as GPT-2, BERT, XLNet, and RoBERTa [19–22] have demonstrated the ability to learn effectively from unlabeled data to perform a wide variety of downstream tasks given supervised discriminative fine-tuning on each specific task.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Several transformer models such as GPT-2, BERT, XLNet, and RoBERTa [19–22] have demonstrated the ability to learn effectively from unlabeled data to perform a wide variety of downstream tasks given supervised discriminative fine-tuning on each specific task.”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 2×
“We refer the interested reader to the numerous publications about transformer models [16, 20–22], and focus on code completion for the remainder of this section.”
BackgroundDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“lue and are most likely lower than the actual user sentiment. Over 150K requests, we have seen a surfacing rate of 9.2% and an acceptance rate of 10%. 11 Knowledge Distillation Knowledge distillation [38] is the model compression technique in which a smaller model – the student network – is trained to reproduce the re…”
InfluentialExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“(3) Language specific control codes, is an approach introduced in [29, 30] that uses target prefixes to facilitate constrained language modeling or multitask learning.”
Background(2004) · Cited by 9,718
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 3×
“Transformers [18–21] are a family of neural networks designed to process ordered sequential data.”
BackgroundDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · cited 1×
“n [38] is the model compression technique in which a smaller model – the student network – is trained to reproduce the results of a larger model – the teacher network. It has been shown in literature [39, 40], that it is possible to reach comparable performance on various tasks using distilled neural networks, resulti…”
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
Flagged first, then by citations
Research works printed but not linked: 7
- Alexey Svyatkovskiy , Sebastian Lee , Anna Hadjitofi , Maik Riechert , Juliana Franco , and Miltiadis Allamanis . Fast and memory-eficient neural code completion , 2020 . Alexey Svyatkovskiy, Sebastian Lee, Anna Hadjitofi, Maik Riechert, Juliana Franco, and Miltiadis Allamanis. Fast and memory-eficient neural code completion, 2020.
- Alec Radford , Jef Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language models are unsupervised multitask learners . 2019 . Alec Radford, Jef Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
- Microsoft Corporation . Ai-assisted development. https://marketplace. visualstudio.com/items?itemName=VisualStudioExptTeam.VSIntelliCode. Visited Jan 2020 . Microsoft Corporation. Ai-assisted development. https://marketplace. visualstudio.com/items?itemName=VisualStudioExptTeam.VSIntelliCode. Visited Jan 2020.
- Alec Radford . Improving language understanding by generative pre-training . 2018 . Alec Radford. Improving language understanding by generative pre-training. 2018.
- Jimmy Lei Ba , Jamie Ryan Kiros, and Geofrey E. Hinton. Layer normalization , 2016 . Jimmy Lei Ba, Jamie Ryan Kiros, and Geofrey E. Hinton. Layer normalization, 2016.
- Eclipse Foundation . Code Recommenders . www.eclipse.org/recommenders. Visited June 2017 . Eclipse Foundation. Code Recommenders. www.eclipse.org/recommenders. Visited June 2017.
- Eclipse Recommenders . Eclipse SnipMatch . http://www.eclipse.org/ recommenders/manual/#snipmatch , 2014 . Visited Jun 2017. Eclipse Recommenders. Eclipse SnipMatch. http://www.eclipse.org/ recommenders/manual/#snipmatch, 2014. Visited Jun 2017.
Cited by
Sorted by citations
Items 101–200 of 480
Federated Few-Shot Learning for Mobile NLP
Cai, D., Wang, S., Wu, Y., Lin, F. X., & Xu, M. (2023) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3570361.3613277 · Cited by 33
“Examples include auto completion, QA, and sentiment analysis [45, 73, 78, 81, 106].”
BackgroundWho evaluates the evaluators? On automatic metrics for assessing AI-based offensive code generators
Liguori, P., Improta, C., Natella, R., Čukić, B., & Cotroneo, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120073 · Cited by 31
From Commit Message Generation to History-Aware Commit Message Completion
Eliseeva, A., Sokolov, Y., Bogomolov, E., Golubev, Y., Dig, D., & Bryksin, T. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00078 · Cited by 36
“Completion systems for both natural languages [21] and programming languages [22] are now widespread and overall well accepted by end users.”
BackgroundMethodInfluentialComprehensive Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Zhang, J., Bao, X., Wang, X., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3532759 · Cited by 30
“In our extended study, we added five larger billion-level LLMs, so we will focus on exploring the impact of PEFT techniques on LLM4APR research, in order to provide guidance on the selection of PEFT techniques for APR research in the era of LLMs.”
Pradel, M., & Chandra, S. (2021) · Communications of the ACM · doi:10.1145/3460348 · Cited by 28
“For example, it is available in TabNine, studied for internal usage at companies such as Facebook, and for widely-used IDEs, such as IntelliJ from JetBrains and Visual Studio’s IntelliCode from Microsoft [34].”
BackgroundMethodInfluentialLiao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H., & Li, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3486195 · Cited by 23
“I N the realm of Software Engineering, the advent of Large Language Models (LLMs) has revolutionized various tasks, including code generation [1], [2], [3], [4], [5].”
Your Code Secret Belongs to Me: Neural Code Completion Tools Can Memorize Hard-Coded Credentials
Huang, Y., Li, Y., Wu, W., Zhang, J., & Lyu, M. R.-T. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660818 · Cited by 35
“Powered by language models [Black et al. 2021; Feng et al. 2020; OpenAI 2023; Svyatkovskiy et al. 2020; Wang et al. 2021] neural code completion breaks the limitations of traditional code completion, which heavily relies on static analysis like type inference and variable name resolving.”
MethodDesign pattern recognition: a study of large language models
Pandey, S. K., Chand, S., Horkoff, J., Staron, M., Ochodek, M., & Durisic, D. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10625-1 · Cited by 27
“Svyatkovskiy et al. (2020) introduce a tool called IntelliCode Compose, a multilingual code completion tool for integrated development environments (IDEs).”
RLCoder: Reinforcement Learning for Repository-Level Code Completion
Wang, Y., Wang, Y., Guo, D., Chen, J., Zhang, R., Ma, Y., & Zheng, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00014 · Cited by 43
CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training
Wang, X., Wang, Y., Wan, Y., Wang, J., Zhou, P., Li, L., Wu, H., & Liu, J. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.80 · Cited by 29
CCRep: Learning Code Change Representations via Pre-Trained Code Model and Query Back
Liu, Z., Tang, Z., Xia, X., & Yang, X. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00014 · Cited by 34
“Recently, researchers also applied model pre-training to code and released a number of pre-trained code models, such as CodeBERT [21], GPT-C [27], PLBART [28] and CodeT5 [29].”
BackgroundMethodCode Execution with Pre-trained Language Models
Liu, C., Lu, S., Chen, W., Jiang, D., Svyatkovskiy, A., Fu, S., Sundaresan, N., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.308 · Cited by 37
Self-citationDeepDev-PERF: a deep learning-based approach for improving software performance
Garg, S., Moghaddam, R. Z., Clement, C. B., Sundaresan, N., & Wu, C. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549096 · Cited by 24
“…approaches have been shown to achieve state-of-the-art performance, not only in various Natural Language Processing (NLP) problems, but also a variety of software engineering tasks such as code-completion [28], documentation generation [7], unit test generation [29], bug detection [11], etc.”
BackgroundSelf-citationHow Important Are Good Method Names in Neural Code Generation? A Model Robustness Perspective
Yang, G., Zhou, Y., Yang, W., Yue, T., Chen, X., & Chen, T. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3630010 · Cited by 34
“, BLEU, CodeBLEU, Success rate), which have been widely used in previous studies of neural code generation [1, 9, 11, 13, 42, 49, 59, 65] and adversarial example generation [4, 34, 60, 68, 69, 71, 75].”
ChatAssert: LLM-Based Test Oracle Generation With External Tools Assistance
Hayet, I., Scott, A., & d’Amorim, M. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3519159 · Cited by 38
“We consider standard NLP metrics used in prior work, namely, BLEU [28], CodeBLEU [29], ROUGE [30], and EditSim [31].”
Explaining mispredictions of machine learning models using rule induction
Cito, J., Dillig, I., Kim, S., Murali, V., & Chandra, S. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468614 · Cited by 26
“For instance, machine learning has become a popular tool for automating code completion tasks [44], predicting which tests to run [28], finding relevant code fragments from large corpora [8], isolating crash inducing event sequences from telemetry [32], performing type inference [20, 34, 47], and…”
MethodIncorporating domain knowledge through task augmentation for front-end JavaScript code generation
Shen, S., Zhu, X. P., Dong, Y., Guo, Q., Zhen, Y., & Li, G. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558965 · Cited by 28
“Some works also refer to statementlevel code completion as code generation [20].”
Ciniselli, M., Pascarella, L., & Bavota, G. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528440 · Cited by 34
“This has been achieved, for example, by (i) exploiting contextual information such as the code surrounding the statements on which code completion is triggered [8, 48] and/or or the recent code change history [48]; and (ii) moving towards more advanced techniques such as Deep Learning (DL) models, that have been widel…”
BackgroundMethodDelving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study
Liu, S., Keung, J., Yang, Z., Liu, F., Zhou, Q., & Liao, Y. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00055 · Cited by 48
“Decoder-only models of code, like GPT-C [51], CodeGen [40], and Code Llama [47], are on the basis of GPT series [45], which predicts text when given the preceding context.”
MethodCognitive Agents Powered by Large Language Models for Agile Software Project Management
Cinkusz, K., Chudziak, J. A., & Niewiadomska‐Szynkiewicz, E. (2024) · Electronics · doi:10.3390/electronics14010087 · Cited by 42
“In the context of software engineering, LLMs enhance cognitive agents by enabling them to automate code generation [36], assist in documentation [37], facilitate communication [23], and support decision-making [24].”
Wang, X., Wu, Q., Zhang, H., Lyu, C., Jiang, X., Zheng, Z., Lyu, L., & Hu, S. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527896 · Cited by 27
“Many source code representation techniques have achieved considerable success in many program comprehension tasks, including code classification [50, 64, 65], malware detection [11, 66], and code generation [67, 68].”
BackgroundPSCVFinder: A Prompt-Tuning Based Framework for Smart Contract Vulnerability Detection
Yu, L., Lu, J., Liu, X., Li, Y., Zhang, F., & Ma, J. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00030 · Cited by 29
Large Language Model-Aware In-Context Learning for Code Generation
Li, J., Tao, C., Li, J., Li, G., Jin, Z., Zhang, H., Fang, Z., & Liu, F. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715908 · Cited by 32
Programming Knowledge Tracing: A Comprehensive Dataset and A New Model
Zhu, R., Zhang, D., Han, C., Gaol, M., Lu, X., Qian, W., & Zhou, A. (2022) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw58026.2022.00048 · Cited by 28
“2020), GPT-C (Svyatkovskiy et al. 2020), and PLBART (Ahmad et al.”
BackgroundGeneration-based Code Review Automation: How Far Are We?
Zhou, X., Kim, K., Xu, B., Han, D., He, J., & Lo, D. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00036 · Cited by 29
“They suggested that an edit distance-based metric [43] could serve as an important proxy for estimating the effort required to edit and fix generated code.”
BackgroundCoLeFunDa: Explainable Silent Vulnerability Fix Identification
Zhou, J., Pacheco, M., Chen, J., Hu, X., Xia, X., Lo, D., & Hassan, A. E. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00214 · Cited by 23
“This model established the use of pre-trained models, and its architecture continues to be used for a variety of programming language processing tasks [47], [48].”
MethodNon-Autoregressive Line-Level Code Completion
Liu, F., Fu, Z., Li, G., Jin, Z., Liu, H., Hao, Y., & Zhang, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3649594 · Cited by 22
“— Transformer [43]: We reproduce Intellicode compose [43], an autoregressive Transformer-based (GPT-2) line-level code completion approach using our datasets under comparable parameter settings with SANAR.”
BackgroundMethodGPT-3-Powered Type Error Debugging: Investigating the Use of Large Language Models for Code Repair
Ribeiro, F., Macedo, J. N., Tsushima, K., Abreu, R., & Saraiva, J. (2023) · ACM SIGPLAN International Conference on Software Language Engineering (SLE) · doi:10.1145/3623476.3623522 · Cited by 27
“However, translating buggy code to fixed code has limitations [15] and general-purpose models supporting code understanding and generation tasks [1, 17, 27, 45] started being considered.”
BackgroundLUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 33
How to Protect Copyright Data in Optimization of Large Language Models?
Chu, T., Song, Z., & Yang, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29741 · Cited by 41
Template-Guided Program Repair in the Era of Large Language Models
Huang, K., Zhang, J., Meng, X., & Liu, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00030 · Cited by 30
“Findings from a prior study [24] indicate that GPT-like models (Decoder-only), including GPT-C [38] and CodeGPT [39], underperform in classification tasks such as this.”
BUI, D. Q. N., Yu, Y., & Jiang, L. (2021) · Open Research Online (The Open University) · Cited by 20
Automating Code-Related Tasks Through Transformers: The Impact of Pre-training
Tufano, R., Pascarella, L., & Bavota, G. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00203 · Cited by 20
“[4], [12], [31]–[48] NSP Next Sentence Prediction Given two sentences (or two statements) asks the model to guess whether they follow each other.”
BackgroundAI-Assisted Programming Tasks Using Code Embeddings and Transformers
Kotsiantis, S. B., Verykios, V. S., & Tzagarakis, M. M. (2024) · Electronics · doi:10.3390/electronics13040767 · Cited by 29
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 26
Kemell, K., Saarikallio, M., Nguyen‐Duc, A., & Abrahamsson, P. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107805 · Cited by 27
Exploiting the Adversarial Example Vulnerability of Transfer Learning of Source Code
Yang, Y., Fan, H., Lin, C., Li, Q., Zhao, Z., & Shen, C. Q. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3402153 · Cited by 20
“Besides the BERT-like architecture, GPT [32] and UniLM [33] are employed to enhance the model performance of code completion tasks.”
MethodGenerative AI for OCL Constraint Generation: Dataset Collection and LLM Fine-tuning
Pan, F., Zolfaghari, V., Wen, L., Petrović, N., Lin, J., & Knoll, A. (2024) · IEEE International Symposium on Systems Engineering · doi:10.1109/isse63315.2024.10741141 · Cited by 19
“As generative AI demonstrates its competence in code generation tasks [16], researchers are exploring its potential for OCL generation.”
BackgroundNatural language-guided programming
Heyman, G., Huysegems, R., Justen, P., & Cutsem, T. V. (2021) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3486607.3486749 · Cited by 19
“A detailed study of GPT-2 autocom-pletion was carried out by Svyatkovskiy et al. [34].”
MethodCodeBERT‐Attack: Adversarial attack against source code deep learning models via pre‐trained model
Zhang, H., Lu, S., Li, Z., Jin, Z. G., Ma, L., Liu, Y., & Li, G. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2571 · Cited by 18
“*14 Emerging from representation models, the large pre-trained models have demonstrated great potential in the natural language processing (NLP) community and is later introduced to solve SE tasks.”
BackgroundPTM-APIRec: Leveraging Pre-trained Models of Source Code in API Recommendation
Li, Z., Li, C., Tang, Z. J., Huang, W., Ge, J., Luo, B., Ng, V., Wang, T., Hu, Y., & Zhang, X. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3632745 · Cited by 18
“As shown in Figure 1, after entering the object name and a dot, IDE (or other API recommendation systems) provides a list of APIs as the recommendation [44].”
BackgroundA Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
A User-centered Security Evaluation of Copilot
Asare, O., Nagappan, M., & Asokan, N. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639154 · Cited by 31
“IntelliCode [19] is another example of a CGT that generates recommendations based on thousands of open-source projects on GitHub.”
BackgroundTowards Automatically Addressing Self-Admitted Technical Debt: How Far Are We?
Mastropaolo, A., Penta, M. D., & Bavota, G. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00103 · Cited by 21
“…to investigate, considering the significant advances in the application of generative approaches to software-related tasks, such as code completion [11], [12], [13], [14], [15], program repair [16], [17], [18], [19], vulnerability patching [20], [21], or code review [22], [23], [24], [25].”
BackgroundDeep learning for code generation: a survey
Zhang, H., Zhang, K., Li, Z., Li, J., Li, J., Li, Y., Zhao, Y., Zhu, Y. S., Liu, F., Li, G., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-023-3956-3 · Cited by 20
“GPT-C (366M) [89] is one of the earliest pretrained transformer decoders on source code.”
InfluentialExploring Smart Contract Recommendation: Towards Efficient Blockchain Development
Jiang, Z., Zheng, Z., Chen, K., Luo, X., Tang, X., & Li, Y. (2022) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2022.3202081 · Cited by 16
“Most of researches on code recommendation, which also refers to code completion, focus on traditional programming languages such as Java, JavaScript, Typescript and Python [13], [14].”
BackgroundReasoning and Planning with Large Language Models in Code Development
Ding, H., Fan, Z., Guehring, I., Gupta, G. P., Ha, W., Huan, J., Liu, L. Q., Omidvar-Tehrani, B., Wang, S., & Zhou, H. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671452 · Cited by 23
“Large Language Models (LLMs) have emerged as a transformative force in the field of code development, ushering in a paradigm shift that promises to revolutionize the way software is created, tested, and maintained [26, 114].”
Generative AI Agents in Autonomous Machines: A Safety Perspective
Jabbour, J., & Reddi, V. J. (2024) · International Conference on Computer Aided Design · doi:10.1145/3676536.3698390 · Cited by 28
“Transformers have redefined sequence modeling and now serve as the backbone of cutting-edge systems in text generation [60, 102], language translation [105, 205], and code generation [16, 89, 92, 177], thanks to their ability to capture long-range dependencies and process sequences in parallel.”
DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code SkeletonsPreprint
Drain, D., Clement, C. B., Serrato, G., & Sundaresan, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.09352 · Cited by 46
“The cross-entropy losses are five times better than those normally reported for generating Python code since the editing task is relatively easy ((Svyatkovskiy et al., 2020; Clement et al., 2020)).”
Self-citationAutomating traffic model enhancement with AI research agent
Guo, X., Yang, X., Peng, M., Lu, H., Zhu, M., & Yang, H. (2025) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2025.105187 · Cited by 18
“These AI systems have already made significant progress in areas such as literature-based discovery 26,27 , code generation 28,29 , data analysis 30–33 as well as chemical experiment design 34 and materials discovery 35–37 .”
Alami, A., & Ernst, N. (2025) · 2025 IEEE/ACM 18th International Conference on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase66643.2025.00016 · Cited by 15
“The versatility and broad applicability of LLMs have the potential to fundamentally alter the landscape of software engineering by either reducing the reliance on human intervention or support in tasks that were previously considered too complex for automation [76], [82].”
Usage of Large Language Model for Code Generation Tasks: A Review
Stefano Bistarelli, B., Fiore, M., Mercanti, I., & Mongiello, M. (2025) · SN Computer Science · doi:10.1007/s42979-025-04241-5 · Cited by 29
Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study
van Dam, T., Izadi, M., & Arie Van Deursen, A. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00035 · Cited by 21
“Although most research on auto-completion is focused on single-token prediction, several studies aimed to complete entire statements or blocks of code [9, 24, 25].”
BackgroundMethodInfluentialHliš, T., Četina, L., Beranič, T., & Pavlič, L. (2023) · Applied Sciences · doi:10.3390/app132413061 · Cited by 20
“It is still under development and exclusive to Microsoft developers [37].”
BackgroundAI Code Generators for Security: Friend or Foe?
Natella, R., Liguori, P., Improta, C., Čukić, B., & Cotroneo, D. (2024) · IEEE Security & Privacy · doi:10.1109/msec.2024.3355713 · Cited by 15
CoSec: On-the-Fly Security Hardening of Code LLMs via Supervised Co-decoding
Li, D., Yan, M. L., Zhang, Y. X., Liu, Z., Liu, C. C., Zhang, X., Chen, T., & Lo, D. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680371 · Cited by 23
Harrath, Y., Adohinzin, O., Kaabi, J., & Saathoff, M. (2025) · Computers · doi:10.3390/computers14090374 · Cited by 18
“Genetic algorithms, neural networks, and transformer-based models such as GPT-3 assist developers by generating code snippets and suggesting fixes for code smells and inefficiencies [157–159].”
GrammarT5: Grammar-Integrated Pretrained Encoder-Decoder Neural Model for Code
Zhu, Q., Liang, Q., Sun, Z., Xiong, Y., Zhang, L., & Cheng, S. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639125 · Cited by 21
Rethinking AI code generation: a one-shot correction approach based on user feedback
Le, K. T., & Andrzejak, A. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00451-y · Cited by 15
“Various approaches, spanning from probabilistic (Bielik et al 2016; Li et al 2017; Schumacher et al 2020) to Machine Learning (Kim et al 2021; Svyatkovskiy et al 2020), have been proposed to validate this hypothesis.”
Assessing Evaluation Metrics for Neural Test Oracle Generation
Shin, J., Hemmati, H., Wei, M., & Wang, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3433463 · Cited by 14
“ES) [39] is a metric that implements Levenshtein distance to measure the distance be-tween two sequences by calculating the smallest number of edit operations (i.e., insertion, deletion, and replacement) required to transform one string into another.”
BackgroundVulAdvisor: Natural Language Suggestion Generation for Software Vulnerability Repair
Zhang, J., Wang, C., Li, A., Wang, W., Li, T., & Liu, Y. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695555 · Cited by 15
LGS-KT: Integrating logical and grammatical skills for effective programming knowledge tracing
Sun, X., Liu, Q., Zhang, K., Shen, S., Zhuang, Y., & Guo, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107164 · Cited by 15
Automated variable renaming: are we there yet?
Mastropaolo, A., Aghajani, E., Pascarella, L., & Bavota, G. (2023) · Empirical Software Engineering · doi:10.1007/s10664-022-10274-8 · Cited by 15
“For example, IntelliCode was introduced by Svyatkovskiy et al. [10], a group of researchers working at the Microsoft Corporation.”
MethodARist: An effective API argument recommendation approach
Nguyen, S., Tran, C., Tran, K. T., Nguyen, T. M., Nguyen, T.-T., Ngo, K.-T., & Vo, H. D. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111786 · Cited by 17
Syntax-aware on-the-fly code completion
Takerngsaksiri, W., Tantithamthavorn, C. K., & Li, Y.-F. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107336 · Cited by 14
(2023) · doi:10.1109/ase56229.2023.00157 · Cited by 14
“This fundamental fact supports researchers in employing language models in various SE tasks [4], [5], [6], [7] to automatically generate/predict code.”
BackgroundAn Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Code
Hort, M., Grishina, A., & Moonen, L. (2023) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem56168.2023.10304803 · Cited by 15
“Answer to RQ1: Out of the reviewed 293 publications, 33% shared source code and 27% shared trained artifacts.”
BackgroundExplaining Transformer-based Code Models: What Do They Learn? When They Do Not Work?
Mohammadkhani, A. H., Tantithamthavorn, C. K., & Hemmatif, H. (2023) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam59687.2023.00020 · Cited by 28
“, automated program repair [25], [26], [27], programming language translation [28], code completion [29]).”
BackgroundToward Automatically Completing GitHub Workflows
Mastropaolo, A., Zampetti, F., Bavota, G., & Penta, M. D. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623351 · Cited by 16
“[48] introduce IntelliCode, a multilingual code completion tool that predicts sequences of arbitrary token types using subtokens to overcome the OoV problem [51].”
MethodAdvances and challenges in artificial intelligence text generation
Li, B., Yang, P., Sun, Y., Hu, Z., & Yi, M. (2024) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2300410 · Cited by 15
Assessing the Effectiveness and Security Implications of AI Code Generators
Taeb, M., Chi, H., & Bernadin, S. (2024) · Journal of The Colloquium for Information Systems Security Education · doi:10.53735/cisse.v11i1.180 · Cited by 13
“IntelliCode Compose was proposed as a general-purpose multilingual code completion tool that is capable of predicting sequences of code tokens of arbitrary types, generating up to entire lines of syntactically correct code, achieving an edit similarity of 86.7% [12].”
BackgroundInfluentialPitfalls in Language Models for Code Intelligence: A Taxonomy and Survey
She, X., Liu, Y., Zhao, Y., He, Y., Li, L., Tantithamthavorn, C. K., Qin, Z., & Wang, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3748647 · Cited by 31
“Systems based on advanced LM4Code, such as GitHub Copilot [40], Amazon CodeWhisperer [4], and Microsoft IntelliCode [139], have already been deployed in real-world IDEs and have garnered a large number of users.”
Exploring and evaluating personalized models for code generation
Zlotchevski, A., Drain, D., Svyatkovskiy, A., Clement, C. B., Sundaresan, N., & Tufano, M. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558959 · Cited by 13
“Neural code generation has generated an intense recent interest in NLP, using Transformer models [31] in particular for code completion [3, 4, 7, 21, 25, 26], code synthesis from examples [6], natural language to code [2, 6, 7], code feature summarization [1, 16, 18, 19, 23, 32], code search [9, 10], unit test generat…”
MethodSelf-citationThe Transformers’ Ability to Implement for Solving Intricacies of Language Processing
Singh, P. N., & Behera, S. (2022) · Asian Conference on Innovation in Technology (ASIANCON) · doi:10.1109/asiancon55314.2022.9909423 · Cited by 15
“Alexey Svyatkovskiy et al. presented a novel approach for AI-assited code completion calling it Pythia [7] and IntelliCode compose [8].”
MethodAll you need is logs: improving code completion by learning from anonymous IDE usage logs
Bibaev, V., Kalina, A., Lomshakov, V., Golubev, Y., Bezzubov, A., Povarov, N. I., & Bryksin, T. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558968 · Cited by 13
“One of the key features that defines an IDE is code completion [1, 10, 18, 40, 42].”
BackgroundMethodMeasuring code maintainability with deep neural networks
Hu, Y., Jiang, H., & Hu, Z. (2023) · Frontiers of Computer Science · doi:10.1007/s11704-022-2313-0 · Cited by 14
Li, Y., Qi, S., Gao, C., Peng, Y., Lo, D., Lyu, M. R.-T., & Xu, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3524461 · Cited by 17
“…for capturing textual semantics through attention mechanisms [9], Transformer has proven highly effective in learning source code representations [10], [11], and has become a state-of-the-art architecture in several code intelligence tasks, including code completion [12], [13], code…”
Grammar-Based Patches Generation for Automated Program Repair
Tang, Y. G., Zhou, L., Blanco, A., Liu, S., Wei, F., Zhou, M., & Yang, M. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.111 · Cited by 13
“Researchers have exploited machine learning to automate several development and maintenance tasks, such as code completion (Svyatkovskiy et al., 2020), comment generation (Hu et al., 2018), code search (Gu et al., 2018), bug localization (Zheng et al., 2016) and fixing (Tufano et al., 2018).”
BackgroundWhat is wrong with your code generated by large language models? An extensive study
Dou, S., Jia, H., Wu, S., Zheng, H., Wu, M., Tao, Y., Zhang, M., Chai, M., Fan, J., Xu, N., et al. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4632-8 · Cited by 102
“Code generation (or program synthesis) aims to accurately generate executable programs from problem specifications such as natural language requirement descriptions and unit test examples [1, 2].”
Improving Text-to-Code Generation with Features of Code Graph on GPT-2
Paik, I., & Wang, J. (2021) · Electronics · doi:10.3390/electronics10212706 · Cited by 14
Broken Promises: Measuring Confounding Effects in Learning-based Vulnerability Discovery
Imgrund, E., Ganz, T., Härterich, M., Pirch, L., Risse, N., & Rieck, K. (2023) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3605764.3623915 · Cited by 13
From Triumph to Uncertainty: The Journey of Software Engineering in the AI Era
Mastropaolo, A., Escobar‐Velásquez, C., & Linares‐Vásquez, M. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709360 · Cited by 23
“Such a sheer amount of data has facilitated the adoption of DL methods to enhance and automate various SE tasks, including –but not limited to– bug-fixing [41, 74, 148, 212], code summarization [73, 109, 116] and code completion [46, 89, 112, 124, 202, 203].”
LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code Completion
Wang, C., Huang, K., Zhang, J., Feng, Y., Zhang, L., Liu, Y., & Peng, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00245 · Cited by 17
“Historically, researchers have explored statistical models [18], [19] [8], [9] because they are similar in token-based prediction.”
Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy
Clement, C., Lu, S., Liu, X., Tufano, M., Drain, D., Duan, N., Sundaresan, N., & Svyatkovskiy, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.387 · Cited by 12
“Namely, the generative pre-trained transformer model for code (Svyatkovskiy et al., 2020) (GPT-C) and the Python method text-to-text transfer trans-former model (Clement et al., 2020) (PyMT5).”
BackgroundMethodInfluentialSelf-citationMPI-RICAL: Data-Driven MPI Distributed Parallelism Assistance with Transformers
Schneider, N., Kadosh, T., Hasabnis, N., Mattson, T. G., Pinter, Y., & Oren, G. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3624062.3624063 · Cited by 12
“Several tools already exist for tasks such as analyzing comments, documentation and generating code summarization, code completion, and code translation [49, 51, 56].”
MethodDCServCG: A data-centric service code generation using deep learning
Alizadehsani, Z., Ghaemi, H., Shahraki, A., González‐Briones, A., & Corchado, J. M. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106304 · Cited by 13
KetGPT – Dataset Augmentation of Quantum Circuits Using Transformers
Apak, B., Bandić, M., Sarkar, A., & Feld, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63778-0_17 · Cited by 17
IRCoCo: Immediate Rewards-Guided Deep Reinforcement Learning for Code Completion
Li, B., Sun, Z., Huang, T., Zhang, H., Wan, Y., Li, G., Jin, Z. G., & Lyu, C. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643735 · Cited by 11
“These LMs serve as the actor to perform code completion, while two quality evaluators for generated code, trained using BLEU and Edit-Sim metrics, act as the critic to produce immediate rewards; these components are incorporated within the IRCoCo framework.”
Source Code Recommender Systems: The Practitioners' Perspective
Ciniselli, M., Pascarella, L., Aghajani, E., Scalabrino, S., Oliveto, R., & Bavota, G. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00182 · Cited by 13
“, [47]), (ii) empirical studies about code generation/completion techniques (e.”
Aligning Offline Metrics and Human Judgments of Value for Code Generation Models
Dibia, V. C., Fourney, A., Bansal, G., Poursabzi-Sangdeh, F., Liu, H., & Amershi, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.540 · Cited by 24
“As one proxy for effort, we computed normalized edit similarity (Svyatkovskiy et al., 2020) as follows: where gen is code generated by a model for a problem in the HumanEval dataset, ref is the hand-written reference solution to the problem and lev is the character Levenshtein edit distance.”
BackgroundMethodInfluentialLearning to Locate and Describe Vulnerabilities
Zhang, J., Liu, S., Wang, X., Li, T., & Liu, Y. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00045 · Cited by 15
“Transformer follows an encoder-decoder structure but uses self-attention mechanisms to process sequences of data (i.e., token sequence) and has also been widely adopted in code or its comment generation [46], [47].”
MethodPlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context
Chen, X., Gong, L., Cheung, A., & Song, D. X. (2021) · doi:10.18653/v1/2021.acl-long.169 · Cited by 13
“Our code synthesis problem is also related to code completion, i.e., autocompleting the program given the code context (Raychev et al., 2014; Li et al., 2018; Svyatkovskiy et al., 2020).”
MethodDeepMig: A transformer-based approach to support coupled library and code migrations
Rocco, J. D., Nguyen, P. T., Sipio, C. D., Rubei, R., Ruscio, D. D., & Penta, M. D. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107588 · Cited by 13
PTM4Tag+: Tag recommendation of stack overflow posts with pre-trained models
He, J., Xu, B., Yang, Z., Han, D., Yang, C., Liu, J., Zhao, Z., & Lo, D. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10576-z · Cited by 11
“Svyatkovskiy et al. trained GPT-C (Svyatkovskiy et al. 2020), a generative model based on GPT-2 architecture, and leveraged GPT-C to build a code completion tool.”
Mutation Testing via Iterative Large Language Model-Driven Scientific Debugging
Straubinger, P., Kreis, M., Lukasczyk, S., & Fraser, G. (2025) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw64639.2025.10962485 · Cited by 17
“In PLP, example applications of LLMs include code completion [18], program repair [19], and code generation [20].”
exLong: Generating Exceptional Behavior Tests with Large Language Models
Zhang, J., Liu, Y., Nie, P., Li, J. J., & Gligoric, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00176 · Cited by 11
“Edit similarity [49], [64]: calculates 1-Levenshtein distance which is the minimum number of character-level edits (insertions, deletions, or substitutions) required to change the prediction into the ground-truth.”
Boosting source code suggestion with self-supervised Transformer Gated Highway
Hussain, Y., Huang, Z., Zhou, Y., & Wang, S. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111553 · Cited by 10
Better Context Makes Better Code Language Models: A Case Study on Function Call Argument Completion
Pei, H., Zhao, J., Lausen, L., Zha, S., & Karypis, G. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25653 · Cited by 17
Towards Code Generation from BDD Test Case Specifications: A Vision
Chemnitz, L., Reichenbach, D., Aldebes, H., Naveed, M., Narasimhan, K., & Mezini, M. (2023) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain58948.2023.00031 · Cited by 11
“[3]–[5], [17] III.”
Metadata
- Authors
- Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N.
- Affiliations
- Microsoft (United States) (US)
- Microsoft, USA
- ORCID
- Sundaresan, Neel
- Venue
- Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2020-11-08
- Pages
- 1433-1443
- OpenAlex
- W3025993830
- Cited by
- 474
- References
- 38
- Citations by year
- 2026: 42 · 2025: 97 · 2024: 117 · 2023: 103 · 2022: 52 · 2021: 56 · 2020: 7
- FWCI
- 41.33
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- code generation, code completion, transformer models, integrated development environment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
