Full Line Code Completion: Bringing AI to Desktop
Open accessAbstract
In recent years, several industrial solutions for the problem of multi-token code completion appeared, each making a great advance in the area but mostly focusing on cloud-based runtime and avoiding working on the end user's device. In this work, we describe our approach for building a multi-token code completion feature for the JetBrains' IntelliJ Platform, which we call Full Line Code Completion. The feature suggests only syntactically correct code and works fully locally, i.e., data querying and the generation of suggestions happens on the end user's machine. We share important time and memory-consumption restrictions, as well as design principles that a code completion engine should satisfy. Working entirely on the end user's device, our code completion engine enriches user experience while being not only fast and compact but also secure. We share a number of useful techniques to meet the stated development constraints and also describe offline and online evaluation pipelines that allowed us to make better decisions. Our online evaluation shows that the usage of the tool leads to 1.3 times more Python code in the IDE being produced by code completion. The described solution was initially started with a help of researchers and was then bundled into all JetBrains IDEs where it is now used by millions of users. Thus, we believe that this work is useful for bridging academia and industry, providing researchers with the knowledge of what happens when complex research-based solutions are integrated into real products.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
3 of 24 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured
- Findingown data
Knowledge distillation does not provide any quality improvement for their model sizes.
“Unlike in many similar circumstances, we do not employ knowledge distillation [56], because we discovered that for our sizes of the models, this technique does not provide any quality improvement.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [56] 56. V. Sanh, L. Debut, J. Chaumond, and T. Wolf, "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter," arXiv preprint arXiv:1910.01108, 2019. source
- Findingreasoning
Quantization from FP32 to INT4 decreases the model size several times with negligible quality loss.
“This transformation decreases the size of the model several times, trading it off to a negligible quality loss.”
Quote in the paperSubstantial claim - Findingnone given
FLCC works fully locally, with data querying and suggestion generation on the end user's machine.
“The feature suggests only syntactically correct code and works fully locally, i.e., data querying and the generation of suggestions happens on the end user's machine.”
- Also stated in: § I. INTRODUCTION · plain
“It works fully locally, i.e. all computations, from code completion invocation to the showing of suggestion, happen on the end user's device.”
Quote in the paperSubstantial claim - Also stated in: § I. INTRODUCTION · plain
- Findingown data
The cache is used in over 90% of completion events.
“According to our observation, the cache is used in over 90% of completion events”
Quote in the paperSubstantial claim - Findingown data
Usage of FLCC leads to 1.3 times more Python code in the IDE being produced by code completion compared to standard code completion only.
“Our online evaluation shows that the usage of the tool leads to 1.3 times more Python code in the IDE being produced by code completion.”
- Also stated in: § I. INTRODUCTION · plain
“Our experiments showed that the ratio of completed code (i.e., the percentage of code symbols written with code completion among all written code) increased 1.3 times for Python users with Full Line Code Completion compared to users with standard code completion only.”
Quote in the paperSubstantial claim - Also stated in: § I. INTRODUCTION · plain
- Findingown data
For Python, the ratio of completed code increased 1.3 times (from 17% to 22%) for users with FLCC enabled compared to users with standard code completion only.
“For Python, the metric bumped 1.3 times (from 17% to 22%) for users with FLCC enabled compared to users with standard code completion only.”
Quote in the paperSubstantial claim - Findingown data
The authors did not observe any statistically significant data showing that users cancel completion selections or delete text more frequently when provided with FLCC.
“We did not observe any statistically significant data that would show that users cancel completion selections or delete text more frequently when they are provided with FLCC.”
Quote in the paperNot a substantial claim - Findingown data
In the authors' setup, n = 20 and k = 3 are the best values for the dynamic beam search stopping criteria.
“Our experiments showed that n = 20 and k = 3 are the best values in our setup.”
Quote in the paperNot a substantial claim - Findingcitation
The average time to show for FLCC is 150 ms.
“The average time to show for FLCC is 150 ms”
Quote in the paperSubstantial claim - Findingown data
A maximum context length of 1,536 tokens performed best according to their benchmarks given memory footprint restrictions.
“For our models, we used the maximum context length of 1,536 tokens, since it performed the best according to our benchmarks, given memory footprint restrictions from target IDEs.”
Quote in the paperSubstantial claim - Findingown data
The authors did not observe any performance degradation when enabling FLCC.
“We did not observe any performance degradation when enabling FLCC.”
Quote in the paperSubstantial claim - Findingown data
In the same mid-2023 benchmark, the mean time from completion invocation to suggestion display with llama.cpp was approximately 100 ms.
“For llama.cpp, the same procedure took approximately 100 ms.”
Quote in the paperNot a substantial claim - Findingown data
The acceptance rate for Full Line Code Completion for Python was about 38%.
“The acceptance rate for Full Line Code Completion for Python was about 38%.”
Quote in the paperNot a substantial claim - Findingown data
In a mid-2023 benchmark on MacOS with an M2 Max processor, the mean time from completion invocation to suggestion display with ONNX Runtime was approximately 75 ms.
“The benchmark showed that the mean time from the invocation of completion to the showing of the suggestion with ONNX Runtime was approximately 75 ms.”
Quote in the paperNot a substantial claim - Findingown data
For Python, the strategy for collecting hypotheses dropped Matched Ratio from 20.6% to 20.3%.
“Matched Ratio only dropped from 20.6% to 20.3%”
Quote in the paperSubstantial claim - Findingnone given
For FLCC, INT4-quantized LLaMA-like models with 100M parameters were used.
“In summary, for Full Line Code Completion, INT4-quantized LLaMA-like models with 100M parameters were used.”
Quote in the paperSubstantial claim - Findingown data
Context processing is observed to be 3-10 times slower than generation, influenced by hardware and numerical precision.
“Context processing, which computes token interrelations, is observed to be 3-10 times slower than generation (influenced by hardware and numerical precision).”
Quote in the paperSubstantial claim - Findingreasoning
Repeating tabulation on different indentation levels can result in many similar tokens in the vocabulary, leading to inefficiencies for the model.
“Insight 1: Repeating tabulation on different indentation levels can result in a lot of similar tokens in the vocabulary, which might lead to various inefficiencies for the model.”
Quote in the paperSubstantial claim - Findingown data
In late 2023, it became possible to infer the same LLaMA model in 50 ms instead of 100 ms.
“it became possible to infer the same LLaMA model in just 50 ms instead of 100 ms”
Quote in the paperSubstantial claim - Findingown data
Quantization decreased the model size from almost 400 MB to slightly more than 100 MB.
“Firstly, the size of the model decreased from almost 400 MB to slightly more than 100 MB, which fits within the desired restrictions.”
Quote in the paperSubstantial claim - Findingown data
For Python, the strategy for collecting hypotheses increased the percentage of Perfect Lines from 13% to 21%.
“for Python, it increased the percentage of Perfect Lines from 13% to 21%”
Quote in the paperNot a substantial claim - Findingown data
FLCC benefits from filtering out incorrect code.
“Our research for the 2023.3PyCharm Pro release showed that FLCC benefits from filtering out incorrect code.”
Quote in the paperSubstantial claim - Findingown data
LLaMA-like models performed best in terms of quality and speed among the architectures experimented with for FLCC.
“Having experimented with several model architectures, including GPT-2 and LLaMA, we ended up with a LLaMA-like model for FLCC for all languages.In our evaluations, LLaMA-like models performed best not only in terms of quality but also speed, thanks to llama.cpp[52] a multiplatform inference engine for LLaMA-like models.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [52] 52. "llama.cpp," [Accessed 01.01.2025]. [Online]. Available: https://github. com/ggerganov/llama.cpp
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
Filtering out incorrect code loses only 1% of valuable suggestions.
“Only 1% of valuable suggestions are lost”
Quote in the paperSubstantial claim - Findingown data
The acceptance rate for standard completion for Python was about 27%.
“For standard completion, the value is about 27%”
Quote in the paperNot a substantial claim - Findingown data
For a vocabulary of 16,384 tokens, as many as 800 tokens consisted of characters from various languages, mostly Chinese, Japanese, Korean, Arabic, and others.
“For a vocabulary of 16,384 tokens, as many as 800 tokens consisted of characters from various languages, mostly Chinese, but also Japanese, Korean, Arabic, and others.”
Quote in the paperSubstantial claim - Findingcitation
FLCC suggests only syntactically correct code.
“The feature suggests only syntactically correct code and works fully locally, i.e., data querying and the generation of suggestions happens on the end user's machine.”
- Also stated in: § I. INTRODUCTION · plain
“a multi-token code completion engine that utilizes a Transformer-based neural network in its core and suggests only syntactically correct code.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [18] 18. JetBrains, "Full Line Code Completion," [Accessed 01.01.2025]. [Online]. Available: https://plugins.jetbrains.com/plugin/14823-full-line- code-completion
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Also stated in: § I. INTRODUCTION · plain
- Study design and conductcitation
For Python, FLCC used a subset of The Stack (StarCoder dataset) processed to keep only permissive licenses.
“For Python, we used the subset of The Stack [38] -the StarCoder dataset [48], which we additionally processed to keep only permissive licenses.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [38] 38. HuggingFace, "The Stack," [Accessed 01.01.2025]. [Online]. Available: https://huggingface.co/datasets/bigcode/the-stack
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [48] 48. BigCode, "StarCoder," [Accessed 01.01.2025]. [Online]. Available: https://huggingface.co/datasets/bigcode/starcoderdata
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationown data
Cache use makes the typical time to show even lower.
“which makes a typical time to show even lower.”
Quote in the paperNot a substantial claim - Interpretationown data
The drop in Matched Ratio from 20.6% to 20.3% is negligible.
“which is quite negligible”
Quote in the paperNot a substantial claim - Interpretationown data
Full Line Code Completion suggests only valid code.
“we establish the distinguishing feature of Full Line Code Completion -suggesting only valid code.”
Quote in the paperSubstantial claim - Study design and conductreasoning
The test set repositories were separated so the model is not trained on them.
“After collecting the dataset, we found a representative set of repositories to be separated into the test part, thus ensuring that the model is not trained on them.”
Quote in the paperSubstantial claim - Interpretationreasoning
Randomly removing each import statement from training files with 50% probability helps the model learn to anticipate necessary imports without their explicit inclusion.
“To achieve this, we randomly remove each import statement from training files with a 50% probability, helping the model to learn to anticipate the necessary imports without their explicit inclusion.”
Quote in the paperSubstantial claim- Rests on an assumption: Developers often write code first and add imports later.
- Rests on an assumption: The auto-import feature in IntelliJ-based IDEs automatically recognizes code from external libraries and adds necessary imports.
- Premisecitation
All multi-token completion products except TabNine and aiXcoder offer only cloud-based solutions.
“More importantly, all such products, except for TabNine and aiXcoder, offer only cloud-based solutions that might have certain restrictions for users behind the firewall, users with limited internet access, and users who have privacy concerns regarding sending their code to the cloud.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [3] 3. Microsoft, "GitHub Copilot," [Accessed 01.01.2025]. [Online]. Available: https://github.com/features/copilot
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [4] 4. TabNine, "TabNine," [Accessed 01.01.2025]. [Online]. Available: https://www.tabnine.com/
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [5] 5. Codeium, "Codeium," [Accessed 01.01.2025]. [Online]. Available: https://codeium.com/
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation
Transformer-based neural networks significantly outperform other architectures in generating sequential data.
“Specifically, Transformerbased [13] neural networks significantly outperform other architectures in generating sequential data [14]- [17].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [14]-[17]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Study design and conductreasoning
No conclusions can be made from comparing the acceptance rate of FLCC and standard completion due to significantly different lengths of suggestions.
“however, no conclusions can be made from this comparison due to significantly different lengths of suggestions.”
Quote in the paperSubstantial claim - Interpretationown data
Sophisticated caching strategies allow speeding up code completion without significant quality loss.
“Insight 10: Sophisticated caching strategies allow to speed up code completion without significant quality loss.”
Quote in the paperSubstantial claim - Interpretationreasoning
This limitation restricts the ability to conduct a direct comparison with other multi-token completion tools.
“This limitation restricts our ability to conduct a direct comparison with other multi-token completion tools”
Quote in the paperNot a substantial claim - Study design and conductnone given
The analysis is limited to a specific group of users: those participating in the EAP.
“our analysis is limited to a specific group of users -those participating in the EAP.”
Quote in the paperSubstantial claim - Interpretationnone given
Full Line Code Completion meets the speed and memory footprint restrictions.
“It meets the speed and memory footprint restrictions”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 19
- May combine two claims. 19
- The quote only partly states this claim. 1
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationThe Stack dataset encompasses over 6TB of code in 358 languages and is extensively used for training large language models due to permissive licensing and wide coverage.
“This dataset, encompassing over 6TB of code in 358 languages, is extensively used for training large language models due to its permissive licensing and wide coverage.”
Quote in the paperNot a substantial claim2 of 3 sources checked- cites [8] 8. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021. source
The Data Collection section (p. 4) describes a different training dataset: 159 GB of filtered Python files from GitHub. Searches for The Stack, 6 TB, 358 languages, and permissive licensing found no passage in our copy addressing those claims or the dataset’s purported use for training. (LLM, conf=0.78)
Experimental — accuracy not yet measured - cites [37] 37. D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M. Ferrandis, Y. Jernite, M. Mitchell, S. Hughes, T. Wolf et al., "The Stack: 3 TB of permissively licensed source code," arXiv preprint arXiv:2211.15533, 2022. source
The Abstract (p. 1) describes The Stack as permissively licensed code intended to support research on code language models, but gives its size as 3.1 TB across 30 languages—not over 6 TB across 358. Appendix B (p. 19) reports 370 languages for the updated v1.1 dataset. The paper reports training its own models on subsets; it does not establish that The Stack was extensively used to train large language models. (LLM, conf=0.88)
Experimental — accuracy not yet measured - cites [38] 38. HuggingFace, "The Stack," [Accessed 01.01.2025]. [Online]. Available: https://huggingface.co/datasets/bigcode/the-stack
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
FindingcitationFLCC is a multi-token code completion engine that utilizes a Transformer-based neural network in its core.
“At JetBrains, we built the Full Line Code Completion (FLCC) feature [18] -a multi-token code completion engine that utilizes a Transformer-based neural network in its core and suggests only syntactically correct code.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [18] 18. JetBrains, "Full Line Code Completion," [Accessed 01.01.2025]. [Online]. Available: https://plugins.jetbrains.com/plugin/14823-full-line- code-completion
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
FindingreasoningThe caching approach significantly increases model inference speed in over 90% of real-world scenarios while keeping quality at almost the same level.
“According to our observations, this allows to significantly increase model inference speed in over 90% of the real-world scenarios, while keeping the quality at almost the same level.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataThe train-validation-test proportions for the Python model were 80%-5%-15%, totaling slightly more than 50 GB.
“For the training of the Python model, the corresponding proportions of the train-validation-test sets of the data were 80%-5%-15%, adding up to slightly more than 50 GB of data.”
Quote in the paperNot a substantial claimMay combine two claims.
FindingreasoningThe quantized model runs 2 times faster than the full precision model on M2 CPU on MacOS and 4 times faster on Intel i9 CPU on Linux.
“Secondly, llama.cppallows us to shorten the CPU inference time due to more efficient usage of the registers for INT4 data: e.g., the quantized model runs 2 times faster compared to full precision model on M2 CPU on MacOS and 4 times faster on Intel i9 CPU on Linux.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
FindingcitationFLCC is bundled into all JetBrains IDEs and enabled by default for various programming languages.
“The feature is implemented in the form of a plugin for the IntelliJ Platform [19], [20] (an open-source platform for building IDEs), and is bundled into all JetBrains IDEs: IntelliJ IDEA [21], PyCharm Pro [22], WebStorm [23], and others, where it is enabled by default for various programming languages.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [19] 19. --, "IntelliJ Platform," JetBrains, [Accessed 01.01.2025]. [Online]. Available: https://www.jetbrains.com/opensource/idea/
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [20] 20. Z. Kurbatova, Y. Golubev, V. Kovalenko, and T. Bryksin, "The IntelliJ platform: A framework for building plugins and mining software data," in 2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW), 2021, pp. 14-17. source
- cites [21] 21. JetBrains, "IntelliJ IDEA," JetBrains, [Accessed 01.01.2025]. [Online]. Available: https://www.jetbrains.com/idea/
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone givenFLCC has been available for Python since version 2023.3 and for other languages in 2024.1 and 2024.2, serving millions of users.
“The feature has been available for Python since version 2023.3 and then became available for other languages too in 2024.1 and 2024.2, now serving millions of users.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Findingnone givenThe resulting implementation has proven its efficiency during A/B testing on real users.
“The resulting implementation has proven its efficiency during A/B testing on real users”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenThe feature's performance is significantly different for different languages.
“Additionally, the feature's performance is significantly different for different languages”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingnone givenThe scope of the implemented feature is limited to a single line of code.
“Also, the scope of the implemented feature is limited to a single line of code.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenThe implementation was bundled into 2024.1 and 2024.2 releases of IntelliJ-based IDEs.
“was bundled into 2024.1 & 2024.2 releases of IntelliJ-based IDEs.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
InterpretationreasoningThe overall approach covers all restrictions and design principles, and the resulting feature is local-based, fast, compact, and properly integrated into the IDE.
“Overall, our approach to implementing Full Line Code Completion allows us to cover all restrictions and design principles described at the beginning of this section, meaning that the resulting feature is local-based, fast, compact, and properly integrated into the IDE.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationCode completion is among the most attractive features in IDEs because it helps developers avoid wasting time on manually typing code and speeds up software engineering activity.
“Code completion is among the most attractive features in integrated development environments (IDEs) [1], [2], since it helps developers to avoid wasting time on manually typing all of the code characters and thus speeds up any software engineering activity.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [1] 1. G. Murphy, M. Kersten, and L. Findlater, "How are Java software developers using the Elipse IDE?" IEEE Software, vol. 23, no. 4, pp. 76-83, 2006. source
- cites [2] 2. S. Amann, S. Proksch, S. Nadi, and M. Mezini, "A study of Visual Studio usage in practice," in 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), vol. 1, 2016, pp. 124-134. source
May combine two claims.
Study design and conductcitationAll comparisons provided in Table I are statistically significant, measured by per-user bootstrap.
“All the provided comparisons are statistically significant, measured by employing per-user bootstrap [65].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [65] 65. B. Efron, "Bootstrap methods: another look at the Jackknife," in Breakthroughs in statistics: Methodology and distribution. Springer, 1992, pp. 569-593. source
May combine two claims.
PremisecitationTabNine and aiXcoder support fully local models and data querying but lack integration features and do not implement code correctness checks.
“While TabNine and aiXcoder support fully local models and data querying, they lack integration features and do not implement code correctness checks.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [4] 4. TabNine, "TabNine," [Accessed 01.01.2025]. [Online]. Available: https://www.tabnine.com/
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [7] 7. aiXcoder, "aiXcoder," [Online. Accessed 01.01.2025]. [Online]. Available: https://www.aixcoder.com/en/#/
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
Interpretationnone givenToken healing resolves generation issues from mid-token positions by reverting to one token prior and constraining the initial generation to the incomplete token.
“This method, called token healing, resolves generation issues from mid-token positions by reverting to one token prior and constraining the initial generation to the incomplete token.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationGitHub Copilot shows an acceptance rate of about 30% for both single-line and multi-line suggestions combined.
“For comparison, GitHub Copilot shows an acceptance rate of about 30% [66] for both single-line and multi-line suggestions combined.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [66] 66. GitHub, "Quantifying GitHub Copilot's impact in the enterprise with accenture," 2024, [Accessed 01.01.2025]. [Online]. Avail- able: https://github.blog/news-insights/research/research-quantifying- github-copilots-impact-in-the-enterprise-with-accenture/
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation150 ms is perfect for the user's visual perception.
“which is perfect for the user's visual perception [67].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [67] 67. R. J. Kosinski, "A literature review on reaction time," Clemson University, vol. 10, no. 1, pp. 337-344, 2008.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
Interpretationown dataFLCC is more effective than the standard single-token approach, offering faster coding and potentially reduced cognitive load for developers.
“The feature, as evidenced by our A/B tests, proves more effective than the standard singletoken approach, offering faster coding and potentially reduced cognitive load for developers.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductnone givenEAP releases were used as a platform for A/B testing, and EAPs of versions 2023.3, 2024.1, and 2024.2 were used for online evaluation of FLCC.
“EAP releases are also used as a platform for A/B testing.The EAPs of versions 2023.3,2024.1, and 2024.2 were used to conduct the online evaluation for Full Line Code Completion.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductcitationThe processes required for the training pipeline and the plugin are implemented to be as similar to each other as possible.
“Thus, we ensure that the processes required for both stages are implemented to be as similar to each other as possible [58].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [58] 58. M. Zinkevich, "Rules of ML," [Accessed 01.01.2025]. [Online]. Available: https://developers.google.com/machine-learning/guides/rules-of-ml
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
The quote only partly states this claim.
Interpretationown dataForcing only ASCII symbols in the vocabulary and merging non-ASCII sequences into a single <UNK> token improves the model's ability to suggest proper tokens.
“This observation led us to force only ASCII symbols to be used for the vocabulary while leaving other symbols as <UNK> tokens and merging sequences of them into a single <UNK> token.”
- Also stated in: § 1) Tokenization: · plain
“Insight 5: Even without the comments, raw code also contains characters that can hinder the model's ability to suggest proper tokens.To combat this, we focused on the ASCII symbols.”
Quote in the paperSubstantial claim- Also stated in: § 1) Tokenization: · plain
May combine two claims.
InterpretationreasoningRemoving leading indentations and substituting them with special tokens <SCOPE_IN> and <SCOPE_OUT> saves space in the BPE vocabulary and better handles token healing.
“Such an approach allows not only to save space in the BPE vocabulary but also to better handle the token healing problem that is described further in Section III-D1.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningOpen-source models like InCoder, StarCoder, CodeLLaMA, etc., are not suitable for local code completion engines like FLCC because they are too big, have restrictive licenses, and only specific architectures have fast inference engines.
“Despite the fact that the models are open-source, they are not suitable for local code completion engines like FLCC for several reasons.First of all, they are usually too big to ship to users and do not meet the memory footprint restrictions for desktop applications like IDEs.Secondly, the license for the model's weights might not be permissive for commercial use.Last but not least, only specific”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductreasoningAll forks of a given repository were placed in the same data segment to avoid potential data leaks.
“Additionally, to avoid potential data leaks, we ensure that all the forks of the given repository are placed in the same data segment.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningA vocabulary size of 16,384 tokens allows decent model quality while maintaining an acceptable memory footprint.
“We chose the vocabulary size of 16,384 tokens because it allows for a decent model quality while maintaining an acceptable memory footprint: for our sizes of models, a significant part of the model's weights depends on the vocabulary size.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Study design and conductnone givenEAP participants typically have a deeper understanding and engagement with the IDE's features, and their usage patterns may differ from average users.
“EAP participants typically have a deeper understanding and engagement with the IDE's features, and their usage patterns may differ from average users.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Interpretationnone givenFull Line Code Completion is available for a wide range of users, including those with limited internet access and those with privacy concerns.
“being available for a wide range of users, including those with limited internet access and those with privacy concerns.”
Quote in the paperNot a substantial claim- Rests on an assumption: There is a need for local-based multi-token code completion among users with limited internet access or privacy concerns.
May combine two claims.
Study design and conductown dataThe offline evaluation dataset remains limited in size and might not represent all code well enough.
“our dataset for offline evaluation remains limited in terms of size and might not represent all the code well enough.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Interpretationnone givenFull Line Code Completion fills a niche of local-based multi-token code completion engines.
“The feature fills a niche of local-based multi-token code completion engines”
Quote in the paperSubstantial claim- Rests on an assumption: There is a need for local-based multi-token code completion among users with limited internet access or privacy concerns.
The paper relies on this without support.
Built-in completion engines usually only work in a single-token mode, generating only one identifier.
“In any case, built-in completion engines usually only work in a single-token mode, i.e., only one identifier is generated (e.g., a variable or a method name).”
The paper relies on this without support.
Developers often write code first and add imports later.
“Developers often write code first and add imports later, a process facilitated by the auto-import feature in IntelliJ-based IDEs -it automatically recognizes code from external libraries and adds the necessary imports.”
The paper relies on this without support.
The auto-import feature in IntelliJ-based IDEs automatically recognizes code from external libraries and adds necessary imports.
“Developers often write code first and add imports later, a process facilitated by the auto-import feature in IntelliJ-based IDEs -it automatically recognizes code from external libraries and adds the necessary imports.”
The paper relies on this without support.
The user expects a code-suggesting tool to show finished constructions.
“Moreover, the user expects a code-suggesting tool to show finished constructions.”
The paper relies on this without support.
None of the multi-token completion solutions have a common API for inline completion.
“Moreover, none of these solutions have a common API for inline completion, which makes them conflict with others and requires re-implementing the same functionality, which leads to poor UX and many confirmed bugs.”
The paper relies on this without support.
There is a need for local-based multi-token code completion among users with limited internet access or privacy concerns.
“being available for a wide range of users, including those with limited internet access and those with privacy concerns.”
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3× · see citing works
“Specifically, Transformer-based [14] neural networks significantly out-perform other architectures in the task of sequential data generation [15]–[18].”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2× · see citing works
“2) Language modelling: Over the past few years, Transformer-based neural networks started dominating in language modelling [15], [16], [25].”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Bootstrap Methods: Another Look at the Jackknife
Hinkley, D. (2008) · doi:10.1007/978-0-387-75692-9_9 · Cited by 9,963 · cited 1×
“This is our main, golden star metric used for the assessment of code completion quality.”
Fang, H. (2006) · doi:10.1201/b18055-8 · Cited by 9,792
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · cited 1× · see citing works
“This limitation restricts our ability to conduct a direct comparison with other multi-token completion tools, which are commonly available as IntelliJ plugins and are frequently used by a broader user base not involved in the EAP.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 5× · see citing works
“In the past years, several products for multi-token code completion have appeared on the market, including Copilot [3], TabNine [4], Codeium [5], Amazon CodeWhisperer [6], Source-graph Cody [7], aiXcoder [8], and others.”
InfluentialEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 2× · see citing works
“A notable development to address these issues is the introduction of The Stack dataset [9], [37], [38] by BigCode [39], a collaboration of over 350 researchers backed by HuggingFace and ServiceNow.”
Bootstrap Methods: Another Look at the Jackknife
Efron, B. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_41 · Cited by 1,374 · see citing works
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219 · cited 1× · see citing works
“Thus, to compress the model’s size and speed up its inference, a well-known technique called quantization [57] was used.”
C-52111, D. O. T. E. C. O. J. 1. J. 2. –. C. (2013) · IIC - International Review of Intellectual Property and Competition Law · doi:10.1007/s40319-013-0096-4 · Cited by 1,175 · cited 3×
“In our approach, the way to obtain the final model has several steps, including tokenization, constructing training examples, training the Transformer-based model, and then quantizing it.”
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · cited 2× · see citing works
“1) Tokenization: In FLCC, we use Transformer-based models for the source code, so we chose to use the byte-pair encoding (BPE) [30] as a standard approach for such models [15], [16], [25].”
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540 · cited 1× · see citing works
“The most famous examples of such models are InCoder [28], [40], StarCoder [29], [41], CodeLLaMA [27], [42], CodeGen and CodeGen-2 [26], [43], [44], CodeParrot [45], and PolyCoder [46], [47].”
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · cited 1× · see citing works
“1) Datasets: Many existing datasets for training code completion models, particularly for Python [32]–[34], have limitations due to licensing issues and quality concerns.”
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · cited 2× · see citing works
“A notable development to address these issues is the introduction of The Stack dataset [9], [37], [38] by BigCode [39], a collaboration of over 350 researchers backed by HuggingFace and ServiceNow.”
How are Java software developers using the Eclipse IDE?
Murphy, G. C., Kersten, M., & Findlater, L. (2006) · IEEE Software · doi:10.1109/ms.2006.105 · Cited by 413 · cited 2× · see citing works
“1) Datasets: Many existing datasets for training code completion models, particularly for Python [32]–[34], have limitations due to licensing issues and quality concerns.”
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · cited 2× · see citing works
“In addition to recent advancements in single-token code completion, there is a niche of multi-token completion that is being actively studied in academia [26]–[29] and employed in products [3]–[6].”
On the naturalness of software
Hindle, A., Barr, E. T., Gabel, M., Su, Z., & Devanbu, P. (2016) · Communications of the ACM · doi:10.1145/2902362 · Cited by 307 · cited 1× · see citing works
“Since source code can be interpreted as sequential data, natural language processing (NLP) methods have found their application for the task [13].”
StarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261 · cited 2× · see citing works
“In addition to recent advancements in single-token code completion, there is a niche of multi-token completion that is being actively studied in academia [26]–[29] and employed in products [3]–[6].”
CodeGen2: Lessons for Training LLMs on Programming and Natural LanguagesPreprint
Nijkamp, E., Hayashi, H., Xiong, C.-M., Savarese, S., & Zhou, Y.-B. (2023) · arXiv.org · doi:10.48550/arxiv.2305.02309 · Cited by 253 · see citing works
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200 · cited 3× · see citing works
“This metric can be defined differently [62], and it accounts for the ratio of code that stays in the file for some time after accepting code completion.”
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200 · see citing works
InCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187 · cited 3× · see citing works
“The numerator of the metric includes code from all code completion sources: standard and inline completions, so it allows us to assess the entire code completion experience.”
Haynes, M. (2020) · C++ Conference · doi:10.32381/prod · Cited by 171
Schmitt, J. (2022) · Encyclopedia of Big Data · doi:10.1007/978-3-319-32001-4_179-1 · Cited by 118 · cited 1×
“Since we are focused on code completion, we filtered out all comments.”
(2000) · International Journal of Productivity and Performance Management · doi:10.1108/ws.2000.07949fab.004 · Cited by 104 · see citing works
Fast and Memory-Efficient Neural Code Completion
Svyatkovskiy, A., Lee, S., Hadjitofi, A., Riechert, M., Franco, J. V., & Allamanis, M. (2021) · 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00045 · Cited by 95 · cited 1× · see citing works
“In recent years, machine learning techniques were studied and applied for the different aspects of the completion problem [9]–[12], including the generation of the completion sequences itself.”
A Study of Visual Studio Usage in Practice
Amann, S., Proksch, S., Nadi, S., & Mezini, M. (2016) · 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) · doi:10.1109/saner.2016.39 · Cited by 77 · cited 2× · see citing works
“Code completion is among the most attractive features in integrated development environments (IDEs) [1], [2], since it helps developers to avoid wasting time on manually typing all of the code characters and thus speeds up any software engineering activity.”
Structural Language Models of CodePreprint
Alon, U. S., Sadaka, R., Levy, O., & Yahav, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.00577 · Cited by 60 · see citing works
Sutherland, R. (2003) · doi:10.1201/9780203912539-7 · Cited by 55 · cited 2×
“A notable development to address these issues is the introduction of The Stack dataset [9], [37], [38] by BigCode [39], a collaboration of over 350 researchers backed by HuggingFace and ServiceNow.”
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · arXiv · doi:10.48550/arxiv.2305.16264 · Cited by 49 · cited 1× · see citing works
“Following recent advancements in choosing the number of training steps given the model and dataset sizes, we used HuggingFace’s approximation [55] for every trained model.”
Program Synthesis and Semantic Parsing with Learned Code IdiomsPreprint
Shin, E. C. R., Allamanis, M., Brockschmidt, M., & Polozov, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10816 · Cited by 49 · cited 1× · see citing works
“Compared to natural language texts, source code primarily consists of common statements or sequences of more than one token [50].”
The Stack: 3 TB of permissively licensed source code
Kocetkov, D., Li, R., Allal, L. B., Li, J., Mou, C., Ferrandis, C. M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.15533 · Cited by 42 · cited 3× · see citing works
“2) Solution’s memory footprint must be feasible and take a reasonable part of the IDE’s memory footprint.”
How Practitioners Expect Code Completion?
Wang, C., Hu, J., Gao, C., Yu, J., Xie, T., Huang, H., Lei, Z., & Deng, Y. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616280 · Cited by 42 · see citing works
A Study of Potential Code Borrowing and License Violations in Java Projects on GitHub
Golubev, Y., Eliseeva, M., Povarov, N. I., & Bryksin, T. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387455 · Cited by 27 · cited 1× · see citing works
“Open-source code, often collected from platforms like GitHub, may have restrictive licenses or none at all [35], which poses legal challenges [36], as seen in GitHub Copilot’s [3] usage.”
Self-citationThe IntelliJ Platform: A Framework for Building Plugins and Mining Software Data
Kurbatova, Z., Golubev, Y., Kovalenko, V., & Bryksin, T. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew52652.2021.00016 · Cited by 17 · cited 1× · see citing works
“The feature is implemented in the form of a plugin for the IntelliJ Platform [20], [21] (an open-source platform for building IDEs), and is bundled into all JetBrains IDEs: IntelliJ IDEA [22], PyCharm Pro [23], WebStorm [24], and others, where it is enabled by default for various programming…”
Self-citationPractitioners' Expectations on Code Completion
Wang, C., Hu, J., Gao, C., Jin, Y., Xie, T., Huang, H., Lei, Z., & Deng, Y. (2023) · arXiv · Cited by 14 · cited 1× · see citing works
“During such A/B testing in late 2023 and 2024, we collected a lot of data to compute different important metrics from literature [62], [63].”
All you need is logs: improving code completion by learning from anonymous IDE usage logs
Bibaev, V., Kalina, A., Lomshakov, V., Golubev, Y., Bezzubov, A., Povarov, N. I., & Bryksin, T. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558968 · Cited by 13 · see citing works
Self-citationTowards Full-line Code Completion with Neural Language ModelsPreprint
Wang, W., Shen, S., Li, G., & Jin, Z. G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.08603 · Cited by 9 · cited 17× · see citing works
“Specifically, Transformer-based [14] neural networks significantly out-perform other architectures in the task of sequential data generation [15]–[18].”
InfluentialDash, A., Meher, P., & Barik, P. (2026) · International Journal of AI Electrical Civil and Mechanical engineering · doi:10.64751/7jkmh432 · Cited by 8 · cited 1×
“This allows us to create a proper integration with the standard IntelliJ completion, JetBrains AI Assistant [61], and third-party code completion providers.”
Ovalle, B. S. T. (2022) · Encuentro Internacional de Educación en Ingeniería · doi:10.26507/paper.2300 · Cited by 4 · cited 4× · see citing works
“In addition to recent advancements in single-token code completion, there is a niche of multi-token completion that is being actively studied in academia [26]–[29] and employed in products [3]–[6].”
Influential
Flagged first, then by citations
32 of the 74 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- [14]14. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
- [17]17. V. J. Hellendoorn, C. Sutton, R. Singh, P. Maniatis, and D. Bieber, "Global relational models of source code," in International conference on learning representations, 2019.
- [67]67. R. J. Kosinski, "A literature review on reaction time," Clemson University, vol. 10, no. 1, pp. 337-344, 2008.
Cited by
Sorted by citations
Improving LLM-based code completion using LR parsing
Atique, M. M. A. B., Moon, H., Sasano, I., & Choi, K. (2025) · Journal of Computer Languages · doi:10.1016/j.cola.2025.101352 · Cited by 1 · see citing works
Towards an experience in AI-driven development for programming applied to multimedia using Tabnine
Llerena-Izquierdo, J., Vargas, I. V. F., Gorozabel, J. A. P., & Arce, A. E. V. (2024) · IEEE URUCON · doi:10.1109/urucon63440.2024.10850207 · Cited by 1 · see citing works
“Fostering a thorough understanding of the ethical and social implications of AI among developers, educators, and users can help raise awareness of the consequences of dishonest practices [19], [27].”
- Authors
- Semenkin, A., Bibaev, V., Sokolov, Y., Krylov, K., Kalina, A., Khannanova, A., Savenkov, D., Rovdo, D., Davidenko, I., Karnaukhov, K., Vakhrushev, M., Kostyukov, M., Podvitskii, M., Surkov, P., Golubev, Y., Povarov, N., & Bryksin, T.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 563-574
- arXiv
- 2405.08704
- OpenAlex
- W4413360205
- Semantic Scholar
- b9eaa37036837fba277c40517a2be7671dbfa768
- Cited by
- 8
- References
- 74
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 2
- FWCI
- 7.16
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- offline evaluation, online assessment, Python code generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
