Repository-Level Prompt Generation for Large Language Models of Code
Open accessAbstract
With the success of large language models (LLMs) of code and their use as code assistants (e.g. Codex used in GitHub Copilot), techniques for introducing domain-specific knowledge in the prompt design process become important. In this work, we propose a framework called Repo-Level Prompt Generator that learns to generate example-specific prompts using prompt proposals. The prompt proposals take context from the entire repository, thereby incorporating both the structure of the repository and the context from other relevant files (e.g. imports, parent class files). Our technique doesn't require any access to the weights of the LLM, making it applicable in cases where we only have black-box access to the LLM. We conduct experiments on the task of single-line code-autocompletion using code repositories taken from Google Code archives. We demonstrate that an oracle constructed from our prompt proposals gives a remarkably high relative improvement of 36% over Codex, showing the quality of these proposals. Further, we show that when we train a model to predict a prompt proposal, we can achieve significant performance gains over Codex and other baselines. We release our code, data, and trained checkpoints at: \url{https://github.com/shrivastavadisha/repo_level_prompt_generation}.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Shrivastava et al. (2022) relies on inflexible heuristics and classifier training for prompt construction.
“Despite its innovative approach, the study relies on inflexible heuristics and classifier training for prompt construction.”
From RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023) · cites this paper as Shrivastava et al. (2022)
The Introduction (p. 1) says RLPG uses a learned Prompt Proposal Classifier to select among prompt proposals, supporting the classifier-training part. But Experimental Details (p. 5) explicitly says the framework can incorporate new prompt proposals, so describing its prompt construction as reliant on inflexible heuristics overstates the source. (LLM, conf=0.90)
Experimental — accuracy not yet measuredPrior work suggests that augmenting prompts with code from the same repository improves performance in code generation tasks.
“Prior work suggests that augmenting prompts with code from the same repository improves performance in code generation tasks [60].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [60]
Shrivastava et al. (2022) explores the scenario of repository-level code completion.
“A study by Shrivastava et al. (2022) also explores the scenario of repository-level code completion.”
From RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023) · cites this paper as Shrivastava et al. (2022)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 47
- Allamanis, M. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software . Association for Computing Machinery, 2019.
- Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021. · arXiv
- Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
- Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., Pieler, M., Prashanth, U. S., Purohit, S., Reynolds, L., Tow, J., Wang, B., and Weinbach, S. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models , 2022.
- Bragg, J., Cohan, A., Lo, K., and Beltagy, I. FLEX: Unifying evaluation for few-shot NLP. In Advances in Neural Information Processing Systems , 2021.
- Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances in Neural Information Processing Systems , 2020.
- Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021. · arXiv
- Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. · arXiv
- Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 , 2020. · arXiv
- Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-t., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999 , 2022. · arXiv
- Gao, T., Fisch, A., and Chen, D. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , 2021.
- Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366 , 2020. · arXiv
- He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2016.
- Hellendoorn, V. J. and Devanbu, P. Are deep neural networks the best choice for modeling source code? In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering , 2017.
- Jiang, E., Toh, E., Molina, A., Olson, K., Kayacik, C., Donsbach, A., Cai, C. J., and Terry, M. Discovering the syntax and strategies of natural language programming with generative language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , 2022.
- Kanade, A., Maniatis, P., Balakrishnan, G., and Shi, K. Learning and evaluating contextual embedding of source code. In Proceedings of the 37th International Conference on Machine Learning , 2020.
- Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., and Lewis, M. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations , 2020.
- Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.
- Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 , 2022. · arXiv
- Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , 2021.
- Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , 2021.
- Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures . Association for Computational Linguistics, 2022.
- Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 2023.
- Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, J. Gpt understands, too. arXiv:2103.10385 , 2021. · arXiv
- Pashakhanloo, P., Naik, A., Dai, H., Maniatis, P., and Naik, M. Learning to walk over relational graphs of source code. In Deep Learning for Code Workshop , 2022a.
- Pashakhanloo, P., Naik, A., Wang, Y., Dai, H., Maniatis, P., and Naik, M. Codetrek: Flexible modeling of code using an extensible relational representation. In International Conference on Learning Representations , 2022b.
- Qin, G. and Eisner, J. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2021.
- Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022. · arXiv
- Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., et al. A generalist agent. arXiv preprint arXiv:2205.06175 , 2022. · arXiv
- Reynolds, L. and McDonell, K. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , 2021.
- Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.
- Schick, T. and Schütze, H. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , 2021.
- Shin, T., Razeghi, Y., IV , R. L. L., Wallace, E., and Singh, S. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP) , 2020.
- Shrivastava, D., Larochelle, H., and Tarlow, D. On-thefly adaptation of source code models. In NeurIPS 2020 Workshop on Computer-Assisted Programming , 2020.
- Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research , 2014.
- Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., and Weisz, J. D. Investigating explainability of generative ai for code through scenario-based design. In 27th International Conference on Intelligent User Interfaces , 2022.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, u., and Polosukhin, I. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems , 2017.
- Wang, C., Yang, Y., Gao, C., Peng, Y., Zhang, H., and Lyu, M. R. No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2022.
- Wang, Y., Shi, E., Du, L., Yang, X., Hu, Y., Han, S., Zhang, H., and Zhang, D. Cocosum: Contextual code summarization with multi-relational graph neural network. arXiv preprint arXiv:2107.01933 , 2021a. · arXiv
- Wang, Y., Wang, W., Joty, S., and Hoi, S. C. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , 2021b.
- Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022. · arXiv
- Wu, Y., Rabe, M. N., Hutchins, D., and Szegedy, C. Memorizing transformers. In International Conference on Learning Representations , 2022.
- Xiao, W., Beltagy, I., Carenini, G., and Cohan, A. PRIMERA: Pyramid-based masked sentence pre-training for multi-document summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2022.
- Xu, F. F., Alon, U., Neubig, G., and Hellendoorn, V. J. A systematic evaluation of large language models of code. arXiv preprint arXiv:2202.13169 , 2022a. · arXiv
- Xu, F. F., He, J., Neubig, G., and Hellendoorn, V. J. Capturing structural locality in non-parametric language models. In International Conference on Learning Representations , 2022b.
- Zhang, J., Panthaplackel, S., Nie, P., Mooney, R. J., Li, J. J., and Gligoric, M. Learning to generate code comments from class hierarchies. arXiv preprint arXiv:2103.13426 , 2021. · arXiv
- Zhou, S., Alon, U., Xu, F. F., Jiang, Z., and Neubig, G. Docprompting: Generating code by retrieving the docs. In International Conference on Learning Representations , 2023.
Cited by
Sorted by citations
Unleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 764 · see citing works
(2025) · doi:10.1109/icse55347.2025.00157 · Cited by 157 · see citing works
RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
Zhang, F., Chen, B., Zhang, Y., Keung, J., Liu, J., Zan, D., Mao, Y., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.151 · Cited by 173 · see citing works
Effective test generation using pre-trained Large Language Models and mutation testing
Dakhel, A. M., Nikanjam, A., Majdinasab, V., Khomh, F., & Desmarais, M. C. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107468 · Cited by 157 · see citing works
Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639183 · Cited by 112 · see citing works
ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification
Mu, F., Shi, L., Wang, S., Yu, Z., Zhang, B., Wang, C., Liu, S., & Wang, Q. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660810 · Cited by 74 · see citing works
“Prompt Learning is one of the most important techniques for achieving this goal [12, 28, 31, 40, 47].”
Fully Autonomous Programming with Large Language Models
Liventsev, V., Grishina, A., Härmä, A., & Moonen, L. (2023) · Proceedings of the Genetic and Evolutionary Computation Conference · doi:10.1145/3583131.3590481 · Cited by 45 · see citing works
LLMeLog: An Approach for Anomaly Detection based on LLM-enriched Log Events
He, M., Jia, T., Duan, C., Cai, H., Li, Y., & Huang, G. (2024) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre62328.2024.00023 · Cited by 46 · see citing works
Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation
Wang, C., Zhang, J., Feng, Y., Li, T., Sun, W., Liu, Y., & Peng, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714462 · Cited by 40 · see citing works
Bias Testing and Mitigation in LLM-based Code Generation
Huang, D., Zhang, J. M., Bu, Q., Xie, X., Chen, J., & Cui, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3724117 · Cited by 38 · see citing works
Project-specific code summarization with in-context learning
Yun, S. M., LIN, S.-J., Gu, X., & Shen, B. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112149 · Cited by 29 · see citing works
Liao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H., & Li, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3486195 · Cited by 23 · see citing works
Search-Based LLMs for Code Optimization
Gao, S., Gao, C., Gu, W., & Lyu, M. R.-T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00021 · Cited by 25 · see citing works
Learning Performance-Improving Code Edits
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2023) · arXiv · doi:10.48550/arxiv.2302.07867 · Cited by 23 · see citing works
“Recent work has demonstrated that retrieval-based mechanisms can improve language models for various tasks requiring factual or procedural knowledge (Liu et al., 2021; Poesia et al., 2021; Rubin et al., 2022; Madaan et al., 2022; Shrivastava et al., 2023).”
BackgroundAgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology
Nguyen, M. H., Chau, T. P., Nguyen, P. X., & Bui, N. D. Q. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00026 · Cited by 34 · see citing works
The Fact Selection Problem in LLM-Based Program Repair
Parasaram, N., Yan, H.-J., Yang, B.-Y., Flahy, Z., Qudsi, A., Ziaber, D., Barr, E. T., & Mechtaev, S. (2024) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00162 · Cited by 39 · see citing works
The Impact of Prompt Programming on Function-Level Code Generation
Khojah, R., de Oliveira, F. D., Mohamad, M., & Leitner, P. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3587794 · Cited by 48 · see citing works
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 17 · see citing works
Bibi, N., Maqbool, A., & Rana, T. A. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101865 · Cited by 12 · see citing works
Retrieval-augmented code completion for local projects using large language models
Hostnik, M., & Robnik‐Šikonja, M. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128596 · Cited by 11 · see citing works
Case study: using AI-assisted code generation in mobile teams
Vasiliniuc, M.-S., & Groza, A. P. (2023) · IEEE International Conference on Intelligent Computer Communication and Processing (ICCP) · doi:10.1109/iccp60212.2023.10398656 · Cited by 8 · see citing works
Li, L., Liang, B., Chen, L., & Zhang, X. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107527 · Cited by 8 · see citing works
Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning
Jie, Z., & Lu, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.668 · Cited by 11 · see citing works
Bukit, T. A., Pillado, E. P. B., Yahya, B. N., & Lee, S. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113793 · Cited by 7 · see citing works
Anchor Attention, Small Cache: Code Generation With Large Language Models
Zhang, X., Zhou, Y., Yang, G., Gall, H. C., & Chen, T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3570680 · Cited by 6 · see citing works
Guiding ChatGPT for Better Code Generation: An Empirical Study
Liu, C., Bao, X., Zhang, H., Zhang, N., Hu, H., Zhang, X., & Yan, M. L. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00018 · Cited by 6 · see citing works
MoonBit: Explore the Design of an AI-Friendly Programming Language
Fei, H., Zhang, Y., Zhang, H., Wang, Y., & Liu, Q. (2024) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3643795.3648376 · Cited by 2 · see citing works
YABLoCo: Yet Another Benchmark for Long Context Code Generation
Valeev, A., Garaev, R., Lomshakov, V., Piontkovskaya, I., Ivanov, V. V., & Adewuyi, I. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00016 · Cited by 2 · see citing works
Does Few-Shot Learning Help LLM Performance in Code Synthesis?
Xu, D., Xie, T., Xia, B., Li, H., Bai, Y., Sun, Y., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2412.02906 · Cited by 3 · see citing works
“…et al., 2023; Guo et al., 2024; Dubey et al., 2024), test case generation (Vikram et al., 2023; Lemieux et al., 2023; Schäfer et al., 2023), debugger interaction (Islam et al., 2024; Ho et al., 2024), and repository-level generation (Shrivastava et al., 2023; Bui et al., 2024; Wang et al., 2024a).”
Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E. T. (2023) · arXiv · Cited by 112 · see citing works
Pal, A., Mathew, M. G., Moorthy, A. V. G., & Babu, C. V. S. (2025) · AIP conference proceedings · doi:10.1063/5.0259612
Jin, H., & Hamdaqa, M. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756954
PlayCoder: Making LLM-Generated GUI Code Playable
Peng, Z., Tao, W., Yin, X., Ying, C., Luo, Y., & Guo, Y. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808097 · Cited by 2 · see citing works
- Authors
- Shrivastava, D., Larochelle, H., & Tarlow, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2206.12839
- OpenAlex
- W4283702402
- Cited by
- 27
- References
- 0
- Citations by year
- 2026: 1 · 2025: 13 · 2024: 8 · 2023: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Canadian Institute for Advanced Research
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- prompt generation, software repositories, code completion, domain knowledge, repository structure, codices, GitHub Copilot
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
