CoChat
Sign Up
PreprintCited by 37Software Engineering ResearchTeaching and Learning ProgrammingSoftware Testing and Debugging Techniques

CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion

Open access
Ding, Yangruibo, Wang, Zijian, Ahmad, Wasi Uddin, Ding, Hantian, Tan, Ming, Jain, Nihal, Ramanathan, Murali Krishna, Nallapati, Ramesh +3 more·2023·arXiv
DOI 10.48550/arxiv.2310.11248arXiv 2310.11248
Open this paper in CoChat — free accountOpen PDF

Abstract

Code completion models have made significant progress in recent years, yet current popular evaluation datasets, such as HumanEval and MBPP, predominantly focus on code completion tasks within a single file. This over-simplified setting falls short of representing the real-world software development scenario where repositories span multiple files with numerous cross-file dependencies, and accessing and understanding cross-file context is often required to complete the code correctly. To fill in this gap, we propose CrossCodeEval, a diverse and multilingual code completion benchmark that necessitates an in-depth cross-file contextual understanding to complete the code accurately. CrossCodeEval is built on a diverse set of real-world, open-sourced, permissively-licensed repositories in four popular programming languages: Python, Java, TypeScript, and C#. To create examples that strictly require cross-file context for accurate completion, we propose a straightforward yet efficient static-analysis-based approach to pinpoint the use of cross-file context within the current file. Extensive experiments on state-of-the-art code language models like CodeGen and StarCoder demonstrate that CrossCodeEval is extremely challenging when the relevant cross-file context is absent, and we see clear improvements when adding these context into the prompt. However, despite such improvements, the pinnacle of performance remains notably unattained even with the highest-performing model, indicating that CrossCodeEval is also capable of assessing model's capability in leveraging extensive context to make better code completion. Finally, we benchmarked various methods in retrieving cross-file context, and show that CrossCodeEval can also be used to measure the capability of code retrievers.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.11248

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Scaling Laws for Neural Language Models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works

    “We see the performance increases following the power law in all settings as expected (Kaplan et al., 2020).”
    Background

Flagged first, then by citations

Research works printed but not linked: 39

  1. Ahmad, W., Chakraborty, S., Ray, B., Chang, K.W., 2021. Unified pre-training for program under- standing and generation, in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Online. pp. 2655-2668. URL: https://aclanthology.org/2021. naacl-main.211, doi:10.18653/v1/2021.naacl-main.211. · DOI · Paper page
  2. Allal, L.B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C.M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al., 2023. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301.03988 URL: https://arxiv.org/abs/2301.03988. · arXiv
  3. Allamanis, M., Sutton, C., 2013. Mining source code repositories at massive scale using language modeling, in: Proceedings of the 10th Working Conference on Mining Software Repositories, IEEE Press. p. 207-216. URL: https://dl.acm.org/doi/pdf/10.5555/2487085.2487127. · DOI
  4. Athiwaratkun, B., Gouda, S.K., Wang, Z., Li, X., Tian, Y., Tan, M., Ahmad, W.U., Wang, S., Sun, Q., Shang, M., Gonugondla, S.K., Ding, H., Kumar, V., Fulton, N., Farahani, A., Jain, S., Giaquinto, R., Qian, H., Ramanathan, M.K., Nallapati, R., Ray, B., Bhatia, P., Sengupta, S., Roth, D., Xiang, B., 2023. Multi-lingual evaluation of code generation models, in: The Eleventh International Conference on Learning Representations. URL: https://openreview. net/forum?id=Bo7eeXm6An8.
  5. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al., 2021. Program synthesis with large language models. ArXiv preprint abs/2108.07732. URL: https://arxiv.org/abs/2108.07732. · arXiv
  6. Barr, E.T., Brun, Y., Devanbu, P., Harman, M., Sarro, F., 2014. The plastic surgery hypothesis, in: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, Association for Computing Machinery, New York, NY, USA. p. 306-317. URL: https://doi.org/10.1145/2635868.2635898, doi:10.1145/2635868.2635898. · DOI
  7. Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., Chen, M., 2022. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255 URL: https: //arxiv.org/abs/2207.14255. · arXiv
  8. Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., Pieler, M., Prashanth, U.S., Purohit, S., Reynolds, L., Tow, J., Wang, B., Weinbach, S., 2022. GPT-NeoX-20B: An open-source autoregressive language model, in: Proceedings of BigScience Episode #5 -Workshop on Challenges & Perspectives in Creating Large Language Models, Association for Computational Linguistics, virtual+Dublin. pp. 95-136. URL: https: //aclanthology.org/2022.bigscience-1.9, doi:10.18653/v1/2022.bigscience-1. · DOI
  9. Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.H., Zi, Y., Anderson, C.J., Feldman, M.Q., Guha, A., Greenberg, M., Jangda, A., 2023. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering 49, 3675-3691. doi:10.1109/TSE.2023.3267446. · DOI · Paper page
  10. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al., 2021. Evaluating large language models trained on code. ArXiv preprint abs/2107.03374. URL: https://arxiv.org/abs/2107.03374. · arXiv
  11. Clement, C., Lu, S., Liu, X., Tufano, M., Drain, D., Duan, N., Sundaresan, N., Svyatkovskiy, A., 2021. Long-range modeling of source code files with eWASH: Extended window access by syntax hierarchy, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Domini- can Republic. pp. 4713-4722. URL: https://aclanthology.org/2021.emnlp-main.387, doi:10.18653/v1/2021.emnlp-main.387. · DOI · Paper page
  12. Ding, H., Kumar, V., Tian, Y., Wang, Z., Kwiatkowski, R., Li, X., Ramanathan, M.K., Ray, B., Bhatia, P., Sengupta, S., et al., 2023. A static evaluation of code completion by large language models. arXiv preprint arXiv:2306.03203 . · arXiv
  13. Ding, Y., Wang, Z., Ahmad, W.U., Ramanathan, M.K., Nallapati, R., Bhatia, P., Roth, D., Xiang, B., 2022. Cocomic: Code completion by jointly modeling in-file and cross-file context. arXiv preprint arXiv:2212.10007 URL: https://arxiv.org/abs/2212.10007. · arXiv
  14. Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., Zhou, M., 2020. CodeBERT: A pre-trained model for programming and natural languages, in: Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online. pp. 1536-1547. URL: https://aclanthology.org/2020. findings-emnlp.139, doi:10.18653/v1/2020.findings-emnlp.139. · DOI · Paper page
  15. Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, S., Zettlemoyer, L., Lewis, M., 2023. Incoder: A generative model for code infilling and synthesis, in: The Eleventh International Conference on Learning Representations. URL: https://openreview. net/forum?id=hQwb-lbM6EL.
  16. Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., Yin, J., 2022. Unixcoder: Unified cross-modal pre-training for code representation, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7212-7225.
  17. Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., Steinhardt, J., 2021. Measuring coding challenge competence with APPS, in: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). URL: https://openreview.net/forum?id=sD93GOzH3i5.
  18. Holtzman, A., Buys, J., Du, L., Forbes, M., Choi, Y., 2020. The curious case of neural text degeneration, in: International Conference on Learning Representations. URL: https: //openreview.net/forum?id=rygGQyrFvH.
  19. Hossain, N., Ghazvininejad, M., Zettlemoyer, L., 2020. Simple and effective retrieve-edit-rerank text generation, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online. pp. 2532-2538. URL: https: //aclanthology.org/2020.acl-main.228, doi:10.18653/v1/2020.acl-main.228. · DOI
  20. Husain, H., Wu, H.H., Gazit, T., Allamanis, M., Brockschmidt, M., 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436 URL: https: //arxiv.org/abs/1909.09436. · arXiv
  21. Kocetkov, D., Li, R., Allal, L.B., Li, J., Mou, C., Ferrandis, C.M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., et al., 2022. The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533 URL: https://arxiv.org/abs/2211.15533. · arXiv
  22. Le Goues, C., Nguyen, T., Forrest, S., Weimer, W., 2012. Genprog: A generic method for automatic software repair. IEEE Trans. Softw. Eng. 38, 54-72. URL: https://doi.org/10.1109/TSE. 2011.104, doi:10.1109/TSE.2011.104. · DOI
  23. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., Carlini, N., 2022. Deduplicating training data makes language models better, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland. pp. 8424-8445. URL: https://aclanthology. org/2022.acl-long.577, doi:10.18653/v1/2022.acl-long.577. · DOI · Paper page
  24. Li, R., Allal, L.B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al., 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 URL: https://arxiv.org/abs/2305.06161. · arXiv
  25. Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A.D., et al., 2022. Competition-level code generation with alphacode. ArXiv preprint abs/2203.07814. URL: https://arxiv.org/abs/2203.07814. · arXiv
  26. Lu, S., Duan, N., Han, H., Guo, D., Hwang, S.w., Svyatkovskiy, A., 2022. ReACC: A retrieval- augmented code completion framework, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computa- tional Linguistics, Dublin, Ireland. pp. 6227-6240. URL: https://aclanthology.org/2022. acl-long.431, doi:10.18653/v1/2022.acl-long.431. · DOI · Paper page
  27. Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., GONG, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S.K., Fu, S., LIU, S., 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation, in: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). URL: https: //openreview.net/forum?id=6lE4dQXaUcb.
  28. Nijkamp, E., Hayashi, H., Xiong, C., Savarese, S., Zhou, Y., 2023a. Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint arXiv:2305.02309 URL: https://arxiv.org/abs/2305.02309. · arXiv
  29. Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., Xiong, C., 2023b. Codegen: An open large language model for code with multi-turn program synthesis, in: Interna- tional Conference on Learning Representations. URL: https://openreview.net/forum?id= iaYcJKpY2B_.
  30. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P.F., Leike, J., Lowe, R., 2022. Training language models to follow instructions with human feedback, in: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 27730-27744. URL: https://proceedings.neurips.cc/paper_files/paper/2022/ file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
  31. Pei, H., Zhao, J., Lausen, L., Zha, S., Karypis, G., 2023. Better context makes better code language models: A case study on function call argument completion, in: Proceedings of the Thirty- Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI Press. URL: https://doi.org/10.1609/aaai.v37i4.25653, doi:10.1609/aaai.v37i4.25653. · DOI
  32. Puri, R., Kung, D.S., Janssen, G., Zhang, W., Domeniconi, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M., Decker, L., Thost, V., Buratti, L., Pujar, S., Ramji, S., Finkler, U., Malaika, S., Reiss, F., 2021. Codenet: A large-scale AI for code dataset for learning a diversity of coding tasks, in: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). URL: https://openreview.net/forum?id=6vZVBkCDrHT.
  33. Raychev, V., Bielik, P., Vechev, M., 2016. Probabilistic model for code with decision trees, in: Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, Association for Computing Machinery, New York, NY, USA. p. 731-747. URL: https://doi.org/10.1145/2983990.2984041, doi:10.1145/2983990.2984041. · DOI
  34. Shrivastava, D., Larochelle, H., Tarlow, D., 2023. Repository-level prompt generation for large language models of code, in: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (Eds.), Proceedings of the 40th International Conference on Machine Learning, PMLR. pp. 31693-31715. URL: https://proceedings.mlr.press/v202/shrivastava23a.html.
  35. Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., Nallapati, R., Ramanathan, M.K., Roth, D., Xiang, B., 2023. ReCode: Robustness evaluation of code generation models, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada. pp. 13818-13843. URL: https://aclanthology.org/2023.acl-long. 773, doi:10.18653/v1/2023.acl-long.773. · DOI · Paper page
  36. Wang, Y., Wang, W., Joty, S., Hoi, S.C., 2021. CodeT5: Identifier-aware unified pre-trained encoder- decoder models for code understanding and generation, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic. pp. 8696-8708. URL: https://aclanthology. org/2021.emnlp-main.685, doi:10.18653/v1/2021.emnlp-main.685. · DOI · Paper page
  37. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., Rush, A., 2020. Transformers: State- of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online. pp. 38-45. URL: https://aclanthology.org/2020.emnlp-demos.6, doi:10.18653/v1/2020.emnlp-demos. · DOI · Paper page
  38. Xu, F.F., Alon, U., Neubig, G., Hellendoorn, V.J., 2022. A systematic evaluation of large language models of code, in: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, Association for Computing Machinery, New York, NY, USA. p. 1-10. URL: https://doi.org/10.1145/3520312.3534862, doi:10.1145/3520312.3534862. · DOI · Paper page
  39. Zhang, F., Chen, B., Zhang, Y., Liu, J., Zan, D., Mao, Y., Lou, J.G., Chen, W., 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570 URL: https://arxiv.org/abs/2303.12570. · arXiv

Cited by

Sorted by citations

Authors
Ding, Y., Wang, Z., Ahmad, W. U., Ding, H., Tan, M., Jain, N., Ramanathan, M. K., Nallapati, R., Bhatia, P., Roth, D., & Xiang, B.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
46701-46723
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2310.11248
arXiv
2310.11248
OpenAlex
W7133187544
Cited by
37
References
1
Citations by year
2026: 18 · 2025: 14 · 2024: 5
FWCI
9.46
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
multilingual programming languages

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review