CoChat
Sign Up
Cited by 11Software Engineering ResearchDomain Adaptation and Few-Shot LearningTopic Modeling

CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Models

Open access
Niu, Changan, Li, Chuanyi, Ng, Vincent, Luo, Bin·2023·2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)
DOI 10.1109/icse48619.2023.00055
Open this paper in CoChat — free accountOpen PDF

Abstract

Despite the recent advances showing that a model pre-trained on large-scale source code data is able to gain appreciable generalization capability, it still requires a sizeable amount of data on the target task for fine-tuning. And the effectiveness of the model generalization is largely affected by the size and quality of the fine-tuning data, which is detrimental for target tasks with limited or unavailable resources. Therefore, cross-task generalization, with the goal of improving the generalization of the model to unseen tasks that have not been seen before, is of strong research and application value. In this paper, we propose a large-scale benchmark that includes 216 existing code-related tasks. Then, we annotate each task with the corresponding meta information such as task description and instruction, which contains detailed information about the task and a solution guide. This also helps us to easily create a wide variety of “training/evaluation” task splits to evaluate the various cross-task generalization capabilities of the model. Then we perform some preliminary experiments to demonstrate that the cross-task generalization of models can be largely improved by in-context learning methods such as few-shot learning and learning from task instructions, which shows the promising prospects of conducting cross-task learning research on our benchmark. We hope that the collection of the datasets and our benchmark will facilitate future work that is not limited to cross-task generalization.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/icse48619.2023.00055

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • DevBench, CrossCodeBench, LiveCodeBench, KernelBench, and NL2Repo each advance scope but none separate proxy from true objective.

    “DevBench (Li et al., 2025), CrossCodeBench (Niu et al., 2023), LiveCodeBench (Jain et al., 2024), KernelBench (Ouyang et al., 2025), and NL2Repo (Ding et al., 2025) each advance scope but none separate proxy from true objective.”

    From SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents (2026) · cites this paper as Niu et al., 2023

    CrossCodeBench does advance benchmark scope: the paper describes 216 tasks across 28 categories and 18 programming languages. But its “Shortcut Methods” section explicitly tests whether models gain actual cross-task capability rather than merely appear to do so through shortcuts; the Results section says the comparison helps distinguish real capability from simple shortcuts. That runs contrary to the claim that CrossCodeBench does not separate a proxy from the objective it evaluates. (LLM, conf=0.78)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. C. Niu, C. Li, B. Luo, and V. Ng, "Deep learning meets software engineering: A survey on pre-trained models of source code," in Pro- ceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, 2022, pp. 5546-5555.
  2. N. T. De Sousa and W. Hasselbring, "Javabert: Training a transformer- based model for the java programming language," in 2021 36th IEEE/ACM International Conference on Automated Software Engineer- ing Workshops (ASEW). IEEE, 2021, pp. 90-95.
  3. N. Chirkova and S. Troshin, "Empirical study of transformers for source code," in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2021, pp. 703-715.
  4. A. Karmakar and R. Robbes, "What do pre-trained code models know about code?" in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 1332-1336.
  5. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, "Codebert: A pre-trained model for programming and natural languages," in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, 2020, pp. 1536-1547.
  6. A. Mastropaolo, S. Scalabrino, N. Cooper, D. N. Palacio, D. Poshy- vanyk, R. Oliveto, and G. Bavota, "Studying the usage of text-to-text transfer transformer to support code-related tasks," in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 336-347.
  7. W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, "Unified pre- training for program understanding and generation," in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 2655-2668.
  8. C. Niu, C. Li, V. Ng, J. Ge, L. Huang, and B. Luo, "Spt-code: Sequence- to-sequence pre-training for learning source code representations," in 2022 IEEE/ACM 44th International Conference on Software Engineer- ing (ICSE), 2022, pp. 01-13.
  9. A. D. Gotmare, J. Li, S. Joty, and S. C. Hoi, "Cascaded fast and slow models for efficient semantic code search," arXiv preprint arXiv:2110.07811, 2021. · arXiv
  10. D. Wang, Z. Jia, S. Li, Y. Yu, Y. Xiong, W. Dong, and X. Liao, "Bridging pre-trained models and downstream tasks for source code understanding," in 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE). IEEE, 2022, pp. 287-298.
  11. M. Yu, X. Guo, J. Yi, S. Chang, S. Potdar, Y. Cheng, G. Tesauro, H. Wang, and B. Zhou, "Diverse few-shot text classification with multiple metrics," in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018, pp. 1206-1215.
  12. O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra et al., "Matching net- works for one shot learning," Advances in neural information processing systems, vol. 29, 2016.
  13. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language mod- els are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
  14. Q. Ye, B. Y. Lin, and X. Ren, "Crossfit: A few-shot learning challenge for cross-task generalization in nlp," in Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, 2021, pp. 7163-7189.
  15. A. Efrat and O. Levy, "The turking test: Can language models understand instructions?" arXiv preprint arXiv:2010.11982, 2020. · arXiv
  16. O. Weller, N. Lourie, M. Gardner, and M. E. Peters, "Learning from task descriptions," in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020, pp. 1361-1375.
  17. J. Gu, Y. Wang, Y. Chen, V. O. Li, and K. Cho, "Meta-learning for low-resource neural machine translation," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 3622-3631.
  18. Z.-Y. Dou, K. Yu, and A. Anastasopoulos, "Investigating meta-learning algorithms for low-resource natural language understanding tasks," in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 1192-1197.
  19. J. A. A. Prenner and R. Robbes, "Making the most of small software engineering datasets with modern machine learning," IEEE Transactions on Software Engineering, 2022.
  20. J. Phang, T. Févry, and S. R. Bowman, "Sentence encoders on stilts: Sup- plementary training on intermediate labeled-data tasks," arXiv preprint arXiv:1811.01088, 2018. · arXiv
  21. J. A. Prenner and R. Robbes, "Automatic program repair with openai's codex: Evaluating quixbugs," arXiv preprint arXiv:2111.03922, 2021. · arXiv
  22. S. D. Kolak, R. Martins, C. Le Goues, and V. J. Hellendoorn, "Patch generation with language models: Feasibility and scaling behavior," in Deep Learning for Code Workshop, 2022.
  23. H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, "Examining zero-shot vulnerability repair with large language models," in 2023 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, 2022, pp. 1-18.
  24. O. Asare, M. Nagappan, and N. Asokan, "Is github's copilot as bad as humans at introducing vulnerabilities in code?" arXiv preprint arXiv:2204.04741, 2022. · arXiv
  25. E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol, and H. Larochelle, "Meta-dataset: A dataset of datasets for learning to learn from few examples," in International Conference on Learning Representations, 2020.
  26. Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Arunkumar, A. Ashok, A. S. Dhanasekaran, A. Naik, D. Stap et al., "Benchmarking generalization via in-context instructions on 1,600+ language tasks," arXiv preprint arXiv:2204.07705, 2022. · arXiv
  27. Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, "Deepface: Closing the gap to human-level performance in face verification," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 1701-1708.
  28. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al., "Mastering the game of go with deep neural networks and tree search," nature, vol. 529, no. 7587, pp. 484-489, 2016.
  29. A. Najberg, "Alibaba ai model tops humans in reading comprehension," Alizila. com, January, vol. 15, 2018.
  30. S. M. Xie, A. Raghunathan, P. Liang, and T. Ma, "An explanation of in-context learning as implicit bayesian inference," in International Conference on Learning Representations, 2021.
  31. Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, "Generalizing from a few examples: A survey on few-shot learning," ACM computing surveys (csur), vol. 53, no. 3, pp. 1-34, 2020.
  32. W. Yin, "Meta-learning for few-shot natural language processing: A survey," arXiv preprint arXiv:2007.09604, 2020. · arXiv
  33. D. Paperno, G. Kruszewski, A. Lazaridou, N.-Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández, "The lam- bada dataset: Word prediction requiring a broad discourse context," in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016, pp. 1525- 1534.
  34. N. Durrani, B. Haddow, P. Koehn, and K. Heafield, "Edinburgh's phrase- based machine translation systems for wmt-14," in Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014, pp. 97-104.
  35. Z. Chen, H. Eavani, W. Chen, Y. Liu, and W. Y. Wang, "Few-shot nlg with pre-trained language model," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 183-190.
  36. G. I. Winata, A. Madotto, Z. Lin, R. Liu, J. Yosinski, and P. Fung, "Language models are few-shot multilingual learners," in Proceedings of the 1st Workshop on Multilingual Representation Learning, 2021, pp. 1-15.
  37. Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela, "Adversarial nli: A new benchmark for natural language understanding," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 4885-4901.
  38. S. Bach, V. Sanh, Z. X. Yong, A. Webson, C. Raffel, N. V. Nayak, A. Sharma, T. Kim, M. S. Bari, T. Févry et al., "Promptsource: An integrated development environment and repository for natural language prompts," in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2022, pp. 93- 104.
  39. C.-Y. Lin and E. Hovy, "Manual and automatic evaluation of sum- maries," in Proceedings of the ACL-02 Workshop on Automatic Sum- marization -Volume 4, 2002, pp. 45-51.
  40. S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. GONG, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. LIU, "CodeXGLUE: A machine learning benchmark dataset for code understanding and generation," in Thirty- fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
  41. A. Elnaggar, W. Ding, L. Jones, T. Gibbs, T. Feher, C. Angerer, S. Severini, F. Matthes, and B. Rost, "Codetrans: Towards cracking the language of silicon's code through self-supervised deep learning and high performance computing," arXiv preprint arXiv:2104.02443, 2021. · arXiv
  42. M. Zhu, A. Jain, K. Suresh, R. Ravindran, S. Tipirneni, and C. K. Reddy, "Xlcost: A benchmark dataset for cross-lingual code intelligence," arXiv preprint arXiv:2206.08474, 2022. · arXiv
  43. R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V. Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker et al., "Codenet: A large- scale ai for code dataset for learning a diversity of coding tasks," in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  44. L. Sgier, "Qualitative data analysis," An Initiat. Gebert Ruf Stift, vol. 19, pp. 19-21, 2012.
  45. O. Chaparro, J. Lu, F. Zampetti, L. Moreno, M. Di Penta, A. Marcus, G. Bavota, and V. Ng, "Detecting missing information in bug descrip- tions," in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, 2017, pp. 396-407.
  46. J. Zhai, X. Xu, Y. Shi, G. Tao, M. Pan, S. Ma, L. Xu, W. Zhang, L. Tan, and X. Zhang, "Cpc: Automatically classifying and propagating natural language comments via program analysis," in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 2020, pp. 1359-1371.
  47. Q. Chen, X. Xia, H. Hu, D. Lo, and S. Li, "Why my code summarization model does not work: Code comment improvement with category pre- diction," ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 30, no. 2, pp. 1-29, 2021.
  48. J. Cohen, "A coefficient of agreement for nominal scales," Educational and psychological measurement, vol. 20, no. 1, pp. 37-46, 1960.
  49. A. J. Viera, J. M. Garrett et al., "Understanding interobserver agreement: the kappa statistic," Fam med, vol. 37, no. 5, pp. 360-363, 2005.
  50. M. Tufano, C. Watson, G. Bavota, M. Di Penta, M. White, and D. Poshyvanyk, "Learning how to mutate source code from bug-fixes," in 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE Computer Society, 2019, pp. 301-312.
  51. J. Svajlenko, J. F. Islam, I. Keivanloo, C. K. Roy, and M. M. Mia, "Towards a big data curated benchmark of inter-project code clones," in Proceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution, 2014, pp. 476-480.
  52. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "Bleu: a method for automatic evaluation of machine translation," in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311-318.

Cited by

Sorted by citations

Authors
Niu, C., Li, C., Ng, V., & Luo, B.
Affiliations
  • State Key Laboratory of New Technology of Computer Software (CN)
  • The University of Texas at Dallas (US)
ORCID
Venue
2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)
Publisher
IEEE
Published
2023-05
Pages
537-549
ISSN
0270-5257 (ISSN-L)
DOI
10.1109/icse48619.2023.00055
OpenAlex
W4384345646
Cited by
11
References
21
Citations by year
2025: 4 · 2024: 3 · 2023: 4
FWCI
2.83
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Natural Science Foundation of China (61802167,2034508)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
cross-task generalization, benchmark, in-context learning, few-shot learning, task instructions

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review