CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
Open accessAbstract
Evaluation metrics play a vital role in the growth of an area as it defines the standard of distinguishing between good and bad models. In the area of code synthesis, the commonly used evaluation metric is BLEU or perfect accuracy, but they are not suitable enough to evaluate codes, because BLEU is originally designed to evaluate the natural language, neglecting important syntactic and semantic features of codes, and perfect accuracy is too strict thus it underestimates different outputs with the same semantic logic. To remedy this, we introduce a new automatic evaluation metric, dubbed CodeBLEU. It absorbs the strength of BLEU in the n-gram match and further injects code syntax via abstract syntax trees (AST) and code semantics via data-flow. We conduct experiments by evaluating the correlation coefficient between CodeBLEU and quality scores assigned by the programmers on three code synthesis tasks, i.e., text-to-code, code translation, and code refinement. Experimental results show that our proposed CodeBLEU can achieve a better correlation with programmer assigned scores compared with BLEU and accuracy.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing works mainly focus on the evaluation of LLMs on one or two tasks and do not discuss the construction of in-context demonstrations in-depth.
“However, these works mainly focus on the evaluation of LLMs on one or two tasks and do not discuss the construction of in-context demonstrations in-depth.”
From What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs? (2023) · cites this paper as [22]
The Experiments section (p. 4) says CodeBLEU is evaluated on three code-synthesis tasks, contrary to the claim that this work focuses on one or two tasks. The paper evaluates an automatic metric against programmers’ quality scores, not LLMs; our copy does not discuss constructing in-context demonstrations. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredExisting match-based evaluation metrics for code translation focus solely on semantics, overlooking executability of the code and the functional equivalence under different implementations.
“Existing match-based evaluation metrics for code translation (Papineni et al., 2002;Ren et al., 2020;Eghbali and Pradel, 2022;Zhou et al., 2023;Tran et al., 2019) focus solely on semantics, overlooking executability of the code and the functional equivalence under different implementations.”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Ren et al., 2020
The Introduction (p. 1) describes n-gram accuracy as token overlap, not a metric focused solely on semantics; it says n-gram accuracy can favor code with logical errors. The Abstract (p. 1) likewise says BLEU neglects syntactic and semantic features. The paper does support the narrower concern that exact-match accuracy underestimates different implementations with the same semantic logic. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredBLEU is used to measure translation accuracy for the four strategies because CodeBLEU does not cover all eight languages in MultilingualTrans.
“Since the CodeBLEU metric (Ren et al., 2020) does not cover all eight languages in MultilingualTrans, we use BLEU to measure translation accuracy for the four strategies.”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Ren et al., 2020
Section 3 (p. 2) identifies this paper as the source of CodeBLEU, and Section 4 (p. 4) evaluates it on Java and Java-to-C# tasks. Neither those passages nor the passages on AST and data-flow matching (Sections 3.2–3.3, p. 3) establish whether CodeBLEU covers all eight languages in MultilingualTrans. The cited paper therefore supports the identification of the metric, but not the stated reason for choosing BLEU. (LLM, conf=0.83)
Experimental — accuracy not yet measuredExisting multilingual datasets use string similarity metrics like BLEU for evaluation rather than verifying functional correctness of generated code.
“existing multilingual datasets [18,28,45] use string similarity metrics like BLEU [21] for evaluation rather than really verify the functional correctness of generated code.”
From CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023) · cites this paper as [28]
Real-world data provides benefits in capturing the complexity of actual webpages.
“Design2Code, which also utilizes real-world data through the C4 dataset[36]from the Common Crawl corpus, exhibits a comparable distribution across these metrics, underscoring the benefits of real-world data in capturing the complexity of actual webpages.”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [36]
The Abstract (p. 1) describes CodeBLEU as a metric for evaluating code synthesis, and §4.1 (pp. 4–5) describes datasets for code-synthesis tasks. Searches for Common Crawl, C4, real-world data, and webpages found no passage addressing whether real-world data captures the complexity of actual webpages. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Design2Code, which utilizes real-world data through the C4 dataset, exhibits a comparable distribution across these metrics to WebCode2M.
“Design2Code, which also utilizes real-world data through the C4 dataset[36]from the Common Crawl corpus, exhibits a comparable distribution across these metrics, underscoring the benefits of real-world data in capturing the complexity of actual webpages.”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [36]
Section 4.1 (p. 4) describes datasets for code-synthesis tasks, while Section 4.3 (pp. 5–6) compares evaluation metrics. Searches for Design2Code, WebCode2M, C4, and Common Crawl found no passage addressing the attributed dataset or distribution comparison. (LLM, conf=0.77)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 1× · see citing works
“As code artificial intelligence receives more and more attention (Allamanis et al. 2015; Yin and Neubig 2017; Al-lamanis et al. 2018; Monperrus 2018; Alon et al. 2019; Svyatkovskiy et al. 2020), the evaluation of code synthesis becomes critical to promote its development.”
BackgroundSelf-citationAn Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation
Tufano, M., Watson, C., Bavota, G., Penta, M. D., White, M., & Poshyvanyk, D. (2019) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3340544 · Cited by 370 · see citing works
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243 · see citing works
Self-citationMapping Language to Code in Programmatic Context
Iyer, S., Konstas, I., Cheung, A., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1192 · Cited by 180 · see citing works
Flagged first, then by citations
Research works printed but not linked: 23
- Allamanis, M.; Barr, E. T.; Devanbu, P.; and Sutton, C. 2018. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR) 51(4): 1-37.
- Allamanis, M.; Tarlow, D.; Gordon, A.; and Wei, Y. 2015. Bimodal modelling of source code and natural language. In International conference on machine learning , 2123-2132.
- Alon, U.; Zilberstein, M.; Levy, O.; and Yahav, E. 2019. code2vec: Learning distributed representations of code. Proceedings of the ACM on Programming Languages 3(POPL): 1-29.
- Bahdanau, D.; Cho, K.; and Bengio, Y. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . · arXiv
- Barone, A. V. M.; and Sennrich, R. 2017. A parallel corpus of Python functions and documentation strings for automated code documentation and code generation. arXiv preprint arXiv:1707.02275 . · arXiv
- Chen, X.; Liu, C.; and Song, D. 2018. Tree-to-tree neural networks for program translation. In Advances in neural information processing systems , 2547-2557.
- Dinella, E.; Dai, H.; Li, Z.; Naik, M.; Song, L.; and Wang, K. 2020. Hoppity: Learning Graph Transformations to Detect and Fix Bugs in Programs. In International Conference on Learning Representations .
- Guo, D.; Ren, S.; Lu, S.; Feng, Z.; Tang, D.; Liu, S.; Zhou, L.; Duan, N.; Yin, J.; Jiang, D.; et al. 2020. GraphCodeBERT: Pre-training Code Representations with Data Flow. arXiv preprint arXiv:2009.08366 . · arXiv
- Guo, D.; Tang, D.; Duan, N.; Zhou, M.; and Yin, J. 2019. Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing. arXiv preprint arXiv:1906.07108 . · arXiv
- Husain, H.; Wu, H.-H.; Gazit, T.; Allamanis, M.; and Brockschmidt, M. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436 . · arXiv
- Kanade, A.; Maniatis, P.; Balakrishnan, G.; and Shi, K. 2019. Pre-trained contextual embedding of source code. arXiv preprint arXiv:2001.00059 . · arXiv
- Karaivanov, S.; Raychev, V.; and Vechev, M. 2014. Phrasebased statistical translation of programming languages. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software , 173-184.
- Lachaux, M.-A.; Roziere, B.; Chanussot, L.; and Lample, G. 2020. Unsupervised Translation of Programming Languages. arXiv preprint arXiv:2006.03511 . · arXiv
- Monperrus, M. 2018. Automatic software repair: a bibliography. ACM Computing Surveys (CSUR) 51(1): 1-24.
- Nguyen, A. T.; Nguyen, T. T.; and Nguyen, T. N. 2015. Divide-and-conquer approach for multi-phase statistical migration for source code (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) , 585-596. IEEE.
- Oda, Y.; Fudaba, H.; Neubig, G.; Hata, H.; Sakti, S.; Toda, T.; and Nakamura, S. 2015. Learning to generate pseudocode from source code using statistical machine translation (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) , 574-584. IEEE.
- Rabinovich, M.; Stern, M.; and Klein, D. 2017. Abstract syntax networks for code generation and semantic parsing. arXiv preprint arXiv:1704.07535 . · arXiv
- Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1(8): 9.
- Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems , 3104-3112.
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. In Advances in Neural Information Processing Systems , 6000-6010.
- Weaver, W. 1955. Translation. Machine translation of languages 14(15-23): 10.
- Yin, P.; and Neubig, G. 2017. A Syntactic Neural Model for General-Purpose Code Generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 440-450. Vancouver, Canada: Association for Computational Linguistics.
- Zhou, L.; Zhang, J.; Zong, C.; and Yu, H. 2019. Sequence generation: From both sides to the middle. In Proceedings of IJCAI 2019 .
Cited by
Sorted by citations
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
“For instance, Ren et al. (2020) finds that BLEU has problems capturing semantic features specific to code, and suggests several semantic modifications to the score.”
BackgroundWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · see citing works
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 263 · see citing works
Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
Pan, R., Ibrahimzada, A. R., Krishna, R., Sankar, D., Wassi, L. P., Merler, M., Sobolev, B., Pavuluri, R., Sinha, S., & Jabbarvand, R. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639226 · Cited by 133 · see citing works
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3267446 · Cited by 109 · see citing works
What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?
Gao, S., Wen, X.-C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00109 · Cited by 106 · see citing works
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
Zhou, S., Alon, U., Agarwal, S., & Neubig, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.859 · Cited by 100 · see citing works
Fully Autonomous Programming with Large Language Models
Liventsev, V., Grishina, A., Härmä, A., & Moonen, L. (2023) · Proceedings of the Genetic and Evolutionary Computation Conference · doi:10.1145/3583131.3590481 · Cited by 45 · see citing works
Multilingual Code Snippets Training for Program Translation
Zhu, M., Suresh, K., & Reddy, C. K. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21434 · Cited by 39 · see citing works
CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
Yan, W., Tian, Y., Li, Y., Chen, Q., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.337 · Cited by 31 · see citing works
WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs
Gui, Y., Li, Z., Wan, Y., Shi, Y., Zhang, H., Chen, B., Su, Y., Chen, D., Wu, S., Zhou, X., et al. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714889 · Cited by 17 · see citing works
Bhattarai, M., Santos, J. E., Jones, S., Biswas, A., Alexandrov, B., & O'Malley, D. (2024) · 2024 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec62836.2024.10938485 · Cited by 13 · see citing works
RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation
Wang, Y., Wang, Y., Wang, S., Guo, D., Chen, J., Grundy, J., Liu, X., Ma, Y., Mao, M., Zhang, H., et al. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3645056 · Cited by 12 · see citing works
Self-citationPlanetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages
Zuo, M., Velez, F. P., Li, X., Littman, M., & Bach, S. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.560 · Cited by 5 · see citing works
“Some benchmarks for code generation use match-based metrics that look for segments that overlap with ground truth code (Pap-ineni et al., 2002; Lin, 2004; Ren et al., 2020), but match-based metrics cannot cover the vast space of equivalent programs (Chen et al., 2021).”
Schuster, T., Kalyan, A., Polozov, O., & Kalai, A. T. (2021) · arXiv · doi:10.48550/arxiv.2106.05784 · Cited by 2 · see citing works
SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
Xiang, Y., Yan, H., Ouyang, S., Gui, L., & He, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.00255 · Cited by 1 · see citing works
“For evaluation, the following components are provided for code execution and metrics calculation: Reference implementation : ground-truth code serving as the reference for CodeBLEU evaluation.”
InfluentialWhat Makes In-Context Examples Effective for Code Generation?
Li, D., Chen, S., Cao, J., & Cheung, S.-C. (2026) · Proceedings of the ACM on Software Engineering · doi:10.1145/3832275
Post-Incorporating Code Structural Knowledge Into Pretrained Models via ICL for Code Translation
Du, Y., Sun, H., & Li, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3605768
Bhattarai, M., Santos, J. E., Jones, S., Biswas, A., Alexandrov, B., & O'Malley, D. (2024) · arXiv
What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?
Gao, S., Wen, X.-C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023) · arXiv
- Authors
- Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., & Ma, S.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2009.10297
- OpenAlex
- W3089307846
- Cited by
- 214
- References
- 5
- Citations by year
- 2026: 10 · 2025: 48 · 2024: 55 · 2023: 72 · 2022: 13 · 2021: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- automatic evaluation metrics, program synthesis, BLEU, abstract syntax tree, data flow analysis, program semantics, program translation, program refinement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
