Execution-Based Evaluation for Open-Domain Code Generation
Open accessAbstract
To extend the scope of coding queries to more realistic settings, we propose ODEX, the first Open-Domain EXecution-based natural language (NL) to Python code generation dataset. ODEX has 945 NL-Code pairs spanning 79 diverse libraries, along with 1,707 human-written test cases for execution. Our NL-Code pairs are harvested from StackOverflow forums to encourage natural and practical coding queries. Moreover, ODEX supports four natural languages as intents, in English, Spanish, Japanese, and Russian. ODEX unveils intriguing behavioral differences among top-performing code language models (LM). While CODEX achieves better overall results, CODEGEN improves effectively via scaling -- CODEGEN 6.1B performs comparably with CODEX 12B. Both models show substantial gaps between open and closed domains, but CODEGEN gaps tend to decrease with model size while CODEX gaps increase. We release ODEX to facilitate research into open-domain problems for the code generation community.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing code generation benchmarks focus mainly on English prompts to multilingual code or cover only a small number of natural languages.
“However, existing benchmarks have mainly concentrated on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs).”
From HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization (2024) · cites this paper as Wang et al., 2022
The Related Work section says ODEX uniquely draws on StackOverflow forums in various languages, and the Abstract says its prompts span English, Spanish, Japanese, and Russian. This supports the point that earlier benchmarks had limited natural-language coverage, but the passages do not establish that existing benchmarks mainly translate English prompts into multiple programming languages. (LLM, conf=0.72)
Experimental — accuracy not yet measuredODEX introduced a dataset with 4 natural languages and Python, averaging 1.8 test cases.
“ODEX (Wang et al., 2022) introduced a dataset comprising 4 NLs and Python, albeit with a limited average of 1.8 test cases.”
From HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization (2024) · cites this paper as Wang et al., 2022
Our approach prioritizes objective evaluation through execution accuracy, providing more rigorous validation than non-executable assessment methodologies.
“Our approach prioritizes objective evaluation through execution accuracy , providing more rigorous validation than non-executable assessment methodologies (Wang et al., 2022; Chen et al., 2021).”
From SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers (2025) · cites this paper as Wang et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 16 · Cited by 20
Research works printed but not linked: 17
- Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 33-43, Berlin, Germany. Association for Computa- tional Linguistics. · DOI · Paper page
- Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing sta- tistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392, Melbourne, Aus- tralia. Association for Computational Linguistics. · DOI · Paper page
- Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. 2023. Out of the bleu: how should we assess quality of the code generation mod- els? Journal of Systems and Software, 203:111741.
- Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin Clement, and Nan Duan. 2022. Execution-based evaluation for data science code generation models. In Proceedings of the Fourth Workshop on Data Science with Human- in-the-Loop (Language Advances), pages 28-36, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1643-1652, Brussels, Bel- gium. Association for Computational Linguistics. · DOI · Paper page
- Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092-1097.
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Chin-Yew Lin and Franz Josef Och. 2004. Auto- matic evaluation of machine translation quality using longest common subsequence and skip-bigram statis- tics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL- 04), pages 605-612, Barcelona, Spain. · DOI · Paper page
- Stephan Lukasczyk and Gordon Fraser. 2022. Pyn- guin: Automated unit test generation for python. In International Conference on Software Engineering: Companion Proceedings.
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In International Conference on Learning Representa- tions.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. · DOI · Paper page
- Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics. · DOI
- Morteza Verdi, Ashkan Sami, Jafar Akhondali, Foutse Khomh, Gias Uddin, and Alireza Karami Motlagh. 2020. An empirical study of c++ vulnerabilities in crowd-sourced code examples. IEEE Transactions on Software Engineering.
- Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. 2021. Concealed data poisoning attacks on NLP models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 139-150, Online. Association for Computational Linguistics. · DOI · Paper page
- Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank F. Xu, and Graham Neubig. 2023. MCoNaLa: A bench- mark for code generation from multiple natural lan- guages. In Findings of the Association for Compu- tational Linguistics: EACL 2023, pages 265-273, Dubrovnik, Croatia. Association for Computational Linguistics.
- Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. Learning to mine aligned code and natural language pairs from stack overflow. In International Conference on Min- ing Software Repositories, pages 476-486. IEEE.
- Pengcheng Yin and Graham Neubig. 2018. TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pages 7-12, Brussels, Belgium. Association for Computational Linguistics. · DOI · Paper page
- Authors
- Wang, Z., Zhou, S., Fried, D., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 1271-1290
- arXiv
- 2212.10481
- OpenAlex
- W4389519979
- Cited by
- 20
- References
- 16
- Citations by year
- 2026: 2 · 2025: 8 · 2024: 8 · 2023: 2
- FWCI
- 5.14
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- multilingual code generation, codices, CodeGen, model size scaling, test case evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
