Learning to mine aligned code and natural language pairs from stack overflow
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
For tasks like code synthesis from natural language, code retrieval, and code summarization, data-driven models have shown great promise. However, creating these models require parallel data between natural language (NL) and code with fine-grained alignments. Stack Overflow (SO) is a promising source to create such a data set: the questions are diverse and most of them have corresponding answers with high quality code snippets. However, existing heuristic methods (e.g., pairing the title of a post with the code in the accepted answer) are limited both in their coverage and the correctness of the NL-code pairs obtained. In this paper, we propose a novel method to mine high-quality aligned data from SO using two sets of features: hand-crafted features considering the structure of the extracted snippets, and correspondence features obtained by training a probabilistic model to capture the correlation between NL and code using neural networks. These features are fed into a classifier that determines the quality of mined NL-code pairs. Experiments using Python and Java as test beds show that the proposed method greatly expands coverage and accuracy over existing mining methods, even when using only a small number of labeled examples. Further, we find that reasonable results are achieved even when training the classifier on one language and testing on another, showing promise for scaling NL-code mining to a wide variety of programming languages beyond those for which we are able to annotate data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This model reports state-of-the-art accuracy on the CoNaLa benchmark dataset [122]”
From In-IDE Code Generation from Natural Language: Promise and Challenges (2022) · cites this paper as [122]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“vent overfitting, on the input of the last softmax layer over target words (p = 0.5), and recurrent dropout [11] on RNNs (p = 0.2). We train the network using the widely used optimization method Adam [18]. To evaluate the neural network, we use the remaining 10% of pairs left aside for testing, retaining the model wit…”
MethodInfluential(2014) · Cited by 33,419
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“So how do neural MT models calculate this probability? We will explain a basic outline of a basic model called the encoder-decoder model [39], and refer readers to references for details [4, 25, 39].”
BackgroundEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 1×
“allows the model to “focus” on particular words in the source snippet S when generating the intent I. The details of attention are beyond the scope of this paper, but interested readers can reference [4, 22]. Why attentional neural MT models?: Attention-based neural MT models are well-suited to the task of learning co…”
BackgroundLearning deep structured semantic models for web search using clickthrough data
Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., & Heck, L. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2505665 · Cited by 2,009 · cited 1×
“Second, they are powerful models that can learn correspondences on a variety of levels; from simple phenomena such as direct word-by-word matches [12], to soft paraphrases [36], to weak correspondences between keywords and large documents for information retrieval [14].”
BackgroundInternational Conference on Learning Representations (ICLR)
Abbas, M. Z. A., Kivinen, J., & Raiko, T. (2016) · Cited by 1,694
A theoretically grounded application of dropout in recurrent neural networks
Gal, Y., & Ghahramani, Z. (2016) · Cambridge University Engineering Department Publications Database · Cited by 1,583
Incorporating Copying Mechanism in Sequence-to-Sequence Learning
Gu, J., Lu, Z., Li, H., & Li, V. O. K. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1154 · Cited by 1,446 · cited 1×
“classifier, as described in the following paragraph. Second, they are powerful models that can learn correspondences on a variety of levels; from simple phenomena such as direct word-by-word matches [12], to soft paraphrases [36], to weak correspondences between keywords and large documents for information retrieval […”
BackgroundStatistical machine translation
(2010) · Choice Reviews Online · doi:10.5860/choice.47-6293 · Cited by 1,309
Recurrent Continuous Translation Models
Kalchbrenner, N., & Blunsom, P. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1176 · Cited by 1,232 · cited 1×
“between natural language and code for the purposes of code summarization [15, 30], code synthesis from natural language [21], and code retrieval [3]. In particular, we use a model based on neural MT [4, 17], a method for MT based on neural networks that is well-suited for this task of learning correspondences for a va…”
Method
Cited by
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1425 · Cited by 1,034
“, 2018), and CoNaLa (Yin et al., 2018).”
BackgroundCodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536
A Neural Model for Generating Natural Language Summaries of Program Subroutines
LeClair, A., Jiang, S., & McMillan, C. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00087 · Cited by 294
“[21], Movshovitz et al.”
A survey on large language models for code generation
Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2024) · arXiv · doi:10.48550/arxiv.2406.00515 · Cited by 257
A Survey on Large Language Models for Code Generation
Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3747588 · Cited by 251
InCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundBig code != big vocabulary: open-vocabulary models for source code
Karampatsis, R.-M., Babii, H., Robbes, R., Sutton, C., & Janes, A. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380342 · Cited by 182
“Yin et al. [90] mine pairs of natural language and code from Stack Overflow to support tasks such as code synthesis from natural language.”
BackgroundCC2Vec: distributed representations of code changes
Hoang, T., Kang, H. J., Lo, D., & Lawall, J. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380361 · Cited by 174
“[62] trained a model to align source code to natural language text from StackOverflow posts.”
MethodEvaluating Large Language Models in Class-Level Code Generation
Du, X., Liu, M., Wang, K., Wang, H., Liu, J., Chen, Y., Feng, J., Sha, C., Peng, X., & Lou, Y. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639219 · Cited by 162
CC2Vec: Distributed Representations of Code Changes
Hoang, T., Kang, H. J., Lawall, J., & Lo, D. (2020) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · doi:10.48550/arxiv.2003.05620 · Cited by 157
“[62] trained a model to align source code to natural language text from StackOverflow posts.”
MethodIn-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2101.11149 · Cited by 153
Self-citationIn-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154
“, recent approaches to align natural language intents with their corresponding code snippets in Stack Overflow for retrieval purposes [123] or use similar deep learning technology as some code generation techniques [120].”
BackgroundMethodSelf-citationA Systematic Literature Review on the Use of Deep Learning in Software Engineering Research
Watson, C., Cooper, N., Palacio, D. N., Moran, K., & Poshyvanyk, D. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3485275 · Cited by 137
“Code Comprehension [9, 10, 16, 64, 73, 77, 93, 97, 119, 121, 130, 144, 180, 185] Souce Code Generation [26, 37, 57, 63, 74, 84, 124, 136, 151, 173] Source Code Retrieval & Traceability [10, 27, 29, 41, 62, 65, 92, 175, 179] Source Code Summarization [11, 29, 77, 95, 162, 179] Bug-Fixing Process [41, 67, 68, 80, 92, 96…”
MethodInfluentialLi, T. J.-J., Radensky, M., Jia, J., Singarajah, K., Mitchell, T. M., & Myers, B. A. (2019) · Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3332165.3347899 · Cited by 111
“This approach also provides more interpretable results and requires less training data than neural network approaches (e.g., [51, 52]).”
BackgroundCodeS: Towards Building Open-source Language Models for Text-to-SQL
Li, H., Zhang, J., Liu, H., Fan, J., Zhang, X., Zhu, J., Wei, R., Pan, H., Li, C., & Chen, H. (2024) · Proceedings of the ACM on Management of Data · doi:10.1145/3654930 · Cited by 144
“To bridge the gap between natural language questions and SQL queries, we incorporate four types of NL-to-code datasets into our pre-training corpus: (1) CoNaLa [74] and StaQC [73] are derived automatically from Stack Overflow, encompasses many NL-to-Python and NL-to-SQL pairs.”
Out of the BLEU: how should we assess quality of the Code Generation models?
Evtikhiev, M., Bogomolov, E., Sokolov, Y., & Bryksin, T. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111741 · Cited by 127
“[39] and consists of 2,879 examples (split into 2,379 training and 500 test examples), crawled from Stack Overflow and then manually curated by human annotators.”
MethodLarge Language Models Meet NL2Code: A Survey
Zan, D., Chen, B., Zhang, F., Lu, D., Wu, B., Guan, B., Yongji, W., & Lou, J.-G. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.411 · Cited by 114
Semantic Evaluation for Text-to-SQL with Distilled Test Suites
Zhong, R., Yu, T., & Klein, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.29 · Cited by 112
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3267446 · Cited by 394
“There are a number of existing single-language NL2Code benchmarks [8], [9], [10].”
BackgroundInfluentialCodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
Zhou, S., Alon, U., Agarwal, S., & Neubig, G. (2023) · arXiv · doi:10.48550/arxiv.2302.05527 · Cited by 100
Self-citationWhat Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?
Gao, S., Wen, X.-C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00109 · Cited by 106
“For program synthesis, we use the CoNaLa [44] dataset for evaluation.”
MethodCode Generation Using Machine Learning: A Systematic Review
Dehaerne, E., Dey, B., Halder, S., Gendt, S. D., & Meert, W. (2022) · IEEE Access · doi:10.1109/access.2022.3196347 · Cited by 163
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
Zhou, S., Alon, U., Agarwal, S., & Neubig, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.859 · Cited by 100
Self-citationExploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing
Suhr, A., Chang, M.-W., Shaw, P., & Lee, K. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.742 · Cited by 73
“Related to the task of semantic parsing is code generation (Oda et al., 2015; Ling et al., 2016; Yin et al., 2018; Lin et al., 2018; Iyer et al., 2018).”
BackgroundOn the importance of building high-quality training datasets for neural code search
Sun, Z., Li, L., Liu, Y., Du, X., & Li, L. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510160 · Cited by 69
“Hence, researchers[52, 54] also collect the titles of StackOverflow questions paired with proper answers containing sample code snippets, which also form the query-code pairs.”
BackgroundExploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models
Weyssow, M., Zhou, X., Kim, K., Lo, D., & Sahraoui, H. A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714461 · Cited by 203
“Since the objective of our study is to specialize LLMs using PEFT techniques, we selected the CoNaLa dataset [80].”
Incorporating External Knowledge through Pre-training for Natural Language to Code Generation
Xu, F. F., Jiang, Z., Yin, P., Vasilescu, B., & Neubig, G. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.538 · Cited by 64
Self-citationKuhail, M. A., Mathew, S. S., Khalil, A., Berengueres, J., & Shah, S. J. H. (2024) · Science of Computer Programming · doi:10.1016/j.scico.2024.103111 · Cited by 68
CoSQA: 20,000+ Web Queries for Code Search and Question Answering
Huang, J., Tang, D., Shou, L., Gong, M., Xu, K., Jiang, D., Zhou, M., & Duan, N. (2021) · doi:10.18653/v1/2021.acl-long.442 · Cited by 156
“A number of open-sourced datasets with a large amount of text-code pairs have been proposed for the purposes of code search (Husain et al., 2019; Gu et al., 2018; Nie et al., 2016) and code question answering (Yao et al., 2018; Yin et al., 2018; Heyman and Cutsem, 2020).”
BackgroundLearning to Update Natural Language Comments Based on Code Changes
Panthaplackel, S., Nie, P., Gligoric, M., Li, J. J., & Mooney, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.168 · Cited by 66
Zhong, L., & Wang, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30185 · Cited by 219
“…code generation of large language models, most of the existing benchmarks focus on the functional correctness of the execution result from the generated code, which means the code is acceptable as long as it is functional for the user’s purpose (Chen et al. 2021; Yin et al. 2018; Lu et al. 2021).”
BackgroundYan, S., Yu, H., Chen, Y., Shen, B., & Jiang, L. (2020) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner48275.2020.9054840 · Cited by 79
“[42] 2,379 training and 500 test examples, 600k mining example Java/python General purposed A dataset of low quality Iyer et al.”
BackgroundReranking for Neural Semantic Parsing
Yin, P., & Neubig, G. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1447 · Cited by 59
“C O N A L A (Yin et al., 2018a) 3 is a newly introduced task for open-domain code generation.”
BackgroundMethodInfluentialSelf-citationBug severity prediction using question-and-answer pairs from Stack Overflow
Tan, Y., Xu, S., Wang, Z., Zhang, T., Xu, Z., & Luo, X. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110567 · Cited by 62
A Survey on Machine Learning Techniques for Source Code AnalysisPreprint
Sharma, T., Kechagia, M., Georgiou, S., Tiwari, R. K., Vats, I., Moazen, H., & Sarro, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09610 · Cited by 56
“Phan and Jannesari [245] used three corpus for their experiments—a large-scale corpus of English-German translation in nlp [201], the Conala corpus [356], which contains Python software documentation as 116,000 English sentences, and the msr 2013 corpus [18].”
BackgroundMethodA Multi-Perspective Architecture for Semantic Code Search
Haldar, R., Wu, L., Xiong, J., & Hockenmaier, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.758 · Cited by 74
“We apply our model to the CoNaLa benchmark dataset (Yin et al., 2018), which consists of Python code snippets and their corresponding annotations in English.”
MethodInfluentialExploitGen: Template-augmented exploit code generation based on CodeBERT
Yang, G., Zhou, Y., Chen, X., Zhang, X., Han, T., & Chen, T. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111577 · Cited by 64
“More recently, several studies (Norouzi et al., 2021; Ahmad et al., 2021; Phan et al., 2021; Wang et al., 2021) focus on the pre-trained language models (PLMs) for code generation tasks, including general-purpose programming languages (such as Java Iyer et al., 2018 and Python Yin et al., 2018).”
BackgroundMethodComparing Large Language Models and Human Programmers for Generating Programming Code
Hou, W., & Ji, Z. (2025) · Advanced Science · doi:10.1002/advs.202412279 · Cited by 48
“The first type designed new benchmark datasets, including HumanEval 16 , MBPP 17 , APPS 18 , CoNaLA 19 , and CodeContests 20 , to evaluate the performance of LLMs.”
Deep Learning Based Program Generation From Requirements Text: Are We There Yet?
Liu, H., Shen, M., Zhu, J., Niu, N., Li, G., & Zhang, L. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3018481 · Cited by 66
“For example, the source code extracted automatically from QA forums may not exactly fulfill the how-to questions in StaQC and CoNaLa.”
InfluentialNatural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.231 · Cited by 46
Natural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · arXiv · doi:10.48550/arxiv.2204.11454 · Cited by 49
JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation
Agashe, R., Iyer, S., & Zettlemoyer, L. (2019) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d19-1546 · Cited by 92
Mining API usage scenarios from stack overflow
Uddin, G., Khomh, F., & Roy, C. K. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106277 · Cited by 57
A survey on machine learning techniques applied to source code
Sharma, T., Kechagia, M., Georgiou, S., Tiwari, R., Vats, I., Moazen, H., & Sarro, F. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111934 · Cited by 173
Code to comment "translation": data, metrics, baselining & evaluation
Gros, D., Sezhiyan, H., Devanbu, P., & Yu, Z. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416546 · Cited by 43
“In addition code synthesis [1, 60] also uses matched pairs of natural language and code; however, these datasets have not been used for generating English from code, and are not used in prior work for this task; so we don’t discuss them further here.”
MethodFrom language to action: a review of large language models as autonomous agents and tool users
Chowa, S. S., Alvi, R., Rahman, S. S., Rahman, M. A., Raiaan, M. A. K., Islam, M. R., Hussain, M., & Azam, S. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-025-11471-9 · Cited by 50
An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities
Yang, Z., Chen, S., Gao, C., Li, Z., Hu, X., Liu, K., & Xia, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3717061 · Cited by 37
“We evaluate the RAF for code generation on three well recognized datasets: CONCODE [32], CoNaLa [72] and HearthStone [45].”
InfluentialEnriching API Documentation with Code Samples and Usage Scenarios from Crowd Knowledge
Zhang, J., Jiang, H., Ren, Z., Zhang, T., & Huang, Z. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2919304 · Cited by 38
“answer in Stack Overflow [23], since the question title is the”
BackgroundCodeRetriever: A Large Scale Contrastive Pre-Training Method for Code Search
Li, X., Gong, Y., Shen, Y., Qiu, X., Zhang, H., Yao, B., Qi, W., Jiang, D., Chen, W., & Duan, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.187 · Cited by 79
Who evaluates the evaluators? On automatic metrics for assessing AI-based offensive code generators
Liguori, P., Improta, C., Natella, R., Čukić, B., & Cotroneo, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120073 · Cited by 43
“, CoNaLa mined dataset contains thousands of training examples mined directly from StackOverflow (Yin et al., 2018)).”
BackgroundMethodInfluentialCODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training
Wang, X., Wang, Y., Wan, Y., Wang, J., Zhou, P., Li, L., Wu, H., & Liu, J. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.80 · Cited by 52
MarianCG: a code generation transformer model inspired by machine translation
Soliman, A. S., Hadhoud, M., & Shaheen, S. I. (2022) · Journal of Engineering and Applied Science · doi:10.1186/s44147-022-00159-4 · Cited by 32
“This is accomplished by fine-tuning MarianMT which is a pre-trained language model with CoNaLa [14] and DJANGO [15] datasets.”
MethodHow Important Are Good Method Names in Neural Code Generation? A Model Robustness Perspective
Yang, G., Zhou, Y., Yang, W., Yue, T., Chen, X., & Chen, T. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3630010 · Cited by 45
“Fine-tuning code generation tasks are typi-140 cally applied to benchmarks that lack test cases, such as CONCODE [42] and CoNaLa [95].”
Survey of Code Search Based on Deep Learning
Xie, Y., Lin, J., Dong, H., Zhang, L., & Wu, Z. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3628161 · Cited by 56
CROKAGE: effective solution recommendation for programming tasks by leveraging crowd knowledge
da Silva, R. F. G., Roy, C. K., Rahman, M. M., Schneider, K. A., Paixão, K. V. R., Dantas, C. E., & Maia, M. D. A. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09863-2 · Cited by 27
“We also use a manually curated dataset provided by a previous work (Yin et al. 2018) containing questions (i.”
MethodInfluentialCan we generate shellcodes via natural language? An empirical study
Liguori, P., Al-Hossami, E., Cotroneo, D., Natella, R., Čukić, B., & Shaikh, S. (2022) · Automated Software Engineering · doi:10.1007/s10515-022-00331-3 · Cited by 35
“[84] incorporated pre-training and fine-tuning of a model to generate Python snippets from natural language using the CoNaLa dataset [87].”
MethodInfluentialEspejel, J. L., Alassan, M. S. Y., Chouham, E. M., Dahhane, W., & Ettifouri, E. H. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100013 · Cited by 28
“CoNaLa (Yin et al., 2018) consists of query-code pairs collected from the website Stack Overflow via a mining method.”
MethodOn Extracting Specialized Code Abilities from Large Language Models: A Feasibility Study
Li, Z., Wang, C., Ma, P., Liu, C., Wang, S., Wu, D., Gao, C., & Liu, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639091 · Cited by 67
Modeling Vocabulary for Big Code Machine LearningPreprint
Babii, H., Janes, A., & Robbes, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01873 · Cited by 24
“[74] mine pairs of natural language and code from Stack Overflow to support tasks like code synthesis from natural language.”
BackgroundTAG : Type Auxiliary Guiding for Code Comment Generation
Cai, R., Liang, Z., Xu, B., Li, Z., Hao, Y., & Chen, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.27 · Cited by 24
Compositional API Recommendation for Library-Oriented Code Generation
Ma, Z., An, S., Xie, B., & Lin, Z. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644403 · Cited by 42
Stephany, F. (2021) · Big Data & Society · doi:10.1177/20539517211003120 · Cited by 22
“Instead research shows that independent professionals, including women, prefer informal, digital, social learning resources like Stack Overflow and tutorial videos to develop new skills (Yin et al., 2018).”
Jointly Learning Semantic Parser and Natural Language Generator via Dual Information Maximization
Ye, H., Li, W., & Wang, L. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1201 · Cited by 24
“For DJANGO and CONALA, the NL utterances are lowercased and tokenized and the tokens in code snippets are separated with space.”
InfluentialCoDesc: A Large Code–Description Parallel Dataset
Hasan, M., Muttaqueen, T., Ishtiaq, A. A., Mehrab, K. S., Haque, M. M. A., Hasan, T., Ahmad, W. U., Iqbal, A., & Shahriyar, R. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.18 · Cited by 51
Execution-Based Evaluation for Open-Domain Code Generation
Wang, Z., Zhou, S., Fried, D., & Neubig, G. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.89 · Cited by 134
Self-citationDeep Code-Comment Understanding and Assessment
Wang, D., Guo, Y., Dong, W., Wang, Z., Liu, H., & Li, S. (2019) · IEEE Access · doi:10.1109/access.2019.2957424 · Cited by 22
“[18] proposed a method for extracting aligned code/natural language pairs from the Q&A website, Stack Overflow.”
MethodExecution-Based Evaluation for Open-Domain Code Generation
Wang, Z., Zhou, S., Fried, D., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2212.10481 · Cited by 20
“Some datasets preserve natural coverage from interactive Jupyter Notebooks (Agashe et al., 2019) or Stack-Overflow posts (Yin et al., 2018; Wang et al., 2022), but face challenges in enabling execution (Lai et al., 2022; Chandel et al., 2022).”
BackgroundMethodInfluentialSelf-citationL2CEval : Evaluating Language-to-Code Generation Capabilities of Large Language Models
Ni, A., Yin, P., Zhao, Y., Riddell, M., Feng, T., Shen, R., Yin, S., Ye, L., Yavuz, S., Xiong, C., et al. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00705 · Cited by 76
“Several code generation benchmarks are collected from raw data from GitHub and StackOverflow, and involve professional annotators to enhance the quality of the data (Iyer et al., 2018; Agashe et al., 2019; Yin et al., 2018).”
Self-citationLearning Based Methods for Code Runtime Complexity Prediction
Sikka, J., Satya, K., Kumar, Y., Uppal, S., Shah, R. R., & Zimmermann, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45439-5_21 · Cited by 19
“The Public Git Archive is a dataset of a large collection of Github repositories [12,16] and [15] are datasets of Question-code pairs mined from Stack Overflow.”
MethodCodeFusion: A Pre-trained Diffusion Model for Code Generation
Singh, M. P., Cambronero, J., Gulwani, S., Le, V., Negreanu, C., & Verbruggen, G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.716 · Cited by 79
A survey on robustness attacks for deep code models
Qu, Y., Huang, S., & Yao, Y. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00464-7 · Cited by 32
Relevance Transformer: Generating Concise Code Snippets with Relevance Feedback
Gemmell, C., Rossetto, F., & Dalton, J. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401215 · Cited by 21
“Results on standard benchmark collections show consistent gains over both retrieval and generation baselines, including significant gains on the realistic CoNaLa dataset [17] based on Stack Overflow questions.”
BackgroundNatural language-guided programming
Heyman, G., Huysegems, R., Justen, P., & Cutsem, T. V. (2021) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3486607.3486749 · Cited by 24
“[49] create a curated dataset called CoNaLa [9] from Stack Overflow posts.”
BackgroundAre Large Language Models Reliable Argument Quality Annotators?
Mirzakhmedova, N., Gohsen, M., Chang, C. H., & Stein, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63536-6_8 · Cited by 64
Associating Natural Language Comment and Source Code Entities
Panthaplackel, S., Gligoric, M., Mooney, R. J., & Li, J. J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6382 · Cited by 14
Code Generation from Natural Language with Less Prior Knowledge and More Monolingual Data
Norouzi, S., Tang, K., & Cao, Y. (2021) · doi:10.18653/v1/2021.acl-short.98 · Cited by 31
“Table 2-3 showcase our SOTA results on Django and CoNaLa.”
MethodInfluentialERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages
Chai, Y., Wang, S., Pang, C., Sun, Y., Tian, H., & Wu, H. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.676 · Cited by 18
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
“COMMITPACKFT Prior work has shown the importance of careful data filtering to maintain quality (Yin et al., 2018; Dhole et al., 2021; Laurençon et al., 2022; Longpre et al., 2023b).”
BackgroundExploring Dynamic Selection of Branch Expansion Orders for Code Generation
Jiang, H., Zhou, C., Meng, F., Zhang, B., Zhou, J., Huang, D., Wu, Q., & Su, J. (2021) · doi:10.18653/v1/2021.acl-long.394 · Cited by 23
“For CONALA, we use the corpus-level BLEU (Yin et al., 2018) as a complementary metric.”
Big Code Search: A Bibliography
Kim, K., Ghatpande, S., Kim, D., Zhou, X., Liu, K., Bissyandé, T. F., Klein, J., & Traon, Y. L. (2023) · ACM Computing Surveys · doi:10.1145/3604905 · Cited by 21
Deep learning for code generation: a survey
Zhang, H., Zhang, K., Li, Z., Li, J., Li, J., Li, Y., Zhao, Y., Zhu, Y. S., Liu, F., Li, G., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-023-3956-3 · Cited by 27
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2208.08227 · Cited by 17
Optimal Neural Program Synthesis from Multimodal Specifications
Ye, X., Chen, Q., Dillig, I., & Durrett, G. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.146 · Cited by 24
Domain-Adaptive Text Classification with Structured Knowledge from Unlabeled Data
Li, T., Chen, X., Dong, Z., Keutzer, K., & Zhang, S. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/585 · Cited by 19
Code to Comment "Translation": Data, Metrics, Baselining & EvaluationPreprint
Gros, D., Sezhiyan, H., Dévanbu, P., & Yu, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01410 · Cited by 16
“In addition code synthesis [1, 60] also uses matched pairs of natural language and code; however, these datasets have not been used for generating English from code, and are not used in prior work for this task; so we don’t discuss them further here.”
MethodMulCode: A Multi-task Learning Approach for Source Code Understanding
Wang, D., Yu, Y., Li, S., Dong, W., Wang, J., & Liao, Q. (2021) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner50967.2021.00014 · Cited by 25
“[40] trained their models on natural language texts and corresponding code from Stack Overflow.”
MethodAPIzation: Generating Reusable APIs from StackOverflow Code Snippets
Terragni, V., & Salza, P. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678576 · Cited by 23
“For completeness, APIZATOR generates a method name for the API from the title of the SO page associated with the code snippet [24].”
BackgroundMethodNeural Code Search Revisited: Enhancing Code Snippet Retrieval through Natural Language IntentPreprint
Heyman, G., & Cutsem, T. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.12193 · Cited by 40
“1: Illustration of the CoNaLa corpus [18] is converted in a benchmark for annotated code search.”
GrammarT5: Grammar-Integrated Pretrained Encoder-Decoder Neural Model for Code
Zhu, Q., Liang, Q., Sun, Z., Xiong, Y., Zhang, L., & Cheng, S. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639125 · Cited by 41
Shellcode_IA32: A Dataset for Automatic Shellcode Generation
Liguori, P., Al-Hossami, E., Cotroneo, D., Natella, R., Čukić, B., & Shaikh, S. (2021) · doi:10.18653/v1/2021.nlp4prog-1.7 · Cited by 19
Zhao, S., Tian, J., Fu, J., Chen, J., & Wen, J. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3395531 · Cited by 17
“We conduct experiments on several code generation and code translation benchmarks to evaluate the efficacy of the proposed method, including the datasets CoNaLa [62], Shell-code_IA32 [29], and CodeTrans [63].”
JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models
Cao, J., Chen, Z., Wu, J., Cheung, S.-C., & Xu, C. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695470 · Cited by 62
Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey
She, X., Liu, Y., Zhao, Y., He, Y., Li, L., Tantithamthavorn, C. K., Qin, Z., & Wang, H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3748647 · Cited by 40
Neural joint attention code search over structure embeddings for software Q&A sites
Hu, G., Peng, M., Zhang, Y., Xie, Q., & Yuan, M. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110773 · Cited by 19
Optimising the fit of stack overflow code snippets into existing code
Reid, B., Treude, C., & Wagner, M. (2020) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3377929.3398087 · Cited by 18
“There has also been much research into using neural networks to map natural language to code snippets [24][23].”
BackgroundAn AST Structure Enhanced Decoder for Code Generation
Jiang, H., Song, L., Ge, Y., Meng, F., Yao, J., & Su, J. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3138717 · Cited by 19
LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead
He, J., Shi, J., Zhuo, T. Y., Treude, C., Sun, J., Du, X., Xing, Z., & Lo, D. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3797276 · Cited by 12
“The LLM-based approach significantly outperformed traditional reference-based metrics (e.g., BLEU, 266:14 J.”
Vector-Quantization-Based Topic Modeling
Gupta, A., & Zhang, Z. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3450946 · Cited by 12
Phan, H., & Jannesari, A. (2020) · doi:10.1145/3416506.3423576 · Cited by 14
“We use the Conala corpus from [50].”
PlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context
Chen, X., Gong, L., Cheung, A., & Song, D. X. (2021) · doi:10.18653/v1/2021.acl-long.169 · Cited by 23
Metadata
- Authors
- Yin, P., Deng, B., Chen, E., Vasilescu, B., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 15th International Conference on Mining Software Repositories
- Publisher
- ACM
- Published
- 2018-05-28
- Pages
- 476-486
- OpenAlex
- W2964315653
- Semantic Scholar
- 1a53e7446274016f737236bdd48e3ff05d966384
- Cited by
- 218
- References
- 51
- Citations by year
- 2026: 17 · 2025: 20 · 2024: 32 · 2023: 33 · 2022: 28 · 2021: 35 · 2020: 37 · 2019: 14 · 2018: 2
- FWCI
- 17.27
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference on Mining Software Repositories (MSR) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- Stack Overflow, code mining, code search, code summarization, program synthesis, stochastic modeling, handcrafted features, cross-lingual generalization, programming languages
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.