Learning to mine aligned code and natural language pairs from stack overflow
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
For tasks like code synthesis from natural language, code retrieval, and code summarization, data-driven models have shown great promise. However, creating these models require parallel data between natural language (NL) and code with fine-grained alignments. Stack Overflow (SO) is a promising source to create such a data set: the questions are diverse and most of them have corresponding answers with high quality code snippets. However, existing heuristic methods (e.g., pairing the title of a post with the code in the accepted answer) are limited both in their coverage and the correctness of the NL-code pairs obtained. In this paper, we propose a novel method to mine high-quality aligned data from SO using two sets of features: hand-crafted features considering the structure of the extracted snippets, and correspondence features obtained by training a probabilistic model to capture the correlation between NL and code using neural networks. These features are fed into a classifier that determines the quality of mined NL-code pairs. Experiments using Python and Java as test beds show that the proposed method greatly expands coverage and accuracy over existing mining methods, even when using only a small number of labeled examples. Further, we find that reasonable results are achieved even when training the classifier on one language and testing on another, showing promise for scaling NL-code mining to a wide variety of programming languages beyond those for which we are able to annotate data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This model reports state-of-the-art accuracy on the CoNaLa benchmark dataset [122]”
From In-IDE Code Generation from Natural Language: Promise and Challenges (2022) · cites this paper as [122]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“vent overfitting, on the input of the last softmax layer over target words (p = 0.5), and recurrent dropout [11] on RNNs (p = 0.2). We train the network using the widely used optimization method Adam [18]. To evaluate the neural network, we use the remaining 10% of pairs left aside for testing, retaining the model wit…”
MethodInfluential(2014) · Cited by 33,419
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“So how do neural MT models calculate this probability? We will explain a basic outline of a basic model called the encoder-decoder model [39], and refer readers to references for details [4, 25, 39].”
BackgroundEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 1×
“allows the model to “focus” on particular words in the source snippet S when generating the intent I. The details of attention are beyond the scope of this paper, but interested readers can reference [4, 22]. Why attentional neural MT models?: Attention-based neural MT models are well-suited to the task of learning co…”
BackgroundLearning deep structured semantic models for web search using clickthrough data
Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., & Heck, L. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2505665 · Cited by 2,009 · cited 1×
“Second, they are powerful models that can learn correspondences on a variety of levels; from simple phenomena such as direct word-by-word matches [12], to soft paraphrases [36], to weak correspondences between keywords and large documents for information retrieval [14].”
BackgroundInternational Conference on Learning Representations (ICLR)
Abbas, M. Z. A., Kivinen, J., & Raiko, T. (2016) · Cited by 1,694
Incorporating Copying Mechanism in Sequence-to-Sequence Learning
Gu, J., Lu, Z., Li, H., & Li, V. O. K. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1154 · Cited by 1,446 · cited 1×
“classifier, as described in the following paragraph. Second, they are powerful models that can learn correspondences on a variety of levels; from simple phenomena such as direct word-by-word matches [12], to soft paraphrases [36], to weak correspondences between keywords and large documents for information retrieval […”
BackgroundStatistical machine translation
(2010) · Choice Reviews Online · doi:10.5860/choice.47-6293 · Cited by 1,309
Recurrent Continuous Translation Models
Kalchbrenner, N., & Blunsom, P. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1176 · Cited by 1,232 · cited 1×
“between natural language and code for the purposes of code summarization [15, 30], code synthesis from natural language [21], and code retrieval [3]. In particular, we use a model based on neural MT [4, 17], a method for MT based on neural networks that is well-suited for this task of learning correspondences for a va…”
MethodGet another label? improving data quality and data mining using multiple, noisy labelers
Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401965 · Cited by 1,165 · cited 1×
“However, the statistical models that power many such applications are only as useful as the data they are trained on, i.e. , garbage in, garbage out [35].”
Background
Research works printed but not linked: 7
- 10.5555/2819009.2819143 · doi:10.5555/2819009.2819143
- Gal Yarin, 2016, Annual Conference on Neural Information Processing Systems (NIPS). 1019--1027
- Diederik, 2014, Kingma and Jimmy Ba
- Lin Xi Victoria, 2018, International Conference on Language Resources and Evaluation (LREC)
- Luong Thang, Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 1412--1421
- 10.5555/2627435.2670313 · doi:10.5555/2627435.2670313
- 10.5555/2666719.2666728 · doi:10.5555/2666719.2666728
Cited by
Items 101–200 of 230
Synergy between Machine/Deep Learning and Software Engineering: How Far Are We?Preprint
Wang, S., Huang, L., Ge, J., Zhang, T., Feng, H., Li, M., Zhang, H., & Ng, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05515 · Cited by 9
“To facilitate maintenance and evolution tasks, researchers have mined various repositories using machine learning algorithms such as Support Vector Machine (SVM), Naive Bayes and Logistic Regression to extract high-quality aligned data [12], identify API issue-related sentences [13], and classify user reviews from app…”
MethodA search-and-fill strategy to code generation for complex software requirements
Dong, Y., Kong, L., Zhang, L., Wang, S., Liu, X., Liu, S., & Chen, M.-C. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107584 · Cited by 11
Optimising the fit of stack overflow code snippets into existing code
Reid, B., Treude, C., & Wagner, M. (2020) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 9
Analysis of Tree-Structured Architectures for Code Generation
Dahal, S., Maharana, A., & Bansal, M. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.384 · Cited by 12
“We adopt the ASDL grammar for Python and transition system (Yin and Neubig, 2018) that decomposes the production of an AST into a sequence of actions.”
BackgroundMethodInfluentialGenerating Adversarial Source Programs Using Important Tokens-based Structural Transformations
Chen, P., Li, Z., Wen, Y. J., & Liu, L. (2022) · IEEE International Conference on Engineering of Complex Computer Systems (ICECCS) · doi:10.1109/iceccs54210.2022.00029 · Cited by 17
Code4ML: a large-scale dataset of annotated Machine Learning code
Drozdova, A., Трофимова, Е. В., Guseva, P., Scherbakova, A., & Ustyuzhanin, A. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1230 · Cited by 14
Pârţachi, P.-P., Dash, S. K., Treude, C., & Barr, E. T. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380440 · Cited by 9
“[33] start from a parallel corpus of Natural Language and Snippet pairs and seek to align it.”
BackgroundTowards Full-line Code Completion with Neural Language ModelsPreprint
Wang, W., Shen, S., Li, G., & Jin, Z. G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.08603 · Cited by 24
“This is important for NL-based code generation tasks like Django (Oda et al. 2015) or CoNaLa (Yin et al. 2018) since in these tasks, the source code corpus is small-scaled and each code sample is short.”
Generating Code with the Help of Retrieved Template Functions and Stack Overflow Answers
Drain, D., Hu, C., Wu, C., Breslav, M., & Sundaresan, N. (2021) · arXiv · doi:10.48550/arxiv.2104.05310
“Among these works, both CODENN (Iyer et al., 2016) and StaQC (Yao et al., 2018) match the question title to the entire code block, while CoNaLa (Yin et al., 2018) matches the question title more precisely to the key lines in the code block.”
BackgroundMethodHumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent
Wu, J. J., & Fard, F. H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715109 · Cited by 53
“To the best of our knowledge, all of the existing benchmarks (e.g., HumanEval [13], CoNaLa [86], Galeras [56], APPS [28], and recent SWE-bench [33]) in code generation are tasked with letting the model generate the code directly as prediction, without giving the model the opportunity to ask for…”
Evaluating the Effectiveness and Efficiency of Demonstration Retrievers in RAG for Coding Tasks
He, P., Wang, S., Chowdhury, S., & Chen, T. H. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00053 · Cited by 19
“We employed the CoNaLa dataset [28], which contains 2,889 NL-code pairs from Stack Overflow (Python).”
Lyra: A Benchmark for Turducken-Style Code Generation
Liang, Q., Sun, Z., Zhu, Q., Zhang, W., Yu, L., Xiong, Y., & Zhang, L. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/588 · Cited by 18
Code Generation From Flowcharts with Texts: A Benchmark Dataset and An Approach
Liu, Z., Hu, X., Zhou, D., Li, L., Zhang, X., & Xiang, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.449 · Cited by 21
“…language directly to executable programs in the form of logical forms (Zelle and Mooney, 1996), database queries (Yu et al., 2018, 2019), general purpose code snippets (Oda et al., 2015; Yin et al., 2018), complete executable code that can solve a specific task (Liu et al., 2020; Li et al., 2022).”
BackgroundJuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code GenerationPreprint
Agashe, R., Iyer, S. V., & Zettlemoyer, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02216 · Cited by 28
Semantic code search using Code2Vec: A bag-of-paths model
Arumugam, L. (2020) · UWSpace (University of Waterloo) · Cited by 11
Developers’ information seeking in Question & Answer websites through a gender lens
Sedhain, A., Diwanji, V., Solomon, H., Leon, S., & Kuttal, S. K. (2024) · Journal of Computer Languages · doi:10.1016/j.cola.2024.101267 · Cited by 10
Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
Zhao, S., Shao, Y., Huang, Y., Song, J., Wan, C., Wan, C., & Ма, Л. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3802824 · Cited by 49
“Dataset CoNaLa [85] DS1000 [35] PNE [88] NQ [34] TriviaQA [31] HotpotQA [84] Task”
Fine-Tuning Large Language Models for Answering Programming Questions with Code Snippets
Lomshakov, V., Kovalchuk, S. V., Omelchenko, M. O., Nikolenko, S., & Aliev, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36021-3_15 · Cited by 9
“In this work, we use several publicly available GPT-based LLMs pretrained on code as backbones for solving closed-book QA in the Stack Overflow domain, evaluating on the CoNaLa dataset [26] and our own QA dataset collected from Stack Overflow .”
Towards Better Multilingual Code Search through Cross-Lingual Contrastive Learning
Huang, X., Ma, Y., Zhou, H., Jiang, Z., Zhang, Y., Wang, T., & Li, S. (2023) · Asia-Pacific Symposium on Internetware · doi:10.1145/3609437.3609439 · Cited by 12
“[21] combined the attention mechanism with an LSTM network model for code summarization, and Yin et al.[43] trained their models on natural language texts and corresponding code from Stack Overflow.”
BackgroundMethodAutomatic bi-modal question title generation for Stack Overflow with prompt learning
Yang, S., Chen, X., Liu, K., Yang, G., & Yu, C. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10466-4 · Cited by 11
“Based on the hybrid prompts (i.e., a mixture of the hard prompts and the soft prompts), our model is prompt-tuned based on a pre-trained Transformer model CodeT5 (Wang et al 2021).”
InfluentialMining software architecture knowledge: Classifying stack overflow posts using machine learning
Ali, M., Mushtaq, H., Rasheed, M. B., Baqir, A., & Alquthami, T. (2021) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.6277 · Cited by 7
Reading StackOverflow Encourages Cheating: Adding Question Text Improves Extractive Code Generation
Orlanski, G., & Gittens, A. (2021) · NLP4PROG · doi:10.18653/v1/2021.nlp4prog-1.8 · Cited by 16
Learning to Represent Programs with Heterogeneous GraphsPreprint
Zhang, K., Wang, W., Zhang, H., Li, G., & Jin, Z. G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.04188 · Cited by 7
“For this task, we use the CoNaLa (Yin et al., 2018) dataset, which contains 2,879 Python code-NL query pairs mined from StackOverflow.”
MethodPrompting with Pseudo-Code Instructions
Mishra, M., Kumar, P., Bhat, R. A., Murthy, R., Contractor, D., & Tamilselvam, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.939 · Cited by 25
Enhancing Project-Specific Code Completion by Inferring Internal API Information
Deng, L., Ren, X., Ni, C., Liang, M., Lo, D., & Liu, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3592823 · Cited by 15
“MostcodeLLMs are designed for independent code completion [10], [11], which refers to independently generating or predicting the next pieces of code based on a given code snippet or functional description, without referencing other information.”
MCodeSearcher: Multi-View Contrastive Learning for Code Search
Li, J., Liu, F., Li, J., Zhao, Y., Li, G., & Jin, Z. G. (2023) · Asia-Pacific Symposium on Internetware · doi:10.1145/3609437.3609456 · Cited by 8
“To evaluate the effectiveness of MCodeSearcher, we conduct extensive experiments on five representative datasets [14, 15, 28, 43, 44].”
MethodOn the effectiveness of large language models for query expansion in code search
Liu, X., Liu, J., Kang, G., Shi, M. Y., Liu, Y., & Yin, Y. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112582 · Cited by 6
XSemPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representations
Zhang, Y., Wang, J., Wang, Z., & Zhang, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.887 · Cited by 17
SE Factual Knowledge in Frozen Giant Code Model: A Study on FQN and Its Retrieval
Huang, Q. Y., Liao, D., Xing, Z., Yuan, Z., Lu, Q., Xu, X., & Lu, J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3436883 · Cited by 15
“Reusing online code snippets is a common practice [28], [29], [30].”
Cleaning StackOverflow for Machine Translation
Rahman, M., Rigby, P. C., Palani, D., & Nguyen, T. N. (2019) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2019.00021 · Cited by 7
“In recent years, MT techniques have been used to translate from natural language to programming languages[10, 22, 23, 24, 28, 40].”
MethodText2PyCode: Machine Translation of Natural Language Intent to Python Source Code
Bonthu, S., Sree, S. R., & Prasad, M. H. M. K. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-84060-0_4 · Cited by 6
Learning to Describe Solutions for Bug Reports Based on Developer Discussions
Panthaplackel, S., Li, J. J., Gligoric, M., & Mooney, R. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.231 · Cited by 8
Human perceiving behavior modeling in evaluation of code generation models
Kovalchuk, S. V., Lomshakov, V., & Aliev, A. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.24 · Cited by 12
Interactive Task and Concept Learning from Natural Language Instructions and GUI DemonstrationsPreprint
Li, T. J.-J., Radensky, M., Jia, J., Singarajah, K., Mitchell, T. M., & Myers, B. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00031 · Cited by 7
“This approach also provides more interpretable results and requires less training data than neural network approaches (e.g., [51, 52]).”
BackgroundYang, C., Liu, Y., & Yin, C. (2024) · Computer Science Review · doi:10.1016/j.cosrev.2024.100637 · Cited by 5
Model Editing for LLMs4Code: How Far are we?
Li, X., Wang, S., Li, S., Ma, J., Yu, J., Liu, X., Wang, J., Ji, B., & Zhang, W. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00049 · Cited by 23
“We build the CNLE and CSNE datasets based on the existing CoNaLa [37] and Code-SearchNet [38] datasets, respectively.”
Koohestani, R., de Bekker, P., Koç, B., & Izadi, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3644183 · Cited by 16
“CodeTrans [57] C#, Java 11,800 TransCoder-ST [162] C++, Java, Python 437,030 CoST [163] 7 programming languages 16,738 AVATAR [164] Java, Python 7,133 / 476 / 1,906 Multilingual-Trans [165] 8 programming languages 30,419 total NicheTrans [165] Various niche languages 236,468 total LLMTrans [165] 8…”
SE-Jury: An LLM-as-Ensemble-Judge Metric for Narrowing the Gap with Human Evaluation in SE
Zhou, X., Kim, K., Zhang, T., Weyssow, M., Gomes, L. F., Yang, G., Liu, K., Xia, X., & Lo, D. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00214 · Cited by 29
Contextualized Data-Wrangling Code Generation in Computational Notebooks
Huang, J., Guo, D., Wang, C., Gu, J., Lu, S., Inala, J. P., Yan, C., Gao, J., Duan, N., & Lyu, M. R.-T. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695503 · Cited by 19
“As data wrangling tasks consume a significant amount of time for data scientists [26], our focus is on automating data wrangling through code generation.”
BackgroundHierarchical Embedding for Code Search in Software Q&A Sites
Li, R., Hu, G., & Peng, M. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207101 · Cited by 11
“In addition, [6], [22] shared their corpora, but did not share the corpora processing method.”
BackgroundIs neural machine translation approach accurate enough for coding assistance?
Akinobu, Y., Obara, M., Kajiura, T., Takano, S., Tamura, M., Tomioka, M., & Kuramitsu, K. (2021) · doi:10.1145/3486949.3486966 · Cited by 6
“and extracting bilingual text from comments or docstrings in source code[2, 18].”
BackgroundNMT-Based Code Generation for Coding Assistance with Natural Language
Akinobu, Y., Kajiura, T., Obara, M., & Kuramitsu, K. (2022) · Journal of Information Processing · doi:10.2197/ipsjjip.30.443 · Cited by 5
AstBERT: Enabling Language Model for Financial Code Understanding with Abstract Syntax Trees
Liang, R., Zhang, T., Lu, Y., Liu, Y., Huang, Z., & Chen, X. (2022) · Workshop on Financial Technology and Natural Language Processing (FinNLP) · doi:10.18653/v1/2022.finnlp-1.2 · Cited by 8
Improving Quality of a Post’s Set of Answers in Stack Overflow
Tavakoli, M., Izadi, M., & Heydarnoori, A. (2020) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa51224.2020.00084 · Cited by 8
“Some researches try to find high-quality questions, answers and code snippets posted on SO to help users find high-quality elements efficiently [12], [13], [14].”
Can NMT understand me? towards perturbation-based evaluation of NMT models for code generation
Liguori, P., Improta, C., Vivo, S. D., Natella, R., Čukić, B., & Cotroneo, D. (2022) · IEEE/ACM International Workshop on Natural Language-Based Software Engineering (NLBSE) · doi:10.1145/3528588.3528653 · Cited by 7
“The Django dataset [36], which is widely used for evaluating neural machine translation task from English to Python [16, 50, 52], contains numerous Python code snippets that are relatively short (e.”
MethodInfluentialExploring Dynamic Selection of Branch Expansion Orders for Code GenerationPreprint
Jiang, H., Zhou, C., Meng, F., Zhang, B., Zhou, J., Huang, D., Wu, Q., & Su, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.00261 · Cited by 5
“For CONALA, we use the corpus-level BLEU (Yin et al., 2018) as a complementary metric.”
Automatic Acquisition of Annotated Training Corpora for Test-Code Generation
Kacmajor, M., & Kelleher, J. D. (2019) · Information · doi:10.3390/info10020066 · Cited by 4
“Another complex method to mine high-quality aligned data from Stack Overflow was described in [17].”
MethodFinding Reusable Machine Learning Components to Build Programming Language Processing Pipelines
Flynn, P. J., Vanderbruggen, T., Liao, C., Lin, P., Emani, M. K., & Shen, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36889-9_27 · Cited by 6
Jiang, S., Shen, J., Wu, S., Cai, Y., Yu, Y., & Zhou, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3214859 · Cited by 4
“Existing studies have shown that NCG is an effective, promising mechanism [6], [7], [8], [9], [27], [28], [29], [30].”
BackgroundJEMMA: An extensible Java dataset for ML4Code applications
Karmakar, A., Allamanis, M., & Robbes, R. (2023) · Empirical Software Engineering · doi:10.1007/s10664-022-10275-7 · Cited by 4
“2016), CoNaLa (Yin et al. 2018), and StaQC (Yao et al.”
BackgroundMethodPython Code Generation by Asking Clarification Questions
Li, H.-S., Mesgar, M., Martins, A. F. T., & Gurevych, I. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.799 · Cited by 35
“Text-to-code generation was first defined through learning on the parallel corpus of NLD-Code pairs (Allamanis et al., 2015; Miceli Barone and Sennrich, 2017; Yin et al., 2018).”
BackgroundAlgoLabel: A Large Dataset for Multi-Label Classification of Algorithmic Challenges
Iacob, R. C. A., Monea, V. C., Radulescu, D. C., Ceapă, A.-F., Rebedea, T. E., & Trăușan-Matu, Ș. (2020) · Mathematics · doi:10.3390/math8111995 · Cited by 6
When Does it Pay Off to Learn a New Skill? Revealing the Complementary Benefit of Cross-SkillingPreprint
Stephany, F. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3717077 · Cited by 4
SciCode: A Research Coding Benchmark Curated by Scientists
Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.13168 · Cited by 15
“Before the emergence of very capable code synthesis models, when most models struggled to produce executable code, datasets like CoNaLa typically included n-gram-based metrics [100].”
MethodZhou, X., Weyssow, M., Widyasari, R., Zhang, T., He, J., Lyu, Y., Chang, J., Zhang, B., Huang, D., & Lo, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06215 · Cited by 3
Text to Code Conversion Using Deep Learning for NLP
Mellah, Y., Kaddari, Z., Berrich, J., Bouchentouf, T., & Belkasmi, M. G. (2020) · International Symposium on Advanced Electrical and Communication Technologies (ISAECT) · doi:10.1109/isaect50560.2020.9523647 · Cited by 3
Knowledge-Enhanced Program Repair for Data Science Code
Ouyang, S., Zhang, J. M., Sun, Z., & Meroño-Peñuela, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00246 · Cited by 27
CodeS : Natural Language to Code Repository via Multi-Layer Sketch
Zan, D., Yu, A., Liu, W., Chen, D., Shen, B., Yao, Y., Li, W., Chen, X., Gong, Y., Guan, B., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3768577 · Cited by 51
“CoNaLa [42] targets line completion.”
A Pilot Study in Surveying Data Challenges of Automatic Software Engineering Tasks
Dong, L., Lu, Q., & Zhu, L. (2024) · doi:10.1145/3663530.3665020 · Cited by 4
Dandamudi, R., & Rodríguez-Pérez, G. (2024) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1145/3691621.3694939 · Cited by 4
The Code Generation Method Based on Gated Attention and InterAction-LSTM
Wang, Y., & Wu, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87571-8_47 · Cited by 3
When Does it Pay Off to Learn a New Skill? Revealing the Complementary Benefit of Cross-SkillingPreprint
Stephany, F. (2021) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/sv9de · Cited by 5
BF++: a language for general-purpose program synthesisPreprint
Liventsev, V., Härmä, A., & Petković, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.09571 · Cited by 6
“A neural model can be pre-trained as a language model and fine-tuned on a translation dataset like CoNaLa [14] or, in the case of AlphaCode [15], a dataset of competitive programming tasks and submitted solutions.”
MethodCROKAGE: effective solution recommendation for programming tasks by leveraging crowd knowledge
Silva, R. (2020) · doi:10.14393/ufu.te.2020.454 · Cited by 4
NoviCode: Generating Programs from Natural Language Utterances by Novices
Mordechai, A. A., Goldberg, Y., & Tsarfaty, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00694 · Cited by 12
“We note that previous works (Yin and Neubig, 2017, Yin and Neubig, 2018) also utilized ASTs to synthesize code better.”
BackgroundMethodResultInfluentialDDPT: Diffusion-Driven Prompt Tuning for Large Language Model Code Generation
Li, J., Hyun, S., & Babar, M. A. (2025) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00030 · Cited by 5
“CoNaLa [39] is a dataset crawled from Stack Overflow and is automatically filtered and manually curated by annotators for evaluating the performance of the system in generating Python code snippets based on natural language description.”
Li, J., Deng, H., Zhang, Y., Zhang, K., Shao, T., Zhao, T., Wang, W., Jin, Z. G., Li, G., Liu, Y., et al. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797106 · Cited by 6
Mapping Language to Programs using Multiple Reward Components with Inverse Reinforcement Learning
Ghosh, S., & Srivastava, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.125 · Cited by 4
Code Generation from Natural Language Using Two-Way Pre-Training
Wang, J., Peng, Y., Le, X., Chen, C., & Guan, X. (2023) · International Conference on Advanced Computational Intelligence (ICACI) · doi:10.1109/icaci58115.2023.10146131 · Cited by 3
“The CoNaLa dataset [28] is a joint project of the Carnegie Mellon University NeuLab and Strudel labs.”
MethodGou, Q., Dong, Y., & Ke, Q. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112140 · Cited by 2
Hu, G., Zeng, X., Yu, W., Peng, M., Yuan, M., & Duan, L. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3686664 · Cited by 3
Towards Improving the Performance of Comment Generation Models by Using Bytecode Information
Huang, Y., Huang, J., Chen, X., & Zheng, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3523713 · Cited by 5
A Globally Normalized Neural Model for Semantic Parsing
Huang, C., Yang, W., Cao, Y., Zai͏̈ane, O. R., & Mou, L. (2021) · Workshop on Structured Prediction for NLP (SPNLP) · doi:10.18653/v1/2021.spnlp-1.7 · Cited by 5
An additional approach to pre-trained code model with multilingual natural languages
Kajiura, T., Souma, N., Sato, M., Takahashi, M., & Kuramitsu, K. (2022) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec57359.2022.00090 · Cited by 2
“To build mPyT5 model, we prepare three Python code datasets (CodeSearchNet [3], CoNaLa [4], and Kaggle).”
MethodDong, Y., Li, G., Jiang, X., & Jin, Z. G. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230317 · Cited by 5
“The scaling factor f ( a , t ) − γ decreases as f ( a , t ) increases, which helps models down-weight actions with large f ( a , t ) and thus focus training on actions with small f ( a , t ) , i.e., antecedent actions.”
Big Transformers for Code Generation
Arutyunov, G. A., & Avdoshin, S. M. (2022) · Proceedings of the Institute for System Programming of RAS · doi:10.15514/ispras-2022-34(4)-6 · Cited by 4
CoSQA: 20,000+ Web Queries for Code Search and Question AnsweringPreprint
Huang, J., Tang, D., Shou, L., Gong, M., Xu, K., Jiang, D., Zhou, M., & Duan, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13239 · Cited by 2
“A number of open-sourced datasets with a large amount of text-code pairs have been proposed for the purposes of code search (Husain et al., 2019; Gu et al., 2018; Nie et al., 2016) and code question answering (Yao et al., 2018; Yin et al., 2018; Heyman and Cutsem, 2020).”
BackgroundLLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Review ⋆Preprint
Rasheed, Z., Muhammad, W., Kemell, K., Saari, M., & Abrahamsson, P. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6332149 · Cited by 3
Richen: Automated enrichment of Git documentation with usage examples and scenarios
Shen, C., Yang, W., Jia, H., Pan, M., & Zhou, Y. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2662 · Cited by 2
“A common practice in many existing studies 11,24 is to regard the question title as the intent of the post on Stack Overflow, as the title conveys the topic under discussion in a straightforward manner.”
Instructive Code Retriever: Learn from Large Language Model's Feedback for Code Intelligence Tasks
Lu, J., Wang, H., Liu, Z., Liang, K., Bao, L., & Yang, X. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3694997 · Cited by 6
Prioritizing Bugs using Stack Overflow Posts
Shatnawi, R. (2025) · International Conference on Software and Computer Applications (ICSCA) · doi:10.1145/3731806.3731807 · Cited by 1
“…detector [20], Conceptions of Refactoring [19] identifying posts related to API problems and patterns [1], code synthesis from natural language, code retrieval, and code summarization [28] [4], improving the quality of code snippets [23], and software testing and bug triaging [21] [25].”
CoSQA + : Enhancing Code Search Evaluation With a Multi-Choice Benchmark and Test-Driven Agents
Gong, J., Wu, Y., Liang, L., Wang, Y., Chen, J., Liu, M., & Zheng, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3631886 · Cited by 25
“While the two models achieve high performances in the CSN benchmark, GraphCodeBERT [28] demonstrates superior results by harnessing data flow encoded in the Abstract Syntax Tree (AST) of code to enrich the structural information during pre-training.”
Weng, S., Feng, Y., Yin, Y., Zhang, Z., & Xu, B. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-41376-3 · Cited by 7
Neuron Patching: Semantic-based Neuron-level Language Model Repair for Code Generation
Gu, J., Aleti, A., Chen, C., & Zhang, H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3799426 · Cited by 11
“(1) Line-level Code Generation: We employ CoNaLa [60], which is pairs of rewritten intents (interline comments) and lines of Python code.”
Context-Aware Code Summarization Using Multimodal Transformer
Saif, S., Yaseen, M., Khattak, U. F., & Rahman, G. (2026) · Engineering Technology & Applied Science Research · doi:10.48084/etasr.15857 · Cited by 1
Extracting Code-relevant Description Sentences Based on Structural Similarity
Cao, Y., Zou, Y., & Xie, B. (2019) · Asia-Pacific Symposium on Internetware · doi:10.1145/3361242.3362699 · Cited by 1
“These Figure 1: An answer post from Stack Overflow context texts are very helpful for developers to understand the functions of the code snippets.”
MethodInfluentialExtension-Compression Learning: A deep learning code search method that simulates reading habits
Gu, L., Wang, Z., Liu, J., Zhang, Y., Yang, D., & Dong, W. (2022) · IEEE International Conference on Engineering of Complex Computer Systems (ICECCS) · doi:10.1109/iceccs54210.2022.00032 · Cited by 1
InstructExcel: A Benchmark for Natural Language Instruction in Excel
Payan, J., Mishra, S. R., Singh, M. P., Negreanu, C., Poelitz, C., Baral, C. R., Roy, S., Chakravarthy, R., Durme, B. V., & Nouri, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.265 · Cited by 13
SIGMA: A Dataset for Text-to-Code Semantic Parsing with Statistical Analysis
Almohaimeed, S., Liu, S., Alsofyani, M., Almohaimeed, S., & Wang, L. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00125 · Cited by 1
“In the CoNaLa [20] dataset, the natural language intents of developers are mapped to code snippets collected from Stack Overflow.”
MethodAn Effective Low-Dimensional Software Code Representation using BERT and ELMo
Majumdar, S., Varshney, A., Das, P. P., Clough, P. D., & Chattopadhyay, S. (2022) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs57517.2022.00082 · Cited by 10
“1) CoNaLa [52] : Collection of Python intent - code snippet pairs and id of the Stack Overflow post.”
BackgroundMethodInfluentialIdentifying Algorithm Names in Code CommentsPreprint
Klainongsuang, J., Nugroho, Y. S., Hata, H., Manaskasemsak, B., Rungsawang, A., Leelaprute, P., & Matsumoto, K. (2019) · arXiv (Cornell University) · doi:10.13140/rg.2.2.26285.84966 · Cited by 1
Code Generation from Natural Language with Less Prior and More Monolingual DataPreprint
Norouzi, S., Tang, K., & Cao, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00259 · Cited by 1
“Table 2-3 showcase our SOTA results on Django and CoNaLa.”
MethodInfluentialOptimal Neural Program Synthesis from Multimodal SpecificationsPreprint
Ye, X., Chen, Q., Dillig, I., & Durrett, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01678 · Cited by 11
“However, because both natural language and input examples can be inherently ambiguous (Devlin et al., 2017; Yin et al., 2018), a recent thread of work on multimodal synthesis attempts to combine”
Measuring The Impact Of Programming Language Distribution
Orlanski, G., Xiao, K., Garcia, X., Hui, J., Howland, J., Malmaud, J., Austin, J., Singh, R., & Catasta, M. (2023) · arXiv · doi:10.48550/arxiv.2302.01973 · Cited by 21
“Related Works Code Evaluation Existing code benchmarks have primarily focused on surface matching evaluation (Lu et al., 2021; Yin et al., 2018; Wang et al., 2022b; Husain et al., 2019).”
BackgroundMining API Usage Scenarios from Stack OverflowPreprint
Uddin, G., Khomh, F., & Roy, C. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08874
Exploiting API Description Information to Improve Code Comment Generation
Yang, G., Zhang, Q., Wu, Y., Tianxing, Z., Liu, H., & Feng, W. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-78618-2_32
Text Classification for Task-based Source Code Related QuestionsPreprint
Vijayaraghavan, S., Song, J., Tomassi, D. A., Punj, S., & Sabet, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.00580
“line model established by the authors[17].”
MethodGenerating Algorithmic Patterns from Semi-structured Input Using a Transition-Based Neural Network
Rikk, R., Várkonyi, T. A., Borsi, Z. R., Pintér, B., & Gregorics, T. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-16078-3_57
Metadata
- Authors
- Yin, P., Deng, B., Chen, E., Vasilescu, B., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 15th International Conference on Mining Software Repositories
- Publisher
- ACM
- Published
- 2018-05-28
- Pages
- 476-486
- OpenAlex
- W2964315653
- Semantic Scholar
- 1a53e7446274016f737236bdd48e3ff05d966384
- Cited by
- 218
- References
- 51
- Citations by year
- 2026: 17 · 2025: 20 · 2024: 32 · 2023: 33 · 2022: 28 · 2021: 35 · 2020: 37 · 2019: 14 · 2018: 2
- FWCI
- 17.27
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference on Mining Software Repositories (MSR) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- Stack Overflow, code mining, code search, code summarization, program synthesis, stochastic modeling, handcrafted features, cross-lingual generalization, programming languages
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.