Multilingual training for software engineering
Open accessThe DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
Well-trained machine-learning models, which leverage large amounts of open-source software data, have now become an interesting approach to automating many software engineering tasks. Several SE tasks have all been subject to this approach, with performance gradually improving over the past several years with better models and training methods. More, and more diverse, clean, labeled data is better for training; but constructing good-quality datasets is time-consuming and challenging. Ways of augmenting the volume and diversity of clean, labeled data generally have wide applicability. For some languages (e.g., Ruby) labeled data is less abundant; in others (e.g., JavaScript) the available data maybe more focused on some application domains, and thus less diverse. As a way around such data bottlenecks, we present evidence suggesting that human-written code in different languages (which performs the same function), is rather similar, and particularly preserving of identifier naming patterns; we further present evidence suggesting that identifiers are a very important element of training data for software engineering tasks. We leverage this rather fortuitous phenomenon to find evidence that available multilingual training data (across different languages) can be used to amplify performance. We study this for 3 different tasks: code summarization, code retrieval, and function naming. We note that this data-augmenting approach is broadly compatible with different tasks, languages, and machine-learning models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Ahmed and Devanbu showed that the performance of the deep learning models depends on the identifiers for the code summarization task [2].”
From Few-shot training LLMs for project-specific code-summarization (2022) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Prior work suggests that language models find more value in identifiers, rather than code structure, when generating code summaries [4].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“They also report that identifiers play a critical role in code summarization tasks.”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [4]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F., Wattenberg, M., Corrado, G., et al. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00065 · Cited by 1,747
Cross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586 · cited 1×
“Foundation models [13, 15, 17, 45, 56] have two stages: i) unsupervised pre-training with corpora at vast scale and ii) fine-tuning with a smaller volume of supervised data for the actual task.”
BackgroundNeural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · cited 1×
“Following [6], the function names are broken into subtokens using BPE [59] (we’ve used BPE tokenization for all tasks).”
Methodcode2vec: Learning distributed representations of code
Alon, U., Zilberstein, M., Levy, O., & Yahav, E. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3290353 · Cited by 962
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
Multilingual Translation with Extensible Multilingual Pretraining and FinetuningPreprint
Tang, Y., Tran, C., Li, X., Chen, P., Goyal, N., Chaudhary, V., Gu, J., & Fan, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.00401 · Cited by 715
Summarizing Source Code using a Neural Attention Model
Iyer, S., Konstas, I., Cheung, A., & Zettlemoyer, L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1195 · Cited by 701
Unified Pre-training for Program Understanding and Generation
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.211 · Cited by 642
Hu, X., Li, G., Xia, X., Lo, D., & Jin, Z. (2018) · Proceedings of the 26th Conference on Program Comprehension · doi:10.1145/3196321.3196334 · Cited by 621
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · cited 4×
“The Dataset: There are several different code summarization datasets; we chose CodeXGLUE6 [47], for two main reasons:”
BackgroundMethodInfluentialCodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483 · cited 9×
“The Dataset: Guo et al. adapt the same CodeSearchNet [29] dataset, with some additional data for candidate codes [22].”
MethodInfluentialUsing Machine Learning for Vulnerability Detection and Classification
Alon, U., Brody, S., Levy, O., & Yahav, E. (2018) · arXiv (Cornell University) · doi:10.4230/oasics.slate.2021.14 · Cited by 467
Cliff´s Delta Calculator: A non-parametric effect size program for two groups of observations
Macbeth, G., Razumiejczyk, E., & Ledesma, R. D. (2010) · Universitas Psychologica · doi:10.11144/javeriana.upsy10-2.cdcp · Cited by 442
Improving automatic source code summarization via deep reinforcement learning
Wan, Y., Zhao, Z., Yang, M., Xu, G., Ying, H., Wu, J., & Yu, P. S. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238206 · Cited by 385
A Transformer-based Approach for Source Code Summarization
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.449 · Cited by 378
A Survey on Similarity Measures in Text Mining
M.K, V., & K, K. (2016) · Machine Learning and Applications: An International Journal · doi:10.5121/mlaij.2016.3103 · Cited by 354
Improved Code Summarization via a Graph Neural Network
LeClair, A., Haque, S., Wu, L., & McMillan, C. (2020) · Proceedings of the 28th International Conference on Program Comprehension · doi:10.1145/3387904.3389268 · Cited by 312
(2004) · doi:10.3115/1220355.1220427 · Cited by 294 · cited 2×
“use smooth BLEU-4 [44] for the evaluations of themodels.”
MethodA Neural Model for Generating Natural Language Summaries of Program Subroutines
LeClair, A., Jiang, S., & McMillan, C. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00087 · Cited by 294
A Survey of Multilingual Neural Machine Translation
Dabre, R., Chu, C., & Kunchukuttan, A. (2021) · ACM Computing Surveys · doi:10.1145/3406095 · Cited by 294
The adverse effects of code duplication in machine learning models of code
Allamanis, M. (2019) · Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3359591.3359735 · Cited by 284
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
Arivazhagan, N., Bapna, A., Firat, O., Lepikhin, D., Johnson, M., Krikun, M., Chen, M. X., Cao, Y., Foster, G., Cherry, C., et al. (2019) · arXiv · doi:10.48550/arxiv.1907.05019 · Cited by 279 · cited 1×
“Several papers show that multilingual-trained models show better performance [36, 62] and are more practical to deploy [9].”
BackgroundDeep code comment generation with hybrid lexical and syntactical information
Hu, X., Li, G., Xia, X., Lo, D., & Jin, Z. (2020) · Empirical Software Engineering · doi:10.1007/s10664-019-09730-9 · Cited by 278
Retrieval-based neural source code summarization
Zhang, J., Wang, X., Zhang, H., Sun, H., & Liu, X. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380383 · Cited by 275
Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks
Mastropaolo, A., Scalabrino, S., Cooper, N., Nader Palacio, D., Poshyvanyk, D., Oliveto, R., & Bavota, G. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00041 · Cited by 247
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Ha, T.-L., Niehues, J., & Waibel, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04798 · Cited by 236
SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code RepresentationPreprint
Wang, X. E., Wang, Y., Mi, F., Zhou, P., Wan, Y., Xiao, L., Li, L., Wu, H., Liu, J., & Jiang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04556 · Cited by 199
DéjàVu: a map of code duplicates on GitHub
Lopes, C. V., Maj, P., Martins, P., Saini, V., Yang, D., Zitny, J., Sajnani, H., & Vitek, J. (2017) · Proceedings of the ACM on Programming Languages · doi:10.1145/3133908 · Cited by 183 · cited 2×
“Sadly, there is a large amount of copied code on GitHUB [46]; inattentively combining different datasets harvested from GitHUB can lead to undesirable levels of duplication in the merged dataset.”
BackgroundCoTexT: Multi-task Learning with Code-Text TransformerPreprint
Phan, L., Tran, H., Le, D., Nguyen, H., Annibal, J., Peltekian, A., & Ye, Y. (2021) · NLP4PROG · doi:10.48550/arxiv.2105.08645 · Cited by 165
Retrieval Augmented Code Generation and Summarization
Parvez, M. R., Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.232 · Cited by 141
Sentiment Analysis for Software Engineering: How Far Can Pre-trained Transformer Models Go?
Zhang, T., Xu, B., Thung, F., Haryono, S. A., Lo, D., & Jiang, L. (2020) · 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme46990.2020.00017 · Cited by 129 · cited 1×
“This approach is known to achieve state-of-the-art performance in both natural language processing, and software-related tasks [3, 4, 11, 18, 22, 32, 33, 35, 50, 75].”
BackgroundA Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine TranslationPreprint
Lakew, S. M., Cettolo, M., & Federico, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.06957 · Cited by 114
Effective identifier names for comprehension and memory
Lawrie, D., Morrell, C. H., Feild, H. A., & Binkley, D. (2007) · Innovations in Systems and Software Engineering · doi:10.1007/s11334-007-0031-2 · Cited by 112
Reassessing automatic evaluation metrics for code summarization tasks
Roy, D., Fakhoury, S., & Arnaoudova, V. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468588 · Cited by 106 · cited 9×
“More than 30 papers have been published in the last five years that follow some form of encoder-decoder architecture [58].”
BackgroundMethodInfluentialReinforcement-Learning-Guided Source Code Summarization Using Hierarchical Attention
Wang, W., Zhang, Y., Sui, Y., Wan, Y., Zhao, Z., Wu, J., Yu, P. S., & Xu, G. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2979701 · Cited by 103
Improved Automatic Summarization of Subroutines via Attention to File Context
Haque, S., LeClair, A., Wu, L., & McMillan, C. (2020) · Proceedings of the 17th International Conference on Mining Software Repositories · doi:10.1145/3379597.3387449 · Cited by 102
Multilingual Neural Machine Translation with Knowledge Distillation
Tan, X., Ren, Y. C., He, D., Qin, T., Zhao, Z. L., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10461 · Cited by 99
(2023) · doi:10.1145/3510003.3510060 · Cited by 88
Cross-Language Clone Detection by Learning Over Abstract Syntax Trees
Pérez, D., & Chiba, S. (2019) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2019.00078 · Cited by 87
Nakov, P., & Ng, H. T. (2009) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1699648.1699682 · Cited by 72
Retrieve and refine: exemplar-based neural comment generation
Wei, B., Li, Y., Li, G., Xia, X., & Jin, Z. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416578 · Cited by 69
Learning type annotation: is big data enough?
Jesse, K., Dévanbu, P., & Ahmed, T. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473135 · Cited by 58
Self-citationAchieving Reliable Sentiment Analysis in the Software Engineering Domain using BERT
Biswas, E., Karabulut, M. E., Pollock, L., & Vijay-Shanker, K. (2020) · 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme46990.2020.00025 · Cited by 55
Code Generation as a Dual Task of Code Summarization
Wei, B., Li, G., Xia, X., Fu, Z., & Jin, Z. G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05923 · Cited by 53
SeCNN: A semantic CNN parser for code comment generation
Li, Z., Wu, Y., Peng, B., Chen, X., Sun, Z., Liu, Y., & Yu, D. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111036 · Cited by 49
DeepCommenter: a deep code comment generation tool with hybrid lexical and syntactical information
Li, B., Yan, M., Xia, X., Hu, X., Li, G., & Lo, D. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417926 · Cited by 35
Qi, W., Gong, Y., Yan, Y., Xu, C., Yao, B., Zhou, B., Cheng, B., Jiang, D., Chen, J., Zhang, R., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08006 · Cited by 33
CoCoSum: Contextual Code Summarization with Multi-Relational Graph Neural NetworkPreprint
Wang, Y.-L., Shi, E.-S., Du, L., Yang, X.-D., Hu, Y., Han, S., Zhang, H.-Y., & Zhang, D.-M. (2021) · arXiv.org · doi:10.48550/arxiv.2107.01933 · Cited by 33
Learning to Find Usages of Library Functions in Optimized Binaries
Ahmed, T., Dévanbu, P., & Sawant, A. A. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3106572 · Cited by 22
Self-citationSYNFIX: Automatically Fixing Syntax Errors using Compiler DiagnosticsPreprint
Ahmed, T., Ledesma, N. R., & Dévanbu, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14671 · Cited by 19
Self-citationNeural Machine Translation for Low-Resource Languages: A SurveyPreprint
Ranathunga, S., Lee, E.-S. A., Skënduli, M. P., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15115 · Cited by 8
Yang, G., Chen, X., Cao, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03644 · Cited by 3
Code Structure Guided Transformer for Source Code SummarizationPreprint
Gao, S., Gao, C., He, Y., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.09340 · Cited by 2
Ensemble Models for Neural Source Code Summarization of SubroutinesPreprint
LeClair, A., Bansal, A., & McMillan, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.11423
Yet Another Combination of IR- and Neural-based Comment GenerationPreprint
Huang, Y., Wei, M., Song, W., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.12938
Code to Comment Translation: A Comparative Study on Model Effectiveness & ErrorsPreprint
Mahmud, J., Faisal, F., Arnob, R. I., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08415
Research works printed but not linked: 16
- 6. Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional at- tention network for extreme summarization of source code. In International conference on machine learning. PMLR, 2091-2100.
- 7. Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019. code2seq: Generating Sequences from Structured Representations of Code. In International Conference on Learning Representations. https://openreview.net/forum?id=H1gKYo09tX
- 10. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65-72.
- 14. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020). · arXiv
- 20. Ekaterina Garmash and Christof Monz. 2016. Ensemble learning for multi-source neural machine translation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. 1409-1418.
- 21. David Gros, Hariharan Sezhiyan, Prem Devanbu, and Zhou Yu. 2020. Code to Comment ?Translation?: Data, Metrics, Baselining & Evaluation. In 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 746-757.
- 22. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, LIU Shujie, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. GraphCodeBERT: Pre-training Code Representations with Data Flow. In International Conference on Learning Representations.
- 25. Masum Hasan, Tanveer Muttaqueen, Abdullah Al Ishtiaq, Kazi Sajeed Mehrab, Md Haque, Mahim Anjum, Tahmid Hasan, Wasi Uddin Ahmad, Anindya Iqbal, and Rifat Shahriyar. 2021. CoDesc: A Large Code-Description Parallel Dataset. arXiv preprint arXiv:2105.14220 (2021). · arXiv
- 31. Paul Jaccard. 1901. Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bull Soc Vaudoise Sci Nat 37 (1901), 547-579.
- 33. Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021. CURE: Code-Aware Neural Machine Translation for Automatic Program Repair. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1161-1173.
- 35. Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning. PMLR, 5110-5121.
- 43. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74-81.
- 60. Ensheng Shi, Yanlin Wang, Lun Du, Junjie Chen, Shi Han, Hongyu Zhang, Dong- mei Zhang, and Hongbin Sun. 2021. Neural Code Summarization: How Far Are We? arXiv preprint arXiv:2107.07112 (2021). · arXiv
- 61. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. 3104- 3112.
- 64. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998-6008.
- 76. Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. arXiv preprint arXiv:1601.00710 (2016). · arXiv
Cited by
Few-shot training LLMs for project-specific code-summarization
Ahmed, T., & Devanbu, P. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3559555 · Cited by 240
Self-citationRecommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models
Ahmed, T., Ghosh, S., Bansal, C., Zimmermann, T., Zhang, X., & Rajmohan, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00149 · Cited by 148
Self-citationMultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3267446 · Cited by 394
“Code generation models have been applied to a variety of tasks, including test generation [18], docstring generation [19], code search [16], [20], type inference [21], [22], [23], and more [24].”
BackgroundMethodAutomatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639183 · Cited by 112
“These models use different representations and sources of information to perform well in code summarization.”
MethodSelf-citationCIRCLE: continual repair across programming languages
Yuan, W., Zhang, Q., He, T., Fang, C., Hung, N. Q. V., Hao, X., & Yin, H. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534219 · Cited by 62
Metadata
- Authors
- Ahmed, T., & Devanbu, P.
- Affiliations
- University of California System (US)
- University of California
- ORCID
- Ahmed, Toufique
- Venue
- Proceedings of the 44th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 1443-1455
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4225716708
- Cited by
- 57
- References
- 34
- Citations by year
- 2026: 7 · 2025: 15 · 2024: 13 · 2023: 19 · 2022: 3
- FWCI
- 11.18
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- NSF (National Science Foundation) (2107592 and 1414172)
- UC Davis (Dean?s Distinguished Graduate Fellowship (DDGF))
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- multilingual training data, data augmentation, software engineering tasks, code summarization, code search, open source software
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.