The DOI matches its Crossref record; 321 of the 321 references it cites were found in a registry or an index.
Large Language Models for Software Engineering: A Systematic Literature Review
Abstract
Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a Systematic Literature Review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We selected and analyzed 395 research articles from January 2017 to January 2024 to answer four key Research Questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, pre-processing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and highlighting promising areas for future study. Our artifacts are publicly available at https://github.com/security-pride/LLM4SE_SLR .
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“To identify relevant benchmarks, we leverage a recent and comprehensive survey on LLMs for SE tasks [33], which analyzed 395 research papers from January 2017 to January 2024.”
From LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks (2025) · cites this paper as [33]
In the citing paper
“SWE-Bench is a benchmark designed to assess the ability of LLMs to solve real-world software engineering tasks collected from 12 real-world GitHub repositories.”
From Context as a Tool: Context Management for Long-Horizon SWE-Agents (2025) · cites this paper as Hou et al., 2024
In the citing paper
“The Verified split is a high-quality subset of SWE-Bench consisting of 500 instances, which are manually curated to provide clearer problem descriptions and more reliable evaluation criteria.”
From Context as a Tool: Context Management for Long-Horizon SWE-Agents (2025) · cites this paper as Hou et al., 2024
In the citing paper
“It is increasingly acting as a community contributor-submitting issues, pull requests, code reviews, and security reports [14, 24, 73].”
From To Ban or Not to Ban? How Open Source Projects Govern GenAI Contributions (2026) · cites this paper as [24]
In the citing paper
“However, LLMs struggle with code-related tasks, particularly in terms of correctness, code quality, and other context-specific limitations [11].”
From Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code (2026) · cites this paper as [11]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Items 201–329 of 329
ChatGPT as a Tool for User Story Quality Evaluation: Trustworthy Out of the Box?
Ronanki, K., Cabrero‐Daniel, B., & Berger, C. (2023) · Lecture notes in business information processing · doi:10.1007/978-3-031-48550-3_17 · Cited by 78
One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization
Wang, D., Chen, B., Li, S., Luo, W., Peng, S., Dong, W., & Liao, X. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00013 · Cited by 77
On the Impact of Sample Duplication in Machine-Learning-Based Android Malware Detection
Zhao, Y., Li, L., Wang, H., Cai, H., Bissyandé, T. F., Klein, J., & Grundy, J. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3446905 · Cited by 76
Self-citationSemantic Compression with Large Language Models
Gilbert, H., Sandborn, M., Schmidt, D. C., Spencer-Smith, J. B., & White, J. (2023) · International Conference on Social Networks Analysis, Management and Security (SNAMS) · doi:10.1109/snams60348.2023.10375400 · Cited by 75
Adaptive Test Generation Using a Large Language ModelPreprint
Schäfer, M., Nadi, S., Eghbali, A., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2302.06527 · Cited by 74
Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities
Ma, W., Liu, S., Zhao, M., Xie, X., Wang, W., Hu, Q., Zhang, J., & Liu, Y. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3664606 · Cited by 74
Self-citationA Systematic Review of Interaction in Search-Based Software Engineering
Ramírez, A., Romero, J. R., & Simons, C. L. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2803055 · Cited by 73
An automated approach to estimating code coverage measures via execution logs
Chen, B., Jian, S., Xu, P., Hu, X. Y., & Jiang, Z. M. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238214 · Cited by 73
The 54th Annual Meeting of the Association for Computational Linguistics
Yung, F., Duh, K., Komura, T., et al. (2016) · The Association for Computational Linguistics · Cited by 71
Proceedings of the 38th Annual Computer Security Applications Conference
(2022) · doi:10.1145/3564625 · Cited by 71
CodeEditor : Learning to Edit Source Code with Pre-trained Models
Li, J., Li, G., Li, Z., Jin, Z. G., Hu, X., Zhang, K., & Fu, Z. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3597207 · Cited by 69
ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks
Sridhara, G., G., R. H., & Mazumdar, S. (2023) · arXiv · doi:10.48550/arxiv.2305.16837 · Cited by 68
Compressing Pre-trained Models of Code into 3 MB
Shi, J., Yang, Z., Xu, B., Kang, H. J., & Lo, D. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556964 · Cited by 67
Self-citationOn the Reliability and Explainability of Language Models for Program Generation
Liu, Y., Tantithamthavorn, C. K., Liu, Y., & Li, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3641540 · Cited by 67
Self-citationPTM4Tag: sharpening tag recommendation of stack overflow posts with pre-trained models
He, J., Xu, B., Yang, Z., Han, D., Yang, C., & Lo, D. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527897 · Cited by 66
Self-citationMaking the most of small Software Engineering datasets with modern machine learning
Prenner, J. A., & Robbes, R. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3135465 · Cited by 64
Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities
Khare, A., Dutta, S., Li, Z., Solko-Breslin, A., Alur, R., & Naik, M. (2023) · arXiv · doi:10.48550/arxiv.2311.16169 · Cited by 64
Proceedings of the 26th Conference on Program Comprehension
(2018) · doi:10.1145/3196321 · Cited by 61
Achieving Reliable Sentiment Analysis in the Software Engineering Domain using BERT
Biswas, E., Karabulut, M. E., Pollock, L., & Vijay‐Shanker, K. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00025 · Cited by 60
On the Effectiveness of Transfer Learning for Code Search
Salza, P., Schwizer, C., Gu, J. N., & Gall, H. C. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3192755 · Cited by 60
Tian, H., Liu, K., Li, Y., Kaboré, A. K., Koyuncu, A., Habib, A., Li, L., Wen, J., Klein, J., & Bissyandé, T. F. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3576039 · Cited by 60
Self-citationYou see what I want you to see: poisoning vulnerabilities in neural code search
Wan, Y., Zhang, S., Zhang, H., Sui, Y., Xu, G., Yao, D., Jin, H., & Sun, L. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549153 · Cited by 60
Towards JavaScript program repair with generative pre-trained transformer (GPT-2)
Lajkó, M., Csuvik, V., & Vidács, L. (2022) · IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1145/3524459.3527350 · Cited by 60
Multilingual Code Co-evolution using Large Language Models
Zhang, J., Nie, P., Li, J. J., & Gligoric, M. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616350 · Cited by 60
KnowLog: Knowledge Enhanced Pre-trained Language Model for Log Understanding
Ma, L., Yang, W., Xu, B., Jiang, S., Fei, B., Liang, J., Zhou, M., & Xiao, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623304 · Cited by 59
Modular Tree Network for Source Code Representation Learning
Wang, W., Li, G., Shen, S., Xia, X., & Jin, Z. G. (2020) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3409331 · Cited by 58
Using pre-trained language models to resolve textual and semantic merge conflicts (experience paper)
Zhang, J., Mytkowicz, T., Kaufman, M., Piskač, R., & Lahiri, S. K. (2022) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3533767.3534396 · Cited by 58
Automatic Model Selection with Large Language Models for Reasoning
Zhao, J., Xie, Y., Kawaguchi, K., He, J., & Xie, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.55 · Cited by 58
An exploratory study on code attention in BERT
Sharma, R., Chen, F., Fard, F. H., & Lo, D. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527921 · Cited by 56
Self-citationAPPT: Boosting Automated Patch Correctness Prediction via Fine-Tuning Pre-Trained Models
Zhang, Q., Fang, C., Sun, W., Liu, Y., He, T., Hao, X., & Chen, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3354969 · Cited by 56
Self-citationModel driven engineering for machine learning components: A systematic literature review
Naveed, H., Arora, C., Khalajzadeh, H., Grundy, J., & Haggag, O. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107423 · Cited by 56
Self-citationEffective API recommendation without historical software repositories
Liu, X., Huang, L., & Ng, V. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238216 · Cited by 55
Alam, A. I., Roy, P. R., Al-omari, F., Roy, C. K., Roy, B., & Schneider, K. A. (2023) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58846.2023.00013 · Cited by 54
DebugBench: Evaluating Debugging Capability of Large Language Models
Tian, R., Ye, Y., Qin, Y., Cong, X., Lin, Y., Pan, Y., Wu, Y., Haotian, H., Weichuan, L., Liu, Z., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.247 · Cited by 54
Ghadhab, L., Jenhani, I., Mkaouer, M. W., & Messaoud, M. B. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106566 · Cited by 53
Development of recommendation systems for software engineering: the CROSSMINER experience
Rocco, J. D., Ruscio, D. D., Sipio, C. D., Nguyen, P. T., & Rubei, R. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09963-7 · Cited by 52
LLMatic: Neural Architecture Search Via Large Language Models And Quality Diversity Optimization
Nasir, M. U., Earle, S., Togelius, J., James, S., & Cleghorn, C. (2024) · Proceedings of the Genetic and Evolutionary Computation Conference · doi:10.1145/3638529.3654017 · Cited by 52
µBert: Mutation Testing using Pre-Trained Language Models
Degiovanni, R., & Papadakis, M. (2022) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw55395.2022.00039 · Cited by 50
Assess and Summarize: Improve Outage Understanding with Large Language Models
Jin, P., Zhang, S., Ma, M., Li, H., Kang, Y., Li, L., Liu, Y., Qiao, B., Zhang, C., Zhao, P., et al. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3613891 · Cited by 50
Self-citationCapturing Failures of Large Language Models Via Human Cognitive Biases
Jones, E., & Steinhardt, J. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-0856 · Cited by 48
Data-driven prototyping via natural-language-based GUI retrieval
Kolthoff, K., Bartelt, C., & Ponzetto, S. P. (2023) · Automated Software Engineering · doi:10.1007/s10515-023-00377-x · Cited by 47
Invalidator: Automated Patch Correctness Assessment via Semantic and Syntactic Reasoning
Le-Cong, T., Luong, D.-M., Le, X.-B. D., Lo, D., Tran, N.-H., Quang-Huy, B., & Huynh, Q. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3255177 · Cited by 46
Self-citationWaDec: Decompiling WebAssembly Using Large Language Model
She, X., Zhao, Y., & Wang, H. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695020 · Cited by 45
Self-citationThe Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation
Nguyen, D., Nam, L., Dau, A. T. V., Nguyen, A., Nghiem, K., Guo, J., & Bui, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.316 · Cited by 44
Capturing Failures of Large Language Models via Human Cognitive Biases
Jones, E., & Steinhardt, J. (2022) · arXiv · doi:10.48550/arxiv.2202.12299 · Cited by 43
Evaluating ChatGPT and GPT-4 for Visual Programming
Singla, A. K. (2023) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3568812.3603474 · Cited by 43
CrashTranslator: Automatically Reproducing Mobile Application Crashes Directly from Stack Trace
Huang, Y., Wang, J., Liu, Z., Wang, Y., Wang, S., Chen, C., Hu, Y., & Wang, Q. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623298 · Cited by 43
Clover: Closed-Loop Verifiable Code Generation
Sun, C., Sheng, Y., Padon, O., & Barrett, C. (2023) · arXiv · doi:10.48550/arxiv.2310.17807 · Cited by 43
Kieker: A monitoring framework for software engineering research
Hasselbring, W., & van Hoorn, A. (2020) · Software Impacts · doi:10.1016/j.simpa.2020.100019 · Cited by 42
Duplicate Bug Report Detection: How Far Are We?
Zhang, T., Han, D., Vinayakarao, V., Irsan, I. C., Xu, B., Thung, F., Lo, D., & Jiang, L. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3576042 · Cited by 42
Self-citationReCode: Robustness Evaluation of Code Generation Models
Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.773 · Cited by 42
Natural Language to Code: How Far Are We?
Wang, S., Geng, M., Lin, B., Sun, Z., Wen, M., Liu, Y., Li, L., Bissyandé, T. F., & Mao, X. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616323 · Cited by 42
Self-citationTeaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation
Wang, C., Zhang, J., Feng, Y., Li, T., Sun, W., Liu, Y., & Peng, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714462 · Cited by 40
Self-citationWhere is your app frustrating users?
Wang, Y., Wang, J., Zhang, H., Ming, X., Shi, L., & Wang, Q. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510189 · Cited by 39
PyTy: Repairing Static Type Errors in Python
Chow, Y. W., Grazia, L. D., & Pradel, M. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639184 · Cited by 39
PentestGPT: An LLM-empowered Automatic Penetration Testing Tool
Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Pinzger, M., & Rass, S. (2023) · arXiv · doi:10.48550/arxiv.2308.06782 · Cited by 37
Self-citationRefBERT: A Two-Stage Pre-trained Framework for Automatic Rename Refactoring
Liu, H., Wang, Y., Zhao, W., Xu, Y., Wang, J., Li, H., & Ji, R. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598092 · Cited by 36
Aligning Offline Metrics and Human Judgments of Value for Code Generation Models
Dibia, V. C., Fourney, A., Bansal, G., Poursabzi-Sangdeh, F., Liu, H., & Amershi, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.540 · Cited by 36
Exploring Distributional Shifts in Large Language Models for Code Analysis
Arakelyan, S., Das, R. J., Mao, Y., & Ren, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1013 · Cited by 36
Capturing source code semantics via tree-based convolution over API-enhanced AST
Chen, L., Ye, W., & Zhang, S. (2019) · ACM International Conference on Computing Frontiers (CF) · doi:10.1145/3310273.3321560 · Cited by 35
Incivility Detection in Open Source Code Review and Issue Discussions
Ferreira, I., Rafiq, A., & Cheng, J. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111935 · Cited by 35
An Empirical Study on Code Comment Completion
Mastropaolo, A., Aghajani, E., Pascarella, L., & Bavota, G. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.26226/morressier.613b5418842293c031b5b5da · Cited by 34
Perfect is the enemy of test oracle
Ibrahimzada, A. R., Varli, Y., Tekinoglu, D., & Jabbarvand, R. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549086 · Cited by 34
Chen, J., Shang, W., Hassan, A. E., Wang, Y., & Lin, J. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00068 · Cited by 33
AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation
Murali, V., Maddila, C., Ahmad, I., Bolin, M., Cheng, D., Ghorbani, N., Fernandez, R., Nagappan, N., & Rigby, P. C. (2023) · arXiv · doi:10.48550/arxiv.2305.12050 · Cited by 33
AutoPruner: transformer-based call graph pruning
Le-Cong, T., Kang, H. J., Nguyen, T. G., Haryono, S. A., Lo, D., Le, X.-B. D., & Huynh, Q. T. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549175 · Cited by 32
Self-citationAddressing Compiler Errors: Stack Overflow or Large Language Models?Preprint
Widjojo, P., & Treude, C. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4529345 · Cited by 32
A Static Evaluation of Code Completion by Large Language Models
Ding, H., Kumar, V., Tian, Y., Wang, Z., Kwiatkowski, R., Li, X., Ramanathan, M. K., Ray, B., Bhatia, P., & Sengupta, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.34 · Cited by 32
Benchmarking Language Models for Code Syntax Understanding
Shen, D., Chen, X., Wang, C., Sen, K., & Song, D. X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.224 · Cited by 32
OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models
Wang, S., Ding, L., Li, S., Luo, Y., Du, B., & Tao, D. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.808 · Cited by 32
Less is More: Summary of Long Instructions is Better for Program Synthesis
Kuznia, K., Mishra, S. R., Parmar, M., & Baral, C. R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.301 · Cited by 31
Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation
Yuan, Z., Liu, J., Zi, Q., Liu, M., Peng, X., & Lou, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.01240 · Cited by 31 · cited 1×
“Alternatively, they merely offer a preliminary exploration of the performance of LLMs in various SE tasks through empirical experiments, without conducting a systematic literature survey [61, 201, 296, 357, 373, 395].”
BackgroundAPIGen: Generative API Method Recommendation
Chen, Y., Gao, C., Zhu, M., Liao, Q., Wang, Y., & Xu, G. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00025 · Cited by 30
SparseCoder: Identifier-Aware Sparse Transformer for File- Level Code Summarization
Wang, Y., Huang, Y., Guo, D., Zhang, H., & Zheng, Z. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00068 · Cited by 30
Latent Execution for Neural Program Synthesis Beyond Domain-Specific LanguagesPreprint
Chen, X., Song, D. X., & Tian, Y. (2021) · arXiv (Cornell University) · Cited by 29
Duplicate Bug Report Detection by Using Sentence Embedding and Fine-tuning
Isotani, H., Washizaki, H., Fukazawa, Y., Nomoto, T., Ouji, S., & Saito, S. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00054 · Cited by 29
DexBERT: Effective, Task-Agnostic and Fine-Grained Representation Learning of Android Bytecode
Sun, T., Allix, K., Kim, K., Zhou, X., Kim, D., Lo, D., Bissyandé, T. F., & Klein, J. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3310874 · Cited by 29
Self-citationRAPID: Zero-Shot Domain Adaptation for Code Search with Pre-Trained Models
Fan, G., Chen, S., Gao, C., Xiao, J., Zhang, T., & Feng, Z. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3641542 · Cited by 28
T-FREX: A Transformer-based Feature Extraction Method from Mobile App Reviews
Motger, Q., Miaschi, A., Dell’Orletta⋄, F., Franch, X., & Marco, J. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00030 · Cited by 28
Gong, L., Zhang, J., Wei, M., Zhang, H., & Huang, Z. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3569934 · Cited by 27
Fine-SE: Integrating Semantic Features and Expert Features for Software Effort Estimation
Li, Y., Ren, Z. M., Wang, Z., Yang, L., Dong, L., Zhong, C., & Zhang, H. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623349 · Cited by 26
Using a Nearest-Neighbour, BERT-Based Approach for Scalable Clone Detection
Chochlov, M., Ahmed, G. A., Patten, J. V., Guo-xian, L., Hou, W., Gregg, D., & Buckley, J. (2022) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme55016.2022.00080 · Cited by 25
A Light Bug Triage Framework for Applying Large Pre-trained Language Model
Lee, J., Han, K., & Yu, H. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556898 · Cited by 24
PTM-APIRec: Leveraging Pre-trained Models of Source Code in API Recommendation
Li, Z., Li, C., Tang, Z. J., Huang, W., Ge, J., Luo, B., Ng, V., Wang, T., Hu, Y., & Zhang, X. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3632745 · Cited by 24
Alhamed, M., & Storer, T. (2022) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme55016.2022.00020 · Cited by 23
Grishina, A., Hort, M., & Moonen, L. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616304 · Cited by 23
Discriminating Human-authored from ChatGPT-Generated Code Via Discernable Feature Analysis
Li, K., Hong, S., Fu, C., Zhang, Y., & Liu, M. (2023) · 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew60843.2023.00059 · Cited by 23
Self-Supervised Query Reformulation for Code Search
Mao, Y., Wan, C., Jiang, Y., & Gu, X. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616306 · Cited by 23
Identification of intra-domain ambiguity using transformer-based machine learning
Moharil, A., & Sharma, A. (2022) · IEEE/ACM International Workshop on Natural Language-Based Software Engineering (NLBSE) · doi:10.1145/3528588.3528651 · Cited by 22
API Entity and Relation Joint Extraction from Text via Dynamic Prompt-tuned Language Model
Huang, Q. Y., Sun, Y., Xing, Z., Yu, M., Xu, X., & Lu, Q. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3607188 · Cited by 22
Log Parsing with Generalization Ability under New Log Types
Yu, S., Wu, Y., Li, Z., He, P., Chen, N., & Liu, C. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616355 · Cited by 22
Saberi, I., Fard, F. H., & Chen, F. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10457-5 · Cited by 22
(2022) · doi:10.2139/ssrn.4496380 · Cited by 20
Weyssow, M., Zhou, X., Kim, K., Lo, D., & Sahraoui, H. A. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616244 · Cited by 20
Self-citationPre-training Code Representation with Semantic Flow Graph for Effective Bug Localization
Du, Y., & Yu, Z. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616338 · Cited by 19
Tricking llms into disobedience: Formalizing, analyzing, and detecting jailbreaks
Rao, A., Vashistha, S., Naik, A., Aditya, S., & Choudhury, M. (2024) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/2ivv5y3ufu9y · Cited by 19
A syntax-guided multi-task learning approach for Turducken-style code generation
Yang, G., Zhou, Y., Chen, X., Zhang, X., Xu, Y., Han, T., & Chen, T. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10372-1 · Cited by 18
A Comparative Study of Code Generation using ChatGPT 3.5 across 10 Programming Languages
Buscemi, A. (2023) · arXiv · doi:10.48550/arxiv.2308.04477 · Cited by 18
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2208.08227 · Cited by 17
Proceedings of the 15th ACM International Conference on Computing Frontiers
Pina, L., Andronidis, A., & Cadar, C. (2018) · doi:10.1145/3203217 · Cited by 16
Zhang, J., Liu, S., Gong, L., Zhang, H., Huang, Z., & Jiang, H. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3227559 · Cited by 16
ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets Using LLMs
Kabir, A., Wang, S., Tian, Y., Chen, T.-H. P., Asaduzzaman, M., & Zhang, W. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3702979 · Cited by 16
Explaining Explanations: An Empirical Study of Explanations in Code Reviews
Widyasari, R., Zhang, T., Bouraffa, A., Maalej, W., & Lo, D. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708518 · Cited by 16
Self-citationChain of Code: Reasoning with a Language Model-Augmented Code Emulator
Li, C., Liang, J., Zeng, A., Chen, X., Hausman, K., Sadigh, D., Levine, S., Fei-Fei, L., Xia, F., & Ichter, B. (2023) · arXiv · doi:10.48550/arxiv.2312.04474 · Cited by 16
Learning to Predict User-Defined Types
Jesse, K., Dévanbu, P., & Sawant, A. A. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3178945 · Cited by 15
Automated Expansion of Abbreviations Based on Semantic Relation and Transfer Expansion
Jiang, Y., Liu, H., Jin, J., & Zhang, L. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2995736 · Cited by 14
Stealthy Backdoor Attack for Code ModelsPreprint
Yang, Z., Xu, B., Zhang, J. M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.02496 · Cited by 14
Self-citationTABASCO: A transformer based contextualization toolkit
Moharil, A., & Sharma, A. (2023) · Science of Computer Programming · doi:10.1016/j.scico.2023.102994 · Cited by 14
Understanding Programs by Exploiting (Fuzzing) Test Cases
Zhao, J., Rong, Y., Guo, Y., He, Y., & Chen, H. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.678 · Cited by 13
Is Self-Repair a Silver Bullet for Code Generation?
Olausson, T. X., Inala, J. P., Wang, C., Gao, J., & Solar-Lezama, A. (2023) · arXiv · doi:10.48550/arxiv.2306.09896 · Cited by 13
Transparent logging as a technique for debugging complex distributed systems
Satyanarayanan, M., Steere, D. C., Kudo, M., et al. (1992) · doi:10.1145/506378.506421 · Cited by 12
Poison Attack and Defense on Deep Source Code Processing ModelsPreprint
Li, J., Li, Z., Zhang, H., Li, G., Jin, Z. G., Hu, X., & Xia, X. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.17029 · Cited by 12
Wang, S., Jean, S., Sengupta, S., Gung, J., Παππάς, Ν., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.478 · Cited by 11
On the Impossible Safety of Large AI Models
El-Mhamdi, E.-M., Farhadkhani, S., Guerraoui, R., Gupta, N., Hoang, L.-N., Pinot, R., Rouault, S., & Stephan, J. (2022) · arXiv · doi:10.48550/arxiv.2209.15259 · Cited by 11
Proceedings of the 37th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages
Hermenegildo, M. V., & Palsberg, J. (2010) · Cited by 9
Assessing and Improving Syntactic Adversarial Robustness of Pre-Trained Models for Code TranslationPreprint
Yang, G., Zhou, Y., Zhang, X., Chen, X., Han, T., & Chen, T. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4623115 · Cited by 9
LLM-Powered Code Vulnerability Repair with Reinforcement Learning and Semantic Reward
Islam, N. T., Khoury, J., Seong, A., Karkevandi, M. B., Parra, G. D. L. T., Bou-Harb, E., & Najafirad, P. (2024) · arXiv · doi:10.48550/arxiv.2401.03374 · Cited by 8
Semantic-Enhanced Static Vulnerability Detection in Baseband Firmware
Liu, Y., Zhang, C., Li, F., et al. (2024) · IEEE/ACM International Conference on Software Engineering (ICSE) · doi:10.1145/3597503.3639158 · Cited by 7
Self-citationAPPT: Boosting Automated Patch Correctness Prediction via Fine-tuning Pre-trained ModelsPreprint
Zhang, Q., Fang, C., Sun, W., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.12453 · Cited by 5
Automatic recognizing relevant fragments of APIs using API references
Wu, D., Feng, Y., Zhang, H., & Xu, B. (2023) · Automated Software Engineering · doi:10.1007/s10515-023-00401-0 · Cited by 5
Helmeczi, R. K. (2024) · doi:10.32920/26866606.v1 · Cited by 5
Making the most of small Software Engineering datasets with modern machine learningPreprint
Prenner, J. A., & Robbes, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15209 · Cited by 4
DebugBench: Evaluating Debugging Capability of Large Language Models
Tian, R., Ye, Y., Qin, Y., Cong, X., Lin, Y., Pan, Y., Wu, Y., Hui, H., Liu, W., Liu, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.04621 · Cited by 2
Banâtre, J.-P. (1992) · Cited by 1
Research works printed but not linked: 271
- Mayank Agarwal Yikang Shen Bailin Wang Yoon Kim and Jie Chen. 2024. Structured code representations enable data-efficient adaptation of code language models (2024). arXiv:2401.10716. Retrieved from https://arxiv.org/abs/2401.10716 · arXiv
- Lakshya Agrawal, Aditya Kanade, Navin Goyal, Shuvendu K. Lahiri, and Sriram Rajamani. 2023. Monitor-guided decoding of code lms with static analysis of repository context. In Advances in Neural Information Processing Systems. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 32270–32298. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2023/file/662b1774ba8845fc1fa3d1fc0177ceeb-Paper-Conference.pdf
- Baleegh Ahmad Shailja Thakur Benjamin Tan Ramesh Karri and Hammond Pearce. 2023. Fixing hardware security bugs with large language models. arXiv:2302.01215. · arXiv
- Toufique Ahmed Kunal Suresh Pai Premkumar Devanbu and Earl T. Barr. 2023. Improving few-shot prompts with relevant static analysis products. arXiv:2304.06815. · arXiv
- Ali Al-Kaswan, Toufique Ahmed, Maliheh Izadi, Anand Ashok Sawant, Premkumar Devanbu, and Arie van Deursen. 2023. Extending source code pre-trained language models to summarise decompiled binarie. In Proceedings of the 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’23). IEEE, 260–271.
- Sven Amann, Sebastian Proksch, Sarah Nadi, and Mira Mezini. 2016. A study of visual studio usage in practice. In Proceedings of the 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER ’16), Vol. 1. IEEE, 124–134.
- Patrick Bareiß Beatriz Souza Marcelo d’Amorim and Michael Pradel. 2022. Code generation tools (almost) for free? A study of few-shot pre-trained language models on code. arXiv:2206.01335. · arXiv
- Nghi D. Q. Bui Hung Le Yue Wang Junnan Li Akhilesh Deepak Gotmare and Steven C. H. Hoi. 2023. Codetf: One-stop transformer library for state-of-the-art code LLM. arXiv:2306.00029. · arXiv
- Jialun Cao Meiziniu Li Ming Wen and Shing-chi Cheung. 2023. A study on prompt design advantages and limitations of chatgpt for deep learning program repair. arXiv:2304.08191. · arXiv
- Aaron Chan Anant Kharkar Roshanak Zilouchian Moghaddam Yevhen Mohylevskyy Alec Helyar Eslam Kamal Mohamed Elkamhawy and Neel Sundaresan. 2023. Transformer-based vulnerability detection in code at edittime: Zero-shot few-shot or fine-tuning? arXiv:2306.01754. · arXiv
- Yiannis Charalambous Norbert Tihanyi Ridhi Jain Youcheng Sun Mohamed Amine Ferrag and Lucas C. Cordeiro. 2023. A new era in software security: Towards self-healing software via large language models and formal verification. arXiv:2305.14752. · arXiv
- Angelica Chen Jérémy Scheurer Tomasz Korbak Jon Ander Campos Jun Shern Chan Samuel R. Bowman Kyunghyun Cho and Ethan Perez. 2023. Improving code generation by training with natural language feedback. arXiv:2303.16749. · arXiv
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating large language models trained on code. arXiv:2107.03374. · arXiv
- Meng Chen Hongyu Zhang Chengcheng Wan Zhao Wei Yong Xu Juhong Wang and Xiaodong Gu. 2023. On the effectiveness of large language models in domain-specific code generation. arXiv:2312.01639. · arXiv
- Xinyun Chen Maxwell Lin Nathanael Schärli and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv:2304.05128. · arXiv
- Xinyun Chen Chang Liu and Dawn Song. 2017. Towards synthesizing complex programs from input-output examples. arXiv:1706.01284. · arXiv
- Agnieszka Ciborowska and Kostadin Damevski. 2023. Too few bug reports? Exploring data augmentation for improved changeset-based bug localization. arXiv:2305.16430. · DOI
- Arghavan Moradi Dakhel Amin Nikanjam Vahid Majdinasab Foutse Khomh and Michel C. Desmarais. 2023. Effective test generation using pre-trained large language models and mutation testing. arXiv:2308.16557. · arXiv
- Pantazis Deligiannis Akash Lal Nikita Mehrotra and Aseem Rastogi. 2023. Fixing rust compilation errors using llms. arXiv:2308.05177. · arXiv
- Yinlin Deng Chunqiu Steven Xia Chenyuan Yang Shizhuo Dylan Zhang Shujing Yang and Lingming Zhang. 2023. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. arXiv:2304.02014. · DOI · Paper page
- Tuan Dinh, Jinman Zhao, Samson Tan, Renato Negrinho, Leonard Lausen, Sheng Zha, and George Karypis. 2024. Large language models of code fail at completing code with potential bugs. In Advances in Neural Information Processing Systems. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 41386–41412. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2023/file/819cebb05f993840e8a52d7564c5c282-Paper-Conference.pdf
- Guanting Dong Hongyi Yuan Keming Lu Chengpeng Li Mingfeng Xue Dayiheng Liu Wei Wang Zheng Yuan Chang Zhou and Jingren Zhou. 2023. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv:2310.05492. · arXiv
- Yihong Dong Jiazheng Ding Xue Jiang Ge Li Zhuo Li and Zhi Jin. 2023. Codescore: Evaluating code generation by learning code execution. arXiv:2301.09043. · arXiv
- Xueying Du Mingwei Liu Juntao Li Hanlin Wang Xin Peng and Yiling Lou. 2023. Resolving crash bugs via large language models: An empirical study. arXiv:2312.10448. · arXiv
- Xueying Du Mingwei Liu Kaixin Wang Hanlin Wang Junwei Liu Yixuan Chen Jiayi Feng Chaofeng Sha Xin Peng and Yiling Lou. 2023. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv:2308.01861. Retrieved from https://arxiv.org/abs/2308.01861 · arXiv
- Aryaz Eghbali and Michael Pradel. 2024. De-hallucinator: Iterative grounding for llm-based code completion. arXiv:2401.01701. · arXiv
- Abdelkarim El-Hajjami Nicolas Fafin and Camille Salinesi. 2023. Which ai technique is better to classify requirements? An experiment with SVM LSTM and ChatGPT. arXiv:2311.11547. · arXiv
- Madeline Endres Sarah Fakhoury Saikat Chakraborty and Shuvendu K. Lahiri. 2023. Formalizing natural language intent into program specifications via large language models. arXiv:2310.01831. Retrieved from https://arxiv.org/abs/2310.01831 · arXiv
- Sarah Fakhoury Saikat Chakraborty Madan Musuvathi and Shuvendu K. Lahiri. 2023. Towards generating functionally correct code edits from natural language issue descriptions. arXiv:2304.03816. Retrieved from https://arxiv.org.abs/2304.03816 · arXiv
- Zhiyu Fan Xiang Gao Abhik Roychoudhury and Shin Hwei Tan. 2022. Automated repair of programs from large language models. arXiv:2205.10583. Retrieved from https://arxiv.org/abs/2205.10583 · arXiv
- Mohamed Amine Ferrag Ammar Battah Norbert Tihanyi Merouane Debbah Thierry Lestable and Lucas C. Cordeiro. 2023. Securefalcon: The next cyber reasoning system for cyber security. arXiv:2307.06616. Retrieved from https://arxiv.org/abs/2307.06616 · arXiv
- Daniel Fried Armen Aghajanyan Jessy Lin Sida Wang Eric Wallace Freda Shi Ruiqi Zhong Wen-tau Yih Luke Zettlemoyer and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv:2204.05999. Retrieved from https://arxiv.org/abs/2204.05999 · arXiv
- Apurva Gandhi Thong Q. Nguyen Huitian Jiao Robert Steen and Ameya Bhatawdekar. 2023. Natural language commanding via program synthesis. arXiv:2306.03460. Retrieved from https://arxiv.org/abs/2306.03460 · arXiv
- Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite Noa Nabeshima Shawn Presser and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv:2101.00027. Retrieved from https://arxiv.org/abs/2101.00027 · arXiv
- Shuzheng Gao Wenxin Mao Cuiyun Gao Li Li Xing Hu Xin Xia and Michael R. Lyu. 2024. Learning in the wild: Towards leveraging unlabeled data for effectively tuning pre-trained code models. arXiv:2401.01060. Retrieved from https://arxiv.org/abs/2401.01060 · arXiv
- Shuzheng Gao Xin-Cheng Wen Cuiyun Gao Wenxuan Wang and Michael R. Lyu. 2023. Constructing effective in-context demonstration for code intelligence tasks: An empirical study. arXiv:2304.07575. Retrieved from https://arxiv.org/abs/2304.07575 · arXiv
- Zeyu Gao Hao Wang Yuchen Zhou Wenyu Zhu and Chao Zhang. 2023. How far have we gone in vulnerability detection using large language models. arXiv:2311.12420. Retrieved from https://arxiv.org/abs/2311.12420 · arXiv
- Jian Gu, Pasquale Salza, and Harald C. Gall. 2022. Assemble foundation models for automatic code summarization. In Proceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’22). IEEE, 935–946.
- Daya Guo Shuo Ren Shuai Lu Zhangyin Feng Duyu Tang Shujie Liu Long Zhou Nan Duan Alexey Svyatkovskiy Shengyu Fu Michele Tufano Shao Kun Deng Colin Clement Dawn Drain Neel Sundaresan Jian Yin Daxin Jiang and Ming Zhou. 2020. Graphcodebert: Pre-training code representations with data flow. arXiv:2009.08366. Retrieved from https://arxiv.org/abs/2009.08366 · arXiv
- Daya Guo Canwen Xu Nan Duan Jian Yin and Julian McAuley. 2023. Longcoder: A long-range pre-trained language model for code completion. arXiv:2306.14893. Retrieved from https://arxiv.org/abs/2306.14893 · arXiv
- Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Y. Wu Y. K. Li Fuli Luo Yingfei Xiong and Wenfeng Liang. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv:2401.14196. Retrieved from https://arxiv.org/abs/2401.14196 · arXiv
- Priyanshu Gupta Avishree Khare Yasharth Bajpai Saikat Chakraborty Sumit Gulwani Aditya Kanade Arjun Radhakrishna Gustavo Soares and Ashish Tiwari. 2023. Grace: Generation using associated code edits. arXiv:2305.14129. Retrieved from https://arxiv.org/abs/2305.14129 · arXiv
- Patrick Hajali and Ignas Budvytis. 2023. Function-constrained program synthesis. arXiv:2311.15500. Retrieved from https://arxiv.org/abs/2311.15500 · arXiv
- Yu Hao Weiteng Chen Ziqiao Zhou and Weidong Cui. 2023. E & v: Prompting large language models to perform static analysis by pseudo-code execution and verification. arXiv:2312.08477. Retrieved from https://arxiv.org/abs/2312.08477 · arXiv
- Junda He Zhou Xin Bowen Xu Ting Zhang Kisub Kim Zhou Yang Ferdian Thung Ivana Irsan and David Lo. 2023. Representation learning for stack overflow posts: How far are we? arXiv:2303.06853. Retrieved from https://arxiv.org/abs/2303.06853 · arXiv
- Jordan Henkel, Denini Silva, Leopoldo Teixeira, Marcelo d’Amorim, and Thomas Reps. 2021. Shipwright: A human-in-the-loop system for dockerfile repair. In Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE ’21). IEEE, 1148–1160.
- Tobias Hey, Jan Keim, Anne Koziolek, and Walter F. Tichy. 2020. Norbert: Transfer learning for requirements classification. In Proceedings of the 2020 IEEE 28th International Requirements Engineering Conference (RE ’20). IEEE, 169–179.
- Sirui Hong Xiawu Zheng Jonathan Chen Yuheng Cheng Jinlin Wang Ceyao Zhang Zili Wang Steven Ka Shing Yau Zijuan Lin Liyang Zhou Chenyu Ran Lingfeng Xiao Chenglin Wu and Jürgen Schmidhuber. 2023. METAGPT: Meta programming for multi-agent collaborative framework. arXiv:2308.00352. Retrieved from https://arxiv.org/abs/2308.00352 · arXiv
- Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the International Conference on Machine Learning. PMLR, 2790–2799.
- Jie Hu Qian Zhang and Heng Yin. 2023. Augmenting greybox fuzzing with generative AI. arXiv:2306.06782. Retrieved from https://arxiv.org/abs/2306.06782 · arXiv
- Xueyu Hu Kun Kuang Jiankai Sun Hongxia Yang and Fei Wu. 2024. Leveraging print debugging to improve code generation in large language models. arXiv:2401.05319. Retrieved from https://arxiv.org/abs/2401.05319 · arXiv
- Dong Huang Qingwen Bu and Heming Cui. 2023. Codecot and beyond: Learning to program and test like a developer. arXiv: 2308.08784. Retrieved from https://arxiv.org/abs/2308.08784 · arXiv
- Dong Huang Qingwen Bu Jie M. Zhang Michael Luck and Heming Cui. 2023. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv: 2312.13010. Retrieved from https://arxiv.org/abs/2312.13010 · arXiv
- Di Huang Ziyuan Nan Xing Hu Pengwei Jin Shaohui Peng Yuanbo Wen Rui Zhang Zidong Du Qi Guo Yewen Pu and Yunji Chen. 2023. anpl: Compiling natural programs with interactive decomposition. arXiv:2305.18498. Retrieved from https://arxiv.org/abs/2305.18498 · arXiv
- Qing Huang Yishun Wu Zhenchang Xing He Jiang Yu Cheng and Huan Jin. 2023. Adaptive intellect unleashed: The feasibility of knowledge transfer in large language models. arXiv:2308.04788. Retrieved from https://arxiv.org/abs/2308.04788 · arXiv
- Qing Huang Jiahui Zhu Zhenchang Xing Huan Jin Changjing Wang and Xiwei Xu. 2023. A chain of ai-based solutions for resolving fqns and fixing syntax errors in partial code. arXiv:2306.11981. Retrieved from https://arxiv.org/abs/2306.11981 · arXiv
- Qing Huang Zhou Zou Zhenchang Xing Zhenkang Zuo Xiwei Xu and Qinghua Lu. 2023. Ai chain on large language model for unsupervised control flow graph generation for statically-typed partial code. arXiv: 2306.00757. Retrieved from https://arxiv.org/abs/2306.00757 · arXiv
- Ali Reza Ibrahimzada Yang Chen Ryan Rong and Reyhaneh Jabbarvand. 2023. Automated bug generation in the era of large language models. arXiv: 2310.02407. Retrieved from https://arxiv.org/abs/2310.02407 · arXiv
- Nafis Tanveer Islam and Peyman Najafirad. 2024. Code security vulnerability repair using reinforcement learning with large language models. arXiv:2401.07031. Retrieved from https://arxiv.org/abs/2401.07031 · arXiv
- Abhinav Jain Chima Adiole Thomas Reps Swarat Chaudhuri and Chris Jermaine. 2024. Coarse-tuning models of code with reinforcement learning feedback. Retrieved from https://openreview.net/forum?id=vLqkCvjHRD
- Naman Jain Tianjun Zhang Wei-Lin Chiang Joseph E Gonzalez Koushik Sen and Ion Stoica. 2023. LLM-assisted code cleaning for training accurate code generators. arXiv:2311.14904. Retrieved from https://arxiv.org/abs/2311.14904 · arXiv
- Prithwish Jana Piyush Jha Haoyang Ju Gautham Kishore Aryan Mahajan and Vijay Ganesh. 2023. Attention compilation and solver-based symbolic analysis are all you need. arXiv:2306.06755. Retrieved from https://arxiv.org/abs/2306.06755 · arXiv
- Zhenlan Ji Pingchuan Ma Zongjie Li and Shuai Wang. 2023. Benchmarking and explaining large language model-based code generation: A causality-centric approach. arXiv:2310.06680. Retrieved from https://arxiv.org/abs/2310.06680 · arXiv
- Nan Jiang Kevin Liu Thibaud Lutellier and Lin Tan. 2023. Impact of code language models on automated program repair. arXiv:2302.05020. Retrieved from https://arxiv.org/abs/2302.05020 · arXiv
- Nan Jiang Chengxiao Wang Kevin Liu Xiangzhe Xu Lin Tan and Xiangyu Zhang. 2023. Nova \({}^{+}\) : Generative language models for binaries. arXiv:2311.13721. Retrieved from https://arxiv.org/abs/2311.13721 · arXiv
- Shuyang Jiang Yuhao Wang and Yu Wang. 2023. Selfevolve: A code evolution framework via large language models. arXiv:2306.02907. Retrieved from https://arxiv.org/abs/2306.02907 · arXiv
- Xue Jiang Yihong Dong Lecheng Wang Qiwei Shang and Ge Li. 2023. Self-planning code generation with large language model. arXiv:2303.06689. Retrieved from https://arxiv.org/abs/2303.06689 · arXiv
- Carlos E. Jimenez John Yang Alexander Wettig Shunyu Yao Kexin Pei Ofir Press and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world GitHub issues? arXiv:2310.06770. Retrieved from https://arxiv.org/abs/2310.06770 · arXiv
- Matthew Jin Syed Shahriar Michele Tufano Xin Shi Shuai Lu Neel Sundaresan and Alexey Svyatkovskiy. 2023. Inferfix: End-to-end program repair with LLMS. arXiv:2303.07263. Retrieved from https://arxiv.org/abs/2303.07263 · arXiv
- Xin Jin Jonathan Larson Weiwei Yang and Zhiqiang Lin. 2023. Binary code summarization: Benchmarking CHATGPT/GPT-4 and other large language models. arXiv:2312.09601. Retrieved from https://arxiv.org/abs/2312.09601 · arXiv
- Md Mahir Asef Kabir Sk Adnan Hassan Xiaoyin Wang Ying Wang Hai Yu and Na Meng. 2023. An empirical study of ChatGPT-3.5 on question answering and code maintenance. arXiv:2310.02104. Retrieved from https://arxiv.org/abs/2310.02104 · arXiv
- Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In Proceedings of the International Conference on Machine Learning. PMLR, 5110–5121.
- Sungmin Kang Gabin An and Shin Yoo. 2023. A preliminary evaluation of LLM-based fault localization. arXiv:2308.05487. Retrieved from https://arxiv.org/abs/2308.05487 · arXiv
- Sungmin Kang Juyeon Yoon Nargiz Askarbekkyzy and Shin Yoo. 2023. Evaluating diverse large language models for automatic and general bug reproduction. arXiv:2311.04532. Retrieved from https://arxiv.org/abs/2311.04532 · arXiv
- Sungmin Kang Juyeon Yoon and Shin Yoo. 2022. Large language models are few-shot testers: Exploring LLM-based general bug reproduction. arXiv:2209.11515. Retrieved from https://arxiv.org/abs/2209.11515 · arXiv
- Jai Kannan. 2023. Can llms configure software tools. arXiv:2312.06121. Retrieved from https://arxiv.org/abs/2312.06121 · arXiv
- Rafael-Michael Karampatsis and Charles Sutton. 2020. Scelmo: Source code embeddings from language models. arXiv:2004.13214. Retrieved from https://arxiv.org/abs/2004.13214 · arXiv
- Adam Khakhar Stephen Mell and Osbert Bastani. 2023. PAC prediction sets for large language models of code. arXiv:2302.08703. Retrieved from https://arxiv.org/abs/2302.08703 · arXiv
- Junaed Younus Khan, Md Tawkat Islam Khondaker, Gias Uddin, and Anindya Iqbal. 2021. Automatic detection of five API documentation smells: Practitioners’ perspectives. In Proceedings of the 2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’21). IEEE, 318–329.
- Junaed Younus Khan and Gias Uddin. 2022. Automatic detection and analysis of technical debts in peer-review documentation of r packages. In Proceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’22). IEEE, 765–776.
- Mohammad Abdullah Matin Khan M. Saiful Bari Xuan Long Do Weishi Wang Md Rizwan Parvez and Shafiq Joty. 2023. Xcodeeval: A large scale multilingual multitask benchmark for code understanding generation translation and retrieval. arXiv:2303.03004. Retrieved from https://arxiv.org/abs/2303.03004 · arXiv
- Muhammad Fawad Akbar Khan Max Ramsdell Erik Falor and Hamid Karimi. 2023. Assessing the promise and pitfalls of chatgpt for automated code generation. arXiv:2311.02640. Retrieved from https://arxiv.org/abs/2311.02640 · arXiv
- Hiroyuki Kirinuki and Haruto Tanno. 2024. Chatgpt and human synergy in black-box testing: A comparative analysis. arXiv:2401.13924. Retrieved from https://arxiv.org/abs/2401.13924 · arXiv
- Barbara Ann Kitchenham and Stuart Charters. 2007. Guidelines for performing systematic literature reviews in software engineering. In Technical Report EBSE 2007-001. Keele University and Durham University Joint Report. Retrieved from https://www.elsevier.com/__data/promis_misc/525444systematicreviewsguide.pdf
- 10.5555/1987360.1987364 · doi:10.5555/1987360.1987364
- Takashi Koide Naoki Fukushi Hiroki Nakano and Daiki Chiba. 2023. Detecting phishing sites using chatgpt. arXiv:2306.05816. Retrieved from https://arxiv.org/abs/2306.05816 · arXiv
- Bonan Kou Muhao Chen and Tianyi Zhang. 2023. Automated summarization of stack overflow posts. arXiv:2305.16680. Retrieved from https://arxiv.org/abs/2305.16680 · arXiv
- Bonan Kou Shengmai Chen Zhijie Wang Lei Ma and Tianyi Zhang. 2023. Is model attention aligned with human attention? An empirical study on large language models for code generation. arXiv:2306.01220. Retrieved from https://arxiv.org/abs/2306.01220 · arXiv
- Shuvendu K Lahiri Aaditya Naik Georgios Sakkas Piali Choudhury Curtis von Veh Madanlal Musuvathi Jeevana Priya Inala Chenglong Wang and Jianfeng Gao. 2022. Interactive code generation via test-driven user-intent formalization. arXiv:2208.05950. Retrieved from https://arxiv.org/abs/2208.05950 · arXiv
- Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In Proceedings of the International Conference on Machine Learning. PMLR, 18319–18345.
- Md Tahmid Rahman Laskar M. Saiful Bari Mizanur Rahman Md Amran Hossen Bhuiyan Shafiq Joty and Jimmy Xiangji Huang. 2023. A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. arXiv:2305.18486. Retrieved from https://arxiv.org/abs/2305.18486 · arXiv
- Hung Le Hailin Chen Amrita Saha Akash Gokul Doyen Sahoo and Shafiq Joty. 2023. Codechain: Towards modular code generation through chain of self-revisions with representative sub-modules. arXiv:2310.08992. Retrieved from https://arxiv.org/abs/2310.08992 · arXiv
- Dong Li Yelong Shen Ruoming Jin Yi Mao Kuan Wang and Weizhu Chen. 2022. Generation-augmented query expansion for code retrieval. arXiv:2212.10692. Retrieved from https://arxiv.org/abs/2212.10692 · arXiv
- Feng-Lin Li, Jennifer Horkoff, John Mylopoulos, Renata S. S. Guizzardi, Giancarlo Guizzardi, Alexander Borgida, and Lin Liu. 2014. Non-functional requirements as qualities, with a spice of ontology. In Proceedings of the 2014 IEEE 22nd International Requirements Engineering Conference (RE ’14). IEEE, 293–302.
- Haochen Li Xin Zhou and Zhiqi Shen. 2024. Rewriting the code: A simple method for large language model augmented code search. arXiv:2401.04514. Retrieved from https://arxiv.org/abs/2401.04514 · arXiv
- Jingyao Li Pengguang Chen and Jiaya Jia. 2023. Motcoder: Elevating large language models with modular of thought for challenging programming tasks. arXiv:2312.15960. Retrieved from https://arxiv.org/abs/2312.15960 · arXiv
- Jingxuan Li, Rui Huang, Wei Li, Kai Yao, and Weiguo Tan. 2021. Toward less hidden cost of code completion with acceptance and ranking models. In Proceedings of the 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME ’21). IEEE, 195–205.
- Jia Li Ge Li Yongmin Li and Zhi Jin. 2023. Enabling programming thinking in large language models toward code generation. arXiv:2305.06599. Retrieved from https://arxiv.org/abs/2305.06599 · arXiv
- Jia Li Ge Li Yongmin Li and Zhi Jin. 2023. Structured chain-of-thought prompting for code generation. arXiv:2305.06599. Retrieved from https://arxiv.org/abs/2305.06599 · arXiv
- Jia Li Ge Li Yunfei Zhao Yongmin Li Zhi Jin Hao Zhu Huanyu Liu Kaibo Liu Lecheng Wang Zheng Fang Lanshen Wang Jiazheng Ding Xuanming Zhang Yihong Dong Yuqi Zhu Bin Gu and Mengfei Yang. 2024. Deveval: Evaluating code generation in practical software projects. arXiv:2401.06401. Retrieved from https://arxiv.org/abs/2401.06401 · arXiv
- Peng Li Tianxiang Sun Qiong Tang Hang Yan Yuanbin Wu Xuanjing Huang and Xipeng Qiu. 2023. Codeie: Large code generation models are better few-shot information extractors. arXiv:2305.05711. Retrieved from https://arxiv.org/abs/2305.05711 · arXiv
- Tsz-On Li Wenxi Zong Yibo Wang Haoye Tian Ying Wang and Shing-Chi Cheung. 2023. Finding failure-inducing test cases with ChatGPT. arXiv:2304.11686. Retrieved from https://arxiv.org/abs/2304.11686 · arXiv
- Tsz-On Li, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, Shing-Chi Cheung, and Jeff Kramer. 2023. Nuances are the key: Unlocking chatgpt to find failure-inducing tests with differential prompting. In Proceedings of the 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE ’23). IEEE, 14–26.
- Xin-Ye Li Jiang-Tian Xue Zheng Xie and Ming Li. 2023. Think outside the code: Brainstorming boosts large language models in code generation. arXiv:2305.10679. Retrieved from https://arxiv.org/abs/2305.10679 · arXiv
- Yichen Li Yintong Huo Zhihan Jiang Renyi Zhong Pinjia He Yuxin Su and Michael R. Lyu. 2023. Exploring the effectiveness of LLMS in automated logging generation: An empirical study. arXiv:2307.05950. Retrieved from https://arxiv.org/abs/2307.05950 · arXiv
- Youjia Li Jianjun Shi and Zheng Zhang. 2023. A novel approach for rapid development based on chatgpt and prompt engineering. arXiv:2312.13115. Retrieved from https://arxiv.org/abs/2312.13115 · arXiv
- Yao Li Tao Zhang Xiapu Luo Haipeng Cai Sen Fang and Dawei Yuan. 2022. Do pre-trained language models indeed understand software engineering tasks? arXiv:2211.10623. Retrieved from https://arxiv.org/abs/2211.10623 · arXiv
- Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Dong Chen, Shuai Wang, and Cuiyun Gao. 2023. Cctest: Testing and repairing code completion systems. In Proceedings of the 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1238–1250.
- Zongjie Li Chaozheng Wang Zhibo Liu Haoxuan Wang Shuai Wang and Cuiyun Gao. 2022. Cctest: Testing and repairing code completion systems. arXiv:2208.08289. Retrieved from https://arxiv.org/abs/2208.08289 · arXiv
- Jinfeng Lin, Yalin Liu, Qingkai Zeng, Meng Jiang, and Jane Cleland-Huang. 2021. Traceability transformed: Generating more accurate links with pre-trained bert models. In Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE ’21). IEEE, 324–335.
- Yu-Chen Lin Akhilesh Kumar Wen-Liang Zhang Norman Chang Muhammad Zakir Rucha Apte Chao Wang and Jyh-Shing Roger Jang. 2023. Applications of large language models in data processing: Innovative approaches to segmenting and renewing information. arXiv:2311.16267. Retrieved from https://arxiv.org/abs/2311.16267 · arXiv
- Chao Liu Xuanlin Bao Hongyu Zhang Neng Zhang Haibo Hu Xiaohong Zhang and Meng Yan. 2023. Improving ChatGPT prompt for code generation. arXiv:2305.08360. Retrieved from https://arxiv.org/abs/2305.08360 · arXiv
- Jinrun Liu Xinyu Tang Linlin Li Panpan Chen and Yepang Liu. 2023. Which is a better programming assistant? A comparative study between chatgpt and stack overflow. arXiv:2308.13851. Retrieved from https://arxiv.org/abs/2308.13851 · arXiv
- Jiawei Liu Chunqiu Steven Xia Yuyao Wang and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? Rigorous evaluation of large language models for code generation. arXiv:2305.01210. Retrieved from https://arxiv.org/abs/2305.01210 · arXiv
- Puzhuo Liu Chengnian Sun Yaowen Zheng Xuan Feng Chuan Qin Yuncheng Wang Zhi Li and Limin Sun. 2023. Harnessing the power of llm to support binary taint analysis. arXiv:2310.08275. Retrieved from https://arxiv.org/abs/2310.08275 · arXiv
- Shangqing Liu Bozhi Wu Xiaofei Xie Guozhu Meng and Yang Liu. 2023. Contrabert: Enhancing code pre-trained models via contrastive learning. arXiv:2301.09072. Retrieved from https://arxiv.org/abs/2301.09072 · arXiv
- Tianyang Liu Canwen Xu and Julian McAuley. 2023. Repobench: Benchmarking repository-level code auto-completion systems. arXiv:2306.03091. Retrieved from https://arxiv.org/abs/2306.03091 · arXiv
- Yi Liu Gelei Deng Yuekang Li Kailong Wang Tianwei Zhang Yepang Liu Haoyu Wang Yan Zheng and Yang Liu. 2023. Prompt injection attack against LLM-integrated applications. arXiv:2306.05499. Retrieved from https://arxiv.org/abs/2306.05499 · arXiv
- Yue Liu Thanh Le-Cong Ratnadira Widyasari Chakkrit Tantithamthavorn Li Li Xuan-Bach D. Le and David Lo. 2023. Refining ChatGPT-generated code: Characterizing and mitigating code quality issues. arXiv:2307.12596. Retrieved from https://arxiv.org/abs/2307.12596 · arXiv
- Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692. Retrieved from https://arxiv.org/abs/1907.11692 · arXiv
- Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang. 2023. Fill in the blank: Context-aware automated text input generation for mobile GUI testing. In Proceedings of the 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE ’23). IEEE, 1355–1367.
- Zhe Liu Chunyang Chen Junjie Wang Mengzhuo Chen Boyu Wu Xing Che Dandan Wang and Qing Wang. 2023. Testing the limits: Unusual text inputs generation for mobile app crash detection with large language model. arXiv:2310.15657. Retrieved from https://arxiv.org/abs/2310.15657 · arXiv
- Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. 2023. Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning. In Proceedings of the 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE ’23). IEEE, 647–658.
- Dipeeka Luitel Shabnam Hassani and Mehrdad Sabetzadeh. 2023. Improving requirements completeness: Automated assistance through large language models. arXiv:2308.03784. Retrieved from https://arxiv.org/abs/2308.03784 · arXiv
- Ziyang Luo Can Xu Pu Zhao Qingfeng Sun Xiubo Geng Wenxiang Hu Chongyang Tao Jing Ma Qingwei Lin and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv:2306.08568. Retrieved from https://arxiv.org/abs/2306.08568 · arXiv
- Lezhi Ma Shangqing Liu Yi Li Xiaofei Xie and Lei Bu. 2024. SpecGen: Automated generation of formal program specifications via large language models. arXiv:2401.08807. Retrieved from https://arxiv.org/abs/2401.08807 · arXiv
- Wei Ma Shangqing Liu Wenhan Wang Qiang Hu Ye Liu Cen Zhang Liming Nie and Yang Liu. 2023. The scope of chatgpt in software engineering: A thorough investigation. arXiv:2305.12138. Retrieved from https://arxiv.org/abs/2305.12138 · arXiv
- Shantanu Mandal Adhrik Chethan Vahid Janfaza S. M. Mahmud Todd A Anderson Javier Turek Jesmin Jahan Tithi and Abdullah Muzahid. 2023. Large language models based automatic synthesis of software specifications. arXiv:2304.09181. Retrieved from https://arxiv.org/abs/2304.09181 · arXiv
- Antonio Mastropaolo, Massimiliano Di Penta, and Gabriele Bavota. 2023. Towards automatically addressing self-admitted technical debt: How far are we?. In Proceedings of the 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE ’23). IEEE, 585–597.
- Mohammad Mahdi Mohajer Reem Aleithan Nima Shiri Harzevili Moshi Wei Alvine Boaye Belle Hung Viet Pham and Song Wang. 2023. Skipanalyzer: An embodied agent for code analysis with large language models. arXiv:2310.18532. Retrieved from https://arxiv.org/abs/2310.18532 · arXiv
- Seungjun Moon Yongho Song Hyungjoo Chae Dongjin Kang Taeyoon Kwon Kai Tzu-iunn Ong Seung-won Hwang and Jinyoung Yeo. 2023. Coffee: Boost your code LLMS by fixing bugs with feedback. arXiv:2311.07215. Retrieved from https://arxiv.org/abs/2311.07215 · arXiv
- 10.5555/1858842.1858883 · doi:10.5555/1858842.1858883
- Fangwen Mu Lin Shi Song Wang Zhuohao Yu Binquan Zhang Chenxue Wang Shichao Liu and Qing Wang. 2023. ClarifyGPT: Empowering LLM-based code generation with intention clarification. arXiv:2310.10996. Retrieved from https://arxiv.org/abs/2310.10996 · arXiv
- Manisha Mukherjee and Vincent J Hellendoorn. 2023. Stack over-flowing with results: The case for domain-specific pre-training over one-size-fits-all models. arXiv:2306.03268. Retrieved from https://arxiv.org/abs/2306.03268 · arXiv
- Daye Nam Andrew Macvean Vincent Hellendoorn Bogdan Vasilescu and Brad Myers. 2023. In-IDE generation-based information support with a large language model. arXiv:2307.08177. Retrieved from https://arxiv.org/abs/2307.08177 · arXiv
- Nathalia Nascimento Paulo Alencar and Donald Cowan. 2023. Comparing software developers with ChatGPT: An empirical investigation. arXiv:2305.11837. Retrieved from https://arxiv.org/abs/2305.11837 · arXiv
- Phuong T. Nguyen Juri Di Rocco Claudio Di Sipio Riccardo Rubei Davide Di Ruscio and Massimiliano Di Penta. 2023. Is this snippet written by chatgpt? An empirical study with a codebert-based classifier. arXiv:2307.09381. Retrieved from https://arxiv.org/abs/2307.09381 · arXiv
- Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In Proceedings of the International Conference on Machine Learning. PMLR, 26106–26128.
- Ansong Ni Pengcheng Yin Yilun Zhao Martin Riddell Troy Feng Rui Shen Stephen Yin Ye Liu Semih Yavuz Caiming Xiong Shafiq Joty Yingbo Zhou Dragomir Radev and Arman Cohan. 2023. L2ceval: Evaluating language-to-code generation capabilities of large language models. arXiv:2309.17446. Retrieved from https://arxiv.org/abs/2309.17446 · arXiv
- Erik Nijkamp Hiroaki Hayashi Caiming Xiong Silvio Savarese and Yingbo Zhou. 2023. CodeGen2: Lessons for training llms on programming and natural languages. arXiv:2305.02309. Retrieved from https://arxiv.org/abs/2305.02309 · arXiv
- Erik Nijkamp Bo Pang Hiroaki Hayashi Lifu Tu Huan Wang Yingbo Zhou Silvio Savarese and Caiming Xiong. 2022. CodeGen: An open large language model for code with multi-turn program synthesis. arXiv:2203.13474. Retrieved from https://arxiv.org/abs/2203.13474 · arXiv
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. CodeGen: An open large language model for code with multi-turn program synthesis. In Proceedings of the International Conference on Learning Representations. Retrieved from https://api.semanticscholar.org/CorpusID:252668917
- David Noever. 2023. Can large language models find and fix vulnerable software? arXiv:2308.10345. Retrieved from https://arxiv.org/abs/2308.10345 · arXiv
- Marcel Ochs, Krishna Narasimhan, and Mira Mezini. 2023. Evaluating and improving transformers pre-trained on ASTS for code completion. In Proceedings of the 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’16). IEEE, 834–844.
- OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https://arxiv.org/abs/2303.08774 · arXiv
- Shuyin Ouyang Jie M. Zhang Mark Harman and Meng Wang. 2023. Llm is like a box of chocolates: The non-determinism of chatgpt in code generation. arXiv:2308.02828. Retrieved from https://arxiv.org/abs/2308.02828 · arXiv
- Jialing Pan Adrien Sadé Jin Kim Eric Soriano Guillem Sole and Sylvain Flamant. 2023. Stelocoder: A decoder-only LLM for multi-language to python code translation. arXiv:2310.15539. Retrieved from https://arxiv.org/abs/2310.15539 · arXiv
- Rangeet Pan Ali Reza Ibrahimzada Rahul Krishna Divya Sankar Lambert Pouguem Wassi Michele Merler Boris Sobolev Raju Pavuluri Saurabh Sinha and Reyhaneh Jabbarvand. 2023. Understanding the effectiveness of large language models in code translation. arXiv:2308.03109. Retrieved from https://arxiv.org/abs/2308.03109 · arXiv
- Bhargavi Paranjape Scott Lundberg Sameer Singh Hannaneh Hajishirzi Luke Zettlemoyer and Marco Tulio Ribeiro. 2023. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv:2303.09014. Retrieved from https://arxiv.org/abs/2303.09014 · arXiv
- Emilio Parisotto Abdel-rahman Mohamed Rishabh Singh Lihong Li Dengyong Zhou and Pushmeet Kohli. 2016. Neuro-symbolic program synthesis. arXiv:1611.01855. Retrieved from https://arxiv.org/abs/1611.01855 · arXiv
- Arkil Patel Siva Reddy Dzmitry Bahdanau and Pradeep Dasigi. 2023. Evaluating in-context learning of libraries for code generation. arXiv:2311.09635. Retrieved from https://arxiv.org/abs/2311.09635 · arXiv
- Rishov Paul Md Mohib Hossain Masum Hasan and Anindya Iqbal. 2023. Automated program repair based on code review: How do pre-trained transformer models perform? arXiv:2304.07840. Retrieved from https://arxiv.org/abs/2304.07840 · arXiv
- Rishov Paul Md. Mohib Hossain Mohammed Latif Siddiq Masum Hasan Anindya Iqbal and Joanna C. S. Santos. 2023. Enhancing automated program repair through fine-tuning and prompt engineering. arXiv:2304.07840. Retrieved from https://arxiv.org/abs/2304.07840 · arXiv
- Tommaso Pegolotti Elias Frantar Dan Alistarh and Markus Püschel. 2023. QIGen: Generating efficient kernels for quantized inference on large language models. arXiv:2307.03738. Retrieved from https://arxiv.org/abs/2307.03738 · arXiv
- Kexin Pei, David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2023. Can large language models reason about program invariants? In Proceedings of the 40th International Conference on Machine Learning (ICML’23). Vol. 202, 27496–27520.
- Sanyogita Piya and Allison Sullivan. 2023. LLM4TDD: Best practices for test driven development using large language models. arXiv:2312.04687. Retrieved from https://arxiv.org/abs/2312.04687 · arXiv
- Amrit Poudel Jinfeng Lin and Jane Cleland-Huang. 2023. Leveraging transformer-based language models to automate requirements satisfaction assessment. arXiv:2312.04463. Retrieved from https://arxiv.org/abs/2312.04463 · arXiv
- Rohith Pudari and Neil A. Ernst. 2023. From copilot to pilot: Towards AI supported software development. arXiv:2303.04142. Retrieved from https://arxiv.org/abs/2303.04142 · arXiv
- Mengnan Qi Yufan Huang Maoquan Wang Yongqiang Yao Zihan Liu Bin Gu Colin Clement and Neel Sundaresan. 2023. Sut: Active defects probing for transcompiler models. arXiv:2310.14209. Retrieved from https://arxiv.org/abs/2310.14209 · arXiv
- Vu Le Anh Quan Chau Thuan Phat Kiet Van Nguyen Phan The Duy and Van-Hau Pham. 2023. XGV-BERT: Leveraging contextualized language model and graph neural network for efficient software vulnerability detection. arXiv:2309.14677. Retrieved from https://arxiv.org/abs/2309.14677 · arXiv
- Alec Radford and Karthik Narasimhan. 2018. Improving language understanding by generative pre-training. Retrieved from https://api.semanticscholar.org/CorpusID:49313245
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- 10.5555/3455716.3455856 · doi:10.5555/3455716.3455856
- Sajjad Rahmani AmirHossein Naghshzan and Latifa Guerrouj. 2023. Improving code example recommendations on informal documentation using bert and query-aware LSH: A comparative study. arXiv:2305.03017. Retrieved from https://arxiv.org/abs/2305.03017 · arXiv
- Abhinav Rao Sachin Vashistha Atharva Naik Somak Aditya and Monojit Choudhury. 2023. Tricking llms into disobedience: Understanding analyzing and preventing jailbreaks. arXiv:2305.14965. Retrieved from https://arxiv.org/abs/2305.14965 · arXiv
- Nikitha Rao Jason Tsay Kiran Kate Vincent J. Hellendoorn and Martin Hirzel. 2023. AI for low-code for AI. arXiv:2305.20015. Retrieved from https://arxiv.org/abs/2305.20015 · arXiv
- Xiaoxue Ren, Xinyuan Ye, Dehai Zhao, Zhenchang Xing, and Xiaohu Yang. 2023. From misuse to mastery: Enhancing code generation with knowledge-driven ai chaining. In Proceedings of the 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE ’23). IEEE, 976–987.
- Tal Ridnik Dedy Kredo and Itamar Friedman. 2024. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv:2401.08500. Retrieved from https://arxiv.org/abs/2401.08500 · arXiv
- Baptiste Roziere Marie-Anne Lachaux Marc Szafraniec and Guillaume Lample. 2021. DOBF: A deobfuscation pre-training objective for programming languages. arXiv:2102.07492. Retrieved from https://arxiv.org/abs/2102.07492 · arXiv
- Fernando Vallecillos Ruiz Anastasiia Grishina Max Hort and Leon Moonen. 2024. A novel approach for automatic program repair using round-trip translation with large language models. arXiv:2401.07994. Retrieved from https://arxiv.org/abs/2401.07994 · arXiv
- Iman Saberi Fatemeh Fard and Fuxiang Chen. 2023. Multilingual adapter-based knowledge aggregation on code summarization for low-resource languages. arXiv:2307.07854. Retrieved from https://arxiv.org/abs/2307.07854 · arXiv
- Ahmed Sadik Antonello Ceravola Frank Joublin and Jibesh Patra. 2023. Analysis of ChatGPT on source code. arXiv:2306.00597. Retrieved from https://arxiv.org/abs/2306.00597 · arXiv
- Anthony Saieva Saikat Chakraborty and Gail Kaiser. 2023. On contrastive learning of semantic similarity forcode to code search. arXiv:2305.03843. Retrieved from https://arxiv.org/abs/2305.03843 · arXiv
- Fardin Ahsan Sakib Saadat Hasan Khan and A. H. M. Karim. 2023. Extending the frontier of ChatGPT: Code generation and debugging. arXiv:2307.08260. Retrieved from https://arxiv.org/abs/2307.08260 · arXiv
- Teven Le Scao Angela Fan Christopher Akiki Ellie Pavlick Suzana Ilić Daniel Hesslow Roman Castagné Alexandra Sasha Luccioni François Yvon Matthias Gallé Jonathan Tow Alexander M. Rush Stella Biderman Albert Webson Pawan Sasanka Ammanamanchi Thomas Wang Benoît Sagot Niklas Muennighoff Albert Villanova del Moral Olatunji Ruwase Rachel Bawden Stas Bekman Angelina McMillan-Major Iz Beltagy Huu Nguyen L ucile Saulnier Samson Tan Pedro Ortiz Suarez Victor Sanh Hugo Laurençon Yacine Jernite Julien Launay Margaret Mitchell Colin Raffel Aaron Gokaslan Adi Simhi Aitor Soroa Alham Fikri Aji Amit Alfassy Anna Rogers Ariel Kreisberg Nitzav Canwen Xu Chenghao Mou Chris Emezue Christopher Klamm Colin Leong Daniel van Strien David Ifeoluwa Adelani Dragomir Radev Eduardo González Ponferrada Efrat Levkovizh Ethan Kim Eyal Bar Natan Francesco De Toni Gérard Dupont Germán Kruszewski Giada Pistilli Hady Elsahar Hamza Benyamina Hieu Tran Ian Yu Idris Abdulmumin Isaac Johnson Itziar Gonzalez-Dios Javier de la Rosa Jenny Chim Jesse Dodge Jian Zhu Jonathan Chang Jörg Frohberg Joseph Tobing Joydeep Bhattacharjee Khalid Almubarak Kimbo Chen Kyle Lo Leandro Von Werra Leon Weber Long Phan Loubna Ben allal Ludovic Tanguy Manan Dey Manuel Romero Muñoz Maraim Masoud María Grandury Mario Šaško Max Huang Maximin Coavoux Mayank Singh Mike Tian-Jian Jiang Minh Chien Vu Mohammad A. Jauhar Mustafa Ghaleb Nishant Subramani Nora Kassner Nurulaqilla Khamis Olivier Nguyen Omar Espejel Ona de Gibert Paulo Villegas et al. 2022. BLOOM: A 176B-parameter open-access multilingual language model. arXiv:2211.05100. Retrieved from https://arxiv.org/abs/2211.05100 · arXiv
- Imanol Schlag Sainbayar Sukhbaatar Asli Celikyilmaz Wen tau Yih Jason Weston Jürgen Schmidhuber and Xian Li. 2023. Large language model programs. arXiv:2305.05364. Retrieved from https://arxiv.org/abs/2305.05364 · arXiv
- Martin Schroder. 2023. AutoScrum: Automating project planning using large language models. arXiv:2306.03197. Retrieved from https://arxiv.org/abs/2306.03197 · arXiv
- Oussama Ben Sghaier and Houari Sahraoui. 2023. A multi-step learning approach to assist code review. In Proceedings of the 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’23). IEEE, 450–460.
- Murray Shanahan. 2022. Talking about large language models. arXiv:2212.03551. Retrieved from https://arxiv.org/abs/2212.03551 · arXiv
- Anton Shapkin Denis Litvinov and Timofey Bryksin. 2023. Entity-augmented code generation. arXiv:2312.08976. Retrieved from https://arxiv.org/abs/2312.08976 · arXiv
- Alexey Shestov Anton Cheshkov Rodion Levichev Ravil Mussabayev Pavel Zadorozhny Evgeny Maslov Chibirev Vadim and Egor Bulychev. 2024. Finetuning large language models for vulnerability detection. arXiv:2401.17010. Retrieved from https://arxiv.org/abs/2401.17010 · arXiv
- Ensheng Shi Yanlin Wang Hongyu Zhang Lun Du Shi Han Dongmei Zhang and Hongbin Sun. 2023. Towards efficient fine-tuning of pre-trained code models: An experimental study and beyond. arXiv:2304.05216. Retrieved from https://arxiv.org/abs/2304.05216 · arXiv
- Ensheng Shi Fengji Zhang Yanlin Wang Bei Chen Lun Du Hongyu Zhang Shi Han Dongmei Zhang and Hongbin Sun. 2023. SoTaNa: The open-source software development assistant. arXiv:2308.13416. Retrieved from https://arxiv.org/abs/2308.13416 · arXiv
- Zejian Shi, Yun Xiong, Xiaolong Zhang, Yao Zhang, Shanshan Li, and Yangyong Zhu. 2022. Cross-modal contrastive learning for code search. In Proceedings of the 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME ’22). IEEE, 94–105.
- Jiho Shin Sepehr Hashtroudi Hadi Hemmati and Song Wang. 2023. Domain adaptation for deep unit test case generation. arXiv:2308.08033. Retrieved from https://arxiv.org/abs/2308.08033 · arXiv
- Jiho Shin Clark Tang Tahmineh Mohati Maleknaz Nayebi Song Wang and Hadi Hemmati. 2023. Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks. arXiv:2310.10508. Retrieved from https://arxiv.org/abs/2310.10508 · arXiv
- Atsushi Shirafuji Yutaka Watanobe Takumi Ito Makoto Morishita Yuki Nakamura Yusuke Oda and Jun Suzuki. 2023. Exploring the robustness of large language models for solving programming problems. arXiv:2306.14583. Retrieved from https://arxiv.org/abs/2306.14583 · arXiv
- Alexander Shypula Aman Madaan Yimeng Zeng Uri Alon Jacob Gardner Milad Hashemi Graham Neubig Parthasarathy Ranganathan Osbert Bastani and Amir Yazdanbakhsh. 2023. Learning performance-improving code edits. arXiv:2302.07867. Retrieved from https://arxiv.org/abs/2302.07867 · arXiv
- Mohammed Latif Siddiq Beatrice Casey and Joanna Santos. 2023. A lightweight framework for high-quality code generation. arXiv:2307.08220. Retrieved from https://arxiv.org/abs/2307.08220 · arXiv
- Mohammed Latif Siddiq Joanna Santos Ridwanul Hasan Tanvir Noshin Ulfat Fahmid Al Rifat and Vinicius Carvalho Lopes. 2023. Exploring the effectiveness of large language models in generating unit tests. arXiv:2305.00418. Retrieved from https://arxiv.org/abs/2305.00418 · arXiv
- André Silva Sen Fang and Martin Monperrus. 2023. Repairllama: Efficient representations and fine-tuned adapters for program repair. arXiv:2312.15698. Retrieved from https://arxiv.org/abs/2312.15698 · arXiv
- Benjamin Steenhoek Michele Tufano Neel Sundaresan and Alexey Svyatkovskiy. 2023. Reinforcement learning from automatic feedback for high-quality unit test generation. arXiv:2310.02368. Retrieved from https://arxiv.org/abs/2310.02368 · arXiv
- Hongjin Su Jungo Kasai Chen Henry Wu Weijia Shi Tianlu Wang Jiayi Xin Rui Zhang Mari Ostendorf Luke Zettlemoyer Noah A. Smith and Tao Yu. 2022. Selective annotation makes language models better few-shot learners. arXiv:2209.01975. Retrieved from https://arxiv.org/abs/2209.01975 · arXiv
- Jiamou Sun Zhenchang Xing Qinghua Lu Xiwei Xu Liming Zhu Thong Hoang and Dehai Zhao. 2023. Silent vulnerable dependency alert prediction with vulnerability key aspect explanation. arXiv:2302.07445. Retrieved from https://arxiv.org/abs/2302.07445 · arXiv
- Weisong Sun Chunrong Fang Yudu You Yuchen Chen Yi Liu Chong Wang Jian Zhang Quanjun Zhang Hanwei Qian Wei Zhao Yang Liu and Zhenyu Chen. 2023. A prompt learning framework for source code summarization. arXiv:2312.16066. Retrieved from https://arxiv.org/abs/2312.16066 · arXiv
- Weisong Sun Chunrong Fang Yudu You Yun Miao Yi Liu Yuekang Li Gelei Deng Shenghan Huang Yuchen Chen Quanjun Zhang Hanwei Qian Yang Liu and Zhenyu Chen. 2023. Automatic code summarization via ChatGPT: How far are we? arXiv:2305.12865. Retrieved from https://arxiv.org/abs/2305.12865 · arXiv
- Yuqiang Sun Daoyuan Wu Yue Xue Han Liu Wei Ma Lyuye Zhang Miaolei Shi and Yang Liu. 2024. LLM4Vuln: A unified evaluation framework for decoupling and enhancing LLMS’ vulnerability reasoning. arXiv:2401.16185. Retrieved from https://arxiv.org/abs/2401.16185 · arXiv
- Yuqiang Sun Daoyuan Wu Yue Xue Han Liu Haijun Wang Zhengzi Xu Xiaofei Xie and Yang Liu. 2023. When GPT meets program analysis: Towards intelligent detection of smart contract logic vulnerabilities in gptscan. arXiv:2308.03314. Retrieved from https://arxiv.org/abs/2308.03314 · arXiv
- Zhensu Sun Xiaoning Du Fu Song Shangwen Wang and Li Li. 2024. When neural code completion models size up the situation: Attaining cheaper and faster completion through dynamic model inference. arXiv:2401.09964. Retrieved from https://arxiv.org/abs/2401.09964 · arXiv
- Chee Wei Tan Shangxin Guo Man Fai Wong and Ching Nam Hang. 2023. Copilot for xcode: Exploring AI-assisted programming by prompting cloud-based large language models. arXiv:2307.14349. Retrieved from https://arxiv.org/abs/2307.14349 · arXiv
200 of 271 shown
Cited by
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“For example, Hou et al. [14] provided an excellent recent SLR covering 229 research papers from 2017 to 2023 reporting SE tasks tackled, data collection and preprocessing techniques, and strategies for optimising LLM performance (such as prompt engineering).”
BackgroundSoftware Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
Large Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.08715 · Cited by 418
Self-citation
Metadata
- Authors
- Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H.
- Affiliations
- Huazhong University of Science and Technology (CN)
- Monash University (AU)
- Singapore Management University (SG)
- Beihang University (CN)
- Hong Kong Polytechnic University (HK)
- ORCID
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2024-11-30
- Volume
- 33
- Issue
- 8
- Pages
- 1-79
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4402665833
- Cited by
- 1,029
- References
- 321
- Citations by year
- 2026: 469 · 2025: 417 · 2024: 135
- FWCI
- 58.78
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
- Funders
- National Natural Science Foundation of China (62072046)
- Key R&D Program of Hubei (2023BAB017, 2023BAB079)
- National Research Foundation, under its Investigatorship (NRF-NRFI08-2022-0002)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, software engineering tasks, LLM evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.