An Empirical Study of Code Smells in Transformer-based Code Generation Techniques

Siddiq, Mohammed Latif; Majumder, Shafayat H.; Mim, Maisha R.; Jajodia, Sourov; Santos, Joanna C. S.

doi:10.1109/scam55253.2022.00014

Cited by 72Software Engineering ResearchSoftware Testing and Debugging TechniquesSoftware Engineering Techniques and Practices

An Empirical Study of Code Smells in Transformer-based Code Generation Techniques

Siddiq, Mohammed Latif, Majumder, Shafayat H., Mim, Maisha R., Jajodia, Sourov, Santos, Joanna C. S.·2022·2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM)

DOI 10.1109/scam55253.2022.00014

Open this paper in CoChat — free account

Abstract

Prior works have developed transformer-based language learning models to automatically generate source code for a task without compilation errors. The datasets used to train these techniques include samples from open source projects which may not be free of security flaws, code smells, and violations of standard coding practices. Therefore, we investigate to what extent code smells are present in the datasets of coding generation techniques and verify whether they leak into the output of these techniques. To conduct this study, we used Pylint and Bandit to detect code smells and security smells in three widely used training sets (CodeXGlue, APPS, and Code Clippy). We observed that Pylint caught 264 code smell types, whereas Bandit located 44 security smell types in these three datasets used for training code generation techniques. By analyzing the output from ten different configurations of the open-source fine-tuned transformer-based GPT-Neo 125M parameters model, we observed that this model leaked the smells and non-standard practices to the generated source code. When analyzing GitHub Copilot's suggestions, a closed source code generation tool, we observed that it contained 18 types of code smells, including substandard coding patterns and 2 security smell types.

Integrity signals

Registry

Crossref · 10.1109/scam55253.2022.00014

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Siddiq et al. showed that for the HumanEval dataset, GitHub Copilot emits certain CWEs in around 2% of cases as measured by Bandit.
“Likewise, Siddiq et al. [7] showed that for the HumanEval dataset (which examine functional capabilities and not security) GitHub Copilot emits certain CWEs in around 2 % of cases as measured by Bandit analysis tool [26].”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [7]
LLM completions may contain critical security vulnerabilities.
“recent work has shown that LLM completions may contain critical security vulnerabilities [7, 8]”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [7]
Literature [7,8] suggests that LLMs should be used with care due to their habit of suggesting vulnerable patterns.
“This is contrary to literature [7, 8] which suggests that LLMs should be used with care due to their habit of suggesting vulnerable patterns.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [7]
This is somewhat surprising given existing published studies on how vulnerable code can be suggested by LLMs.
“This is somewhat surprising given the existing published studies on how vulnerable code can be suggested by the LLMs [7,8].”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [7]
Siddiq et al. (2022) found that GitHub Copilot produced code with similar stylistic issues.
“This finding is in line with a study by Siddiq et al. ( 2022), where they found that GitHub Copilot produced code with similar stylistic issues.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Siddiq et al. (2022)

Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
MizAR 60 for Mizar 50
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Refactoring: Improving the Design of Existing Code
Fowler, M. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45672-4_31 · Cited by 5,615 · see citing works
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · see citing works
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · see citing works
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · see citing works
A Syntactic Neural Model for General-Purpose Code Generation
Yin, P., & Neubig, G. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1041 · Cited by 623 · see citing works
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · see citing works
Object-Oriented Metrics in Practice
Lanza, M., & Marinescu, R. (2006) · Springer eBooks · doi:10.1007/3-540-39538-5 · Cited by 529 · see citing works
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · see citing works
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483 · see citing works
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476 · see citing works
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 1× · see citing works
“These Large Language Learning Models (LLMs) are trained with a large dataset of source code snippets and fine-tuned for a wide range of software engineering applications, such as automated code completion [3]–[5], summarization [6], documentation [7], and generation [8], [9].”
Background
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · see citing works
Program Synthesis
Gulwani, S., Polozov, O., & Singh, R. (2017) · Foundations and Trends® in Programming Languages · doi:10.1561/2500000010 · Cited by 357 · see citing works
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324 · see citing works
On the diffuseness and the impact on maintainability of code smells: a large scale empirical investigation
Palomba, F., Bavota, G., Penta, M. D., Fasano, F., Oliveto, R., & Lucia, A. D. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9535-z · Cited by 307 · see citing works
Detecting code smells using machine learning techniques: Are we there yet?
Nucci, D. D., Palomba, F., Tamburri, D. A., Serebrenik, A., & Lucia, A. D. (2018) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2018.8330266 · Cited by 305 · see citing works
Application of Theorem Proving to Problem Solving
Green, C. (1969) · doi:10.21236/ada459656 · Cited by 290 · see citing works
Toward automatic program synthesis
Manna, Z., & Waldinger, R. J. (1971) · Communications of the ACM · doi:10.1145/362566.362568 · Cited by 285 · see citing works
An Empirical Study of Problem-Aligned Multilingual Code Generation and Directed Code Translation by Large Language Models
Wang, Q., Wen, C., Liu, D., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/oasics.icpec.2023.9 · Cited by 283 · see citing works
The Seven Sins: Security Smells in Infrastructure as Code Scripts
Rahman, A., Parnin, C., & Williams, L. A. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00033 · Cited by 234 · see citing works
Code Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209 · see citing works
TreeGen: A Tree-Based Transformer Architecture for Code Generation
Sun, Z., Zhu, Q., Xiong, Y., Sun, Y., Mou, L., & Zhang, L. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6430 · Cited by 172 · see citing works
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · see citing works
A Cooperative Parallel Search-Based Software Engineering Approach for Code-Smells Detection
Kessentini, W., Kessentini, M., Sahraoui, H. A., Bechikh, S., & Ouni, A. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2331057 · Cited by 132 · see citing works
Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code?
Asare, O., Nagappan, M., & Asokan, N. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10380-1 · Cited by 128 · see citing works
A parallel corpus of Python functions and documentation strings for automated code documentation and code generationPreprint
Barone, A. V. M., & Sennrich, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02275 · Cited by 103 · see citing works
Fast and Memory-Efficient Neural Code Completion
Svyatkovskiy, A., Lee, S., Hadjitofi, A., Riechert, M., Franco, J. V., & Allamanis, M. (2021) · 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00045 · Cited by 95 · see citing works
On the evaluation of code smells and detection tools
Paiva, T., Damasceno, A., Figueiredo, E., & Sant’Anna, C. (2017) · Journal of Software Engineering Research and Development · doi:10.1186/s40411-017-0041-1 · Cited by 91 · see citing works
CodeFill: multi-token code completion by jointly learning from structure and naming sequences
Izadi, M., Gismondi, R., & Gousios, G. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510172 · Cited by 86 · see citing works
StructCoder: Structure-Aware Transformer for Code Generation
Tipirneni, S., Zhu, M., & Reddy, C. K. A. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3636430 · Cited by 71 · see citing works
Code Smells Detection and Visualization: A Systematic Literature Review
dos Reis, J. P., Abreu, F. B. E., Carneiro, G. D. F., & Anslow, C. (2021) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-021-09566-x · Cited by 60 · see citing works
The Prevalence of Code Smells in Machine Learning projects
van Oort, B., Cruz, L., Aniche, M. F., & Arie Van Deursen, A. (2021) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/wain52551.2021.00011 · Cited by 55 · see citing works
Machine Learning Techniques for Code Smells Detection: A Systematic Mapping Study
Caram, F. L., Rodrigues, B. R. D. O., Campanelli, A. S., & Parreiras, F. S. (2019) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s021819401950013x · Cited by 54 · see citing works
Understanding metric-based detectable smells in Python software: A comparative study
Chen, Z., Chen, L., Ma, W., Zhou, X., Zhou, Y., & Xu, B. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.09.011 · Cited by 47 · see citing works
Security Smells in Android
Ghafari, M., Gadient, P., & Nierstrasz, O. (2017) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2017.24 · Cited by 47 · see citing works
Detecting Code Smells in Python Programs
Chen, Z., Chen, L., Ma, W., & Xu, B. (2016) · International Conference on Software Analysis, Testing and Evolution (SATE) · doi:10.1109/sate.2016.10 · Cited by 43 · see citing works
M2TS
Gao, Y., & Lyu, C. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527907 · Cited by 43 · see citing works
Share, But be Aware: Security Smells in Python Gists
Rahman, M. R., Rahman, A., & Williams, L. A. (2019) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2019.00087 · Cited by 30 · see citing works
House of Cards: Code Smells in Open-Source C# Repositories
Sharma, T., Fragkoulis, M., & Spinellis, D. D. (2017) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2017.57 · Cited by 27 · see citing works
A large-scale comparative analysis of Coding Standard conformance in Open-Source Data Science projects
Simmons, A., Barnett, S., Rivera‐Villicana, J., Bajaj, A., & Vasa, R. (2020) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3382494.3410680 · Cited by 26 · see citing works
A Study of Different Coding Styles Affecting Code Readability
Lee, T., Lee, J. B., & In, H. P. (2013) · International Journal of Software Engineering and Its Applications · doi:10.14257/ijseia.2013.7.5.36 · Cited by 25 · see citing works
The Adherence of Open Source Java Programmers to Standard Coding Practices
Elish, M. O., & Offutt, J. (2002) · Cited by 20 · see citing works
StructCoder: Structure-Aware Transformer for Code GenerationPreprint
Tipirneni, S., Zhu, M. F., & Reddy, C. K. A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.05239 · Cited by 7 · see citing works
Code Quality: Examining the Efficacy of Automated Tools
Dasgupta, S., & Hooshangi, S. (2017) · Journal of the Association for Information Systems · Cited by 6 · see citing works
Code-Quality Tools
(2008) · Apress eBooks · doi:10.1007/978-1-4302-0439-8_11 · Cited by 2 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805

Flagged first, then by citations

Sorted by citations

Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643674 · Cited by 149 · see citing works
No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT
Liu, Z., Tang, Y., Luo, X., Zhou, Y., & Zhang, L. F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3392499 · Cited by 128 · see citing works
A Review on Code Generation with LLMs: Application and Evaluation
Wang, J., & Chen, Y. (2023) · IEEE International Conference on Medical Artificial Intelligence (MedAI) · doi:10.1109/medai59581.2023.00044 · Cited by 122 · see citing works
Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?
Savelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588792 · Cited by 109 · see citing works
Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study
Fu, Y., Liang, P., Tahir, A., Li, Z., Shahin, M., Yu, J., & Chen, J. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3716848 · Cited by 75 · see citing works
Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation
Zhong, L., & Wang, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30185 · Cited by 96 · see citing works
A systematic literature review on the impact of AI models on the security of code generation
Negri-Ribalta, C., Géraud, R., Sergeeva, A., & Lenzini, G. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1386720 · Cited by 67 · see citing works
Extending the Frontier of ChatGPT: Code Generation and Debugging
Sakib, F. A., Khan, S. H., & Karim, A. (2024) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet61485.2024.10698405 · Cited by 63 · see citing works
Does ChatGPT Generate Accessible Code? Investigating Accessibility Challenges in LLM-Generated Source Code
Aljedaani, W., Habib, A., Aljohani, A., Eler, M., & Feng, Y. (2024) · Proceedings of the 21st International Web for All Conference · doi:10.1145/3677846.3677854 · Cited by 42 · see citing works
CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models
Hajipour, H., Haßler, K., Holz, T., Schönherr, L., & Fritz, M. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00040 · Cited by 51 · see citing works
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants
Sandoval, G., Pearce, H., Nys, T., Karri, R., Garg, S., & Dolan-Gavitt, B. (2022) · arXiv · doi:10.48550/arxiv.2208.09727 · Cited by 52 · see citing works
Automating the correctness assessment of AI-generated code for security contexts
Cotroneo, D., Foggia, A., Improta, C., Liguori, P., & Natella, R. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112113 · Cited by 34 · see citing works
Just another copy and paste? Comparing the security vulnerabilities of ChatGPT generated code and StackOverflow answers
Hamer, S., d’Amorim, M., & Williams, L. A. (2024) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw63631.2024.00014 · Cited by 37 · see citing works
PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models (LLMs)
Nazzal, M., Khalil, I. M., Khreishah, A. A., & Phan, N. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690298 · Cited by 33 · see citing works
Exploring the problems, their causes and solutions of AI pair programming: A study on GitHub and Stack Overflow
Zhou, X., Liang, P., Zhang, B., Li, Z., Ahmad, A., Shahin, M., & Waseem, M. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112204 · Cited by 30 · see citing works
CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
Peng, J., Cui, L., Huang, K., Yang, J., & Ray, B. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00009 · Cited by 29 · see citing works
CS1 with a Side of AI: Teaching Software Verification for Secure Code in the Era of Generative AI
Fernandez, A. S., & Cornell, K. A. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630817 · Cited by 26 · see citing works
Methodology for Code Synthesis Evaluation of LLMs Presented by a Case Study of ChatGPT and Copilot
Ságodi, Z., Siket, I., & Ferenć, R. (2024) · IEEE Access · doi:10.1109/access.2024.3403858 · Cited by 23 · see citing works
An Investigation into Misuse of Java Security APIs by Large Language Models
Mousavi, Z., Islam, C., Moore, K., Abuadbba, A., & Babar, M. A. (2024) · ACM Asia Conference on Computer and Communications Security · doi:10.1145/3634737.3661134 · Cited by 34 · see citing works
AI-Assisted Programming Tasks Using Code Embeddings and Transformers
Kotsiantis, S. B., Verykios, V. S., & Tzagarakis, M. M. (2024) · Electronics · doi:10.3390/electronics13040767 · Cited by 24 · see citing works
Gotcha! This Model Uses My Code! Evaluating Membership Leakage Risks in Code Models
Yang, Z., Zhao, Z., Wang, C., Shi, J., Kim, D., Han, D., & Lo, D. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3482719 · Cited by 21 · see citing works
Data preparation for Deep Learning based Code Smell Detection: A systematic literature review
Zhang, F., Zhang, Z., Keung, J., Tang, X., Yang, Z., Yu, X., & Hu, W. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112131 · Cited by 15 · see citing works
Exploring the Impact of Generative AI for StandUp Report Recommendations in Software Capstone Project Development
Neyem, A., Sandoval Alcocer, J. P., Mendoza, M., Centellas-Claros, L., Gonzalez, L. A., & Paredes-Robles, C. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630854 · Cited by 14 · see citing works
LLM-based methods for the creation of unit tests in game development
Păduraru, C., Staicu, A.-N., & Ştefănescu, A. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.473 · Cited by 14 · see citing works
Franc: A Lightweight Framework for High-Quality Code Generation
Siddiq, M. L., Casey, B., & Santos, J. C. S. (2024) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam63643.2024.00020 · Cited by 27 · see citing works
Self-citation
Quality In, Quality Out: Investigating Training Data's Role in AI Code Generation
Improta, C., Tufano, R., Liguori, P., Cotroneo, D., & Bavota, G. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00056 · Cited by 11 · see citing works
AI-Powered Multi-Agent Framework for Automated Unit Test Case Generation: Enhancing Software Quality through LLM’s
Garlapati, A., Parmesh, M. N. V. S. S. M., Savitha, & Jaisri, S. (2024) · IEEE Global Conference for Advancement in Technology (GCAT) · doi:10.1109/gcat62922.2024.10923987 · Cited by 10 · see citing works
Vulnerability Handling of AI-Generated Code - Existing Solutions and Open Challenges
Kaniewski, S., Holstein, D., Schmidt, F., & Heer, T. (2024) · Conference on AI, Science, Engineering, and Technology (AIxSET) · doi:10.1109/aixset62544.2024.00026 · Cited by 9 · see citing works
State of the Art of the Security of Code Generated by LLMs: A Systematic Literature Review
Ramírez, L. C., Limón, X., Sánchez-García, Á. J., & Pérez-Arriaga, J. C. (2024) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft63288.2024.00050 · Cited by 9 · see citing works
PwnPilot: Reflections on Trusting Trust in the Age of Large Language Models and AI Code Assistants
Horne, D. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00396 · Cited by 9 · see citing works
Analysis and Comparison of Automatic Code Generation and Transformation Techniques on Low-Code Platforms
Uyanık, B., & Sayar, A. (2023) · International Conference on Software Engineering and Development (ICSED) · doi:10.1145/3637792.3637795 · Cited by 8 · see citing works
Do Code LLMs Understand Design Patterns?
Pan, Z., Song, X., Wang, Y., Cao, R., Li, B., Li, Y., & Liu, H. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00031 · Cited by 10 · see citing works
Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?Preprint
Šavelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09325 · Cited by 7 · see citing works
Exploring the Impact of Intervention Methods on Developers’ Security Behavior in a Manipulated ChatGPT Study
Serafini, R., Yardım, A., & Naiakshina, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713989 · Cited by 6 · see citing works
The Recent Trends of Research on GitHub Copilot: A Systematic Review
Ani, Z. C., Hamid, Z. A., & Zhamri, N. N. (2024) · Communications in Computer and Information Science · doi:10.1007/978-981-99-9589-9_27 · Cited by 5 · see citing works
A Multiple Representation Transformer with Optimized Abstract Syntax Tree for Efficient Code Clone Detection
Yu, T., Li, Y., Lin, L., & He, H. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00050 · Cited by 4 · see citing works
AI-Based Automotive Test Case Generation: An Action Research Study on Integration of Generative AI into Test Automation Frameworks
Karlsson, A., Lindmaa, E., Sun, S., & Staron, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78392-0_4 · Cited by 4 · see citing works
Generating and Reviewing Programming Codes with Large Language Models: A Systematic Mapping Study
de Albuquerque, B. V. L., da Cunha, A. F. S., Souza, L., Siqueira, S. W. M., & dos Santos, R. P. (2024) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3658271.3658342 · Cited by 4 · see citing works
Smells-sus: Sustainability Smells in IaC
Kosbar, S., & Hamdaqa, M. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00117 · Cited by 3 · see citing works
Where Is Self-admitted Code Generated by Large Language Models on GitHub?
Yu, X., Liu, L., Hu, X., Liu, J., & Xia, X. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00047 · Cited by 3 · see citing works
Advancing Secure and Standard Source Code Generation Techniques
Siddiq, M. L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00023 · Cited by 3 · see citing works
Self-citation
Data Preparation for Deep Learning Based Code Smell Detection: A Systematic Literature ReviewPreprint
Zhang, F., Zhang, Z., Keung, J., Tang, X., Yang, Z., Yu, X., & Hu, W. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4693778 · Cited by 3 · see citing works
Lowering the Bar: How Large Language Models Can be Used as a Copilot by Hardware Hackers
Blocklove, J., Pearce, H. A., & Karri, R. (2025) · IEEE Security & Privacy · doi:10.1109/msec.2025.3600140 · Cited by 2 · see citing works
Breaking Barriers in Mobile Accessibility: A Study of LLM-Generated Native Android Interfaces
Rabelo, D. M. F., Martins, R. D. S., Santos, I., Silva, P. H. G. D., da Gama, K. S., & Viana, W. (2025) · IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft) · doi:10.1109/mobilesoft66462.2025.00010 · Cited by 7 · see citing works
Machine Learning Powered Code Smell Detection as a Business Improvement Tool
Siksna, M., Berzina, I., & Romānovs, A. (2023) · International Scientific Conference on Information Technology and Management Science of Riga Technical University (ITMS) · doi:10.1109/itms59786.2023.10317724 · Cited by 2 · see citing works
SmellDSL: A domain-specific language to assist developers in specifying code smell patterns
Silva, R. K., Farias, K., Kunst, R., Carbonera, C. E., & Bolzan, W. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107760 · Cited by 1 · see citing works
The “Question Neighbourhood” Approach for Systematic Evaluation of Code-Generating LLMs
Honarvar, S., Rei, M., & Donaldson, A. F. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3612251 · Cited by 1 · see citing works
Large Language Models for Code Translation: An In-Depth Analysis of Code Smells and Functional Correctness
Feischl, C., & Kern, R. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3777383 · Cited by 3 · see citing works
Execution-as-Configuration: Security Smells in Model Configuration Artifacts
Siddiq, M. L., Johnson, P. N., & Santos, J. C. S. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837470 · Cited by 1 · see citing works
Self-citation
Does LLM Generated Code Smell?
Paul, D. G., Zhu, H. P., & Bayley, I. (2025) · International Conference on Cloud and Big Data Computing (ICCBDC) · doi:10.1109/iccbdc67784.2025.00017 · Cited by 1 · see citing works
Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability
Borg, M., Hewett, D., Hagatulah, N., Couderc, N., Söderberg, E., Graham, D., Kini, U., & Farley, D. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10889-1 · Cited by 4 · see citing works
Investigating the Role of Software Quality Managers and Software Developers in Creating Code Smells
Alzahrani, A. A. H. (2024) · European Journal of Information Technologies and Computer Science · doi:10.24018/compute.2024.4.3.131
User Issues and Concerns in Generative AI: A Mixed-Methods Analysis of App Reviews
Bracamonte, V., Löbner, S., Tronnier, F., Lieberknecht, A.-K., & Pape, S. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-82633-7_19
Analyzing and Detecting Code Design Smells: A Comprehensive Survey
Nashaat, N. R., & Naktal Moaid, E. D. A. N. (2025) · doi:10.1109/csase63707.2025.11054036
109PwnPilot: could an adversary be pair programming with our most trusted software engineers?
Horne, D. (2025) · De Gruyter eBooks · doi:10.1515/9783111436548-006
Comparing Structural Quality of Code Generated by LLMs: A Static Analysis of Code Smells
Sousa, E., Santos, E., Sampaio, A., & Bezerra, C. (2025) · doi:10.5753/eniac.2025.12470
Method Level Refactoring Prediction Using CodeBERT Model
Patnaik, A., Padhy, N., Kumar, L., & Panigrahi, R. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-13009-9_3
Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHubPreprint
Siddiq, M. L., Zhao, X., Lopes, V. C., Casey, B., & Santos, J. C. S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6269349
Self-citation
State of the Art of the Security of Code Generated by LLMs: A Multivocal Literature Review
Ramírez, L. C., Limón, X., Sanchéz-García, Á. J., & Pérez-Arriaga, J. C. (2025) · Programming and Computer Software · doi:10.1134/s0361768825700446
Semantic-Syntactic Feature Fusion via Mixture of Experts for Code Smell Detection
Wang, Y., Ye, C., & Sui, Y. (2025) · International Symposium on Computer Engineering and Intelligent Communications (ISCEIC) · doi:10.1109/isceic67854.2025.11405731
Code Smell Detection Framework Using Self Attention CNN: An Explainable Feature Selection Approach
Tabeeb, F. N., Suchi, K. N. J., Das, D., & Akhter, N. (2025) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11491536
A Comparative Security Evaluation of AI-Generated Code Across Large Language Models and Programming LanguagesPreprint
Aydın, D., & Bahtıyar, Ş. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6968361
Aligning with Human Coding Preferences for Improving Code Generation
Yin, X., Ni, C., & Yang, X. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808113
The transformative potential of AI in software engineering: a case study on LeetCode and ChatGPT
Merkel, M., & Dörpinghaus, J. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10912-5
Can LLMs Keep Up with Library Changes? An Exploratory Study on LLM-Generated Code
Lin, X., Liu, J., & Bao, L. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00088
Evaluating LLMs-Driven Java Code Refactoring from a Developer's Perspective
Freitas, J., Pereira, G., LIMA, L. A., de Sousa, C. R. R., Filho, E., Filho, J. C. D. S., Maia, P. H. M., & Bezerra, C. I. M. (2026) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3786181.3788720
The Impact of AI on Productivity and Code
Yedida, R., Menzies, T., & Novielli, N. (2026) · IEEE Software · doi:10.1109/ms.2026.3683172
AI builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality
Ghammam, A., & Almukhtar, M. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793563
Can LLMs Refactor Their Own Smelly Code? Self-Refactoring Effectiveness and Behavioral Preservation Across Four Open-Source Models
Kim, D. K., Lee, Y.-W., & Jeong, J.-S. (2026) · Applied Sciences · doi:10.3390/app16157733
Improving Secure Code Synthesis with Taxonomy-Guided Agents
Reš, J., Malinka, K., Smrčka, A., & Brnák, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36871-3_8
Towards Tracing the Quality of Code Generated by LLM Back to the Training Code
Wang, C., & Huang, Y. F. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-28454-9_33
Evaluating Large Language Models for Web Accessibility Remediation in Real-World Websites
Aljedaani, W., Gautier, K., Parthasarathy, P. D., & Clemmer, C. (2026) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3797867.3841243

Semantic code smell detection from raw source code using transformers with chunking and data augmentation
Werawatganon, T., & Kantavat, P. (2026) · Machine Learning with Applications · doi:10.1016/j.mlwa.2026.100908
From Fine-tuning to Output: An Empirical Investigation of Test Smells in Transformer-Based Test Code Generation
Aljohani, A., & Do, H. (2024) · ACM Symposium on Applied Computing · doi:10.1145/3605098.3636058 · Cited by 6 · see citing works
FRANC: A Framework for Improving the Quality of Automatically Generated Code
Anonymous, A. (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7972004
SecurityEval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques
Siddiq, M. L., & Santos, J. C. S. (2022) · Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security · doi:10.1145/3549035.3561184 · Cited by 101 · see citing works
Self-citation
Systematically Finding Security Vulnerabilities in Black-Box Code Generation Models.
Hajipour, H., Holz, T., Schönherr, L., & Fritz, M. (2023) · Figshare · doi:10.60882/cispa.32772345

Authors: Siddiq, M. L., Majumder, S. H., Mim, M. R., Jajodia, S., & Santos, J. C. S.
Affiliations: University of Notre Dame (US)
Bangladesh University of Engineering and Technology (BD)
ORCID: Siddiq, Mohammed Latif
Majumder, Shafayat H.
Santos, Joanna C. S.
Venue: 2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM)
Publisher: IEEE
Published: 2022-10
Pages: 71-82
DOI: 10.1109/scam55253.2022.00014
OpenAlex: W4315815628
Cited by: 70
References: 50
Citations by year: 2026: 14 · 2025: 22 · 2024: 28 · 2023: 6
FWCI: 12.52
Type: conference paper
Language: en
Open access: closed
Licence: doi.org/10.15223/policy-029 (stm-asf)
doi.org/10.15223/policy-037 (stm-asf)
Locations: IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) (publishedVersion)
Topics: Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords: code smells, training data quality, GPT-Neo, GitHub Copilot, CodeXGLUE, APP, Pylint, bandits

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

An Empirical Study of Code Smells in Transformer-based Code Generation Techniques

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks