Natural Attack for Pre-trained Models of Code
Open accessAbstract
Pre-trained models of code have achieved success in many important software engineering tasks. However, these powerful models are vulnerable to adversarial attacks that slightly perturb model inputs to make a victim model produce wrong outputs. Current works mainly attack models of code with examples that preserve operational program semantics but ignore a fundamental requirement for adversarial example generation: perturbations should be natural to human judges, which we refer to as naturalness requirement.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The study uses test sets for inference and training sets for selecting ICL demonstrations, aligned with previous ICL work.
“It is important to note that, aligned with previous ICL work [50], we use the test sets of these datasets for inference, while the selection of ICL demonstrations is drawn from the training sets.”
From Demonstration Attack against In-Context Learning for Code Intelligence (2024) · cites this paper as [50]
Cites
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · see citing works
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602 · see citing works
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · see citing works
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495 · see citing works
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 2× · see citing works
“[43] introduce GPT-C, a variant of GPT-2 [37] trained on a large corpus containing multiple programming lan-”
MethodBig code != big vocabulary: open-vocabulary models for source code
Karampatsis, R.-M., Babii, H., Robbes, R., Sutton, C., & Janes, A. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380342 · Cited by 182 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029 · see citing works
Self-citationReCode: Robustness Evaluation of Code Generation Models
Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.773 · Cited by 42 · see citing works
Demonstration Attack against In-Context Learning for Code Intelligence
Ge, Y., Sun, W., Lou, Y., Fang, C., Zhang, Y., Li, Y., Zhang, X., Liu, Y., Zhao, Z., & Chen, Z. (2024) · arXiv · doi:10.48550/arxiv.2410.02841
“A straightforward approach is to adapt existing adversarial attack meth-ods [11, 19, 49, 50, 52] for code to make these modifications.”
Method
- Authors
- Yang, Z., Shi, J., He, J., & Lo, D.
- Affiliations
- Singapore Management University (SG)
- Venue
- Proceedings of the 44th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 1482-1493
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4220722393
- Cited by
- 166
- References
- 7
- Citations by year
- 2026: 27 · 2025: 50 · 2024: 47 · 2023: 35 · 2022: 7
- FWCI
- 19.22
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
- arXiv (Cornell University) (cc-by)
- Funders
- Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) (19-C220-SMU-002)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- adversarial attacks, pre-trained code models, adversarial example generation, code embeddings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
