Fix that Fix Commit: A real-world remediation analysis of JavaScript projects
Abstract
While there is a large body of work on understanding vulnerabilities in the wild, little has been done to understand the dynamics of the remediation phase of the development cycle. To this end, we have done a timeline analysis on 118K commits from 53 of the most used JavaScript projects from GitHub to understand the provenance and prevalence of vulnerabilities in those projects. We used a vulnerability detector (CodeQL) to filter commits that introduced vulnerabilities and the commits that fixed a prior vulnerability. We found that in 82% of the projects, a commit fixing a prior vulnerability, in turn, introduced one or more new vulnerabilities. Among those projects, on average, 18% of the commits intended to fix vulnerabilities, in turn, introduced one or more new vulnerabilities. We also found that 50% of the total vulnerabilities found in those projects originated from a commit meant to fix a prior vulnerability, and 78% of those vulnerabilities could have been avoided if they were to use proper internal testing. We provide critical insights into how proper internal testing can avoid a significant portion of vulnerabilities, increasing organizations' security posture.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Earlier research applied CodeQL to find vulnerable commits in a JavaScript project.
“Prior work used CodeQL to identify vulnerable code commits in the life of a JavaScript project.3”
From Asleep at the keyboard? Assessing the security of GitHub copilot’s code contributions (2025) · cites this paper as [3]
Cites
Capability maturity model, version 1.1
Paulk, M. C., Curtis, B., Chrissis, M. B., & Weber, C. V. (1993) · IEEE Software · doi:10.1109/52.219617 · Cited by 1,263 · cited 1× · see citing works
“The capability maturity model in software engineering defines different levels based on the maturity of the software development processes [18].”
MethodAutomatic Identification of Bug-Introducing Changes
Kim, S., Zimmermann, T., Pan, K., & Whitehead, E. J. (2006) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2006.23 · Cited by 337 · see citing works
Yin, Z., Yuan, D., Zhou, Y., Pasupathy, S., & Bairavasundaram, L. N. (2011) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2025113.2025121 · Cited by 249 · see citing works
A Large-Scale Empirical Study of Security Patches
Li, F., & Paxson, V. (2017) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3133956.3134072 · Cited by 240 · see citing works
Perl, H., Dechand, S., Smith, M., Arp, D., Yamaguchi, F., Rieck, K., Fahl, S., & Acar, Y. (2015) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2810103.2813604 · Cited by 229 · see citing works
Large-scale vulnerability analysis
Frei, S., May, M., Fiedler, U., & Plattner, B. (2006) · doi:10.1145/1162666.1162671 · Cited by 211 · see citing works
Redmiles, E. M., Kross, S., & Mazurek, M. L. (2016) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2976749.2978307 · Cited by 127 · see citing works
When a Patch Goes Bad: Exploring the Properties of Vulnerability-Contributing Commits
Meneely, A., Srinivasan, H., Musa, A., Tejeda, A. R., Mokary, M., & Spates, B. (2013) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.19 · Cited by 91 · see citing works
Discovering bug patterns in JavaScript
Hanam, Q., Brito, F. S. D. M., & Mesbah, A. (2016) · ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE) · doi:10.1145/2950290.2950308 · Cited by 91 · see citing works
How bugs are born: a model to identify how bugs are introduced in software components
Rodríguez-Pérez, G., Robles, G., Serebrenik, A., Zaidman, A., Germán, D. M., & Gonzalez-Barahona, J. M. (2020) · Empirical Software Engineering · doi:10.1007/s10664-019-09781-y · Cited by 75 · see citing works
Piantadosi, V., Scalabrino, S., & Oliveto, R. (2019) · doi:10.1109/icst.2019.00017 · Cited by 45 · see citing works
How Do Developers Act on Static Analysis Alerts? An Empirical Study of Coverity Usage
Imtiaz, N., Murphy, T. B., & Williams, L. A. (2019) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2019.00040 · Cited by 38 · see citing works
Prechelt, L., & Pepper, A. (2014) · Information and Software Technology · doi:10.1016/j.infsof.2014.05.001 · Cited by 29 · see citing works
(2020) · USENIX Security Symposium · Cited by 27 · see citing works
Alomar, N., Wijesekera, P., Qiu, E., & Egelman, S. (2020) · Symposium On Usable Privacy and Security · Cited by 15 · see citing works
Self-citationAn Empirical Study of the Framework Impact on the Security of JavaScript Web Applications
Peguero, K., Zhang, N., & Cheng, X. (2018) · doi:10.1145/3184558.3188736 · Cited by 10 · see citing works
When Do Changes Induce Software Vulnerabilities?
Alohaly, M., & Takabi, H. (2017) · IEEE International Conference on Collaboration and Internet Computing (CIC) · doi:10.1109/cic.2017.00020 · Cited by 7 · see citing works
The Challenges of Labeling Vulnerability-Contributing Commits
Hogan, K., Warford, N., Morrison, R., Miller, D., Malone, S., & Purtilo, J. M. (2019) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew.2019.00083 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476 · see citing works
Examining Zero-Shot Vulnerability Repair with Large Language Models
Pearce, H. A., Tan, B., Ahmad, B., Karri, R., & Dolan-Gavitt, B. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179420 · Cited by 206 · see citing works
(2023) · doi:10.1109/sp46215.2023.10179324 · Cited by 131 · see citing works
Asleep at the keyboard? Assessing the security of GitHub copilot’s code contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2025) · Communications of the ACM · doi:10.1145/3610721 · Cited by 140 · see citing works
“Prior work used CodeQL to identify vulnerable code commits in the life of a JavaScript project [22].”
MethodJust-in-time software vulnerability detection: Are we there yet?
Lomio, F., Iannone, E., Lucia, A. D., Palomba, F., & Lenarduzzi, V. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111283 · Cited by 66 · see citing works
DeVAIC: A tool for security assessment of AI-generated code
Cotroneo, D., Luca, R. D., & Liguori, P. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107572 · Cited by 21 · see citing works
An Empirical Cybersecurity Evaluation of GitHub Copilot's Code ContributionsPreprint
Pearce, H. A., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2021) · arXiv (Cornell University) · Cited by 17 · see citing works
Differential Static Analysis for Detecting Malicious Updates to Open Source Packages
Froh, F. N., Gobbi, M. F., & Kinder, J. (2023) · ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses (SCORED) · doi:10.1145/3605770.3625211 · Cited by 14 · see citing works
Using Semgrep OSS to Find OWASP Top 10 Weaknesses in PHP Applications: A Case Study
Kree, L., Helmke, R., & Winter, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64171-8_4 · Cited by 8 · see citing works
Wang, P., Guemundsson, B. A., & Kotowicz, K. (2021) · IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) · doi:10.1109/eurospw54576.2021.00013 · Cited by 7 · see citing works
An Exploratory Study on Self-Fixed Software Vulnerabilities in OSS Projects
Forootani, S., Sorbo, A. D., & Visaggio, C. A. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00023 · Cited by 5 · see citing works
Akhoundali, J., Hamidi, H., Rietveld, K. F. D., & Gadyatskaya, O. (2025) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3708821.3736220 · Cited by 3 · see citing works
A socio-technical perspective on software vulnerabilities: A causal analysis
Paradis, C. V., Kazman, R., & Konrad, M. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107553 · Cited by 3 · see citing works
An Empirical Investigation of the Security Weaknesses in Open-Source Projects
Al‐Shammare, H., Al-Otaiby, N. F., Al-Otabi, M., & Alshayeb, M. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661279 · Cited by 2 · see citing works
Demo: Large Scale Analysis on Vulnerability Remediation in Open-source JavaScript Projects
Bandara, V., Rathnayake, T., Weerasekara, N., Elvitigala, C., Thilakarathna, K., Wijesekera, P., Zoysa, K. D., & Keppitiyagama, C. (2021) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3460120.3485357 · Cited by 2 · see citing works
Self-citationCodeX: Contextual Flow Tracking for Browser Extensions
Ahmadpanah, M. M., Gobbi, M. F., Hedin, D. S., Kinder, J., & Sabelfeld, A. (2024) · ACM Conference on Data and Application Security and Privacy (CODASPY) · doi:10.1145/3714393.3726495 · Cited by 1 · see citing works
Analyzing the Evolution of Source Code to Predict Vulnerabilities
Ranasinghe, K., & Keppitiyagama, C. (2022) · International Conference on Advances in ICT for Emerging Regions · doi:10.1109/icter58063.2022.10024073 · Cited by 1 · see citing works
Self-citationGENIE: Guarding the npm Ecosystem with Semantic Malware Detection
Gobbi, M. F., & Kinder, J. (2024) · IEEE Secure Development Conference (SecDev) · doi:10.1109/secdev61143.2024.00017 · Cited by 1 · see citing works
Scalable techniques for risk assessment of open-source libraries
Badyal, N. (2023) · doi:10.32657/10356/173692 · Cited by 1 · see citing works
GitHub Profile Rating and Analysis Using Machine Learning
Punjabi, J., Shilwant, N., Ghadge, V., Sonawane, S., & Tingare, B. A. (2024) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2024.62883
Towards Unveiling Vulnerability Remediation Tactics from OSS Community
Zhang, L., Wu, J., Liu, C., Li, K., Chen, S., & Liu, Y. (2025) · IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1109/apr66717.2025.00010
An Empirical Study of Multi-language Security Patches in Open Source Software
Sun, S., Xing, Y., Zou, G., Wang, X., & Sun, K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97623-0_8
- Authors
- Bandara, V., Rathnayake, T., Weerasekara, N., Elvitigala, C., Thilakarathna, K., Wijesekera, P., & Keppitiyagama, C.
- Affiliations
- University of Colombo (LK)
- University of California, Berkeley (US)
- Venue
- 2020 IEEE 20th International Working Conference on Source Code Analysis and Manipulation (SCAM)
- Publisher
- IEEE
- Published
- 2020-09
- Pages
- 198-202
- OpenAlex
- W3098684777
- Cited by
- 23
- References
- 18
- Citations by year
- 2025: 5 · 2024: 7 · 2023: 4 · 2022: 4 · 2021: 3
- FWCI
- 2.09
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- vulnerability remediation, security bugs, vulnerability detection, security posture, CodeQL, GitHub
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
