The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities
Abstract
Security inspection and testing require experts in security who think like an attacker. Security experts need to know code locations on which to focus their testing and inspection efforts. Since vulnerabilities are rare occurrences, locating vulnerable code locations can be a challenging task. We investigated whether software metrics obtained from source code and development history are discriminative and predictive of vulnerable code locations. If so, security experts can use this prediction to prioritize security inspection and testing efforts. The metrics we investigated fall into three categories: complexity, code churn, and developer activity metrics. We performed two empirical case studies on large, widely used open-source projects: the Mozilla Firefox web browser and the Red Hat Enterprise Linux kernel. The results indicate that 24 of the 28 metrics collected are discriminative of vulnerabilities for both projects. The models using all three types of metrics together predicted over 80 percent of the known vulnerable files with less than 25 percent false positives for both projects. Compared to a random selection of files for inspection and testing, these models would have reduced the number of files and the number of lines of code to inspect or test by over 71 and 28 percent, respectively, for both projects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Data Mining: Practical Machine Learning Tools and Techniques
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,437
Community structure in social and biological networks
Girvan, M., & Newman, M. E. J. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.122653799 · Cited by 15,502
On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
Mann, H. B., & Whitney, D. R. (1947) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730491 · Cited by 14,320
Applied Multivariate Statistical Analysis.
Johnson, A. R., & Wichern, D. W. (1988) · Biometrics · doi:10.2307/2531616 · Cited by 11,353
Witten, I. H. (2008) · Cited by 9,143
An Introduction to Variable and Feature Selection.
Guyon, I., & Elisseeff, A. (2003) · Cited by 6,576
McCabe, T. J. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233837 · Cited by 6,059
(2024) · Physics Subject Headings (PhySH) · doi:10.29172/7c2a6982-6d72-4cd8-bba6-2fccb06a7011 · Cited by 3,857
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/153244303322753616 · Cited by 3,502
Martinez, A. M., & Kak, A. C. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908974 · Cited by 3,249
A validation of object-oriented design metrics as quality indicators
Basili, V. R., Briand, L. C., & Melo, W. L. (1996) · IEEE Transactions on Software Engineering · doi:10.1109/32.544352 · Cited by 1,726
Lessmann, S., Baesens, B., Mues, C., & Pietsch, S. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.35 · Cited by 1,226
Data Mining Static Code Attributes to Learn Defect Predictors
Menzies, T., Greenwald, J., & Frank, A. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.256941 · Cited by 1,215
Statistical Comparisons of Classifiers over Multiple Data Sets
DemšarJanez (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248548 · Cited by 1,074
Software Security: Building Security In
McGraw, G. (2006) · 2006 17th International Symposium on Software Reliability Engineering · doi:10.1109/issre.2006.43 · Cited by 785
Mining metrics to predict component failures
Nagappan, N., Ball, T., & Zeller, A. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134349 · Cited by 782
Use of relative code churn measures to predict system defect density
Nagappan, N., & Ball, T. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062514 · Cited by 775 · cited 7×
“Software Eng., vol. 33, no. 1, pp. 2-13, 2007 [8] Nagappan, N., and Ball, T., "Using Software Dependencies and Churn Metrics to Predict Field Failures: An Empirical Case Study," Proc.”
BackgroundInfluentialPredicting fault incidence using software change history
Graves, T. L., Karr, A. F., Marron, J. S., & Siy, H. (2000) · IEEE Transactions on Software Engineering · doi:10.1109/32.859533 · Cited by 753
Predicting the location and number of faults in large software systems
Ostrand, T. J., Weyuker, E. J., & Bell, R. M. (2005) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2005.49 · Cited by 720
Exploring the relationships between design measures and software quality in object-oriented systems
Briand, L. C., Wüst, J., Daly, J. W., & Victor Porter, D. (2000) · Journal of Systems and Software · doi:10.1016/s0164-1212(99)00102-8 · Cited by 710
(2005) · Lecture Notes in Computer Science · doi:10.1007/b106453 · Cited by 636
Network Analysis: Methodological Foundations (Lecture Notes in Computer Science)
Brandes, U., & Erlebach, T. (2005) · Springer eBooks · Cited by 514
Predicting vulnerable software components
Neuhaus, S., Zimmermann, T., Holler, C., & Zeller, A. (2007) · Proceedings of the 14th ACM conference on Computer and communications security · doi:10.1145/1315245.1315311 · Cited by 416
(2003) · CERN Document Server (European Organization for Nuclear Research) · doi:10.1007/b97547 · Cited by 369
The influence of organizational structure on software quality: an empirical case study
Nagappan, N., Murphy, B., & Basili, V. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368160 · Cited by 363
Methodology for validating software metrics
Schneidewind, N. F. (1992) · IEEE Transactions on Software Engineering · doi:10.1109/32.135774 · Cited by 351
Structured testing: a testing methodology using the cyclomatic complexity metric
Wallace, D. R., Watson, A. H., & McCabe, T. J. (1996) · doi:10.6028/nist.sp.500-235 · Cited by 312
Kim, T.-K., & Kittler, J. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.58 · Cited by 290
Software vulnerability analysis
Krsul, I., & Spafford, E. H. (1998) · Cited by 286
Menzies, T., Dekhtyar, A., Distefano, J., & Greenwald, J. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.70721 · Cited by 277
Can developer-module networks predict failures?
Pinzger, M., Nagappan, N., & Murphy, B. (2008) · Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering · doi:10.1145/1453101.1453105 · Cited by 256
Predicting failures with developer networks and social network analysis
Meneely, A., Williams, L., Snipes, W., & Osborne, J. (2008) · Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering · doi:10.1145/1453101.1453106 · Cited by 245
Self-citationMeasuring, analyzing and predicting security vulnerabilities in software systems
Alhazmi, O. H., Malaiya, Y. K., & Ray, I. (2007) · Computers & Security · doi:10.1016/j.cose.2006.10.002 · Cited by 237
Use of relative code churn measures to predict system defect density
Nagappan, N., & Ball, T. (2005) · Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005. · doi:10.1109/icse.2005.1553571 · Cited by 201
Early quality prediction: a case study in telecommunications
Khoshgoftaar, T. M., Allen, E. B., Kalaichelvan, K. S., & Goel, N. (1996) · IEEE Software · doi:10.1109/52.476287 · Cited by 200
Performance of five two-sample location tests for skewed distributions with unequal variances
Fagerland, M. W., & Sandvik, L. (2009) · Contemporary Clinical Trials · doi:10.1016/j.cct.2009.06.007 · Cited by 192
The Effects of Over and Under Sampling on Fault-prone Module Detection
Kamei, Y., Monden, A., Matsumoto, S., Kakimoto, T., & Matsumoto, K. (2007) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2007.28 · Cited by 192
Predicting software defects in varying development lifecycles using Bayesian nets
Fenton, N., Neil, M., Marsh, W., Hearty, P., Marquez, D., Krause, P., & Mishra, R. (2007) · Information and Software Technology · doi:10.1016/j.infsof.2006.09.001 · Cited by 190
Predicting fault-prone components in a java legacy system
Arisholm, E., & Briand, L. C. (2006) · Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering · doi:10.1145/1159733.1159738 · Cited by 187
Implications of ceiling effects in defect predictors
Menzies, T., Turhan, B., Bener, A., Gay, G., Cukic, B., & Jiang, Y. (2008) · Proceedings of the 4th international workshop on Predictor models in software engineering · doi:10.1145/1370788.1370801 · Cited by 185
Applying social network analysis to the information in CVS repositories
Lopez-Fernandez, L. (2004) · "International Workshop on Mining Software Repositories (MSR 2004)" W17S Workshop - 26th International Conference on Software Engineering · doi:10.1049/ic:20040485 · Cited by 179
Using Software Dependencies and Churn Metrics to Predict Field Failures: An Empirical Case Study
Nagappan, N., & Ball, T. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.13 · Cited by 176
(2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.87 · Cited by 170
Weyuker, E. J., Ostrand, T. J., & Bell, R. M. (2008) · Empirical Software Engineering · doi:10.1007/s10664-008-9082-8 · Cited by 169
Program readability: procedures versus comments
Tenny, T. (1988) · IEEE Transactions on Software Engineering · doi:10.1109/32.6171 · Cited by 152
Networks Evolving Step by Step: Statistical Analysis of Dyadic Event Data
Brandes, U., Lerner, J., & Snijders, T. A. B. (2009) · doi:10.1109/asonam.2009.28 · Cited by 142
IEEE Standard for a Software Quality Metrics Methodology
(1992) · IEEE Standards · doi:10.1109/ieeestd.1993.115124 · Cited by 108
(2007) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2007.80 · Cited by 91
Prioritizing software security fortification throughcode-level metrics
Gegick, M., Williams, L. A., Osborne, J. A., & Vouk, M. A. (2008) · ACM Workshop on Quality of Protection (QoP) · doi:10.1145/1456362.1456370 · Cited by 83
Self-citationUsing Historical In-Process and Product Metrics for Early Estimation of Software Failures
Nagappan, N., Ball, T., & Murphy, T. B. (2006) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2006.50 · Cited by 71
Security of open source web applications
Wal, J., Doyle, M., Welch, G. A., & Whelan, M. (2009) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2009.5314215 · Cited by 55
Automating algorithms for the identification of fault-prone files
Ostrand, T. J., Weyuker, E. J., & Bell, R. M. (2007) · International Symposium on Software Testing and Analysis · doi:10.1145/1273463.1273493 · Cited by 48
Project Data Incorporating Qualitative Factors for Improved Software Defect Prediction
Fenton, N., Neil, M., Marsh, W., Hearty, P., Radliński, Ł., & Krause, P. (2007) · Predictive Models in Software Engineering · doi:10.1109/promise.2007.11 · Cited by 46
Toward Non-security Failures as a Predictor of Security Faults and Failures
Gegick, M., Rotella, P., & Williams, L. A. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-00199-4_12 · Cited by 24
Self-citation
Cited by
VulDeePecker: A Deep Learning-Based System for Vulnerability Detection
Li, Z., Zou, D., Xu, S., Ou, X., Jin, H., Wang, S., Deng, Z., & Zhong, Y. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23158 · Cited by 898
SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities
Li, Z., Zou, D., Xu, S., Jin, H., Zhu, Y., & Chen, Z. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2021.3051525 · Cited by 639
Software Vulnerability Detection Using Deep Neural Networks: A Survey
Lin, G., Wen, S., Han, Q.-L., Zhang, J., & Xiang, Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2993293 · Cited by 457
Nam, J., Pan, S. J., & Kim, S. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606584 · Cited by 449
Ghaffarian, S. M., & Shahriari, H. R. (2018) · ACM Computing Surveys · doi:10.1145/3092566 · Cited by 353
Metadata
- Authors
- Shin, Y., Meneely, A., Williams, L., & Osborne, J. A.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2011-11
- Volume
- 37
- Issue
- 6
- Pages
- 772-787
- ISSN
- 0098-5589 (ISSN-L)
- OpenAlex
- W2043837581
- Cited by
- 607
- References
- 55
- FWCI
- 35.24
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.