Empirical analysis of the relationship between CC and SLOC in a large corpus of Java methods and C functions
Open accessAbstract
Measuring the internal quality of source code is one of the traditional goals of making software development into an engineering discipline. Cyclomatic complexity (CC) is an often used source code quality metric, next to source lines of code (SLOC). However, the use of the CC metric is challenged by the repeated claim that CC is redundant with respect to SLOC because of strong linear correlation. We conducted an extensive literature study of the CC/SLOC correlation results. Next, we tested correlation on large Java (17.6 M methods) and C (6.3 M functions) corpora. Our results show that linear correlation between SLOC and CC is only moderate as a result of increasingly high variance. We further observe that aggregating CC and SLOC as well as performing a power transform improves the correlation. Our conclusion is that the observed linear correlation between CC and SLOC of Java methods or C functions is not strong enough to conclude that CC is redundant with SLOC. This conclusion contradicts earlier claims from literature but concurs with the widely accepted practice of measuring of CC next to SLOC. Copyright © 2015 John Wiley & Sons, Ltd.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The two measures (SLOC and CC) tend to be correlated, but not strongly enough to conclude that CC is redundant with SLOC.
“The two measures tend to be correlated, but not strongly enough to conclude that CC is redundant with SLOC [61].”
From In-IDE Code Generation from Natural Language: Promise and Challenges (2022) · cites this paper as [61]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The proof and measurement of association between two things
Spearman, C. (1904) · The American Journal of Psychology · doi:10.2307/1412159 · Cited by 8,039 · see citing works
The proof and measurement of association between two things.
Spearman, C. (2015) · International Journal of Epidemiology · doi:10.1093/ije/dyq191 · Cited by 7,777 · see citing works
McCabe, T. J. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233837 · Cited by 6,059 · see citing works
A metrics suite for object oriented design
Chidamber, S. R., & Kemerer, C. F. (1994) · IEEE Transactions on Software Engineering · doi:10.1109/32.295895 · Cited by 5,698 · see citing works
A Simple Test for Heteroscedasticity and Random Coefficient Variation
Breusch, T. S., & Pagan, A. R. (1979) · Econometrica · doi:10.2307/1911963 · Cited by 5,647 · see citing works
Handbook of Parametric and Nonparametric Statistical Procedures
(2004) · Technometrics · doi:10.1198/tech.2004.s209 · Cited by 4,664 · see citing works
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 1× · see citing works
“The previous literature study can be seen as a restricted form of snowballing [18].”
BackgroundVII. Note on regression and inheritance in the case of two parents
Pearson, K. (1895) · Proceedings of the Royal Society of London · doi:10.1098/rspl.1895.0041 · Cited by 3,473 · see citing works
Graphical Methods for Data Analysis.
Healy, M. J. R., Chambers, J. M., Cleveland, W. S., Kleiner, B., & Tukey, P. A. (1984) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2981587 · Cited by 2,315 · see citing works
Software engineering metrics and models
Conte, S. D., Dunsmore, H. E., & Shen, V. Y. (1986) · Benjamin-Cummings Publishing Co., Inc. eBooks · Cited by 1,062 · see citing works
Comparing measures of sample skewness and kurtosis
JOANES, D. N., & Gill, C. A. (1998) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.1111/1467-9884.00122 · Cited by 1,019 · see citing works
Kvålseth, T. O. (1985) · The American Statistician · doi:10.1080/00031305.1985.10479448 · Cited by 992 · see citing works
The logged dependent variable, heteroscedasticity, and the retransformation problem.
Manning, W. (1998) · Journal of Health Economics · doi:10.1016/s0167-6296(98)00025-3 · Cited by 959 · see citing works
Kvålseth, T. O. (1985) · The American Statistician · doi:10.2307/2683704 · Cited by 761 · see citing works
Predicting fault incidence using software change history
Graves, T., Karr, A. F., Marron, J. S., & Siy, H. P. (2000) · IEEE Transactions on Software Engineering · doi:10.1109/32.859533 · Cited by 753 · see citing works
Quantitative analysis of faults and failures in a complex software system
Fenton, N., & Ohlsson, N. (2000) · IEEE Transactions on Software Engineering · doi:10.1109/32.879815 · Cited by 702 · see citing works
Software errors and complexity: an empirical investigation0
Basili, V. R., & Perricone, B. T. (1984) · Communications of the ACM · doi:10.1145/69605.2085 · Cited by 632 · see citing works
Sensitivity Anaysis as an Ingredient of Modeling
Campolongo, F., Saltelli, A., & Tarantola, S. (2000) · Statistical Science · doi:10.1214/ss/1009213004 · Cited by 607 · see citing works
Sensitivity Analysis as an Ingredient of Modeling
Saltelli, A., Tarantola, S., & Campolongo, F. (2000) · Cited by 604 · see citing works
Program comprehension during software maintenance and evolution
von Mayrhauser, A., & Vans, A. M. (1995) · Computer · doi:10.1109/2.402076 · Cited by 578 · see citing works
The confounding effect of class size on the validity of object-oriented metrics
Emam, K. E., Benlarbi, S., Goel, N., & Rai, S. N. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.935855 · Cited by 422 · see citing works
A large scale study of programming languages and code quality in github
Ray, B., Posnett, D., Filkov, V., & Dévanbu, P. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635922 · Cited by 398 · see citing works
Empirical studies of software engineering
Perry, D. E., Porter, A., & Votta, L. G. (2000) · doi:10.1145/336512.336586 · Cited by 357 · see citing works
The role of replications in Empirical Software Engineering
Shull, F. J., Carver, J. C., Vegas, S., & Juristo, N. (2008) · Empirical Software Engineering · doi:10.1007/s10664-008-9060-1 · Cited by 315 · see citing works
RASCAL: A Domain Specific Language for Source Code Analysis and Manipulation
Klint, P., van der Storm, T., & Vinju, J. J. (2009) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2009.28 · Cited by 304 · see citing works
Self-citationScatterplot Matrix Techniques for Large N
Carr, D. B., Littlefield, R. J., Nicholson, W. L., et al. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478445 · Cited by 242 · see citing works
A critique of cyclomatic complexity as a software metric
Shepperd, M. (1988) · Software Engineering Journal · doi:10.1049/sej.1988.0003 · Cited by 240 · see citing works
Sourcerer: mining and searching internet-scale software repositories
Linstead, E. J., Bajracharya, S. K., Ngo, T. C., Rigor, P., Lopes, C. V., & Baldi, P. (2008) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-008-0118-x · Cited by 237 · see citing works
Cyclomatic complexity density and software maintenance productivity
Gill, G. K., & Kemerer, C. F. (1991) · IEEE Transactions on Software Engineering · doi:10.1109/32.106988 · Cited by 232 · see citing works
A Practical Model for Measuring Maintainability
Heitlager, I., Kuipers, T., & Visser, J. (2007) · International Conference on the Quality of Information and Communications Technology (QUATIC) · doi:10.1109/quatic.2007.8 · Cited by 230 · see citing works
Log transformation: application and interpretation in biomedical research
Feng, C., Wang, H., Lu, N., et al. (2012) · Statistics in Medicine · doi:10.1002/sim.5486 · Cited by 228 · see citing works
On the reliability of mapping studies in software engineering
Wohlin, C., Runeson, P., Neto, P. A. D. M. S., Engström, E., Machado, I., & de Almeida, E. S. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2013.04.076 · Cited by 207 · see citing works
Comparing software metrics tools
Lincke, R., Lundberg, J., & Löwe, W. (2008) · International Symposium on Software Testing and Analysis · doi:10.1145/1390630.1390648 · Cited by 206 · see citing works
Handbook of parametric and nonparametric statistical procedures
(1997) · Choice Reviews Online · doi:10.5860/choice.34-6327 · Cited by 206 · see citing works
A Measure of Control Flow Complexity in Program Text
Woodward, M. R., Hennell, M. A., & Hedley, D. (1979) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1979.226497 · Cited by 199 · see citing works
6th International Conference on the Quality of Information and Communications Technology-TOC
(2007) · International Conference on the Quality of Information and Communications Technology (QUATIC) · doi:10.1109/quatic.2007.7 · Cited by 195 · see citing works
Scatterplot Matrix Techniques for Large N
Carr, D. B., Littlefield, R. J., Nicholson, W. L., et al. (1987) · Journal of the American Statistical Association · doi:10.2307/2289444 · Cited by 194 · see citing works
Effect of violation of normality on the t test of the correlation coefficient.
Edgell, S. E., & Noon, S. M. (1984) · Psychological Bulletin · doi:10.1037/0033-2909.95.3.576 · Cited by 179 · see citing works
Standardized code quality benchmarking for improving software maintainability
Baggen, R., Correia, J. P., Schill, K., et al. (2011) · Software Quality Journal · doi:10.1007/s11219-011-9144-9 · Cited by 158 · see citing works
An Empirical Study of Software Metrics
Li, H. F., & Cheung, W. K. (1987) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1987.233475 · Cited by 151 · see citing works
An extension to the cyclomatic measure of program complexity
Myers, G. J. (1977) · ACM SIGPLAN Notices · doi:10.1145/954627.954633 · Cited by 149 · see citing works
Views on Internal and External Validity in Empirical Software Engineering
Siegmund, J., Siegmund, N., & Apel, S. (2015) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2015.24 · Cited by 136 · see citing works
Making Software: What Really Works, and Why We Believe It
Oram, A., & Wilson, G. (2010) · Cited by 127 · see citing works
Views on Internal and External Validity in Empirical Software Engineering
Siegmund, J., Siegmund, N., & Apel, S. (2015) · Cited by 120 · see citing works
Third time charm: Stronger prediction of programmer performance by software complexity metrics
Curtis, B., Sheppard, S. B., & Milliman, P. (1979) · doi:10.5555/800091.802959 · Cited by 116 · see citing works
Sample size vs. bias in defect prediction
Rahman, F., Posnett, D., Herraiz, I., & Dévanbu, P. (2013) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2491411.2491418 · Cited by 114 · see citing works
Lind, R., & Vairavan, K. (1989) · IEEE Transactions on Software Engineering · doi:10.1109/32.24715 · Cited by 102 · see citing works
An empirical investigation into a large-scale Java open source code repository
Grechanik, M., McMillan, C., DeFerrari, L., et al. (2010) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1852786.1852801 · Cited by 90 · see citing works
Software quality metrics aggregation in industry
Mordal, K., Anquetil, N., Laval, J., et al. (2012) · Journal of Software Evolution and Process · doi:10.1002/smr.1558 · Cited by 87 · see citing works
Ecological inference in empirical software engineering
Posnett, D., Filkov, V., & Dévanbu, P. (2011) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2011.6100074 · Cited by 87 · see citing works
Cyclomatic Complexity and Lines of Code: Empirical Evidence of a Stable Linear Relationship
Jay, G., Hale, J. E., Smith, R., Hale, D. P., Kraft, N. A., & Ward, C. B. (2009) · Journal of Software Engineering and Applications · doi:10.4236/jsea.2009.23020 · Cited by 84 · see citing works
Predicting source-code complexity at the design stage
Henry, S. M., & Selig, C. (1990) · IEEE Software · doi:10.1109/52.50772 · Cited by 79 · see citing works
Analysis of the effects of software reuse on customer satisfaction in an RPG environment
Succi, G., Benedicenti, L., & Vernazza, T. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.922717 · Cited by 75 · see citing works
Towards a Theoretical Model for Software Growth
Herraiz, I., Gonzalez-Barahona, J. M., & Robles, G. (2007) · doi:10.1109/msr.2007.31 · Cited by 74 · see citing works
Determinants of software maintenance profiles: an empirical investigation
Kemerer, C. F., & Slaughter, S. A. (1997) · Journal of Software Maintenance Research and Practice · doi:10.1002/(sici)1096-908x(199707/08)9:4 · Cited by 72 · see citing works
Kitchenham, B. A., & Pickard, L. M. (1987) · Software Engineering Journal · doi:10.1049/sej.1987.0015 · Cited by 63 · see citing works
Revilla, M. A. (2007) · doi:10.1109/issre.2007.12 · Cited by 58 · see citing works
Software Development Effort Estimation Using Fuzzy Logic: A Case Study
Martin, C. L., Pasquier, J. L., Yánez, C., et al. (2006) · Mexican International Conference on Computer Science · doi:10.1109/enc.2005.47 · Cited by 58 · see citing works
Understanding the impact of rapid releases on software quality
Khomh, F., Adams, B., Dhaliwal, T., et al. (2014) · Empirical Software Engineering · doi:10.1007/s10664-014-9308-x · Cited by 57 · see citing works
Vasilescu, B., Serebrenik, A., & van den Brand, M. (2011) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2011.6080798 · Cited by 52 · see citing works
Self-citationA Hybrid Set of Complexity Metrics for Large-Scale Object-Oriented Software Systems
Ma, Y., He, K., Li, B., et al. (2010) · Journal of Computer Science and Technology · doi:10.1007/s11390-010-9398-x · Cited by 48 · see citing works
Some results from an empirical study of computer software
Feuer, A. R., & Fowlkes, E. B. (1979) · doi:10.5555/800091.802958 · Cited by 46 · see citing works
Measurement of data structure complexity
Munson, J. C., & Kohshgoftaar, T. M. (1993) · Journal of Systems and Software · doi:10.1016/0164-1212(93)90065-6 · Cited by 44 · see citing works
Measurement and defect modeling for a legacy software system
Troster, J., & Tian, J. (1995) · Annals of Software Engineering · doi:10.1007/bf02249047 · Cited by 41 · see citing works
Empirical Analysis of the Relationship between CC and SLOC in a Large Corpus of Java Methods
Landman, D., Serebrenik, A., & Vinju, J. J. (2014) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2014.44 · Cited by 39 · see citing works
Self-citationVasilescu, B., Serebrenik, A., & van den Brand, M. (2011) · doi:10.1145/1985374.1985381 · Cited by 38 · see citing works
Self-citationReading Beside the Lines: Indentation as a Proxy for Complexity Metric
Hindle, A., Godfrey, M. W., & Holt, R. C. (2008) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2008.13 · Cited by 35 · see citing works
Program complexity measure for software development management
Sunohara, T., Takano, A., Uehara, K., et al. (1981) · doi:10.5555/800078.802521 · Cited by 32 · see citing works
Debugging effort estimation using software metrics
Gorla, N., Benander, A. C., & Benander, B. A. (1990) · IEEE Transactions on Software Engineering · doi:10.1109/32.44385 · Cited by 30 · see citing works
Proper statistical treatment of species-area data
E.I. DuPont de Nemours and Co., S. R. L., Loehle, C., & USDOE, W. (1988) · doi:10.2172/666215 · Cited by 28 · see citing works
Proper Statistical Treatment of Species-Area Data
Loehle, C. (1990) · Oikos · doi:10.2307/3565750 · Cited by 27 · see citing works
What Does Control Flow Really Look Like? Eyeballing the Cyclomatic Complexity Metric
Vinju, J. J., & Godfrey, M. W. (2012) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2012.17 · Cited by 26 · see citing works
Self-citationAVISPA: a tool for analyzing software process models
Hurtado Alegría, J., Bastarrica Piñeyro, M. C., & Bergel, A. (2014) · Universidad de Chile · doi:10.1002/smr · Cited by 26 · see citing works
An Empirical Study of Lehman's Law on Software Quality Evolution.
Yu, L., & Mishra, A. (2013) · IUScholarWorks (Indiana University) · Cited by 23 · see citing works
A methodology for integrating maintainability using software metrics
Lewis, J. A., & Henry, S. M. (2003) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.1989.65191 · Cited by 21 · see citing works
Beyond Lines of Code: Do We Need More Complexity Metrics?
Herraiz, I., & Hassan, A. E. (2011) · Cited by 19 · see citing works
The Correlation among Software Complexity Metrics with Case StudyPreprint
Tashtoush, Y. M., Al-Maolegi, M., & Arkok, B. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.4523 · Cited by 17 · see citing works
M3: A general model for code analytics in rascal
Basten, B., Hills, M., Klint, P., et al. (2015) · doi:10.1109/swan.2015.7070485 · Cited by 16 · see citing works
First-year results from a research program on human factors in software engineering
Sheppard, S. B., Curtis, B., Milliman, P., et al. (1979) · International Workshop on Managing Requirements Knowledge (MARK) · doi:10.1109/mark.1979.8817108 · Cited by 16 · see citing works
High-MCC Functions in the Linux Kernel
Jbara, A., Matan, A., & Feitelson, D. G. (2013) · Empirical Software Engineering · doi:10.1007/s10664-013-9275-7 · Cited by 15 · see citing works
An empirical study of three common software complexity measures
O'Neal, M. B. (1993) · doi:10.1145/162754.162867 · Cited by 11 · see citing works
Software Measurement and Functional Programming
van den Berg, K. G. (1995) · doi:10.3990/1.9789090082516 · Cited by 11 · see citing works
Applying complexity measures to rule-based prolog programs
Moores, T. T. (1998) · Journal of Systems and Software · doi:10.1016/s0164-1212(98)10042-0 · Cited by 10 · see citing works
A model to predict anti-regressive effort in Open Source Software
Capiluppi, A., & Fernández‐Ramil, J. (2007) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2007.4362632 · Cited by 10 · see citing works
Seven±two software measurement conundrums
Curtis, B., & Carleton, A. D. (2002) · doi:10.1109/metric.1994.344224 · Cited by 5 · see citing works
Software reliability engineering process
Schneidewind, N. F. (2006) · Innovations in Systems and Software Engineering · doi:10.1007/s11334-006-0007-7 · Cited by 4 · see citing works
A Study of Scala Repositories on Github
Coleman, R., & Matthew, A. (2014) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2014.050721 · Cited by 4 · see citing works
Cyclomatic Complexity Number: Analysis of Its Design
Abran, A. (2010) · doi:10.1002/9780470606834.ch6 · Cited by 2 · see citing works
A Large Corpus Of C Source Code Based On Gentoo Packages
Landman, D. (2015) · Data Archiving and Networked Services (DANS) · doi:10.5281/zenodo.208215 · Cited by 2 · see citing works
Self-citationFault-Proneness Estimation and Java Migration: A Preliminary Case Study
Bianco, M., Kaneider, D., Sillitti, A., et al. (2009) · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [25]25. Malhotra R, Singh Y. On the applicability of machine learning techniques for object oriented software fault prediction. Software Engineering: An International Journal 2011; 1:24-37.
- [28]28. Vasilescu B, Serebrenik A, van den Brand MGJ. By no means: a study on aggregating software metrics. 2nd Inter- national Workshop on Emerging Trends in Software Metrics, WETSoM, ACM, 2011; 23-26.
- [36]36. DAVY LANDMAN ET AL. Copyright © 2015 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. 2016; 28:589-618 DOI: 10.1002/smr · DOI
- [63]63. Paige M. A metric for software test planning. Conference Proceedings of COMPSAC 1980; 80:499-504.
- [66]66. EMPIRICAL ANALYSIS OF THE RELATIONSHIP BETWEEN CC AND SLOC 617 Copyright © 2015 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. 2016; 28:589-618 DOI: 10.1002/smr · DOI
Cited by
Sorted by citations
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154 · see citing works
- Authors
- Landman, D., Serebrenik, A., Bouwers, E., & Vinju, J. J.
- Affiliations
- Centrum Wiskunde & Informatica (NL)
- Eindhoven University of Technology (NL)
- Software Improvement Group Amsterdam The Netherlands
- Institut national de recherche en sciences et technologies du numérique (FR)
- Corresponding author
- Landman, Davy
- Venue
- Journal of Software: Evolution and Process
- Publisher
- Wiley
- Published
- 2015-12-09
- Volume
- 28
- Issue
- 7
- Pages
- 589-618
- ISSN
- 2047-7473 (ISSN-L)
- 2047-7481
- OpenAlex
- W2296620627
- Semantic Scholar
- 24d4133443ffcaff10d679486525c65d3458a762
- Cited by
- 29
- References
- 90
- Citations by year
- 2025: 1 · 2024: 2 · 2023: 5 · 2022: 2 · 2021: 2 · 2020: 3 · 2019: 4 · 2018: 4 · 2017: 5 · 2016: 1
- FWCI
- 8.96
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of Software Evolution and Process (publishedVersion)
- Centrum Wiskunde & Informatica (CWI), the national research institute for mathematics and computer science in the Netherlands (submittedVersion)
- TU/e Research Portal (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- TU/e Research Portal (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- cyclomatic complexity, source lines of code, Java methods, c-function, power transformation, metric aggregation, software metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
