The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
A large scale study of programming languages and code quality in github
Abstract
What is the effect of programming languages on software quality? This question has been a topic of much debate for a very long time. In this study, we gather a very large data set from GitHub (729 projects, 80 Million SLOC, 29,000 authors, 1.5 million commits, in 17 languages) in an attempt to shed some empirical light on this question. This reasonably large sample size allows us to use a mixed-methods approach, combining multiple regression modeling with visualization and text analytics, to study the effect of language features such as static v.s. dynamic typing, strong v.s. weak typing on software quality. By triangulating findings from different methods, and controlling for confounding effects such as team size, project size, and project history, we report that language design does have a significant, but modest effect on software quality. Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing. We also find that functional languages are somewhat better than procedural languages. It is worth noting that these modest effects arising from language design are overwhelmingly dominated by the process factors such as project size, team size, and commit size. However, we hasten to caution the reader that even these modest effects might quite possibly be due to other, intangible process factors, e.g., the preference of certain personality types for functional, static and strongly typed languages.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences
Cohen, P., Cohen, P., West, S. G., & Aiken, L. S. (2014) · Psychology Press eBooks · doi:10.4324/9781410606266 · Cited by 20,737
Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses
Vuong, Q. H. (1989) · Econometrica · doi:10.2307/1912557 · Cited by 5,929
Blei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516
Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences.
Fong, G. T., Cohen, J., & Cohen, P. (1985) · Journal of the American Statistical Association · doi:10.2307/2287930 · Cited by 4,475
Types and Programming Languages
Pierce, B. C. (2002) · Cited by 2,226
A theory of type polymorphism in programming
Milner, R. (1978) · Journal of Computer and System Sciences · doi:10.1016/0022-0000(78)90014-4 · Cited by 2,205
Selecting Empirical Methods for Software Engineering Research
Easterbrook, S., Singer, J., Storey, M.-A., & Damian, D. (2008) · Guide to Advanced Empirical Software Engineering · doi:10.1007/978-1-84800-044-5_11 · Cited by 1,167 · cited 1×
“Our approach is best described as mixed-methods, or triangulation [7] approach.”
MethodProceedings of the 2013 International Conference on Software Engineering
Notkin, D. S., Cheng, B. H. C., & Pohl, K. (2013) · Cited by 896
Concurrent Programming in Erlang
Armstrong, J., Williams, M., & Virding, R. (1993) · Cited by 742
Significance Testing of the Spearman Rank Correlation Coefficient
Zar, J. H. (1972) · Journal of the American Statistical Association · doi:10.1080/01621459.1972.10481251 · Cited by 689
Effects of Process Maturity on Quality, Cycle Time, and Effort in Software Product Development
Harter, D. E., Krishnan, M. S., & Slaughter, S. A. (2000) · Management Science · doi:10.1287/mnsc.46.4.451.12056 · Cited by 507
Identifying reasons for software changes using historic databases
Mockus, & Votta (2000) · Proceedings 2000 International Conference on Software Maintenance · doi:10.1109/icsm.2000.883028 · Cited by 472
The principle type-scheme of an object in combinatory logic
Hindley, R. (1969) · Transactions of the American Mathematical Society · doi:10.1090/s0002-9947-1969-0253905-6 · Cited by 448
The confounding effect of class size on the validity of object-oriented metrics
El Emam, K., Benlarbi, S., Goel, N., & Rai, S. N. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.935855 · Cited by 422
Significance Testing of the Spearman Rank Correlation Coefficient
Zar, J. H. (1972) · Journal of the American Statistical Association · doi:10.2307/2284441 · Cited by 396
Don't touch my code!: examining the effects of ownership on software quality
Bird, C., Nagappan, N., Murphy, B., Gall, H., & Devanbu, P. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025119 · Cited by 340
Self-citationThe Principal Type-Scheme of an Object in Combinatory Logic
Hindley, R. (1969) · Transactions of the American Mathematical Society · doi:10.2307/1995158 · Cited by 319
Have things changed now?: an empirical study of bug characteristics in modern open source software
Li, Z., Tan, L., Wang, X., Lu, S., Zhou, Y., & Zhai, C. (2006) · Proceedings of the 1st workshop on Architectural and system support for improving software dependability · doi:10.1145/1181309.1181314 · Cited by 291
Bug characteristics in open source software
Tan, L., Liu, C., Li, Z., Wang, X., Zhou, Y., & Zhai, C. (2014) · Empirical Software Engineering · doi:10.1007/s10664-013-9258-8 · Cited by 239
Odersky, M., Spoon, L., & Venners, B. (2008) · Cited by 218
How, and Why, Process Metrics are better
Rahman, F., & Dévanbu, P. (2013) · Cited by 184
Weyuker, E. J., Ostrand, T. J., & Bell, R. M. (2008) · Empirical Software Engineering · doi:10.1007/s10664-008-9082-8 · Cited by 169
Empirical analysis of programming language adoption
Meyerovich, L. A., & Rabkin, A. S. (2013) · Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications · doi:10.1145/2509136.2509515 · Cited by 158
An experiment to assess different defect detection methods for software requirements inspections
Porter, A. A., & Votta, L. G. (1994) · doi:10.5555/257734.257749 · Cited by 124
Proceedings 2000 International Conference on Software Maintenance
(2000) · doi:10.1109/icsm.2000 · Cited by 87
Hanenberg, S. (2010) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/1869459.1869462 · Cited by 86
Assessing programming language impact on development and maintenance
Bhattacharya, P., & Neamtiu, I. (2011) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1985793.1985817 · Cited by 76
Do static type systems improve the maintainability of software systems? An empirical study
Kleinschmager, S., Robbes, R., Stefik, A., Hanenberg, S., & Tanter, E. (2012) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2012.6240483 · Cited by 73
An empirical study of the influence of static type systems on the usability of undocumented software
Mayer, C., Hanenberg, S., Robbes, R., Tanter, É., & Stefik, A. (2012) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2384616.2384666 · Cited by 62
Pankratius, V., Schmidt, F., & Garretón, G. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227200 · Cited by 51
(2003)
Cited by
Initial coin offerings (ICOs) to finance new ventures
Fisch, C. (2019) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2018.09.007 · Cited by 533
Curating GitHub for engineered software projects
Munaiah, N., Kroh, S., Cabrey, C., & Nagappan, M. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9512-6 · Cited by 365
“In this validation perspective, the performance of score-based and Random Forest classifiers is compared to that of the stargazers-based classifier used in prior literature (Ray et al. 2014).”
BackgroundMethodYilmaz, R., & Karaoglan Yilmaz, F. G. (2023) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2023.100005 · Cited by 344
Le, X. B. D., Lo, D., & Le Goues, C. (2016) · 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) · doi:10.1109/saner.2016.76 · Cited by 332
On the "naturalness" of buggy code
Ray, B., Hellendoorn, V., Godhane, S., Tu, Z., Bacchelli, A., & Devanbu, P. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884848 · Cited by 252
Self-citation
Metadata
- Authors
- Ray, B., Posnett, D., Filkov, V., & Devanbu, P.
- Venue
- Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2014-11-11
- Pages
- 155-165
- OpenAlex
- W2142958724
- Cited by
- 389
- References
- 32
- FWCI
- 25.65
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.