A large scale study of programming languages and code quality in github
The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
What is the effect of programming languages on software quality? This question has been a topic of much debate for a very long time. In this study, we gather a very large data set from GitHub (729 projects, 80 Million SLOC, 29,000 authors, 1.5 million commits, in 17 languages) in an attempt to shed some empirical light on this question. This reasonably large sample size allows us to use a mixed-methods approach, combining multiple regression modeling with visualization and text analytics, to study the effect of language features such as static v.s. dynamic typing, strong v.s. weak typing on software quality. By triangulating findings from different methods, and controlling for confounding effects such as team size, project size, and project history, we report that language design does have a significant, but modest effect on software quality. Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing. We also find that functional languages are somewhat better than procedural languages. It is worth noting that these modest effects arising from language design are overwhelmingly dominated by the process factors such as project size, team size, and commit size. However, we hasten to caution the reader that even these modest effects might quite possibly be due to other, intangible process factors, e.g., the preference of certain personality types for functional, static and strongly typed languages.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences
Cohen, P., Cohen, P., West, S. G., & Aiken, L. S. (2014) · Psychology Press eBooks · doi:10.4324/9781410606266 · Cited by 20,737
Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses
Vuong, Q. H. (1989) · Econometrica · doi:10.2307/1912557 · Cited by 5,929
Blei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516 · cited 1×
“First, we used Latent Dirichlet Allocation(LDA) [7], a well-known topic analysis algorithm, on the text describing project features.”
MethodApplied Multiple Regression/Correlation Analysis for the Behavioral Sciences.
Fong, G. T., Cohen, J., & Cohen, P. (1985) · Journal of the American Statistical Association · doi:10.2307/2287930 · Cited by 4,475
Types and Programming Languages
Pierce, B. C. (2002) · Cited by 2,226
A theory of type polymorphism in programming
Milner, R. (1978) · Journal of Computer and System Sciences · doi:10.1016/0022-0000(78)90014-4 · Cited by 2,205
Selecting Empirical Methods for Software Engineering Research
Easterbrook, S., Singer, J., Storey, M.-A., & Damian, D. (2008) · Guide to Advanced Empirical Software Engineering · doi:10.1007/978-1-84800-044-5_11 · Cited by 1,167 · cited 1×
“Our approach is best described as mixed-methods, or triangulation [7] approach.”
MethodProceedings of the 2013 International Conference on Software Engineering
Notkin, D. S., Cheng, B. H. C., & Pohl, K. (2013) · Cited by 896
Concurrent Programming in Erlang
Armstrong, J., Williams, M., & Virding, R. (1993) · Cited by 742
Cited by
Initial coin offerings (ICOs) to finance new ventures
Fisch, C. (2019) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2018.09.007 · Cited by 533
Curating GitHub for engineered software projects
Munaiah, N., Kroh, S., Cabrey, C., & Nagappan, M. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9512-6 · Cited by 365
“In this validation perspective, the performance of score-based and Random Forest classifiers is compared to that of the stargazers-based classifier used in prior literature (Ray et al. 2014).”
BackgroundMethodYilmaz, R., & Karaoglan Yilmaz, F. G. (2023) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2023.100005 · Cited by 344
Le, X. B. D., Lo, D., & Le Goues, C. (2016) · 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) · doi:10.1109/saner.2016.76 · Cited by 332
On the "naturalness" of buggy code
Ray, B., Hellendoorn, V., Godhane, S., Tu, Z., Bacchelli, A., & Devanbu, P. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884848 · Cited by 252
Self-citationThe ABC of Software Engineering Research
Stol, K.-J., & Fitzgerald, B. (2018) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3241743 · Cited by 245
Navigating the Complexity of Generative AI Adoption in Software Engineering
Russo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652154 · Cited by 217
“” Ensuring the reliability and accuracy of generated code is essential for LLM adoption [67].”
BackgroundSun, D., Boudouaia, A., Zhu, C., & Li, Y. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00446-5 · Cited by 190
DéjàVu: a map of code duplicates on GitHub
Lopes, C. V., Maj, P., Martins, P., Saini, V., Yang, D., Zitny, J., Sajnani, H., & Vitek, J. (2017) · Proceedings of the ACM on Programming Languages · doi:10.1145/3133908 · Cited by 183
“[Ray et al. 2014] studied software quality using the top 50 most popular projects in 17 languages.”
What Attracts Newcomers to Onboard on OSS Projects? TL;DR: Popularity
Fronchetti, F., Wiese, I., Pinto, G., & Steinmacher, I. (2019) · IFIP Advances in Information and Communication Technology · doi:10.1007/978-3-030-20883-7_9 · Cited by 34
“To improve the variety of projects, we first selected the fifteen most popular programming languages used on GitHub [20].”
Landman, D., Serebrenik, A., Bouwers, E., & Vinju, J. J. (2016) · Journal of Software: Evolution and Process · doi:10.1002/smr.1760 · Cited by 29
Zhao, G., Yang, L., Hu, B., & Wang, J. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251336154 · Cited by 27
Furia, C. A., Torkar, R., & Feldt, R. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3611667 · Cited by 14
Automatic analysis of available source code of top artificial intelligence conference papers
Lin, J., Wang, Y., Yu, Y., Zhou, Y., Chen, Y., & Shi, X. (2022) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194022500358 · Cited by 7
Chua, B. B., & Zhang, Y. (2020) · Australasian Journal of Information Systems · doi:10.3127/ajis.v24i0.1714 · Cited by 7
Bhuiyan, M. H. M., Kumar, M. K. B., & Staicu, C.-A. (2026) · Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering · doi:10.1145/3744916.3787766 · Cited by 5
“Prior work shows that stars signal user interest and are widely used in empirical studies to rank and select repositories [12–14, 46, 68].”
Metadata
- Authors
- Ray, B., Posnett, D., Filkov, V., & Devanbu, P.
- Venue
- Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2014-11-11
- Pages
- 155-165
- OpenAlex
- W2142958724
- Cited by
- 389
- References
- 32
- FWCI
- 25.65
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.