The DOI matches its Crossref record; 108 of the 108 references it cites were found in a registry or an index.
Bayesian Data Analysis in Empirical Software Engineering Research
Abstract
Statistics comes in two main flavors: frequentist and Bayesian. For historical and technical reasons, frequentist statistics have traditionally dominated empirical data analysis, and certainly remain prevalent in empirical software engineering. This situation is unfortunate because frequentist statistics suffer from a number of shortcomings-such as lack of flexibility and results that are unintuitive and hard to interpret-that curtail their effectiveness when dealing with the heterogeneous data that is increasingly available for empirical analysis of software engineering practice. In this paper, we pinpoint these shortcomings, and present Bayesian data analysis techniques that provide tangible benefits-as they can provide clearer results that are simultaneously robust and nuanced. After a short, high-level introduction to the basic tools of Bayesian statistics, we present the reanalysis of two empirical studies on the effectiveness of automatically generated tests and the performance of programming languages. By contrasting the original frequentist analyses with our new Bayesian analyses, we demonstrate the concrete advantages of the latter. To conclude we advocate a more prominent role for Bayesian statistical techniques in empirical software engineering research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,245
Baltosser, W. H., & Zar, J. H. (1996) · Ecology · doi:10.2307/2265725 · Cited by 35,166
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597
The Elements of Statistical Learning: Data Mining, Inference, and Prediction
John Lu, Z. Q. (2010) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2010.00646_6.x · Cited by 12,390
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Nonparametric Statistics for the Behavioral Sciences.
Whitfield, J. W., & Siegel, S. (1957) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2342828 · Cited by 7,947
Stan : A Probabilistic Programming Language
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017) · Journal of Statistical Software · doi:10.18637/jss.v076.i01 · Cited by 7,754 · cited 1×
“Tools such as Stan [16] provide scalable generic implementations of these techniques.”
Method
Research works printed but not linked: 12
- 10.1201/b16018 · DOI · Paper page
- gelman, 2016, The problems with $p$p-values are not just with $p$p-values, The Amer Statistician
- downey, 2013, Bayesian Thinking
- dillon, 0, Proc 38th Int Conf Softw Eng
- jalote, 2014, Proc 36th Int Conf Softw Eng
- 10.1145/2884781.2884783 · DOI
- jeffreys, 1998, Theory of Probability
- notkin, 0, Proc 35th Int Conf Softw Eng
- uchitel, 0, Proc 39th Int Conf Softw Eng
- 10.1080/00031305.2016.1154108 · DOI · Paper page
- bertolino, 2015, Proc 37th IEEE/ACM Int Conf Softw Eng, 1
- 10.2307/2331554 · DOI · Paper page
Cited by
Pull Request Governance in Open Source Communities
Alami, A., Pardo, R., Cohn, M. L., & Wasowski, A. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3128356 · Cited by 19
Furia, C. A., Torkar, R., & Feldt, R. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3611667 · Cited by 14
Self-citationSame Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code
Khojah, R., Gomes de Oliveira Neto, F., Mohamad, M., Frattini, J., & Leitner, P. (2026) · Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3832783.3834423
Metadata
- Authors
- Furia, C. A., Feldt, R., & Torkar, R.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2019
- Pages
- 1-1
- ISSN
- 0098-5589
- 1939-3520
- 2326-3881
- Cited by
- 42
- References
- 108
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Bayesian Data Analysis in Empirical Software Engineering Research. https://app.cochat.ai/papers/10.1109/tse.2019.2935974
- The paper
- Furia, C. A., Feldt, R., & Torkar, R. (2019). Bayesian Data Analysis in Empirical Software Engineering Research. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2019.2935974
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
