Effect sizes and their variance for AB/BA crossover design studies
Open accessAbstract
Vegas et al. IEEE Trans Softw Eng 42(2):120:135 (2016) raised concerns about the use of AB/BA crossover designs in empirical software engineering studies. This paper addresses issues related to calculating standardized effect sizes and their variances that were not addressed by the Vegas et al.’s paper. In a repeated measures design such as an AB/BA crossover design each participant uses each method. There are two major implication of this that have not been discussed in the software engineering literature. Firstly, there are potentially two different standardized mean difference effect sizes that can be calculated, depending on whether the mean difference is standardized by the pooled within groups variance or the within-participants variance. Secondly, as for any estimated parameters and also for the purposes of undertaking meta-analysis, it is necessary to calculate the variance of the standardized mean difference effect sizes (which is not the same as the variance of the study). We present the model underlying the AB/BA crossover design and provide two examples to demonstrate how to construct the two standardized mean difference effect sizes and their variances, both from standard descriptive statistics and from the outputs of statistical software. Finally, we discuss the implication of these issues for reporting and planning software engineering experiments. In particular we consider how researchers should choose between a crossover design or a between groups design.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
Fitting Linear Mixed-Effects Models Using <b>lme4</b>
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835 · cited 1× · see citing works
“3 an explanation of how standardized effect sizes and their variances can be calculated from analyzes undertaken using R (R Core Team 2016) with the linear mixed model lme4 package (Bates et al. 2015).”
MethodBorenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225 · see citing works
Wasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,347 · see citing works
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303 · see citing works
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799 · see citing works
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2011) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,177 · see citing works
Morris, S. B., & DeShon, R. P. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.105 · Cited by 2,502 · see citing works
Introduction to Robust Estimation and Hypothesis Testing
Wilcox, R. R. (2012) · Elsevier eBooks · doi:10.1016/c2010-0-67044-1 · Cited by 2,086 · see citing works
(2011) · Cited by 2,025 · see citing works
Meta-analysis of experiments with matched groups or repeated measures designs
Dunlap, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.170 · Cited by 1,693 · cited 3× · see citing works
“Thus, the correlation between repeated measures in the EUBAS data set is rather low compared with that reported by Dunlap et al. (1996) for test-retest measurements.”
BackgroundResultPublication Manual of the American Psychological Association
Yao, R. (2011) · Family and Consumer Sciences Research Journal · doi:10.1111/j.1552-3934.2011.02081.x · Cited by 1,459 · see citing works
A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong
Vargha, A., & Delaney, H. D. (2000) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986025002101 · Cited by 1,164 · see citing works
Synthesizing standardized mean‐change measures
Becker, B. J. (1988) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1988.tb00901.x · Cited by 889 · cited 4× · see citing works
“In the past, researchers have proposed standardizing repeated measures studies using the independent groups variance (see Becker 1988; Dunlap et al. 1996; Borenstein et al. 2009). The reason for this is to make the results of repeated measures studies comparable with the results of independent group studies. This is p…”
ResultInfluentialCross-Over Trials in Clinical Research
Senn, S. J. (1994) · Technometrics · doi:10.1080/00401706.1994.10485798 · Cited by 777 · see citing works
Cross‐over Trials In Clinical Research
Senn, S. (2002) · doi:10.1002/0470854596 · Cited by 653 · see citing works
Arcuri, A., & Briand, L. (2012) · Software Testing Verification and Reliability · doi:10.1002/stvr.1486 · Cited by 628 · see citing works
Cumming, G., & Finch, S. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401614002 · Cited by 528 · cited 2× · see citing works
“Researchers are advised to report CIs (see APA 2010; Cumming and Finch 2001) because they provide a direct link to null hypothesis testing and support meta-analysis.”
BackgroundIntroduction to Robust Estimation and Hypothesis Testing
Horn, P. S., & Wilcox, R. R. (1998) · Technometrics · doi:10.2307/1271402 · Cited by 521 · see citing works
Similarity-Dissimilarity Competition in Disjunctive Classification Tasks
Mathy, F., Haladjian, H. H., Laurent, É., & Goldstone, R. L. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00026 · Cited by 488 · see citing works
A simulation study of the model evaluation criterion mmre
Foss, T., Stensrud, E., Kitchenham, B. A., & Myrtveit, I. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1245300 · Cited by 480 · cited 3× · see citing works
“The appropriate formula for the standardized difference of independent group studies and its variance can be found in Hedges and Olkin (1985). We note that the gRM values from three studies reported by Laitenberger et al. (2001) were used without adjustment in a meta-analysis that involved crossover studies, independe…”
MethodInfluentialSelf-citationA systematic review of effect size in software engineering experiments
Kampenes, V. B., Dybå, T., Hannay, J. E., & Sjøberg, D. I. K. (2007) · Information and Software Technology · doi:10.1016/j.infsof.2007.02.015 · Cited by 370 · see citing works
Comparing software prediction techniques using simulation
Shepperd, M., & Kadoda, G. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.965341 · Cited by 280 · see citing works
The performance of the two‐stage analysis of two‐treatment, two‐period crossover trials
Freeman, P. R. (1989) · Statistics in Medicine · doi:10.1002/sim.4780081202 · Cited by 270 · see citing works
APPLICATIONS OF THE NON-CENTRAL t-DISTRIBUTION
Johnson, N. L., & Welch, B. L. (1940) · Biometrika · doi:10.1093/biomet/31.3-4.362 · Cited by 261 · cited 2× · see citing works
“For this reason, we derived our equations from first principles based on the non-central t distribution (Johnson and Welch 1940), as explained in Section 5.”
BackgroundMethodRobust Statistical Methods for Empirical Software Engineering
Kitchenham, B. A., Madeyski, L., Budgen, D., Keung, J., Pearl Brereton, A., Charters, S., Gibbs, S., & Pohthong, A. (2016) · Empirical Software Engineering · doi:10.1007/s10664-016-9437-5 · Cited by 259 · cited 1× · see citing works
“Since gRM is an unbiased estimate of δ, Kitchenham et al. (2017b) show that this leads to: var(dRM) = df df − 2 ( n1 + n2 2n1n2 + g2 RM ) − g 2 RM [c(df )]2 (49)”
BackgroundSelf-citationUrdan, T. (2001) · Psychology Press eBooks · doi:10.4324/9781410612496 · Cited by 242 · see citing works
The American Psychological Association at Washington
(1925) · Science · doi:10.1126/science.61.1576.294 · Cited by 222 · see citing works
Meta‐analysis combining parallel and cross‐over clinical trials. I: Continuous outcomes
Curtin, F., Altman, D. G., & Elbourne, D. R. (2002) · Statistics in Medicine · doi:10.1002/sim.1205 · Cited by 208 · cited 1× · see citing works
“[1].”
Distribution of the standardized mean change effect size for meta‐analysis on repeated measures
Morris, S. B. (2000) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711000159150 · Cited by 206 · see citing works
Which process metrics can significantly improve defect prediction models? An empirical study
Madeyski, L., & Jureczko, M. (2014) · Software Quality Journal · doi:10.1007/s11219-014-9241-7 · Cited by 180 · see citing works
Self-citationMadeyski, L., Orzeszyna, W., Torkar, R., & Jozala, M. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.44 · Cited by 177 · cited 1× · see citing works
“12Also known as Varga and Delaney’s  metric (see Vargha and Delaney 2000; Arcuri and Briand 2014; Madeyski et al. 2014)”
BackgroundSelf-citationUrdan, T. (2005) · Psychology Press eBooks · doi:10.4324/9781410612816 · Cited by 147 · see citing works
Crossover Designs in Software Engineering Experiments: Benefits and Perils
Vegas, S., Apa, C., & Juristo, N. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2467378 · Cited by 145 · see citing works
Laitenberger, O., Emam, K. E., & Harbich, T. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.922713 · Cited by 117 · see citing works
Abrahão, S., Gravino, C., Insfrán, E., Scanniello, G., & Tortora, G. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.27 · Cited by 71 · cited 2× · see citing works
“We note that many of the papers used effectiveness measures based on a scale from 0 to 1 based on the proportion of questions answered correctly (see, for example, Scanniello et al. 2014; Abrahao et al. 2013).”
BackgroundMethodInfluentialOn the impact of UML analysis models on source-code comprehensibility and modifiability
Scanniello, G., Gravino, C., Genero, M., Cruz-Lemus, J. A., & Tortora, G. (2014) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2491912 · Cited by 47 · see citing works
Madeyski, L. (2009) · Springer eBooks · doi:10.1007/978-3-642-04288-1 · Cited by 43 · see citing works
Self-citationCiolkowski, M. (2009) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2009.5316026 · Cited by 42 · see citing works
Stout, D. E., & Ruble, T. L. (1995) · Journal of Accounting Education · doi:10.1016/0748-5751(95)00010-j · Cited by 25 · see citing works
Cross--Project Defect Prediction With Respect To Code Ownership Model: An Empirical Study
Jureczko, M., & Madeyski, L. (2015) · Prace Naukowe Uniwersytetu Ekonomicznego we Wrocławiu · doi:10.5277/e-inf150102 · Cited by 18 · cited 2× · see citing works
“It is also worth mentioning that in order to streamline the uptake of the research results of this paper, the reproducer R package (Madeyski 2017) complements this paper, as well as Kitchenham et al. (2017a), Madeyski and Jureczko (2015), Jureczko and Madeyski (2015), with the aim of making our work easier to reproduc…”
BackgroundMethodSelf-citationLessons from Conducting a Distributed Quasi-experiment
Budgen, D., Kitchenham, B., Charters, S., Gibbs, S., Pohthong, A., Keung, J., & Brereton, P. (2013) · 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.12 · Cited by 10 · cited 2× · see citing works
“If large sample sizes are not possible, consider planning a distributed experiment as proposed by Budgen et al. (2013). In a distributed experiment, all related experiments must use the same protocol and results are aggregated as a single nested experiment.”
MethodSelf-citationCorrections to effect size variances for continuous outcomes of crossover clinical trials
Kitchenham, B. A., Madeyski, L., & Curtin, F. (2017) · Statistics in Medicine · doi:10.1002/sim.7379 · Cited by 10 · see citing works
Self-citationThe Risk of Using the Q Heterogeneity Estimator for Software Engineering Experiments
Dieste, Ó., ́ndez, E. F., Martínez, R. G., & Juristo, N. (2011) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2011.15 · Cited by 7 · cited 1× · see citing works
“Examples of the use of simulation in software engineering include (Shepperd and Kadoda 2001) who used simulation to compare prediction techniques, Dieste et al. (2011) who investigated the use”
Method
Flagged first, then by citations
1 of the 45 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Ciolkowski M (1999) Evaluating the effectiveness of different inspection techniques on informal requirements documents. PhD thesis, University of Kaiserslautern, Kaiserslautern
- Hedges LV, Olkin I (1985) Statistical methods for meta-analysis. Academic Press Orlando, Florida
- Kitchenham BA, Madeyski L (2016) Meta-analysis. In: Kitchenham B A, Budgen D, Brereton P (eds) Evidence-Based Software Engineering and Systematic Reviews. CRC Press, chap 11, pp 133–154
Cited by
Sorted by citations
Madeyski, L., & Lewowski, T. (2020) · Evaluation and Assessment in Software Engineering · doi:10.1145/3383219.3383264 · Cited by 64 · see citing works
“…primary code smell data set and the auxiliary data set is available on Zenodo at https://doi. org/10.5281/zenodo.3666840, as well as in the subsequent version of the reproducer R package on CRAN, as we did in our previous papers (e.g., [4, 5, 7–9]) to streamline the usage of the research results.”
InfluentialSelf-citationUfer, M., Kelsh, D., Schoedel, K. A., & Dingemanse, J. (2021) · SLEEP · doi:10.1093/sleep/zsab224 · Cited by 45 · see citing works
Dote‐Montero, M., Carneiro‐Barrera, A., Martínez‐Vizcaíno, V., Ruiz, J. R., & Amaro‐Gahete, F. J. (2021) · Scandinavian Journal of Medicine and Science in Sports · doi:10.1111/sms.13999 · Cited by 39 · see citing works
Kitchenham, B. A., Madeyski, L., & Pearl Brereton, A. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09747-0 · Cited by 34 · see citing works
“In addition, there is a relationship between the two standard deviations (Madeyski and Kitchenham 2018b): where r is the Pearson correlation between the repeated measures.”
BackgroundMethodResultInfluentialSelf-citationSevilla-Lorente, R., Carneiro-Barrera, A., Molina-Garcia, P., Ruiz, J. R., & Amaro-Gahete, F. (2023) · Journal of Science and Medicine in Sport · doi:10.1016/j.jsams.2023.03.004 · Cited by 30 · see citing works
Studying the explanations for the automated prediction of bug and non-bug issues using LIME and SHAP
Schulte, L., Ledel, B., & Herbold, S. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10469-1 · Cited by 28 · see citing works
“One might note, that the execution plan could be switched to a AB/BA testing [16].”
Karac, I., Turhan, B., & Juristo, N. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2920377 · Cited by 27 · see citing works
“Next, the factors that are not significant are dropped and the effect sizes are investigated according to the formulations for crossover design studies given in [50] based on the reduced model.”
MethodInfluentialDurst, J., Roesel, I., Sudeck, G., Sassenberg, K., & Krauß, I. (2020) · Journal of Medical Internet Research · doi:10.2196/18233 · Cited by 24 · see citing works
Yarizadh, H., Shab‐Bidar, S., Zamani, B., Nazary-Vannani, A., Baharlooi, H., & Djafarian, K. (2020) · Journal of the American College of Nutrition · doi:10.1080/07315724.2019.1661804 · Cited by 25 · see citing works
McAuley, E., MacLaughlin, H. L., Hannan-Jones, M. T., King, N. A., & ROSS, L. J. (2022) · Nutrition Reviews · doi:10.1093/nutrit/nuac063 · Cited by 23 · see citing works
Recommendations for analysing and meta-analysing small sample size software engineering experiments
Kitchenham, B. A., & Madeyski, L. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10504-1 · Cited by 19 · see citing works
“Our long-term goal is to promote the reproducibility of research in software engineering (Madeyski and Kitchenham 2017) by supporting our research papers with algorithms and data sets published in the reproducer R package (see Kitchenham et al. 2017; Madeyski and Kitchenham 2018; Jureczko and…”
Self-citationProblems with Statistical Practice in Human-Centric Software Engineering Experiments
Kitchenham, B. A., Madeyski, L., & Pearl Brereton, A. (2019) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3319008.3319009 · Cited by 18 · see citing works
Self-citationLaursen, D. R. T., Nejstgaard, C. H., Bjørkedal, E., Frost, A. D., Hansen, M., Paludan‐Müller, A. S., Prosenz, J., Werner, C. P., & Hróbjartsson, A. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000055.pub2 · Cited by 13 · see citing works
Quantitative verification with adaptive uncertainty reduction
Alasmari, N., Călinescu, R., Paterson, C., & Mirandola, R. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111275 · Cited by 10 · see citing works
Affective reactions and test-driven development: Results from three experiments and a survey
Baldassarre, M. T., Caivano, D., Fucci, D., Romano, S., & Scanniello, G. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111154 · Cited by 9 · see citing works
Hanenberg, S., & Mehlhorn, N. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-10077-3 · Cited by 8 · see citing works
“…effect is contained in the measurement of A. Due to this, one does not only speak about the treatment effect but also about the period effect (see Madeyski and Kitchenham 2018, p. 1984) – and there are good reasons why one should be concerned about the period effect (see for example Wellek and…”
BackgroundEffect sizes and their variance for AB/BA crossover design studies
Madeyski, L., & Kitchenham, B. A. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3182556 · Cited by 8 · see citing works
Self-citationThe Importance of the Correlation in Crossover Experiments
Kitchenham, B. A., Madeyski, L., Scanniello, G., et al. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3070480 · Cited by 7 · see citing works
Self-citationCreating Evolving Project Data Sets in Software Engineering
Lewowski, T., & Madeyski, L. (2019) · Studies in computational intelligence · doi:10.1007/978-3-030-26574-8_1 · Cited by 7 · see citing works
Self-citationGutierrez, N. A. C., Melo, O. O., & Martínez, C. A. (2023) · Statistical Papers · doi:10.1007/s00362-022-01391-z · Cited by 6 · see citing works
Mining Developer Dynamics for Agent-Based Simulation of Software Evolution
Herbold, V. (2019) · doi:10.53846/goediss-7551 · Cited by 6 · see citing works
Semiparametric generalized estimating equations for repeated measurements in cross-over designs
Gutierrez, N. A. C., Melo, O. O., & Martínez, C. A. (2023) · Statistical Methods in Medical Research · doi:10.1177/09622802231158736 · Cited by 5 · see citing works
“In both the software R Core Team (2022) is used through adaptation of the package geeM built by McDaniel et al. (2013). Systolic pressure data Estimate Std.”
MethodImperative versus declarative collection processing
Mehlhorn, N., & Hanenberg, S. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3519016 · Cited by 4 · see citing works
“9 However, crossover designs require to take possible carryover effects (respectively periodic effects) into account (see [23, p. 1984]) such as learning effects, fatigue effect, etc.”
BackgroundAmin, T. U., & Shahzad, B. (2024) · Requirements Engineering · doi:10.1007/s00766-024-00425-2 · Cited by 4 · see citing works
“The purpose of descriptive statistics is to describe the characteristics of a data set [61–64].”
Need for Sleep: The Impact of a Night of Sleep Deprivation on Novice Developers’ Performance
Fucci, D., Scanniello, G., Romano, S., & Juristo, N. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2834900 · Cited by 31 · see citing works
“Controlling or compensating for such effect is particularly risky as it implies the use of statistical techniques that hampers the interpretation of the results [75], [76].”
BackgroundTest-Driven Development and Embedded Systems: An Exploratory Investigation
Esposito, M., Romano, S., & Scanniello, G. (2023) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa60479.2023.00045 · Cited by 3 · see citing works
reproducer: Reproduce Statistical Analyses and Meta-Analyses
Madeyski, L., Kitchenham, B., & Lewowski, T. (2015) · doi:10.32614/cran.package.reproducer · Cited by 3 · see citing works
Self-citationHindiyeh, R. I., & Cross, J. (2022) · Team Performance Management · doi:10.1108/tpm-02-2022-0016 · Cited by 2 · see citing works
Brainstorming versus a Scenario-based Approach
Franco, A. F. J., & Giraldo, G. U. (2019) · International Conference on Software and Information Engineering (ICSIE) · doi:10.1145/3328833.3328848 · Cited by 2 · see citing works
“Threats to the validity of our empirical study were mitigated following considerations proposed by Kitchenham et al. [22] and Madeyski and Kitchenham [30].”
MethodOn the use of Test-Driven Development for Embedded Systems
Cassieri, P., Esposito, M., Romano, S., & Scanniello, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107779 · Cited by 1 · see citing works
AI-assisted code refactoring: Where can it be helpful and where do humans outperform it?
Ampatzoglou, A., Arvanitou, E.-M., Almpanopoulos, S., Mittas, N., & Chatzigeorgiou, A. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112862 · Cited by 1 · see citing works
Reichl, J., Hanenberg, S., & Gruhn, V. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00058 · Cited by 1 · see citing works
Hanenberg, S., Morzeck, J., & Gruhn, V. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10531-y · Cited by 1 · see citing works
How do Copilot Suggestions Impact Developers' Frustration and Productivity?
Guglielmi, E., Arnoudova, V., Bavota, G., Oliveto, R., & Scalabrino, S. (2025) · arXiv · doi:10.48550/arxiv.2504.06808 · Cited by 1 · see citing works
“We will also conduct statistical tests for carryover effects to ensure validity of our findings [28], [29].”
DREQUS: an approach for the Discovery of REQuirements Using Scenarios
Franco, J., & Fredy, A. (2020)
Penalized GEE for Complex Carry‐Over in Repeated‐Measures Crossover Designs
Gutierrez, N. A. C., Melo, O. O., & Mylona, K. (2026) · Statistics in Medicine · doi:10.1002/sim.70561
A Methodological Evaluation of Meta-Analyses in tDCS - Motor Learning ResearchPreprint
Alsalti, T., Hussey, I., Elson, M., Krause, R. W., & Pohl, S. (2024) · medRxiv · doi:10.1101/2024.07.26.24311068
Alsalti, T., Hussey, I., Elson, M., Krause, R. W., & Pohl, S. (2026) · European Journal of Neuroscience · doi:10.1111/ejn.70533
“Similarly, there are methods to compute the ESs and their corresponding sampling variances when dealing with crossover studies (Madeyski and Kitchenham 2018).”
Gungormus, D. B., & Pérez‐Mármol, J. M. (2026) · Pain · doi:10.1097/j.pain.0000000000004042
- Authors
- Madeyski, L., & Kitchenham, B.
- Affiliations
- Wrocław University of Science and Technology (PL)
- Keele University (GB)
- Corresponding author
- Madeyski, Lech
- Kitchenham, Barbara
- Venue
- Empirical Software Engineering
- Published
- 2017-12-06
- Volume
- 23
- Issue
- 4
- Pages
- 1982-2017
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W2772922594
- Cited by
- 38
- References
- 45
- Citations by year
- 2026: 4 · 2025: 1 · 2024: 5 · 2023: 6 · 2022: 4 · 2021: 5 · 2020: 5 · 2019: 6 · 2018: 2
- FWCI
- 3.20
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Empirical Software Engineering (publishedVersion, cc-by)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- standardized mean difference, repeated measures designs, meta-analysis, software engineering experiments
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
