CoChat
Sign Up
Cited by 3Software Engineering Techniques and PracticesOptimal Experimental Design MethodsSoftware Engineering Research

Crossover Designs in Software Engineering Experiments: Review of the State of Analysis

Open access
Frattini, Julian, Fucci, Davide, Vegas, Sira·2024·Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement
DOI 10.1145/3674805.3690754
Open this paper in CoChat — free accountOpen PDF

Abstract

Experimentation is an essential method for causal inference in any empirical discipline. Crossover-design experiments are common in Software Engineering (SE) research. In these, subjects apply more than one treatment in different orders. This design increases the amount of obtained data and deals with subject variability but introduces threats to internal validity like the learning and carryover effect. Vegas et al. reviewed the state of practice for crossover designs in SE research and provided guidelines on how to address its threats during data analysis while still harnessing its benefits. In this paper, we reflect on the impact of these guidelines and review the state of analysis of crossover-design experiments in SE publications between 2015 and March 2024. To this end, by conducting a forward snowballing of the guidelines, we survey 136 publications reporting 67 crossover-design experiments and evaluate their data analysis against the provided guidelines. The results show that the validity of data analyses has improved compared to the original state of analysis. Still, despite the explicit guidelines, only 29.5% of all threats to validity were addressed properly. While the maturation and the optimal sequence threats are properly addressed in 35.8% and 38.8% of all studies in our sample respectively, the carryover threat is only modeled in about 3% of the observed cases. The lack of adherence to the analysis guidelines threatens the validity of the conclusions drawn from crossover-design experiments.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3674805.3690754

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • We mitigate crossover-design threats by adjusting for them in our analyses [18], [44], which factors out effects from the phenomenon of interest and allows us to determine that they did not affect our experiment.

    “We mitigate these by adjusting for them in our analyses [18], [44], which not only factors out the effects from the phenomenon of interest but also allows us to determine that they did not affect our experiment.”

    From Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code (2026) · cites this paper as [44]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. Edward M Bennett, Renee Alpert, and AC Goldstein. 1954. Communications through limited-response questioning. Public Opinion Quarterly 18, 3 (1954), 303-308. https://doi.org/10.1086/266520 · DOI
  2. Justus Bogner, Sebastian Kotstein, and Timo Pfaff. 2023. Do RESTful API design rules have an impact on the understandability of Web APIs? Empirical software engineering 28, 6 (2023), 132. https://doi.org/10.1007/s10664-023-10367-y · DOI
  3. Hendrik Bünder and Herbert Kuchen. 2019. Towards behavior-driven graphical user interface testing. ACM SIGAPP Applied Computing Review 19, 2 (2019), 5-17. https://doi.org/10.1145/3357385.3357386 · DOI
  4. Riccardo Coppola, Tommaso Fulcini, Luca Ardito, Marco Torchiano, and Emil Alègroth. 2023. On Effectiveness and Efficiency of Gamified Exploratory GUI Testing. IEEE Transactions on Software Engineering (2023). https://doi.org/10. 1109/TSE.2023.3348036 · DOI
  5. NA Cruz, OO Melo, and CA Martinez. 2024. A correlation structure for the analysis of Gaussian and non-Gaussian responses in crossover experimental designs with repeated measures. Statistical Papers 65, 1 (2024), 263-290. https: //doi.org/10.1007/s00362-022-01391-z · DOI
  6. Nelson Alirio Cruz Gutierrez, Oscar Orlando Melo, and Carlos Alberto Martinez. 2023. Semiparametric generalized estimating equations for repeated measure- ments in cross-over designs. Statistical Methods in Medical Research 32, 5 (2023), 1033-1050. https://doi.org/10.1177/09622802231158736 · DOI
  7. Michelangelo Esposito, Simone Romano, and Giuseppe Scanniello. 2023. Test- Driven Development and Embedded Systems: An Exploratory Investigation. In 2023 49th Euromicro Conference on Software Engineering and Advanced Applica- tions (SEAA). IEEE, 239-246. https://doi.org/10.1109/SEAA60479.2023.00045 · DOI
  8. Guangchao Charles Feng. 2015. Mistakes and how to avoid mistakes in using intercoder reliability indices. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences 11, 1 (2015), 13. https://doi.org/10.1027/1614- 2241/a000086 · DOI
  9. Rita Francese, Michele Risi, and Genoveffa Tortora. 2020. miniJava: Automatic Miniaturization of Java Applications. In Proceedings of the International Conference on Advanced Visual Interfaces. 1-8. https://doi.org/10.1145/3399715.3399847 · DOI
  10. Julian Frattini, Davide Fucci, Richard Torkar, and Daniel Mendez. 2024. A Second Look at the Impact of Passive Voice Requirements on Domain Modeling: Bayesian Reanalysis of an Experiment. arXiv preprint arXiv:2402.10800 (2024). · arXiv
  11. Julian Frattini, Lloyd Montgomery, Davide Fucci, Jannik Fischbach, Michael Un- terkalmsteiner, and Daniel Mendez. 2023. Let's Stop Building at the Feet of Giants: Recovering unavailable Requirements Quality Artifacts. In Joint Proceedings of REFSQ-2023 Workshops, Doctoral Symposium, Posters & Tools Track and Journal Early Feedback co-located with the 28th International Conference on Requirements Engineering: Foundation for Software Quality (REFSQ 2023), Vol. 3378. CEUR-WS. https://doi.org/10.48550/arXiv.2304.04670 · DOI
  12. Julian Frattini, Lloyd Montgomery, Davide Fucci, Michael Unterkalmsteiner, Daniel Mendez, and Jannik Fischbach. 2024. Requirements quality research artifacts: Recovery, analysis, and management guideline. Journal of Systems and Software (2024), 112120. https://doi.org/10.1016/j.jss.2024.112120 · DOI
  13. Davide Fucci, Simone Romano, Maria Teresa Baldassarre, Danilo Caivano, Giuseppe Scanniello, Burak Turhan, and Natalia Juristo. 2018. A longitudinal cohort study on the retainment of test-driven development. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 1-10. https://doi.org/10.1145/3239235.3240502 · DOI
  14. Davide Fucci, Giuseppe Scanniello, Simone Romano, Martin Shepperd, Boyce Sigweni, Fernando Uyaguari, Burak Turhan, Natalia Juristo, and Markku Oivo. 2016. An external replication on the effects of test-driven development using a multi-site blind analysis approach. In Proceedings of the 10th ACM/IEEE Inter- national Symposium on Empirical Software Engineering and Measurement. 1-10. https://doi.org/10.1145/2961111.2962592 · DOI
  15. Jo Hannay and Magne Jørgensen. 2008. The Role of Deliberate Artificial Design Elements in Software Engineering Experiments. IEEE Transactions on Software Engineering 34, 2 (2008), 242-259. https://doi.org/10.1109/TSE.2008.13 · DOI
  16. Carlos Iñiguez-Jarrín, José Ignacio Panach, and Oscar Pastor López. 2020. Im- provement of usability in user interfaces for massive data analysis: an em- pirical study. Multimedia Tools and Applications 79, 17 (2020), 12257-12288. https://doi.org/10.1007/s11042-019-08456-6 · DOI
  17. Vigdis By Kampenes, Tore Dybå, Jo E Hannay, and Dag IK Sjøberg. 2009. A systematic review of quasi-experiments in software engineering. Information and Software Technology 51, 1 (2009), 71-82. https://doi.org/10.1016/j.infsof.2008. 04.006 · DOI
  18. Muhammad Uzair Khan, Hassan Sartaj, Muhammad Zohaib Iqbal, Muhammad Usman, and Numra Arshad. 2019. Aspectocl: using aspects to ease maintenance of evolving constraint specification. Empirical Software Engineering 24 (2019), 2674-2724. https://doi.org/10.1007/s10664-019-09717-6 · DOI
  19. Barbara Kitchenham, John Fry, and Stephen Linkman. 2003. The case against cross-over designs in software engineering. In Eleventh annual international workshop on software technology and engineering practice. IEEE, 65-67. https: //doi.org/10.1109/STEP.2003.32 · DOI
  20. Barbara Kitchenham, Lech Madeyski, and Pearl Brereton. 2019. Problems with statistical practice in human-centric software engineering experiments. In Pro- ceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering. 134-143. https://doi.org/10.1145/3319008.3319009 · DOI
  21. Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello, and Carmine Gravino. 2021. The Importance of the Correlation in Crossover Experiments. IEEE Trans- actions on Software Engineering 48, 8 (2021), 2802-2813. https://doi.org/10.1109/ TSE.2021.3070480 · DOI
  22. Barbara A Kitchenham, Tore Dyba, and Magne Jorgensen. 2004. Evidence-based software engineering. In Proceedings. 26th International Conference on Software Engineering. IEEE, 273-281. https://doi.org/10.1109/ICSE.2004.1317449 · DOI
  23. Lech Madeyski and Barbara Kitchenham. 2018. Effect sizes and their variance for AB/BA crossover design studies. In Proceedings of the 40th International Conference on Software Engineering. 420-420. https://doi.org/10.1007/s10664-017-9574-5 · DOI · Paper page
  24. Tim Menzies and Martin Shepperd. 2019. "Bad smells" in software analytics papers. Information and software technology 112 (2019), 35-47. https://doi.org/ 10.1016/j.infsof.2019.04.005 · DOI
  25. David Moreno-Lumbreras, Gregorio Robles, Daniel Izquierdo-Cortázar, and Je- sus M Gonzalez-Barahona. 2024. Software development metrics: to VR or not to VR. Empirical Software Engineering 29, 2 (2024), 1-49. https://doi.org/10.1007/ s10664-023-10435-3 · DOI
  26. Lesley M Pickard, Barbara A Kitchenham, and Peter W Jones. 1998. Combining empirical results in software engineering. Information and software technology 40, 14 (1998), 811-821. https://doi.org/10.1016/S0950-5849(98)00101-3 · DOI
  27. Paul Ralph, Nauman bin Ali, Sebastian Baltes, Domenico Bianculli, Jessica Diaz, Yvonne Dittrich, Neil Ernst, Michael Felderer, Robert Feldt, Antonio Filieri, et al. 2020. Empirical standards for software engineering research. (2020). arXiv preprint arXiv:2010.03525. · arXiv
  28. Simone Romano, Giuseppe Scanniello, Maria Teresa Baldassarre, and Davide Fucci. 2020. On the Effect of Noise on Software Engineers' Performance: Results from Two Replicated Experiments. In 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 334-341. https://doi.org/ 10.1109/SEAA51224.2020.00062 · DOI
  29. Julio Sandobalin, Emilio Insfran, and Silvia Abrahao. 2020. On the effectiveness of tools to support infrastructure as code: Model-driven versus code-centric. IEEE Access 8 (2020), 17734-17761. https://doi.org/10.1109/ACCESS.2020.2966597 · DOI
  30. Konrad Schneid, Sebastian Thöne, and Herbert Kuchen. 2022. Semi-automated test migration for BPMN-based process-driven applications. In International Conference on Enterprise Design, Operations, and Computing. Springer, 237-254. https://doi.org/10.1007/978-3-031-17604-3_14 · DOI
  31. Marina Trkman, Jan Mendling, Peter Trkman, and Marjan Krisper. 2019. Impact of the conceptual model's representation format on identifying and understanding user stories. Information and software technology 116 (2019), 106169. https: //doi.org/10.1016/j.infsof.2019.08.001 · DOI
  32. Sira Vegas, Cecilia Apa, and Natalia Juristo. 2015. Crossover designs in software engineering experiments: Benefits and perils. IEEE Transactions on Software Engineering 42, 2 (2015), 120-135. https://doi.org/10.1109/TSE.2015.2467378 · DOI · Paper page
  33. Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.

Cited by

Sorted by citations

Authors
Frattini, J., Fucci, D., & Vegas, S.
Affiliations
  • Blekinge Institute of Technology (SE)
  • Universidad Politécnica de Madrid (ES)
ORCID
Venue
Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement
Publisher
ACM
Published
2024-10-24
Pages
482-488
ISSN
1938-6451 (ISSN-L)
DOI
10.1145/3674805.3690754
OpenAlex
W4403413352
Cited by
3
References
2
Citations by year
2026: 3
FWCI
0.97
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Diva portal (Dalarna University Library) (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Funders
  • Blekinge Tekniska Högskola
  • European Regional Development Fund (MCIN/AEI/10)
  • Agencia Estatal de Investigación (13039, AEI/10)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
crossover design, software engineering experiments, residual effects, learning effects, internal validity, experimental design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review