CoChat
Sign Up
Cited by 145Software Engineering Techniques and PracticesOptimal Experimental Design MethodsAdvanced Software Engineering Methodologies

Crossover Designs in Software Engineering Experiments: Benefits and Perils

Vegas, Sira, Apa, Cecilia, Juristo, Natalia·2016·IEEE Transactions on Software Engineering
DOI 10.1109/tse.2015.2467378
Open this paper in CoChat — free account

Abstract

In experiments with crossover design subjects apply more than one treatment. Crossover designs are widespread in software engineering experimentation: they require fewer subjects and control the variability among subjects. However, some researchers disapprove of crossover designs. The main criticisms are: the carryover threat and its troublesome analysis. Carryover is the persistence of the effect of one treatment when another treatment is applied later. It may invalidate the results of an experiment. Additionally, crossover designs are often not properly designed and/or analysed, limiting the validity of the results. In this paper, we aim to make SE researchers aware of the perils of crossover experiments and provide risk avoidance good practices. We study how another discipline (medicine) runs crossover experiments. We review the SE literature and discuss which good practices tend not to be adhered to, giving advice on how they should be applied in SE experiments. We illustrate the concepts discussed analysing a crossover experiment that we have run. We conclude that crossover experiments can yield valid results, provided they are properly designed and analysed, and that, if correctly addressed, carryover is no worse than other validity threats.

Integrity signals

Registry

Crossref · 10.1109/tse.2015.2467378

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The period at which a participant reviewed a particular AOI can potentially affect participants' behavior through a maturation or exhaustion effect, making it necessary to condition on it to avoid bias.

    “In our study, for example, the period (at what time a participant reviewed a particular AOI) can potentially affect participants' behavior in what is called the maturation or exhaustion effect [18], making it necessary to condition on it to avoid bias.”

    From Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code (2026) · cites this paper as [18]

  • The crossover design induces threats to internal validity [18], such as the learning effect over multiple periods or an optimal sequence effect.

    “The crossover design induces threats to internal validity [18], e.g., the learning effect over multiple periods or an optimal sequence effect.”

    From Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code (2026) · cites this paper as [18]

  • We mitigate crossover-design threats by adjusting for them in our analyses [18], [44], which factors out effects from the phenomenon of interest and allows us to determine that they did not affect our experiment.

    “We mitigate these by adjusting for them in our analyses [18], [44], which not only factors out the effects from the phenomenon of interest but also allows us to determine that they did not affect our experiment.”

    From Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code (2026) · cites this paper as [18]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Vegas, S., Apa, C., & Juristo, N.
ORCID
Vegas, Sira
Venue
IEEE Transactions on Software Engineering
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2016-02-01
Volume
42
Issue
2
Pages
120-135
ISSN
  • 0098-5589 (ISSN-L)
  • 1939-3520
DOI
10.1109/tse.2015.2467378
OpenAlex
W2295373235
Cited by
145
References
2
Citations by year
2026: 11 · 2025: 19 · 2024: 16 · 2023: 19 · 2022: 16 · 2021: 19 · 2020: 14 · 2019: 13 · 2018: 9 · 2017: 6 · 2016: 3
FWCI
16.18
Type
journal article
Language
en
Open access
closed
Licence
ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
Locations
IEEE Transactions on Software Engineering (publishedVersion)
Funders
Spanish Ministry of Economy and Competitiveness research (TIN2011-23216)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Software Engineering Methodologies
Keywords
crossover design, software engineering experiments, residual effects, experimental design, threats to validity, experimental analysis, best practices, research methodology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review