The DOI matches its Crossref record; 210 of the 210 references it cites were found in a registry or an index.
On doing relevant and rigorous experiments: Review and recommendations
Open accessAbstract
Although experiments are the gold standard for establishing causality, several threats can undermine the internal validity of experimental findings. In this article, we first discuss these threats, which include the lack of consequential decisions and outcomes, deception, demand effects and unfair comparisons, as well as issues concerning statistical validity (e.g., minimum sample size per cell, estimating variance correctly). We expose each problem, show potential solutions, and bring to the fore issues of relevance of the findings (i.e., external and ecological validity). Thereafter, we take stock of the state‐of‐the‐science regarding validity threats using a representative sample of 468 recent experiments from 258 articles published in top‐tier journals. We compare research practices in three fields of study—management, social psychology, and economics, which regularly use experimental research—to operations management, which has more recently begun to use the experimental paradigm. Our results underscore the importance for journals and authors to follow what we identify to be best‐practice methodological suggestions (i.e., the “ten commandments” of experimental research). We show that—on average—markers of methodological rigor and generalizability positively and significantly predict the citations received by published articles. Finally, given that experiments are infeasible in some settings, we conclude with a brief review of often overlooked quasi‐experimental designs, which are useful for generating strong counterfactuals and hence allow making causal claims in the field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.6.1173 · Cited by 91,532
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Arellano, M., & Bond, S. (1991) · The Review of Economic Studies · doi:10.2307/2297968 · Cited by 33,272
Sample Selection Bias as a Specification Error
Heckman, J. J. (1979) · Econometrica · doi:10.2307/1912352 · Cited by 28,779
(2001) · RePEc: Research Papers in Economics · Cited by 27,577
Initial conditions and moment restrictions in dynamic panel data models
Blundell, R., & Bond, S. (1998) · Journal of Econometrics · doi:10.1016/s0304-4076(98)00009-8 · Cited by 22,571
Specification Tests in Econometrics
Hausman, J. A. (1978) · Econometrica · doi:10.2307/1913827 · Cited by 18,820
Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions
Ryan, R. M., & Deci, E. L. (2000) · Contemporary Educational Psychology · doi:10.1006/ceps.1999.1020 · Cited by 18,666 · cited 1×
“…task), social structures (e.g., a participant’s status in the group), psychological rewards (e.g., praise or recognition), and even biological factors (e.g., hormones) can induce consequences as well (Benabou & Tirole, 2003; Kosfeld, Heinrichs, Zak, Fischbacher, & Fehr, 2005; Ryan & Deci, 2000).”
BackgroundLongitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 18,497 · cited 1×
“…experimental result (Smith, 1976): For instance, whereas framing effects might enhance mundane realism (e.g., cover stories), several researchers have shown that they might bias the behavioral outcomes measured in an experimental study (Andreoni, 1995; Kühberger, 1998; Tversky & Kahneman, 1981).”
Background
Cited by
Cooper, B., Eva, N., Zarea Fazlelahi, F., Newman, A., Lee, A., & Obschonka, M. (2020) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2020.103472 · Cited by 339
Abusive supervision: A systematic review and fundamental rethink
Fischer, T., Tian, A. W., Lee, A., & Hughes, D. J. (2021) · The Leadership Quarterly · doi:10.1016/j.leaqua.2021.101540 · Cited by 318
Podsakoff, P. M., & Podsakoff, N. P. (2019) · The Leadership Quarterly · doi:10.1016/j.leaqua.2018.11.002 · Cited by 312
A critique of the Leader-Member Exchange construct: Back to square one
Gottfredson, R. K., Wright, S. L., & Heaphy, E. D. (2020) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101385 · Cited by 205
Humble leadership: A review and synthesis of leader expressed humility
Kelemen, T. K., Matthews, S. H., Matthews, M. J., & Henry, S. E. (2023) · Journal of Organizational Behavior · doi:10.1002/job.2608 · Cited by 203
How follower traits and cultural values influence the effects of leadership
Matthews, S. H., Kelemen, T. K., & Bolino, M. C. (2021) · The Leadership Quarterly · doi:10.1016/j.leaqua.2021.101497 · Cited by 69
Steffens, N. K., Wolyniec, N., Okimoto, T. G., Mols, F., Haslam, S. A., & Kay, A. A. (2021) · The Leadership Quarterly · doi:10.1016/j.leaqua.2021.101498 · Cited by 61
Lee, A., Inceoglu, I., Hauser, O., & Greene, M. (2022) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101426 · Cited by 48
Do you remember? Rater memory systems and leadership measurement
Hansbrough, T. K., Lord, R. G., Schyns, B., Foti, R. J., Liden, R. C., & Acton, B. P. (2021) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101455 · Cited by 33
Güntner, A. V., Klasmeier, K. N., Klonek, F. E., & Kauffeld, S. (2021) · Journal of Leadership & Organizational Studies · doi:10.1177/15480518211012408 · Cited by 25
Social exchange theory in leadership research: A problematizing review
Madison, K., Eva, N., De Cieri, H., & Goh, Z. (2025) · The Leadership Quarterly · doi:10.1016/j.leaqua.2025.101924 · Cited by 22
A co-creation involving peripheral cues: When and how engagement size generates persuasiveness
Li, P., & Sun, Y. (2025) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2025.104311 · Cited by 6
Metadata
- Authors
- Lonati, S., Quiroga, B. F., Zehnder, C., & Antonakis, J.
- Venue
- Journal of Operations Management
- Publisher
- Wiley
- Published
- 2018-11
- Volume
- 64
- Issue
- 1
- Pages
- 19-40
- ISSN
- 0272-6963 (ISSN-L)
- 1873-1317
- OpenAlex
- W2910905828
- Cited by
- 354
- References
- 210
- FWCI
- 16.99
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.