The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
On doing relevant and rigorous experiments: Review and recommendations
Open accessAbstract
Although experiments are the gold standard for establishing causality, several threats can undermine the internal validity of experimental findings. In this article, we first discuss these threats, which include the lack of consequential decisions and outcomes, deception, demand effects and unfair comparisons, as well as issues concerning statistical validity (e.g., minimum sample size per cell, estimating variance correctly). We expose each problem, show potential solutions, and bring to the fore issues of relevance of the findings (i.e., external and ecological validity). Thereafter, we take stock of the state‐of‐the‐science regarding validity threats using a representative sample of 468 recent experiments from 258 articles published in top‐tier journals. We compare research practices in three fields of study—management, social psychology, and economics, which regularly use experimental research—to operations management, which has more recently begun to use the experimental paradigm. Our results underscore the importance for journals and authors to follow what we identify to be best‐practice methodological suggestions (i.e., the “ten commandments” of experimental research). We show that—on average—markers of methodological rigor and generalizability positively and significantly predict the citations received by published articles. Finally, given that experiments are infeasible in some settings, we conclude with a brief review of often overlooked quasi‐experimental designs, which are useful for generating strong counterfactuals and hence allow making causal claims in the field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Sample Selection Bias as a Specification Error
Heckman, J. J. (1979) · Econometrica · doi:10.2307/1912352 · Cited by 28,779
Initial conditions and moment restrictions in dynamic panel data models
Blundell, R., & Bond, S. (1998) · Journal of Econometrics · doi:10.1016/s0304-4076(98)00009-8 · Cited by 22,571
Specification Tests in Econometrics
Hausman, J. A. (1978) · Econometrica · doi:10.2307/1913827 · Cited by 18,820
Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions
Ryan, R. M., & Deci, E. L. (2000) · Contemporary Educational Psychology · doi:10.1006/ceps.1999.1020 · Cited by 18,666 · cited 1×
“…task), social structures (e.g., a participant’s status in the group), psychological rewards (e.g., praise or recognition), and even biological factors (e.g., hormones) can induce consequences as well (Benabou & Tirole, 2003; Kosfeld, Heinrichs, Zak, Fischbacher, & Fehr, 2005; Ryan & Deci, 2000).”
BackgroundThe Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 18,497 · cited 1×
“…experimental result (Smith, 1976): For instance, whereas framing effects might enhance mundane realism (e.g., cover stories), several researchers have shown that they might bias the behavioral outcomes measured in an experimental study (Andreoni, 1995; Kühberger, 1998; Tversky & Kahneman, 1981).”
BackgroundSelf-Reports in Organizational Research: Problems and Prospects
Podsakoff, P. M., & Organ, D. W. (1986) · Journal of Management · doi:10.1177/014920638601200408 · Cited by 17,102
Initial conditions and moment restrictions in dynamic panel data models
Blundell, R., & Bond, S. (1995) · Working Paper Series · doi:10.1920/wp.ifs.1995.9517 · Cited by 14,755
Design Science in Information Systems Research1
Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004) · MIS Quarterly · doi:10.2307/25148625 · Cited by 10,707
How Much Should We Trust Differences-in-Differences Estimates?
Bertrand, M., Duflo, E., & Mullainathan, S. (2004) · The Quarterly Journal of Economics · doi:10.1162/003355304772839588 · Cited by 10,641
Cited by
Cooper, B., Eva, N., Zarea Fazlelahi, F., Newman, A., Lee, A., & Obschonka, M. (2020) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2020.103472 · Cited by 339
Abusive supervision: A systematic review and fundamental rethink
Fischer, T., Tian, A. W., Lee, A., & Hughes, D. J. (2021) · The Leadership Quarterly · doi:10.1016/j.leaqua.2021.101540 · Cited by 318
Podsakoff, P. M., & Podsakoff, N. P. (2019) · The Leadership Quarterly · doi:10.1016/j.leaqua.2018.11.002 · Cited by 312
A critique of the Leader-Member Exchange construct: Back to square one
Gottfredson, R. K., Wright, S. L., & Heaphy, E. D. (2020) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101385 · Cited by 205
Humble leadership: A review and synthesis of leader expressed humility
Kelemen, T. K., Matthews, S. H., Matthews, M. J., & Henry, S. E. (2023) · Journal of Organizational Behavior · doi:10.1002/job.2608 · Cited by 203
Metadata
- Authors
- Lonati, S., Quiroga, B. F., Zehnder, C., & Antonakis, J.
- Venue
- Journal of Operations Management
- Publisher
- Wiley
- Published
- 2018-11
- Volume
- 64
- Issue
- 1
- Pages
- 19-40
- ISSN
- 0272-6963 (ISSN-L)
- 1873-1317
- OpenAlex
- W2910905828
- Cited by
- 354
- References
- 43
- FWCI
- 16.99
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: On doing relevant and rigorous experiments: Review and recommendations. https://app.cochat.ai/papers/10.1016/j.jom.2018.10.003
- The paper
- Lonati, S., Quiroga, B. F., Zehnder, C., & Antonakis, J. (2018). On doing relevant and rigorous experiments: Review and recommendations. Journal of Operations Management. https://doi.org/10.1016/j.jom.2018.10.003
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.