The DOI matches its Crossref record; 211 of the 211 references it cites were found in a registry or an index.
On doing relevant and rigorous experiments: Review and recommendations
Open accessAbstract
Although experiments are the gold standard for establishing causality, several threats can undermine the internal validity of experimental findings. In this article, we first discuss these threats, which include the lack of consequential decisions and outcomes, deception, demand effects and unfair comparisons, as well as issues concerning statistical validity (e.g., minimum sample size per cell, estimating variance correctly). We expose each problem, show potential solutions, and bring to the fore issues of relevance of the findings (i.e., external and ecological validity). Thereafter, we take stock of the state‐of‐the‐science regarding validity threats using a representative sample of 468 recent experiments from 258 articles published in top‐tier journals. We compare research practices in three fields of study—management, social psychology, and economics, which regularly use experimental research—to operations management, which has more recently begun to use the experimental paradigm. Our results underscore the importance for journals and authors to follow what we identify to be best‐practice methodological suggestions (i.e., the “ten commandments” of experimental research). We show that—on average—markers of methodological rigor and generalizability positively and significantly predict the citations received by published articles. Finally, given that experiments are infeasible in some settings, we conclude with a brief review of often overlooked quasi‐experimental designs, which are useful for generating strong counterfactuals and hence allow making causal claims in the field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.6.1173 · Cited by 91,532
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Arellano, M., & Bond, S. (1991) · The Review of Economic Studies · doi:10.2307/2297968 · Cited by 33,272
Sample Selection Bias as a Specification Error
Heckman, J. J. (1979) · Econometrica · doi:10.2307/1912352 · Cited by 28,779
(2001) · RePEc: Research Papers in Economics · Cited by 27,577
Initial conditions and moment restrictions in dynamic panel data models
Blundell, R., & Bond, S. (1998) · Journal of Econometrics · doi:10.1016/s0304-4076(98)00009-8 · Cited by 22,571
Specification Tests in Econometrics
Hausman, J. A. (1978) · Econometrica · doi:10.2307/1913827 · Cited by 18,820
Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions
Ryan, R. M., & Deci, E. L. (2000) · Contemporary Educational Psychology · doi:10.1006/ceps.1999.1020 · Cited by 18,666 · cited 1×
“…task), social structures (e.g., a participant’s status in the group), psychological rewards (e.g., praise or recognition), and even biological factors (e.g., hormones) can induce consequences as well (Benabou & Tirole, 2003; Kosfeld, Heinrichs, Zak, Fischbacher, & Fehr, 2005; Ryan & Deci, 2000).”
BackgroundLongitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 18,497 · cited 1×
“…experimental result (Smith, 1976): For instance, whereas framing effects might enhance mundane realism (e.g., cover stories), several researchers have shown that they might bias the behavioral outcomes measured in an experimental study (Andreoni, 1995; Kühberger, 1998; Tversky & Kahneman, 1981).”
BackgroundSelf-Reports in Organizational Research: Problems and Prospects
Podsakoff, P. M., & Organ, D. W. (1986) · Journal of Management · doi:10.1177/014920638601200408 · Cited by 17,102 · cited 1×
“Hypothetical studies, similarly to standard questionnaires, suffer from the well-known problems of self-reported measures (Donaldson & Grant-Vallone, 2002; P. M. Podsakoff & Organ, 1986); it is uncertain whether the self-reported intentions will ultimately translate into real actions.”
ResultTheory Building From Cases: Opportunities And Challenges
Eisenhardt, K. M., & Graebner, M. E. (2007) · Academy of Management Journal · doi:10.5465/amj.2007.24160888 · Cited by 16,670
Sources of method bias in social science research and recommendations on how to control it
Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012) · Annual Review of Psychology · doi:10.1146/annurev-psych-120710-100452 · Cited by 15,744
Initial conditions and moment restrictions in dynamic panel data models
Blundell, R., & Bond, S. (1995) · Working Paper Series · doi:10.1920/wp.ifs.1995.9517 · Cited by 14,755
Introductory Econometrics: A Modern Approach
Wooldridge, J. M. (1999) · Cited by 14,451
(2002) · Cited by 13,211
Design Science in Information Systems Research1
Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004) · MIS Quarterly · doi:10.2307/25148625 · Cited by 10,707
How Much Should We Trust Differences-in-Differences Estimates?
Bertrand, M., Duflo, E., & Mullainathan, S. (2004) · The Quarterly Journal of Economics · doi:10.1162/003355304772839588 · Cited by 10,641
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436
Design science in information systems research
(2004) · Journal of the Association for Information Systems · doi:10.5555/2017212.2017217 · Cited by 8,254
A Design Science Research Methodology for Information Systems Research
Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007) · Journal of Management Information Systems · doi:10.2753/mis0742-1222240302 · Cited by 7,832
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 5×
“sensitive to outliers (Aguinis, Gottfredson, & Joo, 2013), are heavily unstable with respect to the analytical strategy chosen (Button et al., 2013), and have low statistical power (Ioannidis, 2005);”
BackgroundInfluential(2005) · PubMed · Cited by 7,011
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Abadie, A., Diamond, A., & Hainmueller, J. (2010) · Journal of the American Statistical Association · doi:10.1198/jasa.2009.ap08746 · Cited by 5,513
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2008) · doi:10.2307/j.ctvcm4j72 · Cited by 5,051
A Practitioner’s Guide to Cluster-Robust Inference
Colin Cameron, A., & Miller, D. L. (2015) · Journal of Human Resources · doi:10.3368/jhr.50.2.317 · Cited by 5,037
Behavioral Study of obedience.
Milgram, S. (1963) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0040525 · Cited by 4,918 · cited 1×
“First, the hierarchical relationship existing between experimenter and participants might push the subjects to naturally conform to her or his authority (see the pioneering studies on the natural obendience to authority of Milgram, 1963).”
BackgroundWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“…& Joo, 2013), are heavily unstable with respect to the analytical strategy chosen (Button et al., 2013), and have low statistical power (Ioannidis, 2005); at the research field level, small sample sizes tend to inflate the effect size of the treatment, given that only significant…”
BackgroundRegression discontinuity designs in economics
Lee, D. S., & Lemieux, T. (2010) · Journal of Economic Literature · doi:10.1257/jel.48.2.281 · Cited by 4,534
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
What Constitutes a Theoretical Contribution?
Whetten, D. A. (1989) · The Academy of Management Review · doi:10.2307/258554 · Cited by 4,129
INDIVIDUAL RISK ATTITUDES: MEASUREMENT, DETERMINANTS, AND BEHAVIORAL CONSEQUENCES
Dohmen, T., Falk, A., Huffman, D., Sunde, U., Schupp, J., & Wagner, G. G. (2011) · Journal of the European Economic Association · doi:10.1111/j.1542-4774.2011.01015.x · Cited by 3,896
Regression discontinuity designs: A guide to practice
Imbens, G. W., & Lemieux, T. (2008) · Journal of Econometrics · doi:10.1016/j.jeconom.2007.05.001 · Cited by 3,694
Oxytocin increases trust in humans
Kosfeld, M., Heinrichs, M., Zak, P. J., Fischbacher, U., & Fehr, E. (2005) · Nature · doi:10.1038/nature03701 · Cited by 3,690
Methodological fit in management field research
Edmondson, A. C., & Mcmanus, S. E. (2007) · Academy of Management Review · doi:10.5465/amr.2007.26586086 · Cited by 3,227
Robust Inference With Multiway Clustering
Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2011) · Journal of Business & Economic Statistics · doi:10.1198/jbes.2010.07136 · Cited by 3,203
Intrinsic and extrinsic motivation
Benabou, R., & Tirole, J. (2003) · Review of Economic Studies · doi:10.1111/1467-937x.00253 · Cited by 2,913
Formulation and estimation of dynamic models using panel data
Anderson, T. W., & Hsiao, C. (1982) · Journal of Econometrics · doi:10.1016/0304-4076(82)90095-1 · Cited by 2,849
Social Desirability Bias and the Validity of Indirect Questioning
Fisher, R. J. (1993) · Journal of Consumer Research · doi:10.1086/209351 · Cited by 2,789
What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?
Levitt, S. D., & List, J. A. (2007) · Journal of Economic Perspectives · doi:10.1257/jep.21.2.153 · Cited by 2,668
Modeling Managerial Behavior: Misperceptions of Feedback in a Dynamic Decision Making Experiment
Sterman, J. D. (1989) · Management Science · doi:10.1287/mnsc.35.3.321 · Cited by 2,500
Endogeneity in Empirical Corporate Finance1
Roberts, M. R., & Whited, T. M. (2013) · Handbook of the Economics of Finance · doi:10.1016/b978-0-44-453594-8.00007-0 · Cited by 2,459
On making causal claims: A review and recommendations
Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2010) · The Leadership Quarterly · doi:10.1016/j.leaqua.2010.10.010 · Cited by 2,441
Self-citationComparative Politics and the Synthetic Control Method
Abadie, A., Diamond, A., & Hainmueller, J. (2015) · American Journal of Political Science · doi:10.1111/ajps.12116 · Cited by 2,406
Organizational Theories: Some Criteria for Evaluation
Bacharach, S. B. (1989) · The Academy of Management Review · doi:10.5465/amr.1989.4308374 · Cited by 2,305
Card, D., & Krueger, A. B. (2000) · American Economic Review · doi:10.1257/aer.90.5.1397 · Cited by 2,206
Pay Enough or Don't Pay at All*
Gneezy, U., & Rustichini, A. (2000) · Quarterly Journal of Economics · doi:10.1162/003355300554917 · Cited by 2,175
Handbook of Research Methods in Social and Personality Psychology
(2024) · Cambridge University Press eBooks · doi:10.1017/9781009170123 · Cited by 2,121
Minimum Wages and Employment: A Case Study of the Fast Food Industry in New Jersey and Pennsylvania
Card, D., & Krueger, A. (1993) · National Bureau of Economic Research · doi:10.3386/w4509 · Cited by 2,063
The Handbook of Experimental Economics
(1995) · Princeton University Press eBooks · doi:10.1515/9780691213255 · Cited by 2,051
Aguinis, H., & Bradley, K. J. (2014) · Organizational Research Methods · doi:10.1177/1094428114547952 · Cited by 2,021
Flammer, C. (2015) · Management Science · doi:10.1287/mnsc.2014.2038 · Cited by 1,949
Handbook of Research Methods in Social and Personality Psychology
(2014) · Cambridge University Press eBooks · doi:10.1017/cbo9780511996481 · Cited by 1,942
Intention-to-treat concept: A review
Gupta, S. (2011) · Perspectives in Clinical Research · doi:10.4103/2229-3485.83221 · Cited by 1,921
Feynman, R. (2017) · The MIT Press eBooks · doi:10.7551/mitpress/11068.001.0001 · Cited by 1,881
Gardner, W. P., Mulvey, E. P., & Shaw, E. C. (1995) · Psychological Bulletin · doi:10.1037//0033-2909.118.3.392 · Cited by 1,836
Sustainable supply chains: An introduction
Linton, J. D., Klassen, R., & Jayaraman, V. (2007) · Journal of Operations Management · doi:10.1016/j.jom.2007.01.012 · Cited by 1,746
Sustainable Operations Management
Kleindorfer, P. R., Singhal, K., & VanWassenhove, L. N. (2005) · Production and Operations Management · doi:10.1111/j.1937-5956.2005.tb00235.x · Cited by 1,746
Latent Curve Models: A Structural Equation Perspective
Bollen, K. A., & Curran, P. J. (2005) · Wiley Series in Probability and Statistics · doi:10.1002/0471746096 · Cited by 1,729
Galinsky, A. D., Gruenfeld, D. H., & Magee, J. C. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.85.3.453 · Cited by 1,680
Building operations management theory through case and field research
Meredith, J. (1998) · Journal of Operations Management · doi:10.1016/s0272-6963(98)00023-0 · Cited by 1,657
Baumeister, R. F., Vohs, K. D., & Funder, D. C. (2007) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2007.00051.x · Cited by 1,629
The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework
Camerer, C. F., Hogarth, R. M., Budescu, D. V., & Eckel, C. (1999) · Elicitation of Preferences · doi:10.1007/978-94-017-1406-8_2 · Cited by 1,545
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,529
Inferences of competence from faces predict election outcomes
Todorov, A., Mandisodza, A. N., Goren, A., & Hall, C. C. (2005) · Science · doi:10.1126/science.1110589 · Cited by 1,508
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework
Camerer, C. F., & Hogarth, R. M. (1999) · Journal of Risk and Uncertainty · doi:10.1023/a:1007850605129 · Cited by 1,428
Barriers to the Advance of Organizational Science: Paradigm Development as a Dependent Variable
Pfeffer, J. (1993) · The Academy of Management Review · doi:10.2307/258592 · Cited by 1,402
Best-practice recommendations for defining, identifying, and handling outliers
Aguinis, H., Gottfredson, R. K., & Joo, H. (2013) · Organizational Research Methods · doi:10.1177/1094428112470848 · Cited by 1,341
Correcting for Endogeneity in Strategic Management Research
Hamilton, B. H., & Nickerson, J. A. (2003) · Strategic Organization · doi:10.1177/1476127003001001218 · Cited by 1,309
Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L., & Bailenson, J. N. (2002) · Psychological Inquiry · doi:10.1207/s15327965pli1302_01 · Cited by 1,280
Experimental research in financial accounting
Libby, R., Bloomfield, R., & Nelson, M. W. (2002) · Accounting, Organizations and Society · doi:10.1016/s0361-3682(01)00011-3 · Cited by 1,201
Experimenter demand effects in economic experiments
Zizzo, D. J. (2010) · Experimental Economics · doi:10.1007/s10683-009-9230-z · Cited by 1,194
Using Randomization in Development Economics Research: A Toolkit
Duflo, E., Glennerster, R., & Kremer, M. (2006) · National Bureau of Economic Research · doi:10.3386/t0333 · Cited by 1,191
Factors relevant to the validity of experiments in social settings.
Campbell, D. T. (1957) · Psychological Bulletin · doi:10.1037/h0040950 · Cited by 1,140
Experimental Economics: Induced Value Theory
(1991) · Papers in Experimental Economics · doi:10.1017/cbo9780511528354.008 · Cited by 1,123
The Influence of Framing on Risky Decisions: A Meta-analysis
Kühberger, A. (1998) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1998.2781 · Cited by 1,121
Grant, A. M. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.1.108 · Cited by 1,114 · cited 1×
“For instance, when intrinsic motivation is large, additional extrinsic motivators will not necessarily be effective and might even backfire (Ariely, Gneezy, Loewenstein, & Mazar, 2009; Gneezy & Rustichini, 2000; Grant, 2008).”
BackgroundDecision Bias in the Newsvendor Problem with a Known Demand Distribution: Experimental Evidence
Schweitzer, M. E., & Cachon, G. P. (2000) · Management Science · doi:10.1287/mnsc.46.3.404.12070 · Cited by 1,083
Understanding Self-Report Bias in Organizational Behavior Research
Donaldson, S. I., & Grant-Vallone, E. J. (2002) · Journal of Business and Psychology · doi:10.1023/a:1019637632584 · Cited by 1,068
Wacker, J. G. (1998) · Journal of Operations Management · doi:10.1016/s0272-6963(98)00019-9 · Cited by 1,042
Lab Experiments Are a Major Source of Knowledge in the Social Sciences
Falk, A., & Heckman, J. J. (2009) · Science · doi:10.1126/science.1168244 · Cited by 1,033
Davis, D. D., & Holt, C. A. (1993) · Princeton University Press eBooks · doi:10.1515/9780691233376 · Cited by 971
Andreoni, J. (1995) · The Quarterly Journal of Economics · doi:10.2307/2118508 · Cited by 912
Power Posing: Brief Nonverbal Displays Affect Neuroendocrine Levels and Risk Tolerance
Carney, D. R., Cuddy, A. J. C., & Yap, A. J. (2010) · Psychological Science · doi:10.1177/0956797610383437 · Cited by 873
How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
Montgomery, J. M., Nyhan, B., & Torres, M. (2018) · American Journal of Political Science · doi:10.1111/ajps.12357 · Cited by 800
ARIELY, D., GNEEZY, U., LOEWENSTEIN, G., & MAZAR, N. (2009) · Review of Economic Studies · doi:10.1111/j.1467-937x.2009.00534.x · Cited by 797
Do Professional Traders Exhibit Myopic Loss Aversion? An Experimental Analysis
HAIGH, M. S., & LIST, J. A. (2005) · The Journal of Finance · doi:10.1111/j.1540-6261.2005.00737.x · Cited by 754
Van Aken, J. E. (2005) · British Journal of Management · doi:10.1111/j.1467-8551.2005.00437.x · Cited by 744
Checking the Success of Manipulations in Marketing Experiments
Perdue, B. C., & Summers, J. O. (1986) · Journal of Marketing Research · doi:10.1177/002224378602300401 · Cited by 740
Semadeni, M., Withers, M. C., & Trevis Certo, S. (2014) · Strategic Management Journal · doi:10.1002/smj.2136 · Cited by 739
Experimental practices in economics: A methodological challenge for psychologists?
Hertwig, R., & Ortmann, A. (2001) · Behavioral and Brain Sciences · doi:10.1017/s0140525x01004149 · Cited by 733
External validity is more than skin deep: Some answers to criticisms of laboratory experiments.
Berkowitz, L., & Donnerstein, E. (1982) · American Psychologist · doi:10.1037//0003-066x.37.3.245 · Cited by 713
Bridging Practice and Theory: A Design Science Approach
Holmström, J., Ketokivi, M., & Hameri, A. (2009) · Decision Sciences · doi:10.1111/j.1540-5915.2008.00221.x · Cited by 674
Experimental practices in economics: A challenge for psychologists?: (683322011-032)
Hertwig, R., & Ortmann, A. (2000) · PsycEXTRA Dataset · doi:10.1037/e683322011-032 · Cited by 668
HOSODA, M., STONE‐ROMERO, E. F., & COATS, G. (2003) · Personnel Psychology · doi:10.1111/j.1744-6570.2003.tb00157.x · Cited by 622
MONETARY REWARDS AND DECISION COST IN EXPERIMENTAL ECONOMICS
SMITH, V. L., & WALKER, J. M. (1993) · Economic Inquiry · doi:10.1111/j.1465-7295.1993.tb00881.x · Cited by 560
Do Workers Work More if Wages Are High? Evidence from a Randomized Field Experiment
Fehr, E., & Goette, L. (2007) · American Economic Review · doi:10.1257/aer.97.1.298 · Cited by 557
Behavior in operations management: Assessing recent findings and revisiting old assumptions
Bendoly, E., Donohue, K., & Schultz, K. L. (2006) · Journal of Operations Management · doi:10.1016/j.jom.2005.10.001 · Cited by 550
de Treville, S., & Antonakis, J. (2006) · Journal of Operations Management · doi:10.1016/j.jom.2005.04.001 · Cited by 549
Self-citationAbbey, J. D., & Meloy, M. G. (2017) · Journal of Operations Management · doi:10.1016/j.jom.2017.06.001 · Cited by 522
Research in the Psychological Laboratory: Truth or Triviality?
Anderson, C. A., Lindsay, J. J., & Bushman, B. J. (1999) · Current Directions in Psychological Science · doi:10.1111/1467-8721.00002 · Cited by 505
Toward a Theory of Behavioral Operations
Gino, F., & Pisano, G. (2008) · Manufacturing & Service Operations Management · doi:10.1287/msom.1070.0205 · Cited by 497
Notes from the Editors: Redefining some methodological criteria for the journal⋆
Guide, V. D. R., & Ketokivi, M. (2015) · Journal of Operations Management · doi:10.1016/s0272-6963(15)00056-x · Cited by 496
The Evolution of Cooperation in Infinitely Repeated Games: Experimental Evidence
Dal Bó, P., & Fréchette, G. R. (2011) · American Economic Review · doi:10.1257/aer.101.1.411 · Cited by 495
Aldad, T. S., Gan, G., Gao, X.-B., & Taylor, H. S. (2013) · Scientific Reports · doi:10.1038/srep01320 · Cited by 487
On doing better science: From thrill of discovery to policy implications
Antonakis, J. (2017) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.01.006 · Cited by 475
Self-citationTent Poles, Tribalism, and Boundary Spanning: The Rigor-Relevance Debate in Management Research
Gulati, R. (2007) · Academy of Management Journal · doi:10.5465/amj.2007.26279170 · Cited by 461
Autoregressive Latent Trajectory (ALT) Models A Synthesis of Two Traditions
Bollen, K. A., & Curran, P. J. (2004) · Sociological Methods & Research · doi:10.1177/0049124103260222 · Cited by 447 · cited 1×
“…refer to more extensive treatments of this topic to ensure consistency of inference in more complex situations, including where clustering is multiway or disturbances are autoregressive (e.g., Angrist & Pischke, 2008; Bollen & Curran, 2004; Cameron, Gelbach, & Miller, 2011; Cameron & Miller, 2015).”
BackgroundFalk, A., & Szech, N. (2013) · Science · doi:10.1126/science.1231566 · Cited by 439
Estimating the Impact of State Policies and Institutions with Mixed-Level Data
Primo, D. M., Jacobsmeier, M. L., & Milyo, J. (2007) · State Politics & Policy Quarterly · doi:10.1177/153244000700700405 · Cited by 437
Fine-Tuning of Algorithms Using Fractional Experimental Designs and Local Search
Adenso-Díaz, B., & Laguna, M. (2006) · Operations Research · doi:10.1287/opre.1050.0243 · Cited by 435
Hartmann, J., & Moeller, S. (2014) · Journal of Operations Management · doi:10.1016/j.jom.2014.01.005 · Cited by 433
Ketokivi, M., & McIntosh, C. N. (2017) · Journal of Operations Management · doi:10.1016/j.jom.2017.05.001 · Cited by 429
Trust in Forecast Information Sharing
Özer, Ö., Zheng, Y., & Chen, K.-Y. (2011) · Management Science · doi:10.1287/mnsc.1110.1334 · Cited by 421
The Role of Representative Design in an Ecological Approach to Cognition.
Dhami, M. K., Hertwig, R., & Hoffrage, U. (2004) · Psychological Bulletin · doi:10.1037/0033-2909.130.6.959 · Cited by 408
Synth: An R Package for Synthetic Control Methods in Comparative Case Studies
Abadie, A., Diamond, A., & Hainmueller, J. (2011) · Journal of Statistical Software · doi:10.18637/jss.v042.i13 · Cited by 378
Building long‐term orientation in buyer–supplier relationships: The moderating role of culture
Cannon, J. P., Doney, P. M., Mullen, M. R., & Petersen, K. J. (2010) · Journal of Operations Management · doi:10.1016/j.jom.2010.02.002 · Cited by 359
The effects of (incentivized) belief elicitation in public goods experiments
Gächter, S., & Renner, E. (2010) · Experimental Economics · doi:10.1007/s10683-010-9246-4 · Cited by 359
Emsley, R., Dunn, G., & White, I. R. (2010) · Statistical Methods in Medical Research · doi:10.1177/0962280209105014 · Cited by 356
Designing experiments that generalize
Highhouse, S. (2009) · Organizational Research Methods · doi:10.1177/1094428107300396 · Cited by 353 · cited 4×
“rather than tools for assessing the size of a statistical effect (Highhouse, 2009; Rubinstein, 2001), they are useful for capturing directional effects of theoretically informed treatments, yet await field tests for a more precise assessment of the practical effect.”
BackgroundInfluentialCamerer, C. F. (2015) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195328325.003.0016 · Cited by 344
Revisiting Truth or Triviality: The External Validity of Research in the Psychological Laboratory
Mitchell, G. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611432343 · Cited by 343
Bodies of Knowledge for Research in Behavioral Operations
Bendoly, E., Croson, R., Goncalves, P., & Schultz, K. (2010) · Production and Operations Management · doi:10.1111/j.1937-5956.2009.01108.x · Cited by 337
A General Panel Model with Random and Fixed Effects: A Structural Equations Approach
Bollen, K. A., & Brand, J. E. (2010) · Social Forces · doi:10.1353/sof.2010.0072 · Cited by 318
Naor, M., Linderman, K., & Schroeder, R. (2010) · Journal of Operations Management · doi:10.1016/j.jom.2009.11.001 · Cited by 312
Podsakoff, P. M., & Podsakoff, N. P. (2019) · The Leadership Quarterly · doi:10.1016/j.leaqua.2018.11.002 · Cited by 312
Leadership Process Models: A Review and Synthesis
Fischer, T., Dietz, J., & Antonakis, J. (2017) · Journal of Management · doi:10.1177/0149206316682830 · Cited by 308
Self-citationAntonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2014) · Oxford Handbooks Online · doi:10.1093/oxfordhb/9780199755615.013.007 · Cited by 305
Self-citationPromises and Perils of Pre-Analysis Plans
Olken, B. A. (2015) · Journal of Economic Perspectives · doi:10.1257/jep.29.3.61 · Cited by 299
Belief elicitation in experiments: is there a hedging problem?
Blanco, M., Engelmann, D., Koch, A. K., & Normann, H.-T. (2010) · Experimental Economics · doi:10.1007/s10683-010-9249-1 · Cited by 297
van Aken, J., Chandrasekaran, A., & Halman, J. (2016) · Journal of Operations Management · doi:10.1016/j.jom.2016.06.004 · Cited by 290
Viewpoint: On the generalizability of lab behaviour to the field
Levitt, S. D., & List, J. A. (2007) · Canadian Journal of Economics/Revue canadienne d'économique · doi:10.1111/j.1365-2966.2007.00412.x · Cited by 289
E XPERIMENTAL M ETHODS IN P OLITICAL S CIENCE
McDermott, R. (2002) · Annual Review of Political Science · doi:10.1146/annurev.polisci.5.091001.170657 · Cited by 285
Grant, A. M., & Wall, T. D. (2009) · Organizational Research Methods · doi:10.1177/1094428108320737 · Cited by 284
Managers and Students as Newsvendors
Bolton, G. E., Ockenfels, A., & Thonemann, U. W. (2012) · Management Science · doi:10.1287/mnsc.1120.1550 · Cited by 278
Fairness in supply chain contracts: A laboratory study
Katok, E., & Pavlov, V. (2013) · Journal of Operations Management · doi:10.1016/j.jom.2013.01.001 · Cited by 274
Methodological Wishes for the Next Decade and How to Make Wishes Come True
Aguinis, H., & Edwards, J. R. (2014) · Journal of Management Studies · doi:10.1111/joms.12058 · Cited by 267
Artificiality: The tension between internal and external validity in economic experiments
Schram, A. (2005) · Journal of Economic Methodology · doi:10.1080/13501780500086081 · Cited by 260
Experimental subjects are not different
Exadaktylos, F., Espín, A. M., & Brañas-Garza, P. (2013) · Scientific Reports · doi:10.1038/srep01213 · Cited by 257
Personality psychology as a truly behavioural science
Furr, R. M. (2009) · European Journal of Personality · doi:10.1002/per.724 · Cited by 255
Gunst, R. F., & Mason, R. L. (2009) · WIREs Computational Statistics · doi:10.1002/wics.27 · Cited by 253
Özer, Ö., Zheng, Y., & Ren, Y. (2014) · Management Science · doi:10.1287/mnsc.2014.1905 · Cited by 245
On Rigor and Relevance: Fostering Dialectic Progress in Management Research
Vermeulen, F. (2005) · Academy of Management Journal · doi:10.5465/amj.2005.19573102 · Cited by 243
DO LAB EXPERIMENTS MISREPRESENT SOCIAL PREFERENCES? THE CASE OF SELF-SELECTED STUDENT SAMPLES
Falk, A., Meier, S., & Zehnder, C. (2013) · Journal of the European Economic Association · doi:10.1111/jeea.12019 · Cited by 243
Self-citation"High-dilution" experiments a delusion
Maddox, J., Randi, J., & Stewart, W. W. (1988) · Nature · doi:10.1038/334287a0 · Cited by 238
Behavioral operations: The state of the field
Croson, R., Schultz, K., Siemsen, E., & Yeo, M. L. (2013) · Journal of Operations Management · doi:10.1016/j.jom.2012.12.001 · Cited by 238
Ranehill, E., Dreber, A., Johannesson, M., Leiberg, S., Sul, S., & Weber, R. A. (2015) · Psychological Science · doi:10.1177/0956797614553946 · Cited by 233
The Issue of Endogeneity within Theory-Based, Quantitative Management Accounting Research
Chenhall, R. H., & Moers, F. (2007) · European Accounting Review · doi:10.1080/09638180701265937 · Cited by 229
Cooper, D. J., Kagel, J. H., Lo, W., & Gu, Q. L. (1999) · American Economic Review · doi:10.1257/aer.89.4.781 · Cited by 220
Sigall, H., & Mills, J. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_5 · Cited by 217
Camerer, C. F. (2011) · SSRN Electronic Journal · doi:10.2139/ssrn.1977749 · Cited by 210
Running Randomized Evaluations: A Practical Guide
GLENNERSTER, R., & TAKAVARASHA, K. (2013) · doi:10.2307/j.ctt4cgd52 · Cited by 209
An Experimental Template for Case Study Research
Gerring, J., & McDermott, R. (2007) · American Journal of Political Science · doi:10.1111/j.1540-5907.2007.00275.x · Cited by 208
From the Editors Publishing Laboratory Research inAMJ: A Question of When, Not If
Colquitt, J. A. (2008) · Academy of Management Journal · doi:10.5465/amj.2008.33664717 · Cited by 206
Cooper, W. H., & Richardson, A. J. (1986) · Journal of Applied Psychology · doi:10.1037//0021-9010.71.2.179 · Cited by 202
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2205186 · Cited by 201
The queen bee: A myth? The effect of top-level female leadership on subordinate females
Arvate, P. R., Galilea, G. W., & Todescat, I. (2018) · The Leadership Quarterly · doi:10.1016/j.leaqua.2018.03.002 · Cited by 197
Experiments in economics: External validity and the robustness of phenomena
Guala, F., & Mittone, L. (2005) · Journal of Economic Methodology · doi:10.1080/13501780500342906 · Cited by 192
Aiken, L. S., West, S. G., & Millsap, R. E. (2008) · American Psychologist · doi:10.1037/0003-066x.63.1.32 · Cited by 187
Multiple Hypothesis Testing in Experimental Economics
List, J., Shaikh, A., & Xu, Y. (2016) · National Bureau of Economic Research · doi:10.3386/w21875 · Cited by 187
Peer effects on worker output in the laboratory generalize to the field
Herbst, D., & Mas, A. (2015) · Science · doi:10.1126/science.aac9555 · Cited by 185
An Experimental Examination of Intrinsic Values as a
Boyce, R. R., Brown, T. C., McClelland, G. H., Peterson, G. L., & Schulze, W. D. (1992) · Cited by 182
Structural Estimation of the Newsvendor Model: An Application to Reserving Operating Room Time
Olivares, M., Terwiesch, C., & Cassorla, L. (2008) · Management Science · doi:10.1287/mnsc.1070.0756 · Cited by 181
“Just Words? Just Speeches?” On the Economic Value of Charismatic Leadership
Antonakis, J., D’Adda, G., Weber, R. A., & Zehnder, C. (2021) · Management Science · doi:10.1287/mnsc.2021.4219 · Cited by 179
Self-citationSession-effects in the laboratory
Fréchette, G. R. (2012) · Experimental Economics · doi:10.1007/s10683-011-9309-1 · Cited by 172
The Costs of Deception: Evidence from Psychology
Ortmann, A., & Hertwig, R. (2002) · Experimental Economics · doi:10.1023/a:1020365204768 · Cited by 171
Field experiments in strategy research
Chatterji, A. K., Findley, M., Jensen, N. M., Meier, S., & Nielson, D. (2016) · Strategic Management Journal · doi:10.1002/smj.2449 · Cited by 168
Leader corruption depends on power and testosterone
Bendahan, S., Zehnder, C., Pralong, F. P., & Antonakis, J. (2015) · The Leadership Quarterly · doi:10.1016/j.leaqua.2014.07.010 · Cited by 168
Self-citationField Experiments in Organizations
Eden, D. (2017) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-041015-062400 · Cited by 167
Will Economics Become an Experimental Science?
Plott, C. R. (1991) · Southern Economic Journal · doi:10.2307/1060322 · Cited by 165
A comprehensive comparison of students and non-students in classic experimental games
Belot, M. V. K., Duch, R. M., & Miller, L. M. (2015) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2015.02.007 · Cited by 165
From the Editors Publishing Laboratory Research in AMJ: A Question of When, Not If
Colquitt, J. A. (2008) · Academy of Management Journal · doi:10.5465/amr.2008.33664717 · Cited by 165
PRACTITIONERS CORNER: An Advantage of the Linear Probability Model over Probit or Logit
Caudill, S. B. (1988) · Oxford Bulletin of Economics and Statistics · doi:10.1111/j.1468-0084.1988.mp50004005.x · Cited by 164
An Experimental Examination of Intrinsic Values as a Source of the WTA-WTP Disparity
Boyce, R. R., Brown, T. C., McClelland, G. H., Peterson, G. L., & Schulze, W. D. (2010) · Cited by 162
The impact of cultural differences on buyer–supplier negotiations: An experimental study
Ribbink, D., & Grimm, C. M. (2014) · Journal of Operations Management · doi:10.1016/j.jom.2014.01.004 · Cited by 160
Laboratory Experimentation in Economics: Six Points of View
(1987) · Cambridge University Press eBooks · doi:10.1017/cbo9780511528316 · Cited by 152
The External Validity of Laboratory Experiments: The Misleading Emphasis on Quantitative Effects
Kessler, J. B., & Lise Vesterlund, C. (2015) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195328325.003.0020 · Cited by 149
Some predictors of SMJ article impact
Bergh, D. D., Perry, J., & hanke, R. (2006) · Strategic Management Journal · doi:10.1002/smj.504 · Cited by 148
What makes articles highly cited?
Antonakis, J., Bastardoz, N., Liu, Y., & Schriesheim, C. A. (2014) · The Leadership Quarterly · doi:10.1016/j.leaqua.2013.10.014 · Cited by 148
Self-citationA theorist's view of experiments
Rubinstein, A. (2001) · European Economic Review · doi:10.1016/s0014-2921(01)00104-0 · Cited by 145
Chen-Ritzo, C.-H., Harrison, T. P., Kwasnica, A. M., & Thomas, D. J. (2005) · Management Science · doi:10.1287/mnsc.1050.0433 · Cited by 145
Experimental Economics and Supply-Chain Management
Croson, R. T. A., & Donohue, K. L. (2002) · INFORMS Journal on Applied Analytics · doi:10.1287/inte.32.5.74.37 · Cited by 128
Experimental Localism and External Validity
Guala, F. (2003) · Philosophy of Science · doi:10.1086/377400 · Cited by 126
To deceive or not to deceive: The effect of deception on behavior in future laboratory experiments
Jamison, J., Karlan, D., & Schechter, L. (2008) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2008.09.002 · Cited by 126
Podsakoff, N. P., Podsakoff, P. M., MACKENZIE, S. B., & Klinger, R. L. (2012) · Journal of Applied Psychology · doi:10.1037/a0029570 · Cited by 121
Running Randomized Evaluations: A Practical Guide
Glennerster, R., & Takavarasha, K. (2013) · Economics Books · Cited by 114
Schaerer, M., du Plessis, C., Yap, A. J., & Thau, S. (2018) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.08.004 · Cited by 114
QUESTIONS TO CONSIDER WHEN SELECTING STUDENT SAMPLES
Stevens, C. K. (2011) · Journal of Supply Chain Management · doi:10.1111/j.1745-493x.2011.03233.x · Cited by 110
Field Experiments and Natural Experiments
Gerber, A. S., & GREEN, D. P. (2009) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199286546.003.0015 · Cited by 109
Psychology and Experimental Economics: A Gap in Abstraction
Dan Ariely, C., & Norton, M. I. (2007) · Current Directions in Psychological Science · doi:10.1111/j.1467-8721.2007.00531.x · Cited by 89
An Additional Rating Method for Journal Articles in the Field of Management
Kacmar, K. M., & Whitfield, J. M. (2000) · Organizational Research Methods · doi:10.1177/109442810034005 · Cited by 86
Using Laboratory Experiments to Build Better Operations Management Models
Katok, E. (2011) · now publishers, Inc. eBooks · doi:10.1561/0200000022 · Cited by 85
Impact of Lean Production on Perceived Job Autonomy and Job Satisfaction: An Experimental Study
Rodríguez, D., Buyens, D., Landeghem, H. V., & Lasio, V. (2015) · Human Factors and Ergonomics in Manufacturing & Service Industries · doi:10.1002/hfm.20620 · Cited by 81
7. Experimental Economics across Subject Populations
Fréchette, G. R. (2016) · Princeton University Press eBooks · doi:10.1515/9781400883172-008 · Cited by 74
On the Use of Experiments in Design Science Research: A Proposition of an Evaluation Framework
Mettler, T., Eurich, M., & Winter, R. (2014) · Communications of the Association for Information Systems · doi:10.17705/1cais.03410 · Cited by 73
Redesigning Teams and Incentives in a Merger: An Experiment with Managers and Students
Montmarquette, C., Rullière, J., Villeval, M. C., & Zeiliger, R. (2004) · Management Science · doi:10.1287/mnsc.1040.0280 · Cited by 69
Nair, V. N., Strecher, V. J., Fagerlin, A., Ubel, P. A., Resnicow, K., Murphy, S. A., Little, R. J. A., Chakraborty, B., & Zhang, A. (2008) · American Journal of Public Health · doi:10.2105/ajph.2007.127563 · Cited by 69
Cited by
Cooper, B., Eva, N., Zarea Fazlelahi, F., Newman, A., Lee, A., & Obschonka, M. (2020) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2020.103472 · Cited by 339
Abusive supervision: A systematic review and fundamental rethink
Fischer, T., Tian, A. W., Lee, A., & Hughes, D. J. (2021) · The Leadership Quarterly · doi:10.1016/j.leaqua.2021.101540 · Cited by 318
Podsakoff, P. M., & Podsakoff, N. P. (2019) · The Leadership Quarterly · doi:10.1016/j.leaqua.2018.11.002 · Cited by 312
A critique of the Leader-Member Exchange construct: Back to square one
Gottfredson, R. K., Wright, S. L., & Heaphy, E. D. (2020) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101385 · Cited by 205
Humble leadership: A review and synthesis of leader expressed humility
Kelemen, T. K., Matthews, S. H., Matthews, M. J., & Henry, S. E. (2023) · Journal of Organizational Behavior · doi:10.1002/job.2608 · Cited by 203
Metadata
- Authors
- Lonati, S., Quiroga, B. F., Zehnder, C., & Antonakis, J.
- Venue
- Journal of Operations Management
- Publisher
- Wiley
- Published
- 2018-11
- Volume
- 64
- Issue
- 1
- Pages
- 19-40
- ISSN
- 0272-6963 (ISSN-L)
- 1873-1317
- OpenAlex
- W2910905828
- Cited by
- 354
- References
- 211
- FWCI
- 16.99
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.