Meta-Analytic Choices and Judgment Calls: Implications for Theory Building and Testing, Obtained Effect Sizes, and Scholarly Impact
Abstract
The authors content analyzed 196 meta-analyses including 5,581 effect-size estimates published in Academy of Management Journal, Journal of Applied Psychology, Journal of Management, Personnel Psychology, and Strategic Management Journal from January 1982 through August 2009 to assess the presumed effects of each of 21 methodological choices and judgment calls on substantive conclusions. Results indicate that, overall, the various meta-analytic methodological choices available and judgment calls involved in the conduct of a meta-analysis have little impact on the resulting magnitude of the meta-analytically derived effect sizes. Thus, the present study, based on actual meta-analyses, casts doubt on previous warnings, primarily based on selective case studies, that judgment calls have an important impact on substantive conclusions. The authors also tested the fit of a multivariate model that includes relationships among theory-building and theory-testing goals, obtained effect sizes, year of publication of the meta-analysis, and scholarly impact (i.e., citations per year). Results indicate that the more a meta-analysis attempts to test an existing theory, the larger the number of citations, whereas the more a meta-analysis attempts to build new theory, the lower the number of citations. Also, in support of scientific particularism, as opposed to scientific universalism, the magnitude of the derived effects is not related to the extent to which a meta-analysis is cited. Taken together, the results provide a comprehensive data-based understanding of how meta-analytic reviews are conducted and the implications of these practices for theory building and testing, obtained effect sizes, and scholarly impact.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · see citing works
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542 · see citing works
Hunter, J., & Schmidt, F. (2004) · doi:10.4135/9781412985031 · Cited by 3,023 · see citing works
Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.5.678 · Cited by 2,126 · cited 2× · see citing works
“…“perhaps one of the largest sources of error in meta-analyses is the multitude of judgment calls made at various stages of the research synthesis process” (p. 993), and Colquitt, LePine, and Noe (2000) referred to the “subjectivity and judgment calls inherent in meta-analytic efforts” (p. 685).”
BackgroundMethodAguinis, H., Beaty, J. C., Boik, R. J., & Pierce, C. A. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.1.94 · Cited by 1,216 · see citing works
Self-citationCautionary note on reporting eta-squared values from multifactor ANOVA designs.
Pierce, C. A., Block, R. A., & Aguinis, H. (2004) · Educational and Psychological Measurement · doi:10.1177/0013164404264848 · Cited by 584 · see citing works
Self-citationWhat do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564 · cited 1× · see citing works
“Conclusions based on meta-analytic findings set the standard for what is considered state of the science, for what we know and do not know, and for which theory is considered valid and which one is not (Schmidt, 1992, 1996).”
BackgroundComparing meta-analytic moderator estimation techniques under realistic conditions.
Steel, P. D., & Kammeyer-Mueller, J. D. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.1.96 · Cited by 309 · cited 1× · see citing works
“…Demaree, & Mulaik, 1986; James, Demaree, Mulaik, & Ladd, 1992; Law, Schmidt, & Hunter, 1994; McDaniel, Rothstein, & Whetzel, 2006; Newman, Jacobs, & Bartram, 2007; Osburn & Callender, 1992; Oswald & Johnson, 1998; Schmidt & Raju, 2007; Schmidt, Shaffer, & Oh, 2008; Steel & Kammeyer-Mueller, 2002).”
Flagged first, then by citations
Cited by
Sorted by citations
Intrinsic motivation and extrinsic incentives jointly predict performance: A 40-year meta-analysis.
Cerasoli, C. P., Nicklin, J. M., & Ford, M. T. (2014) · Psychological Bulletin · doi:10.1037/a0035661 · Cited by 1,981 · see citing works
Dulebohn, J. H., Bommer, W. H., Liden, R. C., Brouer, R. L., & Ferris, G. R. (2012) · Journal of Management · doi:10.1177/0149206311415280 · Cited by 1,625 · see citing works
Hoch, J. E., Bommer, W. H., Dulebohn, J. H., & Wu, D. (2018) · Journal of Management · doi:10.1177/0149206316665461 · Cited by 1,364 · see citing works
Paradox research in management science: Looking back to move forward
Schad, J., Lewis, M. W., Raisch, S., & Smith, W. K. (2016) · Academy of Management Annals · doi:10.5465/19416520.2016.1162422 · Cited by 1,385 · see citing works
Abusive supervision: A meta-analysis and empirical review
Mackey, J. D., Frieder, R. E., Brees, J. R., & Martinko, M. J. (2017) · Journal of Management · doi:10.1177/0149206315573997 · Cited by 706 · see citing works
Busch, T., & Friede, G. (2018) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.1480 · Cited by 345 · see citing works
Mackey, J. D., Parker Ellen, B., McAllister, C. P., & Alexander, K. C. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2020.10.037 · Cited by 320 · see citing works
An Ounce of Prevention Is Worth a Pound of Cure: Improving Research Quality Before Data Collection
Aguinis, H., & Vandenberg, R. J. (2014) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-031413-091231 · Cited by 304 · see citing works
Self-citationMeta-analytic structural equation modeling with moderating effects on SEM parameters.
Jak, S., & Cheung, M. W. L. (2020) · Psychological Methods · doi:10.1037/met0000245 · Cited by 262 · see citing works
Do they matter? A meta-analytic investigation of individual characteristics and guilt judgments.
Devine, D. J., & Caughlin, D. E. (2014) · Psychology, Public Policy, and Law · doi:10.1037/law0000006 · Cited by 245 · see citing works
How to conduct a meta-analysis in eight steps: A practical guide
Hansen, C., Steinmetz, H., & Block, J. (2022) · Management Review Quarterly · doi:10.1007/s11301-021-00247-4 · Cited by 218 · see citing works
Scientific citations favor positive results: a systematic review and meta-analysis.
Duyx, B., Urlings, M. J. E., Swaen, G. M. H., Bouter, L. M., & Zeegers, M. P. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.06.002 · Cited by 197 · see citing works
A meta-analysis of the role of trust in the leadership- performance relationship
Legood, A., van der Werff, L., Lee, A., & Den Hartog, D. (2021) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2020.1819241 · Cited by 193 · see citing works
The Effectiveness of Advertising: A Meta-Meta-Analysis of Advertising Inputs and Outcomes
Eisend, M., & Tarrahi, F. (2016) · Journal of Advertising · doi:10.1080/00913367.2016.1185981 · Cited by 140 · see citing works
The Influence of Active and Passive Procrastination on Academic Performance: A Meta-Analysis
Kooren, N. S., Van Nooijen, C., & Paas, F. (2024) · Education Sciences · doi:10.3390/educsci14030323 · Cited by 33 · see citing works
“A random-effects model approach is in line with most meta-analytic approaches [40].”
Result
- Authors
- Aguinis, H., Dalton, D. R., Bosco, F. A., Pierce, C. A., & Dalton, C. M.
- Affiliations
- Indiana University (US)
- University of Memphis (US)
- Venue
- Journal of Management
- Publisher
- SAGE Publications
- Published
- 2010-07-21
- Volume
- 37
- Issue
- 1
- Pages
- 5-38
- ISSN
- 0149-2063 (ISSN-L)
- 1557-1211
- OpenAlex
- W2161199557
- Cited by
- 302
- References
- 8
- Citations by year
- 2026: 16 · 2025: 17 · 2024: 26 · 2023: 29 · 2022: 25 · 2021: 23 · 2020: 27 · 2019: 19 · 2018: 16 · 2017: 11 · 2016: 19 · 2015: 21 · 2014: 14 · 2013: 13 · 2012: 16
- FWCI
- 12.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Management (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Business Strategies and Management Research
- Keywords
- theory development, theory testing, scholarly impact, citation impact, management research, applied psychology research, strategic management research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
