Apples and Oranges (and Pears, Oh My!): The Search for Moderators in Meta-Analysis
The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
The purpose of this article is to review current practices with respect to detection and estimation of moderators in meta-analysis and to develop recommendations that are driven by the results of this review and previous research. The first purpose was accomplished through a review of the meta-analyses published in Journal of Applied Psychology from 1978 to 1997. Results show, first, that practices with respect to both the execution of and the reporting of results from searches for moderators are highly variable and, second, that findings relevant for detection of moderators (e.g., percentage variance attributable to artifacts, SDρ, etc.) are often highly inconsistent with what has been suggested in the past. These practices held regardless of time of publication, specificity of the question addressed in the paper, and content area. Detailed suggestions for modifications of current practices are offered.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Significance tests and goodness of fit in the analysis of covariance structures.
Bentler, P. M., & Bonett, D. G. (1980) · Psychological Bulletin · doi:10.1037//0033-2909.88.3.588 · Cited by 19,023 · cited 1×
“In structural equation modeling, indices of model fit, such as the normed fit index (Bentler & Bonnet, 1980), involve a comparison of the residual matrix of the hypothesized model to the residual matrix of a null model.2 Although the definition of “null” can be debated (Mulaik, James, Van Alstine,…”
BackgroundThe application of electronic computers to factor analysis
Kaiser, H. F. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000116 · Cited by 12,459 · cited 1×
“This convention stems from work by Kaiser (1960) and Guttman (1954) in which it was shown that a factor with an eigenvalue = 1 has a reliability of zero.”
Evaluating the use of exploratory factor analysis in psychological research.
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999) · Psychological Methods · doi:10.1037/1082-989x.4.3.272 · Cited by 9,587
Huber, P. J. (1981) · Wiley Series in Probability and Statistics · doi:10.1002/0471725250 · Cited by 7,066
Comparison of five rules for determining the number of components to retain.
Zwick, W. R., & Velicer, W. F. (1986) · Psychological Bulletin · doi:10.1037//0033-2909.99.3.432 · Cited by 3,129 · cited 1×
“Although it can certainly be argued that Kaiser’s criterion is overextended (Cattell & Vogelmann, 1977; Fabrigar, Wegener, MacCallum, & Strahan, 1999; Zwick & Velicer, 1986), it is useful as a baseline value for ruling out factors/ components.”
BackgroundMeta-Analytic review of leader–member exchange theory: Correlates and construct issues.
Gerstner, C. R., & Day, D. V. (1997) · Journal of Applied Psychology · doi:10.1037/0021-9010.82.6.827 · Cited by 3,005 · cited 1×
“…be followed up byt tests, informal consideration of percentage variance attributable to artifacts (e.g., Huffcutt, Roth, & McDaniel, 1996), Hedges and Olkin (1985)Q tests (e.g., Gerstner & Day, 1997), or more sophisticated techniques such as hierarchical linear modeling (van Eerde & Thierry, 1996).”
MethodEvaluation of goodness-of-fit indices for structural equation models.
Mulaik, S. A., James, L. R., Van Alstine, J., & Bennett, N. (1989) · Psychological Bulletin · doi:10.1037//0033-2909.105.3.430 · Cited by 2,923
Some necessary conditions for common-factor analysis
Guttman, L. (1954) · Psychometrika · doi:10.1007/bf02289162 · Cited by 1,689 · cited 2×
“This convention stems from work by Kaiser (1960) and Guttman (1954) in which it was shown that a factor with an eigenvalue = 1 has a reliability of zero. Thus, this can be thought of as a “point of no return” criterion such that we may not think much of factors with eigenvalues only slightly larger than 1, but we can…”
The five factor model of personality and job performance in the European Community.
Salgado, J. F. (1997) · Journal of Applied Psychology · doi:10.1037//0021-9010.82.1.30 · Cited by 1,581 · cited 1×
“…which a criterion was specified, 57% specified the “75% rule”, 19% specified aχ2 of some kind (e.g.,Q, U), 2% relied on a “credibility interval does not contain zero” rule, and 6% specified some other rule (e.g., Salgado, 1997: sampling error accounts for more than half of the observed variance).”
BackgroundRole of social desirability in personality testing for personnel selection: The red herring.
Ones, D. S., Viswesvaran, C., & Reiss, A. D. (1996) · Journal of Applied Psychology · doi:10.1037//0021-9010.81.6.660 · Cited by 924 · cited 4×
“For example, a meta-analysis reported in Ones, Viswesvaran, and Reiss (1996) resulted in a percentage variance value of 73%, but the standard deviation ofρ was only .034, resulting in a very small credibility interval.”
BackgroundMethodInfluential
Research works printed but not linked: 22
- 10.1037/0021-9010.82.4.528 · doi:10.1037/0021-9010.82.4.528 · Paper page
- 10.1037/0021-9010.87.3.583 · doi:10.1037/0021-9010.87.3.583 · Paper page
- 10.1037/0033-2909.88.3.588 · DOI · Paper page
- 10.1037/0021-9010.80.2.327 · doi:10.1037/0021-9010.80.2.327 · Paper page
- 10.1037/0021-9010.81.5.459 · doi:10.1037/0021-9010.81.5.459 · Paper page
- 10.1037/0021-9010.71.3.440 · doi:10.1037/0021-9010.71.3.440 · Paper page
- 10.1037/0021-9010.77.1.3 · doi:10.1037/0021-9010.77.1.3 · Paper page
- 10.1037/0021-9010.80.1.94 · doi:10.1037/0021-9010.80.1.94 · Paper page
- 10.1037/0021-9010.82.1.79 · doi:10.1037/0021-9010.82.1.79 · Paper page
- 10.1037/0033-2909.105.3.430 · DOI · Paper page
- 10.1037/0021-9010.81.6.660 · doi:10.1037/0021-9010.81.6.660 · Paper page
- 10.1037/0021-9010.65.4.373 · doi:10.1037/0021-9010.65.4.373 · Paper page
- 10.1037/0021-9010.81.4.380 · doi:10.1037/0021-9010.81.4.380 · Paper page
- 10.1037/0021-9010.68.3.382 · doi:10.1037/0021-9010.68.3.382 · Paper page
- 10.1037/0021-9010.82.1.30 · DOI · Paper page
- 10.3102/0013189X013008006 · doi:10.3102/0013189x013008006 · Paper page
- Society for Industrial and Organizational Psychology. (2000). 16th Annual Conference call for proposals. Bowling Green, OH: Author.
- 10.1037/0021-9010.72.1.3 · doi:10.1037/0021-9010.72.1.3 · Paper page
- 10.1037/0021-9010.81.5.575 · doi:10.1037/0021-9010.81.5.575 · Paper page
- 10.1037/0021-9010.74.2.259 · doi:10.1037/0021-9010.74.2.259 · Paper page
- 10.1037/0003-066X.53.3.300 · doi:10.1037/0003-066x.53.3.300 · Paper page
- 10.1037/0033-2909.99.3.432 · DOI · Paper page
Cited by
CHRISTIAN, M. S., GARZA, A. S., & SLAUGHTER, J. E. (2011) · Personnel Psychology · doi:10.1111/j.1744-6570.2010.01203.x · Cited by 2,903
“Cortina (2003) suggests that when moderators are present, an appropriate method is to break down the effect sizes into categories and test for differences.”
BackgroundMethodPerceived organizational support: a meta-analytic evaluation of organizational support theory.
Kurtessis, J. N., Eisenberger, R., Ford, M. T., Buffardi, L. C., Stewart, K. A., & Adis, C. S. (2017) · Journal of Management · doi:10.1177/0149206315575554 · Cited by 2,354
“Because caution has been urged in removing outliers (e.g., Cortina, 2003) and these two did not substantially impact results, we retained them.”
Intrinsic motivation and extrinsic incentives jointly predict performance: A 40-year meta-analysis.
Cerasoli, C. P., Nicklin, J. M., & Ford, M. T. (2014) · Psychological Bulletin · doi:10.1037/a0035661 · Cited by 1,981
Refining the relationship between personality and subjective well-being.
Steel, P., Schmidt, J., & Shultz, J. (2008) · Psychological Bulletin · doi:10.1037/0033-2909.134.1.138 · Cited by 1,794
A Review of Self-Determination Theory’s Basic Psychological Needs at Work
Van den Broeck, A., Ferris, D. L., Chang, C.-H., & Rosen, C. C. (2016) · Journal of Management · doi:10.1177/0149206316632058 · Cited by 1,465
“Additional subgroup analyses were performed to examine the effects of a priori moderation effects associated with specific study characteristics (Cortina, 2003) with a significant Q statistic.”
Method
Metadata
- Authors
- Cortina, J. M.
- Affiliations
- George Mason University
- Venue
- Organizational Research Methods
- Publisher
- SAGE Publications
- Published
- 2003-10
- Volume
- 6
- Issue
- 4
- Pages
- 415-439
- ISSN
- 1094-4281 (ISSN-L)
- 1552-7425
- OpenAlex
- W2075116339
- Cited by
- 168
- References
- 47
- Citations by year
- 2026: 6 · 2025: 5 · 2024: 6 · 2023: 7 · 2022: 6 · 2021: 8 · 2020: 6 · 2019: 9 · 2018: 9 · 2017: 10 · 2016: 13 · 2015: 11 · 2014: 11 · 2013: 9 · 2012: 9
- FWCI
- 2.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Organizational Research Methods (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- moderators, heterogeneity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Apples and Oranges (and Pears, Oh My!): The Search for Moderators in Meta-Analysis. https://app.cochat.ai/papers/10.1177/1094428103257358
- The paper
- Cortina, J. M. (2003). Apples and Oranges (and Pears, Oh My!): The Search for Moderators in Meta-Analysis. Organizational Research Methods. https://doi.org/10.1177/1094428103257358
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.