Revealed or Concealed? Transparency of Procedures, Decisions, and Judgment Calls in Meta-Analyses
The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
The authors examined the degree to which meta-analyses in the organizational sciences transparently report procedures, decisions, and judgment calls by systematically reviewing all (198) meta-analyses published between 1995 and 2008 in 11 top journals that publish meta-analyses in industrial and organizational psychology and organizational behavior. The authors extracted information on 54 features of each meta-analysis. On average, the meta-analyses in the sample provided 52.8% of the information needed to replicate the meta-analysis or to assess its validity and 67.6% of the information considered to be most important according to expert meta-analysts. More recently published meta-analyses exhibited somewhat more transparent reporting practices than older ones did. Overall transparency of reporting (but not reporting of the most important items) was associated with higher ranked journals; transparency was not significantly related to number of citations. The authors discuss the implications of inadequate reporting of meta-analyses for development of cumulative knowledge and effective practice and make suggestions for improving the current state of affairs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Perceived organizational support: a review of the literature.
Rhoades, L., & Eisenberger, R. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.4.698 · Cited by 7,592
Justice at the millennium: A meta-analytic review of 25 years of organizational justice research.
Colquitt, J. A., Conlon, D. E., Wesson, M. J., Porter, C. O. L. H., & Ng, K. Y. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.3.425 · Cited by 5,942
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Harter, J. K., Schmidt, F. L., & Hayes, T. L. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.2.268 · Cited by 4,623
Trust in leadership: Meta-analytic findings and implications for research and practice.
Dirks, K. T., & Ferrin, D. L. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.4.611 · Cited by 3,550
A meta-analytic examination of the correlates of the three dimensions of job burnout
Lee, R. T., & Ashforth, B. E. (1996) · Journal of Applied Psychology · doi:10.1037/0021-9010.81.2.123 · Cited by 3,262
Personality and leadership: A qualitative and quantitative review
Judge, T. A., Bono, J. E., Ilies, R., & Gerhardt, M. W. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.4.765 · Cited by 3,190
Task versus relationship conflict, team performance, and team member satisfaction: A meta-analysis.
De Dreu, C. K. W., & Weingart, L. R. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.4.741 · Cited by 3,071
Meta-Analytic review of leader–member exchange theory: Correlates and construct issues.
Gerstner, C. R., & Day, D. V. (1997) · Journal of Applied Psychology · doi:10.1037/0021-9010.82.6.827 · Cited by 3,005
Colquitt, J. A., Scott, B. A., & LePine, J. A. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.4.909 · Cited by 2,606
Lepine, J. A., Podsakoff, N. P., & Lepine, M. A. (2005) · Academy of Management Journal · doi:10.5465/amj.2005.18803921 · Cited by 2,438
Meta-analysis: Principles and procedures
Egger, M., Smith, G. D., & Phillips, A. N. (1997) · BMJ · doi:10.1136/bmj.315.7121.1533 · Cited by 2,415
PREDICTORS OF OBJECTIVE AND SUBJECTIVE CAREER SUCCESS: A META‐ANALYSIS
NG, T. W. H., EBY, L. T., SORENSEN, K. L., & FELDMAN, D. C. (2005) · Personnel Psychology · doi:10.1111/j.1744-6570.2005.00515.x · Cited by 2,412
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Humphrey, S. E., Nahrgang, J. D., & Morgeson, F. P. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.5.1332 · Cited by 2,303
Psychological and physical well-being during unemployment: a meta-analytic study
McKee-Ryan, F., Song, Z., Wanberg, C. R., & Kinicki, A. J. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.1.53 · Cited by 2,269
Five-factor model of personality and job satisfaction: A meta-analysis.
Judge, T. A., Heller, D., & Mount, M. K. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.3.530 · Cited by 2,164
Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.5.678 · Cited by 2,126
Podsakoff, N. P., LePine, J. A., & LePine, M. A. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.2.438 · Cited by 1,948
Sources of social support and burnout: a meta-analytic test of the conservation of resources model
Halbesleben, J. R. B. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.5.1134 · Cited by 1,336
Van Wijk, R., Jansen, J. J. P., & Lyles, M. A. (2008) · Journal of Management Studies · doi:10.1111/j.1467-6486.2008.00771.x · Cited by 1,335
Cohesion and Performance in Groups: A Meta-Analytic Clarification of Construct Relations.
Beal, D. J., Cohen, R. R., Burke, M. J., & McLendon, C. L. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.6.989 · Cited by 1,321
Publication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291
The Forgotten Ones? The Validity of Consideration and Initiating Structure in Leadership Research.
Judge, T. A., Piccolo, R. F., & Ilies, R. (2004) · Journal of Applied Psychology · doi:10.1037/0021-9010.89.1.36 · Cited by 1,265 · cited 1×
“Keith and Frese (2008) discussed one of the potential biases of the primary studies as follows: ‘‘None of the currently available studies on EMT have actually measured on-the-job performance as a result of training’’ (p.”
BackgroundEffectiveness of training in organizations: A meta-analysis of design and evaluation features.
Arthur, W., Bennett, W., Edens, P. S., & Bell, S. T. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.2.234 · Cited by 1,252
A meta-analytic examination of the goal orientation nomological net.
Payne, S. C., Youngcourt, S. S., & Beaubien, J. M. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.1.128 · Cited by 1,242
The relationship of age to ten dimensions of job performance.
Ng, T. W. H., & Feldman, D. C. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.2.392 · Cited by 1,224
Leader-member exchange and citizenship behaviors: A meta-analysis
Ilies, R., Nahrgang, J. D., & Morgeson, F. P. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.1.269 · Cited by 1,210
Chapman, D. S., Uggerslev, K. L., Carroll, S. A., Piasentin, K. A., & Jones, D. A. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.5.928 · Cited by 1,182
A Meta-Analysis of Work Demand Stressors and Job Performance: Examining Main and Moderating Effects
GILBOA, S., SHIROM, A., FRIED, Y., & COOPER, C. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00113.x · Cited by 1,072
Estimation of a Single Effect Size: Parametric and Nonparametric Methods
Hedges, L. V., & Olkin, I. (1985) · Statistical Methods for Meta-Analysis · doi:10.1016/b978-0-08-057065-5.50010-5 · Cited by 983
A META‐ANALYSIS OF THE RELATIONS AMONG TRAINING CRITERIA
ALLIGER, G. M., TANNENBAUM, S. I., BENNETT, W., TRAVER, H., & SHOTLAND, A. (1997) · Personnel Psychology · doi:10.1111/j.1744-6570.1997.tb00911.x · Cited by 948
Nguyen, H.-H. D., & Ryan, A. M. (2008) · Journal of Applied Psychology · doi:10.1037/a0012702 · Cited by 913
Baltes, B. B., Briggs, T. E., Huff, J. W., Wright, J. A., & Neuman, G. A. (1999) · Journal of Applied Psychology · doi:10.1037/0021-9010.84.4.496 · Cited by 837
Five-Factor Model of personality and Performance in Jobs Involving Interpersonal Interactions
Mount, M., Barrick, M., & Stewart, G. (1998) · Human Performance · doi:10.1207/s15327043hup1102&3_3 · Cited by 832
Applicant Reactions to Selection Procedures: An Updated Model and Meta‐Analysis
Hausknecht, J. P., Day, D. V., & Thomas, S. C. (2004) · Personnel Psychology · doi:10.1111/j.1744-6570.2004.00003.x · Cited by 816
A meta-analytic review of the distribution of practice effect: Now you see it, now you don’t
Donovan, J. J., & Radosevich, D. J. (1999) · Journal of Applied Psychology · doi:10.1037/0021-9010.84.5.795 · Cited by 696
Information System Success: Individual and Organizational Determinants
Sabherwal, R., Jeyaraj, A., & Chowa, C. (2006) · Management Science · doi:10.1287/mnsc.1060.0583 · Cited by 685
Factors That Influence Skill Decay and Retention: A Quantitative Review and Analysis
Arthur Jr., W., Bennett Jr., W., Stanush, P. L., & McNelly, T. L. (1998) · Human Performance · doi:10.1207/s15327043hup1101_3 · Cited by 664
Intelligence and leadership: A quantitative review and test of theoretical propositions
Judge, T. A., Colbert, A. E., & Ilies, R. (2004) · Journal of Applied Psychology · doi:10.1037/0021-9010.89.3.542 · Cited by 611
Publication Bias in Meta‐Analysis
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch1 · Cited by 603
Self-citationSMITHER, J. W., LONDON, M., & REILLY, R. R. (2005) · Personnel Psychology · doi:10.1111/j.1744-6570.2005.514_1.x · Cited by 601
Sturman, M. C. (2003) · Journal of Management · doi:10.1016/s0149-2063_03_00028-x · Cited by 578
Influence tactics and work outcomes: a meta analysis
Higgins, C. A., Judge, T. A., & Ferris, G. R. (2003) · Journal of Organizational Behavior · doi:10.1002/job.181 · Cited by 573
Reporting standards for research in psychology: why do we need them? What might they be?
APA Publications and Communications Board Working Group on Journal Article Reporting Standards (2008) · American Psychologist · doi:10.1037/0003-066x.63.9.839 · Cited by 571
The causal relation between job attitudes and performance: A meta-analysis of panel studies.
Riketta, M. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.2.472 · Cited by 547
Webber, S. S., & Donahue, L. M. (2001) · Journal of Management · doi:10.1177/014920630102700202 · Cited by 518
HIGHER‐ORDER DIMENSIONS OF THE BIG FIVE PERSONALITY TRAITS AND THE BIG SIX VOCATIONAL INTEREST TYPES
MOUNT, M. K., BARRICK, M. R., SCULLEN, S. M., & ROUNDS, J. (2005) · Personnel Psychology · doi:10.1111/j.1744-6570.2005.00468.x · Cited by 463
Research Synthesis and Meta-Analysis
Cooper, H., Patall, E., & Lindsay, J. (2009) · The SAGE Handbook of Applied Social Research Methods · doi:10.4135/9781483348858.n11 · Cited by 397
Harrison, D. A., Kravitz, D. A., Mayer, D. M., Leslie, L. M., & Lev-Arey, D. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.5.1013 · Cited by 358
ROTH, P. L., BEVIER, C. A., BOBKO, P., SWITZER, F. S., & TYLER, P. (2001) · Personnel Psychology · doi:10.1111/j.1744-6570.2001.tb00094.x · Cited by 325
Aguinis, H., Dalton, D. R., Bosco, F. A., Pierce, C. A., & Dalton, C. M. (2011) · Journal of Management · doi:10.1177/0149206310377113 · Cited by 302
Long work hours: a social identity perspective on meta‐analysis data
Ng, T. W. H., & Feldman, D. C. (2008) · Journal of Organizational Behavior · doi:10.1002/job.536 · Cited by 264
Meta-analyzing the relationship between grades and job performance.
Roth, P. L., BeVier, C. A., Switzer, F. S., & Schippmann, J. S. (1996) · Journal of Applied Psychology · doi:10.1037/0021-9010.81.5.548 · Cited by 259
Barsky, A., & Kaplan, S. A. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.1.286 · Cited by 243
Neubert, M. J. (1998) · Human Performance · doi:10.1207/s15327043hup1104_2 · Cited by 196
SCHMIDT, F. L., SHAFFER, J. A., & OH, I. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00132.x · Cited by 190
Subgroup Differences in Situational Judgment Test Performance: A Meta-Analysis
Whetzel, D. L., McDaniel, M. A., & Nguyen, N. T. (2008) · Human Performance · doi:10.1080/08959280802137820 · Cited by 183
GROUP DIFFERENCES IN PERSONALITY: META‐ANALYSES COMPARING FIVE U.S. RACIAL GROUPS
FOLDES, H. J., DUEHR, E. E., & ONES, D. S. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00123.x · Cited by 180
Ethnic group differences in measures of job performance: a new meta-analysis.
Roth, P. L., Huffcutt, A. I., & Bobko, P. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.4.694 · Cited by 171
Apples and oranges (and pears, oh my!): the search for moderators in meta-analysis
Cortina, J. M. (2003) · Organizational Research Methods · doi:10.1177/1094428103257358 · Cited by 168 · cited 2×
“After examining a variety of sources (Bobko & Roth, 2008; Cortina, 2003; Geyskens et al., 2009, Guzzo et al., 1987; Higgins & Green, 2006; Lipsey & Wilson, 2001; Rothstein & McDaniel, 1989) in addition to the works of Cooper (1982, 1984, 2009) and Wanous et al.”
Result
Cited by
Best-practice recommendations for defining, identifying, and handling outliers
Aguinis, H., Gottfredson, R. K., & Joo, H. (2013) · Organizational Research Methods · doi:10.1177/1094428112470848 · Cited by 1,341
Aguinis, H., Gottfredson, R. K., & Culpepper, S. A. (2013) · Journal of Management · doi:10.1177/0149206313478188 · Cited by 1,090
Abusive supervision: A meta-analysis and empirical review
Mackey, J. D., Frieder, R. E., Brees, J. R., & Martinko, M. J. (2017) · Journal of Management · doi:10.1177/0149206315573997 · Cited by 706
Sample Selection in Systematic Literature Reviews of Management Research
Hiebl, M. R. W. (2023) · Organizational Research Methods · doi:10.1177/1094428120986851 · Cited by 635
Digital transformation: A review and research agenda
Plekhanov, D., Franke, H., & Netland, T. H. (2023) · European Management Journal · doi:10.1016/j.emj.2022.09.007 · Cited by 606
Metadata
- Authors
- Aytug, Z. G., Rothstein, H. R., Zhou, W., & Kern, M. C.
- Venue
- Organizational Research Methods
- Publisher
- SAGE Publications
- Published
- 2011-04-05
- Volume
- 15
- Issue
- 1
- Pages
- 103-133
- ISSN
- 1094-4281 (ISSN-L)
- 1552-7425
- OpenAlex
- W2164715050
- Cited by
- 147
- References
- 61
- FWCI
- 6.42
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.