Revealed or Concealed? Transparency of Procedures, Decisions, and Judgment Calls in Meta-Analyses
The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
The authors examined the degree to which meta-analyses in the organizational sciences transparently report procedures, decisions, and judgment calls by systematically reviewing all (198) meta-analyses published between 1995 and 2008 in 11 top journals that publish meta-analyses in industrial and organizational psychology and organizational behavior. The authors extracted information on 54 features of each meta-analysis. On average, the meta-analyses in the sample provided 52.8% of the information needed to replicate the meta-analysis or to assess its validity and 67.6% of the information considered to be most important according to expert meta-analysts. More recently published meta-analyses exhibited somewhat more transparent reporting practices than older ones did. Overall transparency of reporting (but not reporting of the most important items) was associated with higher ranked journals; transparency was not significantly related to number of citations. The authors discuss the implications of inadequate reporting of meta-analyses for development of cumulative knowledge and effective practice and make suggestions for improving the current state of affairs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Perceived organizational support: a review of the literature.
Rhoades, L., & Eisenberger, R. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.4.698 · Cited by 7,592
Justice at the millennium: A meta-analytic review of 25 years of organizational justice research.
Colquitt, J. A., Conlon, D. E., Wesson, M. J., Porter, C. O. L. H., & Ng, K. Y. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.3.425 · Cited by 5,942
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Harter, J. K., Schmidt, F. L., & Hayes, T. L. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.2.268 · Cited by 4,623
Trust in leadership: Meta-analytic findings and implications for research and practice.
Dirks, K. T., & Ferrin, D. L. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.4.611 · Cited by 3,550
Judge, T. A., & Bono, J. E. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.1.80 · Cited by 3,436
A meta-analytic examination of the correlates of the three dimensions of job burnout
Lee, R. T., & Ashforth, B. E. (1996) · Journal of Applied Psychology · doi:10.1037/0021-9010.81.2.123 · Cited by 3,262
Personality and leadership: A qualitative and quantitative review
Judge, T. A., Bono, J. E., Ilies, R., & Gerhardt, M. W. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.4.765 · Cited by 3,190
Task versus relationship conflict, team performance, and team member satisfaction: A meta-analysis.
De Dreu, C. K. W., & Weingart, L. R. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.4.741 · Cited by 3,071
Meta-Analytic review of leader–member exchange theory: Correlates and construct issues.
Gerstner, C. R., & Day, D. V. (1997) · Journal of Applied Psychology · doi:10.1037/0021-9010.82.6.827 · Cited by 3,005
Cited by
Best-practice recommendations for defining, identifying, and handling outliers
Aguinis, H., Gottfredson, R. K., & Joo, H. (2013) · Organizational Research Methods · doi:10.1177/1094428112470848 · Cited by 1,341
Aguinis, H., Gottfredson, R. K., & Culpepper, S. A. (2013) · Journal of Management · doi:10.1177/0149206313478188 · Cited by 1,090
Abusive supervision: A meta-analysis and empirical review
Mackey, J. D., Frieder, R. E., Brees, J. R., & Martinko, M. J. (2017) · Journal of Management · doi:10.1177/0149206315573997 · Cited by 706
Sample Selection in Systematic Literature Reviews of Management Research
Hiebl, M. R. W. (2023) · Organizational Research Methods · doi:10.1177/1094428120986851 · Cited by 635
Digital transformation: A review and research agenda
Plekhanov, D., Franke, H., & Netland, T. H. (2023) · European Management Journal · doi:10.1016/j.emj.2022.09.007 · Cited by 606
Advancing Theory with Review Articles
Post, C., Sarala, R., Gatrell, C., & Prescott, J. E. (2020) · Journal of Management Studies · doi:10.1111/joms.12549 · Cited by 548
Wang, G., Holmes, R. M., Oh, I., & Zhu, W. (2016) · Personnel Psychology · doi:10.1111/peps.12140 · Cited by 528
What You See Is What You Get? Enhancing Methodological Transparency in Management Research
Aguinis, H., Ramani, R. S., & Alabduljader, N. (2018) · Academy of Management Annals · doi:10.5465/annals.2016.0011 · Cited by 463
Meta-Synthesis of Qualitative Case Studies: An Approach to Theory Building
Hoon, C. (2013) · Organizational Research Methods · doi:10.1177/1094428113484969 · Cited by 424
The antecedents and outcomes of workplace ostracism: A meta-analysis.
Howard, M. C., Cogswell, J. E., & Smith, M. B. (2020) · Journal of Applied Psychology · doi:10.1037/apl0000453 · Cited by 408
Publication Bias in the Organizational Sciences
Kepes, S., Banks, G. C., McDaniel, M., & Whetzel, D. L. (2012) · Organizational Research Methods · doi:10.1177/1094428112452760 · Cited by 398
The process of reinventing a job: A meta–synthesis of qualitative job crafting research
Lazazzara, A., Tims, M., & de Gennaro, D. (2020) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2019.01.001 · Cited by 371
Bergh, D. D., Aguinis, H., Heavey, C., Ketchen, D. J., Boyd, B. K., Su, P., Lau, C. L. L., & Joo, H. (2016) · Strategic Management Journal · doi:10.1002/smj.2338 · Cited by 355
Busch, T., & Friede, G. (2018) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.1480 · Cited by 345
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLOS ONE · doi:10.1371/journal.pone.0215052 · Cited by 334
Mackey, J. D., Parker Ellen, B., McAllister, C. P., & Alexander, K. C. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2020.10.037 · Cited by 320
Principles for the Validation and Use of Personnel Selection Procedures
(2018) · Industrial and Organizational Psychology · doi:10.1017/iop.2018.195 · Cited by 300
Page, M. J., McKenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S., & Forbes, A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035.pub2 · Cited by 281
Meta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265
Qualitative Methods in Organizational Research: An Example of Grounded Theory Data Analysis
Kamasak, R., Kar, A., Yavuz, M., & Baykut, S. (2017) · Advances in Human Resources Management and Organizational Development · doi:10.4018/978-1-5225-2250-8.ch002 · Cited by 259
A Guide to Conducting a Meta-Analysis
Cheung, M. W.-L., & Vijayakumar, R. (2016) · Neuropsychology Review · doi:10.1007/s11065-016-9319-z · Cited by 239
Aguinis, H., Ramani, R. S., & Alabduljader, N. (2023) · Organizational Research Methods · doi:10.1177/1094428120943281 · Cited by 222
How to conduct a meta-analysis in eight steps: A practical guide
Hansen, C., Steinmetz, H., & Block, J. (2022) · Management Review Quarterly · doi:10.1007/s11301-021-00247-4 · Cited by 218
Methods for second order meta-analysis and illustrative applications
Schmidt, F. L., & Oh, I.-S. (2013) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2013.03.002 · Cited by 151
Technology acceptance research: Meta-analysis
Marikyan, D., Papagiannidis, S., & Stewart, G. (2026) · Journal of Information Science · doi:10.1177/01655515231191177 · Cited by 140
Meta-analyses indexed in PsycINFO had a better completeness of reporting when they mention PRISMA
Leclercq, V., Beaudart, C., Ajamieh, S., Rabenda, V., Tirelli, E., & Bruyère, O. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.06.014 · Cited by 69
Brini, S., Brudasca, N. I., Hodkinson, A., Kaluzinska, K., Wach, A., Storman, D., Prokop-Dorner, A., Jemioło, P., & Bala, M. M. (2023) · Clinical Psychology Review · doi:10.1016/j.cpr.2022.102236 · Cited by 44
A hybrid framework for creating artificial intelligence-augmented systematic literature reviews
Malik, F. S., & Terzidis, O. (2026) · Management Review Quarterly · doi:10.1007/s11301-025-00522-8 · Cited by 28
“This level of clarity strengthens the credibility of synthesized knowledge and supports an open and verifiable research process (Aytug et al. 2012; Kraus et al. 2020; Villiger et al. 2022).”
InfluentialMackey, J. D. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2021.03.002 · Cited by 14
Random or Fixed? An Empirical Examination of Meta-Analysis Model Choices
Slaney, K. L., Tafreshi, D., & Hohn, R. (2018) · Review of General Psychology · doi:10.1037/gpr0000140 · Cited by 13
Metadata
- Authors
- Aytug, Z. G., Rothstein, H. R., Zhou, W., & Kern, M. C.
- Venue
- Organizational Research Methods
- Publisher
- SAGE Publications
- Published
- 2011-04-05
- Volume
- 15
- Issue
- 1
- Pages
- 103-133
- ISSN
- 1094-4281 (ISSN-L)
- 1552-7425
- OpenAlex
- W2164715050
- Cited by
- 147
- References
- 66
- FWCI
- 6.42
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.