REVISITING THE FILE DRAWER PROBLEM IN META‐ANALYSIS: AN ASSESSMENT OF PUBLISHED AND NONPUBLISHED CORRELATION MATRICES
The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
The file drawer problem rests on the assumption that statistically non‐significant results are less likely to be published in primary‐level studies and less likely to be included in meta‐analytic reviews, thereby resulting in upwardly biased meta‐analytically derived effect sizes. We conducted 5 studies to assess the extent of the file drawer problem in nonexperimental research. In Study 1, we examined 37,970 correlations included in 403 matrices published in Academy of Management Journal(AMJ),Journal of Applied Psychology(JAP), and Personnel Psychology(PPsych) between 1985 and 2009 and found that 46.81% of those correlations are not statistically significant. In Study 2, we examined 6,935 correlations used as input in 51 meta‐analyses published in AMJ, JAP,PPsych, and elsewhere between 1982 and 2009 and found that 44.31% of those correlations are not statistically significant. In Study 3, we examined 13,943 correlations reported in 167 matrices in nonpublished manuscripts and found that 45.45% of those correlations are not statistically significant. In Study 4, we examined 20,860 correlations reported in 217 matrices in doctoral dissertations and found that 50.78% of those correlations are not statistically significant. In Study 5, we compared the average magnitude of a sample of 1,002 correlations from Study 1 (published articles) versus 1,224 from Study 4 (dissertations) and found that they were virtually identical (i.e., .2270 and .2279, respectively). In sum, our 5 studies provide consistent empirical evidence that the file drawer problem does not produce an inflation bias and does not pose a serious threat to the validity of meta‐analytically derived conclusions as is currently believed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Perceived organizational support: a review of the literature.
Rhoades, L., & Eisenberger, R. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.4.698 · Cited by 7,592
Mathieu, J. E., & Zajac, D. M. (1990) · Psychological Bulletin · doi:10.1037//0033-2909.108.2.171 · Cited by 6,449
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
TETT, R. P., & MEYER, J. P. (1993) · Personnel Psychology · doi:10.1111/j.1744-6570.1993.tb00874.x · Cited by 4,114
Hunter, J., & Schmidt, F. (2004) · doi:10.4135/9781412985031 · Cited by 3,023
Fixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955 · cited 1×
“…linear-regression tests (Egger, Davey-Smith, Schneider, & Minder, 1997; Stern & Egger, 2005), funnel plots (Light & Pillemer, 1984; Macaskill, Walter, & Irwig, 2001; Stern et al., 2005), and the trim and fill method (Duval, 2005; Duval & Tweedie, 2000a, 2000b; Schwarzer, Carpenter, & Rucker, 2010).”
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Meta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
A Fail-Safe N for Effect Size in Meta-Analysis
Orwin, R. G. (1983) · Journal of Educational Statistics · doi:10.2307/1164923 · Cited by 2,064
Cited by
Determinants of Entrepreneurial Intent: A Meta–Analytic Test and Integration of Competing Models
Schlaegel, C., & Koenig, M. (2014) · Entrepreneurship Theory and Practice · doi:10.1111/etap.12087 · Cited by 1,233
Psychological Safety: A Meta‐Analytic Review and Extension
Frazier, M. L., Fainshmidt, S., Klinger, R. L., Pezeshkan, A., & Vracheva, V. (2017) · Personnel Psychology · doi:10.1111/peps.12183 · Cited by 1,231
Aguinis, H., Gottfredson, R. K., & Culpepper, S. A. (2013) · Journal of Management · doi:10.1177/0149206313478188 · Cited by 1,090
Self-citationNielsen, K., Nielsen, M. B., Ogbonnaya, C., Känsälä, M., Saari, E., & Isaksson, K. (2017) · Work & Stress · doi:10.1080/02678373.2017.1304463 · Cited by 821
Correlational effect size benchmarks
Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2015) · Journal of Applied Psychology · doi:10.1037/a0038047 · Cited by 768
“…however, that our analyses are based on the largest individual-level ES database in the field; although they included a wider range of journal sources, other similar content analyses have involved substantially smaller data sets (e.g., Aguinis et al., 2011; Dalton et al., 2012; Shen et al., 2011).”
MethodSelf-citationAbusive supervision: A meta-analysis and empirical review
Mackey, J. D., Frieder, R. E., Brees, J. R., & Martinko, M. J. (2017) · Journal of Management · doi:10.1177/0149206315573997 · Cited by 706
Beugelsdijk, S., Kostova, T., Kunst, V. E., Spadafora, E., & van Essen, M. (2018) · Journal of Management · doi:10.1177/0149206317729027 · Cited by 457
Shockley, K. M., Shen, W., DeNunzio, M. M., Arvan, M. L., & Knudsen, E. A. (2017) · Journal of Applied Psychology · doi:10.1037/apl0000246 · Cited by 451
Testing a continuum structure of self-determined motivation: A meta-analysis.
Howard, J. L., Gagné, M., & Bureau, J. S. (2017) · Psychological Bulletin · doi:10.1037/bul0000125 · Cited by 447
Meta-Synthesis of Qualitative Case Studies: An Approach to Theory Building
Hoon, C. (2013) · Organizational Research Methods · doi:10.1177/1094428113484969 · Cited by 424
Bergh, D. D., Aguinis, H., Heavey, C., Ketchen, D. J., Boyd, B. K., Su, P., Lau, C. L. L., & Joo, H. (2016) · Strategic Management Journal · doi:10.1002/smj.2338 · Cited by 355
Self-citationSherf, E. N., Parke, M. R., & Isaakyan, S. (2021) · Academy of Management Journal · doi:10.5465/amj.2018.1428 · Cited by 349
What’s in a p? Reassessing best practices for conducting and reporting hypothesis-testing research
Meyer, K. E., van Witteloostuijn, A., & Beugelsdijk, S. (2017) · Journal of International Business Studies · doi:10.1057/s41267-017-0078-8 · Cited by 318
The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles
O’Boyle, E. H., Banks, G. C., & Gonzalez-Mulé, E. (2017) · Journal of Management · doi:10.1177/0149206314527133 · Cited by 246
“For example, Dalton et al. (2012) found “that, contrary to the established belief, the file drawer problem is of little, if any, consequence for meta-analytically derived theoretical conclusions and applications in OBHRM, I-O psychology, and related fields” (p. 225).”
BackgroundA meta-analysis of virtual reality training programs for social skill development
Howard, M. C., & Gutworth, M. B. (2020) · Computers & Education · doi:10.1016/j.compedu.2019.103707 · Cited by 234
Galanakis, M. D., & Tsitouri, E. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1022102 · Cited by 229
A century of research on conscientiousness at work
Wilmot, M. P., & Ones, D. S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1908430116 · Cited by 215
De Rijdt, C., Stes, A., van der Vleuten, C., & Dochy, F. (2013) · Educational Research Review · doi:10.1016/j.edurev.2012.05.007 · Cited by 198
Tejada-Gallardo, C., Blasco-Belled, A., Torrelles-Nadal, C., & Alsinet, C. (2020) · Journal of Youth and Adolescence · doi:10.1007/s10964-020-01289-9 · Cited by 195
“Also, in many cases, peer journals preferred the publication of studies with favorable (i.e., significant) results rather than unfavorable (i.e., non-significant) results—this is known as the “file drawer problem” (Dalton et al. 2012).”
BackgroundVan Iddekinge, C. H., Aguinis, H., Mackey, J. D., & DeOrtentiis, P. S. (2018) · Journal of Management · doi:10.1177/0149206317702220 · Cited by 195
Self-citationPaterson, T. A., Harms, P. D., Steel, P., & Credé, M. (2016) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051815614321 · Cited by 194
Leslie, L. M., Mayer, D. M., & Kravitz, D. A. (2014) · Academy of Management Journal · doi:10.5465/amj.2011.0940 · Cited by 183
A Meta-Analysis of Interpersonal and Organizational Workplace Deviance Research
Mackey, J. D., McAllister, C. P., Ellen, B. P., & Carson, J. E. (2021) · Journal of Management · doi:10.1177/0149206319862612 · Cited by 181
Is the Putative Mirror Neuron System Associated with Empathy? A Systematic Review and Meta-Analysis
Bekkali, S., Youssef, G. J., Donaldson, P. H., Albein-Urios, N., Hyde, C., & Enticott, P. G. (2021) · Neuropsychology Review · doi:10.1007/s11065-020-09452-6 · Cited by 128
Keebler, J. R., Lazzara, E. H., Patzer, B. S., Palmer, E. M., Plummer, J. P., Smith, D. C., Lew, V., Fouquet, S., Chan, Y. R., & Riss, R. (2016) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816672309 · Cited by 97
A meta-analytic review of mobile advertising research
Maseeh, H. I., Jebarajakirthy, C., Pentecost, R., Ashaduzzaman, M., Arli, D., & Weaven, S. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2021.06.022 · Cited by 88
Metadata
- Authors
- DALTON, D. R., AGUINIS, H., DALTON, C. M., BOSCO, F. A., & PIERCE, C. A.
- Venue
- Personnel Psychology
- Publisher
- Wiley
- Published
- 2012-05-04
- Volume
- 65
- Issue
- 2
- Pages
- 221-249
- ISSN
- 0031-5826 (ISSN-L)
- 1744-6570
- OpenAlex
- W1932682057
- Cited by
- 152
- References
- 27
- FWCI
- 12.73
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.