PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
This article has 2 goals. First, we discuss publication bias and explain why it presents a potential problem for industrial and organizational psychology. After reviewing the traditional failsafe N, or file drawer analysis, we introduce a more sophisticated method of publication bias analysis (trim and fill), which has been developed in the medical literature but is largely unfamiliar to industrial and organizational psychology researchers. Second, we demonstrate trim and fill by applying it to validity information reported in the technical manuals of 4 test vendors. In doing so, we assess the likelihood that criterion‐related validity information provided by test publishers may overestimate test validity. In our analysis of 18 validity distributions, we found evidence of either no or minimal bias for 2 of the vendors' distributions and evidence of moderate‐to‐severe bias in at least 1 distribution from each of the other 2 vendors. In both cases in which publication bias was found, we noted instances in which the publishers tended to report only statistically significant correlations and that this practice was detected using publication bias methodology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Perceived organizational support: a review of the literature.
Rhoades, L., & Eisenberger, R. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.4.698 · Cited by 7,592
Hunter, J., & Schmidt, F. (2004) · doi:10.4135/9781412985031 · Cited by 3,023
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
A Fail-Safe N for Effect Size in Meta-Analysis
Orwin, R. G. (1983) · Journal of Educational Statistics · doi:10.2307/1164923 · Cited by 2,064
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1×
“There is also some evidence that editorial policy, at least in some journals, favors the publication of significant results (Dickersin, 2005; Greenwald, 1975).”
ResultRegression Methods to Detect Publication and Other Bias in Meta‐Analysis
Sterne, J. A. C., & Egger, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch6 · Cited by 826 · cited 3×
“Precision is often used instead of sample size for the Y axis of the funnel plot (Sterne & Egger, 2005).”
BackgroundMethodAdjusting for publication bias in the presence of heterogeneity
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003) · Statistics in Medicine · doi:10.1002/sim.1461 · Cited by 809
Duval, S. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch8 · Cited by 496 · cited 4×
“As mentioned earlier in this article, Duval (2005) concluded that there was evidence of substantial publication bias in the McDaniel et al. (1994) data on the validity of structured interviews.”
BackgroundMethodInfluentialEmployee Burnout: A Meta-Analysis of the Relationship Between Age or Years of Experience
Brewer, E. W., & Shapard, L. (2004) · Human Resource Development Review · doi:10.1177/1534484304263335 · Cited by 456
Cited by
Bae, T. J., Qian, S., Miao, C., & Fiet, J. O. (2014) · Entrepreneurship Theory and Practice · doi:10.1111/etap.12095 · Cited by 1,759
Mind-sets matter: A meta-analytic review of implicit theories and self-regulation.
Burnette, J. L., O'Boyle, E. H., VanEpps, E. M., Pollack, J. M., & Finkel, E. J. (2013) · Psychological Bulletin · doi:10.1037/a0029531 · Cited by 1,367
The relation between emotional intelligence and job performance: A meta-analysis
O'Boyle, E. H., Humphrey, R. H., Pollack, J. M., Hawver, T. H., & Story, P. A. (2011) · Journal of Organizational Behavior · doi:10.1002/job.714 · Cited by 1,122
Rudolph, C. W., Katz, I. M., Lavigne, K. N., & Zacher, H. (2017) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2017.05.008 · Cited by 951
Golicic, S. L., & Smith, C. D. (2013) · Journal of Supply Chain Management · doi:10.1111/jscm.12006 · Cited by 790
Metadata
- Authors
- McDANIEL, M. A., ROTHSTEIN, H. R., & WHETZEL, D. L.
- Venue
- Personnel Psychology
- Publisher
- Wiley
- Published
- 2006-11-10
- Volume
- 59
- Issue
- 4
- Pages
- 927-953
- ISSN
- 0031-5826 (ISSN-L)
- 1744-6570
- OpenAlex
- W2118402129
- Cited by
- 141
- References
- 12
- FWCI
- 2.63
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS. https://app.cochat.ai/papers/10.1111/j.1744-6570.2006.00059.x
- The paper
- McDANIEL, M. A., ROTHSTEIN, H. R., & WHETZEL, D. L. (2006). PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS. Personnel Psychology. https://doi.org/10.1111/j.1744-6570.2006.00059.x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.