HARKing's Threat to Organizational Research: Evidence From Primary and Meta‐Analytic Sources
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
We assessed presumed consequences of hypothesizing after results are known (HARKing) by contrasting hypothesized versus nonhypothesized effect sizes among 10 common relations in organizational behavior, human resource management, and industrial and organizational psychology research. In Study 1, we analyzed 247 correlations representing 9 relations with individual performance in 136 articles published in Journal of Applied Psychology and Personnel Psychology and provide evidence that correlations are significantly larger when hypothesized compared to nonhypothesized. In Study 2, we analyzed 281 effect sizes from a meta‐analysis on the job satisfaction–job performance relation and provide evidence that correlations are significantly larger when hypothesized compared to nonhypothesized. In addition, in Study 2, we documented that hypothesized variable pairs are more likely to be mentioned in article titles or abstracts. We also ruled out 13 alternative explanations to the presumed HARKing effect pertaining to methodological (e.g., unreliability, publication year, research setting, research design, measure contextualization, publication source) and substantive (e.g., predictor–performance pair, performance measure, satisfaction measure, occupation, job/task complexity) issues. Our results suggest that HARKing seems to pose a threat to research results, substantive conclusions, and practical applications. We offer recommended solutions to the HARKing threat.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META‐ANALYSIS
BARRICK, M. R., & MOUNT, M. K. (1991) · Personnel Psychology · doi:10.1111/j.1744-6570.1991.tb00688.x · Cited by 8,689
The job satisfaction–job performance relationship: A qualitative and quantitative review.
Judge, T. A., Thoresen, C. J., Bono, J. E., & Patton, G. K. (2001) · Psychological Bulletin · doi:10.1037/0033-2909.127.3.376 · Cited by 4,760
Hunter, J., & Schmidt, F. (2004) · doi:10.4135/9781412985031 · Cited by 3,023
Platt, J. R. (1964) · Science · doi:10.1126/science.146.3642.347 · Cited by 2,907 · cited 1×
“In contrast, strong inference, which pits theories against each other (Edwards & Berry, 2010; Platt, 1964), is based on an experimental design paradigm, infrequent in organizational research and therefore offers relatively fewer opportunities for HARKing.”
Barrick, M. R., Mount, M. K., & Judge, T. A. (2001) · International Journal of Selection and Assessment · doi:10.1111/1468-2389.00160 · Cited by 2,356
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Improved tests for a random effects meta‐regression with a single covariate
Knapp, G., & Hartung, J. (2003) · Statistics in Medicine · doi:10.1002/sim.1482 · Cited by 2,075 · cited 1×
“We used the metafor 1.9–3 package for R (Viechtbauer, 2010), which implements the meta-regression procedures proposed by Knapp and Hartung (2003).”
MethodInfluentialHARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 6×
“HARKing’s prevalence has been acknowledged by authors and editors (e.g., Bedeian et al., 2010; Fanelli, 2009), authors’ evolving hypothesis statements within sources over time (O’Boyle et al., in press), and suspiciously low hypothesis falsification rates (e.g., Fanelli, 2010; Francis, Tanzman, &…”
InfluentialA Critical Review and Best‐Practice Recommendations for Control Variable Usage
Bernerth, J. B., & Aguinis, H. (2016) · Personnel Psychology · doi:10.1111/peps.12103 · Cited by 1,731
Self-citationCorrelational effect size benchmarks
Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2015) · Journal of Applied Psychology · doi:10.1037/a0038047 · Cited by 768
Self-citation“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · cited 1×
“…acknowledged by authors and editors (e.g., Bedeian et al., 2010; Fanelli, 2009), authors’ evolving hypothesis statements within sources over time (O’Boyle et al., in press), and suspiciously low hypothesis falsification rates (e.g., Fanelli, 2010; Francis, Tanzman, & Matthews, 2014; Leung, 2011).”
Fostering integrity in research: Definitions, current knowledge, and future directions
Steneck, N. H. (2006) · Science and Engineering Ethics · doi:10.1007/pl00022268 · Cited by 407
An Ounce of Prevention Is Worth a Pound of Cure: Improving Research Quality Before Data Collection
Aguinis, H., & Vandenberg, R. J. (2014) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-031413-091231 · Cited by 304
Self-citationAguinis, H., Dalton, D. R., Bosco, F. A., Pierce, C. A., & Dalton, C. M. (2011) · Journal of Management · doi:10.1177/0149206310377113 · Cited by 302
Self-citationThe Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles
O’Boyle, E. H., Banks, G. C., & Gonzalez-Mulé, E. (2017) · Journal of Management · doi:10.1177/0149206314527133 · Cited by 246
Presenting Post Hoc Hypotheses as A Priori: Ethical and Theoretical Issues
Leung, K. (2011) · Management and Organization Review · doi:10.1111/j.1740-8784.2011.00222.x · Cited by 61
Cited by
Genome-wide association studies
Uffelmann, E., Huang, Q. Q., Munung, N. S., de Vries, J., Okada, Y., Martin, A. R., Martin, H. C., Lappalainen, T., & Posthuma, D. (2021) · Nature Reviews Methods Primers · doi:10.1038/s43586-021-00056-9 · Cited by 1,849
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Using Theory Elaboration to Make Theoretical Advancements
Fisher, G., & Aguinis, H. (2017) · Organizational Research Methods · doi:10.1177/1094428116689707 · Cited by 517
Self-citationOn doing better science: From thrill of discovery to policy implications
Antonakis, J. (2017) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.01.006 · Cited by 475
What You See Is What You Get? Enhancing Methodological Transparency in Management Research
Aguinis, H., Ramani, R. S., & Alabduljader, N. (2018) · Academy of Management Annals · doi:10.5465/annals.2016.0011 · Cited by 463
Self-citation
Metadata
- Authors
- Bosco, F. A., Aguinis, H., Field, J. G., Pierce, C. A., & Dalton, D. R.
- Venue
- Personnel Psychology
- Publisher
- Wiley
- Published
- 2015-09-16
- Volume
- 69
- Issue
- 3
- Pages
- 709-750
- ISSN
- 0031-5826 (ISSN-L)
- 1744-6570
- OpenAlex
- W1795381179
- Cited by
- 130
- References
- 17
- FWCI
- 15.55
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.