The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
When Does HARKing Hurt? Identifying When Different Types of Undisclosed Post Hoc Hypothesizing Harm Scientific Progress
Open accessPublished version of a preprint
Abstract
Hypothesizing after the results are known, or HARKing, occurs when researchers check their research results and then add or remove hypotheses on the basis of those results without acknowledging this process in their research report ( Kerr, 1998 ). In the present article, I discuss 3 forms of HARKing: (a) using current results to construct post hoc hypotheses that are then reported as if they were a priori hypotheses; (b) retrieving hypotheses from a post hoc literature search and reporting them as a priori hypotheses; and (c) failing to report a priori hypotheses that are unsupported by the current results. These 3 types of HARKing are often characterized as being bad for science and a potential cause of the current replication crisis. In the present article, I use insights from the philosophy of science to present a more nuanced view. Specifically, I identify the conditions under which each of these 3 types of HARKing is most and least likely to be bad for science. I conclude with a brief discussion about the ethics of each type of HARKing.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Falsification and the Methodology of Scientific Research Programmes
Lakatos, I. (1970) · Criticism and the Growth of Knowledge · doi:10.1017/cbo9781139171434.009 · Cited by 4,649
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Error and the Growth of Experimental Knowledge
Mayo, D. G. (1996) · doi:10.7208/chicago/9780226511993.001.0001 · Cited by 1,325
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Bem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893
Gigerenzer, G. (2004) · The Journal of Socio-Economics · doi:10.1016/j.socec.2004.09.033 · Cited by 890
Ferguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603
Questionable Research Practices Revisited
Fiedler, K., & Schwarz, N. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550615612150 · Cited by 279 · cited 1×
“…Journal of Applied Psychology during 2005 to 2010 “Whether any changes in hypotheses had occurred between the completion of data collection and subsequent publication” 53 38% Fiedler and Schwarz (2016) German psychologists “Reporting an unexpected finding as having been predicted from the start.””
BackgroundGiner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274
Questions About Questionable Research Practices in the Field of Management: A Guest Commentary
Banks, G. C., O’Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. (2016) · Journal of Management · doi:10.1177/0149206315619011 · Cited by 163
Presenting Post Hoc Hypotheses as A Priori: Ethical and Theoretical Issues
Leung, K. (2011) · Management and Organization Review · doi:10.1111/j.1740-8784.2011.00222.x · Cited by 61
Prediction and accommodation revisited
Worrall, J. (2014) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2013.10.001 · Cited by 37 · cited 4×
“…not been any in-depth consideration of more modern insights about the extent to which evidence provides novel information in relation to hypotheses (Worrall, 1985, 2010, 2014) or the extent to which severe tests of those hypotheses provide diagnostic information (Mayo, 1991, 1996, 2008, 2010,…”
BackgroundInfluential
Research works printed but not linked: 48
- 10.1371/journal.pone.0172792 · DOI · Paper page
- 10.1057/s41267-017-0081-0 · DOI · Paper page
- 10.5465/AMLE.2010.56659889 · DOI · Paper page
- Bem D. J., 1987, The compleat academic: A practical guide for the beginning social scientist, 171
- 10.1111/peps.12111 · DOI
- 10.5465/amle.2015.0201 · DOI
- 10.2307/2216049 · doi:10.2307/2216049
- 10.1016/j.actpsy.2014.02.001 · DOI · Paper page
- 10.1016/j.jmp.2015.10.003 · DOI · Paper page
- 10.1371/journal.pone.0010271 · DOI · Paper page
- 10.1207/s15327957pspr0203_1 · DOI
- 10.1093/bjps/axl004 · DOI
- 10.1207/s15327957pspr0203_2 · DOI · Paper page
- 10.1093/bjps/55.1.1 · DOI · Paper page
- 10.1177/0149206316679487 · DOI · Paper page
- HowsonC. (1988). Accommodation, prediction and Bayesian confirmation theory. Proceedings of the Biennial Meeting of the Philosophy of Science Association (Vol. 2, pp. 381–392). Philosophy of Science Association. · DOI
- 10.1177/0956797611430953 · DOI · Paper page
- 10.1111/iops.12045 · DOI
- 10.1093/acprof:osobl/9780199778188.003.0024 · DOI
- 10.1093/bjps/52.3.575 · DOI
- 10.1080/19312450701641409 · DOI
- 10.1037/a0039400 · DOI · Paper page
- 10.1086/289639 · DOI
- 10.1093/bjps/axn034 · DOI
- Mayo D. G., 2010, Error and inference: Recent exchanges on experimental reasoning, reliability, and the objectivity and rationality of science, 155
- 10.1016/j.shpsa.2013.10.005 · DOI
- 10.1093/bjps/axl003 · DOI · Paper page
- 10.1111/iops.12049 · DOI
- 10.1037/pspa0000084 · DOI
- 10.1177/1088868313496330 · DOI
- 10.1093/bjps/25.1.1 · DOI · Paper page
- O'Boyle E. H., 2017, Journal of Management, 43, 367
- ParkerC. G. (2015). Reframing the reproducibility crisis: Using an error-statistical account to inform the interpretation of replication results in psychological research. Doctoral dissertation, Virginia Polytechnic Institute and State University. Retrieved from https://vtechworks.lib.vt.edu/handle/10919/52963
- Popper K. R., 1979, Objective knowledge: An evolutionary approach
- 10.1037/gpr0000123 · DOI · Paper page
- 10.1080/00048408712342751 · doi:10.1080/00048408712342751
- 10.5465/amle.2016.0039 · DOI
- SimonsD. J., ShodaY. & LindsayD. S. (2017, April 17). Constraints on generality (COG): A proposed addition to all empirical papers. Retrieved from http://osf.io/preprints/psyarxiv/w9e3r · DOI
- 10.1016/j.jesp.2015.09.017 · DOI
- Ṡwiątkowski W., 2017, Revue Internationale de Psychologie Sociale, 30, 111
- 10.3389/fpsyg.2016.01444 · DOI
- 10.1027/1864-9335/a000270 · DOI
- 10.1016/j.jesp.2016.03.004 · DOI · Paper page
- Wagenmakers E. J., 2015, Extrasensory perception: Support, skepticism, and science:, 153
- 10.1037/a0022790 · DOI · Paper page
- 10.1007/978-94-009-6525-6_11 · DOI
- Worrall J., 2010, Error and inference: Recent exchanges on experimental reasoning, reliability, and the objectivity and rationality of science, 125
- 10.1093/bjps/24.2.95 · DOI · Paper page
Cited by
Addressing the theory crisis in psychology
Oberauer, K., & Lewandowsky, S. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01645-2 · Cited by 462
Van den Broeck, A., Howard, J. L., Van Vaerenbergh, Y., Leroy, H., & Gagné, M. (2021) · Organizational Psychology Review · doi:10.1177/20413866211006173 · Cited by 409
Rubin, M. (2021) · Synthese · doi:10.1007/s11229-021-03276-4 · Cited by 371
Self-citationFrom Omics to Multi-Omics: A Review of Advantages and Tradeoffs
Hayes, C. N., Nakahara, H., Ono, A., Tsuge, M., & Oka, S. (2024) · Genes · doi:10.3390/genes15121551 · Cited by 119
Methodology over metrics: current scientific standards are a disservice to patients and society
Van Calster, B., Wynants, L., Riley, R. D., van Smeden, M., & Collins, G. S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.018 · Cited by 105
“HARKing (hypothesizing after the results are known) HARKing means that parts of a publication (such as the introduction and the hypothesis) are written to accommodate the final results [38] .”
Background
Metadata
- Authors
- Rubin, M.
- Venue
- Review of General Psychology
- Publisher
- SAGE Publications
- Published
- 2017-12
- Volume
- 21
- Issue
- 4
- Pages
- 308-320
- ISSN
- 1089-2680 (ISSN-L)
- 1939-1552
- OpenAlex
- W2765220332
- Cited by
- 182
- References
- 16
- FWCI
- 22.57
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.