The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles
Abstract
The issue of a published literature not representative of the population of research is most often discussed in terms of entire studies being suppressed. However, alternative sources of publication bias are questionable research practices (QRPs) that entail post hoc alterations of hypotheses to support data or post hoc alterations of data to support hypotheses. Using general strain theory as an explanatory framework, we outline the means, motives, and opportunities for researchers to better their chances of publication independent of rigor and relevance. We then assess the frequency of QRPs in management research by tracking differences between dissertations and their resulting journal publications. Our primary finding is that from dissertation to journal article, the ratio of supported to unsupported hypotheses more than doubled (0.82 to 1.00 versus 1.94 to 1.00). The rise in predictive accuracy resulted from the dropping of statistically nonsignificant hypotheses, the addition of statistically significant hypotheses, the reversing of predicted direction of hypotheses, and alterations to data. We conclude with recommendations to help mitigate the problem of an unrepresentative literature that we label the “Chrysalis Effect.”
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Published studies reported a ratio of supported to unsupported hypotheses more than twice as high as dissertations, presumably because peer-reviewed publications are more likely to report significant results.
“This trend aligns with findings such as those in O’Boyle et al.’s (2017)’s management review, which found that published studies reported a ratio of supported to unsupported hypotheses that was more than twice as high as those in dissertations, presumably because peer-reviewed publications are more likely to report significant results.”
From A Meta-analysis of University STEM Summer Bridge Program Effectiveness (2021) · cites this paper as O’Boyle et al.’s (2017)
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Mixing Qualitative and Quantitative Methods: Triangulation in Action.
Jick, T. D. (1979) · Administrative Science Quarterly · doi:10.2307/2392366 · Cited by 7,103 · cited 1× · see citing works
“Triangulation is characterized as the use of “multiple reference points to locate an object’s exact position” (Jick, 1979, p. 602).”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
Merton, R. K. (2017) · Gangs · doi:10.4324/9781351157803-1 · Cited by 3,689 · see citing works
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989 · see citing works
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781 · see citing works
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · see citing works
What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564 · cited 1× · see citing works
“There has been a repeated call for management researchers, reviewers, and editors to stop overemphasizing statistical significance in their evaluation of research quality (J. Cohen, 1994; Gigerenzer, 2004; Meehl, 1978; Nickerson, 2000; Schmidt, 1992, 1996; Schmidt & Hunter, 2002; Wagenmakers, 2007).”
BackgroundScientific Misconduct and the Myth of Self-Correction in Science
Stroebe, W., Postmes, T., & Spears, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460687 · Cited by 323 · see citing works
A peculiar prevalence of <i>p</i> values just below .05
Masicampo, E. J., & Lalande, D. R. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.711335 · Cited by 266 · see citing works
Testing for the Presence of Positive-Outcome Bias in Peer Review: A Randomized Controlled Trial
Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J. D., Brand, R. A., & Leopold, S. S. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.406 · Cited by 245 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · see citing works
Self-citationThe role of leadership in a digitalized world: a review
Cortellazzo, L., Bruni, E., & Zampieri, R. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01938 · Cited by 808 · see citing works
Pratt, M. G., Kaplan, S., & Whittington, R. (2020) · Administrative Science Quarterly · doi:10.1177/0001839219887663 · Cited by 458 · see citing works
Questionable research practices in ecology and evolution
Fraser, H., Parker, T., Nakagawa, S., Barnett, A., & Fidler, F. (2018) · PLOS ONE · doi:10.1371/journal.pone.0200303 · Cited by 346 · see citing works
Mackey, J. D., Parker Ellen, B., McAllister, C. P., & Alexander, K. C. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2020.10.037 · Cited by 320 · see citing works
Questions About Questionable Research Practices in the Field of Management: A Guest Commentary
Banks, G. C., O’Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. (2016) · Journal of Management · doi:10.1177/0149206315619011 · Cited by 163 · see citing works
Self-citationHilgard, J., Engelhardt, C. R., & Rouder, J. N. (2017) · Psychological Bulletin · doi:10.1037/bul0000074 · Cited by 162 · see citing works
Editors’ JIF-boosting stratagems – Which are appropriate and which not?
Martin, B. R. (2016) · Research Policy · doi:10.1016/j.respol.2015.09.001 · Cited by 128 · see citing works
Study preregistration: An evaluation of a method for transparent reporting
Toth, A. A., Banks, G. C., Mellor, D., O’Boyle, E. H., Dickson, A., Davis, D. J., DeHaven, A., Bochantin, J., & Borns, J. (2021) · Journal of Business and Psychology · doi:10.1007/s10869-020-09695-3 · Cited by 73 · see citing works
Self-citationA Meta-analysis of University STEM Summer Bridge Program Effectiveness
Bradford, B. C., Beier, M. E., & Oswald, F. L. (2021) · CBE—Life Sciences Education · doi:10.1187/cbe.20-03-0046 · Cited by 73 · see citing works
How follower traits and cultural values influence the effects of leadership
Matthews, S. H., Kelemen, T. K., & Bolino, M. C. (2021) · The Leadership Quarterly · doi:10.1016/j.leaqua.2021.101497 · Cited by 69 · see citing works
Where Are the Self-Correcting Mechanisms in Science?
Vazire, S., & Holcombe, A. O. (2022) · Review of General Psychology · doi:10.1177/10892680211033912 · Cited by 81 · see citing works
A Systems-Based Approach to Fostering Robust Science in Industrial-Organizational Psychology
Grand, J. A., Rogelberg, S. G., Allen, T. D., Landis, R. S., Reynolds, D. H., Scott, J. C., Tonidandel, S., & Truxillo, D. M. (2018) · Industrial and Organizational Psychology · doi:10.1017/iop.2017.55 · Cited by 52 · see citing works
- Authors
- O’Boyle, E. H., Banks, G. C., & Gonzalez-Mulé, E.
- Affiliations
- University of Iowa (US)
- Longwood University (US)
- University of Iowa
- Corresponding author
- O’Boyle, Ernest Hugh
- Venue
- Journal of Management
- Publisher
- SAGE Publications
- Published
- 2016-07-10
- Volume
- 43
- Issue
- 2
- Pages
- 376-399
- ISSN
- 0149-2063 (ISSN-L)
- 1557-1211
- OpenAlex
- W2147516620
- Cited by
- 246
- References
- 12
- Citations by year
- 2026: 12 · 2025: 14 · 2024: 11 · 2023: 24 · 2022: 15 · 2021: 24 · 2020: 28 · 2019: 25 · 2018: 24 · 2017: 29 · 2016: 21 · 2015: 12 · 2014: 6
- FWCI
- 59.28
- Type
- journal article
- Language
- en
- Update notices
- correction (2016-08-26)
- Open access
- closed
- Locations
- Journal of Management (publishedVersion)
- Topics
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- questionable research practices, publication bias, general strain theory, hypothesis revision, data manipulation, management research, research integrity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
