Science or Art? How Aesthetic Standards Grease the Way Through the Publication Bottleneck but Undermine Science
Abstract
The current crisis in psychological research involves issues of fraud, replication, publication bias, and false positive results. I argue that this crisis follows the failure of widely adopted solutions to psychology’s similar crisis of the 1970s. The untouched root cause is an information-economic one: Too many studies divided by too few publication outlets equals a bottleneck. Articles cannot pass through just by showing theoretical meaning and methodological rigor; their results must appear to support the hypothesis perfectly. Consequently, psychologists must master the art of presenting perfect-looking results just to survive in the profession. This favors aesthetic criteria of presentation in a way that harms science’s search for truth. Shallow standards of statistical perfection distort analyses and undermine the accuracy of cumulative data; narrative expectations encourage dishonesty about the relationship between results and hypotheses; criteria of novelty suppress replication attempts. Concerns about truth in research are emerging in other sciences and may eventually descend on our heads in the form of difficult and insensitive regulations. I suggest a more palatable solution: to open the bottleneck, putting structures in place to reward broader forms of information sharing beyond the exquisite art of present-day journal publication.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Researchers can confirm the absence of p-hacking in their research reports through contemporary transparency practices such as actively affirming disclosure of data collection stopping rule, data exclusions, measures, and manipulations; providing logical and principled justifications for nonstandard data exclusions and analytical approaches; providing public access to standard data analysis and to
“researchers can confirm the absence of p -hacking in their research reports by (a) actively affirming the disclosure of their data collection stopping rule, data exclusions, measures, and manipulations (Simmons, Nelson, & Simonsohn, 2012), (b) providing logical and principled justifications for nonstandard data exclusions and analytical approaches (Giner-Sorolla, 2012, p. 568), (c) providing”
From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Giner-Sorolla, 2012, p. 568
Clean papers in which the evidence consistently supports the proposed conclusion are more likely to be published than papers with equivocal evidence, exceptions, or other uncertainties.
“And, clean papers in which the evidence consistently supports the proposed conclusion are more likely to be published than papers with equivocal evidence, exceptions, or other uncertainties 16,17 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [17]
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
The Elaboration Likelihood Model of Persuasion
Petty, R. E., & Cacioppo, J. T. (1986) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60214-2 · Cited by 8,822 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Chaiken, S. (1980) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.39.5.752 · Cited by 5,516 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657 · see citing works
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1× · see citing works
“Confidence in the applied usefulness of the publication process has taken a hit, as clinical trials have demonstrated unusually low success rates when trying to reproduce academic results (Begley & Ellis, 2012; Prinz, Schlange, & Asadullah, 2011; Turner, Matthews, et al., 2008).”
ResultProcessing Fluency and Aesthetic Pleasure: Is Beauty in the Perceiver's Processing Experience?
Reber, R., Schwarz, N., & Winkielman, P. (2004) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0804_3 · Cited by 2,967 · see citing works
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791 · see citing works
Selective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435 · see citing works
Wicker, A. W. (1969) · Journal of Social Issues · doi:10.1111/j.1540-4560.1969.tb00619.x · Cited by 2,152 · see citing works
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989 · see citing works
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
The Evolutionary Psychology of Facial Beauty
Rhodes, G. (2005) · Annual Review of Psychology · doi:10.1146/annurev.psych.57.102904.190208 · Cited by 1,757 · see citing works
Statistical significance in psychological research.
Lykken, D. T. (1968) · Psychological bulletin · doi:10.1037/h0026141 · Cited by 1,363 · see citing works
Changing Order: Replication and Induction in Scientific Practice
Star, S. L., & Collins, H. M. (1988) · Technology and Culture · doi:10.2307/3105095 · Cited by 1,283 · see citing works
Daston, L., & Galison, P. (1992) · Representations · doi:10.2307/2928741 · Cited by 1,269 · see citing works
Artifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,183 · see citing works
Changing order: Replication and induction in scientific practice
(1987) · Journal of Economic Psychology · doi:10.1016/0167-4870(87)90014-6 · Cited by 1,133 · see citing works
The Handbook of Research Synthesis
Hanka, R. (1994) · BMJ · doi:10.1136/bmj.309.6952.488a · Cited by 1,064 · see citing works
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 2× · see citing works
“However, other issues concerned all areas of psychology: limitations of null-hypothesis significance testing, bias toward positive results in publication, and the resulting lack of credibility of the standard research article (Elms, 1975; Greenwald, 1975).”
BackgroundWagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977 · see citing works
Bem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893 · see citing works
Gigerenzer, G. (2004) · The Journal of Socio-Economics · doi:10.1016/j.socec.2004.09.033 · Cited by 890 · see citing works
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · see citing works
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751 · see citing works
The stability of behavior: II. Implications for psychological research.
Epstein, S. J. (1980) · American Psychologist · doi:10.1037/0003-066x.35.9.790 · Cited by 740 · see citing works
Publication Bias in Meta‐Analysis
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch1 · Cited by 603 · see citing works
The Evolution of the Physicist's Picture of Nature
Dirac, P. A. M. (1963) · Scientific American · doi:10.1038/scientificamerican0563-45 · Cited by 540 · see citing works
Why Current Publication Practices May Distort Science
Young, N. S., Ioannidis, J. P. A., & Al‐Ubaydli, O. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050201 · Cited by 504 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 473 · see citing works
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453 · see citing works
Scientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar‐Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 366 · see citing works
Final Collapse of the Neyman-Pearson Decision Theoretic Framework and Rise of the neoFisherian
Hurlbert, S. H., & Lombardi, C. M. (2009) · Annales Zoologici Fennici · doi:10.5735/086.046.0501 · Cited by 277 · see citing works
The crisis of confidence in social psychology.
Elms, A. C. (1975) · American Psychologist · doi:10.1037/0003-066x.30.10.967 · Cited by 251 · see citing works
Exploration in animals and humans
Archer, J. E., & Birke, L. I. A. (1983) · Medical Entomology and Zoology · Cited by 244 · see citing works
Too good to be true: Publication bias in two prominent studies from experimental psychology
Francis, G. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0227-9 · Cited by 210 · see citing works
The Merits of Unconscious Thought in Creativity
Zhong, C., Dijksterhuis, A., & Galinsky, A. D. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02176.x · Cited by 200 · see citing works
Failure to replicate mood-dependent retrieval
Bower, G. H., & Mayer, J. D. (1985) · Bulletin of the Psychonomic Society · doi:10.3758/bf03329773 · Cited by 184 · see citing works
Reforming Science: Methodological and Cultural Reforms
Casadevall, A., & Fang, F. C. (2012) · Infection and Immunity · doi:10.1128/iai.06183-11 · Cited by 156 · see citing works
LeBel, E. P., & Peters, K. R. (2011) · Review of General Psychology · doi:10.1037/a0025172 · Cited by 152 · see citing works
The social psychology of science
Gergen, K. J. (1994) · Choice Reviews Online · doi:10.5860/choice.32-0910 · Cited by 119 · see citing works
Does Sunshine Prime Loyal? Affective Priming in the Naming Task
Klauer, K. C., & Musch, J. (2001) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/713755986 · Cited by 108 · see citing works
Psychology’s Woes and a Partial Cure: The Value of Replication
Roediger, H. L., III (2012) · APS observer · Cited by 101 · see citing works
The great betrayal: fraud in science
(2005) · Choice Reviews Online · doi:10.5860/choice.42-4611 · Cited by 99 · see citing works
Exploring the Cold-to-Hot Empathy Gap in Smokers
Sayette, M. A., Loewenstein, G. F., Griffin, K. M., & Black, J. J. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02178.x · Cited by 96 · see citing works
Misprescription and misuse of one-tailed tests
LOMBARDI, C. M., & HURLBERT, S. H. (2009) · Austral Ecology · doi:10.1111/j.1442-9993.2009.01946.x · Cited by 93 · see citing works
J. B. Watson's Little Albert, Cyril Burt's twins, and the need for a critical science.
Samelson, F. (1980) · American Psychologist · doi:10.1037/0003-066x.35.7.619 · Cited by 92 · see citing works
The Nature of Social and Personality Psychology as Reflected in JPSP, 1965-2000.
Quiñones-Vidal, E., García, J. J. L., Ortega, M. P., & Tortosa-Gil, F. (2004) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.86.3.435 · Cited by 90 · see citing works
Publication Trends in JPSP: A Three-Decade Review
Reis, H. T., & Stiller, J. D. (1992) · Personality and Social Psychology Bulletin · doi:10.1177/0146167292184011 · Cited by 86 · see citing works
Mackintosh, N. J. (1995) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780198523369.001.0001 · Cited by 82 · see citing works
Beware the creeping cracks of bias
Sarewitz, D. (2012) · Nature · doi:10.1038/485149a · Cited by 78 · see citing works
Short, Sweet, and Problematic? The Rise of the Short Report in Psychological Science
Ledgerwood, A., & Sherman, J. W. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611427304 · Cited by 72 · see citing works
Twenty Years of PSPB: Trends in Content, Design, and Analysis
Sherman, R. C., Buddie, A. M., Dragan, K. L., End, C. M., & Finney, L. J. (1999) · Personality and Social Psychology Bulletin · doi:10.1177/0146167299025002004 · Cited by 65 · see citing works
Meta-analytic synthesis for theory development.
Miller, N., & Pollock, V. E. (1994) · Cited by 61 · see citing works
Crisis discussions in psychology—New historical and philosophical perspectives
Sturm, T., & Mülberger, A. (2011) · Studies in History and Philosophy of Science Part C Studies in History and Philosophy of Biological and Biomedical Sciences · doi:10.1016/j.shpsc.2011.11.001 · Cited by 53 · see citing works
The Great Betrayal: Fraud in Science
Wilmshurst, P. T. (2004) · BMJ · doi:10.1136/bmj.329.7471.922 · Cited by 49 · see citing works
The A nna K arenina principle: A way of thinking about success in science
Bornmann, L., & Marx, W. (2012) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22661 · Cited by 49 · see citing works
Bones, A. K. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612441216 · Cited by 48 · see citing works
The Evolution of Sex Differences in Jealousy: Failure to Replicate Previous Results
Grice, J. W., & Seely, E. (2000) · Journal of Research in Personality · doi:10.1006/jrpe.2000.2284 · Cited by 35 · see citing works
Replication, replication, replication
Ritchie, S. J., Wiseman, R. J., & French, C. C. (2012) · University of Hertfordshire Research Archive (University of Hertfordshire) · Cited by 33 · see citing works
AESTHETICS IN SCIENCE AND IN ART
Engler, G. (1990) · The British Journal of Aesthetics · doi:10.1093/bjaesthetics/30.1.24 · Cited by 32 · see citing works
On the Use of Narrative as Argument
Voss, J. F., Wiley, J., & Sandak, R. (1999) · Routledge eBooks · doi:10.4324/9781410603135-17 · Cited by 21 · see citing works
Implicit editorial policies and the integrity of psychology as an empirical science.
Wong, P. T. P. (1981) · American Psychologist · doi:10.1037/0003-066x.36.6.690 · Cited by 14 · see citing works
Social Psychological Research: The Comparison of Four Journals
Witte, E. H., & Brandt, V. C. (2010) · Social Science Open Access Repository (GESIS – Leibniz Institute for the Social Sciences) · doi:10.23668/psycharchives.9080 · Cited by 9 · see citing works
Did Sir Cyril Burt Fake His Research on Heritability of Intelligence? Part I.
Gillie, O. (1977) · Phi Delta Kappan · Cited by 9 · see citing works
An effort at studying Rorschach content symbolism: The frog response.
Sapolsky, A. (1964) · Journal of Consulting Psychology · doi:10.1037/h0046144 · Cited by 8 · see citing works
What to do about fraud charges in science; or, will the Burt affair ever end?
Samelson, F. (1997) · Genetica · doi:10.1007/bf02259518 · Cited by 7 · see citing works
What’s new in the economics of arts and culture?
Potts, J. (2007) · Dialogue A Journal of Mormon Thought · Cited by 4 · see citing works
Flagged first, then by citations
1 of the 70 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · see citing works
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · see citing works
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · see citing works
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · see citing works
The Value of Direct Replication
Simons, D. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514755 · Cited by 630 · see citing works
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563 · see citing works
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510 · see citing works
“Whatever the cause of this over-representation might be, there seems to be too much focus on getting B perfect, ^ significant results (see also Giner-Sorolla, 2012).”
Replication, falsification, and the crisis of confidence in social psychology
Earp, B. D., & Trafimow, D. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00621 · Cited by 472 · see citing works
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471 · see citing works
Deep impact: unintended consequences of journal rank
Brembs, B., Button, K., & Munafò, M. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00291 · Cited by 441 · see citing works
Prevention is better than cure: addressing anti-vaccine conspiracy theories
Jolley, D., & Douglas, K. M. (2017) · Journal of Applied Social Psychology · doi:10.1111/jasp.12453 · Cited by 446 · see citing works
Pre-registration in social psychology—A discussion and suggested template
Veer, A. V. '., & Giner‐Sorolla, R. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.03.004 · Cited by 436 · see citing works
Self-citationSwedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 408 · see citing works
Implications of the Credibility Revolution for Productivity, Creativity, and Progress
Vazire, S. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617751884 · Cited by 361 · see citing works
Fair Statistical Communication in HCI
Dragicevic, P. (2016) · Human-computer interaction series · doi:10.1007/978-3-319-26633-6_13 · Cited by 339 · see citing works
Implementing Reproducible Research
Stodden, V. (2014) · doi:10.1201/b16868 · Cited by 314 · see citing works
Challenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 323 · see citing works
Continuously Cumulating Meta-Analysis and Replicability
Braver, S. L., Thoemmes, F. J., & Rosenthal, R. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614529796 · Cited by 298 · see citing works
Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01142-4 · Cited by 289 · see citing works
A Short (Personal) Future History of Revolution 2.0
Spellman, B. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615609918 · Cited by 213 · see citing works
Sommet, N., Weissman, D. L., Cheutin, N., & Elliot, A. J. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231178728 · Cited by 274 · see citing works
An Agenda for Open Science in Communication
Dienlin, T., Johannes, N., Bowman, N. D., Masur, P. K., Engesser, S., Kümpel, A. S., Lukito, J., Bier, L. M., Zhang, R., Johnson, B. K., et al. (2020) · Journal of Communication · doi:10.1093/joc/jqz052 · Cited by 221 · see citing works
Rubin, M. (2017) · Review of General Psychology · doi:10.1037/gpr0000128 · Cited by 181 · see citing works
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 190 · see citing works
Ejelöv, E., & Luke, T. J. (2019) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2019.103937 · Cited by 175 · see citing works
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168 · see citing works
The Preregistration RevolutionPreprint
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017) · doi:10.31219/osf.io/2dxu5 · Cited by 220 · see citing works
Reconceptualizing replication as a sequence of different studies: A replication typology
Hüffmeier, J., Mazei, J., & Schultze, T. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.009 · Cited by 168 · see citing works
Lessons From Pinocchio: Cues to Deception May Be Highly Exaggerated
Luke, T. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691619838258 · Cited by 135 · see citing works
Let’s Put Our Money Where Our Mouth Is
Maner, J. K. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528215 · Cited by 151 · see citing works
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 135 · see citing works
Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education
Cook, B. G., Lloyd, J., Mellor, D., Nosek, B. A., & Therrien, W. J. (2018) · The Exceptional Child · doi:10.1177/0014402918793138 · Cited by 115 · see citing works
Atkinson, K. M., Koenka, A. C., Sanchez, C. E., Moshontz, H., & Cooper, H. M. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1127 · Cited by 126 · see citing works
How Trustworthy Is the Scientific Literature in Industrial and Organizational Psychology?
Kepes, S., & McDaniel, M. A. (2013) · Industrial and Organizational Psychology · doi:10.1111/iops.12045 · Cited by 125 · see citing works
Increasing the Transparency of Research Papers with Explorable Multiverse Analyses
Dragicevic, P., Jansen, Y., Sarma, A., Kay, M., & Chevalier, F. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300295 · Cited by 123 · see citing works
Understanding the Replication Crisis as a Base Rate Fallacy
Bird, A. (2018) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/axy051 · Cited by 122 · see citing works
P-hacking and publication bias interact to distort meta-analytic effect size estimates
Friese, M., & Frankenbach, J. (2020) · Psychological Methods · doi:10.1037/met0000246 · Cited by 122 · see citing works
Replication Research and Special Education
Travers, J. C., Cook, B. G., Therrien, W. J., & Coyne, M. D. (2016) · Remedial and Special Education · doi:10.1177/0741932516648462 · Cited by 118 · see citing works
Replicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future
Świątkowski, W., & Dompnier, B. (2017) · International Review of Social Psychology · doi:10.5334/irsp.66 · Cited by 116 · see citing works
On the Other Side of the Mirror: Priming in Cognitive and Social Psychology
Doyen, S., Klein, O., Simons, D. J., & Cleeremans, A. (2014) · Social Cognition · doi:10.1521/soco.2014.32.supp.12 · Cited by 112 · see citing works
Quantifying Support for the Null Hypothesis in Psychology: An Empirical Investigation
Aczél, B., Pálfi, B., Szollosi, A., Kovács, M., Szászi, B., Szécsi, P., Zrubka, M., Gronau, Q. F., van den Bergh, D., & Wagenmakers, E. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918773742 · Cited by 111 · see citing works
Some Whys and Hows of Experiments in Human—Computer Interaction
Hornbæk, K. (2013) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000043 · Cited by 109 · see citing works
van Assen, M. A. L. M., van Aert, R. C. M., Nuijten, M. B., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0084896 · Cited by 110 · see citing works
Quality Uncertainty Erodes Trust in Science
Vazire, S. (2017) · Collabra: Psychology · doi:10.1525/collabra.74 · Cited by 106 · see citing works
Prospects of functional magnetic resonance imaging as lie detector
Rusconi, E., & Mitchener-Nissen, T. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00594 · Cited by 106 · see citing works
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98 · see citing works
LeBel, E. P., Borsboom, D., Giner‐Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., & Smith, C. T. (2013) · Perspectives on Psychological Science · doi:10.1177/1745691613491437 · Cited by 103 · see citing works
Self-citationI feel good, therefore I am real: Testing the causal influence of mood on state authenticity
Lenton, A. P., Slabu, L., Sedikides, C., & Power, K. (2013) · Cognition & Emotion · doi:10.1080/02699931.2013.778818 · Cited by 103 · see citing works
A Call for Examining Replication and Bias in Special Education Research
Cook, B. G. (2014) · Remedial and Special Education · doi:10.1177/0741932514528995 · Cited by 97 · see citing works
The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates
Nuijten, M., van Assen, M., Veldkamp, C. L. S., & Wicherts, J. (2015) · Review of General Psychology · doi:10.1037/gpr0000034 · Cited by 60 · see citing works
Why Selective Publication of Statistically Significant Results Can Be Effective
de Winter, J., & Happee, R. (2013) · PLoS ONE · doi:10.1371/journal.pone.0066463 · Cited by 67 · see citing works
Scientists’ Reputations Are Based on Getting It Right, Not Being Right
Ebersole, C. R., Axt, J. R., & Nosek, B. A. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002460 · Cited by 67 · see citing works
Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully
Hoogeveen, S., Sarafoglou, A., & Wagenmakers, E. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920919667 · Cited by 65 · see citing works
Zlatev, J. J., & Miller, D. T. (2016) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2016.08.004 · Cited by 62 · see citing works
Does preregistration improve the credibility of research findings?
Rubin, M. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.4.p376 · Cited by 61 · see citing works
“…(Simmons, Nelson, & Simonsohn, 2012), (b) providing logical and principled justifications for nonstandard data exclusions and analytical approaches (Giner-Sorolla, 2012, p. 568), (c) providing public access to their standard data analysis procedures (e.g., Lin & Green, 2016), (d) providing public…”
Methodvan den Akker, O. R., van Assen, M. A. L. M., Bakker, M., Elsherif, M. M., Wong, T. K., & Wicherts, J. M. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02277-0 · Cited by 78 · see citing works
Into the wild: Field research can increase both replicability and real-world impact
Maner, J. K. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.018 · Cited by 61 · see citing works
Improving Psychological Science through Transparency and Openness: An Overview
Hales, A. H., Wesselmann, E. D., & Hilgard, J. (2018) · Perspectives on Behavior Science · doi:10.1007/s40614-018-00186-8 · Cited by 61 · see citing works
Self-Determination Theory and HCI Games Research: Unfulfilled Promises and Unquestioned Paradigms
Tyack, A., & Mekler, E. D. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3673230 · Cited by 57 · see citing works
Solutions to the problems of incremental validity testing in relationship science
Wang, Y. A., & Eastwick, P. W. (2020) · Personal Relationships · doi:10.1111/pere.12309 · Cited by 57 · see citing works
Open Science Promotes Diverse, Just, and Sustainable Research and Educational Outcomes
Grahe, J. E., Cuccolo, K., Leighton, D. C., & Alvarez, L. D. C. (2019) · Psychology Learning & Teaching · doi:10.1177/1475725719869164 · Cited by 58 · see citing works
From Sex Objects to Human Beings
Bernard, P., Gervais, S. J., Allen, J. M., Delmée, A., & Klein, O. (2015) · Psychology of Women Quarterly · doi:10.1177/0361684315580125 · Cited by 55 · see citing works
Sleep spindles and general cognitive ability – A meta-analysis
Ujma, P. P. (2018) · Sleep Spindles & Cortical Up States · doi:10.1556/2053.2.2018.01 · Cited by 49 · see citing works
Coyne, J. C., & Kwakkenbos, L. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-11 · Cited by 51 · see citing works
Displaced revenge: Can revenge taste “sweet” if it aims at a different target?
Sjöström, A., & Gollwitzer, M. (2014) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2014.09.016 · Cited by 50 · see citing works
Frightened by an Old Scarecrow: The Remarkable Resilience of Demand Characteristics
Sharpe, D., & Whelton, W. J. (2016) · Review of General Psychology · doi:10.1037/gpr0000087 · Cited by 50 · see citing works
Giner‐Sorolla, R. (2018) · European Review of Social Psychology · doi:10.1080/10463283.2018.1542902 · Cited by 48 · see citing works
Self-citationOpen Science: What, Why, and HowPreprint
Spellman, B., Gilbert, E. A., & Corker, K. S. (2017) · doi:10.31234/osf.io/ak6jr · Cited by 50 · see citing works
A Concise Set of Core Recommendations to Improve the Dependability of Psychological Research
Lishner, D. A. (2015) · Review of General Psychology · doi:10.1037/gpr0000028 · Cited by 49 · see citing works
Psychology, replication & beyond
Laws, K. R. (2016) · BMC Psychology · doi:10.1186/s40359-016-0135-2 · Cited by 47 · see citing works
Krishna, A., & Peter, S. M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0203470 · Cited by 46 · see citing works
Too Good to be False: Nonsignificant Results Revisited
Hartgerink, C., Wicherts, J., & van Assen, M. (2017) · Collabra Psychology · doi:10.1525/collabra.71 · Cited by 46 · see citing works
Replication crisis or an opportunity to improve scientific production?
Navarro, M. D. F., Pascual‐Llobell, J., Pascual‐Soler, M., Perezgonzalez, J. D., & Berríos-Riquelme, J. (2020) · European Journal of Education · doi:10.1111/ejed.12417 · Cited by 46 · see citing works
Negativland - a home for all findings in psychology
Laws, K. R. (2013) · BMC Psychology · doi:10.1186/2050-7283-1-2 · Cited by 43 · see citing works
Campbell, L., Loving, T. J., & LeBel, E. P. (2014) · Personal Relationships · doi:10.1111/pere.12053 · Cited by 35 · see citing works
Investigating the replicability of the social and behavioural sciences
Tyner, A. H., Abatayo, A. L., Daley, M., Field, S., Fox, N. W., Haber, N., Hahn, K. M., Struhl, M. K., Mawhinney, B., Miske, O., et al. (2026) · Nature · doi:10.1038/s41586-025-10078-y · Cited by 43 · see citing works
Friesen, J. P., Kawakami, K., Vingilis‐Jaremko, L., Caprara, R., Sidhu, D. M., Williams, A., Hugenberg, K., Rodríguez‐Bailón, R., Cañadas, E., & Niedenthal, P. M. (2019) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000139 · Cited by 40 · see citing works
Stracke, C. M. (2020) · Radical Solutions and Open Science · doi:10.1007/978-981-15-4276-3_2 · Cited by 24 · see citing works
Frequentist and Bayesian inference: A conceptual primer
van Zyl, C. J. J. (2018) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2018.06.004 · Cited by 38 · see citing works
Sommet, N., Weissman, D. L., Cheutin, N., & Elliot, A. (2022) · doi:10.31219/osf.io/xhe3u · Cited by 36 · see citing works
How can preregistration contribute to research in our field?
Jonas, K. J., & Cesario, J. (2015) · Comprehensive Results in Social Psychology · doi:10.1080/23743603.2015.1070611 · Cited by 36 · see citing works
Psychological science’s replicability crisis and what it means for science in the courtroom.
Chin, J. M. (2014) · Psychology Public Policy and Law · doi:10.1037/law0000012 · Cited by 39 · see citing works
Radical Solutions and Open Science
Burgos, D. (2020) · Lecture notes in educational technology · doi:10.1007/978-981-15-4276-3 · Cited by 43 · see citing works
Does the conclusion follow from the evidence? Recommendations for improving research
Hales, A. H. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.011 · Cited by 34 · see citing works
Nichols, A., & Edlund, J. E. (2015) · Review of General Psychology · doi:10.1037/gpr0000027 · Cited by 34 · see citing works
van den Akker, O. R., van Assen, M. A. L. M., Enting, M., de Jonge, M., Ong, H. H., Rüffer, F. F., Schoenmakers, M., Stoevenbelt, A. H., Wicherts, J. M., & Bakker, M. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231187988 · Cited by 39 · see citing works
Gibson, S. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108367943 · Cited by 64 · see citing works
The empirical march: Making science better at self-correction.
Makel, M. C. (2014) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/a0035803 · Cited by 32 · see citing works
An Introduction to Replication Research in Gifted Education
Makel, M. C., & Plucker, J. A. (2015) · Gifted Child Quarterly · doi:10.1177/0016986215578747 · Cited by 31 · see citing works
Bernard, P., Content, J., Servais, L., Wollast, R., & Gervais, S. J. (2020) · Sex Roles · doi:10.1007/s11199-019-01115-y · Cited by 33 · see citing works
Special Interest Group on Transparent Statistics in HCI
Kay, M., Haroz, S., Guha, S., & Dragicevic, P. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2886442 · Cited by 31 · see citing works
Schneider, J. W., Allum, N., Andersen, J. P., Petersen, M. B., Madsen, E. B., Mejlgaard, N., & Zachariae, R. H. (2024) · PLoS ONE · doi:10.1371/journal.pone.0304342 · Cited by 30 · see citing works
Evaluating Registered Reports: A Naturalistic Comparative Study of Article Impact
Hopun, L. T., Singleton Thorn, F., Nosek, B. A., & Errington, T. M. (2017) · doi:10.31219/osf.io/5y8w7 · Cited by 29 · see citing works
Pre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 30 · see citing works
Grosz, M. P., Dufner, M., Back, M. D., & Denissen, J. J. A. (2015) · Journal of Research in Personality · doi:10.1016/j.jrp.2015.05.007 · Cited by 28 · see citing works
Kerr, N. L., Ao, X., Hogg, M. A., & Zhang, J. (2018) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2018.05.001 · Cited by 28 · see citing works
Spellman, B. A., Gilbert, E. A., & Corker, K. S. (2018) · Wiley eBooks · doi:10.1002/9781119170174.epcn519 · Cited by 84 · see citing works
Psychology Exceptionalism and the Multiple Discovery of the Replication Crisis
Nelson, N. C., Chung, J., Ichikawa, K., & Malik, M. M. (2021) · Review of General Psychology · doi:10.1177/10892680211046508 · Cited by 25 · see citing works
Pre-Registration in Social Psychology—a Discussion and Suggested TemplatePreprint
Veer, A. V. '., & Giner‐Sorolla, R. (2016) · doi:10.31234/osf.io/4frms · Cited by 31 · see citing works
Self-citation
- Authors
- Giner-Sorolla, R.
- Affiliations
- University of Kent, United Kingdom
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2012-11
- Volume
- 7
- Issue
- 6
- Pages
- 562-571
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2125515636
- Cited by
- 274
- References
- 70
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 8 · 2023: 21 · 2022: 17 · 2021: 17 · 2020: 40 · 2019: 40 · 2018: 19 · 2017: 19 · 2016: 22 · 2015: 14 · 2014: 19 · 2013: 18 · 2012: 5
- FWCI
- 23.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Kent Academic Repository (University of Kent) (submittedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- publication bias, replication, scientific fraud, false positives, statistical significance thresholds, information economics, open science, research integrity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
