CoChat
Sign Up
Cited by 462Academic and Historical Perspectives in PsychologyPsychology Research and BibliometricsMeta-analysis and systematic reviews

Addressing the theory crisis in psychology

Open access
Oberauer, Klaus, Lewandowsky, Stephan·2019·Psychonomic Bulletin & Review
DOI 10.3758/s13423-019-01645-2
Open this paper in CoChat — free accountOpen PDF

Abstract

A worrying number of psychological findings are not replicable. Diagnoses of the causes of this "replication crisis," and recommendations to address it, have nearly exclusively focused on methods of data collection, analysis, and reporting. We argue that a further cause of poor replicability is the often weak logical link between theories and their empirical tests. We propose a distinction between discovery-oriented and theory-testing research. In discovery-oriented research, theories do not strongly imply hypotheses by which they can be tested, but rather define a search space for the discovery of effects that would support them. Failures to find these effects do not question the theory. This endeavor necessarily engenders a high risk of Type I errors-that is, publication of findings that will not replicate. Theory-testing research, by contrast, relies on theories that strongly imply hypotheses, such that disconfirmation of the hypothesis provides evidence against the theory. Theory-testing research engenders a smaller risk of Type I errors. A strong link between theories and hypotheses is best achieved by formalizing theories as computational models. We critically revisit recommendations for addressing the "replication crisis," including the proposal to distinguish exploratory from confirmatory research, and the preregistration of hypotheses and analysis plans.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3758/s13423-019-01645-2

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., . . . Wicherts, J. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108-119. doi:10.1002/per.1919 · DOI
  2. Barnes, E. C. (2008). The paradox of predictivism. Cambridge: Cambrdige University Press.
  3. Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., . . . Johnson, V. E. (2018). Redefine statistical significance. Nature Human Behaviour, 2(1), 6-10. doi:10.1038/s41562-017-0189-z · DOI · Paper page
  4. Bröder, A., & Schütz, J. (2009). Recognition ROCs are curvilinear -or are they? On premature arguments against the two-high-threshold model of recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 587-606.
  5. de Groot, A. D. (1956/2014). The meaning of "significance" for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. Acta Psychologica, 148, 188-194. doi:10.1016/j.actpsy.2014.02.001 · DOI
  6. Farrell, S., & Lewandowsky, S. (2018). Computational modeling of cognition and behavior. Cambridge: Cambridge University Press.
  7. Ferguson, C. J., & Heene, M. (2012). A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science's Aversion to the Null. Perspectives on Psychological Science, 7(6), 555- 561. doi:10.1177/1745691612459059 · DOI · Paper page
  8. Fiedler, K. (2017). What constitutes strong psychological science? The (neglected) role of diagnosticity and a priori theorizing. Perspectives on Cognitive Science, 12, 46-61. doi:10.1177/1745691616654458 · DOI
  9. Fiedler, K., Kutzner, F., & Krueger, J. I. (2012). The long way from α-error control to validity proper: Problems with a short-sighted false-positive debate. Perspectives on Psychological Science, 7, 661-669. doi:10.1177/1745691612462587 · DOI
  10. Freund, A. M., & Isaacowitz, D. M. (2013). Beyond Age Comparisons: A Plea for the Use of a Modified Brunswikian Approach to Experimental Designs in the Study of Adult Development and Aging. Human Development, 56(6), 351-371.
  11. Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: frequency formats. Psychological Review, 102, 684-704.
  12. Gigerenzer, G., Hoffrage, U., & Kleinbölting, H. (1991). Probabilistic mental models: a Brunswikian theory of confidence. Psychological Review, 98, 506-528.
  13. Jostmann, N. B., Lakens, D., & Schubert, T. W. (2009). Weight as an embodiment of importance. Psychological Science, 20(9), 1169-1174. doi:10.1111/j.1467-9280.2009.02426.x · DOI
  14. Kary, A., Taylor, R., & Donkin, C. (2016). Using Bayes factors to test the predictions of models: A case study in visual working memory. Journal of Mathematical Psychology, 72, 210-219. doi:10.1016/j.jmp.2015.07.002 · DOI
  15. Kellen, D., & Klauer, K. C. (2014). Discrete-state and continuous models of recognition memory: testing core properties under minimal assumptions. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40, 1795-1804. doi:10.1037/xlm0000016 · DOI
  16. Kellen, D., & Klauer, K. C. (2015). Signal detection and threshold modeling of confidence-rating ROCs: a critical test with minimal assumptions. Psychological Review, 122, 542-557. doi:10.1037/a0039251 · DOI
  17. Körner, A., Topolinski, S., & Strack, F. (2015). Routes to embodiment. frontiers in Psychology, 6(940). doi:10.3389/fpsyg.2015.00940 · DOI
  18. Ladyman, J. (2002). Understanding philosophy of science. Oxon: Routledge.
  19. Lewandowsky, S., Duncan, M., & Brown, G. D. A. (2004). Time does not cause forgetting in short-term serial recall. Psychonomic Bulletin & Review, 11, 771-790.
  20. Lewandowsky, S., & Oberauer, K. (2015). Rehearsal in serial recall: An unworkable solution to the non-existent problem of decay. Psychological Review, 122, 674-699. doi:10.1037/a0039684 · DOI
  21. MacLeod, C. M. (1991). Half a century of research on the Stroop effect: an integrative review. Psychological Bulletin, 109, 163-203.
  22. Marsman, M., Schönbrodt, F. D., Morey, R. D., Yao, Y., Gelman, A., & Wagenmakers, E.-J. (2017). A Bayesian bird's eye view of "Replications of important results in social psychology". Royal Society Open Science, 4. Retrieved from doi:10.1098/rsos.160426 · DOI
  23. Miller, J. (2009). What is the probability of replicating a statistically significant effect? Psychonomic Bulletin & Review, 16, 617-640. doi:10.3758/PBR.16.4.617 · DOI
  24. Muthukrishna, M., & Henrich, J. (2019). A problem in theory. Nature Human Behavior. doi:10.1038/s41562-018-0522-1 · DOI
  25. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.1708274114 · DOI · Paper page
  26. O'Reilly, R. C., & Munakata, Y. (2000). Computational explorations in cognitive neuroscience: understanding the mind by simulating the brain. Cambridge, MA: MIT Press.
  27. Oberauer, K., & Lewandowsky, S. (2014). Further evidence against decay in working memory. Journal of Memory and Language, 73, 15-30. doi:10.1016/j.jml.2014.02.003 · DOI
  28. Oberauer, K., & Lin, H.-Y. (2017). An interference model of visual working memory. Psychological Review, 124, 21-59.
  29. Palmer, J. (1990). Attentional limits on the perception and memory of visual information. Journal of Experimental Psychology: Human Perception and Performance, 16, 332-350.
  30. Pashler, H., & Harris, C. R. (2012). Is the replication crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531-536. doi:10.1177/1745691612463401 · DOI · Paper page
  31. Pitt, M. A., Myung, I. J., & Zhang, S. (2002). Toward a method of selecting among computational models of cognition. Psychological Review, 109, 472-491.
  32. Rubin, M. (2017a). Do p values lose their meaning in exploratory analyses? It depends how you define the familywise error rate. Review of General Psychology, 21(3), 269-275. doi:10.1037/gpr0000123 · DOI
  33. Rubin, M. (2017b). When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress. Review of General Psychology, 21, 308-320. doi:10.1037/gpr0000128 · DOI
  34. Sederberg, P. B., Howard, M. C., & Kahana, M. J. (2008). A context-based theory of recency and contiguity in free recall. Psychological Review, 115, 893-912.
  35. Sewell, D. K., Lilburn, S. D., & Smith, P. L. (2014). An information capacity limitation of visual short- term memory. Journal of Experimental Psychology: Human Perception and Performance, 40, 2214-2242. doi:10.1037/a0037744 · DOI
  36. Shiffrin, R. M., Lee, M. D., Kim, W., & Wagenmakers, E.-J. (2008). A survey of model evaluation approaches with a tutorial on hierarchical Bayesian methods. Cognitive Science, 32, 1248- 1284. doi:10.1080/03640210802414826 · DOI
  37. Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., . . . Nosek, B. A. (2018). Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. Advances in Methods and Practices in Psychological Science, 1(3), 337-356. doi:10.1177/2515245917747646 · DOI
  38. Smith, P. L., Corbett, E. A., Lilburn, S. D., & Kyllingsbæk, S. (2018). The power law of visual working memory characterizes attention engagement. Psychological Review, 125, 435-451. doi:10.1037/rev0000098 · DOI
  39. Smith, P. L., Lilburn, S. D., Corbett, E. A., Sewell, D. K., & Kyllingsbæk, S. (2016). The attention- weighted sample-size model of visual short-term memory: Attention capture predicts resource allocation and memory load. Cognitive Psychology, 89, 71-105. doi:10.1016/j.cogpsych.2016.07.002 · DOI
  40. Snyder, L. J. (1994). Is evidence historical? In P. Achinstein & L. H. Snyder (Eds.), Scientific methods: Conceptual and historical problems (pp. 95-117): Krieger.
  41. Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11, 702-712. doi:10.1177/1745691616658637 · DOI
  42. Topolinski, S., & Sparenberg, P. (2012). Turning the hands of time: Clockwise movements increase preference for novelty. Social Psychology and Personality Science, 3, 208-214. doi:10.1177/1948550611419266 · DOI
  43. van den Berg, R., & Ma, W. J. (2018). A resource-rational theory of set size effects in human visual working memory. eLIFE, 7, e34963. Retrieved from https://doi.org/10.7554/eLife.34963 doi:10.7554/eLife.34963 · DOI
  44. van den Berg, R., Shin, H., Chou, W.-C., George, R., & Ma, W. J. (2012). Variability in encoding precision accounts for visual short-term memory limitations. Proceedings of the National Academy of Sciences, 109, 8780-8785.
  45. Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779-804.
  46. Wagenmakers, E.-J., Dutilh, G., & Srafoglou, A. (2018). The creativity-verification cycle in psychological science: New methods to combat old idols. Perspectives on Cognitive Science, 13, 418-427. doi:10.1177/1745691618771357 · DOI
  47. Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., . . . Morey, R. D. (2018). Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications. Psychonomic Bulletin & Review, 25(1), 35-57. doi:10.3758/s13423-017-1343-3 · DOI
  48. Williams, L. E., & Bargh, J. A. (2008). Experiencing physical warmth promotes interpersonal warmth. Science, 322, 606-607. doi:10.1126/science.1162548 · DOI · Paper page
  49. Wills, A. J., & Pothos, E. M. (2012). On the adequacy of current empirical evaluations of formal models of categorization. Psychological Bulletin, 138, 102-125. doi:10.1037/a0025715 · DOI
  50. Wixted, J. T. (2007). Dual-process theory and signal-detection theory of recognition memory. Psychological Review, 114, 152-176.

Cited by

Sorted by citations

Authors
Oberauer, K., & Lewandowsky, S.
Affiliations
  • University of Zurich (CH)
  • The University of Western Australia (AU)
Corresponding author
Oberauer, Klaus
ORCID
Venue
Psychonomic Bulletin & Review
Publisher
Springer Science and Business Media LLC
Published
2019-09-12
Volume
26
Issue
5
Pages
1596-1618
ISSN
  • 1069-9384 (ISSN-L)
  • 1531-5320
DOI
10.3758/s13423-019-01645-2
OpenAlex
W2958577954
Cited by
462
References
13
Citations by year
2026: 59 · 2025: 62 · 2024: 79 · 2023: 85 · 2022: 76 · 2021: 48 · 2020: 47 · 2019: 6
FWCI
2.54
Type
journal article
Language
en
Open access
Licence
Locations
  • Psychonomic Bulletin & Review (publishedVersion)
  • PubMed (publishedVersion)
  • Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, other-oa)
  • Bristol Research (University of Bristol) (submittedVersion, other-oa)
  • UWA Profiles and Research Repository (UWA) (submittedVersion)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, other-oa)
  • Zurich Open Repository and Archive (University of Zurich)
Topics
  • Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
replication crisis, replicability, type I error, preregistration, hypothesis testing
MeSH terms
Scientific Experimental Error, Computer Simulation, Data Collection, Humans, Logic, Psychological Theory, Psychology, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review