A Short (Personal) Future History of Revolution 2.0
The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.
Abstract
Crisis of replicability is one term that psychological scientists use for the current introspective phase we are in—I argue instead that we are going through a revolution analogous to a political revolution. Revolution 2.0 is an uprising focused on how we should be doing science now (i.e., in a 2.0 world). The precipitating events of the revolution have already been well-documented: failures to replicate, questionable research practices, fraud, etc. And the fact that none of these events is new to our field has also been well-documented. I suggest four interconnected reasons as to why this time is different: changing technology, changing demographics of researchers, limited resources, and misaligned incentives. I then describe two reasons why the revolution is more likely to catch on this time: technology (as part of the solution) and the fact that these concerns cut across social and life sciences—that is, we are not alone. Neither side in the revolution has behaved well, and each has characterized the other in extreme terms (although, of course, each has had a few extreme actors). Some suggested reforms are already taking hold (e.g., journals asking for more transparency in methods and analysis decisions; journals publishing replications) but the feared tyrannical requirements have, of course, not taken root (e.g., few journals require open data; there is no ban on exploratory analyses). Still, we have not yet made needed advances in the ways in which we accumulate, connect, and extract conclusions from our aggregated research. However, we are now ready to move forward by adopting incremental changes and by acknowledging the multiplicity of goals within psychological science.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Psychological science is currently navigating a period of crisis and opportunity [17,56,57].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [57]
In the citing paper
“Therefore, contemporary science could be more accurately portrayed as facing ' new opportunities and challenges ' or even a ' revolution ' (45).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [45]
Citations
Cites
The Structure of Scientific Revolutions
Kuhn, T. S., & Schlegel, R. (1963) · Physics Today · doi:10.1063/1.3050879 · Cited by 35,538
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“Relatedly, the “file drawer problem” is a well-known major flaw in all meta-analysis (Rosenthal, 1979).”
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
The Structure of Scientific Revolutions
Hall, M. B., & Kuhn, T. S. (1963) · The American Historical Review · doi:10.2307/1847040 · Cited by 7,479
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“…in particular, seemed to be suffering from the “spotlight effect” (i.e., believing that everyone was staring at them for these mishaps), it turns out that problems of nonreplicability had been running rampant across the biological sciences and, most scarily, in medicine for years (Ioannidis, 2005).”
Calamita, E., Schmid, M., Kunz, M., Ndebele-Murisa, M. R., Magadza, C. H. D., Nyambe, I., & Wehrli, B. (2019) · Figshare · doi:10.26181/6153922fe919c · Cited by 4,222
WEIRD languages have misled us, too
Majid, A., & Levinson, S. C. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1000018x · Cited by 4,076
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Journal of the American Statistical Association
Fildes, R. (1989) · International Journal of Forecasting · doi:10.1016/0169-2070(89)90106-4 · Cited by 3,874
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · cited 1×
“…dozens of journals in psychology and other sciences (including Science ) have signed onto the Transparency and Openness guidelines, a framework for developing publication practices that would incrementally increase openness and (hopefully) consequently replicability in science (Nosek et al., 2015).”
Self-citationMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475 · cited 1×
“Neuroscience had been famously attacked for the analysis methods used by many researchers (Vul, Harris, Winkielman, & Pashler, 2009).”
Statistical significance in psychological research.
Lykken, D. T. (1968) · Psychological Bulletin · doi:10.1037/h0026141 · Cited by 1,350
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · cited 1×
“They have been documented and discussed in many places (see the many papers in the November 2012 special section on replicability in Perspectives , Pashler & Wagenmakers, 2012, and the blogs of Ed Yong, e.g., Yong, 2012), so let me just quickly note a few:”
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · cited 4×
“Psychfiledrawer .org (established 2012; see Spellman, 2012a) has shown how unpublishable replications can be made public (even though people were often afraid to post their research).”
InfluentialNegative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · cited 1×
“I believe that the (relatively) early creation of the Open Science Framework (OSF; established 2011) has provided an existence proof that preregistration, open methods, data, and analysis plans can be implemented without much trouble.”
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Theory-Testing in Psychology and Physics: A Methodological Paradox
Meehl, P. E. (1967) · Philosophy of Science · doi:10.1086/288135 · Cited by 962
Bem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · cited 1×
“Technology now is quite different from the extant technology when “the rules of the game called psychological science” (Bakker, van Dijk, & Wicherts, 2012) were developed.”
Replications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734
The fallacy of the null-hypothesis significance test.
Rozeboom, W. W. (1960) · Psychological Bulletin · doi:10.1037/h0042040 · Cited by 729
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697
Perspectives on Psychological Science
Blase, A. E. K. A. S. L. (2011) · APS observer · Cited by 628
Ferguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603 · cited 1×
“This information can help us to find support for new theories, refine our current theories, and devalue or discard less-supported theories (something we are not at all good at; Ferguson & Heene, 2012; Greenwald, 2012).”
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563 · cited 1×
“And we need to be better and faster at self-correction (Ioannidis, 2012).”
The ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455
King, G. (1995) · PS Political Science & Politics · doi:10.1017/s1049096500057607 · Cited by 436
Fraley, R. C., & Vazire, S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109019 · Cited by 409
Yong, E. (2012) · Nature · doi:10.1038/485298a · Cited by 405
Scientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar-Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 368
Continuously Cumulating Meta-Analysis and Replicability
Braver, S. L., Thoemmes, F. J., & Rosenthal, R. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614529796 · Cited by 297
There Is Nothing So Theoretical as a Good Method
Greenwald, A. G. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611434210 · Cited by 281
Effect sizes and p values: What should be reported and what should be replicated?
Greenwald, A., Gonzalez, R., Harris, R. J., & Guthrie, D. (1996) · Psychophysiology · doi:10.1111/j.1469-8986.1996.tb02121.x · Cited by 274
Giner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274 · cited 2×
“But the rewards (credit, tenure, fame) only come if the research is published, and that research is only published if it is novel, hypothesis-confirming, and the data are “pristine” (Giner-Sorolla, 2012).”
Eich, E. (2014) · Psychological Science · doi:10.1177/0956797613512465 · Cited by 255
The Psychology of Replication and Replication in Psychology
Francis, G. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459520 · Cited by 238
Voodoo Correlations Are Everywhere—Not Only in Neuroscience
Fiedler, K. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611400237 · Cited by 235
Fiedler, K., Kutzner, F., & Krueger, J. I. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462587 · Cited by 229
An Introduction to Registered Replication Reports at Perspectives on Psychological Science
Simons, D. J., Holcombe, A. O., & Spellman, B. A. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614543974 · Cited by 184
Self-citationMills, J. L. (1993) · New England Journal of Medicine · doi:10.1056/nejm199310143291613 · Cited by 182
Loftus, G. R. (1993) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03204506 · Cited by 168
Priem, J. (2013) · Nature · doi:10.1038/495437a · Cited by 159
Writing the Empirical Journal Article
Bem, D. J. (2021) · The Compleat Academic · doi:10.4324/9781315808314-10 · Cited by 150
King, G. (1995) · PS Political Science & Politics · doi:10.2307/420301 · Cited by 145
A Longitudinal Study of the New York Times Science Times Section
Clark, F., & Illman, D. L. (2006) · Science Communication · doi:10.1177/1075547006288010 · Cited by 116
Vazire, S. (2015) · Social Psychological and Personality Science · doi:10.1177/1948550615603955 · Cited by 106
The Compleat academic : a practical guide for the beginning social scientist
mark-p-zanna-and-john-m-darley (1986) · Cited by 70
Short, Sweet, and Problematic? The Rise of the Short Report in Psychological Science
Ledgerwood, A., & Sherman, J. W. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611427304 · Cited by 69
(1976) · Journal of Personality and Social Psychology · doi:10.1037/h0078635 · Cited by 69
The effect of context on visual representational momentum
Hubbard, T. L. (1993) · Memory & Cognition · doi:10.3758/bf03211169 · Cited by 67
Crisis discussions in psychology—New historical and philosophical perspectives
Sturm, T., & Mülberger, A. (2011) · Studies in History and Philosophy of Science Part C Studies in History and Philosophy of Biological and Biomedical Sciences · doi:10.1016/j.shpsc.2011.11.001 · Cited by 59
Introduction to the Special Section: Data, Data, Everywhere . . . Especially in My File Drawer
Spellman, B. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611432124 · Cited by 58
Self-citationBones, A. K. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612441216 · Cited by 54
Psychologists Are Open to Change, yet Wary of Rules
Fuchs, H. M., Jenny, M., & Fiedler, S. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459521 · Cited by 50
Past and Future American Psychological Association Guidelines for Statistical Practice
Finch, S., Thomason, N., & Cumming, G. (2002) · Theory & Psychology · doi:10.1177/0959354302126005 · Cited by 40
Galak, J., & Meyvis, T. (2011) · Journal of Experimental Psychology General · doi:10.1037/a0021447 · Cited by 34
Scientific Disintegrity as a Public Bad
Engel, C. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615577865 · Cited by 32
Ranganath, K. A., Spellman, B. A., & Joy-Gaba, J. A. (2010) · Perspectives on Psychological Science · doi:10.1177/1745691610361604 · Cited by 24
Self-citationZanna, M. P., & Darley, J. M. (2021) · Psychology Press eBooks · doi:10.4324/9781315808314 · Cited by 19
Bakker, M., Cramer, A. O. J., Matzke, D., Kievit, R., van der Maas, H. L. J., Wagenmakers, E., & Borsboom, D. (2013) · European Journal of Personality · doi:10.1002/per.1920 · Cited by 16
You Could Have Just Asked: Reply to Francis (2012)
Galak, J., & Meyvis, T. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463079 · Cited by 15
Memos to the President From a “Council of Psychological Science Advisers”
Teachman, B. A., Norton, M. I., & Spellman, B. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615605829 · Cited by 15
Self-citationScientific Utopia … or Too Much Information? Comment on Nosek and Bar-Anan
Spellman, B. A. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.701161 · Cited by 14
Self-citationWhat Lies Within: Superscripting References to Reveal Research Trends
Anicich, E. M. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614549772 · Cited by 11
Replications of Important Results in Social Psychology: Special Issue of Social PsychologyPreprint
Nosek, B. A., & Lakens, D. (2013) · Cited by 8
Course Syllabi for Open and Reproducible MethodsPreprint
Ball, R., Bowman, S. D., Christensen, G., Frank, M. C., Funder, D. C., Hinsen, K., Janz, N., Medeiros, N., Miguel, E. A., & Nosek, B. A. (2015) · OSF Preprints (OSF Preprints) · Cited by 5
Introduction to the Special Section on Methods: Odds and End
Spellman, B. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615582201 · Cited by 4
Self-citationLetter From the Incoming Editor
Spellman, B. A. (2010) · Perspectives on Psychological Science · doi:10.1177/1745691609356780 · Cited by 2
Self-citationStryer, D. B., Browner, W., & Newman, T. (1994) · New England Journal of Medicine · doi:10.1056/nejm199403243301212 · Cited by 1
Research works printed but not linked: 2
- American Psychological Association, 2001, Publication manual of the American Psychological Association
- Nosek B. A., 2014, Social Psychology, 45, 137
Cited by
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Small Effects: The Indispensable Foundation for a Cumulative Psychological Science
Götz, F. M., Gosling, S. D., & Rentfrow, P. J. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620984483 · Cited by 538
“If we are to progress as a science, we must adjust our expectations and align our incentive structures to reward accurate and meaningful effects rather than exaggerated and unreliable effects (De Boeck & Jeon, 2018; Lindsay, 2020; Munafò et al., 2017; Spellman, 2015).”
BackgroundZwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 444
Is science really facing a reproducibility crisis, and do we need it to?
Fanelli, D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708272114 · Cited by 416
Why Hypothesis Testers Should Spend Less Time Testing Hypotheses
Scheel, A. M., Tiokhin, L., Isager, P. M., & Lakens, D. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620966795 · Cited by 382
“Reforms proposed in reaction to the crisis tried to repair the HD machinery by making methods more rigorous (Spellman, 2015).”
BackgroundMethod
Metadata
- Authors
- Spellman, B. A.
- Affiliations
- University of Virginia (US)
- Corresponding author
- Spellman, Barbara A.
- ORCID
- Spellman, Barbara A.
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2015-11
- Volume
- 10
- Issue
- 6
- Pages
- 886-899
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2313516711
- Cited by
- 213
- References
- 81
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 23 · 2023: 25 · 2022: 15 · 2021: 37 · 2020: 20 · 2019: 25 · 2018: 32 · 2017: 15 · 2016: 12 · 2015: 1
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- replication crisis, replication failure, questionable research practices, scientific fraud, scientific revolution, research transparency, open data, exploratory analysis, research incentives
- MeSH terms
- Humans, Psychology, Publishing, Research Design, United States, Access to Information, Behavioral Research, History, 20th Century, History, 21st Century
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.