The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
The case for formal methodology in scientific reform
Open accessAbstract
Current attempts at methodological reform in sciences come in response to an overall lack of rigor in methodological and scientific practices in experimental sciences. However, most methodological reform attempts suffer from similar mistakes and over-generalizations to the ones they aim to address. We argue that this can be attributed in part to lack of formalism and first principles. Considering the costs of allowing false claims to become canonized, we argue for formal statistical rigor and scientific nuance in methodological reform. To attain this rigor and nuance, we propose a five-step formal approach for solving methodological problems. To illustrate the use and benefits of such formalism, we present a formal statistical analysis of three popular claims in the metascientific literature: (i) that reproducibility is the cornerstone of science; (ii) that data must not be used twice in any analysis; and (iii) that exploratory projects imply poor statistical practice. We show how our formal approach can inform and shape debates about such methodological claims.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Representing and Intervening: Introductory Topics in the Philosophy of Natural Science
Hacking, I. (1983) · doi:10.1017/cbo9780511814563 · Cited by 4,814
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Circular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · cited 7×
“What is the statistical problem that preregistration aims to address? Statistically, preregistration is offered as a solution to the problemof using datamore than once and issues of validity of statistical procedures resulting from this usage [13,22,92,93].”
BackgroundMethodInfluentialExploratory Research in the Social Sciences
Stebbins, R. (2001) · doi:10.4135/9781412984249 · Cited by 1,840
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · cited 1×
“In well-cited articles in methodological reform literature, high reproducibility of a result is often interpreted as evidence that the result is true ( Nosek et al., 2012 ; Open Science Collaboration, 2015 ; Pashler and Wagenmakers, 2012 ; Zwaan et al., 2018 ).”
BackgroundNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · cited 1×
“We limit our discussion to statistical reproducibility of results only (similar to results reproducibility in [26]), and exclude other types such as computational or methods reproducibility—whether the materials, methods, procedures, algorithms, analyses used in an original study are reported in a sufficiently detaile…”
ResultWhy We (Usually) Don't Have to Worry About Multiple Comparisons
Gelman, A., Hill, J., & Yajima, M. (2012) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.618213 · Cited by 1,274
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · cited 10×
“What is the statistical problem that preregistration aims to address? Statistically, preregistration is offered as a solution to the problemof using datamore than once and issues of validity of statistical procedures resulting from this usage [13,22,92,93].”
BackgroundMethodInfluentialMeasurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
Flake, J. K., & Fried, E. I. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920952393 · Cited by 969
Inventing Temperature: Measurement and Scientific Progress
Chang, H. (2004) · doi:10.1093/0195171276.001.0001 · Cited by 883
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716
The Value of Direct Replication
Simons, D. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514755 · Cited by 630
Measurement error and the replication crisis
Loken, E., & Gelman, A. (2017) · Science · doi:10.1126/science.aal3618 · Cited by 628
Repeatability of published microarray gene expression analyses
Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly, I., Cui, X., Culhane, A. C., Falchi, M., Furlanello, C., Game, L., Jurman, G., et al. (2009) · Nature Genetics · doi:10.1038/ng.295 · Cited by 551
The past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499
Principles and procedures of exploratory data analysis.
Behrens, J. T. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.2.131 · Cited by 465
Addressing the theory crisis in psychology
Oberauer, K., & Lewandowsky, S. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01645-2 · Cited by 462
Bayesian just-so stories in psychology and neuroscience.
Bowers, J. S., & Davis, C. J. (2012) · Psychological Bulletin · doi:10.1037/a0026450 · Cited by 436
Lack of theory building and testing impedes progress in the factor and network literature
Fried, E. I. (2020) · Psychological Inquiry · doi:10.1080/1047840x.2020.1853461 · Cited by 317
Meng, X.-L. (2018) · The Annals of Applied Statistics · doi:10.1214/18-aoas1161sf · Cited by 316 · cited 1×
“Example: statistical methods assume probabilistic sampling and do not make any claims in a non-probabilistic sampling framework [43].”
Lakens, D., & Evers, E. R. K. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528520 · Cited by 266 · cited 1×
“In an article that makes practical recommendations to improve the methodology of psycho-335 logical science, Lakens and Evers ( 2014 ) argue that “One of the goals of psychological science is to differentiate among all possible truths” and suggest that one way to achieve this goal is to improve the…”
MethodPublication bias and the canonization of false facts
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016) · eLife · doi:10.7554/elife.21451 · Cited by 266
Patil, P., Peng, R. D., & Leek, J. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616646366 · Cited by 243
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021) · Royal Society Open Science · doi:10.1098/rsos.211037 · Cited by 197
Scientific progress despite irreproducibility: A seeming paradox
Shiffrin, R. M., Börner, K., & Stigler, S. M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1711786114 · Cited by 79 · cited 1×
“Indeed, the history of science is full of examples of fields going through arduous sequence of experiments yielding failures such as non-reproducible results to eventually arrive at scientific regularities ( Barwich, 2019 ; Chang, 2004 ; Shiffrin et al., 2018 ).”
Background
Cited by
van Rooij, I., & Baggio, G. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620970604 · Cited by 354
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
Does preregistration improve the credibility of research findings?
Rubin, M. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.4.p376 · Cited by 61
Metadata
- Authors
- Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E.
- Venue
- Royal Society Open Science
- Publisher
- The Royal Society
- Published
- 2021-03
- Volume
- 8
- Issue
- 3
- Pages
- 200805-200805
- ISSN
- 2054-5703 (ISSN-L)
- OpenAlex
- W3148605487
- Cited by
- 188
- References
- 31
- FWCI
- 21.33
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.