CoChat
Sign Up
PreprintCited by 195Meta-analysis and systematic reviewsAcademic Publishing and Open AccessResearch Data Management Practices

The DOI matches its Crossref record.

The case for formal methodology in scientific reform

Open access
Devezer, Berna, Navarro, Danielle J., Vandekerckhove, Joachim, Buzbas, Erkan Ozge·2020·Royal Society Open Science
DOI 10.1101/2020.04.26.048306PMC PMC8101540
Open this paper in CoChat — free account

Abstract

Current attempts at methodological reform in sciences come in response to an overall lack of rigor in methodological and scientific practices in experimental sciences. However, most methodological reform attempts suffer from similar mistakes and over-generalizations to the ones they aim to address. We argue that this can be attributed in part to lack of formalism and first principles. Considering the costs of allowing false claims to become canonized, we argue for formal statistical rigor and scientific nuance in methodological reform. To attain this rigor and nuance, we propose a five-step formal approach for solving methodological problems. To illustrate the use and benefits of such formalism, we present a formal statistical analysis of three popular claims in the metascientific literature: (a) that reproducibility is the cornerstone of science; (b) that data must not be used twice in any analysis; and (c) that exploratory projects imply poor statistical practice. We show how our formal approach can inform and shape debates about such methodological claims.

Overview

Integrity signals

Registry

Crossref · 10.1101/2020.04.26.048306

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Preregistration does not clarify the distinction between confirmatory and exploratory analyses.

    “However, the distinction between confirmatory and exploratory analyses is problematic, and preregistration does not clarify it”

    From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Devezer et al., 2020

  • The distinction between confirmatory and exploratory analyses is problematic, and preregistration does not clarify it.

    “However, the distinction between confirmatory and exploratory analyses is problematic, and preregistration does not clarify it (see also Devezer et al., 2020).”

    From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Devezer et al., 2020

  • None of the previous articles that questioned the added value of preregistration considered the full range of advantages supposedly associated with preregistration.

    “However, none of these articles have distinguished between historical and contemporary transparency, and none have considered the full range of advantages that are supposedly associated with preregistration.”

    From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Devezer et al., 2020

  • None of the previous articles that questioned the added value of preregistration distinguished between historical and contemporary transparency.

    “However, none of these articles have distinguished between historical and contemporary transparency, and none have considered the full range of advantages that are supposedly associated with preregistration.”

    From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Devezer et al., 2020

  • Low replication rates have been attributed to more than just questionable research practices.

    “low replication rates have been attributed to more than just questionable research practices.”

    From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Devezer et al., 2020

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 113

  1. Arabatzis T. Experiment. In: The Routledge Companion to Philosophy of Science Routledge; 2013. p. 223–234.
  2. 10.1073/pnas.1708285114 · DOI
  3. 2019, The Value of Failure in Science: The Story of Grandmother Cells in Neuroscience, Frontiers in neuroscience, 13, 1121, 10.3389/fnins.2019.01121 · DOI
  4. 10.1161/CIRCRESAHA.114.303819 · DOI · Paper page
  5. 10.1037//1082-989X.2.2.131 · DOI · Paper page
  6. 10.2307/2346101 · DOI
  7. 2013, Valid post-selection inference, The Annals of Statistics, 41, 802
  8. 10.2307/1990053 · DOI
  9. Białek M. Replications can cause distorted belief in scienti1c progress. Behavioral and Brain Sciences. 2018; 41. · DOI
  10. 2018, Deep analogical inference as the origin of hypotheses, Journal of Problem Solving, 11, 1
  11. 10.1037/a0026450 · DOI · Paper page
  12. 10.2307/2286841 · DOI
  13. 2019, Need of mathematical formalism in proposals for robust modeling, Computational Brain & Behavior, 2, 197, 10.1007/s42113-019-00057-8 · DOI
  14. Casella G , Berger R. Statistical inference second edition; 2002.
  15. 10.31222/osf.io/43298 · DOI
  16. Chang H. Inventing temperature: Measurement and scienti1c progress. Oxford University Press; 2004. · DOI · Paper page
  17. Chater N , Oaksford M , et al. The probabilistic mind: Prospects for Bayesian cognitive science. OUP Oxford; 2008. · DOI
  18. Claesen A , Gomes SLBT , Tuerlinckx F , Vanpaemel W , Preregistration: Comparing Dream to Reality. PsyArXiv; 2019. https://osf.io/n3axs/download.
  19. 2013, Prediction in M-complete Problems with Limited Sample Size, Bayesian Analysis, 8, 647
  20. Darnieder WF . Bayesian methods for data-dependent priors. PhD thesis, The Ohio State University; 2011.
  21. 2019, Errors in statistical inference under model misspeci1cation: evidence, hypothesis testing, and AIC, Frontiers in Ecology and Evolution, 7, 372, 10.3389/fevo.2019.00372 · DOI
  22. 10.1371/journal.pone.0216125 · DOI
  23. 2008, Reproducible research in computational harmonic analysis, Computing in Science & Engineering, 11, 8
  24. 1953, Sequential decision problems for processes with continuous time parameter. Testing hypotheses, The Annals of Mathematical Statistics, 24, 254, 10.1214/aoms/1177729031 · DOI
  25. 1942, On the Liapunov limit error in the theory of probability, Ark Mat Astr Fys, 28, 1
  26. 10.31222/osf.io/5vnhj · DOI
  27. 10.31234/osf.io/hs7wm · DOI
  28. 2019, Psychologists psychologizing scienti1c psychology: An epistemological reading of the replication crisis, Theory & Psychology, 29, 158, 10.1177/0959354319835322 · DOI
  29. 10.31234/osf.io/zg84s · DOI
  30. 2003, A Bayesian formulation of exploratory data analysis and goodness-of-1t testing, International Statistical Review, 71, 369, 10.1111/j.1751-5823.2003.tb00203.x · DOI
  31. 2004, Exploratory data analysis for complex models, Journal of Computational and Graphical Statistics, 13, 755, 10.1198/106186004X11435 · DOI
  32. 2012, Why we (usually) don’t have to worry about multiple comparisons, Journal of Research on Educational Effectiveness, 5, 189, 10.1080/19345747.2011.618213 · DOI · Paper page
  33. 10.31234/osf.io/jcs6e · DOI
  34. 10.1214/11-STS356 · DOI
  35. 10.1086/289110 · DOI
  36. 10.1126/scitranslmed.aaf5027 · DOI · Paper page
  37. 10.1097/GIM.0b013e3180676d79 · DOI
  38. 10.31234/osf.io/rybh9 · DOI
  39. Hacking I. The Self-Vindication of the Laboratory Sciences. In: Pickering A , editor. Science as Practice and Culture University of Chicago Press; 1992.p. 29–64.
  40. Hacking I , Hacking J , et al. Representing and intervening: Introductory topics in the philosophy of natural science. Cambridge university press; 1983. · DOI · Paper page
  41. 10.31234/osf.io/v784s · DOI
  42. 10.1207/s15327906mbr4003_2 · DOI
  43. Haig BD . Inference to the best explanation: A neglected approach to theory appraisal in psychology. The American journal of psychology. 2009; p. 219–234. · DOI
  44. Hayes BK , Banner S , Forrester S , Navarro DJ . Selective sampling and inductive inference: Draw-ing inferences based on observed and missing evidence. Cognitive Psychology. 2019; 113. doi: https://doi.org/10.1016/j.cogpsych.2019.05.003. · DOI
  45. 2019, Statistical analyses for studying replication: Meta-analytic perspectives, Psychological Methods, 24, 557, 10.1037/met0000189 · DOI
  46. 2018, Why the reward structure of science makes reproducibility problems inevitable, The Journal of Philosophy, 115, 661, 10.5840/jphil20181151239 · DOI
  47. 10.1017/S0140525X0999152X · DOI · Paper page
  48. 2018, Meta-research: Why research on research matters, PLoS biology, 16, e2005468, 10.1371/journal.pbio.2005468 · DOI
  49. 10.1038/ng.295 · DOI · Paper page
  50. 10.1017/S0140525X10003134 · DOI · Paper page
  51. 2018, A practical guide for transparency in psychological science, Collabra: Psychology, 4, 1
  52. 10.1038/nn.2303 · DOI · Paper page
  53. 2018, Justify your alpha, Nature Human Behaviour, 2, 168, 10.1038/s41562-018-0311-x · DOI
  54. 10.1177/1745691614528520 · DOI · Paper page
  55. 2019, An overview of scienti1c reproducibility: Consideration of relevant issues for behavior science/analysis, Perspectives on Behavior Science, 42, 33, 10.1007/s40614-019-00193-3 · DOI
  56. 10.1007/s42113-019-00029-y · DOI
  57. 2017, Is Most Published Research Really False?, Annual Review of Statistics and Its Application, 4, 109, 10.1146/annurev-statistics-060116-054104 · DOI
  58. Lehmann EL , Casella G. Theory of point estimation, 2nd Ed. Springer-Verlag: New York; 1998.
  59. Leonelli S. Rethinking Reproducibility as a Criterion for Research Quality. In: Including a Symposium on Mary Morgan: Curiosity, Imagination, and Surprise Emerald Publishing Limited; 2018. p. 129–146. · DOI
  60. 10.1111/1467-9884.00238 · DOI
  61. Lindley DV . Understanding Uncertainty. Wiley; 2006. · DOI
  62. Loftus JR , Taylor JE . A signi1cance test for forward stepwise model selection. arXiv preprint 14053920. 2014;.
  63. 10.1126/science.aal3618 · DOI · Paper page
  64. 2019, Preregistration of modeling exercises may not be useful, Computational Brain & Behavior, 2, 179, 10.1007/s42113-019-00038-x · DOI
  65. 10.2307/2283149 · DOI
  66. 10.1080/19312450701641409 · DOI
  67. 2017, Exploratory reports: A new article type for Cortex, Cortex, 96, A1, 10.1016/j.cortex.2017.07.014 · DOI
  68. McNutt M , Journals unite for reproducibility. American Association for the Advancement of Science; 2014. · DOI
  69. 10.1080/00031305.2018.1527253 · DOI
  70. 2018, Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election, The Annals of Applied Statistics, 12, 685
  71. 2006, MVUE for the mean with one observation: Normal with same mean and variance, The American Statistician, 60, 71, 10.1198/000313006X90774 · DOI
  72. 2017, A manifesto for reproducible science, Nature Human Behaviour, 1, 0021, 10.1038/s41562-016-0021 · DOI · Paper page
  73. Muthukrishna M , Henrich J. A problem in theory. Nature Human Behaviour. 2019; p. 1. · DOI
  74. 2019, Between the devil and the deep blue sea: Tensions between scienti1c judgement and statistical model selection, Computational Brain & Behavior, 2, 28, 10.1007/s42113-018-0019-z · DOI
  75. 10.7554/eLife.21451 · DOI · Paper page
  76. Nosek BA , Ebersole CR , DeHaven AC , Mellor DT . The preregistration revolution. Proceedings of the National Academy of Sciences. 2018; p. 201708274. · DOI
  77. 10.1027/1864-9335/a000192 · DOI · Paper page
  78. 10.1177/1745691612459058 · DOI
  79. Oaksford M , Chater N. Rational models of cognition. Oxford University Press Oxford; 1998.
  80. 2019, Addressing the theory crisis in psychology, Psychonomic bulletin & review, 26, 1596, 10.3758/s13423-019-01645-2 · DOI · Paper page
  81. 10.1177/1745691612462588 · DOI
  82. 10.1126/science.aac4716 · DOI · Paper page
  83. 10.1177/1745691612465253 · DOI · Paper page
  84. 10.1177/1745691616646366 · DOI · Paper page
  85. 10.31234/osf.io/4tgve · DOI
  86. 2019, A Statistical Model to Investigate the Reproducibility Rate Based on Replication Experiments, International Statistical Review, 87, 68, 10.1111/insr.12273 · DOI
  87. Peirce CS , The Collected Papers of Charles S. Peirce. Eds. C. Hartshorne , P. Weiss , and A.W. Burks . Cambridge, MA: Harvard University Press; 1974.
  88. 2019, Rinse and repeat: Understanding the value of replication across different ways of knowing. Publications, 7, 52
  89. 10.31235/osf.io/4dsqa · DOI
  90. 2017, Theory and methodology of exploratory social science research, International Journal of Science and Research Methodology, 5, 129
  91. 10.31234/osf.io/7qbpr · DOI
  92. 10.31234/osf.io/rnw9q · DOI
  93. 10.1027/1864-9335/a000428 · DOI
  94. 2017, Do p values lose their meaning in exploratory analyses? It depends how you de1ne the familywise error rate, Review of General Psychology, 21, 269, 10.1037/gpr0000123 · DOI
  95. 2017, An evaluation of four solutions to the forking paths problem: Adjusted alpha, preregistration, sensitivity analyses, and abandoning the Neyman-Pearson approach, Review of General Psychology, 21, 321, 10.1037/gpr0000135 · DOI
  96. Russell B. A History of Western philosophy: Collectors edition. NY: Simon and Schuster; 1945. Fourth Printing.
  97. 2014, Metascience could rescue the ‘replication crisis’, Nature, 515, 9, 10.1038/515009a · DOI
  98. 10.1073/pnas.1711786114 · DOI · Paper page
  99. 10.1177/0956797611417632 · DOI · Paper page
  100. 10.1177/1745691613514755 · DOI · Paper page
  101. Srivastava S. Veri1ability is a core principle of science. Behavioral and Brain Sciences. 2018; 41. · DOI
  102. Stebbins RA . Exploratory research in the social sciences, vol. 48. Sage; 2001. · DOI · Paper page
  103. 10.1177/1745691616658637 · DOI
  104. Strack F , Stroebe W. What have we learned? What can we learn? Behavioral and Brain Sciences. 2018; 41. · DOI
  105. 10.1177/1745691613514450 · DOI · Paper page
  106. Szollosi A , Donkin C , Arrested theory development: The misguided distinction between exploratory and con1rmatory research. PsyArXiv; 2019. https://psyarxiv.com/suzej/. · DOI
  107. Szollosi A , Kellen D , Navarro DJ , Shiffrin R , van Rooij I , Van Zandt T , Donkin C. Is preregistration worthwhile? Trends in Cognitive Sciences. 2019;. · DOI
  108. 10.1037/rev0000052 · DOI
  109. 10.2307/2682991 · DOI
  110. 2016, Pre-registration in social psychology—A discussion and suggested template, Journal of Experimental Social Psychology, 67, 2, 10.1016/j.jesp.2016.03.004 · DOI
  111. 10.1177/1745691612463078 · DOI · Paper page
  112. 2020, Bayesian hypothesis testing for Gaussian graphical models: Conditional independence and order constraints, Journal of Mathematical Psychology, 99, 102441, 10.1016/j.jmp.2020.102441 · DOI
  113. Zwaan RA , Etz A , Lucas RE , Donnellan MB . Making replication mainstream. Behavioral and Brain Sciences. 2018; 41. · DOI

Cited by

Sorted by citations

Metadata

Authors
Devezer, B., Navarro, D. J., Vandekerckhove, J., & Buzbas, E. O.
Affiliations
  • University of Idaho (US)
  • UNSW Sydney (AU)
  • University of California, Irvine (US)
Corresponding author
Devezer, Berna
ORCID
Venue
Royal Society Open Science
Publisher
openRxiv
Published
2020-04-28
Volume
8
Issue
3
Pages
200805-200805
ISSN
2054-5703 (ISSN-L)
DOI
10.1101/2020.04.26.048306
PMC
PMC8101540
OpenAlex
W3148605487
Cited by
195
References
0
Citations by year
2026: 9 · 2025: 12 · 2024: 64 · 2023: 37 · 2022: 28 · 2021: 28 · 2020: 17
FWCI
22.75
Type
journal article
Language
en
Open access
Licence
Locations
  • Royal Society Open Science (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • eScholarship (California Digital Library) (submittedVersion)
Funders
  • National Science Foundation (P20GM104420, 1850849, 1658303)
  • National Institutes of Health
  • National Institute of General Medical Sciences (P20GM104420)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
Keywords
methodological reform, first principles, reproducibility, data reuse, exploratory analysis, meta-research, scientific claims

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review