CoChat
Sign Up
Cited by 0Research Data Management PracticesMeta-analysis and systematic reviewsAcademic Publishing and Open Access

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.

Preregistration of Studies with Existing Data

Open access
Mertens, Gaëtan, Krypotos, Angelos-Miltiadis·2022·Integrity of Scientific Research
DOI 10.1007/978-3-030-99680-2_36

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Preregistration of research plans is becoming an increasingly popular and common tool to enhance the transparency of a study’s methodology. In a preregistration, researchers document their research plans and register them to a public repository prior to conducting their research. In this chapter, we provide arguments for why preregistration can protect scientific findings against Questionable Research Practices (QRPs), such as outcome swapping, selective reporting of conditions, unwarranted data exclusions, and post-hoc changing of hypotheses. Furthermore, we place particular emphasis on preregistering research plans when using existing data and we give an overview of preregistration templates and public repositories for different types of research designs. We conclude this chapter with highlighting some of the common criticisms of preregistration and our counter-arguments, and provide future reflections.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/978-3-030-99680-2_36

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. De Groot AD (2014) The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. Acta Psychol 148:188–194 · DOI
  2. Popper K (1959) The logic of scientific discovery. Hutchinson, London
  3. Mogie M (2004) In support of null hypothesis significance testing. Proc R Soc Lond Ser B Biol Sci 7:271. https://royalsocietypublishing.org/doi/10.1098/rsbl.2003.0105 · DOI
  4. Lakens D (2021) The practical alternative to the p value is the correctly used p value. Perspect Psychol Sci 16:639–648. http://journals.sagepub.com/doi/10.1177/1745691620958012 · DOI
  5. Carver R (1978) The case against statistical significance testing. Harv Educ Rev 48(3):378–399. https://meridian.allenpress.com/her/article/48/3/378/21418/The-Case-Against-Statistical-Significance-Testing · DOI · Paper page
  6. McShane BB, Gal D, Gelman A, Robert C, Tackett JL (2019) Abandon statistical significance. Am Stat 73:235–245. https://www.tandfonline.com/doi/full/10.1080/00031305.2018.1527253 · DOI
  7. Wagenmakers E-J, Wetzels R, Borsboom D, van der Maas HLJ (2011) Why psychologists must change the way they analyze their data: the case of psi: Comment on Bem (2011). J Pers Soc Psychol 100(3):426–432. http://doi.apa.org/getdoi.cfm?doi=10.1037/a0022790 · DOI · Paper page
  8. Tendeiro JN, Kiers HAL (2019) A review of issues about null hypothesis Bayesian testing. Psychol Methods 24(6):774–795. http://doi.apa.org/getdoi.cfm?doi=10.1037/met0000221 · DOI · Paper page
  9. John LK, Loewenstein G, Prelec D (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 23(5):524–532. http://journals.sagepub.com/doi/10.1177/0956797611430953 · DOI · Paper page
  10. LoBue V, Reider LB, Kim E, Burris JL, Oleas DS, Buss KA et al (2020) The importance of using multiple outcome measures in infant research. Infancy 25(4):420–437. https://onlinelibrary.wiley.com/doi/abs/10.1111/infa.12339 · DOI
  11. Wagenmakers E-J (2007) A practical solution to the pervasive problems of p values. Psychon Bull Rev 14(5):779–804. http://www.springerlink.com/index/10.3758/BF03194105 · DOI · Paper page
  12. Cumming G (2014) The new statistics. Psychol Sci 25(1):7–29. http://journals.sagepub.com/doi/10.1177/0956797613504966 · DOI · Paper page
  13. Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers E-J, Berk R et al (2018) Redefine statistical significance. Nat Hum Behav 2(1):6–10. http://www.nature.com/articles/s41562-017-0189-z · DOI · Paper page
  14. Lakens D (2014) Performing high-powered studies efficiently with sequential analyses. Eur J Soc Psychol 44(7):701–710. http://doi.wiley.com/10.1002/ejsp.2023 · DOI · Paper page
  15. Schönbrodt FD, Wagenmakers E-J, Zehetleitner M, Perugini M (2017) Sequential hypothesis testing with Bayes factors: efficiently testing mean differences. Psychol Methods 22(2):322–339. http://doi.apa.org/getdoi.cfm?doi=10.1037/met0000061 · DOI
  16. Morís Fernández L, Vadillo MA (2020) Flexibility in reaction time analysis: many roads to a false positive? R Soc Open Sci 7(2):190831. https://royalsocietypublishing.org/doi/10.1098/rsos.190831 · DOI
  17. Kerr NL (1998) HARKing: hypothesizing after the results are known. Personal Soc Psychol Rev 2(3):196–217 · DOI · Paper page
  18. Neuroskeptic (2012) The nine circles of scientific hell. Perspect Psychol Sci 7(6):643–644. http://journals.sagepub.com/doi/10.1177/1745691612459519 · DOI
  19. Simmons JP, Nelson LD, Simonsohn U (2011) False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 22(11):1359–1366. http://journals.sagepub.com/doi/10.1177/0956797611417632 · DOI · Paper page
  20. JPA I (2005) Why most published research findings are false. PLoS Med 2(8):e124. https://dx.plos.org/10.1371/journal.pmed.0020124 · DOI · Paper page
  21. Scheel AM, Schijen M, Lakens D (2021) An excess of positive results: comparing the standard psychology literature with registered reports. Adv Methods Pract Psychol Sci 4:25152459211007467
  22. Pashler H, Wagenmakers E (2012) Editors’ introduction to the special section on replicability in psychological science. Perspect Psychol Sci 7(6):528–530. http://journals.sagepub.com/doi/10.1177/1745691612465253 · DOI · Paper page
  23. Wen H, Wang H-Y, He X, Wu C-I (2018) On the low reproducibility of cancer studies. Natl Sci Rev 5(5):619–624. https://academic.oup.com/nsr/article/5/5/619/4835582 · DOI
  24. Sorkin BC, Kuszak AJ, Williamson JS, Hopp DC, Betz JM (2016) The challenge of reproducibility and accuracy in nutrition research: resources and pitfalls. Adv Nutr 7(2):383–389. https://academic.oup.com/advances/article/7/2/383/4558081 · DOI
  25. Botvinik-Nezer R, Holzmeister F, Camerer CF, Dreber A, Huber J, Johannesson M et al (2020) Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582(7810):84–88. http://www.nature.com/articles/s41586-020-2314-9 · DOI
  26. Wingen T, Berkessel JB, Englich B (2020) No replication, no trust? How low replicability influences trust in psychology. Soc Psychol Personal Sci 11(4):454–463. http://journals.sagepub.com/doi/10.1177/1948550619877412 · DOI
  27. Weston SJ, Ritchie SJ, Rohrer JM, Przybylski AK (2019) Recommendations for increasing the transparency of analysis of preexisting data sets. Adv Methods Pract Psychol Sci 2(3):214–227. http://journals.sagepub.com/doi/10.1177/2515245919848684 · DOI
  28. Mertens G, Krypotos A-M (2019) Preregistration of analyses of preexisting data. Psychol Belg 59(1):338–352. http://www.psychologicabelgica.com/articles/10.5334/pb.493/ · DOI
  29. van den Akker O, Weston SJ, Campbell L, Chopik WJ, Damian RI, Davis-Kean PE et al (2019) Preregistration of secondary data analysis: a template and tutorial. PsyArXiv. https://doi.org/10.31234/osf.io/hvfmr · DOI
  30. Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N et al (2017) A manifesto for reproducible science. Nat Hum Behav 1(1):0021. https://doi.org/10.1038/s41562-016-0021 · DOI · Paper page
  31. Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ et al (2015) Promoting an open research culture. Science 348(6242):1422–1425. http://www.sciencemag.org/cgi/doi/10.1126/science.aab2374 · DOI · Paper page
  32. Nosek BA, Beck ED, Campbell L, Flake JK, Hardwicke TE, Mellor DT et al (2019) Preregistration is hard, and worthwhile. Trends Cogn Sci 23(10):815–818. https://linkinghub.elsevier.com/retrieve/pii/S1364661319301846 · DOI
  33. Krypotos A-M, Klugkist I, Mertens G, Engelhard IM (2019) A step-by-step guide on preregistration and effective data sharing for psychopathology research. J Abnorm Psychol 128(6):517–527 · DOI
  34. van ’t Veer AE, Giner-Sorolla R (2016) Pre-registration in social psychology—a discussion and suggested template. J Exp Soc Psychol 67:2–12. https://doi.org/10.1016/j.jesp.2016.03.004 · DOI
  35. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT (2018) The preregistration revolution. Proc Natl Acad Sci 115(11):2600–2606. http://www.pnas.org/lookup/doi/10.1073/pnas.1708274114 · DOI · Paper page
  36. Lakens D (2019) The value of preregistration for psychological science: a conceptual analysis. Japanese Psychol Rev 62(3):221–230
  37. Polonioli A, Vega-Mendoza M, Blankinship B, Carmel D (2021) Reporting in experimental philosophy: current standards and recommendations for future practice. Rev Philos Psychol 12:49–73. http://link.springer.com/10.1007/s13164-018-0414-3 · DOI
  38. Haven TL, Errington TM, Gleditsch KS, van Grootel L, Jacobs AM, Kern FG et al (2020) Preregistering qualitative research: a Delphi Study. Int J Qual Methods 1(19):160940692097641. http://journals.sagepub.com/doi/10.1177/1609406920976417 · DOI
  39. DeAngelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R et al (2005) Clinical trial registration. Arch Dermatol 141(1):76–77. http://archderm.jamanetwork.com/article.aspx?doi=10.1001/archderm.141.1.76 · DOI
  40. Chambers CD (2013) Registered reports: a new publishing initiative at Cortex. Cortex 49(3):609–610. https://doi.org/10.1016/j.cortex.2012.12.016 · DOI
  41. Pham MT, Oh TT (2021) Preregistration is neither sufficient nor necessary for good science. J Consum Psychol 31(1):163–176. https://onlinelibrary.wiley.com/doi/10.1002/jcpy.1209 · DOI
  42. Claesen A, Gomes S, Tuerlinckx F, Vanpaemel W (2019) Preregistration: comparing dream to reality. PsyArXiv
  43. Bakker M, Veldkamp CLS, van Assen MALM, Crompvoets EAV, Ong HH, Nosek BA et al (2020) Ensuring the quality and specificity of preregistrations. PLoS Biol 18(12):e3000937. https://dx.plos.org/10.1371/journal.pbio.3000937 · DOI · Paper page
  44. Grant DB, Kovács G, Spens K (2018) Questionable research practices in academia: antecedents and consequences. Eur Bus Rev 30(2):101–127. https://www.emerald.com/insight/content/doi/10.1108/EBR-12-2016-0155/full/html · DOI
  45. Szollosi A, Kellen D, Navarro DJ, Shiffrin R, van Rooij I, Van Zandt T et al (2020) Is preregistration worthwhile? Trends Cogn Sci 24(2):94–95. https://linkinghub.elsevier.com/retrieve/pii/S1364661319302852 · DOI
  46. Fanelli D (2010) Do pressures to publish increase scientists’ bias? An empirical support from US States data. PLoS One 5(4):e10271. https://dx.plos.org/10.1371/journal.pone.0010271 · DOI · Paper page

Metadata

Authors
Mertens, G., & Krypotos, A.-M.
Affiliations
  • Tilburg University (NL)
  • Utrecht University (NL)
Corresponding author
Mertens, Gaëtan
ORCID
Venue
Integrity of Scientific Research
Publisher
Springer International Publishing
Published
2022
Pages
361-370
DOI
10.1007/978-3-030-99680-2_36
OpenAlex
W4312396305
Cited by
0
References
10
FWCI
0.00
Type
book chapter
Language
en
Open access
Licence
Locations
  • Springer eBooks (publishedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
  • PsyArXiv (OSF Preprints) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords
preregistration, research transparency, questionable research practices, selective reporting, data exclusion, existing data, public repositories, research methodology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review