What’s in a Badge? A Computational Reproducibility Investigation of the Open Data Badge Policy in One Issue of Psychological Science
Open accessPublished version of a preprint
The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
In April 2019, Psychological Science published its first issue in which all Research Articles received the Open Data badge. We used that issue to investigate the effectiveness of this badge, focusing on the adherence to its aim at Psychological Science : sharing both data and code to ensure reproducibility of results. Twelve researchers of varying experience levels attempted to reproduce the results of the empirical articles in the target issue (at least three researchers per article). We found that all 14 articles provided at least some data and six provided analysis code, but only one article was rated to be exactly reproducible, and three were rated as essentially reproducible with minor deviations. We suggest that researchers should be encouraged to adhere to the higher standard in force at Psychological Science . Moreover, a check of reproducibility during peer review may be preferable to the disclosure method of awarding badges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Whether shared data are findable, accessible, interoperable, and reusable (FAIR) is yet another question, and some evidence, at least in the field of psychology, suggests that this is often not the case [49,50].”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [50]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Other researchers then accessed those open datasets and tried to reproduce the results reported in each paper, but had a low rate of success [50].”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423 · cited 1×
“Authors can also ensure machine-actionable reusability of their data by following the findable, accessible, interoperable, and reusable (FAIR) guidelines (Wilkinson et al., 2016).”
MethodReproducibility and Replicability in Science
Committee on Reproducibility and Replicability in Science, Board on Behavioral, C., Committee on National Statistics, Education, D. O. B. A. S. S. A., Nuclear and Radiation Studies Board, Studies, D. O. E. A. L., Board on Mathematical Sciences and Analytics, Committee on Applied and Theoretical Statistics, Sciences, D. O. E. A. P., Board on Research Data and Information, et al. (2019) · doi:10.17226/25303 · Cited by 924 · cited 1×
“…for successful reproduction in advance, for a study to count as reproducible, its results should at least be reproducible by a competent external researcher (National Academies of Sciences, Engineering, and Medicine, 2019), such as a PhD student with some experience and training in a similar field.”
Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., et al. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002456 · Cited by 700 · cited 1×
“Kidwell et al. (2016) found that introducing badges at Psychological Science led to an increase in sharing, which indicates the superficial success of this policy— particularly compared with other initiatives (see Rowhani-Farid & Barnett, 2018, and Rowhani-Farid et al., 2020, who found lower and no…”
ResultData reuse and the open data citation advantage
Piwowar, H. A., & Vision, T. J. (2013) · PeerJ · doi:10.7717/peerj.175 · Cited by 603
Enhancing reproducibility for computational methods
Stodden, V., McNutt, M., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 411
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369 · cited 1×
“Stodden et al. (2018) assessed the effectiveness of a policy of mandatory sharing on request at the journal Science and found that, despite this policy, they received data for only 44% of articles.”
BackgroundThe citation advantage of linking publications to research data
Colavizza, G., Hrynaszkiewicz, I., Staden, I., Whitaker, K., & McGillivray, B. (2020) · PLOS ONE · doi:10.1371/journal.pone.0230416 · Cited by 335
Data Sharing in Psychology: A Survey on Barriers and Preconditions
Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T. E., & Wagenmakers, E.-J. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917751886 · Cited by 283 · cited 1×
“Moreover, the provision of further incentives, in particular by funding agencies and institutions, may help make data sharing more common and effective (Houtkoop et al., 2018).”
BackgroundEich, E. (2014) · Psychological Science · doi:10.1177/0956797613512465 · Cited by 255 · cited 1×
“the badges in 2014 (Eich, 2014), and, in April 2019, published its first issue in which all 14 Research Articles received Open Data badges (Volume 30, Issue 4).”
BackgroundGabelica, M., Bojčić, R., & Puljak, L. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.05.019 · Cited by 249 · cited 1×
“5 Regardless, whether authors fill in their disclosure items appropriately should continue to be monitored—a recent study found low adherence even to mandatory data availability statements in biomedical research manuscripts (Gabelica et al., 2022).”
BackgroundA Practical Guide for Transparency in Psychological Science
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · Collabra: Psychology · doi:10.1525/collabra.158 · Cited by 233 · cited 1×
“…in-depth recommendations and tutorials for authors to ensure that their shared data and code are eligible for an Open Data badge are provided by, for example, Arslan (2019), Eberle (2022), Klein et al. (2018), Levenstein and Lyle (2018), Peikert and Brandmaier (2021), and Van Lissa et al. (2021).”
BackgroundCollective Emotions and Social Resilience in the Digital Traces After a Terrorist Attack
Garcia, D., & Rimé, B. (2019) · Psychological Science · doi:10.1177/0956797619831964 · Cited by 211 · cited 1×
“…Science , the journal’s first issue in which all Research Articles were awarded the Open Data badge (Bae & Luck, 2019; Dorfman et al., 2019; Garcia & Rimé, 2019; Geniole et al., 2019; Hakim et al., 2019; Hilgard et al., 2019; Johnson & Wilson, 2019; Lindsay et al., 2019; Obaidi et al.,…”
BackgroundThe Prevalence of Marginally Significant Results in Psychology Over Time
Olsson-Collentine, A., van Assen, M. A. L. M., & Hartgerink, C. H. J. (2019) · Psychological Science · doi:10.1177/0956797619830326 · Cited by 178 · cited 1×
“…Dorfman et al., 2019; Garcia & Rimé, 2019; Geniole et al., 2019; Hakim et al., 2019; Hilgard et al., 2019; Johnson & Wilson, 2019; Lindsay et al., 2019; Obaidi et al., 2019; Olsson-Collentine et al., 2019; Vardy & Atkinson, 2019; Wójcik et al., 2019; Woolley & Fishbach, 2019; Yousif & Keil, 2019).”
Ten simple rules for writing Dockerfiles for reproducible data science
Nüst, D., Sochat, V., Marwick, B., Eglen, S. J., Head, T., Hirst, T., & Evans, B. D. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008316 · Cited by 142
Reactivation of Previous Experiences in a Working Memory Task
Bae, G., & Luck, S. J. (2019) · Psychological Science · doi:10.1177/0956797619830398 · Cited by 138 · cited 1×
“…in the April 2019 issue of Psychological Science , the journal’s first issue in which all Research Articles were awarded the Open Data badge (Bae & Luck, 2019; Dorfman et al., 2019; Garcia & Rimé, 2019; Geniole et al., 2019; Hakim et al., 2019; Hilgard et al., 2019; Johnson & Wilson, 2019;…”
BackgroundA large-scale study on research code quality and execution
Trisovic, A., Lau, M. K., Pasquier, T., & Crosas, M. (2022) · Scientific Data · doi:10.1038/s41597-022-01143-6 · Cited by 135 · cited 1×
“…recommendations for improving the specific badge policy at Psychological Science and comparable initiatives at other journals (for further general recommendations on improving data sharing and computational reproducibility, see Stodden et al., 2016; Trisovic et al., 2022; Wilson et al., 2017).”
BackgroundAnalysis of Open Data and Computational Reproducibility in Registered Reports in Psychology
Obels, P., Lakens, D., Coles, N. A., Gottfried, J., & Green, S. A. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920918872 · Cited by 127 · cited 2×
“Whereas Hardwicke et al. (2021) and Obels et al. (2020) were concerned with computational or analytic reproducibility per se, we focused on computational reproducibility as a measure of the effectiveness of the Psychological Science Open Data badge policy.”
BackgroundHakim, N., Adam, K. C. S., Günseli, E., Awh, E., & Vogel, E. K. (2019) · Psychological Science · doi:10.1177/0956797619830384 · Cited by 124 · cited 1×
“…in which all Research Articles were awarded the Open Data badge (Bae & Luck, 2019; Dorfman et al., 2019; Garcia & Rimé, 2019; Geniole et al., 2019; Hakim et al., 2019; Hilgard et al., 2019; Johnson & Wilson, 2019; Lindsay et al., 2019; Obaidi et al., 2019; Olsson-Collentine et al., 2019; Vardy &…”
BackgroundGroup-Based Relative Deprivation Explains Endorsement of Extremism Among Western-Born Muslims
Obaidi, M., Bergh, R., Akrami, N., & Anjum, G. (2019) · Psychological Science · doi:10.1177/0956797619834879 · Cited by 114 · cited 1×
“…Dorfman et al., 2019; Garcia & Rimé, 2019; Geniole et al., 2019; Hakim et al., 2019; Hilgard et al., 2019; Johnson & Wilson, 2019; Lindsay et al., 2019; Obaidi et al., 2019; Olsson-Collentine et al., 2019; Vardy & Atkinson, 2019; Wójcik et al., 2019; Woolley & Fishbach, 2019; Yousif & Keil, 2019).”
Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021) · Royal Society Open Science · doi:10.1098/rsos.201494 · Cited by 108 · cited 5×
“In comparison, Hardwicke et al. (2021) focused only on the numerical results of a subset of substantive findings for each article, meaning that reproducibility was not as fully evaluated as in our study.”
BackgroundResultInfluentialDid awarding badges increase data sharing in BMJ Open? A randomized controlled trial
Rowhani-Farid, A., Aldcroft, A., & Barnett, A. G. (2020) · Royal Society Open Science · doi:10.1098/rsos.191818 · Cited by 88 · cited 1×
“…led to an increase in sharing, which indicates the superficial success of this policy— particularly compared with other initiatives (see Rowhani-Farid & Barnett, 2018, and Rowhani-Farid et al., 2020, who found lower and no increase in data sharing at Biostatistics and BMJ Open , respectively).”
ResultGeniole, S. N., Procyshyn, T. L., Marley, N., Ortiz, T. L., Bird, B. M., Marcellus, A. L., Welker, K. M., Bonin, P. L., Goldfarb, B., Watson, N. V., et al. (2019) · Psychological Science · doi:10.1177/0956797619826970 · Cited by 84
WORCS: A workflow for open reproducible code in science
Lissa, C. J. V., Brandmaier, A. M., Brinkman, L., Lamprecht, A.-L., Peikert, A., Struiksma, M. E., & Vreede, B. M. I. (2021) · Data Science · doi:10.3233/ds-210031 · Cited by 71
Vardy, T., & Atkinson, Q. D. (2019) · Psychological Science · doi:10.1177/0956797619826972 · Cited by 71 · cited 1×
“…Dorfman et al., 2019; Garcia & Rimé, 2019; Geniole et al., 2019; Hakim et al., 2019; Hilgard et al., 2019; Johnson & Wilson, 2019; Lindsay et al., 2019; Obaidi et al., 2019; Olsson-Collentine et al., 2019; Vardy & Atkinson, 2019; Wójcik et al., 2019; Woolley & Fishbach, 2019; Yousif & Keil, 2019).”
Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science
Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0114876 · Cited by 69 · cited 1×
“Alternatively, authors could provide proof of a successful reproduction attempt, either independently or from within the research team (which would be an improvement, as analyses are commonly carried out by single team members; Veldkamp et al., 2014).”
BackgroundComputational Reproducibility via Containers in Psychology
Clyburne-Sherin, A., Fei, X., & Green, S. A. (2019) · Meta-Psychology · doi:10.15626/mp.2018.892 · Cited by 69
How to Automatically Document Data With the codebook Package to Facilitate Data Reuse
Arslan, R. C. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919838783 · Cited by 67 · cited 1×
“…in-depth recommendations and tutorials for authors to ensure that their shared data and code are eligible for an Open Data badge are provided by, for example, Arslan (2019), Eberle (2022), Klein et al. (2018), Levenstein and Lyle (2018), Peikert and Brandmaier (2021), and Van Lissa et al. (2021).”
BackgroundShared Plates, Shared Minds: Consuming From a Shared Plate Promotes Cooperation
Woolley, K., & Fishbach, A. (2019) · Psychological Science · doi:10.1177/0956797619830633 · Cited by 66 · cited 1×
“…Dorfman et al., 2019; Garcia & Rimé, 2019; Geniole et al., 2019; Hakim et al., 2019; Hilgard et al., 2019; Johnson & Wilson, 2019; Lindsay et al., 2019; Obaidi et al., 2019; Olsson-Collentine et al., 2019; Vardy & Atkinson, 2019; Wójcik et al., 2019; Woolley & Fishbach, 2019; Yousif & Keil, 2019).”
Null Effects of Game Violence, Game Difficulty, and 2D:4D Digit Ratio on Aggressive Behavior
Hilgard, J., Engelhardt, C. R., Rouder, J. N., SEGERT, I., & Bartholow, B. D. (2019) · Psychological Science · doi:10.1177/0956797619829688 · Cited by 65 · cited 1×
“…Articles were awarded the Open Data badge (Bae & Luck, 2019; Dorfman et al., 2019; Garcia & Rimé, 2019; Geniole et al., 2019; Hakim et al., 2019; Hilgard et al., 2019; Johnson & Wilson, 2019; Lindsay et al., 2019; Obaidi et al., 2019; Olsson-Collentine et al., 2019; Vardy & Atkinson, 2019;…”
BackgroundCausal Inference About Good and Bad Outcomes
Dorfman, H. M., Bhui, R., Hughes, B. L., & Gershman, S. J. (2019) · Psychological Science · doi:10.1177/0956797619828724 · Cited by 65 · cited 1×
“…2019 issue of Psychological Science , the journal’s first issue in which all Research Articles were awarded the Open Data badge (Bae & Luck, 2019; Dorfman et al., 2019; Garcia & Rimé, 2019; Geniole et al., 2019; Hakim et al., 2019; Hilgard et al., 2019; Johnson & Wilson, 2019; Lindsay et al.,…”
BackgroundUnconscious Detection of One’s Own Image
Wójcik, M. J., Nowicka, M. M., Bola, M., & Nowicka, A. (2019) · Psychological Science · doi:10.1177/0956797618822971 · Cited by 56 · cited 1×
“…Dorfman et al., 2019; Garcia & Rimé, 2019; Geniole et al., 2019; Hakim et al., 2019; Hilgard et al., 2019; Johnson & Wilson, 2019; Lindsay et al., 2019; Obaidi et al., 2019; Olsson-Collentine et al., 2019; Vardy & Atkinson, 2019; Wójcik et al., 2019; Woolley & Fishbach, 2019; Yousif & Keil, 2019).”
A Reproducible Data Analysis Workflow
Peikert, A., & Brandmaier, A. M. (2021) · Quantitative and Computational Methods in Behavioral Sciences · doi:10.5964/qcmb.3763 · Cited by 56 · cited 1×
“…in-depth recommendations and tutorials for authors to ensure that their shared data and code are eligible for an Open Data badge are provided by, for example, Arslan (2019), Eberle (2022), Klein et al. (2018), Levenstein and Lyle (2018), Peikert and Brandmaier (2021), and Van Lissa et al. (2021).”
BackgroundLevenstein, M. C., & Lyle, J. A. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918758319 · Cited by 51 · cited 1×
“…in-depth recommendations and tutorials for authors to ensure that their shared data and code are eligible for an Open Data badge are provided by, for example, Arslan (2019), Eberle (2022), Klein et al. (2018), Levenstein and Lyle (2018), Peikert and Brandmaier (2021), and Van Lissa et al. (2021).”
BackgroundDealing With Software Collapse
Hinsen, K. (2019) · Computing in Science & Engineering · doi:10.1109/mcse.2019.2900945 · Cited by 43 · cited 1×
“However, the reproducibility of the articles that shared data and analysis code likely decreased since publication (because of issues such as “software rot”; Hinsen, 2019).”
BackgroundThe Additive-Area Heuristic: An Efficient but Illusory Means of Visual Area Approximation
Yousif, S. R., & Keil, F. C. (2019) · Psychological Science · doi:10.1177/0956797619831617 · Cited by 39 · cited 1×
“…Dorfman et al., 2019; Garcia & Rimé, 2019; Geniole et al., 2019; Hakim et al., 2019; Hilgard et al., 2019; Johnson & Wilson, 2019; Lindsay et al., 2019; Obaidi et al., 2019; Olsson-Collentine et al., 2019; Vardy & Atkinson, 2019; Wójcik et al., 2019; Woolley & Fishbach, 2019; Yousif & Keil, 2019).”
A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses
Seibold, H., Czerny, S., Decke, S., Dieterle, R., Eder, T., Fohr, S., Hahn, N., Hartmann, R., Heindl, C., Kopper, P., et al. (2021) · PLOS ONE · doi:10.1371/journal.pone.0251194 · Cited by 37 · cited 1×
“This particularly applies to complex analyses, as verbal descriptions are unlikely to cover the information necessary for exact or essential reproduction (as demonstrated by our difficulties reproducing Article 112; see also Seibold et al., 2021).”
BackgroundLindsay, L., Gambi, C., & Rabagliati, H. (2019) · Psychological Science · doi:10.1177/0956797618822802 · Cited by 36 · cited 2×
“…& Luck, 2019; Dorfman et al., 2019; Garcia & Rimé, 2019; Geniole et al., 2019; Hakim et al., 2019; Hilgard et al., 2019; Johnson & Wilson, 2019; Lindsay et al., 2019; Obaidi et al., 2019; Olsson-Collentine et al., 2019; Vardy & Atkinson, 2019; Wójcik et al., 2019; Woolley & Fishbach, 2019;…”
Badges for sharing data and code at Biostatistics: an observational study
Rowhani-Farid, A., & Barnett, A. G. (2018) · F1000Research · doi:10.12688/f1000research.13477.1 · Cited by 31 · cited 1×
“…Science led to an increase in sharing, which indicates the superficial success of this policy— particularly compared with other initiatives (see Rowhani-Farid & Barnett, 2018, and Rowhani-Farid et al., 2020, who found lower and no increase in data sharing at Biostatistics and BMJ Open ,…”
ResultRacial Bias in Perceptions of Size and Strength: The Impact of Stereotypes and Group Differences
Johnson, D. J., & Wilson, J. P. (2019) · Psychological Science · doi:10.1177/0956797619827529 · Cited by 27 · cited 1×
“…the Open Data badge (Bae & Luck, 2019; Dorfman et al., 2019; Garcia & Rimé, 2019; Geniole et al., 2019; Hakim et al., 2019; Hilgard et al., 2019; Johnson & Wilson, 2019; Lindsay et al., 2019; Obaidi et al., 2019; Olsson-Collentine et al., 2019; Vardy & Atkinson, 2019; Wójcik et al., 2019;…”
BackgroundBadges for sharing data and code at Biostatistics: an observational study
Rowhani-Farid, A., & Barnett, A. G. (2018) · F1000Research · doi:10.12688/f1000research.13477.2 · Cited by 26
Psychological Science Stepping Up a Level
Bauer, P. J. (2022) · Psychological Science · doi:10.1177/09567976221078527 · Cited by 18 · cited 2×
“Badge-awarding journals requiring not only data but also code could more explicitly require authors to provide working code— where necessary—that enables straightforward reproducibility and produces clearly annotated output (see Bauer, 2022, for a reaffirmation of this requirement).”
BackgroundGood Enough Practices in Scientific Computing
Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., & Teal, T. K. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005510 · Cited by 8 · cited 1×
“…recommendations for improving the specific badge policy at Psychological Science and comparable initiatives at other journals (for further general recommendations on improving data sharing and computational reproducibility, see Stodden et al., 2016; Trisovic et al., 2022; Wilson et al., 2017).”
BackgroundImproving the Computational Reproducibility of Clinical Science: Tools for Open Data and CodePreprint
Eberle, J. W. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/bf28t · Cited by 6
Cited by
The dire disregard of measurement invariance testing in psychological science.
Maassen, E., D'Urso, E. D., van Assen, M. A. L. M., Nuijten, M. B., Roover, K. D., & Wicherts, J. M. (2025) · Psychological Methods · doi:10.1037/met0000624 · Cited by 152
Open Science 2.0: Towards a truly collaborative research ecosystem
Thibault, R. T., Amaral, O. B., Argolo, F., Bandrowski, A. E., Davidson, A. R., & Drude, N. I. (2023) · PLOS Biology · doi:10.1371/journal.pbio.3002362 · Cited by 72
Computational reproducibility of Jupyter notebooks from biomedical publications
Samuel, S., & Mietchen, D. (2024) · GigaScience · doi:10.1093/gigascience/giad113 · Cited by 93
Rethlefsen, M. L., Brigham, T., Price, C. L., Moher, D., Bouter, L. M., Kirkham, J. J., Schroter, S., & Zeegers, M. P. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.111229 · Cited by 77
Transparency Is Now the Default at Psychological Science
Hardwicke, T. E., & Vazire, S. (2023) · Psychological Science · doi:10.1177/09567976231221573 · Cited by 57
“Most pertinently, three studies have reported difficulties reproducing the results of articles published in Psychological Science (Crüwell et al., 2023; Hardwicke et al., 2021; Homewood, 2023).”
Metadata
- Authors
- Crüwell, S., Apthorp, D., Baker, B. J., Colling, L., Elson, M., Geiger, S. J., Lobentanzer, S., Monéger, J., Patterson, A., Schwarzkopf, D. S., Zaneva, M., & Brown, N. J. L.
- Affiliations
- University of Cambridge (GB)
- Australian National University (AU)
- Temple University (US)
- University of Sussex (GB)
- Ruhr University Bochum (DE)
- University of Vienna (AT)
- University Hospital Heidelberg (DE)
- Centre National de la Recherche Scientifique (FR)
- University of Sheffield (GB)
- University of Auckland (NZ)
- University of Oxford (GB)
- Linnaeus University (SE)
- Corresponding author
- Crüwell, Sophia
- ORCID
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2023-02-02
- Volume
- 34
- Issue
- 4
- Pages
- 512-522
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W4319062942
- Cited by
- 54
- References
- 43
- Citations by year
- 2026: 15 · 2025: 9 · 2024: 16 · 2023: 14
- FWCI
- 12.67
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (unspecified)
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Psychological Science (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DiVA (Linkoping University) (submittedVersion, other-oa)
- ResearchSpace (University of Auckland) (submittedVersion)
- Funders
- Economic and Social Research Council (Data Analytics & Society Centre for Doctoral Train)
- Ministerium für Kultur und Wissenschaft des Landes Nordrhein-Westfalen (1706dgn006)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- reproducibility, research data sharing, reproducibility assessment, peer review, open science policy
- MeSH terms
- Humans, Periodicals as Topic, Psychology, Research, Reproducibility of Results, Information Dissemination, Editorial Policies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.