Tracking Replicability as a Method of Post-Publication Open Evaluation
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Recent reports have suggested that many published results are unreliable. To increase the reliability and accuracy of published papers, multiple changes have been proposed, such as changes in statistical methods. We support such reforms. However, we believe that the incentive structure of scientific publishing must change for such reforms to be successful. Under the current system, the quality of individual scientists is judged on the basis of their number of publications and citations, with journals similarly judged via numbers of citations. Neither of these measures takes into account the replicability of the published findings, as false or controversial results are often particularly widely cited. We propose tracking replications as a means of post-publication evaluation, both to help researchers identify reliable findings and to incentivize the publication of reliable results. Tracking replications requires a database linking published studies that replicate one another. As any such database is limited by the number of replication attempts published, we propose establishing an open-access journal dedicated to publishing replication attempts. Data quality of both the database and the affiliated journal would be ensured through a combination of crowd-sourcing and peer review. As reports in the database are aggregated, ultimately it will be possible to calculate replicability scores, which may be used alongside citation counts to evaluate the quality of work published in individual journals. In this paper, we lay out a detailed description of how this system could be implemented, including mechanisms for compiling the information, ensuring data quality, and incentivizing the research community to participate.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mixed-effects modeling with crossed random effects for subjects and items
Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008) · Journal of Memory and Language · doi:10.1016/j.jml.2007.12.005 · Cited by 8,540
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 5×
“Ioannidis (2005a) found that of 34 highly cited clinical research studies for which replication attempts had been published, seven (20”
BackgroundInfluentialCategorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models
Jaeger, T. F. (2008) · Journal of Memory and Language · doi:10.1016/j.jml.2007.11.007 · Cited by 3,317 · cited 3×
“Individuals can solve the Tragedy of the Commons by adopting common rules or changing incentive structures. give a recent example, Jaeger (2008), Baayen et al. (2008), and others convinced many language processing researchers to switch from ANOVAs to mixed effects models, in part by convincing…”
BackgroundBelief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246 · cited 2×
“Consistent with this claim, there is some evidence that highly cited papers are less likely to replicate (Ioannidis, 2005a)andthatpublicationbiasaffectshigh-impactjournalsmore severely(Ioannidis,2005a;Munafòetal.,2009).”
BackgroundPublication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 3×
“This has been extensively documented in the medical literature (Dickersin et al., 1987, 1992; Easterbrook et al., 1991; Callaham et al., 1998; Misakian and Bero, 1998; Olson et al., 2002; Dwan et al., 2008; Sena et al., 2010), with additional reports in political science (Gerberg et al., 2001),…”
BackgroundCircular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696
Lohmueller, K. E., Pearce, C. L., Pike, M., Lander, E. S., & Hirschhorn, J. N. (2003) · Nature Genetics · doi:10.1038/ng1071 · Cited by 1,900 · cited 3×
“Recent studies have reported that relatively few genetic association links can be replicated (Ioannidis et al., 2001, 2003; Hirschhorn et al., 2002; Lohmueller et al., 2003; Trikalinos et al., 2004).”
BackgroundCheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
A comprehensive review of genetic association studies
Hirschhorn, J. N., Lohmueller, K., Byrne, E., & Hirschhorn, K. (2002) · Genetics in Medicine · doi:10.1097/00125817-200203000-00002 · Cited by 1,674 · cited 2×
“Recent studies have reported that relatively few genetic association links can be replicated (Ioannidis et al., 2001, 2003; Hirschhorn et al., 2002; Lohmueller et al., 2003; Trikalinos et al., 2004).”
Background
Cited by
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
The economics of reproducibility in preclinical research
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002165 · Cited by 1,055
“Indeed, there are many different perspectives on the size of this problem, and published estimates of irreproducibility range from 51% [5] to 89% [6] (Fig 1).”
BackgroundRecommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823
Hartshorne, J. K., & Germine, L. T. (2015) · Psychological Science · doi:10.1177/0956797614567339 · Cited by 788
Self-citationFacts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609
“Several recent publications have begun to discuss the hurdles and disincentives to conducting replications that appear to be endemic to the social science research infrastructure (e.g., Carpenter, 2012; Hartshome & Schachner, 2012; Makel, 2014; Makel & Plucker, 2014; Schmidt, 2009; Spellman, 2012).”
BackgroundAn Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
Open Science Collaboration (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462588 · Cited by 614
Deep impact: unintended consequences of journal rank
Brembs, B., Button, K., & Munafò, M. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00291 · Cited by 441
Theodoulidis, B., Diaz, D., Crotto, F., & Rancati, E. (2017) · Tourism Management · doi:10.1016/j.tourman.2017.03.018 · Cited by 333
Rewarding Replications: A Sure and Simple Way to Improve Psychological Science
Koole, S. L., & Lakens, D. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462586 · Cited by 281
Marsden, E., Morgan‐Short, K., Thompson, S., & Abugaber, D. (2018) · Language Learning · doi:10.1111/lang.12286 · Cited by 268
Frank, M. C., & Saxe, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460686 · Cited by 175
Teixeira da Silva, J. A., & Dobránszki, J. (2015) · Accountability in Research · doi:10.1080/08989621.2014.899909 · Cited by 158
Oxygenation in Cell Culture: Critical Parameters for Reproducibility Are Routinely Not Reported
Al-Ani, A., Toms, D., Kondro, D., Thundathil, J., Yu, Y., & Ungrin, M. (2018) · PLOS ONE · doi:10.1371/journal.pone.0204269 · Cited by 152
Reproducibility: changing the policies and culture of cell line authentication
Freedman, L. P., Gibson, M. C., Ethier, S. P., Soule, H. R., Neve, R. M., & Reid, Y. A. (2015) · Nature Methods · doi:10.1038/nmeth.3403 · Cited by 136
Irreproducibility in Preclinical Biomedical Research: Perceptions, Uncertainties, and Knowledge Gaps
Jarvis, M. F., & Williams, M. (2016) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2015.12.001 · Cited by 129
“Additionally, most of the available data represented input from a small group of scientists (estimated to be less than 250) involving only three discrete therapeutic areas: oncology, hematology, and psychology [15,16,31,32].”
BackgroundEstimating the Reproducibility of Psychological Science
Nosek, B. A., Cohoon, J., Kidwell, M., & Spies, J. R. (2016) · doi:10.31219/osf.io/447b3 · Cited by 88
“Scientific incentives—funding, publication, or awards—can be tuned to encourage an optimal balance in the collective effort of discovery (36, 37).”
BackgroundImproving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2019) · Journal of Neuroscience Research · doi:10.1002/jnr.24340 · Cited by 83
Standard Proteoforms and Their Complexes for Native Mass Spectrometry
Schachner, L. F., Ives, A. N., McGee, J. P., Melani, R. D., Kafader, J. O., Compton, P. D., Patrie, S. M., & Kelleher, N. L. (2019) · Journal of the American Society for Mass Spectrometry · doi:10.1007/s13361-019-02191-w · Cited by 48
The learning curve of murine subretinal injection among clinically trained ophthalmic surgeons
Huang, P., Narendran, S., Pereira, F., Fukuda, S., Nagasaka, Y., Apicella, I., Yerramothu, P., Marion, K. M., Cai, X., Sadda, S. R., et al. (2022) · Translational Vision Science & Technology · doi:10.1167/tvst.11.3.13 · Cited by 10
Metadata
- Authors
- Hartshorne, J. K., & Schachner, A.
- Venue
- Frontiers in Computational Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2012
- Volume
- 6
- Pages
- 8-8
- ISSN
- 1662-5188 (ISSN-L)
- OpenAlex
- W2148613765
- Cited by
- 107
- References
- 31
- FWCI
- 23.36
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.