The DOI matches its Crossref record.
Again, and Again, and Again …
Abstract
. . . Replication—The confirmation of results and conclusions from one study obtained independently in another—is considered the scientific gold standard. New tools and technologies, massive amounts of data, long-term studies, interdisciplinary approaches, and the complexity of the questions being asked are complicating replication efforts, as are increased pressures on scientists to advance their research. The five Perspectives in this section (and associated News and Careers stories, Readers' Poll, and Editorial) explore some of the issues associated with replicating results across various fields. Ryan (p. 1229) highlights the excitement and challenges that come with field-based research. In particular, observing processes as they occur in nature allows for discovery but makes replication difficult, because the precise conditions surrounding the observations are unique. Further, although laboratory research allows for the specification of experimental conditions, the conclusions may not apply to the real world. Debate about the merits of lab-based and field-based studies has been a persistent theme over time. Tomasello and Call (p. 1227) further contribute to this debate in their discussion of some obvious barriers to replication in primate cognition and behavior research (small numbers of subjects, expense, and ethics issues) as well as more subtle ones, such as the nontrivial challenge of designing tasks that elicit complex cognitive behaviors. New technologies continue to produce a deluge of data of different varieties, raising expectations for new knowledge that will translate into meaningful therapeutics and insights into health. Ioannidis and Khoury (p. 1230) outline multiple steps for validating such large-scale data on the path to clinical utility and make suggestions for incentives (and penalties) that could enhance the availability of reliable data and analyses. Peng (p. 1226) discusses the need for a minimum standard of reproducibility in computer sciences, arguing that enough information about methods and code should be available for independent researchers to reach consistent conclusions using original raw data. Specifically, he describes a model that one journal has used to make this a reality. The need to convince the public that data are replicable has grown as science and public policy-making intersect, an issue that has has beset climate change studies. As Santer et al. (p. 1232) describe, having multiple groups examining the same data and generating new data has led to robust conclusions. The importance of replication and reproducibility for scientists is unquestioned. Sometimes attempts to replicate reveal scientific uncertainties. This is one of the main ways that science progresses (see associated News stories of faster-than-light neutrinos and sirtuins, pp. 1200 and 1194). Unfortunately, in rare instances (compared to the body of scientific work), it can also indicate fraud (see the Editorial by Crocker and Cooper, p. 1182). How do we promote the publication of replicable data? The authors in this section come up with possibilities that are targeted at funders, journals, and the research culture itself. In the Readers'Poll, you can make your views known as well.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Aguinis, H., & Bradley, K. J. (2014) · Organizational Research Methods · doi:10.1177/1094428114547952 · Cited by 2,021
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Ten Simple Rules for Reproducible Computational Research
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003285 · Cited by 880
The conundrum of sharing research data
Borgman, C. L. (2012) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22634 · Cited by 880
Replications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750
The Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & van 't Veer, A. (2014) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 705
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609
An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
Open Science Collaboration (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462588 · Cited by 614
Top 10 Replicated Findings From Behavioral Genetics
Plomin, R., DeFries, J. C., Knopik, V. S., & Neiderhiser, J. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691615617439 · Cited by 517
Wallis, J. C., Rolando, E., & Borgman, C. L. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067332 · Cited by 495
Challenges to the measurement of estradiol: an Endocrine Society Position Statement
Rosner, W., Hankinson, S. E., Sluss, P. M., Vesper, H. W., & Wierman, M. E. (2013) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2012-3780 · Cited by 408
Blockchain technology for improving clinical research quality
Benchoufi, M., & Ravaud, P. (2017) · Trials · doi:10.1186/s13063-017-2035-z · Cited by 367
An open investigation of the reproducibility of cancer biology research
Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J., & Nosek, B. A. (2014) · eLife · doi:10.7554/elife.04333 · Cited by 312
Baker, D., Lidster, K., Sottomayor, A., & Amor, S. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001756 · Cited by 305
Marsden, E., Morgan‐Short, K., Thompson, S., & Abugaber, D. (2018) · Language Learning · doi:10.1111/lang.12286 · Cited by 268
Replication and robustness in developmental research
Duncan, G. J., Engel, M., Claessens, A., & Dowsett, C. J. (2014) · Developmental Psychology · doi:10.1037/a0037996 · Cited by 207
Carroll, K. M., Kiluk, B. D., Nich, C., Gordon, M. A., Portnoy, G. A., Marino, D. R., & Ball, S. A. (2014) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2013.13070987 · Cited by 177
“…this study represents the fi rst replication, via randomized clinical trial, of a computer-assisted therapy for addiction (20), which is signi fi cant because replication studies, while critical to the advancement of science (36), are comparatively rare in the clinical science literature (37).”
BackgroundNew methods for the assessment of Parkinson's disease (2005 to 2015): A systematic review
Sánchez‐Ferro, Á., Elshehabi, M., Godinho, C., Salkovic, D., Hobert, M. A., Domingos, J., van Uem, J. M., Ferreira, J. J., & Maetzler, W. (2016) · Movement Disorders · doi:10.1002/mds.26723 · Cited by 145
The Usability Construct: A Dead End?
Tractinsky, N. (2018) · Human–Computer Interaction · doi:10.1080/07370024.2017.1298038 · Cited by 87
Reproducing Statistical Results
Stodden, V. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020127 · Cited by 77
Faggion, C. M., Huivin, R., Aranda, L., Pandis, N., & Alarcon, M. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.02.011 · Cited by 49
(2022) · doi:10.1109/saner53432.2022.00148 · Cited by 28
Metadata
- Authors
- Jasny, B. R., Chin, G., Chong, L., & Vignieri, S.
- Venue
- Science
- Published
- 2011-12-02
- Volume
- 334
- Issue
- 6060
- Pages
- 1225-1225
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2019866696
- Cited by
- 245
- References
- 0
- FWCI
- 53.28
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.