The DOI matches its Crossref record.
An invitation to reproducible computational research
Open accessAbstract
I am genuinely thrilled to see Biostatistics make a formal venture into computational reproducibility, and I congratulate the editors of Biostatistics on taking this much needed step. I find the policies being adopted by Biostatistics eminently practical, and I hope that many authors will begin using this option. In my comments, I will try to explain how I came to believe in the importance of reproducibility and why I think others may find it in their interest and in the community interest. I will then briefly mention some efforts in other disciplines. Computation-based science publication is currently a doubtful enterprise because there is not enough support for identifying and rooting out sources of error in computational work. In my own experience, error is ubiquitous in scientific computing, and one needs to work very diligently and energetically to eliminate it. One needs a very clear idea of what has been done in order to know where to look for likely sources of error. I often cannot really be sure what a student or colleague has done from his/her own presentation, and in fact often his/her description does not agree with my own understanding of what has been done, once I look carefully at the scripts. Actually, I find that researchers quite generally forget what they have done and misrepresent their computations. Computing results are now being presented in a very loose, “breezy” way—in journal articles, in conferences, and in books. All too often one simply takes computations at face value. This is spectacularly against the evidence of my own experience. I would much rather that at talks and in referee reports, the possibility of such error were seriously examined. I was inspired more than 15 years ago by John Claerbout, an earth scientist at Stanford, to begin practicing reproducible computational science. See Claerbout and Karrenbach (1992). He pointed out to me, in a way paraphrased in Buckheit and Donoho (1995): “an article about computational result is advertising, not scholarship. The actual scholarship is the full software environment, code and data, that produced the result.” This struck me as getting to the heart of a central problem in modern scientific publication. Most of the work in a modern research project is hidden in computational scripts that go to produce the reported results. If these scripts are not brought out into the open, no one really knows what was done in a certain project; for example, tables and figures can be subtly miscomputed or mislabeled. I saw the good sense in making a commitment to transparency as the best way both to avoid and correct errors. Over the years, I have persuaded numerous graduate students and postdocs to work in a way that made clear precisely how each figure in a paper was being computed (Donoho and others, 2009). Over time, this effort has paid multiple dividends; I believe anyone who understands the process and the benefits will eventually be moved to practice it. Computational reproducibility is not an afterthought—it is something that must be designed into a project from the beginning. One does need to develop a whole set of programming and research disciplines with the end result in mind and stick with them. A simple example is visible in the framework provided by Biostatistics: one has to plan to create a single script, in the R language, that generates all the figures and tables used in one's paper. Many individuals might otherwise work “interactively,” and without planning for the end result. They might tend naturally to improvise commands at the keyboard to produce an analysis in bits and pieces, which they then transcribe and piece together into evidence for a paper. To be branded reproducible by Biostatistics, such an informal approach will never be accepted. Does reproducibility sound like “extra work”? It can be, particularly when one is first trying to do it, that is, to break one's own previous nonreproducible habits. But here are some reasons we as “scientists” will want to work in a computationally reproducible way. (a) Improved work and work habits. If we commit from the outset to make a given project computationally reproducible, we work differently, in a way that is much more transparent to ourselves and to others. We know where the project is headed—a published set of R scripts—and will have better focus while we are developing those scripts. Knowing that these scripts will be available to others, we will be motivated to polish and improve our scripts to raise them to a higher level of quality than we would have done if we thought “no one is looking.” (b) Improved teamwork. If we are part of a team that has committed to work this way, we will be able to communicate much more efficiently with other team members about our developing results; they can see what we are doing and propose improvements. Many questions team members may have about what we actually did in a computation will be settled much more efficiently and suggestion
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
phyloseq: An R Package for Reproducible Interactive Analysis and Graphics of Microbiome Census Data
McMurdie, P. J., & Holmes, S. (2013) · PLoS ONE · doi:10.1371/journal.pone.0061217 · Cited by 21,322
Waste Not, Want Not: Why Rarefying Microbiome Data Is Inadmissible
McMurdie, P. J., & Holmes, S. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003531 · Cited by 2,921
Dipy, a library for the analysis of diffusion MRI data
Garyfallidis, E., Brett, M., Amirbekian, B., Rokem, A., van der Walt, S., Descoteaux, M., Nimmo-Smith, I., & Contributors, D. (2014) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2014.00008 · Cited by 1,371
“In addition, since the code is entirely free and open, Dipy is ideally suited to facilitate reproducible research (Donoho, 2010).”
BackgroundHow to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
The case for open computer programs
Ince, D. C., Hatton, L., & Graham-Cumming, J. (2012) · Nature · doi:10.1038/nature10836 · Cited by 535
Metadata
- Authors
- Donoho, D. L.
- Venue
- Biostatistics
- Publisher
- Oxford University Press (OUP)
- Published
- 2010-06-10
- Volume
- 11
- Issue
- 3
- Pages
- 385-388
- ISSN
- 1465-4644 (ISSN-L)
- 1468-4357
- OpenAlex
- W2142491263
- Cited by
- 162
- References
- 0
- FWCI
- 16.27
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An invitation to reproducible computational research. https://app.cochat.ai/papers/10.1093/biostatistics/kxq028
- The paper
- Donoho, D. L. (2010). An invitation to reproducible computational research. Biostatistics. https://doi.org/10.1093/biostatistics/kxq028
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.