The Scientific Method in Practice: Reproducibility in the Computational Sciences
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Since the 1660’s the scientific method has included reproducibility as a mainstay in its effort to root error from scientific discovery. With the explosive growth of digitization in scientific research and communication, it is easier than ever to satisfy this requirement. In computational research experimental details and methods can be recorded in code and scripts, data is digital, papers are frequently online, and the result is the potential for “really reproducible research.” Imagine the ability to routinely inspect code and data and recreate others’ results: Every step taken to achieve the findings can potentially be transparent. Now imagine anyone with an Internet connection and the capability of running the code being able to do this.This paper investigates the obstacles blocking the sharing of code and data to understand conditions under which computational scientists reveal their full research compendium. A survey of registrants at a top machine learning conference (NIPS) was used to discover the strength of underlying factors that affect the decision to reveal code, data, and ideas. Sharing of code and data is becoming more common as about a third of respondents post some on their websites, and about 85% self report to have some code or data publicly available on the web. Contrary to theoretical expectations, the decision to share work is grounded in communitarian norms, although when work remains hidden private incentives dominate the decision. We find that code, data, and ideas are each regarded differently in terms of how they are revealed and that guidance from scientific norms varies with pervasiveness of computation in the field. The largest barriers to sharing are time involved in preparation of work and the legal Intellectual Property framework scientists face.This paper does two things. It provides evidence in the debate about whether scientists’ research revealing behavior is wholly governed by considerations of personal impact or whether the reasoning behind the revealing decision involves larger scientific ideals, and secondly, this research describes the actual sharing behavior in the Machine Learning community.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 18,497 · cited 1×
“(Tversky and Kahneman, 1981).”
Toward a new economics of science
Partha, D., & David, P. A. (1994) · Research Policy · doi:10.1016/0048-7333(94)01002-1 · Cited by 2,448
Lakhani, K. R., & Wolf, R. G. (2003) · doi:10.2139/ssrn.443040 · Cited by 1,531 · cited 2×
“In a comprehensive survey of coders who participate on the open [9-Feb-10 source website SourceForge.com, Lakhani and Wolf (2005) found that many programmers shared their code simply because they enjoyed learning from the coding experience. von Hippel and von Krogh (2006) found that being part of a…”
Lakhani, K. R., & Wolf, R. G. (2005) · Perspectives on Free and Open Source Software · doi:10.7551/mitpress/5326.003.0005 · Cited by 1,530
Harhoff, D., Henkel, J., & von Hippel, E. (2003) · Research Policy · doi:10.1016/s0048-7333(03)00061-1 · Cited by 772 · cited 1×
“Innovators may choose not to protect their product, through either patenting, trade secrets, or trademarking, because the cost of doing so outweighs the potential profit (Harhoff et al. 2003).”
Cited by
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
“Default practice can shift to openness while guidelines are developed for the justification to keep data closed or delay their release (Stodden, 2010).”
An introduction to Docker for reproducible research
Boettiger, C. (2015) · ACM SIGOPS Operating Systems Review · doi:10.1145/2723872.2723882 · Cited by 1,059
Data Quality: Some Comments on the NASA Software Defect Datasets
Shepperd, M., Song, Q., Sun, Z., & Mair, C. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.11 · Cited by 570
The Consequences of the Internet for Politics
Farrell, H. (2012) · Annual Review of Political Science · doi:10.1146/annurev-polisci-030810-110815 · Cited by 387
Reproducibility and replicability of rodent phenotyping in preclinical studies
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R., Golani, I., Gomez-Marin, A., Heller, R., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.01.003 · Cited by 215
Self-citation
Metadata
- Authors
- Stodden, V.
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2010
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W4235777167
- Cited by
- 93
- References
- 5
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.