The fundamental principles of reproducibility
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Reproducibility is a confused terminology. In this paper, I take a fundamental view on reproducibility rooted in the scientific method. The scientific method is analysed and characterized in order to develop the terminology required to define reproducibility. Furthermore, the literature on reproducibility and replication is surveyed, and experiments are modelled as tasks and problem solving methods. Machine learning is used to exemplify the described approach. Based on the analysis, reproducibility is defined and three different degrees of reproducibility as well as four types of reproducibility are specified. This article is part of the theme issue ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico’.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Object Detection With Deep Learning: A Review
Zhao, Z.-Q., Zheng, P., Xu, S.-T., & Wu, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2876865 · Cited by 5,545 · cited 1×
“Although the analysis (low errors by deep learning methods on visual benchmarking tasks) have been reproduced many times [16], the claim that deep learning achieves super-human performance is still”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018) · Science · doi:10.1126/science.aar6404 · Cited by 3,562
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 1×
“s and the impressive results that have been achieved. Deep learning was the main component in the AI’s that beat top level human players of Atari [44], go, chess, shogi [45], poker [46] and Starcraft [47]. These systems are called end-to-end as they learn to play these games without human intervention. Deep models lea…”
BackgroundTorralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440 · cited 1×
“s super-human performance is still debated [17]. (x) Observe the world systematically: To be a trusted source of knowledge, experiments must be designed to remove biases. As many data-sets are biased [18], the bias can be reduced by conducting the experiment on several data-sets, similar to what is done in the example…”
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 2×
“rrie and Svehla [26] Repeatability is when the same team repeats the same experiment in the same lab and reproducibility is the when another team performs the same experiment in a different lab. Peng [27] Reproducibility spectrum that goes from not reproducible to gold standard depended on whether code and data are sh…”
BackgroundMethodWhat does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · cited 4×
“can reveal different information that allows for different interpretation, which means that different investigators might interpret the analysis in different ways and thus reach different conclusions [34]. So, to conclude, our beliefs in scientific theories and hypotheses are strengthened with new evidence. Reproducing…”
BackgroundMethodResultInfluentialShall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125 · cited 4×
“ity is independent investigators subjecting the original data to their own analyses and interpretations. Drummond [23] "Reproducibility requires changes; replicability avoids them." Schmidt [24] Distinguishes between two fundamental levels of replication: 1) the narrow bounded notion that is a repetition of…”
BackgroundMethodInfluentialReproducibility and Replicability in Science
Committee on Reproducibility and Replicability in Science, Board on Behavioral, C., Committee on National Statistics, Education, D. O. B. A. S. S. A., Nuclear and Radiation Studies Board, Studies, D. O. E. A. L., Board on Mathematical Sciences and Analytics, Committee on Applied and Theoretical Statistics, Sciences, D. O. E. A. P., Board on Research Data and Information, et al. (2019) · doi:10.17226/25303 · Cited by 924
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · cited 4×
“When only using one term to describe different degrees of reproducibility or replication, the authors Schmidt [24] and Nosek & Lakens [33] use replication while Goodman et al.”
BackgroundMethodInfluentialSuperhuman AI for multiplayer poker
Brown, N., & Sandholm, T. (2019) · Science · doi:10.1126/science.aay2400 · Cited by 592 · cited 1×
“players of Atari [44], go, chess, shogi [45], poker [46] and Starcraft [47].”
BackgroundState of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293
Self-citationA precise extragalactic test of General Relativity
Collett, T. E., Oldham, L. J., Smith, R. J., Auger, M. W., Westfall, K. B., Bacon, D., Nichol, R. C., Masters, K. L., Koyama, K., & van den Bosch, R. (2018) · Science · doi:10.1126/science.aao2469 · Cited by 157
Cited by
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
Reproducibility in machine‐learning‐based research: Overview, barriers, and drivers
Semmelrock, H., Ross‐Hellauer, T., Kopeinik, S., Theiler, D., Haberl, A., Thalmann, S., & Kowald, D. (2025) · AI Magazine · doi:10.1002/aaai.70002 · Cited by 118
Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers
Semmelrock, H., Ross-Hellauer, T., Kopeinik, S., Theiler, D., Haberl, A., Thalmann, S., & Kowald, D. (2024) · arXiv · doi:10.48550/arxiv.2406.14325 · Cited by 88
What is reproducibility in artificial intelligence and machine learning research?
Desai, A., Abdelhamid, M., & Padalkar, N. R. (2025) · AI Magazine · doi:10.1002/aaai.70004 · Cited by 42
What is Reproducibility in Artificial Intelligence and Machine Learning Research?
Desai, A., Abdelhamid, M., & Padalkar, N. R. (2024) · arXiv · doi:10.48550/arxiv.2407.10239 · Cited by 36
Metadata
- Authors
- Gundersen, O. E.
- Venue
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
- Publisher
- The Royal Society
- Published
- 2021-03-29
- Volume
- 379
- Issue
- 2197
- Pages
- 20200210-20200210
- ISSN
- 1364-503X (ISSN-L)
- 1471-2962
- OpenAlex
- W3109268737
- Cited by
- 90
- References
- 17
- FWCI
- 9.16
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.