On Reproducible AI: Towards Reproducible Research, Open Science, and Digital Scholarship in AI Publications
Open accessAbstract
Artificial intelligence, like any science, must rely on reproducible experiments to validate results. Our objective is to give practical and pragmatic recommendations for how to document AI research so that results are reproducible. Our analysis of the literature shows that AI publications currently fall short of providing enough documentation to facilitate reproducibility. Our suggested best practices are based on a framework for reproducibility and recommendations for best practices given by scientific organizations, scholars, and publishers. We have made a reproducibility checklist based on our investigation and described how every item in the checklist can be documented by authors and examined by reviewers. We encourage authors and reviewers to use the suggested best practices and author checklist when considering submissions for AAAI publications and conferences.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Ubiquity and free availability are not equal to openness and transparency.
“Ubiquity and free availability are not equal to openness and transparency [3, 9, 21].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [21]
The R3: Method Reproducible recommendations require metadata, a reuse license, and a persistent identifier in addition to making data accessible. The paper thus supports the point that free availability alone is insufficient for open, understandable research, but our copy does not address ubiquity as a separate condition. (LLM, conf=0.80)
Experimental — accuracy not yet measuredWhen generative AI follows the release-by-blogpost model, it is reaping the benefits of mimicking scientific communication without actually doing the work.
“When generative AI follows the release-by-blogpost model, it is reaping the benefits of mimicking scientific communication -including associations of reproducibility and rigour [21, 23]- without actually doing the work.”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [21]
Some systems are open in name only.
“some systems are open in name only.”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [21]
The 'State of the Art: How AI Research Is' section reports that few surveyed AI papers shared source code or results. The 'Recommendations for Source Code' section describes what researchers should make available. Neither passage says that systems described as open are open in name only; I searched for nominal openness and inaccessible code or data. (LLM, conf=0.69)
Experimental — accuracy not yet measuredSome degree of run-to-run variability is unavoidable in deep learning frameworks, particularly when dealing with Language Models.
“recent works and official documentation emphasize that some degree of run-to-run variability is unavoidable in deep learning frameworks (Desai et al., 2025; Gundersen et al., 2018), in particular when dealing with Language Models (Semmelrock et al., 2025).”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Gundersen et al., 2018
The paper’s discussion of reproducibility in AI and its definition of repeatability (State of the Art: How AI Research Is; page unavailable) do not address whether run-to-run variability in deep learning frameworks is unavoidable. A passage in Result Descripton and Analysis describes different results when datasets or parameters change, not variability between otherwise identical runs. Searches for randomness, nondeterminism, and deep learning frameworks found no passage establishing the attributed point; the language-model qualification is cited to a different source. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · see citing works
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 3× · see citing works
“Reproducibility is challenging for many sciences, for example when the variability of physical samples and reagents can significantly affect the outcome (Begley and Ellis 2012; Lithgow et al. 2017). In computer science, a large portion of the experiments are fully conducted on computers, making the experiments more st…”
BackgroundInfluentialPromoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · see citing works
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685 · see citing works
Sharing Detailed Research Data Is Associated with Increased Citation Rate
Piwowar, H. A., Day, R. S., & Fridsma, D. B. (2007) · PLoS ONE · doi:10.1371/journal.pone.0000308 · Cited by 881 · see citing works
Enhancing reproducibility for computational methods
Stodden, V., Mcnutt, M. K., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 414 · see citing works
Self-citationImplementing Reproducible Research
Stodden, V. (2014) · doi:10.1201/b16868 · Cited by 314 · see citing works
State of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293 · see citing works
Self-citationTen Simple Rules for the Care and Feeding of Scientific Data
Goodman, A. A., Pepe, A., Blocker, A. W., Borgman, C. L., Cranmer, K. S., Crosas, M., Stefano, R. D., Gil, Y., Groth, P., Hedstrom, M. L., et al. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003542 · Cited by 231 · see citing works
Self-citationGil, Y., David, C. H., Demir, İ., Essawy, B. T., Fulweiler, R. W., Goodall, J. L., Karlstrom, L., Lee, H., Mills, H. J., Oh, J., et al. (2016) · Earth and Space Science · doi:10.1002/2015ea000136 · Cited by 182 · see citing works
Self-citationScholarly Context Not Found: One in Five Articles Suffers from Reference Rot
Klein, M., de Sompel, H. V., Sanderson, R., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0115253 · Cited by 182 · see citing works
A long journey to reproducible results
Lithgow, G. J., Driscoll, M., & Phillips, P. (2017) · Nature · doi:10.1038/548387a · Cited by 157 · see citing works
CODATA-ICSTI Task Group on Data Citation Standards and Practices (2013) · Data Science Journal · doi:10.2481/dsj.osom13-043 · Cited by 148 · see citing works
Achieving human and machine accessibility of cited data in scholarly publications
Starr, J., Castro, E., Crosas, M., Dumontier, M., Downs, R. R., Duerr, R. E., Laurel, H., Haendel, M. A., Herman, I., Hodson, S., et al. (2015) · PeerJ Computer Science · doi:10.7717/peerj-cs.1 · Cited by 129 · see citing works
The Anatomy of a Data Citation: Discovery, Reuse, and Credit
Mooney, H., & Newton, M. P. (2012) · Journal of Librarianship and Scholarly Communication · doi:10.7710/2162-3309.1035 · Cited by 128 · see citing works
Quantifying Reproducibility in Computational Biology: The Case of the Tuberculosis Drugome
Garijo, D., Kinnings, S., Xie, L., Xie, L., Zhang, Y., Bourne, P. E., & Gil, Y. (2013) · PLoS ONE · doi:10.1371/journal.pone.0080278 · Cited by 114 · see citing works
Self-citationOffspring from Reproduction Problems: What Replication Failure Teaches Us
Fokkens, A. S., van Erp, M., Postma, M., Ted Pedersen, A., Vossen, P. T. J. M., & Freire, N. (2013) · Cited by 105 · see citing works
Joint declaration of data citation principles
Callaghan, S. (2014) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7356758 · Cited by 105 · see citing works
Uhlir, P. E. (2012) · Cited by 61 · see citing works
Intention-aware routing to minimise delays at electric vehicle charging stations
Weerdt, M. M. D., Gerding, E., Stein, S., Robu, V., & Jennings, N. R. (2013) · doi:10.1145/2516911.2516923 · Cited by 55 · see citing works
A Proposed Standard for the Scholarly Citation of Quantitative Data
Altman, M., & King, G. (2006) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3783163 · Cited by 47 · see citing works
Data Stewardship in the Earth Sciences
Downs, R. R., Duerr, R. E., Hills, D. J., et al. (2015) · D-Lib Magazine · doi:10.1045/july2015-downs · Cited by 19 · see citing works
DeRisi, S., Kennison, R., & Twyman, N. (2003) · PLoS Biology · doi:10.1371/journal.pbio.0000057 · Cited by 19 · see citing works
Gil, Y. (2017) · Data Science · doi:10.3233/ds-170011 · Cited by 19 · see citing works
Self-citationOn the State and Importance of Reproducible Experimental Research in Parallel ComputingPreprint
Hunold, S., & Träff, J. L. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1308.3648 · Cited by 16 · see citing works
Committing to Publishing Data in the Earth and Space Sciences
Hanson, B., Lehnert, K., & Cutcher‐Gershenfeld, J. (2015) · Eos · doi:10.1029/2015eo022207 · Cited by 16 · see citing works
Open science and community norms
Joly, Y., Dove, E. S., Kennedy, K. L., et al. (2012) · Medical Law International · doi:10.1177/0968533212458431 · Cited by 15 · see citing works
Towards Continuous Scientific Data Analysis and Hypothesis Evolution
Gil, Y., Garijo, D., Ratnakar, V., Mayani, R., Adusumilli, R., Boyce, H., Srivastava, A., & Mallick, P. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11157 · Cited by 14 · see citing works
Self-citationA Survey on Reproducibility in Parallel ComputingPreprint
Hunold, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04217 · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Baker, M. 2016. Is There a Reproducibility Crisis? Nature 533(7604): 452-54.
- Bouquet, P.; Serafini, L.; Zanobini, S.; and Benerecetti, M. 2003. An Algorithm for Semantic Coordination. Paper pre- sented at the Second International Semantic Integration Workshop. Sanibel Island, FL, October 20-23.
- Braun, M. L., and Ong, C. S. 2014. Open Science in CA: FORCE11. doi.org/10.25490/a97f-egyk. · DOI
Cited by
Sorted by citations
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612 · see citing works
Transparency and reproducibility in artificial intelligence
Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., et al. (2020) · Nature · doi:10.1038/s41586-020-2766-y · Cited by 540 · see citing works
Liesenfeld, A., Lopez, A., & Dingemanse, M. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3604316 · Cited by 111 · see citing works
Rethinking open source generative AI: open washing and the EU AI Act
Liesenfeld, A., & Dingemanse, M. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659005 · Cited by 67 · see citing works
Paolini, L., Vahdati, S., Di Iorio, A., Wardenga, R., Heibi, I., & Peroni, S. (2025) · Scientometrics · doi:10.1007/s11192-025-05418-8 · Cited by 8 · see citing works
- Authors
- Gundersen, O. E., Gil, Y., & Aha, D. W.
- Affiliations
- Association for the Advancement of Artificial Intelligence (US)
- United States Naval Research Laboratory (US)
- Venue
- AI Magazine
- Publisher
- Wiley
- Published
- 2018-09
- Volume
- 39
- Issue
- 3
- Pages
- 56-68
- ISSN
- 0738-4602 (ISSN-L)
- 2371-9621
- OpenAlex
- W2890056718
- Cited by
- 99
- References
- 34
- Citations by year
- 2026: 14 · 2025: 6 · 2024: 13 · 2023: 26 · 2022: 16 · 2021: 13 · 2020: 7 · 2019: 4
- FWCI
- 8.52
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- AI Magazine (publishedVersion, cc-by)
- Funders
- National Science Foundation (1440323)
- Norges Teknisk-Naturvitenskapelige Universitet
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- reproducibility, artificial intelligence research, research documentation, reproducibility checklist, open science, digital scholarship, peer review, author guidelines, best practices
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
