The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Transparency and reproducibility in artificial intelligence
Open accessAbstract
Open Access articles citing this article. Deep learning in the stock market—a systematic survey of practice, backtesting, and applications Kenniy Olorunnimbe & Herna Viktor Artificial Intelligence Review Open Access 30 June 2022 Machine learning generalizability across healthcare settings: insights from multi-site COVID-19 screening Jenny Yang , Andrew A. S. Soltan & David A. Clifton npj Digital Medicine Open Access 07 June 2022 Bringing machine learning to research on intellectual and developmental disabilities: taking inspiration from neurological diseases Chirag Gupta , Pramod Chandrashekar … Daifeng Wang Journal of Neurodevelopmental Disorders Open Access 02 May 2022
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In addition, AI's transparency and interpretability are challenged by factors such as patient privacy, algorithm interpretability, publication bias, etc. 39,40.”
From Artificial intelligence in pancreatic cancer (2022) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“highlighting the risk of using corporate models in a sensitive area such as medicine, where patient privacy, transparency and reliability are essential74,76”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [76]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“When generative AI follows the release-by-blogpost model, it is reaping the benefits of mimicking scientific communication -including associations of reproducibility and rigour [21, 23]- without actually doing the work.”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [23]
In the citing paper
“One of the most effective methods of mitigating AI biases is to ensure the use of diverse and representative datasets during AI development and training [67, 87].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [87]
In the citing paper
“A critical component of AI evaluation and development more generally is ensuring transparency and reproducibility, allowing independent research groups to review and build on prior scientific progress (196).”
From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [196]
In the citing paper
“In addition, the lack of transparency and reproducibility of AI computational algorithm methods and code in many published studies threatens scientific progress and discovery, which require that independent scientists are able to scrutinize and reproduce results98.”
From Smart wearable devices in cardiovascular care: Where we are and how to move forward (2021) · cites this paper as [98]
In the citing paper
“Because of the particular significance of addressing the data-related concerns, different countries have developed policies and laws [160].”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [160]
In the citing paper
“Unfortunately, though, the majority of existing AI/ML models remain in specialized journals, with poorly maintained code and variable rigor in validation.283”
From Advancing Targeted Protein Degradation via Multiomics Profiling and Artificial Intelligence (2023) · cites this paper as [283]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal Of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,852
(2015) · neural information processing systems · Cited by 912
A curated mammography data set for use in computer-aided detection and diagnosis research
Lee, R. S., Gimenez, F., Hoogi, A., Miyake, K. K., Gorovoy, M., & Rubin, D. L. (2017) · Scientific Data · doi:10.1038/sdata.2017.177 · Cited by 859
Artificial intelligence faces reproducibility crisis
Hutson, M. (2018) · Science · doi:10.1126/science.359.6377.725 · Cited by 573
Enhancing reproducibility for computational methods
Stodden, V., Mcnutt, M. K., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 464
Self-citationBluemke, D. A., Moy, L., Bredella, M. A., Ertl-Wagner, B. B., Fowler, K. J., Goh, V. J., Halpern, E. F., Hess, C. P., Schiebler, M. L., & Weiss, C. R. (2020) · Radiology · doi:10.1148/radiol.2019192515 · Cited by 343
Reproducible Research Practices and Transparency across the Biomedical Literature
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002333 · Cited by 310
Self-citationState of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293 · cited 1×
“Even with extensive description, reproducing complex computational pipelines based purely on text is a subjective and challenging task 9 .”
BackgroundWallach, J. D., Boyack, K. W., & Ioannidis, J. P. A. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2006930 · Cited by 269
Self-citationExploration, Inference, and Prediction in Neuroscience and Biomedicine
Bzdok, D., & Ioannidis, J. P. A. (2019) · Trends in Neurosciences · doi:10.1016/j.tins.2019.02.001 · Cited by 216
Self-citationReproducibility of computational workflows is automated using continuous analysis
Beaulieu‐Jones, B. K., & Greene, C. S. (2017) · Nature Biotechnology · doi:10.1038/nbt.3780 · Cited by 151
Self-citationGundersen, O. E., Gil, Y., & Aha, D. W. (2018) · AI Magazine · doi:10.1609/aimag.v39i3.2816 · Cited by 99
Crane, M. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00018 · Cited by 74
Toward unrestricted use of public genomic data
Amann, R., Baichoo, S., Blencowe, B. J., Bork, P., Borodovsky, M. Y., Brooksbank, C., Chain, P., Colwell, R. R., Daffonchio, D. G., Danchin, A., et al. (2019) · Science · doi:10.1126/science.aaw1280 · Cited by 56
Self-citationPutting oncology patients at risk.
Carlson, B. (2012) · PubMed · Cited by 3
Research works printed but not linked: 1
- Sculley, D. et al. in Advances in Neural Information Processing Systems 28 (eds Cortes, C. et al.) 2503–2511 (Curran Associates, Inc., 2015).
Cited by
Sorted by citations
Crosby, D., Bhatia, S., Brindle, K. M., Coussens, L. M., Dive, C., Emberton, M., Esener, S., Fitzgerald, R. C., Gambhir, S. S., Kuhn, P., et al. (2022) · Science · doi:10.1126/science.aay9040 · Cited by 1,379
“This is an area where there is an opportunity for AI and ML to help detect cancer (114) and to predict risk of progression (115), although issues of transparency and reproducibility must be addressed (116).”
Smart wearable devices in cardiovascular care: Where we are and how to move forward
Bayoumy, K., Gaber, M., Elshafeey, A., Mhaimeed, O., Dineen, E. H., Marvel, F. A., Martin, S. S., Muse, E. D., Turakhia, M. P., Tarakji, K. G., et al. (2021) · Nature Reviews Cardiology · doi:10.1038/s41569-021-00522-7 · Cited by 829
Artificial Intelligence in Cancer Research and Precision Medicine
Bhinder, B., Gilvary, C., Madhukar, N. S., & Elemento, O. (2021) · Cancer Discovery · doi:10.1158/2159-8290.cd-21-0090 · Cited by 781
Artificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798
Metadata
- Authors
- Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., Mason, C. E., Jones, W., Dopazo, J., Furlanello, C., Waldron, L., Wang, B., McIntosh, C., Goldenberg, A., Kundaje, A., Greene, C. S., Broderick, T., Hoffman, M. M., Leek, J. T., Korthauer, K., Huber, W., Brazma, A., Pineau, J., Tibshirani, R., Hastie, T., Ioannidis, J. P. A., Quackenbush, J., & Aerts, H. J. W. L.
- Affiliations
- University of Toronto (CA)
- Brigham and Women's Hospital (US)
- University Health Network (CA)
- National Center for Toxicological Research (US)
- University of Oxford (GB)
- SAS Institute (United States) (US)
- Q2 Solutions (United States) (US)
- City University of New York (US)
- Canadian Institute for Advanced Research (CA)
- Alex's Lemonade Stand Foundation (US)
- University of British Columbia (CA)
- European Bioinformatics Institute (GB)
- Stanford Medicine (US)
- ORCID
- Haibe-Kains, Benjamin
- Adam, George Alexandru
- Hosny, Ahmed
- Khodakarami, Farnoosh
- Kusko, Rebecca
- Sansone, Susanna-Assunta
- Tong, Weida
- Mason, Christopher E.
- Jones, Wendell
- Furlanello, Cesare
- Waldron, Levi
- Wang, Bo
- McIntosh, Chris
- Goldenberg, Anna
- Kundaje, Anshul
- Greene, Casey S.
- Broderick, Tamara
- Hoffman, Michael M.
- Leek, Jeffrey T.
- Korthauer, Keegan
- Huber, Wolfgang
- Brazma, Alvis
- Pineau, Joelle
- Tibshirani, Robert
- Hastie, Trevor
- Ioannidis, John P. A.
- Quackenbush, John
- Aerts, Hugo J. W. L.
- Venue
- Nature
- Published
- 2020-10-14
- Volume
- 586
- Issue
- 7829
- Pages
- E14-E16
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC8144864
- OpenAlex
- W3008058947
- Cited by
- 540
- References
- 18
- Citations by year
- 2026: 83 · 2025: 91 · 2024: 97 · 2023: 110 · 2022: 79 · 2021: 70 · 2020: 10
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Nature (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- PubMed Central (acceptedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- RISalud Institutional Health Repository of Andalucía (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Funders
- Biotechnology and Biological Sciences Research Council (BB/L005069/2, BB/E025080/1, BB/L005069/1, BB/J020265/1, BB/L005050/1)
- National Cancer Institute (U24CA180996, R01CA237170, CA237170)
- National Institute of General Medical Sciences (DP2GM123485)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- reproducibility, explainability, deep learning, backtesting, generalizability, machine learning in healthcare, intellectual and developmental disabilities
- MeSH terms
- Algorithms, Artificial Intelligence, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
