The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
An empirical analysis of journal policy effectiveness for computational reproducibility
Open accessAbstract
A key component of scientific communication is sufficient information for other researchers in the field to reproduce published findings. For computational and data-enabled research, this has often been interpreted to mean making available the raw data from which results were generated, the computer code that generated the findings, and any additional information needed such as workflows and input parameters. Many journals are revising author guidelines to include data and code availability. This work evaluates the effectiveness of journal policy that requires the data and code necessary for reproducibility be made available postpublication by the authors upon request. We assess the effectiveness of such a policy by ( i ) requesting data and code from authors and ( ii ) attempting replication of the published findings. We chose a random sample of 204 scientific papers published in the journal Science after the implementation of their policy in February 2011. We found that we were able to obtain artifacts from 44% of our sample and were able to reproduce the findings for 26%. We find this policy—author remission of data and code postpublication upon request—an improvement over no policy, but currently insufficient for reproducibility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Indeed, the reproducibility of published computational research has been reported as similarly dismal (~25%).4”
From Reproducibility of Quantitative Systems Pharmacology Models: Current Challenges and Future Opportunities (2019) · cites this paper as [4]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449
Repeatability of published microarray gene expression analyses
Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly, I., Cui, X., Culhane, A. C., Falchi, M., Furlanello, C., Game, L., Jurman, G., et al. (2009) · Nature Genetics · doi:10.1038/ng.295 · Cited by 551
King, G. (1995) · PS Political Science & Politics · doi:10.1017/s1049096500057607 · Cited by 473
Enhancing reproducibility for computational methods
Stodden, V., Mcnutt, M. K., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 464
Self-citationStatistical Analyses and Reproducible Research
Gentleman, R. C., & Lang, D. T. (2007) · Journal of Computational and Graphical Statistics · doi:10.1198/106186007x178663 · Cited by 364
Gilbert, K. J., Andrew, R. L., Bock, D. G., Franklin, M. T., Kane, N. C., Moore, J., Moyers, B. T., Renaut, S., Rennison, D. J., Veen, T., et al. (2012) · Molecular Ecology · doi:10.1111/j.1365-294x.2012.05754.x · Cited by 290
Reproducible Research in Computational Harmonic Analysis
Donoho, D. L., Maleki, A., Rahman, I. U., Shahram, M., & Stodden, V. (2008) · Computing in Science & Engineering · doi:10.1109/mcse.2009.15 · Cited by 274
Self-citationStodden, V., Guo, P., & Ma, Z. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067111 · Cited by 257
Self-citationIs Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say 'Usually Not'
Chang, A. C., & Li, P. (2015) · doi:10.2139/ssrn.2669564 · Cited by 217
Research works printed but not linked: 2
- ; American Association for the Advancement of Science Science journals: Editorial Policies. Available at www.sciencemag.org/authors/science-journals-editorial-policies. Accessed September 11 2011. (2011).
- V Stodden JM Borwein DH Bailey “Setting the default to reproducible’ in Computational Science Research. SIAM News . Available at https://sinews.siam.org/Aboutthe-Author/setting-the-default-to-reproducible-in-computational-science-research. Accessed February 16 2018. (2013).
Cited by
Sorted by citations
Items 101–200 of 369
Data Sharing and Scientific Impact in Eddy Covariance Research
Bond‐Lamberty, B. (2018) · Journal of Geophysical Research Biogeosciences · doi:10.1002/2018jg004502 · Cited by 25
Hensel, W. M. (2020) · European Journal for Philosophy of Science · doi:10.1007/s13194-020-00317-6 · Cited by 28
Farooq, S., & Zezéll, D. M. (2023) · Plasmonics · doi:10.1007/s11468-023-02109-z · Cited by 27
Bolibaugh, C., Vanek, N., & Marsden, E. J. (2021) · Bilingualism Language and Cognition · doi:10.1017/s1366728921000535 · Cited by 30
Promoting Open Science Through Research Data Management
Borghi, J., & Gulick, A. E. V. (2022) · Harvard Data Science Review · doi:10.1162/99608f92.9497f68e · Cited by 36
Open spatial sciences: an introduction
Páez, A. (2021) · Journal of Geographical Systems · doi:10.1007/s10109-021-00364-4 · Cited by 26
Monks, T., & Harper, A. L. (2023) · Journal of Simulation · doi:10.1080/17477778.2023.2260772 · Cited by 27
Investigating the reproducibility of the social and behavioural sciences
Miske, O., Abatayo, A. L., Daley, M., Dirzo, M., Fox, N. W., Haber, N., Hahn, K. M., Struhl, M. K., Mawhinney, B., Silverstein, P., et al. (2026) · Nature · doi:10.1038/s41586-026-10203-5 · Cited by 38
Páez, A. (2021) · Geographical Analysis · doi:10.1111/gean.12307 · Cited by 25
Reproducibility in Computing Research: An Empirical Study
Raghupathi, W., Raghupathi, V., & Ren, J. (2022) · IEEE Access · doi:10.1109/access.2022.3158675 · Cited by 33
Predicting peritoneal recurrence in gastric cancer with serosal invasion using a pathomics nomogram
Chen, D., Lai, J., Cheng, J., Fu, M., Lin, L., Chen, F., Huang, R. H., Chen, J., Lu, J., Chen, Y., et al. (2023) · iScience · doi:10.1016/j.isci.2023.106246 · Cited by 23
Borg, D. N., Bon, J. J., Sainani, K. L., Baguley, B. J., Tierney, N. J., & Drovandi, C. (2020) · Sports Medicine · doi:10.1007/s40279-020-01298-5 · Cited by 29
Repository Approaches to Improving the Quality of Shared Data and Code
Trišović, A., Mika, K. A., Boyd, C., Feger, S. S., & Crosas, M. (2021) · Data · doi:10.3390/data6020015 · Cited by 28
Schofield, P. N., Kulka, U., Tapio, S., & Grosche, B. (2019) · International Journal of Radiation Biology · doi:10.1080/09553002.2019.1589026 · Cited by 24
Open Science is for Aging Research, Too
Isaacowitz, D. M., & Lind, M. (2019) · Innovation in Aging · doi:10.1093/geroni/igz028 · Cited by 25
Proceedings of the 17th Python in Science Conference
(2018) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-4af1f417-018 · Cited by 24
Perspectives on Data Reproducibility and Replicability in Paleoclimate and Climate Science
Bush, R. T., Dutton, A. L., Evans, M. N., Loft, R., & Schmidt, G. A. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.00cd8f85 · Cited by 23
No evidence that mandatory open data policies increase error correction
Berberi, I., & Roche, D. G. (2022) · Nature Ecology & Evolution · doi:10.1038/s41559-022-01879-9 · Cited by 26
Reproducible Research Using Biomodels
Mendes, P. (2018) · Bulletin of Mathematical Biology · doi:10.1007/s11538-018-0498-z · Cited by 20
Opening the Door to Registered Reports: Census of Journals Publishing Registered Reports (2013–2020)
Montoya, A. K., Krenzer, W. L. D., & Fossum, J. L. (2021) · Collabra Psychology · doi:10.1525/collabra.24404 · Cited by 25
Open Data Practices among Users of Primary Biodiversity Data
Mandeville, C. P., Koch, W., Nilsen, E. B., & Finstad, A. G. (2021) · BioScience · doi:10.1093/biosci/biab072 · Cited by 31
A guide for social science journal editors on easing into open science
Silverstein, P., Elman, C., Montoya, A. K., McGillivray, B. C., Pennington, C. R., Harrison, C. H., Steltenpohl, C. N., Röer, J. P., Corker, K. S., Charron, L., et al. (2024) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00141-5 · Cited by 28
Schnell, S. D. (2018) · Bulletin of Mathematical Biology · doi:10.1007/s11538-018-0500-9 · Cited by 25
Collaborating with our community to increase code sharing
Cadwallader, L., Papin, J. A., Gabhann, F. M., & Kirk, R. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008867 · Cited by 22
Current status on the need for improved accessibility to climate models code
Añel, J. A., García‐Rodríguez, M., & Iglesias, J. R. (2021) · Geoscientific model development · doi:10.5194/gmd-14-923-2021 · Cited by 27
Enabling the Verification of Computational Results
Stodden, V., Krafczyk, M. S., & Bhaskar, A. (2018) · International Workshop on Practical Reproducible Evaluation of Computer Systems (P-RECS) · doi:10.1145/3214239.3214242 · Cited by 26
Self-citationPre-registration of mathematical models
Ioannidis, J. P. A. (2022) · Mathematical Biosciences · doi:10.1016/j.mbs.2022.108782 · Cited by 20
Klöcking, M., Wyborn, L., Lehnert, K. A., Ware, B., Prent, A., Profeta, L. R., Kohlmann, F., Noble, W. P., Bruno, I. J., Lambart, S., et al. (2022) · Geochimica et Cosmochimica Acta · doi:10.1016/j.gca.2023.04.024 · Cited by 23
Analysis of Open Data and Computational Reproducibility in Registered Reports in PsychologyPreprint
Obels, P., Lakens, D., Coles, N. A., Gottfried, J., & Green, S. A. (2019) · doi:10.31234/osf.io/fk8vh · Cited by 24
Clinical relevance assessment of animal preclinical research (RAA) tool: development and explanation
Gurusamy, K. S., Moher, D., Loizidou, M., Ahmed, I., Avey, M. T., Barron, C. C., Davidson, B. R., Dwek, M. V., Gluud, C. N., Jell, G., et al. (2021) · PeerJ · doi:10.7717/peerj.10673 · Cited by 18
A quality framework for statistical algorithms
Yung, W., Tam, S., Buelens, B., Chipman, H., Dumpert, F., Ascari, G., Rocci, F., Burger, J. M. S., & Choi, I. (2021) · Statistical Journal of the IAOS · doi:10.3233/sji-210875 · Cited by 18
Mangul, S., Mosqueiro, T., Abdill, R. J., Duong, D., Mitchell, K., Sarwal, V., Hill, B., Brito, J., Littman, R. J., Statz, B., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/452532 · Cited by 20
Computational Reproducibility via Containers in Social PsychologyPreprint
Clyburne-Sherin, A., Fei, X., & Green, S. A. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/mf82t · Cited by 21
Improving Process Descriptions in Research by Model-Based Analysis
Shaked, A., & Reich, Y. (2021) · IEEE Systems Journal · doi:10.1109/jsyst.2020.2990488 · Cited by 17
Teaching for Large-Scale Reproducibility Verification
Vilhuber, L., Son, H. H., Welch, M., Wasser, D., & Darisse, M. (2022) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2022.2074582 · Cited by 17
Care to share? Experimental evidence on code sharing behavior in the social sciences
Krähmer, D., Schächtele, L., & Schneck, A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0289380 · Cited by 18
Data reuse in agricultural genomics research: challenges and recommendations
Hafner, A., DeLeo, V. L., Deng, C. H., Elsik, C. G., Fleming, D. S., Harrison, P. W., Kalbfleisch, T., Petry, B., Pucker, B., Quezada-Rodríguez, E. H., et al. (2025) · GigaScience · doi:10.1093/gigascience/giae106 · Cited by 17
Samota, E. K., & Davey, R. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.678554 · Cited by 23
Four erroneous beliefs thwarting more trustworthy research
Yarborough, M. A., Nadon, R., & Karlin, D. (2019) · eLife · doi:10.7554/elife.45261 · Cited by 17
The State of Play of Reproducibility in Statistics: An Empirical Analysis
Xiong, X., & Cribben, I. (2022) · The American Statistician · doi:10.1080/00031305.2022.2131625 · Cited by 24
Kambouris, S., Wilkinson, D. P., Smith, E. T., & Fidler, F. M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0300333 · Cited by 23
Willoughby, C., & Frey, J. G. (2022) · Digital Discovery · doi:10.1039/d1dd00046b · Cited by 14
Integrating GRASS GIS and Jupyter Notebooks to facilitate advanced geospatial modeling education
Haedrich, C. E., Petráš, V., Petrášová, A., Blumentrath, S., & Mitášová, H. (2023) · Transactions in GIS · doi:10.1111/tgis.13031 · Cited by 16
Borg, D. N., Bon, J. J., Sainani, K. L., Baguley, B. J., Tierney, N. J., & Drovandi, C. (2020) · doi:10.31236/osf.io/ftdgj · Cited by 14
Improving statistical reporting in psychology
Schubert, A., Snijder-Steinhilber, M., Kang, H., & Quintana, D. S. (2025) · Communications Psychology · doi:10.1038/s44271-025-00356-w · Cited by 20
Dawning of a new age? Economics journals’ data policies on the test bench
Vlaeminck, S. (2021) · LIBER Quarterly The Journal of the Association of European Research Libraries · doi:10.53377/lq.10940 · Cited by 13
Code sharing in ecology and evolution increases citation rates but remains uncommon
Maitner, B., Andrade, P. E. S., Lei, L., Kass, J. M., Owens, H. L., Barbosa, G. C. G., Boyle, B. L., Castorena, M., Enquist, B. J., Feng, X., et al. (2024) · Ecology and Evolution · doi:10.1002/ece3.70030 · Cited by 17
Jemioło, P., Storman, D., Mamica, M., Szymkowski, M., Żabicka, W., Wojtaszek-Główka, M., & Ligęza, A. (2022) · Sensors · doi:10.3390/s22072538 · Cited by 14
Czapanskiy, M. F., Ponganis, P. J., Fahlbusch, J. A., Schmitt, T. L., & Goldbogen, J. A. (2022) · Journal of Experimental Biology · doi:10.1242/jeb.243872 · Cited by 13
Reproducibility improves exponentially over 63 years of social learning researchPreprint
Minocher, R., Atmaca, S., Bavero, C., McElreath, R. L., & Beheim, B. (2020) · doi:10.31234/osf.io/4nzc7 · Cited by 13
Gonzalez, J. A. H., & Colom, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1491823 · Cited by 20
Cognitive Modeling Using Artificial Intelligence
Frank, M. C., & Goodman, N. D. (2025) · Annual Review of Psychology · doi:10.1146/annurev-psych-030625-040748 · Cited by 24
Huang, Y.-N., Jaiswal, P., Rajes, A., Yadav, A., Yu, D., Liu, F., Scheg, G., Shih, E., Boldirev, G., Nakashidze, I., et al. (2025) · Genome biology · doi:10.1186/s13059-025-03725-0 · Cited by 20
van Kampen, A. H. C., Mahamune, U., Jongejan, A., van Schaik, B. D. C., Balashova, D., Lashgari, D., Pras‐Raves, M. L., Wever, E. J. M., Dane, A. D., García-Valiente, R., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-52446-8 · Cited by 21
Ambitious Data Science Can Be Painless
Monajemi, H., Murri, R., Jonas, E., Liang, P., Stodden, V., & Donoho, D. L. (2019) · Harvard Data Science Review · doi:10.1162/99608f92.02ffc552 · Cited by 12
Self-citationGrowing the FAIR Community at the Intersection of the Geosciences and Pure and Applied Chemistry
Stall, S., McEwen, L., Wyborn, L., Hoebelheinrich, N. J., & Bruno, I. J. (2019) · Data Intelligence · doi:10.1162/dint_a_00036 · Cited by 11
Goldstein, N. D., Hamra, G. B., & Harper, S. B. (2019) · Epidemiology · doi:10.1097/ede.0000000000001149 · Cited by 11
Open Science in Developmental Science
Gennetian, L. A., Frank, M. C., & Tamis‐LeMonda, C. S. (2022) · Annual Review of Developmental Psychology · doi:10.1146/annurev-devpsych-120920-042335 · Cited by 12
Crüwell, S., Apthorp, D., Baker, B. J., Colling, L. J., Elson, M., Geiger, S. J., Lobentanzer, S., Monéger, J., Patterson, A., Schwarzkopf, D. S., et al. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/729qt · Cited by 11
Operationalizing validity of empirical software engineering studies
Härtel, J., & Lämmel, R. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10370-3 · Cited by 10
Di Lauro, S., Tursunbayeva, A., Antonelli, G., & Moschera, L. (2025) · Personnel Review · doi:10.1108/pr-11-2023-0927 · Cited by 12
Nine quick tips for open meta-analyses
Moreau, D., & Wiebels, K. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012252 · Cited by 11
Penerapan Reproducible Research pada RStudio dengan Bahasa R dan Paket Knitr
Budiaji, W. (2019) · Khazanah Informatika Jurnal Ilmu Komputer dan Informatika · doi:10.23917/khif.v5i1.7202 · Cited by 10
Open Up – the Mission Statement of the Control of Impulsive Action (Ctrl-ImpAct) Lab on Open Science
Reimer, C. B., Chen, Z., Bundt, C., Eben, C., London, R. E., & Vardanian, S. (2019) · Psychologica Belgica · doi:10.5334/pb.494 · Cited by 11
Recommendations on data sharing in HIV drug resistance research
Inzaule, S. C., Siedner, M. J., Little, S. J., Ávila‐Ríos, S., Ayitewala, A., Bosch, R. J., Cálvez, V., Ceccherini‐Silberstein, F., Charpentier, C., Descamps, D., et al. (2023) · PLoS Medicine · doi:10.1371/journal.pmed.1004293 · Cited by 9
Tier-based standards for FAIR sequence data and metadata sharing in microbiome research
Kim, L., Lavrinienko, A., Sebechlebská, Z., Stoltenberg, S., & Bokulich, N. A. (2025) · Nucleic Acids Research · doi:10.1093/nar/gkaf777 · Cited by 10
Halász, G., Schmahl, J., Negron, N., Ni, M., Lim, W. K., Atwal, G. S., Bai, Y., & Glass, D. J. (2025) · Nature Communications · doi:10.1038/s41467-025-65022-5 · Cited by 17
An Empirical Consistent Redshift Bias: A Possible Direct Observation of Zwicky’s TL Theory
Shamir, L. (2024) · Particles · doi:10.3390/particles7030041 · Cited by 9
Marwick, B. (2025) · Journal of Archaeological Science · doi:10.1016/j.jas.2025.106281 · Cited by 18
From policy to practice: progress towards data- and code-sharing in ecology and evolution
Ivimey‐Cook, E. R., Sánchez‐Tójar, A., Berberi, I., Čulina, A., Roche, D. G., Almeida, R. A., Amin, B., Bairos‐Novak, K. R., Balti, H., Bertram, M. G., et al. (2025) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2025.1394 · Cited by 22
Heather, A., Monks, T., Harper, A. L., Mustafee, N., & Mayne, A. (2025) · Journal of Simulation · doi:10.1080/17477778.2025.2552177 · Cited by 8
Data Availability of Open T-Cell Receptor Repertoire Data, a Systematic Assessment
Huang, Y.-N., Patel, N. A., Mehta, J. H., Ginjala, S., Brodin, P., Gray, C. M., Patel, Y. M., Cowell, L. G., Burkhardt, A. M., & Mangul, S. (2022) · Frontiers in Systems Biology · doi:10.3389/fsysb.2022.918792 · Cited by 7
Promoting and Enabling Reproducible Data Science Through a Reproducibility Challenge
Liu, J., Carlson, J., Josh Pasek, B., Puchala, B., Rao, A., & Jagadish, H. V. (2022) · Harvard Data Science Review · doi:10.1162/99608f92.9624ea51 · Cited by 9
Uncertainty propagation in matrix population models: Gaps, importance and guidelines
Simmonds, E. G., & Jones, O. R. (2023) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.14276 · Cited by 7
Horsburgh, J. S., Hooper, R. P., Bales, J. D., Hedstrom, M. L., Imker, H. J., Lehnert, K. A., Shanley, L., & Stall, S. (2020) · Wiley Interdisciplinary Reviews Water · doi:10.1002/wat2.1422 · Cited by 8
Schölzel, C., Blesius, V., Ernst, G., Goesmann, A., & Dominik, A. (2021) · bioRxiv · doi:10.1371/journal.pone.0254749 · Cited by 7
poliastro: a Python library for interactive astrodynamics
Rodríguez, J., & Garrido, J. (2022) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-212e5952-015 · Cited by 6
Smith, G. (2023) · doi:10.1093/oso/9780192868459.001.0001 · Cited by 6
Large-Scale Asymmetry in the Distribution of Galaxy Spin Directions—Analysis and Reproduction
Shamir, L. (2023) · Symmetry · doi:10.3390/sym15091704 · Cited by 6
Recommendations for a standardized publication protocol for a QSP model
Weddell, J. C., Gulati, A., & Yamada, A. (2024) · Journal of Pharmacokinetics and Pharmacodynamics · doi:10.1007/s10928-024-09943-6 · Cited by 6
Sharing study materials in health and medical research
DeVito, N., Morton, C. E., Cashin, A. G., Richards, G. C., & Lee, H. (2022) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2022-111987 · Cited by 7
From policy to practice: Lessons learned from an open science funding initiative
Ratan, K., McIntosh, S., Shah, H. V., Lewis, M., Dumanis, S. B., Vines, T., Schekman, R. W., & Riley, E. A. U. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011626 · Cited by 5
Portability of Scientific Workflows in NGS Data Analysis: A Case StudyPreprint
Schiefer, C., Bux, M., Brandt, J., Messerschmidt, C., Reinert, K., Beule, D., & Leser, U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03104 · Cited by 6
Reproduce to validate: A comprehensive study on the reproducibility of economics research
Herbert, S., Kingi, H., Stanchi, F., & Vilhuber, L. (2024) · Canadian Journal of Economics/Revue canadienne d économique · doi:10.1111/caje.12728 · Cited by 40
Götz, M., & O’Boyle, E. H. (2023) · Emerald Group Publishing Limited eBooks · doi:10.1108/s0742-730120230000041004 · Cited by 8
Open science practices in the false memory literature
Wiechert, S., Leistra, P., Ben‐Shakhar, G., Pertzov, Y., & Verschuère, B. (2024) · Memory · doi:10.1080/09658211.2024.2387108 · Cited by 5
Assessing computational reproducibility in Behavior Research Methods
Ellis, D. A., Towse, J. N., Brown, O., Cork, A., Davidson, B. I., Devereux, S., Hinds, J. M., Ivory, M. R., Nightingale, S. J., Parry, D. A., et al. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02501-5 · Cited by 11
YARD: A Tool for Curating Research Outputs
Peer, L., & Dull, J. (2020) · Data Science Journal · doi:10.5334/dsj-2020-028 · Cited by 5
Common-sense approaches to sharing tabular data alongside publication
Tierney, N. J., & Ram, K. (2021) · Patterns · doi:10.1016/j.patter.2021.100368 · Cited by 5
Software testing in microbial bioinformatics: a call to action
van der Putten, B. C. L., Mendes, C. I., Talbot, B. M., de Korne-Elenbaas, J., Mamede, R., Vila-Cerqueira, P., Coelho, L. P., Gulvik, C. A., Katz, L. S., & participants, T. A. N. 2. H. (2022) · Microbial Genomics · doi:10.1099/mgen.0.000790 · Cited by 5
A decentralized framework for cultivating research lifecycle transparency
Jeng, W., Wang, S.-H., Chen, H., Huang, P., Chen, Y., & Hsiao, H. (2020) · PLoS ONE · doi:10.1371/journal.pone.0241496 · Cited by 6
Open Science for Computer SimulationPreprint
Monks, T., Harper, A. L., Anagnostou, A., & Taylor, S. (2022) · doi:10.31219/osf.io/zpxtm · Cited by 5
Hamilton, D. G., Hong, K., Fraser, H. S., Rowhani-Farid, A., Fidler, F. M., & Page, M. J. (2023) · medRxiv · doi:10.1101/2023.03.22.23287607 · Cited by 5
Optimizing murine sample sizes for RNA-seq studies revealed from large-scale comparative analysisPreprint
Halász, G., Schmahl, J., Negron, N., Ni, M., Atwal, G. S., & Glass, D. J. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.07.08.602525 · Cited by 5
Reproducibility in systems biology modelling
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.08.07.239855 · Cited by 131
The computational reproducibility of articles published under the Open Data + FAIR policy of IJGIS
Kedron, P., Li, Z., & Liu, L. (2026) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2025.2603586 · Cited by 4
Cheimarios, N. (2025) · Frontiers in Physics · doi:10.3389/fphy.2025.1711356 · Cited by 4
Semantic tools for development of high-level interactive applications for supercomputers
Gorodnichev, M., & Lebedev, D. V. (2021) · The Journal of Supercomputing · doi:10.1007/s11227-021-03731-6 · Cited by 4
Laying Foundations to Quantify the “Effort of Reproducibility”
Akella, A. P., Koop, D., & Alhoori, H. (2023) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl57899.2023.00018 · Cited by 5
Bhaskar, A., & Stodden, V. (2024) · ACM-REP · doi:10.1145/3641525.3663629 · Cited by 15
Self-citation
Metadata
- Authors
- Stodden, V., Seiler, J., & Ma, Z.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Columbia University (US)
- Corresponding author
- Stodden, Victoria
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2018-03-12
- Volume
- 115
- Issue
- 11
- Pages
- 2584-2589
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC5856507
- OpenAlex
- W2793993424
- Cited by
- 369
- References
- 26
- Citations by year
- 2026: 45 · 2025: 34 · 2024: 36 · 2023: 53 · 2022: 40 · 2021: 50 · 2020: 36 · 2019: 48 · 2018: 26
- FWCI
- 64.95
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- Alfred P Sloan Foundation (PG004545)
- National Science Foundation (1153384)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- computational reproducibility, editorial policy, data availability, code availability, replication, research artifacts, scientific journals, author guidelines
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
