An empirical analysis of journal policy effectiveness for computational reproducibility

Promoting an open research culture

Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · see citing works

Reproducible research in computational science

Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · see citing works

Repeatability of published microarray gene expression analyses

Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly, I., Cui, X., Culhane, A. C., Falchi, M., Furlanello, C., Game, L., Jurman, G., et al. (2009) · Nature Genetics · doi:10.1038/ng.295 · Cited by 551 · see citing works

Replication, Replication

King, G. (1995) · PS Political Science & Politics · doi:10.1017/s1049096500057607 · Cited by 455 · see citing works

Enhancing reproducibility for computational methods

Stodden, V., Mcnutt, M. K., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 414 · see citing works

Statistical Analyses and Reproducible Research

Gentleman, R. C., & Lang, D. T. (2007) · Journal of Computational and Graphical Statistics · doi:10.1198/106186007x178663 · Cited by 364 · see citing works

Recommendations for utilizing and reporting population genetic analyses: the reproducibility of genetic clustering using the program structure

Gilbert, K. J., Andrew, R. L., Bock, D. G., Franklin, M. T., Kane, N. C., Moore, J., Moyers, B. T., Renaut, S., Rennison, D. J., Veen, T., et al. (2012) · Molecular Ecology · doi:10.1111/j.1365-294x.2012.05754.x · Cited by 290 · see citing works

Reproducible Research in Computational Harmonic Analysis

Donoho, D. L., Maleki, A., Rahman, I. U., Shahram, M., & Stodden, V. (2008) · Computing in Science & Engineering · doi:10.1109/mcse.2009.15 · Cited by 274 · see citing works

Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals

Stodden, V., Guo, P., & Ma, Z. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067111 · Cited by 257 · see citing works

Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say 'Usually Not'

Chang, A. C., & Li, P. (2015) · doi:10.2139/ssrn.2669564 · Cited by 217 · see citing works

Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say "Usually Not"

Chang, A. C., & Li, P. (2015) · Finance and Economics Discussion Series · doi:10.17016/feds.2015.083 · Cited by 216 · see citing works

Making Data Maximally Available

Hanson, B., Sugden, A. M., & Alberts, B. (2011) · Science · doi:10.1126/science.1203354 · Cited by 172 · see citing works

The Legal Framework for Reproducible Scientific Research: Licensing and Copyright

Stodden, V. (2008) · Computing in Science & Engineering · doi:10.1109/mcse.2009.19 · Cited by 153 · see citing works

A synthetic dataset primer for the biobehavioural sciences to promote reproducibility and hypothesis generation

Quintana, D. S. (2020) · eLife · doi:10.7554/elife.53275 · Cited by 142 · see citing works

Privacy, Big Data, and the Public Good

(2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781107590205 · Cited by 138 · see citing works

Computing environments for reproducibility: Capturing the “Whole Tale”

Brinckman, A., Chard, K., Gaffney, N. I., Hategan, M., Jones, M. B., Kowalik, K., Kulasekaran, S., Ludäscher, B., Mecum, B., Nabrzyski, J., et al. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2017.12.029 · Cited by 134 · see citing works

A primer on provenance

Carata, L., Akoush, S., Balakrishnan, N., Bytheway, T., Sohan, R., Seltzer, M., & Hopper, A. (2014) · Communications of the ACM · doi:10.1145/2596628 · Cited by 85 · see citing works

Creating and sharing reproducible research code the workflowr way

Blischak, J., Carbonetto, P. S., & Stephens, M. (2019) · F1000Research · doi:10.12688/f1000research.20843.1 · Cited by 76 · see citing works

Definitions of dataset in the scientific and technical literature

Renear, A. H., Sacchi, S., & Wickett, K. M. (2010) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.14504701240 · Cited by 65 · see citing works

Certificates of Confidentiality: Protecting Human Subject Research Data in Law and Practice

Wolf, L. E., Patel, M. J., Tarver, B. A. W., Austin, J. L., Dame, L. A., & Beskow, L. M. (2015) · The Journal of Law Medicine & Ethics · doi:10.1111/jlme.12302 · Cited by 41 · see citing works

Enabling Reproducibility in Big Data Research: Balancing Confidentiality and Scientific Transparency

Stodden, V. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781107590205.007 · Cited by 26 · see citing works

Open Science is for Aging Research, Too

Isaacowitz, D. M., & Lind, M. (2019) · Innovation in Aging · doi:10.1093/geroni/igz028 · Cited by 25 · see citing works

Differential Privacy: A Cryptographic Approach to Private Data Analysis

Dwork, C. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781107590205.018 · Cited by 20 · see citing works

A Primer on Provenance

Carata, L., Akoush, S., Balakrishnan, N., Bytheway, T., Sohan, R., Seltzer, M., & Hopper, A. (2014) · Queue · doi:10.1145/2602649.2602651 · Cited by 17 · see citing works

Reproducing Epidemiologic Research and Ensuring Transparency

Coughlin, S. S. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx065 · Cited by 10 · see citing works

Opinion: set the default to "open"

Bailey, D. H., Borwein, J. M., & Stodden, V. (2013) · Notices of the American Mathematical Society · Cited by 9 · see citing works

Perturbation Methods for Protecting Numerical Data: Evolution and Evaluation

Sarathy, R., & Muralidhar, K. (2012) · Handbook of statistics · doi:10.1016/b978-0-44-451875-0.00019-1 · Cited by 7 · see citing works

Leakage and the reproducibility crisis in machine-learning-based science

Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387 · see citing works

Preferred reporting items for systematic reviews and meta‐analyses in ecology and evolutionary biology: a PRISMA extension

O’Dea, R. E., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Gurevitch, J., Page, M. J., Stewart, G. B., Moher, D., et al. (2021) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12721 · Cited by 771 · see citing works

The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support

Razavi, S., Jakeman, A. J., Saltelli, A., Prieur, C., Iooss, B., Borgonovo, E., Plischke, E., Piano, S. L., Iwanaga, T., Becker, W., et al. (2020) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2020.104954 · Cited by 651 · see citing works

Modules for Experiments in Stellar Astrophysics (MESA): Time-dependent Convection, Energy Conservation, Automatic Differentiation, and Infrastructure

Jermyn, A. S., Bauer, E. B., Schwab, J., Farmer, R., Ball, W. H., Bellinger, E. P., Dotter, A., Joyce, M., Marchant, P., Mombarg, J. S. G., et al. (2023) · The Astrophysical Journal Supplement Series · doi:10.3847/1538-4365/acae8d · Cited by 604 · see citing works

Investigating the replicability of preclinical cancer biology

Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 393 · see citing works

Binder 2.0 - Reproducible, interactive, sharable environments for science at scale

Project Jupyter, Bussonnier, M., Forde, J. Z., Freeman, J., Granger, B., Head, T., Holdgraf, C., Kelley, K., Nalvarte, G., Osheroff, A., et al. (2018) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-4af1f417-011 · Cited by 303 · see citing works

Bringing Molecular Dynamics Simulation Data into View

Hildebrand, P. W., Rose, A., & Tiemann, J. K. S. (2019) · Trends in Biochemical Sciences · doi:10.1016/j.tibs.2019.06.004 · Cited by 303 · see citing works

The Climate Data Toolbox for MATLAB

Greene, C. A., Thirumalai, K., Kearney, K. A., Delgado, J. M., Schwanghart, W., Wolfenbarger, N. S., Thyng, K. M., Gwyther, D. E., Gardner, A., & Blankenship, D. D. (2019) · Geochemistry Geophysics Geosystems · doi:10.1029/2019gc008392 · Cited by 295 · see citing works

Identifying and Overcoming Threats to Reproducibility, Replicability, Robustness, and Generalizability in Microbiome Research

Schloss, P. D. (2018) · mBio · doi:10.1128/mbio.00525-18 · Cited by 275 · see citing works

A Practical Guide for Transparency in Psychological Science

Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · Collabra Psychology · doi:10.1525/collabra.158 · Cited by 233 · see citing works

Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers

Wratten, L., Wilm, A., & Göke, J. (2021) · Nature Methods · doi:10.1038/s41592-021-01254-9 · Cited by 228 · see citing works

Open science, reproducibility, and transparency in ecology

Powers, S. M., & Hampton, S. E. (2018) · Ecological Applications · doi:10.1002/eap.1822 · Cited by 224 · see citing works

Open Science principles for accelerating trait-based science across the Tree of Life

Gallagher, R. V., Falster, D. S., Maitner, B., Salguero‐Gómez, R., Vandvik, V., Pearse, W. D., Schneider, F. D., Kattge, J., Poelen, J. H., Madin, J. S., et al. (2020) · Nature Ecology & Evolution · doi:10.1038/s41559-020-1109-6 · Cited by 218 · see citing works

Eight grand challenges in socio-environmental systems modeling

Elsawah, S., Filatova, T., Jakeman, A. J., Kettner, A. J., Zellner, M. L., Athanasiadis, I. N., Hamilton, S. H., Axtell, R. L., Brown, D. G., Gilligan, J. M., et al. (2020) · Socio-Environmental Systems Modelling · doi:10.18174/sesmo.2020a16226 · Cited by 205 · see citing works

On the prevalence of uninformative parameters in statistical models applying model selection in applied ecology

Leroux, S. (2019) · PLoS ONE · doi:10.1371/journal.pone.0206711 · Cited by 192 · see citing works

Journal of Computational Neuroscience

(2013) · Journal of Computational Neuroscience · doi:10.1007/10827.1573-6873 · Cited by 190 · see citing works

Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017)

Hardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620979806 · Cited by 187 · see citing works

Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development

Scheuerman, M. K., Hanna, A., & Denton, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476058 · Cited by 216 · see citing works

Systematic benchmarking of omics computational tools

Mangul, S., Martin, L. S., Hill, B. L., Lam, A. K.-M., Distler, M. G., Zelikovsky, A., Eskin, E., & Jonathan Flint, C. (2019) · Nature Communications · doi:10.1038/s41467-019-09406-4 · Cited by 168 · see citing works

Assessing data availability and research reproducibility in hydrology and water resources

Stagge, J., Rosenberg, D., Abdallah, A., Akbar, H., Attallah, N. A., & James, R. (2019) · Scientific Data · doi:10.1038/sdata.2019.30 · Cited by 119 · see citing works

Why don't we share data and code? Perceived barriers and benefits to public archiving practices

Gomes, D. G. E., Pottier, P., Crystal‐Ornelas, R., Hudgins, E. J., Foroughirad, V., Sánchez‐Reyes, L. L., Turba, R., Martinez, P. A., Moreau, D., Bertram, M. G., et al. (2022) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2022.1113 · Cited by 174 · see citing works

Recommendations to enhance rigor and reproducibility in biomedical research

Brito, J., Li, J. Z., Moore, J. H., Greene, C. S., Nogoy, N., Garmire, L. X., & Mangul, S. (2020) · GigaScience · doi:10.1093/gigascience/giaa056 · Cited by 154 · see citing works

To Sobol or not to Sobol? The effects of sampling schemes in systems biology applications

Renardy, M., Joslyn, L. R., Millar, J. A., & Kirschner, D. E. (2021) · Mathematical Biosciences · doi:10.1016/j.mbs.2021.108593 · Cited by 154 · see citing works

From Open Data to Open Science

Ramachandran, R., Bugbee, K., & Murphy, K. (2021) · Earth and Space Science · doi:10.1029/2020ea001562 · Cited by 150 · see citing works

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Kapoor, S., Cantrell, E. M., Peng, K., Pham, T. H., Bail, C. A., Gundersen, O. E., Hofman, J. M., Hullman, J. R., Lones, M. A., Malik, M. M., et al. (2024) · Science Advances · doi:10.1126/sciadv.adk3452 · Cited by 192 · see citing works

Cross-Linguistic Data Formats, advancing data sharing and re-use in comparative linguistics

Forkel, R., List, J., Greenhill, S. J., Rzymski, C., Bank, S., Cysouw, M., Hammarström, H., Haspelmath, M., Kaiping, G. A., & Gray, R. D. (2018) · Scientific Data · doi:10.1038/sdata.2018.205 · Cited by 147 · see citing works

A synthetic dataset primer for the biobehavioural sciences to promote reproducibility and hypothesis generation

Quintana, D. S. (2020) · eLife · doi:10.7554/elife.53275 · Cited by 142 · see citing works

Reproducibility in systems biology modelling

Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2021) · Molecular Systems Biology · doi:10.15252/msb.20209982 · Cited by 131 · see citing works

A large-scale study on research code quality and execution

Trišović, A., Lau, M. K., Pasquier, T., & Crosas, M. (2022) · Scientific Data · doi:10.1038/s41597-022-01143-6 · Cited by 136 · see citing works

Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Schweinsberg, M., Feldman, M., Staub, N., van den Akker, O. R., van Aert, R. C. M., van Assen, M. A. L. M., Liu, Y., Althoff, T., Heer, J., Kale, A. M., et al. (2021) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2021.02.003 · Cited by 129 · see citing works

Analysis of Open Data and Computational Reproducibility in Registered Reports in Psychology

Obels, P., Lakens, D., Coles, N. A., Gottfried, J., & Green, S. A. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920918872 · Cited by 127 · see citing works

Low availability of code in ecology: A call for urgent action

Čulina, A., van den Berg, I., Evans, S. R., & Sánchez‐Tójar, A. (2020) · PLoS Biology · doi:10.1371/journal.pbio.3000763 · Cited by 112 · see citing works

Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail

Miłkowski, M., Hensel, W. M., & Hohol, M. (2018) · Journal of Computational Neuroscience · doi:10.1007/s10827-018-0702-z · Cited by 112 · see citing works

Reproducibility and replicability: opportunities and challenges for geospatial research

Kedron, P., Li, W., Fotheringham, S., & Goodchild, M. (2021) · International Journal of Geographical Information Science · doi:10.1080/13658816.2020.1802032 · Cited by 108 · see citing works

Analytic reproducibility in articles receiving open data badges at the journal <i>Psychological Science</i> : an observational study

Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021) · Royal Society Open Science · doi:10.1098/rsos.201494 · Cited by 108 · see citing works

Calibrating the Scientific Ecosystem Through Meta-Research

Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98 · see citing works

40-Years of Lake Urmia Restoration Research: Synthesis and Next Steps

Parsinejad, M., Rosenberg, D. E., Ghale, Y. A. G., Khazaei, B., Null, S. E., Raja, O., Safaie, A., Sima, S., Sorooshian, A., & Wurtsbaugh, W. A. (2021) · Digital Commons - USU (Utah State University) · doi:10.1016/j.scitotenv.2022.155055

Challenges and recommendations to improve the installability and archival stability of omics computational tools

Mangul, S., Mosqueiro, T., Abdill, R. J., Duong, D., Mitchell, K., Sarwal, V., Hill, B., Brito, J., Littman, R. J., Statz, B., et al. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000333 · Cited by 90 · see citing works

Prevalence and predictors of data and code sharing in the medical and health sciences: systematic review with meta-analysis of individual participant data

Hamilton, D. G., Hong, K., Fraser, H., Rowhani-Farid, A., Fidler, F., & Page, M. J. (2023) · BMJ · doi:10.1136/bmj-2023-075767 · Cited by 86 · see citing works

Available and missing data to model impact of climate change on European forests

Ruiz‐Benito, P., Vacchiano, G., Lines, E. R., Reyer, C., Ratcliffe, S., Morin, X., Hartig, F., Mäkelä, A., Yousefpour, R., Chaves, J. E., et al. (2019) · Ecological Modelling · doi:10.1016/j.ecolmodel.2019.108870 · Cited by 89 · see citing works

Reproducibility in Neuroimaging Analysis: Challenges and Solutions

Botvinik‐Nezer, R., & Wager, T. D. (2022) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2022.12.006 · Cited by 87 · see citing works

Experimenting with reproducibility: a case study of robustness in bioinformatics

Kim, Y.-M., Poline, J., & Dumas, G. (2018) · GigaScience · doi:10.1093/gigascience/giy077 · Cited by 74 · see citing works

Creating and sharing reproducible research code the workflowr way

Blischak, J., Carbonetto, P. S., & Stephens, M. (2019) · F1000Research · doi:10.12688/f1000research.20843.1 · Cited by 76 · see citing works

Practical Reproducibility in Geography and Geosciences

Nüst, D., & Pebesma, E. J. (2020) · Annals of the American Association of Geographers · doi:10.1080/24694452.2020.1806028 · Cited by 73 · see citing works

New trends in bibliometric APIs: A comparative analysis

Vélez-Estévez, A., Pérez, I. J., García‐Sánchez, P., Moral-Muñoz, J. A., & Cobo, M. J. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103385 · Cited by 84 · see citing works

Heterogeneity in effect size estimates

Holzmeister, F., Johannesson, M., BOHM, R. M., Dreber, A., Huber, J., & Kirchler, M. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2403490121 · Cited by 82 · see citing works

Improving the usability and archival stability of bioinformatics software

Mangul, S., Martin, L. S., Eskin, E., & Blekhman, R. (2019) · Genome biology · doi:10.1186/s13059-019-1649-8 · Cited by 71 · see citing works

Computational Reproducibility via Containers in Psychology

Clyburne-Sherin, A., Fei, X., & Green, S. A. (2019) · Meta-Psychology · doi:10.15626/mp.2018.892 · Cited by 69 · see citing works

Microbiome Metadata Standards: Report of the National Microbiome Data Collaborative’s Workshop and Follow-On Activities

Vangay, P., Burgin, J., Johnston, A., Beck, K. L., Berrios, D. C., Blumberg, K., Canon, R. S., Chain, P., Chandonia, J., Christianson, D., et al. (2021) · mSystems · doi:10.1128/msystems.01194-20 · Cited by 68 · see citing works

Developing image analysis methods for digital pathology

Bankhead, P. (2022) · The Journal of Pathology · doi:10.1002/path.5921 · Cited by 68 · see citing works

Empirical evidence of widespread exaggeration bias and selective reporting in ecology

Kimmel, K. A., Avolio, M. L., & Ferraro, P. J. (2023) · Nature Ecology & Evolution · doi:10.1038/s41559-023-02144-3 · Cited by 73 · see citing works

Smart Contracts for Research Data Rights Management over the Ethereum Blockchain Network

Pãnescu, A.-T., & Manta, V. (2018) · Science & Technology Libraries · doi:10.1080/0194262x.2018.1474838 · Cited by 65 · see citing works

Addressing barriers in FAIR data practices for biomedical data

Hughes, L. D., Tsueng, G., DiGiovanna, J., Horvath, T. D., Rasmussen, L. V., Savidge, T., Stoeger, T., Turkarslan, S., Wu, Q., Wu, C., et al. (2023) · Scientific Data · doi:10.1038/s41597-023-01969-8 · Cited by 65 · see citing works

Reproducible image handling and analysis

Miura, K., & Nørrelykke, S. F. (2021) · The EMBO Journal · doi:10.15252/embj.2020105889 · Cited by 57 · see citing works

A funder-imposed data publication requirement seldom inspired data sharing

Couture, J. L., Blake, R. E., McDonald, G. G., & Ward, C. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199789 · Cited by 60 · see citing works

On code sharing and model documentation of published individual and agent-based models

Janssen, M. A., Pritchard, C., & Lee, A. (2020) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2020.104873 · Cited by 57 · see citing works

Share the code, not just the data: A case study of the reproducibility of articles published in the Journal of Memory and Language under the open data policy

Laurinavichyute, A., Yadav, H., & Vasishth, S. (2022) · Journal of Memory and Language · doi:10.1016/j.jml.2022.104332 · Cited by 60 · see citing works

Examining the generalizability of research findings from archival data

Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120377119 · Cited by 56 · see citing works

Tellurium notebooks—An environment for reproducible dynamical modeling in systems biology

Medley, J., Choi, K., König, M., Smith, L. P., Gu, S., Hellerstein, J., Sealfon, S., & Sauro, H. (2018) · PLoS Comput. Biol. · doi:10.1371/journal.pcbi.1006220 · Cited by 58 · see citing works

Publication reform to safeguard wildlife from researcher harm

Field, K. A., Paquet, P. C., Artelle, K. A., Proulx, G., Brook, R. K., & Darimont, C. T. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000193 · Cited by 55 · see citing works

What’s in a Badge? A Computational Reproducibility Investigation of the Open Data Badge Policy in One Issue of <i>Psychological Science</i>

Crüwell, S., Apthorp, D., Baker, B. J., Colling, L., Elson, M., Geiger, S. J., Lobentanzer, S., Monéger, J., Patterson, A., Schwarzkopf, D. S., et al. (2023) · Psychological Science · doi:10.1177/09567976221140828 · Cited by 54 · see citing works

Reproducible Analysis Pipeline for Data Streams: Open-Source Software to Process Data Collected With Mobile Devices

Vega, J., Li, M., Aguillera, K., Goel, N., Joshi, E., Khandekar, K., Durica, K. C., Kunta, A. R., & Low, C. A. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.769823 · Cited by 54 · see citing works

Limited engagement with transparent and open science standards in the policies of pain journals: a cross-sectional evaluation

Cashin, A. G., Bagg, M. K., Richards, G. C., Toomey, E. C., McAuley, J. H., & Lee, H. (2020) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2019-111296 · Cited by 51 · see citing works

Enforcing public data archiving policies in academic publishing: A study of ecology journals

Sholler, D., Ram, K., Boettiger, C., & Katz, D. S. (2019) · Big Data & Society · doi:10.1177/2053951719836258 · Cited by 53 · see citing works

Successes and Struggles with Computational Reproducibility: Lessons from the Fragile Families Challenge

Liu, D. M., & Salganik, M. J. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023119849803 · Cited by 47 · see citing works

The New Common

Aarts, E., Fleuren, H. A., Sitskoorn, M. M., & Wilthagen, T. (2021) · Springer eBooks · doi:10.1007/978-3-030-65355-2 · Cited by 51 · see citing works

Neuroscience Cloud Analysis As a Service: An open-source platform for scalable, reproducible data analysis

Abe, T., Kinsella, I., Saxena, S., Buchanan, E. K., Couto, J., Briggs, J., Kitt, S. L., Glassman, R., Zhou, J., Paninski, L., et al. (2022) · Neuron · doi:10.1016/j.neuron.2022.06.018 · Cited by 51 · see citing works

Principles for data analysis workflows

Stoudt, S., Vásquez, V., & Martinez, C. C. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008770 · Cited by 53 · see citing works

Reproducibility of Quantitative Systems Pharmacology Models: Current Challenges and Future Opportunities

Kirouac, D. C., Cicali, B., & Schmidt, S. (2019) · CPT: Pharmacometrics &amp; Systems Pharmacology · doi:10.1002/psp4.12390 · Cited by 46 · see citing works

The Brazilian Reproducibility Initiative

Amaral, O. B., Neves, K., Wasilewska‐Sampaio, A. P., & Carneiro, C. F. D. (2019) · eLife · doi:10.7554/elife.41602 · Cited by 56 · see citing works

Reproducibility of research: Issues and proposed remedies

Allison, D. B., Shiffrin, R. M., & Stodden, V. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1802324115 · Cited by 48 · see citing works

Learning from reproducing computational results: introducing three principles and the Reproduction Package

Krafczyk, M. S., Shi, A., Bhaskar, A., Marinov, D., & Stodden, V. (2021) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2020.0069 · Cited by 46 · see citing works

Reproducibility in Evolutionary Computation

López‐Ibáñez, M., Branke, J., & Paquete, L. (2021) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3466624 · Cited by 45 · see citing works

(Why) Are Open Research Practices the Future for the Study of Language Learning?

Marsden, E. J., & Morgan‐Short, K. (2023) · Language Learning · doi:10.1111/lang.12568 · Cited by 49 · see citing works

The PLOS ONE collection on machine learning in health and biomedicine: Towards open code and open data

Celi, L. A., Citi, L., Ghassemi, M., & Pollard, T. (2019) · PLoS ONE · doi:10.1371/journal.pone.0210232 · Cited by 45 · see citing works

Recovering data from summary statistics: Sample Parameter Reconstruction via Iterative TEchniques (SPRITE)

Heathers, J. A., Anaya, J., van der Zee, T., & Brown, N. J. (2018) · PeerJ Preprints · doi:10.7287/peerj.preprints.26968v1 · Cited by 44 · see citing works

Reproducibility and Replicability in Economics

Vilhuber, L. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.4f6b9e67 · Cited by 43 · see citing works

The case for free and open source software in research and scholarship

Fortunato, L., & Galassi, M. C. (2021) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2020.0079 · Cited by 47 · see citing works

pyCSEP: A Python Toolkit for Earthquake Forecast Developers

Savran, W. H., Bayona, J. A., Iturrieta, P. C., Asim, K. M., Bao, H., Bayliss, K., Herrmann, M., Schorlemmer, D., Maechling, P. J., & Werner, M. J. (2022) · Seismological Research Letters · doi:10.1785/0220220033 · Cited by 40 · see citing works

Certify reproducibility with confidential data

Pérignon, C., Gadouche, K., Hurlin, C., Silberman, R., & Debonnel, E. (2019) · Science · doi:10.1126/science.aaw2825 · Cited by 40 · see citing works

Enhancing Climate Change Research With Open Science

Tai, T. C., & Robinson, J. P. W. (2018) · Frontiers in Environmental Science · doi:10.3389/fenvs.2018.00115 · Cited by 40 · see citing works

Enhancing Data Integration, Interoperability, and Reuse to Address Complex and Emerging Environmental Health Problems

Heacock, M. L., Lopez, A. R., Amolegbe, S. M., Carlin, D. J., Henry, H. F., Trottier, B. A., Velasco, M. L., & Suk, W. A. (2022) · Environmental Science & Technology · doi:10.1021/acs.est.1c08383 · Cited by 37 · see citing works

A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses

Seibold, H., Czerny, S., Decke, S., Dieterle, R., Eder, T., Fohr, S., Hahn, N., Hartmann, R., Heindl, C., Kopper, P., et al. (2021) · PLoS ONE · doi:10.1371/journal.pone.0251194 · Cited by 37 · see citing works

BCPT perspectives on studies involving natural products, traditional Chinese medicine and systems pharmacology

Tveden‐Nyborg, P., Yang, B., Simonsen, U., & Lykkesfeldt, J. (2024) · Basic & Clinical Pharmacology & Toxicology · doi:10.1111/bcpt.14109 · Cited by 36 · see citing works

Journal data policies: Exploring how the understanding of editors and authors corresponds to the policies themselves

Christian, T.-M. L., Gooch, A., Vision, T., & Hull, E. (2020) · PLoS ONE · doi:10.1371/journal.pone.0230281 · Cited by 36 · see citing works

Reproducibility and Replicability in the Context of the Contested Identities of Geography

Sui, D. Z., & Kedron, P. (2020) · Annals of the American Association of Geographers · doi:10.1080/24694452.2020.1806024 · Cited by 36 · see citing works

A practical guide for transparency in psychological sciencePreprint

Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · doi:10.31234/osf.io/rtygm · Cited by 46 · see citing works

Open science practices in clinical psychology journals: An audit study.

Nutu, D., Gentili, C., Naudet, F., & Cristea, I. A. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000414 · Cited by 35 · see citing works

WordSeg: Standardizing unsupervised word form segmentation from text

Bernard, M., Thiollière, R., Saksida, A., Loukatou, G. R., Larsen, E., Johnson, M., Fibla, L., Emmanuel Dupoux, A., Daland, R., Cao, X., et al. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01223-3 · Cited by 35 · see citing works

Improving norms in research culture to incentivize transparency and rigor

Mellor, D. T. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1902329 · Cited by 36 · see citing works

Data policies of highly-ranked social science journalsPreprint

Crosas, M., Gautier, J., Karcher, S., Kirilova, D., Otalora, G., & Schwartz, A. (2018) · doi:10.31235/osf.io/9h7ay · Cited by 35 · see citing works

Long-term availability of data associated with articles in PLOS ONE

Federer, L. (2022) · PLoS ONE · doi:10.1371/journal.pone.0272845 · Cited by 35 · see citing works

Integrating multiple evidence streams to understand insect biodiversity change

Cooke, R., Outhwaite, C. L., Bladon, A. J., Millard, J. W., Rodger, J. G., Dong, Z., Dyer, E. E., Edney, S., Murphy, J. F., Dicks, L. V., et al. (2025) · Science · doi:10.1126/science.adq2110 · Cited by 34 · see citing works

The Next Frontier: Making Research More Reproducible

Rosenberg, D. E., Filion, Y., Teasley, R. L., Sandoval-Solís, S., Hecht, J. S., van Zyl, J. E., McMahon, G. F., Horsburgh, J. S., Kasprzyk, J. R., & Tarboton, D. G. (2020) · Journal of Water Resources Planning and Management · doi:10.1061/(asce)wr.1943-5452.0001215 · Cited by 33 · see citing works

Relational data paradigms: What do we learn by taking the materiality of databases seriously?

Thomer, A. K., & Wickett, K. M. (2020) · Big Data & Society · doi:10.1177/2053951720934838 · Cited by 32 · see citing works

Paths towards greater consensus building in experimental biology

Roche, D. G., Raby, G. D., Norin, T., Ern, R., Scheuffele, H., Skeeles, M. R., Morgan, R., Andreassen, A. H., Clements, J. C., Louissaint, S., et al. (2022) · Journal of Experimental Biology · doi:10.1242/jeb.243559 · Cited by 33 · see citing works

Accessible molecular phylogenomics at no cost: obtaining 14 new mitogenomes for the ant subfamily Pseudomyrmecinae from public data

Vieira, G. A., & Prosdocimi, F. (2019) · PeerJ · doi:10.7717/peerj.6271 · Cited by 31 · see citing works

Estimating the reproducibility of social learning research published between 1955 and 2018

Minocher, R., Atmaca, S., Bavero, C., McElreath, R. L., & Beheim, B. (2021) · Royal Society Open Science · doi:10.1098/rsos.210450 · Cited by 38 · see citing works

Why don’t we share data and code? Perceived barriers and benefits to public archiving practicesPreprint

Gomes, D. G. E., Pottier, P., Crystal‐Ornelas, R., Hudgins, E. J., Foroughirad, V., Sánchez‐Reyes, L. L., Turba, R., Martinez, P. A., Moreau, D., Bertram, M. G., et al. (2022) · doi:10.31222/osf.io/gaj43 · Cited by 29 · see citing works

How Reproducible Are QM/MM Simulations? Lessons from Computational Studies of the Covalent Inhibition of the SARS-CoV-2 Main Protease by Carmofur

Giudetti, G., Polyakov, I. V., Grigorenko, B. L., Faraji, S., Nemukhin, A. V., & Krylov, A. I. (2022) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.2c00286 · Cited by 28 · see citing works

Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals

Stodden, V., Guo, P., & Ma, Z. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067111 · Cited by 257 · see citing works

Reproduce to validate: A comprehensive study on the reproducibility of economics research

Herbert, S., Kingi, H., Stanchi, F., & Vilhuber, L. (2024) · Canadian Journal of Economics/Revue canadienne d économique · doi:10.1111/caje.12728 · Cited by 40 · see citing works

Reproducibility in Management Science

Fišar, M., Greiner, B., Huber, C., Katok, E., Ozkes, A., Abraham, D., Adams, G., Adbi, A., Addoum, J., Adena, M., et al. (2024) · Apollo (University of Cambridge) · doi:10.17863/cam.125367

The Reproducibility of Economics Research: A Case StudyPreprint

Herbert, S., Kingi, H., Stanchi, F., & Vilhuber, L. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4325149 · Cited by 35 · see citing works

Verification Reports: A new article type at Cortex

Chambers, C. (2020) · Cortex · doi:10.1016/j.cortex.2020.04.020 · Cited by 34 · see citing works