Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Journal policy on research data and code availability is an important part of the ongoing shift toward publishing reproducible computational science. This article extends the literature by studying journal data sharing policies by year (for both 2011 and 2012) for a referent set of 170 journals. We make a further contribution by evaluating code sharing policies, supplemental materials policies, and open access status for these 170 journals for each of 2011 and 2012. We build a predictive model of open data and code policy adoption as a function of impact factor and publisher and find higher impact journals more likely to have open data and code policies and scientific societies more likely to have open data and code policies than commercial publishers. We also find open data policies tend to lead open code policies, and we find no relationship between open data and code policies and either supplemental material policies or open access journal status. Of the journals in this study, 38% had a data policy, 22% had a code policy, and 66% had a supplemental materials policy as of June 2012. This reflects a striking one year increase of 16% in the number of data policies, a 30% increase in code policies, and a 7% increase in the number of supplemental materials policies. We introduce a new dataset to the community that categorizes data and code sharing, supplemental materials, and open access policies in 2011 and 2012 for these 170 journals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Policies that mandate sharing of analysis scripts have already been implemented at some journals, for example in economics and political science [19,45,46].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“An increasing number of publishers and funding bodies require authors or grant holders to make their data publicly available [247].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [247]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Minimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
King, G. (1995) · PS Political Science & Politics · doi:10.1017/s1049096500057607 · Cited by 436
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424 · cited 1×
“…has focused on changes needed in scientific publishing and the role of journal open data requirements in fostering scientific reliability [4,5,6,7] but little research has focused on availability of the code needed to replicate computational findings, although this has been associated with…”
BackgroundScientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar-Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 368
Evolution of Translational Omics: Lessons Learned and the Path Forward
Micheel, C., Nass, S. J., & Omenn, G. S. (2013) · Cited by 365
Public Availability of Published Research Data in High-Impact Journals
Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., & Ioannidis, J. P. A. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024357 · Cited by 363 · cited 1×
“…attention has focused on changes needed in scientific publishing and the role of journal open data requirements in fostering scientific reliability [4,5,6,7] but little research has focused on availability of the code needed to replicate computational findings, although this has been associated…”
BackgroundPublish your computer code: it is good enough
Barnes, N. (2010) · Nature · doi:10.1038/467753a · Cited by 315
Accessible Reproducible Research
Mesirov, J. P. (2010) · Science · doi:10.1126/science.1179653 · Cited by 307
Reproducible Research in Computational Harmonic Analysis
Donoho, D. L., Maleki, A., Rahman, I. U., Shahram, M., & Stodden, V. (2009) · Computing in Science & Engineering · doi:10.1109/mcse.2009.15 · Cited by 274
Self-citation
Research works printed but not linked: 2
- Reproducible Research. Special Issue, Computing in Science and Engineering 14: 4 11–56. http://ieeexplore.ieee.org/xpl/tocresult.jsp?reload=true&isnumber=6241356&punumber=5992 (last accessed May 8, 2013).
- Editorial (1998) The gatekeepers. Nature Structural Biology 5: 3 164. http://www.nature.com/nsmb/wilma/v5n3.892130820.html (last accessed May 8, 2013).
Cited by
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035
How Open Science Helps Researchers Succeed
McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., McDougall, D., Nosek, B. A., Ram, K., Soderberg, C. K., et al. (2016) · eLife · doi:10.7554/elife.16800 · Cited by 798
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
“Sharing of data, protocols, materials, and software has been promoted in several -omics fields, creating a substrate for reproducible data practices [28–31].”
BackgroundDonoho, D. (2017) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2017.1384734 · Cited by 695
Reporting animal research: Explanation and Elaboration for the ARRIVE guidelines 2019
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/703355 · Cited by 559
Enhancing reproducibility for computational methods
Stodden, V., McNutt, M., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 411
Self-citationITK: enabling reproducible research and open science
McCormick, M., Liu, X., Jomier, J., Marion, C., & Ibanez, L. (2014) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2014.00013 · Cited by 376
Park, J. E., Park, S. Y., Kim, H. J., & Kim, H. S. (2019) · Korean Journal of Radiology · doi:10.3348/kjr.2018.0070 · Cited by 388
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369
Self-citationChallenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 267
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336
State of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293
Marwick, B. (2017) · Journal of Archaeological Method and Theory · doi:10.1007/s10816-015-9272-9 · Cited by 179
Open science, reproducibility, and transparency in ecology
Powers, S. M., & Hampton, S. E. (2019) · Ecological Applications · doi:10.1002/eap.1822 · Cited by 224
Reproducibility and replicability of rodent phenotyping in preclinical studies
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R., Golani, I., Gomez-Marin, A., Heller, R., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.01.003 · Cited by 215
Self-citationCohen-Boulakia, S., Belhajjame, K., Collin, O., Chopard, J., Froidevaux, C., Gaignard, A., Hinsen, K., Larmande, P., Bras, Y. L., Lemoine, F., et al. (2017) · Future Generation Computer Systems · doi:10.1016/j.future.2017.01.012 · Cited by 181
Gil, Y., David, C. H., Demir, I., Essawy, B. T., Fulweiler, R. W., Goodall, J. L., Karlstrom, L., Lee, H., Mills, H. J., Oh, J., et al. (2016) · Earth and Space Science · doi:10.1002/2015ea000136 · Cited by 180
Springate, D. A., Kontopantelis, E., Ashcroft, D. M., Olier, I., Parisi, R., Chamapiwa, E., & Reeves, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0099825 · Cited by 179
Big Data and Biomedical Informatics: A Challenging Opportunity
Bellazzi, R. (2014) · Yearbook of Medical Informatics · doi:10.15265/iy-2014-0024 · Cited by 184
Tools and techniques for computational reproducibility
Piccolo, S. R., & Frampton, M. B. (2016) · GigaScience · doi:10.1186/s13742-016-0135-4 · Cited by 169
Stodden, V., & Miguez, S. (2014) · Journal of Open Research Software · doi:10.5334/jors.ay · Cited by 139
Self-citationPromises and pitfalls of data sharing in qualitative research
Tsai, A. C., Kohrt, B. A., Matthews, L. T., Betancourt, T. S., Lee, J. K., Papachristos, A. V., Weiser, S. D., & Dworkin, S. L. (2016) · Social Science & Medicine · doi:10.1016/j.socscimed.2016.08.004 · Cited by 132
Reproducible and reusable research: are journal data sharing policies meeting the mark?
Vasilevsky, N. A., Minnier, J., Haendel, M. A., & Champieux, R. E. (2017) · PeerJ · doi:10.7717/peerj.3208 · Cited by 165
Howison, J., & Bullard, J. (2016) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23538 · Cited by 159
Computing environments for reproducibility: Capturing the “Whole Tale”
Brinckman, A., Chard, K., Gaffney, N., Hategan, M., Jones, M. B., Kowalik, K., Kulasekaran, S., Ludäscher, B., Mecum, B. D., Nabrzyski, J., et al. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2017.12.029 · Cited by 132
Self-citationRethomics: An R framework to analyse high-throughput behavioural data
Geissmann, Q., Garcia Rodriguez, L., Beckwith, E. J., & Gilestro, G. F. (2019) · PLOS ONE · doi:10.1371/journal.pone.0209331 · Cited by 149
Low availability of code in ecology: A call for urgent action
Culina, A., van den Berg, I., Evans, S., & Sánchez-Tójar, A. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000763 · Cited by 118
How to make more published research true
Ioannidis, J. P. A. (2015) · Acimed · Cited by 138
Better Software, Better Research
Goble, C. (2014) · IEEE Internet Computing · doi:10.1109/mic.2014.88 · Cited by 121
Naldi, A., Hernandez, C., Levy, N., Stoll, G., Monteiro, P. T., Chaouiya, C., Helikar, T., Zinovyev, A. Y., Calzone, L., Cohen‐Boulakia, S., et al. (2018) · Frontiers in Physiology · doi:10.3389/fphys.2018.00680 · Cited by 114
Marwick, B., Barton, C. M., Bates, L., Bollwerk, E., Bocinsky, R. K., Carter, A. K., Conrad, C., Costa, S., Crema, E. R., Davies, B., et al. (2017) · doi:10.31235/osf.io/72n8g · Cited by 17
A Beginner's Guide to Conducting Reproducible Research
Alston, J. M., & Rick, J. A. (2021) · Bulletin of the Ecological Society of America · doi:10.1002/bes2.1801 · Cited by 99
Elevating The Status of Code in Ecology
Mislan, K. A. S., Heer, J. M., & White, E. P. (2016) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2015.11.006 · Cited by 87
Agasisti, T., & Bowers, A. J. (2017) · Handbook of Contemporary Education Economics · doi:10.4337/9781785369070.00014 · Cited by 83
Effect of impact factor and discipline on journal data sharing policies
Resnik, D. B., Morales, M., Landrum, R., Shi, M., Minnier, J., Vasilevsky, N. A., & Champieux, R. E. (2019) · Accountability in Research · doi:10.1080/08989621.2019.1591277 · Cited by 82
A review of European models to assess the sustainability performance of livestock production systems
van der Linden, A., de Olde, E. M., Mostert, P. F., & de Boer, I. J. M. (2020) · Agricultural Systems · doi:10.1016/j.agsy.2020.102842 · Cited by 65
Digital open science—Teaching digital tools for reproducible and transparent research
Toelch, U., & Ostwald, D. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2006022 · Cited by 80
Research data sharing: Developing a stakeholder‐driven model for journal policies
Sturges, P., Bamkin, M., Anders, J. H. S., Hubbard, B., Hussain, A. R., & Heeley, M. J. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23336 · Cited by 79
Hardwicke, T. E., & Ioannidis, J. P. A. (2018) · PLOS ONE · doi:10.1371/journal.pone.0201856 · Cited by 78
Reproducing Statistical Results
Stodden, V. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020127 · Cited by 77
Self-citationIs Most Published Research Really False?
Leek, J. T., & Jager, L. R. (2016) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-060116-054104 · Cited by 90
Bridging the Chasm: A Survey of Software Engineering Practice in Scientific Programming
Storer, T. (2017) · ACM Computing Surveys · doi:10.1145/3084225 · Cited by 93
Rethlefsen, M. L., Brigham, T., Price, C. L., Moher, D., Bouter, L. M., Kirkham, J. J., Schroter, S., & Zeegers, M. P. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.111229 · Cited by 77
Can We Be Wrong? The Problem of Textual Evidence in a Time of Data
Piper, A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108922036 · Cited by 90
Austin, C. C., Brown, S., Fong, N., Humphrey, C., Leahey, A., & Webster, P. M. (2016) · IASSIST Quarterly · doi:10.29173/iq904 · Cited by 66
Hardwicke, T. E., Mathur, M. B., MacDonald, K. E., Nilsonne, G., Banks, G. C., Kidwell, M., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Tessler, M. H., et al. (2018) · doi:10.31222/osf.io/39cfb · Cited by 52
A survey of biodiversity informatics: Concepts, practices, and challenges
Gadelha, L. M. R., Jr., de Siracusa, P. C., Dalcin, E., da Silva, L. A. E., Augusto, D. A., Krempser, E., Affe, H. M. D. J., Costa, R. L., Mondelli, M. L., Meirelles, P. M., et al. (2020) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1394 · Cited by 72
Sciunits: Reusable Research Objects
That, D. H. T., Fils, G., Yuan, Z., & Malik, T. (2017) · IEEE International Conference on eScience · doi:10.1109/escience.2017.51 · Cited by 60
The pediatric template of brain perfusion
Avants, B. B., Duda, J. T., Kilroy, E., Krasileva, K., Jann, K., Kandel, B. T., Tustison, N. J., Yan, L., Jog, M., Smith, R., et al. (2015) · Scientific Data · doi:10.1038/sdata.2015.3 · Cited by 60
Page, M. J., Nguyen, P., Hamilton, D. G., Haddaway, N., Kanukula, R., Moher, D., & McKenzie, J. E. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.03.003 · Cited by 60
The case for free and open source software in research and scholarship
Fortunato, L., & Galassi, M. C. (2021) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2020.0079 · Cited by 76
Stodden, V. (2020) · Communications of the ACM · doi:10.1145/3360646 · Cited by 69
Self-citationTowards computational reproducibility: researcher perspectives on the use and sharing of software
AlNoamany, Y., & Borghi, J. (2018) · PeerJ Computer Science · doi:10.7717/peerj-cs.163 · Cited by 60
Rousi, A. M., & Laakso, M. (2020) · Scientometrics · doi:10.1007/s11192-020-03467-9 · Cited by 54
How to make models more useful
Barton, C. M., Lee, A., Janssen, M. A., van der Leeuw, S., Tucker, G. E., Porter, C. H., Greenberg, J., Swantek, L. A., Frank, K., Chen, M., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2202112119 · Cited by 53
Krafczyk, M. S., Shi, A., Bhaskar, A., Marinov, D., & Stodden, V. (2021) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2020.0069 · Cited by 61
Self-citationScience is about corroborating empirical evidence, even in academic business research journals
Babin, B. J., Ortinau, D. J., Herrmann, J.-L., & López, C. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.06.002 · Cited by 55
Waltemath, D., & Wolkenhauer, O. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2555481 · Cited by 49
Sharing Data Increases Citations
Drachen, T. M., Ellegaard, O., Larsen, A. V., & Dorch, B. F. (2016) · LIBER Quarterly The Journal of the Association of European Research Libraries · doi:10.18352/lq.10149 · Cited by 48
Reproducibility and Replicability in Economics
Vilhuber, L. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.4f6b9e67 · Cited by 71
Reasons, challenges, and some tools for doing reproducible transportation research
Zheng, Z. (2021) · Communications in Transportation Research · doi:10.1016/j.commtr.2021.100004 · Cited by 47
Groom, Q., Strubbe, D., Adriaens, T., Davis, A. J. S., Desmet, P., Oldoni, D., Reyserhove, L., Roy, H. E., & Vanderhoeven, S. (2019) · Citizen Science: Theory and Practice · doi:10.5334/cstp.238 · Cited by 45
“It also reduces costs by avoiding duplication of effort (Stodden, Guo, and Ma 2013; Jarnevich et al. 2015).”
BackgroundColella, J. P., Stephens, R. B., Campbell, M., Kohli, B. A., Parsons, D. J., & McLean, B. S. (2020) · BioScience · doi:10.1093/biosci/biaa146 · Cited by 46
ODDPub – a Text-Mining Algorithm to Detect Data Sharing in Biomedical Publications
Riedel, N., Kip, M., & Bobrov, E. (2020) · Data Science Journal · doi:10.5334/dsj-2020-042 · Cited by 44
Estimating the reproducibility of social learning research published between 1955 and 2018
Minocher, R., Atmaca, S., Bavero, C., McElreath, R. L., & Beheim, B. (2021) · Royal Society Open Science · doi:10.1098/rsos.210450 · Cited by 48
Christian, T.-M. L., Gooch, A., Vision, T., & Hull, E. (2020) · PLoS ONE · doi:10.1371/journal.pone.0230281 · Cited by 42
Morris, A. H., Stagg, B. C., Lanspa, M. J., Orme, J. F., Clemmer, T. P., Weaver, L. K., Thomas, F., Grissom, C. K., Hirshberg, E. L., East, T. D., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa294 · Cited by 44
Data policies of highly-ranked social science journalsPreprint
Crosas, M., Gautier, J., Karcher, S., Kirilova, D., Otalora, G., & Schwartz, A. (2018) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/9h7ay · Cited by 42
Heacock, M. L., Lopez, A. R., Amolegbe, S. M., Carlin, D. J., Henry, H. F., Trottier, B. A., Velasco, M. L., & Suk, W. A. (2022) · Environmental Science & Technology · doi:10.1021/acs.est.1c08383 · Cited by 46
Marwick, B. (2017) · ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) · doi:10.17605/osf.io/3d6xx · Cited by 41
Ribes, D. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998344 · Cited by 36
Parti, K., & Szigeti, Á. (2021) · Cogent Social Sciences · doi:10.1080/23311886.2021.1970880 · Cited by 48
Open science practices in clinical psychology journals: An audit study.
Nutu, D., Gentili, C., Naudet, F., & Cristea, I. A. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000414 · Cited by 34
MLitB: machine learning in the browser
Meeds, E., Hendriks, R., Faraby, S. A., Bruntink, M., & Welling, M. (2015) · PeerJ Computer Science · doi:10.7717/peerj-cs.11 · Cited by 40
Reproducible research in statistics: A review and guidelines for the Biometrical Journal
Hofner, B., Schmid, M., & Edler, L. (2016) · Biometrical Journal · doi:10.1002/bimj.201500156 · Cited by 35
Kim, J., Kim, S., Cho, H. M., Chang, J. H., & Kim, S. Y. (2020) · PeerJ · doi:10.7717/peerj.9924 · Cited by 39
Towards Bayesian Model-Based Demography: Agency, Complexity and Uncertainty in Migration Studies
Bijak, J. (2021) · Methodos series · doi:10.1007/978-3-030-83039-7 · Cited by 48
A Minimum Standard for Publishing Computational Results in the Weather and Climate Sciences
Irving, D. (2015) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-15-00010.1 · Cited by 34
Reproducibility in Computing Research: An Empirical Study
Raghupathi, W., Raghupathi, V., & Ren, J. (2022) · IEEE Access · doi:10.1109/access.2022.3158675 · Cited by 43
Stodden, V., Miguez, S., & Seiler, J. (2015) · Computing in Science & Engineering · doi:10.1109/mcse.2015.18 · Cited by 33
Self-citationCurrent and planned adoption of data sharing policies by editors of Korean scholarly journals
Kim, S. Y., Yi, H. J., & Huh, S. Y. (2019) · Science Editing · doi:10.6087/kcse.151 · Cited by 31
Tool-Driven Revolutions in Archaeological Science
Schmidt, S. C., & Marwick, B. (2020) · Journal of Computer Applications in Archaeology · doi:10.5334/jcaa.29 · Cited by 41
Zheng, J., Harris, M. R., Masci, A. M., Lin, Y., Hero, A. O., Smith, B., & He, Y. O. (2016) · Journal of Biomedical Semantics · doi:10.1186/s13326-016-0100-2 · Cited by 31
Statistical contributions to bioinformatics: Design, modelling, structure learning and integration
Morris, J. S., & Baladandayuthapani, V. (2017) · Statistical Modelling · doi:10.1177/1471082x17698255 · Cited by 32
Reproducibility in Research: Systems, Infrastructure, Culture
Crick, T., Hall, B. A., & Ishtiaq, S. S. (2017) · Journal of Open Research Software · doi:10.5334/jors.73 · Cited by 30
GigaDB: promoting data dissemination and reproducibility
Sneddon, T. P., Zhe, X. S., Edmunds, S., Li, P., Goodman, L., & Hunter, C. (2014) · Database · doi:10.1093/database/bau018 · Cited by 28
Kurata, K., Matsubayashi, M., & Mine, S. (2017) · SAGE Open · doi:10.1177/2158244017717301 · Cited by 29
Current status on the need for improved accessibility to climate models code
Añel, J. A., García‐Rodríguez, M., & Iglesias, J. R. (2021) · Geoscientific model development · doi:10.5194/gmd-14-923-2021 · Cited by 26
Enabling Reproducibility in Big Data Research: Balancing Confidentiality and Scientific Transparency
Stodden, V. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781107590205.007 · Cited by 26
Self-citationBaeza, J. A. (2022) · Journal of Crustacean Biology · doi:10.1093/jcbiol/ruac012 · Cited by 28
Stodden, V., Krafczyk, M. S., & Bhaskar, A. (2018) · International Workshop on Practical Reproducible Evaluation of Computer Systems (P-RECS) · doi:10.1145/3214239.3214242 · Cited by 27
Self-citationEmbracing Transparency Through Data Sharing
Boué, S., Byrne, M., Hayes, A. W., Hoeng, J., & Peitsch, M. C. (2018) · International Journal of Toxicology · doi:10.1177/1091581818803880 · Cited by 30
Anticipating consequences of sharing raw data and code and of awarding badges for sharing
Ioannidis, J. P. A. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.015 · Cited by 25
Schnell, S. D. (2018) · Bulletin of Mathematical Biology · doi:10.1007/s11538-018-0500-9 · Cited by 28
Yu, H. J., Nysak, S., Garg, N., Ong, E., Ye, X., Zhang, X., & He, Y. O. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-2729-1 · Cited by 26
The New Reality of Reproducibility: The Role of Data Work in Scientific Research
Feinberg, M., Sutherland, W., Nelson, S. B., Jarrahi, M. H., & Rajasekar, A. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392840 · Cited by 28
It's time! Ten reasons to start replicating simulation studies
Lohmann, A., Astivia, O. L. O., Morris, T. P., & Groenwold, R. H. H. (2022) · Frontiers in Epidemiology · doi:10.3389/fepid.2022.973470 · Cited by 29
Utilizing Provenance in Reusable Research Objects
Yuan, Z., That, D. H. T., Kothari, S., Fils, G., & Malik, T. (2018) · Informatics · doi:10.3390/informatics5010014 · Cited by 24
Data Sharing and Engineering Faculty: An Analysis of Selected Publications
Wiley, C. (2018) · Science & Technology Libraries · doi:10.1080/0194262x.2018.1516596 · Cited by 24
Metadata
- Authors
- Stodden, V., Guo, P., & Ma, Z.
- Affiliations
- Columbia University (US)
- Corresponding author
- Stodden, Victoria
- ORCID
- Stodden, Victoria
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-06-21
- Volume
- 8
- Issue
- 6
- Pages
- e67111
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3689732
- OpenAlex
- W2031934506
- Cited by
- 257
- References
- 25
- Citations by year
- 2026: 9 · 2025: 12 · 2024: 11 · 2023: 11 · 2022: 19 · 2021: 23 · 2020: 33 · 2019: 25 · 2018: 26 · 2017: 30 · 2016: 18 · 2015: 21 · 2014: 15 · 2013: 2
- FWCI
- 54.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Columbia Academic Commons (Columbia University)
- Funders
- National Science Foundation (1153384)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- reproducible research, open data, open code, open access journals, journal impact factor
- MeSH terms
- Models, Theoretical, Periodicals as Topic, Reproducibility of Results, Computational Biology, Codes of Ethics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


