Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Journal policy on research data and code availability is an important part of the ongoing shift toward publishing reproducible computational science. This article extends the literature by studying journal data sharing policies by year (for both 2011 and 2012) for a referent set of 170 journals. We make a further contribution by evaluating code sharing policies, supplemental materials policies, and open access status for these 170 journals for each of 2011 and 2012. We build a predictive model of open data and code policy adoption as a function of impact factor and publisher and find higher impact journals more likely to have open data and code policies and scientific societies more likely to have open data and code policies than commercial publishers. We also find open data policies tend to lead open code policies, and we find no relationship between open data and code policies and either supplemental material policies or open access journal status. Of the journals in this study, 38% had a data policy, 22% had a code policy, and 66% had a supplemental materials policy as of June 2012. This reflects a striking one year increase of 16% in the number of data policies, a 30% increase in code policies, and a 7% increase in the number of supplemental materials policies. We introduce a new dataset to the community that categorizes data and code sharing, supplemental materials, and open access policies in 2011 and 2012 for these 170 journals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Policies that mandate sharing of analysis scripts have already been implemented at some journals, for example in economics and political science [19,45,46].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“An increasing number of publishers and funding bodies require authors or grant holders to make their data publicly available [247].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [247]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Minimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
King, G. (1995) · PS Political Science & Politics · doi:10.1017/s1049096500057607 · Cited by 436
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424 · cited 1×
“…has focused on changes needed in scientific publishing and the role of journal open data requirements in fostering scientific reliability [4,5,6,7] but little research has focused on availability of the code needed to replicate computational findings, although this has been associated with…”
BackgroundScientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar-Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 368
Evolution of Translational Omics: Lessons Learned and the Path Forward
Micheel, C., Nass, S. J., & Omenn, G. S. (2013) · Cited by 365
Public Availability of Published Research Data in High-Impact Journals
Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., & Ioannidis, J. P. A. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024357 · Cited by 363 · cited 1×
“…attention has focused on changes needed in scientific publishing and the role of journal open data requirements in fostering scientific reliability [4,5,6,7] but little research has focused on availability of the code needed to replicate computational findings, although this has been associated…”
BackgroundPublish your computer code: it is good enough
Barnes, N. (2010) · Nature · doi:10.1038/467753a · Cited by 315
Accessible Reproducible Research
Mesirov, J. P. (2010) · Science · doi:10.1126/science.1179653 · Cited by 307
Reproducible Research in Computational Harmonic Analysis
Donoho, D. L., Maleki, A., Rahman, I. U., Shahram, M., & Stodden, V. (2009) · Computing in Science & Engineering · doi:10.1109/mcse.2009.15 · Cited by 274
Self-citation
Research works printed but not linked: 2
- Reproducible Research. Special Issue, Computing in Science and Engineering 14: 4 11–56. http://ieeexplore.ieee.org/xpl/tocresult.jsp?reload=true&isnumber=6241356&punumber=5992 (last accessed May 8, 2013).
- Editorial (1998) The gatekeepers. Nature Structural Biology 5: 3 164. http://www.nature.com/nsmb/wilma/v5n3.892130820.html (last accessed May 8, 2013).
Cited by
Items 101–200 of 276
Samota, E. K., & Davey, R. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.678554 · Cited by 23
From policy to practice: progress towards data- and code-sharing in ecology and evolution
Ivimey‐Cook, E. R., Sánchez‐Tójar, A., Berberi, I., Čulina, A., Roche, D. G., Almeida, R. A., Amin, B., Bairos‐Novak, K. R., Balti, H., Bertram, M. G., et al. (2025) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2025.1394 · Cited by 25
Citations for Software: Providing Identification, Access and Recognition for Research Software
Soito, L., & Hwang, L. (2017) · International Journal of Digital Curation · doi:10.2218/ijdc.v11i2.390 · Cited by 23
Spying on the Floating Point Behavior of Existing, Unmodified Scientific Applications
Dinda, P. A., Bernat, A., & Hetland, C. (2020) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3369583.3392673 · Cited by 21
Gonzalez, J. A. H., & Colom, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1491823 · Cited by 26
Quality Control for Scientific Research: Addressing Reproducibility, Responsiveness, and Relevance
Hubbard, D. W., & Carriquiry, A. L. (2019) · The American Statistician · doi:10.1080/00031305.2018.1543138 · Cited by 23
On the Convergence of Epidemiology, Biostatistics, and Data Science
Goldstein, N. D., LeVasseur, M. T., & McClure, L. A. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.9f0215e6 · Cited by 21
A new framework for host-pathogen interaction research
Yu, H. J., Li, L., Huffman, A., Beverley, J. R., Hur, J., Merrell, E. C., Huang, H.-H., Wang, Y., Liu, Y., Ong, E., et al. (2022) · Frontiers in Immunology · doi:10.3389/fimmu.2022.1066733 · Cited by 28
Open Data Policies among Library and Information Science Journals
Jackson, B. (2021) · Publications · doi:10.3390/publications9020025 · Cited by 21
Osakpolor, S. E., Kattwinkel, M., Schirmel, J., Feckler, A., Manfrin, A., & Schäfer, R. B. (2021) · Ecological Modelling · doi:10.1016/j.ecolmodel.2021.109710 · Cited by 20
Experiences in integrated data and research object publishing using GigaDB
Edmunds, S., Li, P., Hunter, C., Zhe, X. S., Davidson, R., Nogoy, N., & Goodman, L. (2016) · International Journal on Digital Libraries · doi:10.1007/s00799-016-0174-6 · Cited by 20
From Reproducibility to Translation in Neurodegenerative Disease
Zeiss, C. J. (2017) · ILAR Journal · doi:10.1093/ilar/ilx006 · Cited by 21
A meta-epidemiological assessment of transparency indicators of infectious disease models
Zavalis, E. A., & Ioannidis, J. P. A. (2022) · PLoS ONE · doi:10.1371/journal.pone.0275380 · Cited by 22
Ethical perspectives on data and software sharing in the sciences: A research agenda
Darch, P. T., & Knox, E. J. M. (2017) · Library & Information Science Research · doi:10.1016/j.lisr.2017.11.008 · Cited by 19
Integrating GRASS GIS and Jupyter Notebooks to facilitate advanced geospatial modeling education
Haedrich, C. E., Petráš, V., Petrášová, A., Blumentrath, S., & Mitášová, H. (2023) · Transactions in GIS · doi:10.1111/tgis.13031 · Cited by 16
An Exploratory Study of Research Data Governance in the U.S.
Kouper, I., Raymond, A. H., & Giroux, S. A. (2020) · Open Information Science · doi:10.1515/opis-2020-0010 · Cited by 16
Reproducibility and replicability of rodent phenotyping in preclinical studiesPreprint
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R. T., Golani, I., Gómez-Marín, À., Heller, R., et al. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/079350 · Cited by 16
Self-citationSzkuta, K., & Osimo, D. (2016) · foresight · doi:10.1108/fs-06-2014-0040 · Cited by 16
Cohen, J. T., & Wong, J. B. (2017) · Medical Care · doi:10.1097/mlr.0000000000000811 · Cited by 14
Functional R code is rare in species distribution and abundance papers
Kellner, K. F., Doser, J. W., & Belant, J. L. (2024) · Ecology · doi:10.1002/ecy.4475 · Cited by 15
""""Share and Enjoy"""": Publishing Useful and Usable Scientific Models
Crick, T., Hall, B. A., Ishtiaq, S. S., & Takeda, K. (2014) · IEEE/ACM International Conference on Utility and Cloud Computing (UCC) · doi:10.1109/ucc.2014.156 · Cited by 14
Sellami, A., Réau, M. F., Montès, M., & Lagarde, N. (2022) · Frontiers in Endocrinology · doi:10.3389/fendo.2022.986016 · Cited by 14
Data as Social Capital and the Gift Culture in Research
Klump, J. (2017) · Data Science Journal · doi:10.5334/dsj-2017-014 · Cited by 15
Yang, B., Rousseau, R., Wang, X., & Huang, S. (2018) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24031 · Cited by 14
Broussard, M. (2015) · Digital Journalism · doi:10.1080/21670811.2015.1074863 · Cited by 14
For long-term sustainable software in bioinformatics
Coelho, L. P. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011920 · Cited by 13
contextual: Evaluating Contextual Multi-Armed Bandit Problems in RPreprint
van Emden, R., & Kaptein, M. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01926 · Cited by 13
Huang, M., Law, H., & Tam, S. Y. (2025) · Cancers · doi:10.3390/cancers17193121 · Cited by 16
Open Science, Replicability, and Transparency in Modelling
Prike, T. (2021) · Methodos series · doi:10.1007/978-3-030-83039-7_10 · Cited by 12
Openness and Computational Reproducibility in Plant Pathology: Where We Stand and a Way Forward
Sparks, A., Ponte, E. M. D., Alves, K. S., Foster, Z., & Grünwald, N. J. (2023) · Phytopathology · doi:10.1094/phyto-10-21-0430-per · Cited by 11
Access to Research Data: Addressing the Problem through Journal Data Sharing Policies
Sturges, P., Bamkin, M., Anders, J. H. S., & Hussain, A. R. (2014) · Purdue e-Pubs (Purdue University System) · Cited by 11
Schirrwagen, J., Cimiano, P., Ayer, V., Pietsch, C., Wiljes, C., Vompras, J., & Pieper, D. (2019) · Data Science Journal · doi:10.5334/dsj-2019-006 · Cited by 11
Now Is the Time for Transparency in Value-Based Healthcare Decision Modeling
Hay, J. W. (2019) · Value in Health · doi:10.1016/j.jval.2019.01.009 · Cited by 11
Mondelli, M. L., Peterson, A. T., & Gadelha, L. M. R., Jr. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-34146-6_3 · Cited by 13
Schneider, M. V., Griffin, P. C., Tyagi, S., Flannery, M., Dayalan, S., Gladman, S. L., Watson‐Haigh, N. S., Bayer, P. E., Charleston, M., Cooke, I., et al. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx071 · Cited by 10
Software Curation in Research Libraries: Practice and Promise
Chassanoff, A., Borghi, J., AlNoamany, Y., & Thornton, K. (2018) · Journal of Librarianship and Scholarly Communication · doi:10.7710/2162-3309.2239 · Cited by 10
Weber, N., & Thomer, A. K. (2014) · Advances in human and social aspects of technology book series · doi:10.4018/978-1-4666-6002-1.ch005 · Cited by 12
Costa, V., Righelli, D., Russo, F., Berardinis, P. D., Angelini, C., & D’Apice, L. (2017) · International Journal of Molecular Sciences · doi:10.3390/ijms18030494 · Cited by 9
Simulation Design: Trans-Paradigm Best-Practice from Software Engineering
Rossiter, S. (2015) · Journal of Artificial Societies and Social Simulation · doi:10.18564/jasss.2842 · Cited by 8
Navarro‐Molina, C., & Melero, R. (2019) · El Profesional de la Informacion · doi:10.3145/epi.2019.sep.16 · Cited by 10
Sharing data in Ecology: how to add more value to data
Pérez‐Luque, A. J., & Ros‐Candeira, A. (2019) · Ecosistemas · doi:10.7818/ecos.1838 · Cited by 8
Status and factors associated with the adoption of data sharing policies in Asian journals
Kim, J., & Bai, S. Y. (2022) · Science Editing · doi:10.6087/kcse.274 · Cited by 8
Wang, Y., Chen, B., Zhao, L., & Zeng, Y. (2022) · Learned Publishing · doi:10.1002/leap.1437 · Cited by 8
Transparency in Infectious Disease Research: Meta-research Survey of Specialty Journals
Zavalis, E. A., Contopoulos‐Ioannidis, D. G., & Ioannidis, J. P. A. (2023) · The Journal of Infectious Diseases · doi:10.1093/infdis/jiad130 · Cited by 7
Seeking Equilibrium in Data Reuse: A Study of Knowledge Satisficing
York, J. (2022) · Deep Blue (University of Michigan) · doi:10.7302/6170 · Cited by 6
Stodden, V. (2020) · International Workshop on Practical Reproducible Evaluation of Computer Systems (P-RECS) · doi:10.1145/3391800.3398172 · Cited by 8
Self-citationCiencia reproducible: ¿qué, por qué, cómo?
Rodríguez‐Sánchez, F., Pérez‐Luque, A. J., Bartomeus, Í., & Varela, S. (2016) · Ecosistemas · doi:10.7818/ecos.2016.25-2.11 · Cited by 16
He, Y. O. (2022) · Methods in molecular biology · doi:10.1007/978-1-0716-2265-0_12 · Cited by 7
The poor availability of syntaxes of structural equation modeling
Wicherts, J. M., & Crompvoets, E. A. V. (2017) · Accountability in Research · doi:10.1080/08989621.2017.1396214 · Cited by 6
Chard, K., Gaffney, N. I., Jones, M. B., Kowalik, K., Ludäscher, B., McPhillips, T., Nabrzyski, J., Stodden, V., Taylor, I., Thelen, T., et al. (2019) · International Conference on e-Science · doi:10.1109/escience.2019.00068 · Cited by 6
Self-citationA workflow language for research e-infrastructures
Candela, L., Grossi, V., Manghi, P., & Trasarti, R. (2021) · International Journal of Data Science and Analytics · doi:10.1007/s41060-020-00237-x · Cited by 6
Andalib, S. Y., Monsur, M., Cook, C., Lemon, M., Zawarus, P. W., & Loon, L. (2025) · Education Sciences · doi:10.3390/educsci15080992 · Cited by 5
Reading the fine print: A review and analysis of business journals’ data sharing policies
Dosch, B. R., & Martindale, T. (2020) · Journal of Business & Finance Librarianship · doi:10.1080/08963568.2020.1847549 · Cited by 5
Confronting the Reproducibility Crisis: A Case Study of Challenges in Cybersecurity AI
Moulton, R. H., McCully, G. A., & Hastings, J. D. (2024) · Cyber Awareness and Research Symposium (CARS) · doi:10.1109/cars61786.2024.10778911 · Cited by 9
Escape Room: A Configurable Testbed for Hierarchical Reinforcement Learning
Menashe, J., & Stone, P. (2019) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/juvy9794 · Cited by 5
cchsflow: an open science approach to transform and combine population health surveys
Yusuf, W., Vyuha, R., Bennett, C. L., Sequeira, Y., Maskerine, C., & Manuel, D. G. (2021) · Canadian Journal of Public Health · doi:10.17269/s41997-020-00470-8 · Cited by 5
The Digital Science Field of Design Science Research
Storey, V. C., & Baskerville, R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-82405-1_33 · Cited by 6
Data Communities: Empowering Researcher-Driven Data Sharing in the Sciences
Springer, R., & Cooper, D. (1970) · International Journal of Digital Curation · doi:10.2218/ijdc.v15i1.695 · Cited by 5
Digital Open Science – Teaching digital tools for reproducible and transparent researchPreprint
Toelch, U., & Ostwald, D. (2017) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/fbvep · Cited by 5
Open science practices in clinical psychology journals: an audit studyPreprint
Cristea, I. A., Naudet, F., Nutu, D., & Gentili, C. (2018) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/t2k56 · Cited by 5
Godwin, H. J., Dewis, H., Darch, P. T., Hout, M. C., Ernst, D., Broadbent, P., Papesh, M. H., & Wolfe, J. M. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02759-3 · Cited by 4
Király, A. (2017) · Archaeologiai Értesítő · doi:10.1556/0208.2017.142.8 · Cited by 4
Simulation studies of social systems: telling the story based on provenance patterns
Wilsdorf, P., Reinhardt, O., Prike, T., Hinsch, M., Bijak, J., & Uhrmacher, A. M. (2024) · Royal Society Open Science · doi:10.1098/rsos.240258 · Cited by 5
Park, J. R., Bae, I., Ryu, S., Kim, M., Bang, H., Won, J., & Lee, H. (2025) · Integrative Medicine Research · doi:10.1016/j.imr.2025.101229 · Cited by 3
Revealing Earth science code and data-use practices using the Throughput Graph Database
Thomer*, A. K., Wofford, M. F., Lenard, M., Vidaña, S. D., & Goring, S. J. (2023) · Geological Society of America eBooks · doi:10.1130/2022.2558(10 · Cited by 3
Tooling for Reproducible Research: Considerations for the Past and Future of Data Analysis
Peng, R. D. (2025) · Austrian Journal of Statistics · doi:10.17713/ajs.v54i3.2052 · Cited by 3
Pixel: a content management platform for quantitative omics data
Denecker, T., Durand, W., Maupetit, J., Hébert, C., Camadro, J., Poulain, P., & Lelandais, G. (2019) · PeerJ · doi:10.7717/peerj.6623 · Cited by 3
Towards a Culture of Open Science and Data Sharing in Health and Medical Research
Rowhani-Farid, A. (2018) · Queensland University of Technology · doi:10.5204/thesis.eprints.119697 · Cited by 7
OpenHeart Project—An Open‐Source Research Community in the Field of Mechanical Circulatory Support
Pauls, J. P., Miotto, A., Stephens, A. F., Gregory, S. D., & Tansley, G. D. (2018) · Artificial Organs · doi:10.1111/aor.13361 · Cited by 3
Simulation Studies of Social Systems – Telling the Story Based on Provenance PatternsPreprint
Reinhardt, O., Prike, T., Hinsch, M., Bijak, J., Wilsdorf, P., & Uhrmacher, A. M. (2023) · doi:10.36227/techrxiv.20209844 · Cited by 3
A conceptual review of uses and meanings of reproducibility and replicationPreprint
Ulpts, S., & Schneider, J. W. (2024) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/entu4 · Cited by 3
Wood, J., & van Schalkwyk, I. (2025) · Journal of Transportation Technologies · doi:10.4236/jtts.2025.151010 · Cited by 4
A comprehensive review of chemoinformatics: From molecular representation to data-driven design
Islam, M. S., Sarker, P., Islam, M. M., Shamim, S. U., Malitha, S. B., Nakib, K. I. A., Alam, M. Z., & Chowdhury, A. M. S. (2026) · Next research. · doi:10.1016/j.nexres.2026.102362 · Cited by 2
Reproducibility literature analysis - a federal information professional perspective
Antognoli, E., Avila, R. L., Sears, J., Christiansen, L. L., Tieman, J., & Hart, J. (2020) · IASSIST Quarterly · doi:10.29173/iq967 · Cited by 2
FAIR Principles and Why They Matter
Russell, K. (2021) · Wiley eBooks · doi:10.1002/9783527825042.ch5 · Cited by 2
A Study in Reproducibility: The Congruent Matching Cells Algorithm and cmcR Package
Zemmels, J., VanderPlas, S., & Hofmann, H. (2023) · The R Journal · doi:10.32614/rj-2023-014 · Cited by 3
Open data policy and journal impacts: a synthetic difference-in-differences approach
Huang, H.-C. R., Ma, Y., & Wang, Y. (2023) · Applied Economics Letters · doi:10.1080/13504851.2023.2187017 · Cited by 2
Fundamentación ética del deber sobre el acceso abierto a fuentes de información científica
Valverde, A., & González-Miranda, J. A. (2024) · Revista Espiga · doi:10.22458/re.v23i47.5136 · Cited by 3
Radiomics in non-oncologic musculoskeletal diseases: from pixels to practice
Oh, J., Hong, S. H., Choi, J., Yoo, H. J., & Chae, H.-D. (2025) · Clinical Radiology · doi:10.1016/j.crad.2025.107106 · Cited by 3
Version Control Systems to Facilitate Research Collaboration in Economics
Rodrigues, B. (2015) · Computational Economics · doi:10.1007/s10614-015-9513-8 · Cited by 2
Data expertise and service development in geoscience data centers and academic libraries
Thompson, C. A. (2017) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 4
An analysis of open research data practice, drivers, and barriers in Japan
Ikeuchi, U. (2018) · Library and Information Science · doi:10.46895/lis.79.21 · Cited by 2
Gayle, V., & Connelly, R. (2022) · Methodological Innovations · doi:10.1177/20597991221111681 · Cited by 2
Towards "Reproducibility-as-a-Service".Preprint
Crick, T., Ishtiaq, S. S., & Hall, B. A. (2015) · arXiv (Cornell University) · Cited by 2
The Poor Availability of Syntaxes of Structural Equation ModelingPreprint
Wicherts, J. M., & Crompvoets, E. A. V. (2017) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/qns25 · Cited by 2
Tool-driven Revolutions in Archaeological SciencePreprint
Schmidt, S. C., & Marwick, B. (2019) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/4nkxv · Cited by 3
Methodological landscape of qualitative works on open sciencePreprint
Manco, A. (2022) · MediArXiv (OSF Preprints) · doi:10.33767/osf.io/ac2zd · Cited by 2
Simulation Studies of Social Systems -- Telling the Story Based on Provenance PatternsPreprint
Reinhardt, O., Prike, T., Hinsch, M., Bijak, J., Uhrmacher, A. M., & Wilsdorf, P. (2023) · doi:10.36227/techrxiv.20209844.v2 · Cited by 2
A survey of experimental stimulus presentation code sharing in major areas of psychology
Rawal, A., Truong, V., Lo, Y., Tseng, L.-Y., & Duncan, N. W. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02390-8 · Cited by 1
Krishnapur, P. K., Chaitra, C. M., Arjun, B., & Ananthakrishnan, M. (2026) · Cureus · doi:10.7759/cureus.103008 · Cited by 4
Overcoming barriers to computational reproducibility
Hornung, R., Németh, L., Zadorozhnyi, O., Ullmann, T., Kammer, M., Killick, R., Paciorek, C., Chiquet, J., Herrmann, M., Batinović, L., et al. (2026) · Royal Society Open Science · doi:10.1098/rsos.252489 · Cited by 2
“It makes sense to me”: Examining data file column names in the visual cognitive literature
Sordo, G. C. D., Dewis, H., Darch, P. T., Hout, M. C., & Godwin, H. J. (2026) · Behavior Research Methods · doi:10.3758/s13428-026-03104-y · Cited by 1
A review of journal policies for sharing research data across disciplines
Ikeuchi, U., Nishiura, M., & Itsumura, H. (2014) · Figshare · doi:10.6084/m9.figshare.945108 · Cited by 1
Reproducibility in Computational Science
Stodden, V. (2012) · doi:10.7916/d8f76mtz · Cited by 1
Self-citationCoalescent: an open-science framework for importance sampling in coalescent theory
Tewari, S., & Spouge, J. L. (2015) · PeerJ · doi:10.7717/peerj.1203 · Cited by 1
Data Sharing And Data Reuse: An Investigation Of Descriptive Information Facilitators And Inhibitors
Murillo, A. P. (2016) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/a7vg-fp11 · Cited by 1
Do we consent to rules of consent and confidentiality?
Wegscheider, K., & Friede, T. (2017) · Biometrical Journal · doi:10.1002/bimj.201600150 · Cited by 1
Denaxas, S., González-Izquierdo, A., Pikoula, M., Direk, K., Fitzpatrick, N. K., Hemingway, H., & Smeeth, L. L. (2017) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms.2017.74 · Cited by 1
Gabb, H. A. (2019) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 1
Parti, K., & Szigeti, Á. (2021) · Socio hu · doi:10.18030/socio.hu.2021.1.147 · Cited by 1
Metadata
- Authors
- Stodden, V., Guo, P., & Ma, Z.
- Affiliations
- Columbia University (US)
- Corresponding author
- Stodden, Victoria
- ORCID
- Stodden, Victoria
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-06-21
- Volume
- 8
- Issue
- 6
- Pages
- e67111
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3689732
- OpenAlex
- W2031934506
- Cited by
- 257
- References
- 25
- Citations by year
- 2026: 9 · 2025: 12 · 2024: 11 · 2023: 11 · 2022: 19 · 2021: 23 · 2020: 33 · 2019: 25 · 2018: 26 · 2017: 30 · 2016: 18 · 2015: 21 · 2014: 15 · 2013: 2
- FWCI
- 54.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Columbia Academic Commons (Columbia University)
- Funders
- National Science Foundation (1153384)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- reproducible research, open data, open code, open access journals, journal impact factor
- MeSH terms
- Models, Theoretical, Periodicals as Topic, Reproducibility of Results, Computational Biology, Codes of Ethics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


