Best Practices for Scientific Computing
Open accessThe DOI matches its Crossref record; 73 of the 75 references it cites were found in a registry or an index.
Abstract
The Community Page is a forum for organizations and societies
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The creation of MIMIC-IV followed best practices in scientific computing23.”
From MIMIC-IV, a freely accessible electronic health record dataset (2023) · cites this paper as [23]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
5 retracted
Chang, G. A., Roth, C., Reyes, C. L., Pornillos, O., Chen, Y.-J., & Chen, A. P. (2006) · Science · doi:10.1126/science.314.5807.1875b · Cited by 137
RetractedRETRACTED: Structure of the multidrug resistance efflux transporter EmrE from Escherichia coli
Ma, C., & Chang, G. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0400137101 · Cited by 102
RetractedChang, G. A. (2007) · Journal of Molecular Biology · doi:10.1016/j.jmb.2003.05.001 · Cited by 22
RetractedMa, C., & Chang, G. A. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0700711104 · Cited by 18
RetractedHuijgen, R., Boekholdt, S. M., Arsenault, B. J., Bao, W., Davaine, J., Tabet, F., Petrides, F., Rye, K.-A., DeMicco, D. A., Barter, P. J., et al. (2013) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.04.002 · Cited by 1
RetractedIPython: A System for Interactive Scientific Computing
Perez, F., & Granger, B. E. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.53 · Cited by 4,009 · cited 1×
“Common approaches to this include this use of knitr [63] and IPython Notebooks [64].”
MethodCockburn, A. (2001) · UCLA - Biblioteca de Ciencias y Tecnologia Felix Morales Bueno · Cited by 1,858
Verification and Validation in Scientific Computing
Oberkampf, W. L., & Roy, C. J. (2010) · doi:10.1017/cbo9780511760396 · Cited by 1,556 · cited 1×
“Exactly how they check correctness depends on the researcher’s understanding of the problem at hand [24, 32, 46].”
BackgroundDesign and code inspections to reduce errors in program development
Fagan, M. E. (1976) · IBM Systems Journal · doi:10.1147/sj.153.0182 · Cited by 1,492
Agile Software Development, Principles, Patterns, and Practices
Martin, R. (2002) · Cited by 1,466
Research works printed but not linked: 4
- Z Merali, 2010, Error: why scientific programming does not compute, Nature, 467, 775
- Baddeley A, Eysenck MW, Anderson MC (2009) Memory. New York: Psychology Press.
- Ray DS, Ray EJ (2009) Unix and Linux: visual quickstart guide. 4th edition. San Francisco: Peachpit Press.
- Kelly D, Sanders R (2008) Assessing the quality of scientific software. In: Proceedings First International Workshop on Software Engineering for Computational Science and Engineering. Available: http://secse08.cs.ua.edu/Papers/Kelly.pdf.
Cited by
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
The nf-core framework for community-curated bioinformatics pipelines
Ewels, P. A., Peltzer, A., Fillinger, S., Patel, H., Alneberg, J., Wilm, A., Garcia, M. U., Di Tommaso, P., & Nahnsen, S. (2020) · Nature Biotechnology · doi:10.1038/s41587-020-0439-x · Cited by 4,050
MIMIC-IV, a freely accessible electronic health record dataset
Johnson, A. E. W., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. (2023) · Scientific Data · doi:10.1038/s41597-022-01899-x · Cited by 3,401
“The creation of MIMIC-IV followed best practices in scientific computing 23 .”
MethodMDTraj: A Modern Open Library for the Analysis of Molecular Dynamics Trajectories
McGibbon, R. T., Beauchamp, K. A., Harrigan, M. P., Klein, C., Swails, J. M., Hernández, C. X., Schwantes, C. R., Wang, L.-P., Lane, T. J., & Pande, V. S. (2015) · Biophysical Journal · doi:10.1016/j.bpj.2015.08.015 · Cited by 2,373
MDAnalysis: A Python Package for the Rapid Analysis of Molecular Dynamics Simulations
Gowers, R., Linke, M., Barnoud, J., Reddy, T., Melo, M., Seyler, S., Domański, J., Dotson, D., Buchoux, S., Kenney, I., et al. (2016) · Proceedings of the Python in Science Conference · doi:10.25080/majora-629e541a-00e · Cited by 2,026
Pollard, T. J., Johnson, A. E. W., Raffa, J. D., Celi, L. A., Mark, R. G., & Badawi, O. (2018) · Scientific Data · doi:10.1038/sdata.2018.178 · Cited by 1,993
MNE software for processing MEG and EEG data
Gramfort, A., Luessi, M., Larson, E., Engemann, D. A., Strohmeier, D., Brodbeck, C., Parkkonen, L., & Hämäläinen, M. S. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.10.027 · Cited by 1,951
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816
“Creation of MIMIC-CXR followed good software practices, including source code control, continuous integration, and test-driven development 19 .”
MethodA comparison of single-cell trajectory inference methods
Saelens, W., Cannoodt, R., Todorov, H., & Saeys, Y. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0071-9 · Cited by 1,670
Tackling Exascale Software Challenges in Molecular Dynamics Simulations with GROMACS
Páll, S., Abraham, M. J., Kutzner, C., Hess, B., & Lindahl, E. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-15976-8_1 · Cited by 1,208
“Ideally, software engineering on such moderately large multi-purpose code bases would take place within the context of effective abstractions [26].”
BackgroundWeaver, S., Shank, S. D., Spielman, S. J., Li, M., Muse, S. V., & Kosakovsky Pond, S. L. (2018) · Molecular Biology and Evolution · doi:10.1093/molbev/msx335 · Cited by 983
MOOSE: Enabling massively parallel multiphysics simulation
Permann, C. J., Gaston, D. R., Andrš, D., Carlsen, R. W., Kong, F., Lindsay, A. D., Miller, J. M., Peterson, J. W., Slaughter, A. E., Stogner, R. H., et al. (2020) · SoftwareX · doi:10.1016/j.softx.2020.100430 · Cited by 660
Iterative near-term ecological forecasting: Needs, opportunities, and challenges
Dietze, M. C., Fox, A., Beck-Johnson, L. M., Betancourt, J. L., Hooten, M. B., Jarnevich, C. S., Keitt, T. H., Kenney, M. A., Laney, C. M., Larsen, L. G., et al. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1710231115 · Cited by 604
Self-citationFrom big data analysis to personalized medicine for all: challenges and opportunities
Alyass, A., Turcotte, M., & Meyre, D. (2015) · BMC Medical Genomics · doi:10.1186/s12920-015-0108-y · Cited by 557
“There is a need for reliable and maintainable computer codes through best practices for scientific computing [41].”
BackgroundThe SunPy Project: Open Source Development and Status of the Version 1.0 Core Package
Community, T. S., Barnes, W. T., Bobra, M. G., Christe, S. D., Freij, N., Hayes, L. A., Ireland, J., Mumford, S., Perez-Suarez, D., Ryan, D. F., et al. (2020) · The Astrophysical Journal · doi:10.3847/1538-4357/ab4f7a · Cited by 494
“…a community-led, free, and open source38 core Python package (sunpy), supports an ecosystem of affiliated packages (see Section 6) consistent with best practices (Wilson et al. 2014), and engages with the community through mailing lists, chat rooms, tutorials, summer programs, and mentorship.”
BackgroundCockett, R., Kang, S., Heagy, L. J., Pidlisecky, A., & Oldenburg, D. W. (2015) · Computers & Geosciences · doi:10.1016/j.cageo.2015.09.015 · Cited by 431
“Throughout its development, we have focused on modularity, usability, documentation, and extensive unit-testing (Wilson et al., 2014; Holscher et al., 2010; Kalderimis and Meyer, 2011; Merwin et al., 2015).”
BackgroundThe MIMIC Code Repository: enabling reproducibility in critical care research
Johnson, A. E. W., Stone, D. J., Celi, L. A., & Pollard, T. J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx084 · Cited by 442
Bayesian WorkflowPreprint
Gelman, A., Vehtari, A., Simpson, D., Margossian, C. C., Carpenter, B., Yao, Y., Kennedy, L., Gabry, J., Bürkner, P., & Modrák, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01808 · Cited by 708
Dalmaijer, E. S., Mathôt, S., & Van der Stigchel, S. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0422-2 · Cited by 381
PlantCV v2: Image analysis software for high-throughput plant phenotyping
Gehan, M. A., Fahlgren, N., Abbasi, A., Berry, J. C., Callen, S. T., Chavez, L., Doust, A. N., Feldman, M. J., Gilbert, K. B., Hodge, J. G., et al. (2017) · PeerJ · doi:10.7717/peerj.4088 · Cited by 365
Hamman, J. J., Nijssen, B., Bohn, T. J., Gergel, D. R., & Mao, Y. (2018) · Geoscientific Model Development · doi:10.5194/gmd-11-3481-2018 · Cited by 322
“…the use of established software development methods has grown in recent years in the fields of hydrology and the earth sciences to promote reproducibility of model and analysis results (e.g., Wilson et al., 2014; Ceola et al., 2015; Fienen and Bakker, 2016; Gil et al., 2016; Hutton et al., 2016).”
MethodA checklist for maximizing reproducibility of ecological niche models
Feng, X., Park, D. S., Walker, C., Peterson, A. T., Merow, C., & Papeş, M. (2019) · Nature Ecology & Evolution · doi:10.1038/s41559-019-0972-5 · Cited by 323
Automatic music transcription: challenges and future directions
Benetos, E., Dixon, S., Giannoulis, D., Kirchhoff, H., & Klapuri, A. (2013) · Journal of Intelligent Information Systems · doi:10.1007/s10844-013-0258-3 · Cited by 311
“Among them, spectrogram factorisation is rapidly growing in popularity and could potentially establish itself as the mainstream, even though at present a large number of approaches involve the use of signal processing and feature extraction based techniques.”
lab.js: A free, open, online study builder
Henninger, F., Shevchenko, Y., Mertens, U. K., Kieslich, P. J., & Hilbig, B. E. (2022) · Behavior Research Methods · doi:10.3758/s13428-019-01283-5 · Cited by 294
Smith, A. M., Katz, D. S., Niemeyer, K. E., & FORCE11 Software Citation Working Group (2016) · PeerJ Computer Science · doi:10.7717/peerj-cs.86 · Cited by 276
“Research results, including data, increasingly depend on software (Hannay et al., 2009), and thus may depend on the specific version used (Sandve et al., 2013; Wilson et al., 2014).”
BackgroundBurek, P., Satoh, Y., Kahil, T., Tang, T., Greve, P., Smilovic, M., Guillaumot, L., Zhao, F., & Wada, Y. (2020) · Geoscientific Model Development · doi:10.5194/gmd-13-3267-2020 · Cited by 260
“We hope that we have appropriately represented CWatM and its use of best practices in research software as stated in Wilson et al. (2014) and Jiménez et al. (2017).”
MethodThe Spack package manager: bringing order to HPC software chaos
Gamblin, T., LeGendre, M., Collette, M. R., Lee, G. L., Moody, A., de Supinski, B. R., & Futral, S. (2015) · Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/2807591.2807623 · Cited by 279
“Unfortunately, building scientific software is notoriously complex, with immature build systems that are difficult to adapt to new machines [13, 23, 42].”
BackgroundOur path to better science in less time using open data science tools
Lowndes, J. S. S., Best, B. D., Scarborough, C., Afflerbach, J. C., Frazier, M. R., O’Hara, C. C., Jiang, N., & Halpern, B. S. (2017) · Nature Ecology & Evolution · doi:10.1038/s41559-017-0160 · Cited by 286
“Coding is “as important to modern scientific research as telescopes and test tubes” 50 , but it is critical to “dispel the misconception that these skills are intuitive, obvious, or in any way inherent” 41 .”
BackgroundMethodA Large-Scale Study About Quality and Reproducibility of Jupyter Notebooks
Pimentel, J. F., Murta, L., Braganholo, V., & Freire, J. (2019) · 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2019.00077 · Cited by 246
“[9], scientists should write programs for people and not for computers.”
BackgroundSecondary Analysis of Electronic Health Records
Data, M. C. (2016) · OAPEN (The OAPEN Foundation) · doi:10.1007/978-3-319-43742-2 · Cited by 246
Marwick, B. (2017) · Journal of Archaeological Method and Theory · doi:10.1007/s10816-015-9272-9 · Cited by 179
“This method of interaction is a formidable obstacle to reproducibility because mouse gestures leave few traces that are enduring and accessible to others (Wilson et al .”
BackgroundMethodSimulating Illumina metagenomic data with InSilicoSeq
Gourlé, H., Karlsson-Lindsjö, O., Hayer, J., & Bongcam-Rudloff, E. (2019) · Bioinformatics · doi:10.1093/bioinformatics/bty630 · Cited by 244
Git can facilitate greater reproducibility and increased transparency in science
Ram, K. (2013) · Source Code for Biology and Medicine · doi:10.1186/1751-0473-8-7 · Cited by 230
Towards reproducible computational drug discovery
Schaduangrat, N., Lampa, S., Simeon, S., Gleeson, M. P., Spjuth, O., & Nantasenamat, C. (2020) · Journal of Cheminformatics · doi:10.1186/s13321-020-0408-x · Cited by 208
VIP: Vortex Image Processing Package for High-contrast Direct Imaging
Gomez Gonzalez, C. A., Wertz, O., Absil, O., Christiaens, V., Defrère, D., Mawet, D., Milli, J., Absil, P.-A., Van Droogenbroeck, M., Cantalloube, F., et al. (2017) · The Astronomical Journal · doi:10.3847/1538-3881/aa73d7 · Cited by 211
“The design and development of VIP follow modern practices for scientific software development such as code modularity, the active use of a version control system ( git ) and extensive documentation (Wilson et al. 2014 et al. 2014).”
MethodHow functional traits influence plant growth and shade tolerance across the life cycle
Falster, D. S., Duursma, R. A., & FitzJohn, R. G. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1714044115 · Cited by 204
Melton, J. R., Arora, V. K., Wisernig-Cojoc, E., Seiler, C., Fortier, M., Chan, E., & Teckentrup, L. (2020) · Geoscientific Model Development · doi:10.5194/gmd-13-2825-2020 · Cited by 167
The Tao of open science for ecology
Hampton, S. E., Anderson, S. S., Bagby, S. C., Gries, C., Han, X., Hart, E. M., Jones, M. B., Lenhardt, W. C., MacDonald, A., Michener, W. K., et al. (2015) · Ecosphere · doi:10.1890/es14-00402.1 · Cited by 187
“…description of a scientific process to enable meaningful public scrutiny and examination, with nothing intentionally obscured by technology or process version control: a system that manages snapshots (and hence “revisions” or “versioning”) of code and data for a project (Wilson et al. 2014).”
BackgroundA Quick Introduction to Version Control with Git and GitHub
Blischak, J. D., Davenport, E. R., & Wilson, G. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004668 · Cited by 192
“By tracking your code development with a VCS and hosting it online, you are performing science that is more transparent, reproducible, and open to collaboration [1,2].”
BackgroundSelf-citationTroubling Trends in Scientific Software Use
Joppa, L. N., McInerny, G., Harper, R., Salido, L., Takeda, K., O'Hara, K., Gavaghan, D., & Emmott, S. (2013) · Science · doi:10.1126/science.1231535 · Cited by 189
Inference in the age of big data: Future perspectives on neuroscience
Bzdok, D., & Yeo, B. T. T. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.04.061 · Cited by 189
Data Integration Challenges for Machine Learning in Precision Medicine
Martínez-García, M., & Hernández-Lemus, E. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2021.784455 · Cited by 185
Software Carpentry: lessons learned
Wilson, G. (2016) · F1000Research · doi:10.12688/f1000research.3-62.v2 · Cited by 174
Self-citationWhy don't we share data and code? Perceived barriers and benefits to public archiving practices
Gomes, D. G. E., Pottier, P., Crystal‐Ornelas, R., Hudgins, E. J., Foroughirad, V., Sánchez‐Reyes, L. L., Turba, R., Martinez, P. A., Moreau, D., Bertram, M. G., et al. (2022) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2022.1113 · Cited by 232
Enhancing Rigor in Quantitative Entrepreneurship Research
Maula, M., & Stam, W. (2020) · Entrepreneurship Theory and Practice · doi:10.1177/1042258719891388 · Cited by 167
“Additionally, similar to software programming, errors can occur easily (Wilson et al., 2014).”
MethodTools and techniques for computational reproducibility
Piccolo, S. R., & Frampton, M. B. (2016) · GigaScience · doi:10.1186/s13742-016-0135-4 · Cited by 169
“It is often also useful to store code in a version control system (VCS) [8, 9, 47], and to share it via Web-based services like GitHub.com or Bitbucket.org [52].”
BackgroundWhat's Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities
Chattopadhyay, S., Prasad, I., Henley, A. Z., Sarma, A., & Barik, T. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376729 · Cited by 159
Stodden, V., & Miguez, S. (2014) · Journal of Open Research Software · doi:10.5334/jors.ay · Cited by 178
Short communication: Landlab v2.0: a software package for Earth surface dynamics
Barnhart, K. R., Hutton, E. W. H., Tucker, G. E., Gasparini, N. M., İstanbulluoğlu, E., Hobley, D. E. J., Lyons, N. J., Mouchené, M., Nudurupati, S. S., Adams, J. M., et al. (2020) · Earth Surface Dynamics · doi:10.5194/esurf-8-379-2020 · Cited by 184
EQcorrscan: Repeating and Near‐Repeating Earthquake Detection and Analysis in Python
Chamberlain, C. J., Hopp, C. J., Boese, C. M., Warren‐Smith, E., Chambers, D., Chu, S. X., Michailos, K., & Townend, J. (2018) · Seismological Research Letters · doi:10.1785/0220170151 · Cited by 160
Reproducibility in Scientific Computing
Ivie, P., & Thain, D. (2018) · ACM Computing Surveys · doi:10.1145/3186266 · Cited by 198
Zwier, M. C., Adelman, J. L., Kaus, J. W., Pratt, A. J., Wong, K. F., Rego, N. B., Suárez, E., Lettieri, S., Wang, D. W., Grabe, M., et al. (2015) · Journal of Chemical Theory and Computation · doi:10.1021/ct5010615 · Cited by 154
Embracing heterogeneity: coalescing the Tree of Life and the future of phylogenomics
Bravo, G. A., Antonelli, A., Bacon, C. D., Bartoszek, K., Blom, M. P. K., Huynh, S., Jones, G., Knowles, L. L., Lamichhaney, S., Marcussen, T., et al. (2019) · PeerJ · doi:10.7717/peerj.6399 · Cited by 154
“Given their key role in phylogenomic research, it is advisable that software development, documentation, and availability follow the best possible practices (e.g., Leprevost et al., 2014;Wilson et al., 2014; Guang et al., 2016; Darriba, Flouri & Stamatakis, 2018; DeBiasse & Ryan, 2018).”
Ten Simple Rules for the Open Development of Scientific Software
Prlić, A., & Procter, J. B. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002802 · Cited by 141
PIC, a paediatric-specific intensive care database
Zeng, X., Yu, G., Lu, Y., Tan, L., Wu, X., Shi, S., Duan, H., Shu, Q., & Li, H. (2020) · Scientific Data · doi:10.1038/s41597-020-0355-4 · Cited by 138
Image Analysis in Plant Sciences: Publish Then Perish
Lobet, G. (2017) · Trends in Plant Science · doi:10.1016/j.tplants.2017.05.002 · Cited by 134
Four simple recommendations to encourage best practices in research software
Jiménez, R. C., Kuzak, M., Alhamdoosh, M., Barker, M., Batut, B., Borg, M., Capella-Gutierrez, S., Chue Hong, N., Cook, M., Corpas, M., et al. (2017) · F1000Research · doi:10.12688/f1000research.11407.1 · Cited by 140
“Strategies to increase software quality usually target software developers, focusing on training and adoption of best practices (Wilson et al., 2014).”
BackgroundInfluentialSwitch 2.0: A modern platform for planning high-renewable power systems
Johnston, J., Henríquez-Auba, R., Maluenda, B., & Fripp, M. (2019) · SoftwareX · doi:10.1016/j.softx.2019.100251 · Cited by 153
Using a suite of ontologies for preserving workflow-centric research objects
Belhajjame, K., Zhao, J., Garijo, D., Gamble, M., Hettne, K. M., Palma, R., Mina, E., Corcho, Ó., Gómez-Pérez, J. M., Bechhofer, S., et al. (2015) · Journal of Web Semantics · doi:10.1016/j.websem.2015.01.003 · Cited by 145
“Using a workflow, an experiment can be defined as a graph where the nodes represent analysis operations, which can be supplied locally or accessible remotely, and edges specify dependencies between the operations.”
BackgroundMethodThe Past, Present and Future of the ELKH Cloud
Héder, M., Rigó, E., Medgyesi, D., Lovas, R., Tenczer, S., Török, F., Farkas, A., Emődi, M., Kadlecsik, J., Mező, G., et al. (2022) · Információs Társadalom · doi:10.22503/inftars.xxii.2022.2.8 · Cited by 143
“In their seminal paper, Wilson et al. (2014) explored the landscape of scien - tific computing and its relationship to science in general.”
BackgroundDevelopment of Open-Source Porous Media Simulators: Principles and Experiences
Bilke, L., Flemisch, B., Kalbacher, T., Kolditz, O., Helmig, R., & Nagel, T. (2019) · Transport in Porous Media · doi:10.1007/s11242-019-01310-1 · Cited by 137
Swarm v3: towards tera-scale amplicon clustering
Mahé, F., Czech, L., Stamatakis, A., Quince, C., de Vargas, C., Dunthorn, M., & Rognes, T. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab493 · Cited by 137
Sustainable computational science: the ReScience initiative
Rougier, N. P., Hinsen, K., Alexandre, F., Arildsen, T., Barba, L. A., Benureau, F. C. Y., Brown, C. T., de Buyl, P., Caglayan, O., Davison, A. P., et al. (2017) · PeerJ Computer Science · doi:10.7717/peerj-cs.142 · Cited by 120
“…educational and methodological levels, things have started to change with a growing literature on best practices for making computations reproducible (Sandve, Nekrutenko, Taylor, & Hovig, 2013; Crook, Davison, & Plesser, 2013; Wilson et al., 2014; Halchenko & Hanke, 2015; Janz, 2015; Hinsen, 2015).”
Self-citationBornAgain : software for simulating and fitting grazing-incidence small-angle scattering
Pospelov, G., Herck, W. V., Burle, J., Loaiza, J. M. C., Durniak, C., Fisher, J. M., Ganeva, M., Yurov, D., & Wuttke, J. (2020) · Journal of Applied Crystallography · doi:10.1107/s1600576719016789 · Cited by 126
Approximate Bayesian Computation for infectious disease modelling
Minter, A., & Retkutė, R. (2019) · Epidemics · doi:10.1016/j.epidem.2019.100368 · Cited by 146
Ten simple rules for writing Dockerfiles for reproducible data science
Nüst, D., Sochat, V., Marwick, B., Eglen, S. J., Head, T., Hirst, T., & Evans, B. D. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008316 · Cited by 142
“This is an important part of common practice for scientific computing [3,4].”
BackgroundTen simple rules for making research software more robust
Taschuk, M., & Wilson, G. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005412 · Cited by 110
“Version control is essential to sustainable software development [17,18].”
BackgroundSelf-citationPackaging Data Analytical Work Reproducibly Using R (and Friends)
Marwick, B., Boettiger, C., & Mullen, L. A. (2017) · The American Statistician · doi:10.1080/00031305.2017.1375986 · Cited by 123
“We see evidence of this need in the proliferation of commentary and how-to publications across different areas of science (Wilson et al. 2014; Stodden and Miguez 2014; Stanisic and Legrand 2014; LeVeque, Mitchell, and Stodden 2012; Donoho 2010; Sandve et al. 2013; Rokem, Marwick, and Staneva 2017;…”
BackgroundToward standard practices for sharing computer code and programs in neuroscience
Eglen, S. J., Marwick, B., Halchenko, Y. O., Hanke, M., Sufi, S., Gleeson, P., Silver, R. A., Davison, A. P., Lanyon, L., Abrams, M. B., et al. (2017) · Nature Neuroscience · doi:10.1038/nn.4550 · Cited by 128
OpenCASA: A new open-source and scalable tool for sperm quality analysis
Alquézar-Baeta, C., Gimeno‐Martos, S., Miguel-Jiménez, S., Santolaria, P., Yániz, J. L., Palacín, I., Casao, A., Cebrián-Pérez, J. Á., Muiño-Blanco, T., & Pérez‐Pé, R. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006691 · Cited by 109
van Gent, P., Farah, H., van Nes, N., & van Arem, B. (2019) · Journal of Open Research Software · doi:10.5334/jors.241 · Cited by 112
Skills and Knowledge for Data-Intensive Environmental Research
Hampton, S. E., Jones, M. B., Wasser, L. A., Schildhauer, M. P., Supp, S. R., Brun, J., Hernandez, R. R., Boettiger, C., Collins, S. L., Gross, L. J., et al. (2017) · BioScience · doi:10.1093/biosci/bix025 · Cited by 105
“A fundamental concept of clean, well-organized code is the don’t repeat yourself (DRY) principle (Wilson et al. 2014 ).”
BackgroundMethodSelf-citationHagen, O., Flück, B., Fopp, F., Cabral, J. S., Hartig, F., Pontarp, M., Rangel, T. F., & Pellissier, L. (2021) · PLoS Biology · doi:10.1371/journal.pbio.3001340 · Cited by 119
“Gen3sis follows best practices for scientific computing [71], including high modularization; consistent naming, style, and formatting; single and meaningful authoritative representation; automated workflows; version control; continuous integration; and extensive documentation.”
BackgroundMetaresearch for Evaluating Reproducibility in Ecology and Evolution
Fidler, F., Chee, Y. E., Wintle, B. C., Burgman, M. A., McCarthy, M. A., & Gordon, A. (2017) · BioScience · doi:10.1093/biosci/biw159 · Cited by 94
FRETBursts: An Open Source Toolkit for Analysis of Freely-Diffusing Single-Molecule FRET
Ingargiola, A., Lerner, E., Chung, S., Weiss, S., & Michalet, X. (2016) · PLoS ONE · doi:10.1371/journal.pone.0160716 · Cited by 98
Towards Effective Foraging by Data Scientists to Find Past Analysis Choices
Kery, M. B., John, B. E., O'Flaherty, P., Horvath, A., & Myers, B. A. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300322 · Cited by 95
“It may help the data scientists be more aware and productive if, just as natural scientists keep a lab notebook of their experiments, they too had support to quickly record and reference what they already tried, under what assumptions, and with what results [27, 31].”
BackgroundBetter Software, Better Research
Goble, C. (2014) · IEEE Internet Computing · doi:10.1109/mic.2014.88 · Cited by 121
Sequenceserver: a modern graphical user interface for custom BLAST databasesPreprint
Priyam, A., Woodcroft, B. J., Rai, V., Munagala, A., Moghul, I., Ter, F., Gibbins, M. A., Moon, H., Leonard, G., Rumpf, W., et al. (2015) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/033142 · Cited by 107
Giannoulatou, E., Park, S. H., Humphreys, D. T., & Ho, J. W. K. (2014) · BMC Bioinformatics · doi:10.1186/1471-2105-15-s16-s15 · Cited by 108
“Most importantly, there have been numerous published papers that attempt to train scientists to adopt best practices for scientific computing [13-17].”
BackgroundProceedings of the 15th Python in Science Conference
(2016) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-629e541a-014 · Cited by 115
PanACoTA: A modular tool for massive microbial comparative genomicsPreprint
Perrin, A., & Rocha, E. P. C. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1093/nargab/lqaa106 · Cited by 124
“PanACoTA was developed in Python3, trying to follow the best practices for scientific software development (53,54).”
BackgroundPerceived Corruption and Individuals’ Life Satisfaction: The Mediating Role of Institutional Trust
Ciziceno, M., & Travaglino, G. A. (2018) · Social Indicators Research · doi:10.1007/s11205-018-1850-2 · Cited by 117
“1 To run analyses with software other than SPSS (Wilson et al. 2014), data from study 1 are available from the corresponding author upon request, whereas data from study 2 are available online at http://www.worldvaluessurvey.org/wvs.jsp.”
MethodComputational Chemistry Methods for Nanoporous Materials
Evans, J. D., Fraux, G., Gaillac, R., Kohen, D., Trousselet, F., Vanson, J.-M., & Coudert, F. (2016) · Chemistry of Materials · doi:10.1021/acs.chemmater.6b02994 · Cited by 92
Understanding and improving the quality and reproducibility of Jupyter notebooks
Pimentel, J. F., Murta, L., Braganholo, V. P., & Freire, J. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09961-9 · Cited by 128
“…reinforce prior work, which has emphasized the negative impact of the lack of best practices of Software Engineering in scientific computing software (Wilson et al. 2014), regarding separation of concerns (Hürsch and Lopes 1995), tests (Myers et al. 2004), and maintenance (Horwitz and Reps 1992).”
BackgroundReproducibility in Neuroimaging Analysis: Challenges and Solutions
Botvinik‐Nezer, R., & Wager, T. D. (2022) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2022.12.006 · Cited by 116
NG-meta-profiler: fast processing of metagenomes using NGLess, a domain-specific language
Coelho, L. P., Alves, R., Monteiro, P., Huerta-Cepas, J., Freitas, A. T., & Bork, P. (2019) · Microbiome · doi:10.1186/s40168-019-0684-8 · Cited by 87
Measuring the microbiome: Best practices for developing and benchmarking microbiomics methods
Bokulich, N. A., Ziemski, M., Robeson, M. S., & Kaehler, B. D. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.11.049 · Cited by 114
“Write simple, well-commented code that does things well and is understandable by other people, rather than devise a solution that is over-optimized but hard to follow due to its complexity [174].”
BackgroundTeaching Creative and Practical Data Science at Scale
Donoghue, T., Voytek, B., & Ellis, S. (2020) · Journal of Statistics and Data Science Education · doi:10.1080/10691898.2020.1860725 · Cited by 102
“Within projects, students must have opportunities to write good and well-documented code that accomplishes their analysis goals (Wilson et al. 2014, 2017).”
BackgroundElevating The Status of Code in Ecology
Mislan, K. A. S., Heer, J., & White, E. P. (2015) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2015.11.006 · Cited by 85
Self-citationA “datathon” model to support cross-disciplinary collaboration
Aboab, J., Celi, L. A., Charlton, P. H., Feng, M., Ghassemi, M. M., Marshall, D. C., Mayaud, L., Naumann, T., McCague, N., Paik, K., et al. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aad9072 · Cited by 89
The do’s and don’ts of infrastructure code: A systematic gray literature review
Kumara, I., Garriga, M., Romeu, A. U., Nucci, D. D., Palomba, F., Tamburri, D. A., & van den Heuvel, W. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106593 · Cited by 115
“For the candidate practice categories, we used the practice categories proposed in the existing literature [48, 49, 50, 51, 52, 53].”
Eilmer: An open-source multi-physics hypersonic flow solver
Gibbons, N., Damm, K. A., Jacobs, P. A., & Gollan, R. (2022) · Computer Physics Communications · doi:10.1016/j.cpc.2022.108551 · Cited by 77
“[25] that call for scientific computing to be more responsibly conducted and better scrutinised, noting as they do that the risk of mistakes in our field is high and the consequences potentially severe.”
BackgroundshinyGEO: a web-based application for analyzing gene expression omnibus datasets
Dumas, J., Gargano, M. A., & Dancik, G. M. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw519 · Cited by 84
“The user can also export the data and generate the R code used for each analysis, following best practices in scientific computing (Wilson et al. , 2014) to ensure reproducibility.”
MethodDeveloping an automated iterative near‐term forecasting system for an ecological study
White, E. P., Yenni, G. M., Taylor, S. D., Christensen, E. M., Bledsoe, E. K., Simonis, J. L., & Ernest, S. K. M. (2018) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13104 · Cited by 85
Self-citationAfrica, P. C. (2022) · SoftwareX · doi:10.1016/j.softx.2022.101252 · Cited by 77
Creating and sharing reproducible research code the workflowr way
Blischak, J. D., Carbonetto, P., & Stephens, M. (2019) · F1000Research · doi:10.12688/f1000research.20843.1 · Cited by 77
The Case for Open Preprints in Biology
Desjardins-Proulx, P., White, E. P., Adamson, J. J., Ram, K., Poisot, T., & Gravel, D. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001563 · Cited by 73
“Git is primarily used for software development [4] but it provides a powerful tool way to collaborate on every step of the manuscript development process [16].”
MethodSelf-citationExperimenting with reproducibility: a case study of robustness in bioinformatics
Kim, Y.-M., Poline, J., & Dumas, G. (2018) · GigaScience · doi:10.1093/gigascience/giy077 · Cited by 94
“…by suggesting tools and best practices following the programming best practices of Wilson et al., such as modularizing and re-using code, unit testing, document design, data management, and project organization [2, 27], as well as keeping data provenance and recording all intermediate results [28].”
Containers for computational reproducibility
Moreau, D., Wiebels, K., & Boettiger, C. (2023) · Nature Reviews Methods Primers · doi:10.1038/s43586-023-00236-9 · Cited by 121
“Proper commenting and documentation can help make containers more readable and maintainable and facilitate the use and utility of containerized software 108 .”
BackgroundMoulds, S., Buytaert, W., & Mijić, A. (2015) · Geoscientific model development · doi:10.5194/gmd-8-3215-2015 · Cited by 80
“Acknowledging that (Joppa et al., 2013; Wilson et al., 2014), we hope this final goal will ensure the software is useful for educational purposes as well as scientific research.”
Metadata
- Authors
- Wilson, G., Aruliah, D. A., Brown, C. T., Chue Hong, N. P., Davis, M., Guy, R. T., Haddock, S. H. D., Huff, K. D., Mitchell, I. M., Plumbley, M. D., Waugh, B., White, E. P., & Wilson, P.
- Affiliations
- Mozilla Foundation (US)
- Ontario Tech University (CA)
- Michigan State University (US)
- Space Telescope Science Institute (US)
- University of Toronto (CA)
- Monterey Bay Aquarium Research Institute (US)
- University of California, Berkeley (US)
- University of British Columbia (CA)
- Queen Mary University of London (GB)
- University College London (GB)
- Utah State University (US)
- University of Wisconsin–Madison (US)
- Corresponding author
- Wilson, Greg
- ORCID
- Venue
- PLoS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-01-07
- Volume
- 12
- Issue
- 1
- Pages
- e1001745
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC3886731
- OpenAlex
- W2131966673
- Cited by
- 723
- References
- 75
- Citations by year
- 2026: 63 · 2025: 28 · 2024: 49 · 2023: 42 · 2022: 56 · 2021: 51 · 2020: 70 · 2019: 69 · 2018: 67 · 2017: 62 · 2016: 63 · 2015: 54 · 2014: 27
- FWCI
- 128.23
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion, cc-by)
- Surrey Research Insight Open Access (The University of Surrey) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (0953694, CAREER, EP/H043160/1)
- Alfred P. Sloan Foundation
- Natural Sciences and Engineering Research Council of Canada
- Engineering and Physical Sciences Research Council (EP/H043160/1, EP/H043101/1, EP/G007144/1)
- National Human Genome Research Institute (R01HG007513)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
- Keywords
- scientific computing, best practices
- MeSH terms
- Computing Methodologies, Humans, Software, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.