Best Practices for Scientific Computing
Open accessThe DOI matches its Crossref record; 73 of the 75 references it cites were found in a registry or an index.
Abstract
The Community Page is a forum for organizations and societies
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The creation of MIMIC-IV followed best practices in scientific computing23.”
From MIMIC-IV, a freely accessible electronic health record dataset (2023) · cites this paper as [23]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
5 retracted
Chang, G. A., Roth, C., Reyes, C. L., Pornillos, O., Chen, Y.-J., & Chen, A. P. (2006) · Science · doi:10.1126/science.314.5807.1875b · Cited by 137
RetractedRETRACTED: Structure of the multidrug resistance efflux transporter EmrE from Escherichia coli
Ma, C., & Chang, G. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0400137101 · Cited by 102
RetractedChang, G. A. (2007) · Journal of Molecular Biology · doi:10.1016/j.jmb.2003.05.001 · Cited by 22
RetractedMa, C., & Chang, G. A. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0700711104 · Cited by 18
RetractedHuijgen, R., Boekholdt, S. M., Arsenault, B. J., Bao, W., Davaine, J., Tabet, F., Petrides, F., Rye, K.-A., DeMicco, D. A., Barter, P. J., et al. (2013) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.04.002 · Cited by 1
RetractedIPython: A System for Interactive Scientific Computing
Perez, F., & Granger, B. E. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.53 · Cited by 4,009 · cited 1×
“Common approaches to this include this use of knitr [63] and IPython Notebooks [64].”
MethodCockburn, A. (2001) · UCLA - Biblioteca de Ciencias y Tecnologia Felix Morales Bueno · Cited by 1,858
Verification and Validation in Scientific Computing
Oberkampf, W. L., & Roy, C. J. (2010) · doi:10.1017/cbo9780511760396 · Cited by 1,556 · cited 1×
“Exactly how they check correctness depends on the researcher’s understanding of the problem at hand [24, 32, 46].”
BackgroundDesign and code inspections to reduce errors in program development
Fagan, M. E. (1976) · IBM Systems Journal · doi:10.1147/sj.153.0182 · Cited by 1,492
Agile Software Development, Principles, Patterns, and Practices
Martin, R. (2002) · Cited by 1,466
Research works printed but not linked: 4
- Z Merali, 2010, Error: why scientific programming does not compute, Nature, 467, 775
- Baddeley A, Eysenck MW, Anderson MC (2009) Memory. New York: Psychology Press.
- Ray DS, Ray EJ (2009) Unix and Linux: visual quickstart guide. 4th edition. San Francisco: Peachpit Press.
- Kelly D, Sanders R (2008) Assessing the quality of scientific software. In: Proceedings First International Workshop on Software Engineering for Computational Science and Engineering. Available: http://secse08.cs.ua.edu/Papers/Kelly.pdf.
Cited by
Items 201–300 of 727
Using prototyping to choose a bioinformatics workflow management system
Jackson, M., Wallace, E. W. J., & Kavoussanakis, K. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008622 · Cited by 29
“The development of RiboViz has also been strongly guided by Wilson and colleagues’ best practices for scientific computing [17].”
MethodLawlor, B., & Walsh, P. (2015) · Bioengineered · doi:10.1080/21655979.2015.1050162 · Cited by 28
“with respect to those choices.(15) Figure 1 shows the essential elements of software engineering practice, and illustrates the dependencies between them.”
BackgroundToward open science in marketing research
Deer, L., Adler, S. J., Datta, H., Mizik, N., & Sarstedt, M. (2024) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2024.12.005 · Cited by 28
General guidelines for biomedical software development
Silva, L. A. B., Jiménez, R. C., Blomberg, N., & Oliveira, J. L. (2017) · F1000Research · doi:10.12688/f1000research.10750.2 · Cited by 30
Dolgopolovas, V., Jevsikova, T., & Dagienė, V. (2017) · Computer Applications in Engineering Education · doi:10.1002/cae.21862 · Cited by 27
“Another important field which should be mentioned and which is widely used in various engineering educational programs is scientific computing [35], where C programming is extremely important.”
BackgroundSoftware engineering for scientific big data analysis
Grüning, B. A., Lampa, S., Vaudel, M., & Blankenberg, D. J. (2019) · GigaScience · doi:10.1093/gigascience/giz054 · Cited by 29
Eriksen, J. J. (2016) · Molecular Physics · doi:10.1080/00268976.2016.1271155 · Cited by 27
GPU Computing with Python: Performance, Energy Efficiency and Usability
Holm, H. H., Brodtkorb, A. R., & Sætra, M. L. (2020) · Computation · doi:10.3390/computation8010004 · Cited by 27
“Researchers spend a large portion of their time writing computer programs [36], and compiled languages such as C/C++ and Fortran have been the de facto standard within scientific computing for decades.”
BackgroundInstitutionalizing Transparency
Jeremy Freese, A., & King, M. M. (2018) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023117739216 · Cited by 27
“Better still is that the resources available to learn contemporary practices are now abundant, including many wise papers about how to integrate better practices into social science work flow (e.g., Christensen 2016; Gentzkow and Shapiro 2014; Healy 2016; Long 2009; Wilson et al. 2014).”
BackgroundPredicting rock type from MWD tunnel data using a reproducible ML-modelling process
Hansen, T. F., Liu, Z., & Tørresen, J. (2024) · Tunnelling and Underground Space Technology · doi:10.1016/j.tust.2024.105843 · Cited by 31
Modflow-setup: Robust automation of groundwater model construction
Leaf, A. T., & Fienen, M. N. (2022) · Frontiers in Earth Science · doi:10.3389/feart.2022.903965 · Cited by 30
“workflows takes time (e.g., Donoho et al., 2008; Wilson et al., 2014) that is most efficiently spent developing general code that can be reused in many different contexts.”
BackgroundHow to Share Data for Collaboration
Ellis, S., & Leek, J. T. (2017) · The American Statistician · doi:10.1080/00031305.2017.1375987 · Cited by 37
USID and Pycroscopy -- Open frameworks for storing and analyzing spectroscopic and imaging dataPreprint
Somnath, S., Smith, C. R., Laanait, N., Vasudevan, R. K., Ievlev, A. V., Belianinov, A., Lupini, A. R., Shankar, M. A., Kalinin, S. V., & Jesse, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.09515 · Cited by 32
Predicting microbial community structure and temporal dynamics by using graph neural network models
Andersen, K. S., Zhao, K., Agerskov, A. D. L., Sørensen, C. B., Holmager, T. J., Nierychlo, M., Peces, M., Guo, C., & Nielsen, P. H. (2025) · Nature Communications · doi:10.1038/s41467-025-64175-7 · Cited by 25
Data Sharing and Scientific Impact in Eddy Covariance Research
Bond‐Lamberty, B. (2018) · Journal of Geophysical Research Biogeosciences · doi:10.1002/2018jg004502 · Cited by 26
“The use of best practices (Marwick et al., 2017; Wilson et al., 2014) can ameliorate, but not currently solve, this fundamental problem of scientific computing.”
BackgroundVaST: A variability search toolkit
Sokolovsky, K. V., & Lebedev, A. A. (2017) · Astronomy and Computing · doi:10.1016/j.ascom.2017.12.001 · Cited by 37
“Some of the best practices for scientific computing were reviewed by Wilson et al. (2012).”
BackgroundDeveloping Open-Source Scientific Practice *
Millman, K. J., & Pérez, F. (2018) · Implementing Reproducible Research · doi:10.1201/9781315373461-6 · Cited by 26
“For a more detailed discussion, we recommend [17, 13, 1].”
BackgroundComputational reproducibility in computational social science
Schoch, D., Chan, C.-H., Wagner, C., & Bleier, A. (2024) · EPJ Data Science · doi:10.1140/epjds/s13688-024-00514-w · Cited by 36
“Researcher can probably write executable code but have varying skills in standard software development practices such as using unit tests and continuous integration [Wilson et al., 2014] to show correctness.”
Open-source Tools in R for Landscape Ecology
Hesselbarth, M. H. K., Nowosad, J., Signer, J., & Graham, L. (2021) · Current Landscape Ecology Reports · doi:10.1007/s40823-021-00067-y · Cited by 32
Using interactive digital notebooks for bioscience and informatics education
Davies, A., Hooley, F., Freeman, P., Eleftheriou, I., & Moulton, G. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008326 · Cited by 25
“There are calls for journal editors and reviewers to enforce computational reproducibility [33]; however, many scientists use and write code on a regular basis but often lack formal training in good software engineering practices [34].”
BackgroundDeveloping educational resources for population genetics in R: an open and collaborative approach
Kamvar, Z. N., López-Uribe, M. M., Coughlan, S., Grünwald, N. J., Lapp, H., & Manel, S. (2016) · Molecular Ecology Resources · doi:10.1111/1755-0998.12558 · Cited by 27
“…vignettes in the text-based R Markdown format (Baumer et al. 2014); using continuous integration for automatically testing and building the site (Wilson et al. 2014); using Docker containers for doing so reproducibly (Boettiger 2015); and using GitHub pull requests for open peer review of all…”
MethodBest publishing practices to improve user confidence in scientific software
Poisot, T. (2015) · Ideas in Ecology and Evolution · doi:10.4033/iee.2015.8.8.f · Cited by 24
Harnessing Big Data in Critical Care: Exploring a new European Dataset
Rodemund, N., Wernly, B., Jung, C., Cozowicz, C., & Koköfer, A. (2024) · Scientific Data · doi:10.1038/s41597-024-03164-9 · Cited by 24
Where next for the reproducibility agenda in computational biology?
Lewis, J., Breeze, C. E., Charlesworth, J., Maclaren, O. J., & Cooper, J. (2016) · BMC Systems Biology · doi:10.1186/s12918-016-0288-x · Cited by 23
“Aspects of standard good practice in software development [7, 8] (version control, modular design) also make research more reproducible.”
BackgroundMethodThe Challenge and Promise of Software Citation for Credit, Identification, Discovery, and Reuse
Niemeyer, K. E., Smith, A. M., & Katz, D. S. (2016) · Journal of Data and Information Quality · doi:10.1145/2968452 · Cited by 28
“Provenance of research results and data requires, among other things, a record of the software used to generate or process that data [Sandve et al. 2013; Wilson et al. 2014].”
MethodAn open-source spatio-dynamic wetland model of plant community responses to hydrological pressures
Martínez‐López, J., Fernández, J. M., Naimi, B., Carreño, F., & Selma, M. Á. E. (2014) · Ecological Modelling · doi:10.1016/j.ecolmodel.2014.11.024 · Cited by 23
Source Code Level Word Embeddings in Aiding Semantic Test-to-Code Traceability
Csuvik, V., Kicsi, A., & Vidács, L. (2019) · IEEE/ACM International Symposium on Software and Systems Traceability (SST) · doi:10.1109/sst.2019.00016 · Cited by 23
“For more information about good coding practices, please refer to [5].”
BackgroundA framework for streamlining research workflow in neuroscience and psychology
Kubilius, J. (2014) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2013.00052 · Cited by 22
Running an open experiment: transparency and reproducibility in soil and ecosystem science
Bond‐Lamberty, B., Smith, A. P., & Bailey, V. (2016) · Environmental Research Letters · doi:10.1088/1748-9326/11/8/084004 · Cited by 23
Thinking Too Classically: Research Topics in Human-Quantum Computer Interaction
Ashktorab, Z., Weisz, J. D., & Ashoori, M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300486 · Cited by 26
Water Science Software Institute: Agile and Open Source Scientific Software Development
Ahalt, S. C., Band, L., Christopherson, L., Idaszak, R., Lenhardt, W. C., Minsker, B. S., Palmer, M. A., Shelley, M., Tiemann, M., & Zimmerman, A. (2014) · Computing in Science & Engineering · doi:10.1109/mcse.2014.5 · Cited by 22
"Who Has Plots?": Contextualizing Scientific Software, Practice, and Visualizations
Paine, D., & Lee, C. P. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134720 · Cited by 35
“…there are significant cultural differences between scientists and software engineers (SEs) [21, 27, 45, 56], much discourse about scientific software development has prioritized identifying “missing” SE practices and teaching scientists to develop software like software engineers [42, 59, 61, 62].”
BackgroundModel-R: A Framework for Scalable and Reproducible Ecological Niche Modeling
Sánchez‐Tapia, A., de Siqueira, M. F., Lima, R. O., Barros, F. S. M., Gall, G., Gadelha, L. M. R., Jr., da Silva, L. A. E., & Osthoff, C. (2017) · Communications in computer and information science · doi:10.1007/978-3-319-73353-1_15 · Cited by 23
“The inclusion of this feature is motivated by many academic journals recommending that authors of computational studies should also provide the required data sets, tools, and workflows used to generate the results [19] [20] so that reviewers and readers could better validate them.”
Backgroundautoman: A Python-Based Automation Framework for Numerical Computing
Ramachandran, P. (2018) · Computing in Science & Engineering · doi:10.1109/mcse.2018.05329818 · Cited by 23
“[11] provide very general guidelines to make this workflow more productive and reliable.”
BackgroundTen simple rules for quick and dirty scientific programming
Balaban, G., Grytten, I., Rand, K. D., Scheffer, L., & Sandve, G. K. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008549 · Cited by 30
Choices and trade-offs in inference with infectious disease models
Funk, S., & King, A. A. (2019) · Epidemics · doi:10.1016/j.epidem.2019.100383 · Cited by 26
Visualising data science workflows to support third-party notebook comprehension: an empirical study
Ramasamy, D., Sarasua, C., Bacchelli, A., & Bernstein, A. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10289-9 · Cited by 31
“By augmenting previous guidelines for making scientific analyses reproducible (Wilson et al. 2014), Rule et al. (2019) developed a set of ten rules in 2019, particularly for computational notebooks like Jupyter.”
MethodTeaching Computational Reproducibility for Neuroimaging
Millman, J., Brett, M., Barnowski, R., & Poline, J. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00727 · Cited by 24
“The best known may be Software Carpentry (Wilson, 2014).”
BackgroundOn the evaluation of research software: the CDUR procedure
Gomez-Diaz, T., & Recio, T. (2019) · F1000Research · doi:10.12688/f1000research.19994.2 · Cited by 33
Why do biomedical researchers learn to program? An exploratory investigation
Deardorff, A. (2020) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2020.819 · Cited by 20
“Unfortunately, many scientists do not learn to program as part of their graduate training and must seek additional courses and workshops to use these new techniques [5].”
BackgroundNine best practices for research software registries and repositories
Garijo, D., Ménager, H., Hwang, L., Trišović, A., Hucka, M., Morrell, T. E., Allen, A., & Task Force on Best Practices for Software Registries (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1023 · Cited by 32
“INTRODUCTION Research software is an essential constituent in scientific investigations (Wilson et al., 2014; Momcheva & Tollerud, 2015; Hettrick, 2018; Lamprecht et al., 2020), as it is often used to transform and prepare data, perform novel analyses on data, automate manual processes, and visualize results reported…”
MethodBirkel, G. W., Ghosh, A., Kumar, V. S., Weaver, D. S., Ando, D., Backman, T. W. H., Arkin, A. P., Keasling, J. D., & Martín, H. G. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1615-y · Cited by 20
“In order to facilitate reproducibility, the lack of which represents one of the main problems afflicting biological research [32], all results and tutorials are provided in the form of Jupyter notebooks [33, 34] (see Table 1 and jupyter.org).”
BackgroundSoftware and the Scientist: Coding and Citation Practices in Geodynamics
Hwang, L., Fish, A. E., Soito, L., Smith, M., & Kellogg, L. H. (2017) · Earth and Space Science · doi:10.1002/2016ea000225 · Cited by 20
“…standard, and target best practices cover version control, coding, portability of con fi guration and build, testing, Earth and Space Science 10.1002/2016EA000225 documentation, and user work fl ow in line with current thoughts on best practices for scienti fi c computing (Wilson et al., 2014).”
BackgroundInfluentialThe basic matrix library (BML) for quantum chemistry
Bock, N., Negre, C. F. A., Mniszewski, S. M., Mohd-Yusof, J., Aradi, B., Fattebert, J., Osei-Kuffuor, D., Germann, T. C., & Niklasson, A. M. N. (2018) · The Journal of Supercomputing · doi:10.1007/s11227-018-2533-0 · Cited by 20
SFRmaker and Linesink‐Maker: Rapid Construction of Streamflow Routing Networks from Hydrography Data
Leaf, A. T., Fienen, M. N., & Reeves, H. W. (2021) · Ground Water · doi:10.1111/gwat.13095 · Cited by 20
“In developing and maintaining these programs, we have sought to leverage open-source software tools and best practices (e.g., Wilson et al. 2014; https://nsls-ii.github.io/ scientific-python-cookiecutter) to improve the robustness, quality, transparency, and maintainability of the code.”
MethodCollaborative experience between scientific software projects using Agile Scrum development
Baxter, A. L., BenZvi, S., Bonivento, W. M., Brazier, A., Clark, M., Coleiro, A., Collom, D., Molla, M. C., Cousins, B., Orellana, A. D., et al. (2022) · Software Practice and Experience · doi:10.1002/spe.3120 · Cited by 31
Teaching introductory GIS programming to geographers using an open source Python approach
Etherington, T. R. (2015) · Journal of Geography in Higher Education · doi:10.1080/03098265.2015.1086981 · Cited by 20
“…is both worrying, as scienti fi c fi ndings may become erroneous when code is not well tested or documented (Merali, 2010), and unfortunate as some simple formal training in basic programming best practices relating to design and documentation could help to avoid many issues (Wilson et al., 2014).”
BackgroundCPL library — A minimal framework for coupled particle and continuum simulation
Smith, E. R., Trevelyan, D., Fernández, E. R., Sufian, A., O’Sullivan, C., & Dini, D. (2019) · Computer Physics Communications · doi:10.1016/j.cpc.2019.107068 · Cited by 19
Improving bioinformatics software quality through incorporation of software engineering practices
Noor, A. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.839 · Cited by 26
“This phenomenon is further supported by quantities of research showing this lack of education and training among bioinformatics engineers (Baxter et al., 2006; Hannay et al., 2009; Merali, 2010; Joppa et al., 2013; Wilson, 2014; Wilson et al., 2014; Lawlor & Walsh, 2015; List, Ebert & Albrecht, 2017; Mulder et al., 20…”
BackgroundResultRampant software errors undermine scientific results
Soergel, D. A. W. (2014) · F1000Research · doi:10.12688/f1000research.5930.1 · Cited by 25
Guidelines for the estimation and reporting of plasmid conjugation rates
Kosterlitz, O., & Huisman, J. S. (2023) · Plasmid · doi:10.1016/j.plasmid.2023.102685 · Cited by 23
How to Professionally Develop Reusable Scientific Software—And When Not To
Adorf, C. S., Ramasubramani, V., Anderson, J. A., & Glotzer, S. C. (2018) · Computing in Science & Engineering · doi:10.1109/mcse.2018.2882355 · Cited by 19
“& AS COMPUTATIONAL SCIENCE continues to play a rapidly growing role across nearly all scientific disciplines, quality scientific software becomes ever more critical to research productivity.(1) The development of such software is impeded by the fact that most scientific software is developed by nonexperts for whom the…”
BackgroundNguyen, S. L., Greenberg, J. W., Wang, H., Collaer, B. W., Wang, J., & Petroff, M. G. (2019) · PLoS ONE · doi:10.1371/journal.pone.0218270 · Cited by 20
Ten recommendations for software engineering in research
Hastings, J., Haug, K., & Steinbeck, C. (2014) · GigaScience · doi:10.1186/2047-217x-3-31 · Cited by 19
OPEN SOURCE APPROACH TO URBAN GROWTH SIMULATION
Petrášová, A., Petráš, V., Berkel, D. V., Harmon, B. A., Mitášová, H., & Meentemeyer, R. K. (2016) · The international archives of the photogrammetry, remote sensing and spatial information sciences/International archives of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprs-archives-xli-b7-953-2016 · Cited by 19
How Scientists Use Large Language Models to Program
O’Brien, G. E. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713668 · Cited by 42
“Existing communities that are professionalizing research software and up-skilling scientific developers [13, 91, 104, 105] may be an ideal locus for discussions about what Code LLM practices should be acceptable in science, what code quality-checking mechanisms should exist, and how to advocate for…”
bertha: Project skeleton for scientific software
Riesch, M., Nguyen, T. D., & Jirauschek, C. (2020) · PLoS ONE · doi:10.1371/journal.pone.0230557 · Cited by 25
“By using a VCS, the “Make Incremental Changes” paradigm [5, 6] can be implemented easily and the intrinsically generated development history may serve as rudimentary documentation of design decisions [4].”
BackgroundMethodInfluentialResponsible modelling: Unit testing for infectious disease epidemiology
Lucas, T. C. D., Pollington, T. M., Davis, E. L., & Hollingsworth, T. D. (2020) · Epidemics · doi:10.1016/j.epidem.2020.100425 · Cited by 18
Ellipsoids (v1.0): 3-D magnetic modelling of ellipsoidal bodies
Takahashi, D., & Oliveira, V. C. (2017) · Geoscientific model development · doi:10.5194/gmd-10-3591-2017 · Cited by 20
“We refer the readers interested in best practices for scientific computing to Wilson et al. (2014) The numerical simulations presented here were generated with the Jupyter Notebook (http://jupyter.org), which is a web application that allows creating and sharing documents that contain live code,…”
MethodA Grid for Multidimensional and Multivariate Spatial Representation and Data Processing
Stål, T., & Reading, A. M. (2020) · Journal of Open Research Software · doi:10.5334/jors.287 · Cited by 18
A practical guide to writing a radiative transfer code
Korkin, S. V., Sayer, A. M., Ibrahim, A. I., & Lyapustin, A. (2021) · Computer Physics Communications · doi:10.1016/j.cpc.2021.108198 · Cited by 18
Loenneker, H. D., Buchanan, E. M., Martinovici, A., Primbs, M. A., Elsherif, M. M., Baker, B. J., Dudda, L. A., Đurđević, D. F., Mišić, K., Peetz, H. K., et al. (2023) · Cortex · doi:10.1016/j.cortex.2023.11.012 · Cited by 22
A Survey of Open Source Multiphysics Frameworks in Engineering
Babur, Ö., Šmilauer, V., Verhoeff, T. T., & van den Brand, M. (2015) · Procedia Computer Science · doi:10.1016/j.procs.2015.05.273 · Cited by 17
“A considerable number of teams are apparently using software engineering tools such as source code repositories and bug trackers, which are considered to be among the best practices for scientific computing [17].”
BackgroundPerspective on coarse-graining, cognitive load, and materials simulation
Jankowski, E., Ellyson, N., Fothergill, J. W., Henry, M. M., Leibowitz, M. H., Miller, E. D., Alberts, M., Chesser, S., Guevara, J. D., JONES, C. D., et al. (2019) · Computational Materials Science · doi:10.1016/j.commatsci.2019.109129 · Cited by 19
“Recurring themes in recommended readings [72,27,73] around best practices and considerations for community-driven scientific software development include:”
BackgroundBionitio: demonstrating and facilitating best practices for bioinformatics command-line software
Georgeson, P., Syme, A., Sloggett, C., Chung, J., Dashnow, H., Milton, M., Lonsdale, A., Powell, D., Seemann, T., & Pope, B. J. (2019) · GigaScience · doi:10.1093/gigascience/giz109 · Cited by 17
“It is typically better to find an established library or package that solves a problem than to attempt to write one’s own routines for well-established problems [3], and this also improves reusability [10].”
Rademacher, T., Seyednasrollah, B., Basler, D., Cheng, J., Mandra, T., Miller, E. W., Lin, Z., Orwig, D. A., Pederson, N., Pfister, H., et al. (2021) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13717 · Cited by 18
Zaytsev, Y. V., & Morrison, A. (2013) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2012.00031 · Cited by 17
Figueiredo, L., Scherer, C., & Cabral, J. S. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010356 · Cited by 19
“Our kit adds to the efforts of Wilson and colleagues in promoting the “best” [9] or at least “good enough” [10] practices in scientific computing.”
MethodKim, A. Y., Herrmann, V., Barreto, R., Calkins, B., Gonzalez‐Akre, E. B., Johnson, D. J., Jordan, J. A., Magee, L. J., McGregor, I. R., Montero, N., et al. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13982 · Cited by 16
“Needed are safeguards to test the reproducibility of this pipeline that both provide timely notifications of break-downs and require a minimal number of active human interventions (Beaulieu-Jones & Greene, 2017; Stodden et al., 2016; Stodden & Miguez, 2014; Wilson et al., 2014).”
BackgroundDistant Viewing Toolkit: A Python Package for the Analysis of Visual Culture
Arnold, T., & Tilton, L. (2020) · The Journal of Open Source Software · doi:10.21105/joss.01800 · Cited by 25
“Development of the Distant Viewing Toolkit follows best-practices for open-source software development (Wilson et al., 2014).”
MethodReproducible Research in Vadose Zone Sciences
Skaggs, T. H., Young, M. H., & Vrugt, J. A. (2015) · Vadose Zone Journal · doi:10.2136/vzj2015.06.0088 · Cited by 15
“Although any computing system can be used to develop reproducible research, the use of higher-level, reusable code and systems has been recommended as leading to more reliable scientific software (Wilson et al., 2014).”
MethodCareful considerations for mHealth app development: lessons learned from QuestExplore
Subramaniam, A., Hensley, E., Stojancic, R., Vaughn, J., & Shah, N. (2022) · mHealth · doi:10.21037/mhealth-21-51 · Cited by 15
“This can be enhanced by methodological, thorough documentation designed to “help people understand code” (19).”
BackgroundComputational Methods in Landscape Ecology
Hesselbarth, M. H. K., Nowosad, J., de Flamingh, A., Simpkins, C., Jung, M., Gerber, G., & Bosch, M. (2024) · Current Landscape Ecology Reports · doi:10.1007/s40823-024-00104-6 · Cited by 30
“Computational science analyzes abstracted core mechanisms of research questions using data and algorithms and is one of the most important tools of modern science [4].”
A comparative study of programming languages for next-generation astrodynamics systems
Eichhorn, H., Cano, J. L., Mclean, F., & Anderl, R. (2017) · CEAS Space Journal · doi:10.1007/s12567-017-0170-8 · Cited by 15
“Although this violates the “Do Not Repeat Yourself” rule [11], another possibility would be to extend the interface of the Newton–Raphson solver to accept an additional array of double-valued parameters.”
BackgroundTop 10 metrics for life science software good practices
Artaza, H., Hong, N. C., Corpas, M., Corpuz, A., Hooft, R. W. W., Jiménez, R. C., Leskošek, B. L., Olivier, B. G., Štourač, J., Vařeková, R. S., et al. (2016) · F1000Research · doi:10.12688/f1000research.9206.1 · Cited by 15
Costa, R. L., Gadelha, L. M. R., Jr., Alves, M. R., & Porto, F. A. M. (2017) · PeerJ · doi:10.7717/peerj.3509 · Cited by 18
“The implementation follows best practices for scientific software development (Wilson et al., 2014).”
MethodSeven quick tips for analysis scripts in neuroimaging
van Vliet, M. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007358 · Cited by 16
“There is a large body of literature on managing complexity and reducing the chance for programmer error in software [16–20].”
BackgroundQuantitative Measurement of Scientific Software Quality: Definition of a Novel Quality Model
Koteska, B., Mishev, A., & Pejov, L. (2018) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194018500146 · Cited by 15
Knight, V., Harper, M., Glynatsi, N. E., & Campbell, O. (2018) · PLoS ONE · doi:10.1371/journal.pone.0204981 · Cited by 22
Coding Code: Qualitative Methods for Investigating Data Science Skills
Theobold, A. S., Wickstrom, M. H., & Hancock, S. A. (2023) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2023.2277847 · Cited by 22
“The theme of “efficiency” was found in a similar vein, by recognizing code within the theme of data wrangling and data visualization which did/did not adhere to the “don’t repeat yourself” principle (Wilson et al. 2014).”
Backgroundvan Kampen, A. H. C., Mahamune, U., Jongejan, A., van Schaik, B. D. C., Balashova, D., Lashgari, D., Pras‐Raves, M. L., Wever, E. J. M., Dane, A. D., García-Valiente, R., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-52446-8 · Cited by 24
A reproducible approach to high-throughput biological data acquisition and integration
Börnigen, D., Moon, Y. S., Rahnavard, A., Waldron, L., McIver, L. J., Shafquat, A., Franzosa, E. A., Miropolsky, L., Sweeney, C. J., Morgan, X. C., et al. (2015) · PeerJ · doi:10.7717/peerj.791 · Cited by 17
Wang, Q., Shi, F., Kowalczyk, A. P., Campbell, R. M., Goudey, B., Rawlinson, D., Harwood, A., Ferrá, H., & Kowalczyk, A. (2014) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2014.6999192 · Cited by 14
“In order to fully exploit the potential of these devices for medical and biotechnological research, efficient software tools are required to reduce implementation and performance difficulties allowing researchers to focus on the comparison of statistical results rather than wrestling with the intricacies of software d…”
BackgroundAnt genomics (Hymenoptera: Formicidae): challenges to overcome and opportunities to seize
Nygaard, S., & Wurm, Y. (2015) · Research at the University of Copenhagen (University of Copenhagen) · doi:10.25849/myrmecol.news_021:059 · Cited by 16
Script of Scripts: A pragmatic workflow system for daily computational research
Wang, G., & Peng, B. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006843 · Cited by 16
“To make scripts better suited for reproducible research, theories and implementations of scientific workflow systems [3, 5] have been introduced for script management and execution, creating what are referred to as “pipeline languages” or “pipeline tools.””
MethodSimulation Foundry: Automated and F.A.I.R. Molecular Modeling
Gygli, G., & Pleiss, J. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c00018 · Cited by 18
Fostering Better Coding Practices for Data Scientists
Pruim, R. J., Gîrjău, M.-C., & Horton, N. J. (2023) · Harvard Data Science Review · doi:10.1162/99608f92.97c9f60f · Cited by 19
Shalom, E. S., Kim, H., van der Heijden, R. A., Ahmed, Z., Patel, R., Hormuth, D. A., DiCarlo, J. C., Yankeelov, T. E., Sisco, N. J., Dortch, R. D., et al. (2023) · Magnetic Resonance in Medicine · doi:10.1002/mrm.29909 · Cited by 17
Garcés, M. (2018) · Springer eBooks · doi:10.1007/978-3-319-75140-5_8 · Cited by 14
Best versus Good Enough Practices for Open Music Research
Jensenius, A. R. (2021) · Empirical Musicology Review · doi:10.18061/emr.v16i1.7646 · Cited by 16
“The FAIR principles or the advice presented in the paper Best practices for scientific computing (Wilson et al., 2014) are at a level beyond what most empirical music researchers can implement.”
BackgroundInfluentialEvaluation of Textual Similarity Techniques in Code Level Traceability
Csuvik, V., Kicsi, A., & Vidács, L. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-24305-0_40 · Cited by 14
“Using good coding practices [41] can make the task easier and with proper naming conventions [35] very accurate results can be achieved.”
BackgroundHow Scientists Use Large Language Models to Program
O'Brien, G. (2025) · arXiv · doi:10.48550/arxiv.2502.17348 · Cited by 14
“Existing communities that are professionalizing research software and up-skilling scientific developers [13, 91, 104, 105] may be an ideal locus for discussions about what Code LLM practices should be acceptable in science, what code quality-checking mechanisms should exist, and how to advocate for…”
Writing Accessible Theory in Ecology and Evolution: Insights from Cognitive Load Theory
Ou, W. J.-A., Henriques, G. J. B., Senthilnathan, A., Ke, P., Grainger, T. N., & Germain, R. M. (2021) · BioScience · doi:10.1093/biosci/biab133 · Cited by 16
Wimmler, M., Bathmann, J., Vollhüter, J., & Berger, U. (2024) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2024.105973 · Cited by 13
Grandchamp, A., Piégu, B., & Monget, P. (2019) · Genome Biology and Evolution · doi:10.1093/gbe/evz078 · Cited by 15
The Database Normalization Theory and the Theory of Normalized Systems: Finding a Common Ground
Eessaar, E. (2016) · Cited by 13
Simplifying the development of portable, scalable, and reproducible workflows
Piccolo, S., Ence, Z. E., Anderson, E. C., Chang, J. T., & Bild, A. H. (2021) · eLife · doi:10.7554/elife.71069 · Cited by 13
“Software is fundamental to modern scientific research ( Hey et al., 2009 ; Wilson, 2014 ).”
BackgroundCharacterising reproducibility debt in scientific software: A systematic literature review
Hassan, Z., Treude, C., Norrish, M., Williams, G., & Potanin, A. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112327 · Cited by 19
Continuous Reproducibility in GNSS Signal Processing
Fernández–Prades, C., Vilà‐Valls, J., Arribas, J., & Ramos, A. (2018) · IEEE Access · doi:10.1109/access.2018.2822835 · Cited by 15
Implementing hybrid simulations that integrate DES+SD in Python
Palmer, G., & Tian, Y. (2021) · Journal of Simulation · doi:10.1080/17477778.2021.1992312 · Cited by 12
“The authors have strived to use best practices in research software development (Benureau and Rougier (2018); Percival (2014); Wilson et al. (2014)) such as modularisation, documentation and readability, accessibility, and testing (which can contribute to simulation model verification).”
Method
Metadata
- Authors
- Wilson, G., Aruliah, D. A., Brown, C. T., Chue Hong, N. P., Davis, M., Guy, R. T., Haddock, S. H. D., Huff, K. D., Mitchell, I. M., Plumbley, M. D., Waugh, B., White, E. P., & Wilson, P.
- Affiliations
- Mozilla Foundation (US)
- Ontario Tech University (CA)
- Michigan State University (US)
- Space Telescope Science Institute (US)
- University of Toronto (CA)
- Monterey Bay Aquarium Research Institute (US)
- University of California, Berkeley (US)
- University of British Columbia (CA)
- Queen Mary University of London (GB)
- University College London (GB)
- Utah State University (US)
- University of Wisconsin–Madison (US)
- Corresponding author
- Wilson, Greg
- ORCID
- Venue
- PLoS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-01-07
- Volume
- 12
- Issue
- 1
- Pages
- e1001745
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC3886731
- OpenAlex
- W2131966673
- Cited by
- 723
- References
- 75
- Citations by year
- 2026: 63 · 2025: 28 · 2024: 49 · 2023: 42 · 2022: 56 · 2021: 51 · 2020: 70 · 2019: 69 · 2018: 67 · 2017: 62 · 2016: 63 · 2015: 54 · 2014: 27
- FWCI
- 128.23
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion, cc-by)
- Surrey Research Insight Open Access (The University of Surrey) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (0953694, CAREER, EP/H043160/1)
- Alfred P. Sloan Foundation
- Natural Sciences and Engineering Research Council of Canada
- Engineering and Physical Sciences Research Council (EP/H043160/1, EP/H043101/1, EP/G007144/1)
- National Human Genome Research Institute (R01HG007513)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
- Keywords
- scientific computing, best practices
- MeSH terms
- Computing Methodologies, Humans, Software, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.