Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Replicability and reproducibility of computational models has been somewhat understudied by "the replication movement." In this paper, we draw on methodological studies into the replicability of psychological experiments and on the mechanistic account of explanation to analyze the functions of model replications and model reproductions in computational neuroscience. We contend that model replicability, or independent researchers' ability to obtain the same output using original code and data, and model reproducibility, or independent researchers' ability to recreate a model without original code, serve different functions and fail for different reasons. This means that measures designed to improve model replicability may not enhance (and, in some cases, may actually damage) model reproducibility. We claim that although both are undesirable, low model reproducibility poses more of a threat to long-term scientific progress than low model replicability. In our opinion, low model reproducibility stems mostly from authors' omitting to provide crucial information in scientific papers and we stress that sharing all computer code and data is not a solution. Reports of computational studies should remain selective and include all and only relevant bits of code.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“…of results are the cornerstones of science, so it is no surprise that recent doubts about researchers ’ ability to consistently duplicate findings in a number of scientific fields have caused quite a stir in the scientific community (Button et al. 2013; Loken and Gelman 2017; Maxwell et al. 2015).”
BackgroundMachamer, P., Darden, L., & Craver, C. F. (2000) · Philosophy of Science · doi:10.1086/392759 · Cited by 3,179 · cited 1×
“According to this approach, to explain a phenomenon in terms of how it is constituted is to elucidate its underlying causal structure and organized components and operations of the mechanism responsible for the phenomenon (Craver 2007; Machamer et al. 2000).”
BackgroundReconstruction and Simulation of Neocortical Microcircuitry
Markram, H., Muller, E., Ramaswamy, S., Reimann, M. W., Abdellah, M., Sanchez, C. A., Ailamaki, A., Alonso-Nanclares, L., Antille, N., Arsever, S., et al. (2015) · Cell · doi:10.1016/j.cell.2015.09.029 · Cited by 1,557 · cited 1×
“Their proposal also helps to deal with highly complex models, such as detailed simulations of neocortical microcircuitry (Markram et al. 2015) or functional simulations of the whole brain (Eliasmith et al. 2012).”
BackgroundReproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 1×
“A number of authors in the computational science community are drawing attention to problems with model replicability and reproducibility (Hutson 2018; Peng 2011; Rougier et al. 2017; Sandve et al. 2013).”
BackgroundShall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125 · cited 2×
“One distinction that has become influential is the contrast between direct and conceptual replications (Hüffmeier et al. 2016; Schmidt2009 ; Stroebe and Strack 2014; Zwaan et al. 2018).”
BackgroundTen Simple Rules for Reproducible Computational Research
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003285 · Cited by 880 · cited 5×
“A number of authors in the computational science community are drawing attention to problems with model replicability and reproducibility (Hutson 2018; Peng 2011; Rougier et al. 2017; Sandve et al. 2013).”
BackgroundMethodInfluentialMeasurement error and the replication crisis
Loken, E., & Gelman, A. (2017) · Science · doi:10.1126/science.aal3618 · Cited by 628 · cited 1×
“…of results are the cornerstones of science, so it is no surprise that recent doubts about researchers ’ ability to consistently duplicate findings in a number of scientific fields have caused quite a stir in the scientific community (Button et al. 2013; Loken and Gelman 2017; Maxwell et al. 2015).”
BackgroundThe alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369
Research works printed but not linked: 51
- Arbib, M. A. (2003). Rana computatrix to human language: Towards a computational neuroethology of language evolution. Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences, 361(1811), 2345–2379. https://doi.org/10.1098/rsta.2003.1248 . · DOI
- Baetu, T. M. (2015). The completeness of mechanistic explanations. Philosophy of Science, 82(5), 775–786. https://doi.org/10.1086/683279 . · DOI
- Braitenberg, V. (1984). Vehicles, experiments in synthetic psychology. Cambridge: MIT Press.
- Buckheit, J. B., & Donoho, D. L. (1995). WaveLab and reproducible research. In Wavelets and statistics (pp. 55–81). New York: Springer. https://doi.org/10.1007/978-1-4612-2544-7_5 . · DOI · Paper page
- Busemeyer, J. R., & Diederich, A. (2010). Cognitive modeling. Los Angeles: Sage.
- Cannon, R. C., Gleeson, P., Crook, S., Ganapathy, G., Marin, B., Piasini, E., & Silver, R. A. (2014). LEMS: A language for expressing complex biological models in concise and hierarchical form and its use in underpinning NeuroML 2. Frontiers in Neuroinformatics, 8, 79. https://doi.org/10.3389/fninf.2014.00079 . · DOI · Paper page
- Chirimuuta, M. (2014). Minimal models and canonical neural computations: The distinctness of computational explanation in neuroscience. Synthese, 191(2), 127–153. https://doi.org/10.1007/s11229-013-0369-y . · DOI
- Chomsky, N. (1959). A review of B. F. Skinner’s Verbal behavior. Language, 35(1), 26–58. · DOI
- Claerbout, J. F., & Karrenbach, M. (1992). Electronic documents give reproducible research a new meaning. SEG Expanded Abstracts, 11, 601–604. https://doi.org/10.1190/1.1822162 . · DOI · Paper page
- Connett, J. E. (2008). Repeatability and reproducibility. In R. B. D’Agostino, L. Sullivan, & J. Massaro (Eds.), Wiley encyclopedia of clinical trials, https://doi.org/10.1002/9780471462422.eoct512 . · DOI
- Craver, C. F. (2007). Explaining the brain: Mechanisms and the mosaic unity of neuroscience. Oxford: Oxford University Press. · DOI · Paper page
- Craver, C. F., & Kaplan, D. M. (2018). Are more details better? On the norms of completeness for mechanistic explanations. The British Journal for the Philosophy of Science. https://doi.org/10.1093/bjps/axy015 . · DOI · Paper page
- Crook, S. M., Davison, A. P., & Plesser, H. E. (2013). Learning from the past: Approaches for reproducibility in computational neuroscience. In 20 years of computational neuroscience (pp. 73–102). New York: Springer. https://doi.org/10.1007/978-1-4614-1424-7_4 . · DOI
- Davison, A. P., Brüderle, D., Eppler, J., Kremkow, J., Muller, E., Pecevski, D., … Yger, P. (2008). PyNN: A common interface for neuronal network simulators. Frontiers in Neuroinformatics, 2, 11. https://doi.org/10.3389/neuro.11.011.2008 · DOI · Paper page
- De Pittà, M., Goldberg, M., Volman, V., Berry, H., & Ben-Jacob, E. (2009). Glutamate regulation of calcium and IP3 oscillating and pulsating dynamics in astrocytes. Journal of Biological Physics, 35(4), 383–411. https://doi.org/10.1007/s10867-009-9155-y . · DOI · Paper page
- De Schutter, E. (2008). Why are computational neuroscience and systems biology so separate? PLoS Computational Biology, 4(5), e1000078. https://doi.org/10.1371/journal.pcbi.1000078 . · DOI · Paper page
- Delling, D., Demetrescu, C., Johnson, D. S., & Vitek, J. (2016). Rethinking experimental methods in computing. Schloss Dagstuhl—Leibniz-Zentrum fuer Informatik GmbH, Wadern/Saarbruecken, Germany. https://doi.org/10.4230/DagRep.6.3.24 . · doi:10.4230/dagrep.6.3.24
- Dodge, R. (1903). Five types of eye movement in the horizontal meridian plane of the field of regard. American Journal of Physiology-Legacy Content, 8(4), 307–329. https://doi.org/10.1152/ajplegacy.1903.8.4.307 . · DOI · Paper page
- Does your code stand up to scrutiny? (2018). Nature, 555(7695), 142. 10/gc5tf9.
- Drummond, D. C. (2009). Replicability is not reproducibility: Nor is it good science. In Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML. Montreal, Canada: National Research Council. http://cogprints.org/7691/ . Accessed 17 Feb 2018.
- Dupont, G., Loomekandja Lokenye, E. F., & Challiss, R. A. J. (2011). A model for Ca2+ oscillations stimulated by the type 5 metabotropic glutamate receptor: An unusual mechanism based on repetitive, reversible phosphorylation of the receptor. Biochimie, 93(12), 2132–2138. https://doi.org/10.1016/j.biochi.2011.09.010 . · doi:10.1016/j.biochi.2011.09.010
- Easterbrook, S. M. (2014). Open code for open science? Nature Geoscience, 7(11), ngeo2283. https://doi.org/10.1038/ngeo2283 . · DOI
- Eliasmith, C., Stewart, T. C., Choo, X., Bekolay, T., DeWolf, T., Tang, Y., & Rasmussen, D. (2012). A large-scale model of the functioning brain. Science, 338(6111), 1202–1205. https://doi.org/10.1126/science.1225266 . · DOI · Paper page
- Gleeson, P., Crook, S., Cannon, R. C., Hines, M. L., Billings, G. O., Farinella, M., Morse, T. M., Davison, A. P., Ray, S., Bhalla, U. S., Barnes, S. R., Dimitrova, Y. D., & Silver, R. A. (2010). NeuroML: A language for describing data driven models of neurons and networks with a high degree of biological detail. PLoS Computational Biology, 6(6), e1000815. https://doi.org/10.1371/journal.pcbi.1000815 . · DOI · Paper page
- Gleeson, P., Davison, A. P., Silver, R. A., & Ascoli, G. A. (2017). A commitment to open source in neuroscience. Neuron, 96(5), 964–965. https://doi.org/10.1016/j.neuron.2017.10.013 . · DOI
- Hochstein, E. (2015). One mechanism, many models: A distributed theory of mechanistic explanation. Synthese, 193, 1387–1407. https://doi.org/10.1007/s11229-015-0844-8 . · DOI
- Hüffmeier, J., Mazei, J., & Schultze, T. (2016). Reconceptualizing replication as a sequence of different studies: A replication typology. Journal of Experimental Social Psychology, 66, 81–92. https://doi.org/10.1016/j.jesp.2015.09.009 . · DOI · Paper page
- Hutson, M. (2018). Missing data hinder replication of artificial intelligence studies. Science. https://doi.org/10.1126/science.aat3298 . · DOI
- Johansson, R., & Johansson, M. (2014). Look here, eye movements play a functional role in memory retrieval. Psychological Science, 25(1), 236–242. https://doi.org/10.1177/0956797613498260 . · DOI · Paper page
- Kuhn, T. S. (1957). The Copernican revolution: Planetary astronomy in the development of western thought. Cambridge: Harvard University Press.
- Kuhn, T. S. (1962). The structure of scientific revolutions. Chicago: University of Chicago Press.
- Laudan, L. (1977). Progress and its problems: Towards a theory of scientific growth. Berkeley: University of California Press.
- Lavrentovich, M., & Hemkin, S. (2008). A mathematical model of spontaneous calcium(II) oscillations in astrocytes. Journal of Theoretical Biology, 251(4), 553–560. https://doi.org/10.1016/j.jtbi.2007.12.011 . · DOI
- Legéndi, R. O., Gulyás, L., & Mansury, Y. (2013). Model replication in the context of agent-based simulation: Lessons learnt from two case studies. European Quarterly of Political Attitudes and Mentalities, 2(1), 74–85.
- Lyons, L. (1986). Statistics for nuclear and particle physicists. Cambridge: Cambridge University Press. · DOI · Paper page
- Manninen, T., Hituri, K., Toivari, E., & Linne, M.-L. (2011). Modeling signal transduction leading to synaptic plasticity: Evaluation and comparison of five models. EURASIP Journal on Bioinformatics and Systems Biology, 2011(1), 797250. https://doi.org/10.1155/2011/797250 . · DOI
- Manninen, T., Havela, R., & Linne, M.-L. (2017). Reproducibility and comparability of computational models for astrocyte calcium excitability. Frontiers in Neuroinformatics, 11, 11. https://doi.org/10.3389/fninf.2017.00011 . · DOI
- Maunsell, J. (2010). Announcement regarding supplemental material. Journal of Neuroscience, 30(32), 10599–10600.
- Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70(6), 487–498. https://doi.org/10.1037/a0039400 . · DOI · Paper page
- McDougal, R. A., Bulanova, A. S., & Lytton, W. W. (2016). Reproducibility in computational neuroscience models and simulations. IEEE Transactions on Bio-Medical Engineering, 63(10), 2021–2035. https://doi.org/10.1109/TBME.2016.2539602 . · DOI · Paper page
- Migliore, M., Morse, T. M., Davison, A. P., Marenco, L., Shepherd, G. M., & Hines, M. L. (2003). ModelDB. Neuroinformatics, 1(1), 135–139. https://doi.org/10.1385/NI:1:1:135 . · DOI
- Miłkowski, M. (2016). Explanatory completeness and idealization in large brain simulations: A mechanistic perspective. Synthese, 193(5), 1457–1478. https://doi.org/10.1007/s11229-015-0731-3 . · DOI
- Miller, J. N., & Miller, J. C. (2010). Statistics and chemometrics for analytical chemistry (6th ed.). Harlow: Pearson.
- Nordlie, E., Gewaltig, M.-O., & Plesser, H. E. (2009). Towards reproducible descriptions of neuronal network models. PLoS Computational Biology, 5(8), e1000456. https://doi.org/10.1371/journal.pcbi.1000456 . · DOI · Paper page
- Parr, T., & Friston, K. J. (2018). Active inference and the anatomy of oculomotion. Neuropsychologia, 111, 334–343. https://doi.org/10.1016/j.neuropsychologia.2018.01.041 . · DOI · Paper page
- Plesser, H. E. (2018). Reproducibility vs. replicability: A brief history of a confused terminology. Frontiers in Neuroinformatics, 11, 76. https://doi.org/10.3389/fninf.2017.00076 . · DOI · Paper page
- Riera, J., Hatanaka, R., Ozaki, T., & Kawashima, R. (2011). Modeling the spontaneous Ca2+ oscillations in astrocytes: Inconsistencies and usefulness. Journal of Integrative Neuroscience, 10(04), 439–473. https://doi.org/10.1142/S0219635211002877 . · DOI
- Rosenthal, R. (1966). Experimenter effects in behavioral research. East Norwalk: Appleton-Century-Crofts.
- Rougier, N. P., Hinsen, K., Alexandre, F., Arildsen, T., Barba, L. A., Benureau, F. C. Y., Brown, C. T., de Buyl, P., Caglayan, O., Davison, A. P., Delsuc, M. A., Detorakis, G., Diem, A. K., Drix, D., Enel, P., Girard, B., Guest, O., Hall, M. G., Henriques, R. N., Hinaut, X., Jaron, K. S., Khamassi, M., Klein, A., Manninen, T., Marchesi, P., McGlinn, D., Metzner, C., Petchey, O., Plesser, H. E., Poisot, T., Ram, K., Ram, Y., Roesch, E., Rossant, C., Rostami, V., Shifman, A., Stachelek, J., Stimberg, M., Stollmeier, F., Vaggi, F., Viejo, G., Vitay, J., Vostinar, A. E., Yurchak, R., & Zito, T. (2017). Sustainable computational science: The ReScience initiative. PeerJ Computer Science, 3, e142. https://doi.org/10.7717/peerj-cs.142 . · DOI · Paper page
- Zeigler, B. P. (1976). Theory of modelling and simulation. New York: Wiley.
- Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41, e120. https://doi.org/10.1017/S0140525X17001972 . · DOI · Paper page
Cited by
NeuroKit2: A Python toolbox for neurophysiological signal processing
Makowski, D., Pham, T., Lau, Z. J., Brammer, J. C., Lespinasse, F., Pham, H., Schölzel, C., & Chen, S. H. A. (2021) · Behavior Research Methods · doi:10.3758/s13428-020-01516-y · Cited by 1,388
How Computational Modeling Can Force Theory Building in Psychological Science
Guest, O., & Martin, A. E. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620970585 · Cited by 460
Application of the Hierarchical Bootstrap to Multi-Level Data in Neuroscience
Saravanan, V., Berman, G. J., & Sober, S. J. (2019) · doi:10.1101/819334 · Cited by 170
Reproducibility in systems biology modelling
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2021) · Molecular Systems Biology · doi:10.15252/msb.20209982 · Cited by 131
Liu, D. M., & Salganik, M. J. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023119849803 · Cited by 47
Metadata
- Authors
- Miłkowski, M., Hensel, W. M., & Hohol, M.
- Venue
- Journal of Computational Neuroscience
- Published
- 2018-10-31
- Volume
- 45
- Issue
- 3
- Pages
- 163-172
- ISSN
- 0929-5313 (ISSN-L)
- 1573-6873
- OpenAlex
- W2899002317
- Cited by
- 112
- References
- 9
- FWCI
- 14.43
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- Funders
- National Science Centre, Poland (2014/14/E/HS1/00803, 2015/19/B/HS1/03310)
- Foundation for Polish Science (START Scholarship)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail. https://app.cochat.ai/papers/10.1007/s10827-018-0702-z
- The paper
- Miłkowski, M., Hensel, W. M., & Hohol, M. (2018). Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail. Journal of Computational Neuroscience. https://doi.org/10.1007/s10827-018-0702-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.