CoChat

Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail

Open access
Miłkowski, Marcin, Hensel, Witold M., Hohol, Mateusz · 2018 · Journal of Computational Neuroscience
Cited by 112

The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.

Abstract

Replicability and reproducibility of computational models has been somewhat understudied by "the replication movement." In this paper, we draw on methodological studies into the replicability of psychological experiments and on the mechanistic account of explanation to analyze the functions of model replications and model reproductions in computational neuroscience. We contend that model replicability, or independent researchers' ability to obtain the same output using original code and data, and model reproducibility, or independent researchers' ability to recreate a model without original code, serve different functions and fail for different reasons. This means that measures designed to improve model replicability may not enhance (and, in some cases, may actually damage) model reproducibility. We claim that although both are undesirable, low model reproducibility poses more of a threat to long-term scientific progress than low model replicability. In our opinion, low model reproducibility stems mostly from authors' omitting to provide crucial information in scientific papers and we stress that sharing all computer code and data is not a solution. Reports of computational studies should remain selective and include all and only relevant bits of code.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10827-018-0702-z

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 9 references

Research works printed but not linked: 51

  1. Arbib, M. A. (2003). Rana computatrix to human language: Towards a computational neuroethology of language evolution. Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences, 361(1811), 2345–2379. https://doi.org/10.1098/rsta.2003.1248 . · DOI
  2. Baetu, T. M. (2015). The completeness of mechanistic explanations. Philosophy of Science, 82(5), 775–786. https://doi.org/10.1086/683279 . · DOI
  3. Braitenberg, V. (1984). Vehicles, experiments in synthetic psychology. Cambridge: MIT Press.
  4. Buckheit, J. B., & Donoho, D. L. (1995). WaveLab and reproducible research. In Wavelets and statistics (pp. 55–81). New York: Springer. https://doi.org/10.1007/978-1-4612-2544-7_5 . · DOI · Paper page
  5. Busemeyer, J. R., & Diederich, A. (2010). Cognitive modeling. Los Angeles: Sage.
  6. Cannon, R. C., Gleeson, P., Crook, S., Ganapathy, G., Marin, B., Piasini, E., & Silver, R. A. (2014). LEMS: A language for expressing complex biological models in concise and hierarchical form and its use in underpinning NeuroML 2. Frontiers in Neuroinformatics, 8, 79. https://doi.org/10.3389/fninf.2014.00079 . · DOI · Paper page
  7. Chirimuuta, M. (2014). Minimal models and canonical neural computations: The distinctness of computational explanation in neuroscience. Synthese, 191(2), 127–153. https://doi.org/10.1007/s11229-013-0369-y . · DOI
  8. Chomsky, N. (1959). A review of B. F. Skinner’s Verbal behavior. Language, 35(1), 26–58. · DOI
  9. Claerbout, J. F., & Karrenbach, M. (1992). Electronic documents give reproducible research a new meaning. SEG Expanded Abstracts, 11, 601–604. https://doi.org/10.1190/1.1822162 . · DOI · Paper page
  10. Connett, J. E. (2008). Repeatability and reproducibility. In R. B. D’Agostino, L. Sullivan, & J. Massaro (Eds.), Wiley encyclopedia of clinical trials, https://doi.org/10.1002/9780471462422.eoct512 . · DOI
  11. Craver, C. F. (2007). Explaining the brain: Mechanisms and the mosaic unity of neuroscience. Oxford: Oxford University Press. · DOI · Paper page
  12. Craver, C. F., & Kaplan, D. M. (2018). Are more details better? On the norms of completeness for mechanistic explanations. The British Journal for the Philosophy of Science. https://doi.org/10.1093/bjps/axy015 . · DOI · Paper page
  13. Crook, S. M., Davison, A. P., & Plesser, H. E. (2013). Learning from the past: Approaches for reproducibility in computational neuroscience. In 20 years of computational neuroscience (pp. 73–102). New York: Springer. https://doi.org/10.1007/978-1-4614-1424-7_4 . · DOI
  14. Davison, A. P., Brüderle, D., Eppler, J., Kremkow, J., Muller, E., Pecevski, D., … Yger, P. (2008). PyNN: A common interface for neuronal network simulators. Frontiers in Neuroinformatics, 2, 11. https://doi.org/10.3389/neuro.11.011.2008 · DOI · Paper page
  15. De Pittà, M., Goldberg, M., Volman, V., Berry, H., & Ben-Jacob, E. (2009). Glutamate regulation of calcium and IP3 oscillating and pulsating dynamics in astrocytes. Journal of Biological Physics, 35(4), 383–411. https://doi.org/10.1007/s10867-009-9155-y . · DOI · Paper page
  16. De Schutter, E. (2008). Why are computational neuroscience and systems biology so separate? PLoS Computational Biology, 4(5), e1000078. https://doi.org/10.1371/journal.pcbi.1000078 . · DOI · Paper page
  17. Delling, D., Demetrescu, C., Johnson, D. S., & Vitek, J. (2016). Rethinking experimental methods in computing. Schloss Dagstuhl—Leibniz-Zentrum fuer Informatik GmbH, Wadern/Saarbruecken, Germany. https://doi.org/10.4230/DagRep.6.3.24 . · doi:10.4230/dagrep.6.3.24
  18. Dodge, R. (1903). Five types of eye movement in the horizontal meridian plane of the field of regard. American Journal of Physiology-Legacy Content, 8(4), 307–329. https://doi.org/10.1152/ajplegacy.1903.8.4.307 . · DOI · Paper page
  19. Does your code stand up to scrutiny? (2018). Nature, 555(7695), 142. 10/gc5tf9.
  20. Drummond, D. C. (2009). Replicability is not reproducibility: Nor is it good science. In Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML. Montreal, Canada: National Research Council. http://cogprints.org/7691/ . Accessed 17 Feb 2018.
  21. Dupont, G., Loomekandja Lokenye, E. F., & Challiss, R. A. J. (2011). A model for Ca2+ oscillations stimulated by the type 5 metabotropic glutamate receptor: An unusual mechanism based on repetitive, reversible phosphorylation of the receptor. Biochimie, 93(12), 2132–2138. https://doi.org/10.1016/j.biochi.2011.09.010 . · doi:10.1016/j.biochi.2011.09.010
  22. Easterbrook, S. M. (2014). Open code for open science? Nature Geoscience, 7(11), ngeo2283. https://doi.org/10.1038/ngeo2283 . · DOI
  23. Eliasmith, C., Stewart, T. C., Choo, X., Bekolay, T., DeWolf, T., Tang, Y., & Rasmussen, D. (2012). A large-scale model of the functioning brain. Science, 338(6111), 1202–1205. https://doi.org/10.1126/science.1225266 . · DOI · Paper page
  24. Gleeson, P., Crook, S., Cannon, R. C., Hines, M. L., Billings, G. O., Farinella, M., Morse, T. M., Davison, A. P., Ray, S., Bhalla, U. S., Barnes, S. R., Dimitrova, Y. D., & Silver, R. A. (2010). NeuroML: A language for describing data driven models of neurons and networks with a high degree of biological detail. PLoS Computational Biology, 6(6), e1000815. https://doi.org/10.1371/journal.pcbi.1000815 . · DOI · Paper page
  25. Gleeson, P., Davison, A. P., Silver, R. A., & Ascoli, G. A. (2017). A commitment to open source in neuroscience. Neuron, 96(5), 964–965. https://doi.org/10.1016/j.neuron.2017.10.013 . · DOI
  26. Hochstein, E. (2015). One mechanism, many models: A distributed theory of mechanistic explanation. Synthese, 193, 1387–1407. https://doi.org/10.1007/s11229-015-0844-8 . · DOI
  27. Hüffmeier, J., Mazei, J., & Schultze, T. (2016). Reconceptualizing replication as a sequence of different studies: A replication typology. Journal of Experimental Social Psychology, 66, 81–92. https://doi.org/10.1016/j.jesp.2015.09.009 . · DOI · Paper page
  28. Hutson, M. (2018). Missing data hinder replication of artificial intelligence studies. Science. https://doi.org/10.1126/science.aat3298 . · DOI
  29. Johansson, R., & Johansson, M. (2014). Look here, eye movements play a functional role in memory retrieval. Psychological Science, 25(1), 236–242. https://doi.org/10.1177/0956797613498260 . · DOI · Paper page
  30. Kuhn, T. S. (1957). The Copernican revolution: Planetary astronomy in the development of western thought. Cambridge: Harvard University Press.
  31. Kuhn, T. S. (1962). The structure of scientific revolutions. Chicago: University of Chicago Press.
  32. Laudan, L. (1977). Progress and its problems: Towards a theory of scientific growth. Berkeley: University of California Press.
  33. Lavrentovich, M., & Hemkin, S. (2008). A mathematical model of spontaneous calcium(II) oscillations in astrocytes. Journal of Theoretical Biology, 251(4), 553–560. https://doi.org/10.1016/j.jtbi.2007.12.011 . · DOI
  34. Legéndi, R. O., Gulyás, L., & Mansury, Y. (2013). Model replication in the context of agent-based simulation: Lessons learnt from two case studies. European Quarterly of Political Attitudes and Mentalities, 2(1), 74–85.
  35. Lyons, L. (1986). Statistics for nuclear and particle physicists. Cambridge: Cambridge University Press. · DOI · Paper page
  36. Manninen, T., Hituri, K., Toivari, E., & Linne, M.-L. (2011). Modeling signal transduction leading to synaptic plasticity: Evaluation and comparison of five models. EURASIP Journal on Bioinformatics and Systems Biology, 2011(1), 797250. https://doi.org/10.1155/2011/797250 . · DOI
  37. Manninen, T., Havela, R., & Linne, M.-L. (2017). Reproducibility and comparability of computational models for astrocyte calcium excitability. Frontiers in Neuroinformatics, 11, 11. https://doi.org/10.3389/fninf.2017.00011 . · DOI
  38. Maunsell, J. (2010). Announcement regarding supplemental material. Journal of Neuroscience, 30(32), 10599–10600.
  39. Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70(6), 487–498. https://doi.org/10.1037/a0039400 . · DOI · Paper page
  40. McDougal, R. A., Bulanova, A. S., & Lytton, W. W. (2016). Reproducibility in computational neuroscience models and simulations. IEEE Transactions on Bio-Medical Engineering, 63(10), 2021–2035. https://doi.org/10.1109/TBME.2016.2539602 . · DOI · Paper page
  41. Migliore, M., Morse, T. M., Davison, A. P., Marenco, L., Shepherd, G. M., & Hines, M. L. (2003). ModelDB. Neuroinformatics, 1(1), 135–139. https://doi.org/10.1385/NI:1:1:135 . · DOI
  42. Miłkowski, M. (2016). Explanatory completeness and idealization in large brain simulations: A mechanistic perspective. Synthese, 193(5), 1457–1478. https://doi.org/10.1007/s11229-015-0731-3 . · DOI
  43. Miller, J. N., & Miller, J. C. (2010). Statistics and chemometrics for analytical chemistry (6th ed.). Harlow: Pearson.
  44. Nordlie, E., Gewaltig, M.-O., & Plesser, H. E. (2009). Towards reproducible descriptions of neuronal network models. PLoS Computational Biology, 5(8), e1000456. https://doi.org/10.1371/journal.pcbi.1000456 . · DOI · Paper page
  45. Parr, T., & Friston, K. J. (2018). Active inference and the anatomy of oculomotion. Neuropsychologia, 111, 334–343. https://doi.org/10.1016/j.neuropsychologia.2018.01.041 . · DOI · Paper page
  46. Plesser, H. E. (2018). Reproducibility vs. replicability: A brief history of a confused terminology. Frontiers in Neuroinformatics, 11, 76. https://doi.org/10.3389/fninf.2017.00076 . · DOI · Paper page
  47. Riera, J., Hatanaka, R., Ozaki, T., & Kawashima, R. (2011). Modeling the spontaneous Ca2+ oscillations in astrocytes: Inconsistencies and usefulness. Journal of Integrative Neuroscience, 10(04), 439–473. https://doi.org/10.1142/S0219635211002877 . · DOI
  48. Rosenthal, R. (1966). Experimenter effects in behavioral research. East Norwalk: Appleton-Century-Crofts.
  49. Rougier, N. P., Hinsen, K., Alexandre, F., Arildsen, T., Barba, L. A., Benureau, F. C. Y., Brown, C. T., de Buyl, P., Caglayan, O., Davison, A. P., Delsuc, M. A., Detorakis, G., Diem, A. K., Drix, D., Enel, P., Girard, B., Guest, O., Hall, M. G., Henriques, R. N., Hinaut, X., Jaron, K. S., Khamassi, M., Klein, A., Manninen, T., Marchesi, P., McGlinn, D., Metzner, C., Petchey, O., Plesser, H. E., Poisot, T., Ram, K., Ram, Y., Roesch, E., Rossant, C., Rostami, V., Shifman, A., Stachelek, J., Stimberg, M., Stollmeier, F., Vaggi, F., Viejo, G., Vitay, J., Vostinar, A. E., Yurchak, R., & Zito, T. (2017). Sustainable computational science: The ReScience initiative. PeerJ Computer Science, 3, e142. https://doi.org/10.7717/peerj-cs.142 . · DOI · Paper page
  50. Zeigler, B. P. (1976). Theory of modelling and simulation. New York: Wiley.
  51. Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41, e120. https://doi.org/10.1017/S0140525X17001972 . · DOI · Paper page

Cited by

See all 112

Metadata

Authors
Miłkowski, M., Hensel, W. M., & Hohol, M.
Venue
Journal of Computational Neuroscience
Published
2018-10-31
Volume
45
Issue
3
Pages
163-172
ISSN
  • 0929-5313 (ISSN-L)
  • 1573-6873
OpenAlex
W2899002317
Cited by
112
References
9
FWCI
14.43
Type
journal article
Language
en
Open access
hybrid
Licence
creativecommons.org/licenses/by/4.0 (tdm)
Funders
  • National Science Centre, Poland (2014/14/E/HS1/00803, 2015/19/B/HS1/03310)
  • Foundation for Polish Science (START Scholarship)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail. https://app.cochat.ai/papers/10.1007/s10827-018-0702-z
The paper
Miłkowski, M., Hensel, W. M., & Hohol, M. (2018). Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail. Journal of Computational Neuroscience. https://doi.org/10.1007/s10827-018-0702-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.