Measuring the Performance of Neural Models
Open accessThe DOI matches its Crossref record.
Abstract
Good metrics of the performance of a statistical or computational model are essential for model comparison and selection. Here, we address the design of performance metrics for models that aim to predict neural responses to sensory inputs. This is particularly difficult because the responses of sensory neurons are inherently variable, even in response to repeated presentations of identical stimuli. In this situation, standard metrics (such as the correlation coefficient) fail because they do not distinguish between explainable variance (the part of the neural response that is systematically dependent on the stimulus) and response variability (the part of the neural response that is not systematically dependent on the stimulus, and cannot be explained by modeling the stimulus-response relationship). As a result, models which perfectly describe the systematic stimulus-response relationship may appear to perform poorly. Two metrics have previously been proposed which account for this inherent variability: Signal Power Explained (SPE, Sahani and Linden, 2003), and the normalized correlation coefficient (CC norm , Hsu et al., 2004). Here, we analyze these metrics, and show that they are intimately related. However, SPE has no lower bound, and we show that, even for good models, SPE can yield negative values that are difficult to interpret. CC norm is better behaved in that it is effectively bounded between -1 and 1, and values below zero are very rare in practice and easy to interpret. However, it was hitherto not possible to calculate CC norm directly; instead, it was estimated using imprecise and laborious resampling techniques. Here, we identify a new approach that can calculate CC norm quickly and accurately. As a result, we argue that it is now a better choice of metric than SPE to accurately evaluate the performance of neural models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V., & McDermott, J. H. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.044 · Cited by 544
“We measured the amount of BOLD variance predicted in left-out sounds, correcting for both the reliability of the measured voxel response and the reliability of the predicted voxel response (Schoppe et al., 2016; Spearman, 1904).”
de Vries, S. E. J., Lecoq, J. A., Buice, M. A., Groblewski, P. A., Ocker, G. K., Oliver, M., Feng, D., Cain, N., Ledochowitsch, P., Millman, D., et al. (2020) · Nature Neuroscience · doi:10.1038/s41593-019-0550-9 · Cited by 353
Weak but Critical Links between Primary Somatosensory Centers and Motor Cortex during Movement
Wei, P., Bao, R., Lv, Z., & Jing, B. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00001 · Cited by 264
The representation of prediction error in auditory cortex
Rubin, J., Ulanovsky, N., Nelken, I., & Tishby, N. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005058 · Cited by 87
A framework of hybrid model development with identification of plant-model mismatch
Chen, Y., & Ierapetritou, M. (2020) · AIChE Journal · doi:10.1002/aic.16996 · Cited by 72
Metadata
- Authors
- Schoppe, O., Harper, N. S., Willmore, B. D. B., King, A. J., & Schnupp, J. W. H.
- Venue
- Frontiers in Computational Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2016-02-10
- Volume
- 10
- Pages
- 10-10
- ISSN
- 1662-5188 (ISSN-L)
- OpenAlex
- W2298475536
- Cited by
- 95
- References
- 0
- FWCI
- 4.70
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Measuring the Performance of Neural Models. https://app.cochat.ai/papers/10.3389/fncom.2016.00010
- The paper
- Schoppe, O., Harper, N. S., Willmore, B. D. B., King, A. J., & Schnupp, J. W. H. (2016). Measuring the Performance of Neural Models. Frontiers in Computational Neuroscience. https://doi.org/10.3389/fncom.2016.00010
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.