High-Dimensional Cluster Analysis with the Masked EM Algorithm
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Cluster analysis faces two problems in high dimensions: the “curse of dimensionality” that can lead to overfitting and poor generalization performance and the sheer time taken for conventional algorithms to process large amounts of high-dimensional data. We describe a solution to these problems, designed for the application of spike sorting for next-generation, high-channel-count neural probes. In this problem, only a small subset of features provides information about the cluster membership of any one data vector, but this informative feature subset is not the same for all data points, rendering classical feature selection ineffective. We introduce a “masked EM” algorithm that allows accurate and time-efficient clustering of up to millions of points in thousands of dimensions. We demonstrate its applicability to synthetic data and to real-world high-channel-count spike sorting data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
Model-Based Clustering, Discriminant Analysis, and Density Estimation
Fraley, C., & Raftery, A. E. (2002) · Journal of the American Statistical Association · doi:10.1198/016214502760047131 · Cited by 4,257
Unsupervised spike detection and sorting with wavelets and superparamagnetic clustering
Quiroga, R. Q., Nadasdy, Z., & Ben-Shaul, Y. (2004) · Neural Computation · doi:10.1162/089976604774201631 · Cited by 2,119
A review of methods for spike sorting: the detection and classification of neural action potentials
Lewicki, M. S. (1998) · Network Computation in Neural Systems · doi:10.1088/0954-898x/9/4/001 · Cited by 1,370
Harris, K. D., Henze, D. A., Csicsvari, J., Hirase, H., & Buzsáki, G. (2000) · Journal of Neurophysiology · doi:10.1152/jn.2000.84.1.401 · Cited by 1,151
Self-citationThe EM algorithm for mixtures of factor analyzers
Ghahramani, Z., & Hinton, G. E. (1996) · Cambridge University Engineering Department Publications Database · Cited by 629
Comparing Clusterings by the Variation of Information
Meilă, M. (2003) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-45167-9_14 · Cited by 604
Cited by
Arousal and Locomotion Make Distinct Contributions to Cortical Activity Patterns and Visual Encoding
Vinck, M., Batista-Brito, R., Knoblich, U., & Cardin, J. A. (2015) · Neuron · doi:10.1016/j.neuron.2015.03.028 · Cited by 819
Spike sorting for large, dense electrode arrays
Rossant, C., Kadir, S. N., Goodman, D. F. M., Schulman, J., Hunter, M. L. D., Saleem, A. B., Grosmark, A., Belluscio, M., Denfield, G. H., Ecker, A. S., et al. (2016) · Nature Neuroscience · doi:10.1038/nn.4268 · Cited by 785
Self-citationKilosort: realtime spike-sorting for extracellular electrophysiology with hundreds of channels
Pachitariu, M., Steinmetz, N., Kadir, S., Carandini, M., & Kenneth D., H. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/061481 · Cited by 740
Self-citationA Fully Automated Approach to Spike Sorting
Chung, J. E., Magland, J. F., Barnett, A. H., Tolosa, V. M., Tooker, A. C., Lee, K. Y., Shah, K. G., Felix, S. H., Frank, L. M., & Greengard, L. F. (2017) · Neuron · doi:10.1016/j.neuron.2017.08.030 · Cited by 490
Sliced Human Cortical Organoids for Modeling Distinct Cortical Layer Formation
Qian, X., Su, Y., Adam, C. D., Deutschmann, A. U., Pather, S. R., Goldberg, E. M., Su, K., Li, S., Lu, L., Jacob, F., et al. (2020) · Cell Stem Cell · doi:10.1016/j.stem.2020.02.002 · Cited by 479
“Single unit clusters were initially isolated from the 1 ms filtered, thresholded recordings using an unsupervised masked expectation-maximization (EM) algorithm that uses principle component analysis of spike features to sort spikes (KlustaKwik) (Kadir et al., 2014).”
MethodPast, present and future of spike sorting techniques
Rey, H. G., Pedreira, C., & Quian Quiroga, R. (2015) · Brain Research Bulletin · doi:10.1016/j.brainresbull.2015.04.007 · Cited by 462
Yger, P., Spampinato, G. L., Esposito, E., Lefebvre, B., Deny, S., Gardella, C., Stimberg, M., Jetter, F., Zeck, G., Picaud, S., et al. (2018) · eLife · doi:10.7554/elife.34518 · Cited by 441
Network Homeostasis and State Dynamics of Neocortical Sleep
Watson, B. O., Levenstein, D., Greene, J. P., Gelinas, J. N., & Buzsáki, G. (2016) · Neuron · doi:10.1016/j.neuron.2016.03.036 · Cited by 375
“After separation of LFP and spike data, spikes were clustered into single units (Kadir et al., 2014) and separated into putative pyramidal cells and putative interneurons (Supplemental Methods).”
MethodEntorhinal-CA3 Dual-Input Control of Spike Timing in the Hippocampus by Theta-Gamma Coupling
Fernández-Ruiz, A., Oliva, A., Nagy, G. A., Maurer, A. P., Berényi, A., & Buzsáki, G. (2017) · Neuron · doi:10.1016/j.neuron.2017.02.017 · Cited by 352
Hippocampal ripples down-regulate synapses
Norimoto, H., Makino, K., Gao, M., Shikano, Y., Okamoto, K., Ishikawa, T., Sasaki, T., Hioki, H., Fujisawa, S., & Ikegaya, Y. (2018) · Science · doi:10.1126/science.aao0702 · Cited by 262
Spatial representations of self and other in the hippocampus
Danjo, T., Toyoizumi, T., & Fujisawa, S. (2018) · Science · doi:10.1126/science.aao3898 · Cited by 258
Coherent encoding of subjective spatial position in visual cortex and hippocampus
Saleem, A. B., Diamanti, E. M., Fournier, J., Harris, K. D., & Carandini, M. (2018) · Nature · doi:10.1038/s41586-018-0516-1 · Cited by 254
Self-citationCriticality between Cortical States
Fontenele, A. J., de Vasconcelos, N. A. P., Feliciano, T., Aguiar, L. A. A., Soares-Cunha, C., Coimbra, B., Dalla Porta, L., Ribeiro, S., Rodrigues, A. J., Sousa, N., et al. (2019) · Physical Review Letters · doi:10.1103/physrevlett.122.208101 · Cited by 237
An independent, landmark-dominated head-direction signal in dysgranular retrosplenial cortex
Jacob, P.-Y., Casali, G., Spieser, L., Page, H., Overington, D., & Jeffery, K. (2017) · Nature Neuroscience · doi:10.1038/nn.4465 · Cited by 227
Emotional Mirror Neurons in the Rat’s Anterior Cingulate Cortex
Carrillo, M., Han, Y., Migliorati, F., Liu, M., Gazzola, V., & Keysers, C. (2019) · Current Biology · doi:10.1016/j.cub.2019.03.024 · Cited by 227
Organization of prefrontal network activity by respiration-related oscillations
Biskamp, J., Bartos, M., & Sauer, J.-F. (2017) · Scientific Reports · doi:10.1038/srep45508 · Cited by 197
Place cells are more strongly tied to landmarks in deep than in superficial CA1
Geiller, T., Fattahi, M., Choi, J.-S., & Royer, S. (2017) · Nature Communications · doi:10.1038/ncomms14531 · Cited by 163
“Spike sorting was performed semi-automatically, using KlustaKwik (klustakwik.sourceforge.net) 32 , followed by manual adjustment of the clusters with Klusters 33 .”
Treatment during a vulnerable developmental period rescues a genetic epilepsy
Marguet, S. L., Le-Schulte, V. T. Q., Merseburg, A., Neu, A., Eichler, R., Jakovcevski, I., Ivanov, A., Hanganu-Opatz, I. L., Bernard, C., Morellini, F., et al. (2015) · Nature Medicine · doi:10.1038/nm.3987 · Cited by 125
Pan, L., Zhang, P., Daengngam, C., Peng, S., & Chongcheawchamnan, M. (2022) · Journal of Raman Spectroscopy · doi:10.1002/jrs.6225 · Cited by 116
“Therefore it is impossible to construct clusters based on distance in high-dimensional space [46], [47].”
BackgroundSharif, F., Tayebi, B., Buzsáki, G., Royer, S., & Fernandez-Ruiz, A. (2021) · Neuron · doi:10.1016/j.neuron.2020.10.034 · Cited by 111
Glutamatergic synaptic integration of locomotion speed via septoentorhinal projections
Justus, D., Dalügge, D., Bothe, S., Fuhrmann, F., Hannes, C., Kaneko, H., Friedrichs, D., Sosulina, L., Schwarz, I., Elliott, D. A., et al. (2017) · Nature Neuroscience · doi:10.1038/nn.4447 · Cited by 82
Place field repetition and spatial learning in a multicompartment environment
Grieves, R. M., Jenkins, B. W., Harland, B. C., Wood, E. R., & Dudchenko, P. A. (2016) · Hippocampus · doi:10.1002/hipo.22496 · Cited by 72
“The energy, first principal component, peak amplitude, time at peak and width of waveform were used to sort spikes using the Klustakwik spike sorting algorithms (Kadir et al., 2014).”
MethodHippocampal place cells encode global location but not connectivity in a complex space
Duvelle, É., Grieves, R. M., Liu, A., Jedidi-Ayoub, S., Holeniewska, J., Harris, A., Nyberg, N., Donnarumma, F., Lefort, J. M., Jeffery, K. J., et al. (2021) · Current Biology · doi:10.1016/j.cub.2021.01.005 · Cited by 48
Tanni, S., de Cothi, W., & Barry, C. (2022) · Current Biology · doi:10.1016/j.cub.2022.06.046 · Cited by 37
Electrode-size dependent thresholds in subretinal neuroprosthetic stimulation
Corna, A., Herrmann, T., & Zeck, G. (2018) · Journal of Neural Engineering · doi:10.1088/1741-2552/aac1c8 · Cited by 42
Grid cells on steeply sloping terrain: evidence for planar rather than volumetric encoding
Hayman, R. M. A., Casali, G., Wilson, J. J., & Jeffery, K. J. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00925 · Cited by 40
“Single units were initially isolated using a cluster cutting program (Tint, Axona) in combination with KlustaKwik (Kadir et al., 2014) Automatic clustering of the data using KlustaKwik was fine-tuned and cleaned up manually (mainly to merge clusters falsely identified as separate).”
Method
Metadata
- Authors
- Kadir, S. N., Goodman, D. F. M., & Harris, K. D.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2014-11-01
- Volume
- 26
- Issue
- 11
- Pages
- 2379-2394
- ISSN
- 0899-7667 (ISSN-L)
- OpenAlex
- W2153886236
- Cited by
- 291
- References
- 27
- FWCI
- 31.69
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.