The DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index.
Deterministic annealing for clustering, compression, classification, regression, and related optimization problems
Abstract
The deterministic annealing approach to clustering and its extensions has demonstrated substantial performance improvement over standard supervised and unsupervised learning methods in a variety of important applications including compression, estimation, pattern recognition and classification, and statistical regression. The application-specific cost is minimized subject to a constraint on the randomness of the solution, which is gradually lowered. We emphasize the intuition gained from analogy to statistical physics. Alternatively the method is derived within rate-distortion theory, where the annealing process is equivalent to computation of Shannon's rate-distortion function, and the annealing temperature is inversely proportional to the slope of the curve. The basic algorithm is extended by incorporating structural constraints to allow optimization of numerous popular structures including vector quantizers, decision trees, multilayer perceptrons, radial basis functions, and mixtures of experts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
Optimization by Simulated Annealing
Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983) · Science · doi:10.1126/science.220.4598.671 · Cited by 45,134
Equation of State Calculations by Fast Computing Machines
Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., & Teller, E. (1953) · The Journal of Chemical Physics · doi:10.1063/1.1699114 · Cited by 37,192
Neural Networks: A Comprehensive Foundation
Haykin, S. (1998) · Cited by 30,070
(1967) · Cited by 23,719
Stochastic relaxation, gibbs distributions, and the bayesian restoration of images.
Geman, S., & Geman, D. (1984) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1984.4767596 · Cited by 17,949
Least squares quantization in PCM
Lloyd, S. (1982) · IEEE Transactions on Information Theory · doi:10.1109/tit.1982.1056489 · Cited by 15,836
Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686
Information theory and statistical mechanics
Jaynes, E. T. (1957) · Physical Review · doi:10.1103/physrev.106.620 · Cited by 12,931
(1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895
An Algorithm for Vector Quantizer Design
Linde, Y., Buzo, A., & Gray, R. (1980) · IEEE Transactions on Communications · doi:10.1109/tcom.1980.1094577 · Cited by 7,201
Vector Quantization and Signal Compression
Gersho, A., & Gray, R. M. (1992) · Springer eBooks · doi:10.1007/978-1-4615-3626-0 · Cited by 7,003
(1975) · CERN Document Server (European Organization for Nuclear Research) · Cited by 5,958
Information Theory and Reliable Communication.
Lee, P. M., & Gallager, R. T. (1970) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344305 · Cited by 5,399
Adaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384
A Fuzzy Relative of the ISODATA Process and Its Use in Detecting Compact Well-Separated Clusters
Dunn, J. C. (1973) · Journal of Cybernetics · doi:10.1080/01969727308546046 · Cited by 5,049
Maximum Likelihood from Incomplete Data via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1976) · Cited by 4,517
Pattern Classification and Scene Analysis
Thompson, M., Duda, R. O., & Hart, P. E. (1974) · Leonardo · doi:10.2307/1573081 · Cited by 4,495
Baum, L. E., Petrie, T., Soules, G., & Weiss, N. (1970) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177697196 · Cited by 4,297
Fast Learning in Networks of Locally-Tuned Processing Units
Moody, J., & Darken, C. J. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.281 · Cited by 4,194
Information Theory and Reliable Communication
Bell, D. A. (1969) · Electronics and Power · doi:10.1049/ep.1969.0319 · Cited by 4,012
Hierarchical Mixtures of Experts and the EM Algorithm
Jordan, M. I., & Jacobs, R. A. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.181 · Cited by 2,673
Mixture models : inference and applications to clustering
McLachlan, G. J., & Basford, K. E. (1988) · M. Dekker eBooks · Cited by 2,094
Quantizing for minimum distortion
Max, J. (1960) · IEEE Transactions on Information Theory · doi:10.1109/tit.1960.1057548 · Cited by 2,038
Hedonic housing prices and the demand for clean air
Harrison, D., & Rubinfeld, D. L. (1978) · Journal of Environmental Economics and Management · doi:10.1016/0095-0696(78)90006-2 · Cited by 1,854
Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy
Shore, J., & Johnson, R. (1980) · IEEE Transactions on Information Theory · doi:10.1109/tit.1980.1056144 · Cited by 1,774
'Neural-gas' network for vector quantization and its application to time-series prediction
Martinetz, T. M., Berkovich, S. G., & Schulten, K. J. (1993) · IEEE Transactions on Neural Networks · doi:10.1109/72.238311 · Cited by 1,431
Computation of channel capacity and rate-distortion functions
Blahut, R. (1972) · IEEE Transactions on Information Theory · doi:10.1109/tit.1972.1054855 · Cited by 1,381
A graduated assignment algorithm for graph matching
Gold, S., & Rangarajan, A. (1996) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.491619 · Cited by 1,090
Principles and practice of information theory
Blahut, R. E. (1987) · Medical Entomology and Zoology · Cited by 1,008
A Convergence Theorem for the Fuzzy ISODATA Clustering Algorithms
Bezdek, J. C. (1980) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1980.4766964 · Cited by 980
Stochastic Complexity and Modeling
Rissanen, J. (1986) · The Annals of Statistics · doi:10.1214/aos/1176350051 · Cited by 922
A clustering technique for summarizing multivariate data
Ball, G. H., & Hall, D. J. (1967) · Behavioral Science · doi:10.1002/bs.3830120210 · Cited by 912
An algorithm for computing the capacity of arbitrary discrete memoryless channels
Arimoto, S. (1972) · IEEE Transactions on Information Theory · doi:10.1109/tit.1972.1054753 · Cited by 838
An analogue approach to the travelling salesman problem using an elastic net method
Durbin, R., & Willshaw, D. (1987) · Nature · doi:10.1038/326689a0 · Cited by 797
Neural Networks and Related Methods for Classification
Ripley, B. D. (1994) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1994.tb01990.x · Cited by 627
Entropy-constrained vector quantization
Chou, P. A., Lookabaugh, T. D., & Gray, R. M. (1989) · IEEE Transactions on Acoustics Speech and Signal Processing · doi:10.1109/29.17498 · Cited by 584
On the training of radial basis function classifiers
Musavi, M. T., Ahmed, W., Chan, K. H., Faris, K. B., & Hummels, D. M. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80038-3 · Cited by 569
Speech coding based upon vector quantization
Buzo, A., Gray, A., Gray, R., & Markel, J. (1980) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1980.1163445 · Cited by 559
A study of vector quantization for noisy channels
Farvardin, N. (1990) · IEEE Transactions on Information Theory · doi:10.1109/18.53739 · Cited by 479
Classification and Regression Trees
Moisen, G. G. (2008) · Encyclopedia of Ecology · doi:10.1016/b978-008045405-4.00149-x · Cited by 468
Statistical mechanics and phase transitions in clustering
Rose, K., Gurewitz, E., & Fox, G. C. (1990) · Physical Review Letters · doi:10.1103/physrevlett.65.945 · Cited by 458
Self-citationPairwise data clustering by deterministic annealing
Hofmann, T., & Buhmann, J. M. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.566806 · Cited by 451
Parallel and deterministic algorithms from MRFs: surface reconstruction
Geiger, D., & Girosi, F. (1991) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.134040 · Cited by 447
Statistical pattern recognition with neural networks: benchmarking studies.
Kohonen, T., Barna, G., & Chrisley, R. (1988) · Cited by 409
Optimal pruning with applications to tree-structured source coding and modeling
Chou, P. A., Lookabaugh, T., & Gray, R. M. (1989) · IEEE Transactions on Information Theory · doi:10.1109/18.32124 · Cited by 407
A deterministic annealing approach to clustering
Rose, K., Gurewitz, E., & Fox, G. (1990) · Pattern Recognition Letters · doi:10.1016/0167-8655(90)90010-y · Cited by 382
Self-citationStatistical pattern recognition with neural networks: benchmarking studies
Kohonen, Barna, & Chrisley (1988) · IEEE International Conference on Neural Networks · doi:10.1109/icnn.1988.23829 · Cited by 352
Regression modeling in back-propagation and projection pursuit learning
Hwang, J.-N., Lay, S.-R., Maechler, M., Martin, R. D., & Schimert, J. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.286906 · Cited by 278
On the performance and complexity of channel-optimized vector quantizers
Farvardin, N., & Vaishampayan, V. (1991) · IEEE Transactions on Information Theory · doi:10.1109/18.61130 · Cited by 276
Vector quantization by deterministic annealing
Rose, K., Gurewitz, E., & Fox, G. C. (1992) · IEEE Transactions on Information Theory · doi:10.1109/18.144705 · Cited by 262
Self-citationNeural network approaches to image compression
Dony, R. D., & Haykin, S. (1995) · Proceedings of the IEEE · doi:10.1109/5.364461 · Cited by 258
NONLINEAR GATED EXPERTS FOR TIME SERIES: DISCOVERING REGIMES AND AVOIDING OVERFITTING
WEIGEND, A. S., MANGEAS, M., & SRIVASTAVA, A. N. (1995) · International Journal of Neural Systems · doi:10.1142/s0129065795000251 · Cited by 253
Optimal partitioning for classification and regression trees
Chou, P. A. (1991) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.88569 · Cited by 248
Generalized Deformable Models, Statistical Physics, and Matching Problems
Yuille, A. L. (1990) · Neural Computation · doi:10.1162/neco.1990.2.1.1 · Cited by 222
Competitive learning and soft competition for vector quantizer design
Yair, E., Zeger, K., & Gersho, A. (1992) · IEEE Transactions on Signal Processing · doi:10.1109/78.124940 · Cited by 199
Instabilities of Regression Estimates Relating Air Pollution to Mortality
McDonald, G. C., & Schwing, R. C. (1973) · Technometrics · doi:10.1080/00401706.1973.10489073 · Cited by 197
An Analysis of the Elastic Net Approach to the Traveling Salesman Problem
Durbin, R., Szeliski, R., & Yuille, A. (1989) · Neural Computation · doi:10.1162/neco.1989.1.3.348 · Cited by 184
Learning piecewise control strategies in a modular neural network architecture
Jacobs, R. A., & Jordan, M. I. (1993) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.229447 · Cited by 177
The design of joint source and channel trellis waveform coders
Ayanoglu, E., & Gray, R. (1987) · IEEE Transactions on Information Theory · doi:10.1109/tit.1987.1057376 · Cited by 175
Constrained clustering as an optimization method
Rose, K., Gurewitz, E., & Fox, G. C. (1993) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.236251 · Cited by 168
Self-citationA construction of vector quantizers for noisy channels
Kumazawa, H., Kasahara, M., & Namekawa, T. (1984) · Electronics and Communications in Japan (Part I Communications) · doi:10.1002/ecja.4400670406 · Cited by 152
A tutorial survey of theory and applications of simulated annealing
Hajek, B. (1985) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.1985.268599 · Cited by 148
Riskin, E. A., & Gray, R. M. (1991) · IEEE Transactions on Signal Processing · doi:10.1109/78.98004 · Cited by 147
A mapping approach to rate-distortion computation and analysis
Rose, K. H. (1994) · IEEE Transactions on Information Theory · doi:10.1109/18.340468 · Cited by 140
Self-citationStatistical mechanics as the underlying theory of ‘elastic’ and ‘neural’ optimisations
Simić, P. (1990) · Network Computation in Neural Systems · doi:10.1088/0954-898x_1_1_007 · Cited by 123
Gradient algorithms for designing predictive vector quantizers
Chang, P., & Gray, R. M. (1986) · IEEE Transactions on Acoustics Speech and Signal Processing · doi:10.1109/tassp.1986.1164905 · Cited by 107
Derivation of a class of training algorithms
Luttrell, S. P. (1990) · IEEE Transactions on Neural Networks · doi:10.1109/72.80234 · Cited by 97
Vector quantization with complexity costs
Buhmann, J. M., & Kühnel, H. (1993) · IEEE Transactions on Information Theory · doi:10.1109/18.243432 · Cited by 96
A global optimization technique for statistical classifier design
Miller, D. J., Rao, A. V., Rose, K. H., & Gersho, A. (1996) · IEEE Transactions on Signal Processing · doi:10.1109/78.553484 · Cited by 94
Self-citationPhase transitions in stochastic self-organizing maps
Graepel, T., Burger, M., & Obermayer, K. (1997) · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · doi:10.1103/physreve.56.3876 · Cited by 89
A least biased fuzzy clustering method
Beni, G., & Liu, X. (1994) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.310694 · Cited by 88
Multiple local optima in vector quantizers (Corresp.)
Gray, R. M., & Karnin, E. D. (1982) · IEEE Transactions on Information Theory · doi:10.1109/tit.1982.1056471 · Cited by 82
Deterministic Annealing Variant of the EM Algorithm
Ueda, N., & Nakano, R. (1994) · neural information processing systems · Cited by 72
Constrained Nets for Graph Matching and Other Quadratic Assignment Problems
Simić, P. (1991) · Neural Computation · doi:10.1162/neco.1991.3.2.268 · Cited by 70
Wong, Y.-F. (1993) · Neural Computation · doi:10.1162/neco.1993.5.1.89 · Cited by 66
Vector quantizer design for memoryless noisy channels
Zeger, K., & Gersho, A. (2003) · International Conference on Communications · doi:10.1109/icc.1988.13817 · Cited by 65
Mixture of experts regression modeling by deterministic annealing
Rao, A. V., Miller, D. J., Rose, K. H., & Gersho, A. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650107 · Cited by 60
Self-citationA deterministic annealing approach for parsimonious design of piecewise regression models
Rao, A. V., Miller, D. J., Rose, K. H., & Gersho, A. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.748824 · Cited by 59
Self-citationCombined source-channel vector quantization using deterministic annealing
Miller, D. J., & Rose, K. H. (1994) · IEEE Transactions on Communications · doi:10.1109/tcomm.1994.577056 · Cited by 41
Self-citationCustomized ECG beat classifier using mixture of experts
Hu, Y. H., Palreddy, S., & Tompkins, W. J. (2002) · IEEE Workshop on Neural Networks for Signal Processing (NNSP) · doi:10.1109/nnsp.1995.514920 · Cited by 7
Design of robust HMM speech recognizers using deterministic annealing
Rao, A. V., Rose, K. H., & Gersho, A. (2002) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.1997.659125 · Cited by 6
Self-citationA deterministic annealing approach to discriminative hidden Markov model design
Rao, A. V., Rose, K. H., & Gersho, A. (2002) · Machine learning for signal processing ... · doi:10.1109/nnsp.1997.622407 · Cited by 4
Self-citation
Cited by
Statistical pattern recognition: a review
Jain, A. K., Duin, P. W., & Mao, J. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.824819 · Cited by 6,767
Survey of Clustering Algorithms
Xu, R., & WunschII, D. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2005.845141 · Cited by 6,197
Alon, U., Barkai, N., Notterman, D. A., Gish, K., Ybarra, S., Mack, D., & Levine, A. J. (1999) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.96.12.6745 · Cited by 4,167
Statistical shape analysis: clustering, learning, and testing
Srivastava, A., Joshi, S. H., Mio, W., & Liu, X. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.86 · Cited by 2,727
Unsupervised learning of finite mixture models
Figueiredo, M. A. T., & Jain, A. K. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.990138 · Cited by 2,125
Metadata
- Authors
- Rose, K.
- Venue
- Proceedings of the IEEE
- Published
- 1998
- Volume
- 86
- Issue
- 11
- Pages
- 2210-2239
- ISSN
- 0018-9219 (ISSN-L)
- OpenAlex
- W2161877964
- Cited by
- 876
- References
- 86
- FWCI
- 17.94
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.