High-Dimensional Unsupervised Selection and Estimation of a Finite Generalized Dirichlet Mixture Model Based on Minimum Message Length
The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
We consider the problem of determining the structure of high-dimensional data, without prior knowledge of the number of clusters. Data are represented by a finite mixture model based on the generalized Dirichlet distribution. The generalized Dirichlet distribution has a more general covariance structure than the Dirichlet distribution and offers high flexibility and ease of use for the approximation of both symmetric and asymmetric distributions. This makes the generalized Dirichlet distribution more practical and useful. An important problem in mixture modeling is the determination of the number of clusters. Indeed, a mixture with too many or too few components may not be appropriate to approximate the true model. Here, we consider the application of the minimum message length (MML) principle to determine the number of clusters. The MML is derived so as to choose the number of clusters in the mixture model which best describes the data. A comparison with other selection criteria is performed. The validation involves synthetic data, real data clustering, and two interesting real applications: classification of web pages, and texture database summarization for efficient retrieval.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879 · cited 1×
“Ç”
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444 · cited 1×
“The EM algorithm [44] is a general approach to ML in the presence of incomplete data.”
MethodEstimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033 · cited 1×
“Ç”
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
McLachlan, G., & Peel, D. (2000) · Wiley Series in Probability and Statistics · doi:10.1002/0471721182 · Cited by 10,991
UCI Repository of machine learning databases
Blake, C. L. (1998) · Medical Entomology and Zoology · Cited by 10,582
Finding Groups in Data: An Introduction to Cluster Analysis
Kaufman, L., & Rousseeuw, P. J. (1990) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316801 · Cited by 7,344
Statistical pattern recognition: a review
Jain, A. K., Duin, P. W., & Mao, J. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.824819 · Cited by 6,767 · cited 1×
“Fields in which mixture models have been successfully applied include image processing, pattern recognition, machine learning, and remote sensing [ 4 ].”
BackgroundModeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011 · cited 3×
“The methods we compare with are the MDL proposed by Rissanen in [ 15 ], the MMDL [18], the AIC [13], the MML-like criterion, which we call MMLlike, proposed by Figueiredo and Jain in [12], and the LEC [2].”
MethodInfluentialThe EM Algorithm and Extensions
Kushary, D. (1998) · Technometrics · doi:10.1080/00401706.1998.10485534 · Cited by 5,417
Wiley Series in Probability and Statistics
(2008) · Wiley series in probability and statistics · doi:10.1002/series1345 · Cited by 4,217
Sphere Packings, Lattices and Groups.
Coxeter, H. S. M., Conway, J. H., & Sloane, N. J. A. (1989) · The American Mathematical Monthly · doi:10.2307/2323992 · Cited by 3,377
Automatic text processing: the transformation, analysis, and retrieval of information by computer
(1989) · Choice Reviews Online · doi:10.5860/choice.27-0351 · Cited by 3,188 · cited 1×
“The goal of this application is to understand the textual content of a Web page based on statistical features by considering single-word statistics, that is, the frequency of word occurrence [58].”
Background(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/jmlr.2003.3.4-5.993 · Cited by 3,024
How Many Clusters? Which Clustering Method? Answers Via Model-Based Cluster Analysis
Fraley, C. (1998) · The Computer Journal · doi:10.1093/comjnl/41.8.578 · Cited by 2,549 · cited 1×
“The Ruspini [52] data set contains 2D data in four groups (see Fig. 3a). Chevron is another 2D data set [ 53 ].”
BackgroundUnsupervised learning of finite mixture models
Figueiredo, M. A. T., & Jain, A. K. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.990138 · Cited by 2,125 · cited 2×
“The complete-data Fisher information matrix has a block-diagonal structure and its determinant is equal to the product of the determinant of the information matrix for each component times the determinant of the information matrix of ~P jF ð Þj ’ jF ð~P Þj YM j¼1 jF ð~ jÞj; ð13Þ where jF ð~P Þj is…”
BackgroundOn Bayesian Analysis of Mixtures with an Unknown Number of Components (with discussion)
Richardson, S., & Green, P. J. (1997) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00095 · Cited by 1,898
QBIC project: querying images by content, using color, texture, and shape:
Niblack, C. W., Barber, R., Equitz, W., Flickner, M. D., Glasman, E. H., Petkovic, D., Yanker, P., Faloutsos, C., & Taubin, G. (1993) · SPIE Proceedings · doi:10.1117/12.143648 · Cited by 1,747
VisualSEEk: a fully automated content-based image query system
Smith, J. R., & Chang, S. (1996) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/244130.244151 · Cited by 1,745 · cited 2×
“developed to handle pictorial queries, for example, QBIC [60], Photobook [61], Blobworld [62], VisualSeek [63], and Atlas [64].”
MethodImage indexing using color correlograms
Huang, J., Kumar, S. R., Mitra, M., Zhu, W.-J., & Zabih, R. (2002) · Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.1997.609412 · Cited by 1,646 · cited 1×
“In order to determine the vector of characteristics for each image, we have computed a set of features derived from the correlogram [67].”
MethodPhotobook: Content-based manipulation of image databases
Pentland, A., Picard, R. W., & Sclaroff, S. (1996) · International Journal of Computer Vision · doi:10.1007/bf00123143 · Cited by 1,562 · cited 2×
“developed to handle pictorial queries, for example, QBIC [60], Photobook [61], Blobworld [62], VisualSeek [63], and Atlas [64].”
MethodBlobworld: image segmentation using expectation-maximization and its application to image querying
Carson, C., Belongie, S., Greenspan, H., & Malik, J. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1023800 · Cited by 1,410 · cited 2×
“developed to handle pictorial queries, for example, QBIC [60], Photobook [61], Blobworld [62], VisualSeek [63], and Atlas [64].”
MethodFiltering for texture classification: a comparative study
Randen, T., & Husoy, J. H. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.761261 · Cited by 1,397 · cited 1×
“Some studies show that considering the following neighborhoods is sufficient for co-occurrence matrices, in the case of gray-level images, to obtain good results in general: ð1; 0Þ, ð1; 4Þ, ð1; 2Þ, and ð1; 3 4 Þ [68].”
BackgroundEveritt, B. S. (2005) · Encyclopedia of Statistics in Behavioral Science · doi:10.1002/0470013192.bsa231 · Cited by 1,263
Everitt, B. S., & Hand, D. J. (1981) · Springer eBooks · doi:10.1007/978-94-009-5897-5 · Cited by 1,058
The minimum description length principle in coding and modeling
Barron, A., Rissanen, J., & Yu, B. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.720554 · Cited by 1,019 · cited 1×
“Ç”
An Information Measure for Classification
Wallace, C. S., & Boulton, D. M. (1968) · The Computer Journal · doi:10.1093/comjnl/11.2.185 · Cited by 933 · cited 2×
“Ç”
BackgroundUniversal coding, information, prediction, and estimation
Rissanen, J. (1984) · IEEE Transactions on Information Theory · doi:10.1109/tit.1984.1056936 · Cited by 898 · cited 1×
“Ç”
Sphere Packings, Lattices and Groups
Conway, J. H., & Sloane, N. J. A. (1993) · Grundlehren der mathematischen Wissenschaften · doi:10.1007/978-1-4757-2249-9 · Cited by 880
Image classification for content-based indexing
Vailaya, A., Figueiredo, M. A. T., Jain, A. K., & Zhang, H.-J. (2001) · IEEE Transactions on Image Processing · doi:10.1109/83.892448 · Cited by 762 · cited 1×
“Summarizing the database is very important because it simplifies the task of retrieval by restricting the search for similar images to a smaller domain of the database [65].”
BackgroundMcLachlan, G. J. (1987) · Applied Statistics · doi:10.2307/2347790 · Cited by 708 · cited 1×
“Index Terms—Finite mixture models, generalized Dirichlet mixture, EM, information theory, MML, AIC, MDL, MMDL, LEC, data clustering, image database summarization, Web mining.”
MethodSum and Difference Histograms for Texture Classification
Unser, M. (1986) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1986.4767760 · Cited by 529
Estimation and Inference by Compact Coding
Wallace, C. S., & Freeman, P. R. (1987) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1987.tb01695.x · Cited by 523 · cited 1×
“In the following sections, we will calculate the determinant of the Fisher information matrix jF ð Þj and the prior pdf hð Þ for a mixture of generalized Dirichlet distributions.”
MethodPractical Bayesian Density Estimation Using Mixtures of Normals
Roeder, K., & Wasserman, L. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474044 · Cited by 503 · cited 1×
“The first is the implementationofmodelselectioncriteria[5],[ 6 ].Thesecond is fully Bayesian and consists of resampling from the full posteriordistribution,withthenumberofclustersconsidered to be unknown [7].”
BackgroundAdvances in Minimum Description Length: Theory and Applications
Grünwald, P., Myung, I. J., & Pitt, M. A. (2005) · Data Archiving and Networked Services (DANS) · Cited by 408
Statistical and Inductive Inference by Minimum Message Length
(2005) · Information Science and Statistics · doi:10.1007/0-387-27656-4 · Cited by 349
Minimum Message Length and Kolmogorov Complexity
Wallace, C. S. (1999) · The Computer Journal · doi:10.1093/comjnl/42.4.270 · Cited by 347 · cited 1×
“In the following sections, we will calculate the determinant of the Fisher information matrix jF ð Þj and the prior pdf hð Þ for a mixture of generalized Dirichlet distributions.”
MethodSjölander, K., Karplus, K., Brown, M., Hughey, R., Krogh, A., Mian, I. S., & Haussler, D. (1996) · Bioinformatics · doi:10.1093/bioinformatics/12.4.327 · Cited by 330
Bayesian approaches to Gaussian mixture modeling
Roberts, S. J., Husmeier, D., Rezek, I., & Penny, W. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.730550 · Cited by 317 · cited 1×
“Ç”
Model selection for probabilistic clustering using cross-validated likelihood
Smyth, P. (2000) · Statistics and Computing · doi:10.1023/a:1008940618127 · Cited by 285 · cited 1×
“To select the number of clusters, resampling schemes [8] and cross-validation approaches [ 9 ] have also been used.”
MethodInference in model-based cluster analysis
Bensmail, H., Celeux, G., Raftery, A. E., & Robert, C. P. (1997) · Statistics and Computing · doi:10.1023/a:1018510926151 · Cited by 199 · cited 1×
“The validation involves synthetic data, real data clustering, and two interesting real applications: classification of Web pages, and texture database summarization for efficient retrieval.”
MethodBouguila, N., Ziou, D., & Vaillancourt, J. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2004.834664 · Cited by 195 · cited 1×
“The estimation of the number of clusters is carried out by finding the minimum with regards to of the message length MessLen.”
BackgroundSelf-citationAn attempt to define the nature of chemical diabetes using a multidimensional analysis
Reaven, G. M., & Miller, R. G. (1979) · Diabetologia · doi:10.1007/bf00423145 · Cited by 188 · cited 1×
“Diabetes is a 3D data set involving 145 observations used for diabetes diagnosis [54].”
BackgroundA stochastic approximation type EM algorithm for the mixture problem
Celeux, G., & Diebolt, J. (1992) · Stochastics and stochastics reports · doi:10.1080/17442509208833797 · Cited by 172 · cited 1×
“This rapid convergence can be improved by introducing a stochastic step in the EM algorithm [48], [49] that prevents the sequence of estimates t from staying near an unstable stationary point of the likelihood function [49].”
BackgroundMML clustering of multi-state, Poisson, von Mises circular and Gaussian distributions
Wallace, C. S., & Dowe, D. L. (2000) · Statistics and Computing · doi:10.1023/a:1008992619036 · Cited by 167 · cited 6×
“The message length is defined as minus the logarithm of the posterior probability (interesting discussion about the difference between maximizing a density andmaximizing a probability can be found in [ 28 ]).”
BackgroundMethodInfluentialStatistical and Inductive Inference by Minimum Message Length (Information Science and Statistics)
Wallace, C. S. (2005) · Springer eBooks · Cited by 99
Finding overlapping components with MML
Baxter, R. A., & Oliver, J. (2000) · Statistics and Computing · doi:10.1023/a:1008928315401 · Cited by 83 · cited 2×
“In the following sections, we will calculate the determinant of the Fisher information matrix jF ð Þj and the prior pdf hð Þ for a mixture of generalized Dirichlet distributions.”
MethodCombining positive and negative examples in relevance feedback for content-based image retrieval
Kherfi, M. L., Ziou, D., & Bernardi, A. (2003) · Journal of Visual Communication and Image Representation · doi:10.1016/s1047-3203(03)00043-9 · Cited by 51 · cited 2×
“developed to handle pictorial queries, for example, QBIC [60], Photobook [61], Blobworld [62], VisualSeek [63], and Atlas [64].”
MethodSelf-citationUnsupervised Learning of Correlated Multivariate Gaussian Mixture Models Using MML
Agusta, Y., & Dowe, D. L. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-24581-0_40 · Cited by 27 · cited 1×
“The superiority of MML is also reported by Agusta and Dowe in [57], where they show that MML beats both AIC and BIC, in bit costing, with multivariate Gaussian mixtures, for the Iris and Diabetes data sets.”
MethodBouguila, N., & Ziou, D. (2006) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvpr.2005.493 · Cited by 26
Self-citationSingle factor analysis in MML mixture modelling
Edwards, R. T., & Dowe, D. L. (1998) · Lecture notes in computer science · doi:10.1007/3-540-64383-4_9 · Cited by 23 · cited 1×
“Then, we are using a partial assignment that is different of the total assignment used in [11], [50].”
MethodUnsupervised learning of Gamma mixture models using Minimum Message Length
Agusta, Y., & Dowe, D. L. (2003) · Cited by 20
MML Clustering of Continuous-Valued Data Using Gaussian and t Distributions
Agusta, Y., & Dowe, D. L. (2002) · Lecture notes in computer science · doi:10.1007/3-540-36187-1_13 · Cited by 11 · cited 1×
“The deterministic methods can them- selves be classified in two main classes.”
MethodProbability Density Estimation
Scott, D. W. (2015) · Elsevier eBooks · doi:10.1016/b978-0-08-097086-8.42066-0 · Cited by 11
Metadata
- Authors
- Bouguila, N., & Ziou, D.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2007-10
- Volume
- 29
- Issue
- 10
- Pages
- 1716-1731
- ISSN
- 0162-8828 (ISSN-L)
- OpenAlex
- W2096784803
- Cited by
- 169
- References
- 54
- FWCI
- 13.00
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.