The DOI matches its Crossref record; 8 of the 9 references it cites were found in a registry or an index.
Predictive automatic relevance determination by expectation propagation
Abstract
In many real-world classification problems the input contains a large number of potentially ir-relevant features. This paper proposes a new Bayesian framework for determining the rele-vance of input features. This approach extends one of the most successful Bayesian methods for feature selection and sparse learning, known as Automatic Relevance Determination (ARD). ARD finds the relevance of features by optimiz-ing the model marginal likelihood, also known as the evidence. We show that this can lead to over-fitting. To address this problem, we propose Pre-dictive ARD based on estimating the predictive performance of the classifier. While the actual leave-one-out predictive performance is generally very costly to compute, the expectation propaga-tion (EP) algorithm proposed by Minka provides an estimate of this predictive performance as a side-effect of its iterations. We exploit this in our algorithm to do feature selection, and to select data points in a sparse Bayesian kernel classifier. Moreover, we provide two other improvements to previous algorithms, by replacing Laplace’s approximation with the generally more accurate EP, and by incorporating the fast optimization algorithm proposed by Faul and Tipping. Our experiments show that our method based on the EP estimate of predictive performance is more accurate on test data than relevance determina-tion by optimizing the evidence.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Mackay, D. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.415 · Cited by 4,440
Bayesian Learning for Neural Networks
Neal, R. M. (1996) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-0745-0 · Cited by 4,276
Expectation Propagation for approximate Bayesian inferencePreprint
Minka, T. P. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.2294 · Cited by 1,391
Self-citationRätsch, G., Onoda, T., & Müller, K. R. (2001) · Machine Learning · doi:10.1023/a:1007618119488 · Cited by 1,303
Tipping, M. E. (1999) · Cited by 869
Analysis of Sparse Bayesian Learning
Faul, A., & Tipping, M. E. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0054 · Cited by 289
Gaussian Processes for Classification: Mean-Field Algorithms
Opper, M., & Winther, O. (2000) · Neural Computation · doi:10.1162/089976600300014881 · Cited by 209
Bayesian automatic relevance determination algorithms for classifying gene expression data
Li, Y., Campbell, C., & Tipping, M. E. (2002) · Bioinformatics · doi:10.1093/bioinformatics/18.10.1332 · Cited by 174
Cited by
Sorted by citations
(2010) · doi:10.5555/1756006.1859921 · Cited by 1,924
Zhang, Y., Tang, Q., Zhang, Y., Wang, J., Stimming, U., & Lee, A. A. (2020) · Nature Communications · doi:10.1038/s41467-020-15235-7 · Cited by 889
Sparse multinomial logistic regression: fast algorithms and generalization bounds
Krishnapuram, B., Carin, L., Figueiredo, M. A. T., & Hartemink, A. J. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.127 · Cited by 883
Bayesian Inference and Optimal Design for the Sparse Linear Model
Seeger, M. (2008) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.5555/1390681.1390707 · Cited by 278
Cawley, G. C., & Talbot, N. L. C. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl386 · Cited by 243
Metadata
- Authors
- Qi, Y. (., Minka, T. P., Picard, R. W., & Ghahramani, Z.
- Affiliations
- MIT Media Lab (US)
- Microsoft Research (United Kingdom) (GB)
- Venue
- Twenty-first international conference on Machine learning - ICML '04
- Publisher
- ACM Press
- Published
- 2004
- Pages
- 85
- OpenAlex
- W2156140776
- Cited by
- 110
- References
- 9
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 3 · 2023: 2 · 2022: 3 · 2021: 3 · 2020: 4 · 2018: 1 · 2017: 2 · 2016: 1 · 2015: 6 · 2014: 3 · 2013: 9 · 2012: 9
- FWCI
- 5.83
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- Funders
- MIT Media Lab
- Gatsby Charitable Foundation
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Keywords
- automatic relevance determination, expectation propagation, feature selection, sparse Bayesian learning, overfitting, evidence, Laplace approximation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
