The DOI matches its Crossref record; 8 of the 9 references it cites were found in a registry or an index.
Predictive automatic relevance determination by expectation propagation
Abstract
In many real-world classification problems the input contains a large number of potentially ir-relevant features. This paper proposes a new Bayesian framework for determining the rele-vance of input features. This approach extends one of the most successful Bayesian methods for feature selection and sparse learning, known as Automatic Relevance Determination (ARD). ARD finds the relevance of features by optimiz-ing the model marginal likelihood, also known as the evidence. We show that this can lead to over-fitting. To address this problem, we propose Pre-dictive ARD based on estimating the predictive performance of the classifier. While the actual leave-one-out predictive performance is generally very costly to compute, the expectation propaga-tion (EP) algorithm proposed by Minka provides an estimate of this predictive performance as a side-effect of its iterations. We exploit this in our algorithm to do feature selection, and to select data points in a sparse Bayesian kernel classifier. Moreover, we provide two other improvements to previous algorithms, by replacing Laplace’s approximation with the generally more accurate EP, and by incorporating the fast optimization algorithm proposed by Faul and Tipping. Our experiments show that our method based on the EP estimate of predictive performance is more accurate on test data than relevance determina-tion by optimizing the evidence.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Mackay, D. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.415 · Cited by 4,440
Bayesian Learning for Neural Networks
Neal, R. M. (1996) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-0745-0 · Cited by 4,276
Expectation Propagation for approximate Bayesian inferencePreprint
Minka, T. P. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.2294 · Cited by 1,391
Self-citationRätsch, G., Onoda, T., & Müller, K. R. (2001) · Machine Learning · doi:10.1023/a:1007618119488 · Cited by 1,303
Tipping, M. E. (1999) · Cited by 869
Analysis of Sparse Bayesian Learning
Faul, A., & Tipping, M. E. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0054 · Cited by 289
Gaussian Processes for Classification: Mean-Field Algorithms
Opper, M., & Winther, O. (2000) · Neural Computation · doi:10.1162/089976600300014881 · Cited by 209
Bayesian automatic relevance determination algorithms for classifying gene expression data
Li, Y., Campbell, C., & Tipping, M. E. (2002) · Bioinformatics · doi:10.1093/bioinformatics/18.10.1332 · Cited by 174
Cited by
Sorted by citations
Items 101–110 of 110
Model Selection: An Empirical Study on Two Kernel Classifiers
Chu, W. (2006) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2006.1716309
Infinite Continuous Feature Model for Psychiatric Comorbidity Analysis
Valera, I., Ruiz, F. J. R., Olmos, P. M., Blanco, C., & Pérez‐Cruz, F. (2015) · Neural Computation · doi:10.1162/neco_a_00805
Cognitive-inspired speaker affective state profiling
Norhaslinda, K. (2012) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/51057
Variational Bayesian Multinomial Probit Regression with
Girolami, M., & Rogers, S. (2006)
Selection of Relevant Features for Raman Spectroscopy
Kemmler, M., & Denzler, J. (2015)
Brao, V., & Lozano, J. S. (2008) · RECERCAT (Consorci de Serveis Universitaris de Catalunya)
A Bayesian Practitioner’s Guide to Expectation Propagation
Zhou, J., Grazian, C., & Ormerod, J. T. (2026) · Statistical Science · doi:10.1214/24-sts957
Werhli, A. V., Lopes, P. P., Arce, O. E. A., Aderhold, A., & Machado, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15993-9_14
EigenGP: KL-expansion based Gaussian process learningPreprint
Qi, Y., Dai, B., & Zhu, Y. T. (2012) · arXiv (Cornell University)
Self-citation
Metadata
- Authors
- Qi, Y. (., Minka, T. P., Picard, R. W., & Ghahramani, Z.
- Affiliations
- MIT Media Lab (US)
- Microsoft Research (United Kingdom) (GB)
- Venue
- Twenty-first international conference on Machine learning - ICML '04
- Publisher
- ACM Press
- Published
- 2004
- Pages
- 85
- OpenAlex
- W2156140776
- Cited by
- 110
- References
- 9
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 3 · 2023: 2 · 2022: 3 · 2021: 3 · 2020: 4 · 2018: 1 · 2017: 2 · 2016: 1 · 2015: 6 · 2014: 3 · 2013: 9 · 2012: 9
- FWCI
- 5.83
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- Funders
- MIT Media Lab
- Gatsby Charitable Foundation
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Keywords
- automatic relevance determination, expectation propagation, feature selection, sparse Bayesian learning, overfitting, evidence, Laplace approximation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
