The Variational Bayesian EM Algorithm for Incomplete Data: With Application to Scoring Graphical Model Structures
Abstract
Abstract We present an efficient procedure for estimating the marginal likelihood of probabilistic models with latent variables or incomplete data. This method constructs and optimizes a lower bound on the marginal likelihood using variational calculus, resulting in an iterative algorithm which generalizes the EM algorithm by maintaining posterior distributions over both latent variables and parameters. We define the family of conjugate-exponential models—which includes finite mixtures of exponential family models, factor analysis, hidden Markov models, linear state-space models, and other models of interest—for which this bound on the marginal likelihood can be computed very simply through a modification of the standard EM algorithm. In particular, we focus on applying these bounds to the problem of scoring discrete directed graphical model structures (Bayesian networks). Extensive simulations comparing the variational bounds to the usual approach based on the Bayesian Information Criterion (BIC) and to a sampling-based gold standard method known as Annealed Importance Sampling (AIS) show that variational bounds substantially outperform BIC in finding the correct model structure at relatively little computational cost, while approaching the performance of the much more costly AIS procedure. Using AIS allows us to provide the first serious case study of the tightness of variational bounds. We also analyze the performance of AIS through a variety of criteria, and outline directions in which this work can be extended.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Free Energy method approached the performance of AIS and clearly outperformed BIC in comparisons based on simulation of graphical model structures.
“In other comparisons, based on simulation of graphical model structures [6] the Free Energy method approached the performance of AIS and clearly outperformed BIC.”
From Comparing families of dynamic causal models (2010) · cites this paper as [6]
Previous work in graphical models favours the free energy approach over BIC, and work on biochemical models finds AIS to be the best of the more computationally expensive sampling methods.
“Previous work in graphical models favours the free energy approach over BIC [6] and work on biochemical models finds AIS to be the best of the more computationally expensive sampling methods.”
From Comparing families of dynamic causal models (2010) · cites this paper as [6]
The Free Energy is a better approximation to the model evidence than BIC.
“This adaptability makes the Free Energy a better approximation to the model evidence, as has been shown empirically [6],[31].”
From Comparing families of dynamic causal models (2010) · cites this paper as [6]
Cites
Maximum Likelihood from Incomplete Data Via the <i>EM</i> Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444 · see citing works
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033 · see citing works
(1988) · Cited by 16,582 · see citing works
Lauritzen, S. L., & Spiegelhalter, D. J. (1988) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1988.tb01721.x · Cited by 3,907 · see citing works
An Introduction to Variational Methods for Graphical Models
Jordan, M. I., Zoubin Ghahramani, A., Jaakkola, T., & Saul, L. K. (1999) · Machine Learning · doi:10.1023/a:1007665907178 · Cited by 3,665 · see citing works
A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants
Neal, R. M., & Hinton, G. E. (1998) · Learning in Graphical Models · doi:10.1007/978-94-011-5014-9_12 · Cited by 2,187 · see citing works
Marginal Likelihood from the Gibbs Output
Chib, S. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476635 · Cited by 1,815 · see citing works
A Tutorial on Learning with Bayesian Networks
Heckerman, D. E. (2008) · Studies in computational intelligence · doi:10.1007/978-3-540-85066-3_3 · Cited by 1,712 · see citing works
(2003) · Cited by 1,680 · see citing works
Neal, R. M. (2001) · Statistics and Computing · doi:10.1023/a:1008923215028 · Cited by 1,208 · see citing works
Keeping the neural networks simple by minimizing the description length of the weights
Hinton, G. E., & van Camp, D. (1993) · Proceedings of the sixth annual conference on Computational learning theory - COLT '93 · doi:10.1145/168304.168306 · Cited by 994 · see citing works
Simulating normalizing constants: from importance sampling to bridge sampling to path sampling
Gelman, A., & Meng, X. (1998) · Statistical Science · doi:10.1214/ss/1028905934 · Cited by 986 · see citing works
Mackay, D. J. C. (1995) · Network Computation in Neural Systems · doi:10.1088/0954-898x/6/3/011 · Cited by 915 · see citing works
Generalized Belief Propagation
Yedidia, J. S., Freeman, W. T., & Weiss, Y. (2000) · Cited by 894 · see citing works
A Variational Baysian Framework for Graphical Models
Attias, H. T. (1999) · neural information processing systems · Cited by 689 · see citing works
Ockham's Razor and Bayesian Analysis
Jefferys, W. H., & Berger, J. O. (1992) · American Scientist · Cited by 541 · see citing works
The Bayesian Structural EM AlgorithmPreprint
Friedman, N. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.7373 · Cited by 511 · see citing works
Probabilistic Networks and Expert Systems
Cowell, R., Dawid, A., Lauritzen, S. L., & Spiegelhalter, D. J. (1999) · Information Science and Statistics · doi:10.1007/b97670 · Cited by 497 · see citing works
Variational Inference for Bayesian Mixtures of Factor Analysers
Ghahramani, Z., & Beal, M. J. (1999) · Cited by 380 · see citing works
Self-citationEfficient Approximations for the Marginal Likelihood of Bayesian Networks with Hidden Variables
Chickering, D. M., & Heckerman, D. E. (1997) · Machine Learning · doi:10.1023/a:1007469629108 · Cited by 287 · see citing works
Propagation Algorithms for Variational Bayesian Learning
Ghahramani, Z., & Beal, M. J. (2000) · Cited by 281 · see citing works
Self-citationBayesian Methods for Mixtures of Experts
Waterhouse, S. R., Mackay, D., & Robinson, A. J. (1995) · Cited by 175 · see citing works
Rasmussen, C. E., & Ghahramani, Z. (2000) · neural information processing systems · Cited by 132 · see citing works
Self-citationVariational methods for inference and estimation in graphical models
Jaakkola, T., & Jordan, M. I. (1997) · Cited by 91 · see citing works
Asymptotic Model Selection for Directed Networks with Hidden Variables
Geiger, D., Heckerman, D. E., & Meek, C. (1998) · Springer eBooks · doi:10.1007/978-94-011-5014-9_16 · Cited by 77 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Hierarchical Models in the Brain
Friston, K. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000211 · Cited by 952 · see citing works
Comparing families of dynamic causal models
Penny, W. D., Stephan, K. E., Daunizeau, J., Rosa, M. J., Friston, K. J., Schofield, T. M., & Leff, A. P. (2010) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000709 · Cited by 688 · see citing works
Comparing dynamic causal models using AIC, BIC and free energy
Penny, W. D. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.07.039 · Cited by 298 · see citing works
DEM: A variational treatment of dynamic systems
Friston, K. J., Trujillo-Barreto, N., & Daunizeau, J. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.02.054 · Cited by 278 · see citing works
Model averaging, optimal inference, and habit formation
FitzGerald, T. H. B., Dolan, R. J., & Friston, K. J. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00457 · Cited by 121 · see citing works
Laplace approximation in sparse Bayesian learning for structural damage detection
Wang, X., Hou, R., Xia, Y., & Zhou, X. (2020) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2020.106701 · Cited by 32 · see citing works
- Authors
- Beal, M. J., & Ghahramani, Z.
- Affiliations
- Gatsby Computational Neuroscience Unit, UCL. UK
- Venue
- Bayesian Statistics 7
- Publisher
- Oxford University PressOxford
- Published
- 2003-07-03
- Pages
- 453-463
- OpenAlex
- W1570770495
- Cited by
- 458
- References
- 25
- Citations by year
- 2026: 3 · 2025: 10 · 2024: 8 · 2023: 10 · 2022: 20 · 2021: 22 · 2020: 31 · 2019: 21 · 2018: 22 · 2017: 31 · 2016: 24 · 2015: 25 · 2014: 35 · 2013: 25 · 2012: 20
- FWCI
- 21.97
- Type
- book chapter
- Language
- en
- Open access
- closed
- Locations
- Cambridge University Engineering Department Publications Database (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Methods and Mixture Models
- Physical Sciences › Mathematics › Statistics and Probability › Markov Chains and Monte Carlo Methods
- Keywords
- marginal likelihood estimation, missing data, latent variables, variational lower bound, Bayesian network structure learning, Bayesian information criterion, annealed importance sampling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
