Bayes and big data: the consensus Monte Carlo algorithm
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
A useful definition of ‘big data’ is data that is too big to process comfortably on a single machine, either because of processor, memory, or disk bottlenecks. Graphics processing units can alleviate the processor bottleneck, but memory or disk bottlenecks can only be eliminated by splitting data across multiple machines. Communication between large numbers of machines is expensive (regardless of the amount of data being communicated), so there is a need for algorithms that perform distributed approximate Bayesian analyses with minimal communication. Consensus Monte Carlo operates by running a separate Monte Carlo algorithm on each machine, and then averaging individual Monte Carlo draws across machines. Depending on the model, the resulting draws can be nearly indistinguishable from the draws that would have been obtained by running a single-machine algorithm for a very long time. Examples of consensus Monte Carlo are shown for simple models where single-machine solutions are available, for large single-layer hierarchical models, and for Bayesian additive regression trees (BART).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
Multivariate Adaptive Regression Splines
Friedman, J. H. (1991) · The Annals of Statistics · doi:10.1214/aos/1176347963 · Cited by 8,000
Rue, H., Martino, S., & Chopin, N. (2009) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2008.00700.x · Cited by 5,279
MCMC Methods for Multi-Response Generalized Linear Mixed Models: The MCMCglmmR Package
Hadfield, J. D. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i02 · Cited by 4,799
Pregel: a system for large-scale graph processing
Malewicz, G., Austern, M. H., Bik, A. J. C., Dehnert, J. C., Horn, I., Leiser, N., & Czajkowski, G. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807184 · Cited by 3,515
White, T. (2009) · Cited by 2,800
Keulen, D. S. (2014) · Cited by 1,677
BART: Bayesian additive regression trees
Chipman, H. A., George, E. I., & McCulloch, R. E. (2010) · The Annals of Applied Statistics · doi:10.1214/09-aoas285 · Cited by 1,542
Self-citationKeeping the neural networks simple by minimizing the description length of the weights
Hinton, G. E., & van Camp, D. (1993) · Proceedings of the sixth annual conference on Computational learning theory - COLT '93 · doi:10.1145/168304.168306 · Cited by 994 · cited 1×
“© google, Inc. 2016. all rights reserved KEYWORDS Bayesian inference; Markov chain Monte carlo; distributed computing; big data; embarrassingly parallel ARTICLE HISTORY received 31 october 2013 accepted 1 december 2015 CONTACT Steven l. Scott stevescott@google.com”
MethodBART: Bayesian Additive Regression Trees
Chipman, H., George, E. I., & McCulloch, R. E. (2006) · ScholarlyCommons (University of Pennsylvania) · Cited by 889
Self-citationA case for NOW (Networks of Workstations)
Anderson, T. E., Culler, D. E., & Patterson, D. (1995) · IEEE Micro · doi:10.1109/40.342018 · Cited by 798
Bayesian parameter estimation via variational methods
Jaakkola, T. S., & Jordan, M. I. (2000) · Statistics and Computing · doi:10.1023/a:1008932416310 · Cited by 603
Asymptotics in statistics : some basic concepts
Cam, L. M. L., & Yang, G. L. (1990) · Springer eBooks · Cited by 444
Automatic Bayesian Curve Fitting
Denison, D. G. T., Mallick, B. K., & Smith, A. F. M. (1998) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00128 · Cited by 394
A Scalable Bootstrap for Massive Data
Kleiner, A., Talwalkar, A., Sarkar, P., & Jordan, M. I. (2014) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12050 · Cited by 389
Propagation Algorithms for Variational Bayesian Learning
Ghahramani, Z., & Beal, M. J. (2000) · Cited by 283
Communication-efficient algorithms for statistical optimization
Zhang, Y., Duchi, J. C., & Wainwright, M. J. (2012) · 2012 IEEE 51st IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2012.6426691 · Cited by 283
Le Cam, L., & Lo Yang, G. (2000) · Springer Series in Statistics · doi:10.1007/978-1-4612-1166-2 · Cited by 253
Lee, A., Yau, C., Giles, M. B., et al. (2009) · Oxford University Research Archive (ORA) (University of Oxford) · Cited by 251
Scalable Bayes via Barycenter in Wasserstein SpacePreprint
Srivastava, S., Li, C., & Dunson, D. B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.05880 · Cited by 207
Lee, A., Yau, C., Giles, M. B., Doucet, A., & Holmes, C. C. (2010) · Journal of Computational and Graphical Statistics · doi:10.1198/jcgs.2010.10039 · Cited by 205
Communication-efficient algorithms for statistical optimization
(2013) · Journal of Machine Learning Research · doi:10.5555/2567709.2567769 · Cited by 195
Suchard, M. A., Wang, Q., Chan, C., Frelinger, J., Cron, A., & West, M. (2010) · Journal of Computational and Graphical Statistics · doi:10.1198/jcgs.2010.10016 · Cited by 179
Asymptotically Exact, Embarrassingly Parallel MCMC
Neiswanger, W., Wang, C., & Xing, E. P. (2013) · arXiv (Cornell University) · doi:10.1184/r1/6475481 · Cited by 176
Advances in neural information processing systems 12: Proceedings of the 1999 conference
(2001) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(01)90031-8 · Cited by 113
Sampling for Bayesian Computation with Large DatasetsPreprint
Huang, Z., & Gelman, A. (2007) · SSRN Electronic Journal · doi:10.2139/ssrn.1010107 · Cited by 67
Parallelizing MCMC with Random Partition TreesPreprint
Wang, X., Guo, F., Heller, K., & Dunson, D. B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03164 · Cited by 58
The potential and perils of preprocessing: Building new foundations
Blocker, A. W., & Meng, X. (2013) · Bernoulli · doi:10.3150/13-bejsp16 · Cited by 48
Self-citation
Cited by
Marketing Analytics for Data-Rich Environments
Wedel, M., & Kannan, P. K. (2016) · Journal of Marketing · doi:10.1509/jm.15.0413 · Cited by 1,261
Communication-Efficient Distributed Statistical Inference
Jordan, M. I., Lee, J. D., & Yang, Y. (2019) · Journal of the American Statistical Association · doi:10.1080/01621459.2018.1429274 · Cited by 442
Metadata
- Authors
- Scott, S. L., Blocker, A. W., Bonassi, F. V., Chipman, H. A., George, E. I., & McCulloch, R. E.
- Venue
- International Journal of Management Science and Engineering Management
- Publisher
- Informa UK Limited
- Published
- 2016-02-16
- Volume
- 11
- Issue
- 2
- Pages
- 78-88
- ISSN
- 1750-9653 (ISSN-L)
- 1750-9661
- OpenAlex
- W2305001871
- Cited by
- 332
- References
- 28
- FWCI
- 50.31
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.