The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Parallel Multi-Block ADMM with o(1 / k) Convergence
Open accessAbstract
This paper introduces a parallel and distributed algorithm for solving the following minimization problem with linear constraints: minimizef1(x1)+⋯+fN(xN)subject toA1x1+⋯+ANxN=c,x1∈X1,…,xN∈XN,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {minimize} ~~&f_1(\mathbf{x}_1) + \cdots + f_N(\mathbf{x}_N)\\ \text {subject to}~~&A_1 \mathbf{x}_1 ~+ \cdots + A_N\mathbf{x}_N =c,\\&\mathbf{x}_1\in {\mathcal {X}}_1,~\ldots , ~\mathbf{x}_N\in {\mathcal {X}}_N, \end{aligned}$$\end{document}where N≥2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N \ge 2$$\end{document}, fi\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_i$$\end{document} are convex functions, Ai\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_i$$\end{document} are matrices, and Xi\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {X}}_i$$\end{document} are feasible sets for variable xi\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{x}_i$$\end{document}. Our algorithm extends the alternating direction method of multipliers (ADMM) and decomposes the original problem into N smaller subproblems and solves them in parallel at each iteration. This paper shows that the classic ADMM can be extended to the N-block Jacobi fashion and preserve convergence in the following two cases: (i) matrices Ai\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_i$$\end{document} are mutually near-orthogonal and have full column-rank, or (ii) proximal terms are added to the N subproblems (but without any assumption on matrices Ai\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_i$$\end{document}). In the latter case, certain proximal terms can let the subproblem be solved in more flexible and efficient ways. We show that ‖xk+1-xk‖M2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Vert {\mathbf {x}}^{k+1} - {\mathbf {x}}^k\Vert _M^2$$\end{document} converges at a rate of o(1 / k) where M is a symmetric positive semi-definte matrix. Since the parameters used in the convergence analysis are conservative, we introduce a strategy for automatically tuning the parameters to substantially accelerate our algorithm in practice. We implemented our algorithm (for the case ii above) on Amazon EC2 and tested it on basis pursuit problems with >300 GB of distributed data. This is the first time that successfully solving a compressive sensing problem of such a large scale is reported.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2011) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000016 · Cited by 17,075
Boyd, S. (2010) · now publishers, Inc. eBooks · doi:10.1561/9781601984616 · Cited by 12,677
Parallel and Distributed Computation: Numerical Methods
Bertsekas, D. P., & Tsitsiklis, J. N. (1989) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 5,293
A dual algorithm for the solution of nonlinear variational problems via finite element approximation
Gabay, D., & Mercier, B. (1976) · Computers & Mathematics with Applications · doi:10.1016/0898-1221(76)90003-1 · Cited by 2,514
Smooth minimization of non-smooth functions
Nesterov, Y. (2005) · Mathematical Programming · doi:10.1007/s10107-004-0552-5 · Cited by 2,455
Splitting Algorithms for the Sum of Two Nonlinear Operators
Lions, P. L., & Mercier, B. (1979) · SIAM Journal on Numerical Analysis · doi:10.1137/0716071 · Cited by 1,972
Numerical Methods for Nonlinear Variational Problems
Glowinski, R. (1984) · Springer eBooks · doi:10.1007/978-3-662-12613-4 · Cited by 1,728
Minimization Methods for Non-Differentiable Functions
Shor, N. Z. (1985) · Springer Series in Computational Mathematics · doi:10.1007/978-3-642-82118-9 · Cited by 1,551
Numerical Methods for Nonlinear Variational Problems
Glowinski, R., & Oden, J. T. (1985) · Journal of Applied Mechanics · doi:10.1115/1.3169136 · Cited by 1,541
Generalized Lagrange Multiplier Method for Solving Problems of Optimum Allocation of Resources
Everett, H. (1963) · Operations Research · doi:10.1287/opre.11.3.399 · Cited by 1,274
Alternating Direction Algorithms for $\ell_1$-Problems in Compressive Sensing
Yang, J., & Zhang, Y. (2011) · SIAM Journal on Scientific Computing · doi:10.1137/090777761 · Cited by 1,201
Glowinski, R., & Marroco, A. (1975) · Revue française d'automatique, informatique, recherche opérationnelle. Analyse numérique · doi:10.1051/m2an/197509r200411 · Cited by 1,105
On the $O(1/n)$ Convergence Rate of the Douglas–Rachford Alternating Direction Method
He, B., & Yuan, X. (2012) · SIAM Journal on Numerical Analysis · doi:10.1137/110836936 · Cited by 881
RASL: Robust Alignment by Sparse and Low-Rank Decomposition for Linearly Correlated Images
Peng, Y., Ganesh, A., Wright, J., Xu, W., & Ma, Y. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.282 · Cited by 817
Fast Alternating Direction Optimization Methods
Goldstein, T., O'Donoghue, B., Setzer, S., & Baraniuk, R. (2014) · SIAM Journal on Imaging Sciences · doi:10.1137/120896219 · Cited by 785
Chen, C., He, B., Ye, Y., & Yuan, X. (2014) · Mathematical Programming · doi:10.1007/s10107-014-0826-5 · Cited by 719
On the linear convergence of the alternating direction method of multipliers
Hong, M., & Luo, Z.-Q. (2017) · Mathematical Programming · doi:10.1007/s10107-016-1034-2 · Cited by 690
On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers
Deng, W., & Yin, W. (2016) · Journal of Scientific Computing · doi:10.1007/s10915-015-0048-x · Cited by 650
Self-citationRecovering Low-Rank and Sparse Components of Matrices from Incomplete and Noisy Observations
Tao, M., & Yuan, X. (2011) · SIAM Journal on Optimization · doi:10.1137/100781894 · Cited by 453
A Unified Primal-Dual Algorithm Framework Based on Bregman Iteration
Zhang, X., Burger, M., & Osher, S. (2011) · Journal of Scientific Computing · doi:10.1007/s10915-010-9408-8 · Cited by 419
A proximal-based decomposition method for convex minimization problems
Chen, G., & Teboulle, M. (1994) · Mathematical Programming · doi:10.1007/bf01582566 · Cited by 392
A Class of Projection and Contraction Methods for Monotone Variational Inequalities
He, B. (1997) · Applied Mathematics & Optimization · doi:10.1007/bf02683320 · Cited by 379
Latent variable graphical model selection via convex optimization
Chandrasekaran, V., Parrilo, P. A., & Willsky, A. S. (2012) · The Annals of Statistics · doi:10.1214/11-aos949 · Cited by 377
D-ADMM: A Communication-Efficient Distributed Algorithm for Separable Optimization
Mota, J. F. C., Xavier, J. M. F., Aguiar, P. M. Q., & Puschel, M. (2013) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2013.2254478 · Cited by 363
Alternating Direction Method with Gaussian Back Substitution for Separable Convex Programming
He, B., Tao, M., & Yuan, X. (2012) · SIAM Journal on Optimization · doi:10.1137/110822347 · Cited by 360
On the O(1=k) convergence of asynchronous distributed alternating Direction Method of Multipliers
Wei, E., & Ozdaglar, A. (2013) · 2013 IEEE Global Conference on Signal and Information Processing · doi:10.1109/globalsip.2013.6736937 · Cited by 342
On non-ergodic convergence rate of Douglas–Rachford alternating direction method of multipliers
He, B., & Yuan, X. (2014) · Numerische Mathematik · doi:10.1007/s00211-014-0673-6 · Cited by 299
A Note on the Alternating Direction Method of Multipliers
Han, D., & Yuan, X. (2012) · Journal of Optimization Theory and Applications · doi:10.1007/s10957-012-0003-z · Cited by 244
Convergence Rate Analysis of Several Splitting Schemes
Davis, D., & Yin, W. (2016) · Scientific Computation · doi:10.1007/978-3-319-41589-5_4 · Cited by 206
Self-citationBlock splitting for distributed optimization
Parikh, N., & Boyd, S. (2013) · Mathematical Programming Computation · doi:10.1007/s12532-013-0061-8 · Cited by 132
On the Linear Convergence of the Alternating Direction Method of Multipliers
Luo, T. (2013) · Cited by 130
On Full Jacobian Decomposition of the Augmented Lagrangian Method for Separable Convex Programming
He, B., Hou, L., & Yuan, X. (2015) · SIAM Journal on Optimization · doi:10.1137/130922793 · Cited by 128
Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions
Davis, D., & Yin, W. (2017) · Mathematics of Operations Research · doi:10.1287/moor.2016.0827 · Cited by 123
Self-citationParallel and distributed sparse optimization
Peng, Z., Yan, M., & Yin, W. (2013) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/acssc.2013.6810364 · Cited by 87
Self-citationOn the Convergence Rate of Multi-Block ADMMPreprint
Lin, T., Ma, S., & Zhang, S. (2014) · arXiv (Cornell University) · Cited by 24
Cited by
Global Convergence of ADMM in Nonconvex Nonsmooth Optimization
Wang, Y., Yin, W., & Zeng, J. (2019) · Journal of Scientific Computing · doi:10.1007/s10915-018-0757-z · Cited by 1,268
Self-citationDecentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges
Martínez Beltrán, E. T., Pérez, M. Q., Sánchez, P. M. S., Bernal, S. L., Bovet, G., Pérez, M. G., Pérez, G. M., & Celdrán, A. H. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3315746 · Cited by 625
“Some examples are Decentralized Gradient Descent (DGD) [104], decentralized Alternating Direction Method of Multipliers (ADMM) [105], EXTRA [106], and ADMM based on Jacobi-Proximal [107].”
Energy-Efficient UAV-Assisted Mobile Edge Computing: Resource Allocation and Trajectory Optimization
Li, M., Cheng, N., Gao, J., Wang, Y., Zhao, L., & Shen, X. (2020) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2020.2968343 · Cited by 508
Hong, D., Hu, J., Yao, J., Chanussot, J., & Zhu, X. X. (2021) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2021.05.011 · Cited by 326
Exogenous Cost Allocation in Peer-to-Peer Electricity Markets
Baroche, T., Pinson, P., Latimier, R. L. G., & Ahmed, H. B. (2019) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2019.2896654 · Cited by 314
Metadata
- Authors
- Deng, W., Lai, M.-J., Peng, Z., & Yin, W.
- Venue
- Journal of Scientific Computing
- Published
- 2016-11-12
- Volume
- 71
- Issue
- 2
- Pages
- 712-736
- ISSN
- 0885-7474 (ISSN-L)
- 1573-7691
- OpenAlex
- W2277973662
- Cited by
- 455
- References
- 35
- FWCI
- 48.53
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.