CoChat

Joint Parameter-and-Bandwidth Allocation for Improving the Efficiency of Partitioned Edge Learning

Open access
Wen, Dingzhu, Bennis, Mehdi, Huang, Kaibin · 2020 · IEEE Transactions on Wireless Communications
Cited by 64

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.

Abstract

To leverage data and computation capabilities of mobile devices, machine learning algorithms are deployed at the network edge for training artificial intelligence (AI) models, resulting in the new paradigm of edge learning. In this paper, we consider the framework of partitioned edge learning for iteratively training a large-scale model using many resource-constrained devices (called workers). To this end, in each iteration, the model is dynamically partitioned into parametric blocks, which are downloaded to worker groups for updating using data subsets. Then, the local updates are uploaded to and cascaded by the server for updating a global model. To reduce resource usage by minimizing the total learning-and-communication latency, this work focuses on the novel joint design of parameter (computation load) allocation and bandwidth allocation (for downloading and uploading). Two design approaches are adopted. First, a practical sequential approach, called partially integrated parameter-and-bandwidth allocation (PABA), yields two schemes, namely bandwidth aware parameter allocation and parameter aware bandwidth allocation. The former minimizes the load for the slowest (in computing) of worker groups, each training a same parametric block. The latter allocates the largest bandwidth to the worker being the latency bottleneck. Second, PABA are jointly optimized. Despite it being a nonconvex problem, an efficient and optimal solution algorithm is derived by intelligently nesting a bisection search and solving a convex problem. Experimental results using real data demonstrate that integrating PABA can substantially improve the performance of partitioned edge learning in terms of latency (by e.g., 46%) and accuracy (by e.g., 4% given the latency of 100 seconds).

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/twc.2020.3021177

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 16

  1. 10.1109/CVPR.2016.165 · DOI
  2. chen, 2016, Revisiting distributed synchronous SGD, arXiv 1604 00981
  3. kamp, 2018, Efficient decentralized deep learning by dynamic model averaging, Proc European Conf Machine Learning and Knowledge Discovery in Databases (ECML PKDD), 394
  4. chen, 2018, Lag: Lazily aggregated gradient for communication-efficient distributed learning, Proc Conf Neural Inf Process Syst (NIPS), 5050
  5. 10.1109/TWC.2019.2946245 · DOI · Paper page
  6. 10.1016/B978-1-55860-377-6.50048-7 · DOI · Paper page
  7. li, 2013, Parameter server for distributed machine learning, NIPS Big Learning Workshop, 2
  8. 10.1145/2640087.2644155 · DOI · Paper page
  9. ho, 2013, More effective distributed ml via a stale synchronous parallel parameter server, Proc Conf Neural Inf Process Syst (NIPS), 1223
  10. 10.1145/3343180.3343192 · doi:10.1145/3343180.3343192
  11. 10.1109/MCOM.001.1900103 · DOI · Paper page
  12. carreira-perpinan, 2014, Distributed optimization of deeply nested systems, Proc Int Workshop Artif Intell Statist (AISTATS), 10
  13. 10.1109/JSAC.2019.2934006 · DOI
  14. 10.1109/TCOMM.2019.2944169 · DOI · Paper page
  15. ren, 2019, Accelerating DNN training in wireless federated edge learning system, arXiv 1905 09712
  16. ruder, 2016, An overview of gradient descent optimization algorithms, arXiv 1609 04747

Cited by

See all 64

Metadata

Authors
Wen, D., Bennis, M., & Huang, K.
Venue
IEEE Transactions on Wireless Communications
Published
2020-12
Volume
19
Issue
12
Pages
8272-8286
ISSN
  • 1536-1276 (ISSN-L)
  • 1558-2248
OpenAlex
W3086073975
Cited by
64
References
13
FWCI
4.71
Type
journal article
Language
en
Open access
green
Funders
  • Hong Kong Research Grants Council (17208319, 17209917)
  • Innovation and Technology Fund (GHP/016/18GD, 17209917)
  • Guangdong Basic and Applied Basic Research Foundation (2019B1515130003, 17209917)
  • European Union CHIST-ERA (EU-CHISTERA) Program projects CONNECT and LeadingEdge

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.