Supervised Learning Under Distributed Features
Open accessThe DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Abstract
This paper studies the problem of learning under both large datasets and large-dimensional feature space scenarios. The feature information is assumed to be spread across agents in a network, where each agent observes some of the features. Through local cooperation, the agents are supposed to interact with each other to solve an inference problem and converge towards the global minimizer of an empirical risk. We study this problem exclusively in the primal domain, and propose new and effective distributed solutions with guaranteed convergence to the minimizer with linear rate under strong convexity. This is achieved by combining a dynamic diffusion construction, a pipeline strategy, and variance-reduced techniques. Simulation results illustrate the conclusions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Consensus algorithms were also developed in [158] to jointly learn a model under the decentralized network while keeping the distributed data features locally.”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [158]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2011) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000016 · Cited by 17,075
Boyd, S. (2010) · now publishers, Inc. eBooks · doi:10.1561/9781601984616 · Cited by 12,677
Computer Architecture: A Quantitative Approach
Hennessy, J. L., & Patterson, D. A. (1989) · UCLA - Biblioteca de Ciencias y Tecnologia Felix Morales Bueno · Cited by 9,375
(2010) · Cited by 8,261
Parallel and Distributed Computation: Numerical Methods
Bertsekas, D. P., & Tsitsiklis, J. N. (1989) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 5,293
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
Distributed Subgradient Methods for Multi-Agent Optimization
Nedic, A., & Ozdaglar, A. (2009) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2008.2009515 · Cited by 3,719 · cited 3×
“under some mild conditions on A [19], [46]–[49].”
BackgroundMethodInfluential
Research works printed but not linked: 2
- 10.1109/tsp.2018.2875883 · DOI
- horn, 1990, Matrix Analysis
Cited by
Edge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“Consensus algorithms were also developed in [158] to jointly learn a model under the decentralized network while keeping the distributed data features locally.”
MethodFedBCD: A Communication-Efficient Collaborative Learning Framework for Distributed Features
Liu, Y., Zhang, X., Kang, Y., Li, L., Chen, T., Hong, M., & Yang, Q. (2022) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3198176 · Cited by 257
“[19] propose a dynamic diffusion approach which requires communications between parties with its related neighbours.”
BackgroundA Sharp Estimate on the Transient Time of Distributed Stochastic Gradient Descent
Pu, S., Olshevsky, A., & Paschalidis, I. C. (2021) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2021.3126253 · Cited by 72
“gent i only, and all the agents communicate and exchange information over a network. Problems in the form of (1.1) find applications in multi-agent target seeking [31, 8], distributed machine learning [13, 24, 10, 2, 45, 1, 4], and wireless networks [9, 20, 2], among other scenarios. In order to solve (1.1), we assume…”
BackgroundInfluentialPu, S., Olshevsky, A., & Paschalidis, I. C. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975212 · Cited by 61
“There is a great need for techniques that train such models quickly and reliably in a distributed fashion over networks where the individual processors or GPUs may be scattered across the globe and communicate over an unreliable network, which may suffer from message losses, delays, and asynchrony (see [1], [2], [29],…”
BackgroundHierarchical Federated Learning for Hybrid Data Partitioning Across Multitype Sensors
Su, L., & Lau, V. K. N. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3051382 · Cited by 37
“Recently, in [12]–[14], vertical data partitioning has been studied to extend FL to embrace systems with distributed features.”
BackgroundMethodInfluentialArtificial Intelligence Methodologies for Data Management
Serey, J., Quezada, L., Alfaro, M. D., Fuertes, G., Vargas, M., Ternero, R., Sabattin, J., Durán, C., & Gutiérrez, S. (2021) · Symmetry · doi:10.3390/sym13112040 · Cited by 36
Learning Privately over Distributed Features: An ADMM Sharing Approach
Hu, Y., Liu, P., Kong, L., & Niu, D. (2019) · arXiv · doi:10.48550/arxiv.1907.07735 · Cited by 30
“A few recent studies [8, 9, 10, 11, 12, 13, 14] have revisited vertically partitioned features under the setting of distributed learning, which is motivated by the increase in feature dimensionality as well as the opportunity of cooperation between multiple parties that may hold different aspects of information about…”
BackgroundMethodInfluentialDistributed artificial intelligence: Taxonomy, review, framework, and reference architecture
Janbi, N. F., Katib, I. A., & Mehmood, R. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2023.200231 · Cited by 44
Distributed Primal-Dual Method for Convex Optimization With Coupled Constraints
Su, Y., Wang, Q., & Sun, C. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3123888 · Cited by 38
Secure Distributed On-Device Learning Networks with Byzantine Adversaries
Dong, Y., Cheng, J., Hossain, M. J., & Leung, V. C. M. (2019) · IEEE Network · doi:10.1109/mnet.2019.1900025 · Cited by 28
“When the parameter server experiences unprecedented outage,1 the DLN becomes a promising on-device learning network since the DLN does not require a central parameter server [2].”
BackgroundImproving the Transient Times for Distributed Stochastic Gradient Methods
Huang, K., & Pu, S. (2022) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3201141 · Cited by 28
“Distributed optimization problems in the form of (1) find applications in distributed machine learning [3], [4], [5], [6], [7], [8], [9], multiagent target seeking [10], and wireless networks [6], [11], [12], among many other scenarios.”
Dual Consensus Proximal Algorithm for Multi-Agent Sharing Problems
Alghunaim, S. A., Lyu, Q., Yan, M., & Sayed, A. H. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3114978 · Cited by 23
Self-citationMusluoglu, C. A., & Bertrand, A. (2023) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2023.3275272 · Cited by 17
“If y is interpreted as a feature vector, this type of separation of the observed data is also known as feature partitioning or distributed features [30], [31], [32], [33], [34], [35].”
BackgroundCommunication-Efficient Vertical Federated Learning via Compressed Error Feedback
Valdeira, P., Xavier, J., Soares, C., & Chi, Y. (2025) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2025.3540655 · Cited by 18
“In particular, [27] performed multiple local updates be-tween communication rounds, [28] updated the local models asynchronously, and [29] proposed a fully decentralized approach.”
The Internet of Things as a Deep Neural Network
Du, R., Magnússon, S., & Fischione, C. (2020) · IEEE Communications Magazine · doi:10.1109/mcom.001.2000015 · Cited by 13
“Distributed learning algorithms that allow decomposing the features among nodes have been considered in the literature, such as the ones based on dynamic diffusion approach [11] and adaptive direction multiplier method [12].”
MethodA Non-Asymptotic Analysis of Network Independence for Distributed Stochastic Gradient DescentPreprint
Olshevsky, A., Paschalidis, I. C., & Pu, S. (2019) · arXiv (Cornell University) · Cited by 13
Cho, M., Lai, L., & Xu, W. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3078495 · Cited by 12
“We remark that the regularized loss minimization problem considered in [9]–[11] is a different problem from the consensus problems considered in [19]–[23] in the aspect of separability.”
BackgroundZhang, J., Ge, S., Chang, T., & Luo, Z.-Q. T. (2022) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3184772 · Cited by 13
“To evaluate the practical performance of the proposed algorithms, we consider the less studied distributed vertical learning problem [30], [31].”
BackgroundMethodDistributed Learning on Heterogeneous Resource-Constrained DevicesPreprint
Rapp, M., Khalili, R., & Henkel, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05403 · Cited by 10
“Some studies consider a different type of heterogeneous learning where not the training data but the features are distributed [16], i.”
BackgroundAdaptive diffusion networks: An overview
Tiglea, D. G., Candido, R., & Silva, M. T. M. (2024) · Signal Processing · doi:10.1016/j.sigpro.2024.109570 · Cited by 8
A Multi-Agent Primal-Dual Strategy for Composite Optimization over Distributed Features
Alghunaim, S. A., Yan, M., & Sayed, A. H. (2020) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco47968.2020.9287370 · Cited by 11
“The works [19], [20] establish linear convergence for strongly convex objectives and a smooth coupling function g.”
BackgroundSelf-citationHovine, C., & Bertrand, A. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3474168 · Cited by 9
“…consensus inner loop that requires the data associated with a single batch of samples to be retransmitted and iterated over multiple times, and often require that the inner iterative loop associated with a single batch of samples converges before moving on to the next batch [10], [14], [15], [16].”
BackgroundCommunication-efficient Vertical Federated Learning via Compressed Error Feedback
Valdeira, P., Xavier, J., Soares, C., & Chi, Y. (2024) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco63174.2024.10715377 · Cited by 6
Camouflage Learning: Feature Value Obscuring Ambient Intelligence for Constrained Devices
Nguyen, L. N., Sigg, S., Lietzén, J., Findling, R. D., & Ruttik, K. (2021) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2021.3092271 · Cited by 7
“Data can be fragmented horizontally (distribution of data instances) or vertically (distribution of feature values) [43], [44], [45] (cf. Fig.”
BackgroundMethodDynamic Average Diffusion With Randomized Coordinate Updates
Ying, B., Yuan, K., & Sayed, A. H. (2019) · IEEE Transactions on Signal and Information Processing over Networks · doi:10.1109/tsipn.2019.2942191 · Cited by 7
“Likewise, in learning problem formulations involving feature vectors and parameter models that are distributed over space, or loss functions that are expressed in the form of sums [9]–[14], we encounter optimization problems of the form”
BackgroundSelf-citationAsymptotic Network Independence in Distributed Optimization for Machine LearningPreprint
Olshevsky, A., Paschalidis, I. C., & Pu, S. (2019) · arXiv (Cornell University) · Cited by 6
Privacy-Preserving Distributed Learning with Nonsmooth Objective Functions
Gauthier, F., Gratton, C., Venkategowda, N. K. D., & Werner, S. (2020) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/ieeeconf51394.2020.9443287 · Cited by 5
“Distributed machine learning algorithms have garnered significant research attention recently because of their capacity to process massive amounts of data over a network of agents [1], [2].”
BackgroundA Byzantine-Resilient Dual Subgradient Method for Vertical Federated Learning
Yuan, K., Wu, Z., & Ling, Q. H. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747270 · Cited by 7
Self-citationMusluoglu, C. A., Moonen, M., & Bertrand, A. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909686 · Cited by 4
“This type of partitioning has also been studied under the name of feature partitioning or distributed features in previous works, such as [7]–[10].”
Decentralized EM to Learn Gaussian Mixtures from Datasets Distributed by FeaturesPreprint
Valdeira, P., Soares, C., & Xavier, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.09965 · Cited by 4
“Note that these and our work consider unsupervised learning, yet VP supervised learning has also been studied (e.g. Ying et al., 2018).”
BackgroundSenthil Kumaran, S., Balakannan, S. P., & Li, J. (2021) · The Journal of Supercomputing · doi:10.1007/s11227-021-04064-0 · Cited by 3
“Smart traffic functions enable efficient traffic management to avoid accidents [8].”
BackgroundSu, L., & Lau, V. K. N. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3177449 · Cited by 4
“…T m φ m ( x mi ) + b , and the smoothed hinge loss can then be derived as 2) Multinomial Logistic Regression (LR) With Cross-Entropy Loss [11], [12]: Consider split LR NNs for N f -class classification tasks, the local output from the m th type sensor is Here, the basis function is the linear…”
MethodFedHD: Communication-efficient federated learning from hybrid data
Gao, H., Ge, S., & Chang, T. (2023) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2023.06.039 · Cited by 4
Distributed Learning over Networks with Non-Smooth Regularizers and Feature Partitioning
Gratton, C., Venkategowda, N. K. D., Arablouei, R., & Werner, S. (2021) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco54536.2021.9616045 · Cited by 2
Distributed Adaptive Spatial Filtering With Inexact Local Solvers
Hovine, C., & Bertrand, A. (2025) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2025.3546484 · Cited by 5
“This usually requires solving a problem in a primal and/or dual internal variable with a dimension equal to the number of observed samples [14], [18], [19], [20].”
Decentralized Semi-supervised Learning over Multitask Graphs
Issa, M., Nassif, R., Rizk, E., & Sayed, A. H. (2022) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/ieeeconf56349.2022.10052032 · Cited by 1
“We are interested in a heterogeneous system setting as in [10], [11] where agents might have access to different types and numbers of attributes or features.”
Self-citationAccelerated Federated Learning Over Wireless Fading Channels With Adaptive Stochastic Momentum
Su, L., & Lau, V. K. N. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3340014 · Cited by 4
“The assumptions of vanishing noise covariance made in [11], [12], and [13] are to simplify the convergence of the algorithm but they are too restrictive and not applicable to practical training scenarios.”
BackgroundVertical asynchronous federated learning: algorithms and theoretic guarantees
Chen, T., Jin, X. P., Sun, Y., & Yin, W. (2024) · Elsevier eBooks · doi:10.1016/b978-0-44-319037-7.00020-x · Cited by 1
Network Classifiers With Output SmoothingPreprint
Rizk, E., Nassif, R., & Sayed, A. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04870 · Cited by 1
“Different from the works in [19, 20] where the network corresponds to one classifier and supervised learning is performed under a distributed features assumption, in the current work we assume that each agent corresponds to one classifier and that the outputs of the classifiers are smooth over the graph.”
Self-citation2019 Index IEEE Transactions on Signal Processing Vol. 67
(2019) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2020.2968163 · Cited by 1
Learning with Vertically-Partitioned Data, Binary Feedback, and Random Parameter Update
Nguyen, L. N., & Sigg, S. (2019) · IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) · doi:10.1109/infcomw.2019.8845203
“This setting can be called vertically-partitioned data [5] or feature-wise data distribution [6].”
BackgroundMethodResultInfluentialOnline Supervised Learning with Distributed Features over Multiagent System
An, X., He, B., Hu, C., & Liu, B. (2020) · Complexity · doi:10.1155/2020/8830359
“In batch learning settings, some algorithms have been proposed for distributed features, such as variance-reduced dynamic diffusion (VRD 2 ) [12], feature distributed machine learning (FDML) [1], and the ADMM (alternating direction method of multipliers) sharing [23].”
BackgroundMethodInfluentialEfficient Distributed Training in Heterogeneous Mobile Networks with Active Sampling
Guo, Y., Yu, X., Chaudhuri, K., & Rosing, T. (2020) · International Conference on Mobility, Sensing and Networking (MSN) · doi:10.1109/msn50589.2020.00041
“One characteristic of learning with heterogeneous mobile devices is that the feature vectors, rather than the examples, are distributed across devices [6], [7].”
BackgroundInfluentialDistributed Edge Learning with Inter-Type and Intra-Type Over-the-Air Collaboration
Su, L., & Lau, V. K. N. (2021) · International Conference on Communications · doi:10.1109/icc42927.2021.9500991
“Similar to recent literatures in distributed edge learning [5], [7], [13], we consider the minimization of a regularized empirical risk function with respect to the model weights w where l i ( · ) is the sample-wise loss function on the i -th data sample w.r.t. the model weights { w , b } , and r (…”
BackgroundMethodInfluential(2021) · International Journal For Innovative Engineering and Management Research · doi:10.48047/ijiemr/v10/i11/13
Time-varying dual accelerated gradient ascent: A fast network optimization algorithm
Monifi, E., & Mahdavi‐Amiri, N. (2022) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2022.03.014
Redes de difusão adaptativas podem se sair melhor com menos dados?
Tiglea, D. G. (2024) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.3.2024.tde-20082024-094623
Distributed Dual Coordinate Ascent in a Ring Network for Distributed Machine Learning Process
Cho, M., Ali, S., Asadinia, M., Lai, L., & Xu, W. (2025) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/ieeeconf67917.2025.11443696
“This contrasts with the consensus problems studied in [46–50], where the regularization term is defined as =1 r ( w k ) , with r ( · ) denoting a regularization function and K representing the number of distributed nodes.”
Reiffers-Masson, A., Menasché, D. S., Shimkin, N., & Altman, E. (2026) · Performance Evaluation · doi:10.1016/j.peva.2026.102587
Fu, J., Li, Y., Zhang, H., Zhu, S., Chen, X. D., Wang, J., Li, H., & Lu, R. (2026) · Engineering Research Express · doi:10.1088/2631-8695/aeaac4
“Dynamic diffusion methods restrict communication to related parties (Ying et al 2018).”
Metadata
- Authors
- Ying, B., Yuan, K., & Sayed, A. H.
- Affiliations
- École Polytechnique Fédérale de Lausanne (CH)
- Venue
- IEEE Transactions on Signal Processing
- Published
- 2019-02-15
- Volume
- 67
- Issue
- 4
- Pages
- 977-992
- ISSN
- 1053-587X (ISSN-L)
- 1941-0476
- OpenAlex
- W2805185133
- Cited by
- 50
- References
- 67
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 4 · 2023: 4 · 2022: 8 · 2021: 13 · 2020: 7 · 2019: 9
- FWCI
- 4.49
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CCF-1011918)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Distributed Control Multi-Agent Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed Sensor Networks and Detection Algorithms
- Keywords
- distributed learning, supervised learning, large-scale datasets, high-dimensional feature space, multi-agent systems, strong convexity, linear convergence, variance reduction, empirical risk minimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.