Inference Under Information Constraints I: Lower Bounds From Chi-Square Contraction
Open accessThe DOI matches its Crossref record; 56 of the 59 references it cites were found in a registry or an index.
Abstract
Multiple players are each given one independent sample, about which they can only provide limited information to a central referee. Each player is allowed to describe its observed sample to the referee using a channel from a family of channels W, which can be instantiated to capture, among others, both the communication and privacy-constrained settings. The referee uses the players' messages to solve an inference problem on the unknown distribution that generated the samples. We derive lower bounds for the sample complexity of learning and testing discrete distributions in this informationconstrained setting. Underlying our bounds is a characterization of the contraction in chi-square distance between the observed distributions of the samples when information constraints are placed. This contraction is captured in a local neighborhood in terms of chi-square and decoupled chi-square fluctuations of a given channel, two quantities we introduce. The former captures the average distance between distributions of channel output for two product distributions on the input, and the latter for a product distribution and a mixture of product distribution on the input. Our bounds are tight for both publicand privatecoin protocols. Interestingly, the sample complexity of testing is order-wise higher when restricted to private-coin protocols.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This can enable differentially private protocols with far smaller error than what is possible in the local model, while relying on weaker trust assumptions than in the central model, e.g., [120, 178, 45, 201, 204, 200, 202, 203, 110].”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [178]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“shuffling models of differential privacy can use broader classes of local randomizers, and can even select these local randomizers adaptively [178]”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [178]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Differential Privacy: A Survey of Results
Dwork, C. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-79228-4_1 · Cited by 3,542 · cited 1×
“[21], [35], [9], [19]): For ρ > 0, W (y | x1) W (y | x2) ≤ e, ∀x1, x2 ∈ [k],∀y ∈ {0, 1}∗.”
A Mathematical Introduction to Compressive Sensing
Foucart, S., & Rauhut, H. (2013) · Applied and Numerical Harmonic Analysis · doi:10.1007/978-0-8176-4948-7 · Cited by 2,548 · cited 1×
“where the first inequality is by the decoupling inequality E e T Mθ ≤ E e T Mθ (used in [27] as well) and the second uses sub-Gaussianity once again.”
MethodConcentration Inequalities: A Nonasymptotic Theory of Independence
Boucheron, S., Lugosi, G., & Massart, P. (2013) · doi:10.1093/acprof:oso/9780199535255.001.0001 · Cited by 1,768 · cited 1×
“…where V 1 , . . . , V k/ 2 are independent and distributed uniformly over {− 1 , +1 , so that we can bound the decoupled chi-square fluctuation using Hoeffding’s Lemma ( cf. [12]) as Thus, Lemma III.6 implies that Ω( √ k/ε 2 ) samples are needed for testing (in particular, for uniformity testing).”
MethodInformation Theory: Coding Theorems for Discrete Memoryless Systems
Csiszár, I., & Körner, J. (2011) · Cambridge University Press eBooks · doi:10.1017/cbo9780511921889 · Cited by 1,308
Distributed detection with multiple sensors I. Fundamentals
Viswanathan, R., & Varshney, P. K. (1997) · Proceedings of the IEEE · doi:10.1109/5.554208 · Cited by 1,066 · cited 1×
“Interestingly, such setups have received a lot of attention in the sensor network literature where a fusion center combines local decisions using simple rules such as majority; see [51] for an early review.”
BackgroundLocal Privacy and Statistical Minimax Rates
Duchi, J. C., Jordan, M. I., & Wainwright, M. J. (2013) · 2013 IEEE 54th Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2013.53 · Cited by 954 · cited 3×
“Distribution learning under LDP constraints has been studied in [4], [21], [37], [52], [56], all providing sample-optimal schemes with different merits.”
BackgroundDistributed detection with multiple sensors I. Advanced topics
Blum, R. S., Kassam, S. A., & Poor, H. V. (1997) · Proceedings of the IEEE · doi:10.1109/5.554209 · Cited by 727
Kasiviswanathan, S. P., Lee, H. K., Nissim, K., Raskhodnikova, S., & Smith, A. (2008) · 2008 49th Annual IEEE Symposium on Foundations of Computer Science · doi:10.1109/focs.2008.27 · Cited by 334
Yu, B. (1997) · Festschrift for Lucien Le Cam · doi:10.1007/978-1-4612-1880-7_29 · Cited by 309 · cited 1×
“We can relate the accuracy of a probability estimate to the probability of error for the multiple hypothesis testing problem with hypotheses given by p z using the standard Fano’s method ( cf. [55]).”
MethodOn the Choice of the Number of Class Intervals in the Application of the Chi Square Test
Mann, H. B., & Wald, A. (1942) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731569 · Cited by 275 · cited 2×
“[39]), with particular interest on the large-support and small-sample case where the usual normal approximations of statistics do not apply (cf.”
BackgroundZhang, Y., Duchi, J. C., Jordan, M. I., & Wainwright, M. J. (2013) · arXiv (Cornell University) · Cited by 252
On the best constants in the Khinchin inequality
Szarek, S. (1976) · Studia Mathematica · doi:10.4064/sm-58-2-197-208 · Cited by 249 · cited 1×
“For Z = V Y described above, we have Therefore, by Khintchine’s inequality ( cf. [46]), where v 1 , . . . , v k/ 2 denote the row vectors of the matrix V .”
BackgroundHypothesis testing with communication constraints
Ahlswede, R., & Csiszar, I. (1986) · IEEE Transactions on Information Theory · doi:10.1109/tit.1986.1057194 · Cited by 242 · cited 1×
“In a parallel thread, distributed statistical inference under communication constraints was initially studied in the information theory community [5], [30], [31], with the objective to characterize the asymptotic error exponents as a function of the communication rate.”
BackgroundStatistical inference under multiterminal data compression
Han, T. S., & Amari, S. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.720540 · Cited by 215 · cited 1×
“In a parallel thread, distributed statistical inference under communication constraints was initially studied in the information theory community [5], [30], [31], with the objective to characterize the asymptotic error exponents as a function of the communication rate.”
BackgroundA Coincidence-Based Test for Uniformity Given Very Sparsely Sampled Discrete Data
Paninski, L. (2008) · IEEE Transactions on Information Theory · doi:10.1109/tit.2008.928987 · Cited by 208 · cited 14×
“Then, the sample complexity of ( k, ε/ 3) -distribution testing must be at least When P is set to be Paninski’s perturbed family given in Eq.”
BackgroundMethodInfluentialDiscrete Distribution Estimation under Local Privacy
Kairouz, P., Bonawitz, K., & Ramage, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07387 · Cited by 197 · cited 2×
“Distribution learning under LDP constraints has been studied in [4], [21], [37], [52], [56], all providing sample-optimal schemes with different merits.”
BackgroundStatistical Algorithms and a Lower Bound for Detecting Planted Cliques
Feldman, V., Grigorescu, E., Reyzin, L., Vempala, S., & Xiao, Y. (2017) · Journal of the ACM · doi:10.1145/3046674 · Cited by 195
An Invitation to Compressive Sensing
Foucart, S., & Rauhut, H. (2013) · Applied and Numerical Harmonic Analysis · doi:10.1007/978-0-8176-4948-7_1 · Cited by 185
Distributed Private Data Analysis: Simultaneously Solving How and What
Beimel, A., Kobbi Nissim, B., & Omri, E. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85174-5_25 · Cited by 163
Hypothesis testing with multiterminal data compression
Han, T. (1987) · IEEE Transactions on Information Theory · doi:10.1109/tit.1987.1057383 · Cited by 161 · cited 1×
“In a parallel thread, distributed statistical inference under communication constraints was initially studied in the information theory community [5], [30], [31], with the objective to characterize the asymptotic error exponents as a function of the communication rate.”
BackgroundOptimal schemes for discrete distribution estimation under locally differential privacy
Ye, M., & Barg, A. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2018.2809790 · Cited by 158
Braverman, M., Garg, A., Ma, T., Nguyen, H. L., & Woodruff, D. P. (2016) · Proceedings of the forty-eighth annual ACM symposium on Theory of Computing · doi:10.1145/2897518.2897582 · Cited by 131
Pollard, D., Torgersen, E. N., & Yang, G. L. (1997) · Springer eBooks · doi:10.1007/978-1-4612-1880-7 · Cited by 106
A Survey on Distribution Testing: Your Data is Big. But is it Blue?Preprint
Canonne, C. L. (2015) · Electronic colloquium on computational complexity · Cited by 97
Self-citationAn Automatic Inequality Prover and Instance Optimal Identity Testing
Valiant, G., & Valiant, P. (2017) · SIAM Journal on Computing · doi:10.1137/151002526 · Cited by 93 · cited 2×
“3The sample complexity for a fixed q has been studied under the “instanceoptimal” setting (see [12], [50]): while the question is not fully resolved, nearly tight upper and lower bounds are known.”
BackgroundHadamard Response: Estimating Distributions Privately, Efficiently, and with Little Communication
Acharya, J., Sun, Z., & Zhang, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04705 · Cited by 86 · cited 2×
“Distribution learning under LDP constraints has been studied in [4], [21], [37], [52], [56], all providing sample-optimal schemes with different merits.”
BackgroundSelf-citationTaming big probability distributions
Rubinfeld, R. (2012) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/2331042.2331052 · Cited by 81 · cited 1×
“We refer the reader to surveys [7], [15], [18], [43] for a comprehensive review of recent results on discrete distribution learning and testing.”
MethodGeometric Lower Bounds for Distributed Parameter Estimation Under Communication Constraints
Han, Y., Özgür, A., & Weissman, T. (2021) · IEEE Transactions on Information Theory · doi:10.1109/tit.2021.3108952 · Cited by 73
Fundamental Limits of Online and Distributed Algorithms for Statistical Learning and Estimation
Shamir, O. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1311.3494 · Cited by 67 · cited 1×
“This is along the line of recent work on distributed mean estimation under communication constraints [14], [28], [45], [55], [58], although some of these works consider more general communication models than what we allow.”
BackgroundDistance-based and continuum Fano inequalities with applications to statistical estimationPreprint
Duchi, J. C., & Wainwright, M. J. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1311.2669 · Cited by 62 · cited 2×
“We avoid this complication by relying instead on a slightly modified form of the classic Fano’s argument from [22]; this form of Fano’s argument was used in [34] as well to obtain a lower bound for the sample complexity of learning under communication constraints.”
BackgroundMethodA General Characterization of the Statistical Query ComplexityPreprint
Feldman, V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.02198 · Cited by 62 · cited 1×
“7 We observe that the unifying treatment based on chi-square distance is reminiscent of the lower bounds for learning under statistical queries (SQ) derived in [24], [25], [47].”
BackgroundUniformly Powerful Goodness of Fit Tests
Barron, A. R. (1989) · The Annals of Statistics · doi:10.1214/aos/1176347005 · Cited by 59 · cited 1×
“[8], [40]).”
Mutual Information Optimally Local Private Discrete Distribution Estimation
Wang, S., Huang, L., Wang, P., Nie, Y., Xu, H., Yang, W., Li, X., & Qiao, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.08025 · Cited by 58 · cited 2×
“Distribution learning under LDP constraints has been studied in [4], [21], [37], [52], [56], all providing sample-optimal schemes with different merits.”
BackgroundHypothesis testing for high-dimensional multinomials: A selective review
Balakrishnan, S., & Wasserman, L. (2018) · The Annals of Applied Statistics · doi:10.1214/18-aoas1155sf · Cited by 52
Memory, Communication, and Statistical QueriesPreprint
Steinhardt, J., Valiant, G. J., & Wager, S. (2015) · Electronic colloquium on computational complexity · Cited by 52
Inference Under Information Constraints II: Communication Constraints and Shared Randomness
Acharya, J., Canonne, C. L., & Tyagi, H. (2020) · IEEE Transactions on Information Theory · doi:10.1109/tit.2020.3028439 · Cited by 48
Self-citationInformation-Theoretic Lower Bounds on Bayes Risk in Decentralized Estimation
Xu, A., & Raginsky, M. (2017) · IEEE Transactions on Information Theory · doi:10.1109/tit.2016.2646342 · Cited by 48
On Communication Cost of Distributed Statistical Estimation and Dimensionality
Garg, A., Ma, T., & Nguyễn, H. L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.1665 · Cited by 46 · cited 1×
“This is along the line of recent work on distributed mean estimation under communication constraints [14], [28], [45], [55], [58], although some of these works consider more general communication models than what we allow.”
BackgroundInteractive hypothesis testing against independence
Yu, X., & Kim, Y.-H. (2013) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2013.6620744 · Cited by 44 · cited 1×
“Recent results in this area have focused on more complicated communication models [53], [54] and, more recently, on the minimum communication requirements for large sample sizes [6], [44].”
BackgroundSeparable Statistics in a Polynomial Scheme. I
Medvedev, Y. I. (1977) · Theory of Probability and Its Applications · doi:10.1137/1122001 · Cited by 36 · cited 1×
“The focus in this line of research has always been on the relation of the performance to the support size ( cf. [37]), with particular interest on the large-support and small-sample case where the usual normal approximations of statistics do not apply ( cf. [38], [8]).”
BackgroundLearning structured distributions
Diakonikolas, I. (2016) · doi:10.1201/b19567-21 · Cited by 36 · cited 1×
“We refer the reader to surveys [7], [15], [18], [43] for a comprehensive review of recent results on discrete distribution learning and testing.”
MethodTest without Trust: Optimal Locally Private Distribution TestingPreprint
Acharya, J., Canonne, C. L., Freitag, C., & Tyagi, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.02174 · Cited by 36
Self-citationThe Uniform Distribution Is Complete with Respect to Testing Identity to a Fixed Distribution
Goldreich, O. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-43662-9_10 · Cited by 35 · cited 1×
“result of [29] provides a formal reduction of arbitrary q to the uniform distribution case.”
BackgroundThe Consistency and Ultimate Distribution of Optimum Statistics
Hotelling, H. (1930) · Transactions of the American Mathematical Society · doi:10.1090/s0002-9947-1930-1501565-9 · Cited by 34 · cited 1×
“The latter goes beyond the discrete setting considered here, but often starts with a quantization to a uniform reference distribution (see [35], [39]).”
BackgroundDistributed Statistical Estimation of High-Dimensional and Nonparametric Distributions
Han, Y., Mukherjee, P., Özgür, A., & Weissman, T. (2018) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2018.8437818 · Cited by 31
Testing against independence with multiple decision centers
Wigger, M., & Timo, R. (2016) · International Conference on Signal Processing and Communications (SPCOM) · doi:10.1109/spcom.2016.7746613 · Cited by 29 · cited 1×
“Recent results in this area have focused on more complicated communication models [53], [54] and, more recently, on the minimum communication requirements for large sample sizes [6], [44].”
BackgroundDistribution Testing Lower Bounds via Reductions from Communication Complexity
Blais, E., Canonne, C. L., & Gur, T. (2019) · ACM Transactions on Computation Theory · doi:10.1145/3305270 · Cited by 25
Self-citationCommunication-Efficient Distributed Learning of Discrete Distributions
Diakonikolas, I., Grigorescu, E., Li, J. Z., Natarajan, A., Onak, K., & Schmidt, L. (2017) · neural information processing systems · Cited by 24
Locally Private Hypothesis TestingPreprint
Sheffet, O. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03441 · Cited by 24 · cited 1×
“We remark that distributed uniformity testing under LDP constraints has been studied recently in [46], however the lower bounds derived there are significantly weaker than what we obtain.”
ResultDistribution testing lower bounds via reductions from communication complexity
Blais, E., Canonne, C. L., & Gur, T. (2017) · Apollo (University of Cambridge) · doi:10.5555/3135595.3135623 · Cited by 22
Self-citationCommunication and Memory Efficient Testing of Discrete DistributionsPreprint
Diakonikolas, I., Gouleakis, T., Kane, D. M., & Karingula, S. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04709 · Cited by 14 · cited 1×
“constraints has been studied in [20],6 and, subsequent to an earlier version of this work, the distribution testing problem has been considered in [19].”
BackgroundDistributed Uniformity Testing
Fischer, O., Meir, U., & Oshman, R. (2018) · ACM Symposium on Principles of Distributed Computing (PODC) · doi:10.1145/3212734.3212772 · Cited by 8
Extra Samples can Reduce the Communication for Independence Testing
Sahasranand, K. R., & Tyagi, H. (2018) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2018.8437584 · Cited by 8 · cited 1×
“Recent results in this area have focused on more complicated communication models [53], [54] and, more recently, on the minimum communication requirements for large sample sizes [6], [44].”
BackgroundSelf-citationTwo Party Distribution Testing: Communication and SecurityPreprint
Andoni, A., Malkin, T., & Nosatzki, N. S. (2018) · arXiv (Cornell University) · doi:10.4230/lipics.icalp.2019.15 · Cited by 6
Statistical Windows in Testing for the Initial Distribution of a Reversible Markov ChainPreprint
Berthet, Q., & Kanade, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.01857 · Cited by 4
Computational Complexity and Property Testing
(2020) · Lecture notes in computer science · doi:10.1007/978-3-030-43662-9
Research works printed but not linked: 6
- ingster, 1986, A minimax test of nonparametric hypotheses on the density of a distribution in $L_{p}$ metrics, Teor Veroyatnost i Primenen, 31, 384
- han, 2018, Geometric lower bounds for distributed parameter estimation under communication constraints, Proc 31st Conf Learn Theory COLT, 75, 3163
- acharya, 2019, Inference under information constraints III: Local privacy constraints, arXiv 2007 10979
- cover, 2006, Elements of Information Theory
- tsitsiklis, 1993, Decentralized detection, Advances in Statistical Signal Processing, 2, 297
- pollard, 2003, Asymptopia
Cited by
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
Communication-Efficient Federated Learning via Optimal Client Sampling
Ribero, M., & Vikalo, H. (2020) · LatinX in AI at International Conference on Machine Learning (LXAI) · doi:10.52591/lxai2020071310 · Cited by 132
Breaking the Communication-Privacy-Accuracy Trilemma
Chen, W., Kairouz, P., & Özgür, A. (2022) · IEEE Transactions on Information Theory · doi:10.1109/tit.2022.3218772 · Cited by 94
“If we further assume X are drawn from some discrete distribution p, then the problem falls into distribution estimation under local differential privacy [9], [13], [34], [36], [42], [43], [44], [45], [46] and limited communication [11], [25], [45], [46], [47], [48], [49], [50].”
BackgroundGeometric Lower Bounds for Distributed Parameter Estimation Under Communication Constraints
Han, Y., Özgür, A., & Weissman, T. (2021) · IEEE Transactions on Information Theory · doi:10.1109/tit.2021.3108952 · Cited by 73
Communication-Efficient Federated Learning With Binary Neural Networks
Yang, Y., Zhang, Z., & Yang, Q. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118415 · Cited by 69
“Therefore, many research efforts have been made to theoretically characterize the fundamental trade-offs between the performance of the trained models and the communication costs of the FL framework[5][6][7][8][9].”
Metadata
- Authors
- Acharya, J., Canonne, C. L., & Tyagi, H.
- Affiliations
- Cornell University (US)
- Robert Bosch (India) (IN)
- Venue
- IEEE Transactions on Information Theory
- Published
- 2020-12
- Volume
- 66
- Issue
- 12
- Pages
- 7835-7855
- ISSN
- 0018-9448 (ISSN-L)
- 1557-9654
- OpenAlex
- W2907436114
- Cited by
- 59
- References
- 59
- Citations by year
- 2025: 4 · 2024: 7 · 2023: 9 · 2022: 10 · 2021: 15 · 2020: 7
- FWCI
- 4.03
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Information Theory (publishedVersion)
- NOT FOUND REPOSITORY (Indian Institute of Science Bangalore) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- NSF-CCF (NSF-CCF-1846300 (CAREER), NSF-CCF-1815893)
- Google Faculty Fellowship
- Motwani Postdoctoral Fellowship at Stanford University
- Research Grant from the Robert Bosch Center for Cyberphysical Systems (RBCCPS), Indian Institute of Science, Bengaluru
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed Sensor Networks and Detection Algorithms
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- information constraints, sample complexity, distribution testing, public-coin protocols, lower bounds
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.