CoChat
Sign Up
Cited by 209

The DOI matches its Crossref record.

Power-law distributions in binned empirical data

Virkar, Yogesh, Clauset, Aaron·2014·The Annals of Applied Statistics
DOI 10.1214/13-aoas710
Open this paper in CoChat — free account

Abstract

Many man-made and natural phenomena, including the intensity of earthquakes, population of cities and size of international wars, are believed to follow power-law distributions. The accurate identification of power-law patterns has significant consequences for correctly understanding and modeling complex systems. However, statistical evidence for or against the power-law hypothesis is complicated by large fluctuations in the empirical distribution’s tail, and these are worsened when information is lost from binning the data. We adapt the statistically principled framework for testing the power-law hypothesis, developed by Clauset, Shalizi and Newman, to the case of binned data. This approach includes maximum-likelihood fitting, a hypothesis test based on the Kolmogorov–Smirnov goodness-of-fit statistic and likelihood ratio tests for comparing against alternative explanations. We evaluate the effectiveness of these methods on synthetic binned data with known structure, quantify the loss of statistical power due to binning, and apply the methods to twelve real-world binned data sets with heavy-tailed patterns.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1214/13-aoas710

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 48

  1. Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability 57. Chapman & Hall, New York. · DOI · Paper page
  2. Hill, B. M. (1975). A simple general approach to inference about the tail of a distribution. Ann. Statist. 3 1163–1174. · DOI
  3. Breiman, L., Stone, C. J. and Kooperberg, C. (1990). Robust confidence bounds for extreme upper quantiles. J. Stat. Comput. Simul. 37 127–149. · DOI
  4. Clauset, A., Shalizi, C. R. and Newman, M. E. J. (2009). Power-law distributions in empirical data. SIAM Rev. 51 661–703. · DOI · Paper page
  5. Clauset, A. and Woodard, R. (2013). Estimating the historical and future probabilities of large terrorist events. Ann. Appl. Stat. 7 1838–1865. · DOI
  6. Dekkers, A. L. M. and de Haan, L. (1993). Optimal choice of sample fraction in extreme-value estimation. J. Multivariate Anal. 47 173–195. · DOI
  7. Drees, H. and Kaufmann, E. (1998). Selecting the optimal sample fraction in univariate extreme value estimation. Stochastic Process. Appl. 75 149–172. · DOI
  8. Stumpf, M. P. H. and Porter, M. A. (2012). Critical truths about power laws. Science 335 665–666. · DOI
  9. Vuong, Q. H. (1989). Likelihood ratio tests for model selection and nonnested hypotheses. Econometrica 57 307–333. · DOI · Paper page
  10. Danielsson, J., de Haan, L., Peng, L. and de Vries, C. G. (2001). Using a bootstrap method to choose the sample fraction in tail index estimation. J. Multivariate Anal. 76 226–248. · DOI
  11. Clauset, A., Young, M. and Gleditsch, K. S. (2007). On the frequency of severe terrorist events. Journal of Conflict Resolution 51 58–87. · DOI
  12. Goldstein, M. L., Morris, S. A. and Yen, G. G. (2004). Problems with fitting to the power-law distribution. Eur. Phys. J. B 41 255–258. · DOI · Paper page
  13. Aban, I. B. and Meerschaert, M. M. (2004). Generalized least-squares estimators for the thickness of heavy tails. J. Statist. Plann. Inference 119 341–352. · DOI
  14. Asal, V. and Rethemeyer, R. K. (2008). The nature of the beast: Organizational structures and the lethality of terrorist attacks. The Journal of Politics 70 437–449. · DOI
  15. Beirlant, J. and Teugels., J. L. (1989). Asymptotic normality of Hill’s estimator. Extreme Value Theory 51 148–155. · DOI
  16. Cadez, I. V., Smyth, P., McLachlan, G. J. and McLaren, C. E. (2002). Maximum likelihood estimation of mixture of densities for binned and truncated multivariate data. Machine Learning 47 7–34. · DOI
  17. Chapuis, A. and Tetzlaff, T. (2012). The variability of tidewater-glacier calving: Origin of event-size and interval distributions. Available at arXiv:1205.1640. · arXiv
  18. Cramér, H. (1946). A contribution to the theory of statistical estimation. Skand. Aktuarietidskr. 29 85–94. · DOI
  19. Gabaix, X. (2009). Power laws in economics and finance. Annual Review of Economics 1 255–293. · DOI · Paper page
  20. Goh, K.-I., Cusick, M. E., Valle, D., Childs, B., Vidal, M. and Barabási, A.-L. (2007). The human disease network. Proc. Natl. Acad. Sci. USA 104 8685–8690. · DOI · Paper page
  21. Hall, P. (1982). On some simple estimates of an exponent of regular variation. J. R. Stat. Soc. Ser. B Stat. Methodol. 44 37–42. · DOI
  22. Handcock, M. S. and Jones, J. H. (2004). Likelihood-based inference for stochastic models of sexual network evolution. Theoretical Population Biology 65 413–422. · DOI
  23. Horn, S. D. (1977). Goodness-of-fit tests for discrete data: A review and an application to a health development scale. Biometrics 33 237–247. · DOI
  24. Kass, R. E. and Raftery, A. E. (1994). Bayes factors. J. Amer. Statist. Assoc. 90 773–795. · DOI · Paper page
  25. Kratz, M. and Resnick, S. I. (1996). The $\mathrm{QQ}$-estimator and heavy tails. Comm. Statist. Stochastic Models 12 699–724. · DOI
  26. McLachlan, G. J. and Jones, P. N. (1988). Fitting mixture models to grouped and truncated data via the EM algorithm. Biometrics 44 571–578. · DOI
  27. Mitzenmacher, M. (2004). A brief history of generative models for power law and lognormal distributions. Internet Math. 1 226–251. · DOI · Paper page
  28. Mitzenmacher, M. (2006). The future of power law research. Internet Math. 2 525–534. · DOI
  29. Newman, M. E. J. (2005). Power laws, Pareto distributions and Zipf’s law. Contemporary Physics 46 323–351. · DOI
  30. Noether, G. E. (1963). Note on the Kolmogorov statistic in the discrete case. Metrika 7 115–116. · DOI
  31. Persing, J. and Montgomery, M. T. (2003). Hurricane superintensity. J. Atmospheric Sci. 60 2349–2371. · DOI
  32. Rao, C. R. (1947). Minimum variance and the estimation of several parameters. Proc. Cambridge Philos. Soc. 43 280–283. · DOI
  33. Schultze, J. and Steinebach, J. (1996). On least squares estimates of an exponential tail coefficient. Statist. Decisions 14 353–372. · DOI
  34. Shinokazi, K., Yoda, K., Hozumi, K. and Kira, T. (1964). A quantitative analysis of plant form—The pipe model theory II: Further evidence of the theory and its application in forest ecology. Japanese Journal of Ecology 14 133–139.
  35. Stoev, S. A., Michailidis, G. and Taqqu, M. S. (2011). Estimating heavy-tail exponents through max self-similarity. IEEE Trans. Inform. Theory 57 1615–1636. · DOI
  36. Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. J. R. Stat. Soc. Ser. B Stat. Methodol. 36 111–147. · DOI · Paper page
  37. Tate, M. W. and Hye, L. A. (1973). Inaccuracy of the $\chi^{2}$ test of goodness of fit when expected frequencies are small. J. Amer. Statist. Assoc. 68 836–841. · DOI
  38. Virkar, Y. and Clauset, A. (2014). Supplement to “Power-law distributions in binned empirical data.” DOI:10.1214/13-AOAS710SUPP. · DOI
  39. West, G. B., Enquist, B. J. and Brown, J. H. (2009). A general quantitative theory of forest structure and dynamics. Proc. Natl. Acad. Sci. USA 106 7040–7045. · DOI
  40. Yamamoto, K. and Kobayashi, S. (1993). Analysis of crown structure based on the pipe model theory. Journal of the Japanese Forestry Society 75 445–448.
  41. Wasserman, L. (2004). All of Statistics: A Concise Course in Statistical Inference. Springer, New York. · DOI · Paper page
  42. Grünwald, P. D. (2007). The Minimum Length Description Principle. MIT Press, Cambridge, MA. · DOI · Paper page
  43. Press, W. H., Teukolsky, S. A., Vetterling, W. T. and Flannery, B. P. (1992). Numerical Recipes in C: The Art of Scientific Computing, 2nd ed. Cambridge Univ. Press, Cambridge.
  44. Barndorff-Nielsen, O. E. and Cox, D. R. (1995). Inference and Asymptotics. Chapman & Hall, London. · DOI
  45. Reed, W. J. and Hughes, B. D. (2002). From gene families and genera to income and internet file sizes: Why power laws are so common in nature. Phys. Rev. E (3) 66 067103. · DOI
  46. Reiss, R.-D. and Thomas, M. (2007). Statistical Analysis of Extreme Values with Applications to Insurance, Finance, Hydrology and Other Fields, 3rd ed. Birkhäuser, Basel. · DOI
  47. Richardson, L. F. (1960). Statistics of Deadly Quarrels. The Boxwood Press, Pittsburgh.
  48. Sornette, D. (2006). Critical Phenomena in Natural Sciences: Chaos, Fractals, Selforganization and Disorder: Concepts and Tools, 2nd ed. Springer, Berlin.

Cited by

Sorted by citations

Metadata

Authors
Virkar, Y., & Clauset, A.
Venue
The Annals of Applied Statistics
Publisher
Institute of Mathematical Statistics
Published
2014-03-01
Volume
8
Issue
1
Pages
89-119
ISSN
  • 1932-6157
  • 1941-7330
DOI
10.1214/13-aoas710
Cited by
209
References
0

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review