The DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.
Predictability, Complexity, and Learning
Abstract
We define predictive information I pred(T) as the mutual information between the past and the future of a time series. Three qualitatively different behaviors are found in the limit of large observation times T: I pred(T) can remain finite, grow logarithmically, or grow as a fractional power law. If the time series allows us to learn a model with a finite number of parameters, then I pred(T) grows logarithmically with a coefficient that counts the dimensionality of the model space. In contrast, power-law growth is associated, for example, with the learning of infinite parameter (or non-parametric) models such as continuous functions with smoothness constraints. There are connections between the predictive information and measures of complexity that have been defined both in learning theory and the analysis of physical systems through statistical mechanics and dynamical systems theory. Furthermore, in the same way that entropy provides the unique measure of available information consistent with some simple and plausible conditions, we argue that the divergent part of I pred(T) provides the unique measure for the complexity of dynamics underlying a time series. Finally, we discuss how these ideas may be useful in problems in physics, statistics, and biology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Information Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer Series in Statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 17,781
Scaling Concepts in Polymer Physics
de Gennes, P. G., & Witten, T. A. (1980) · Physics Today · doi:10.1063/1.2914118 · Cited by 9,348
Modeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011
Casella, G., & Berger, R. (2024) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003456285 · Cited by 5,711
Statistical learning by 8-month-old infants
Saffran, J. R., Aslin, R. N., & Newport, E. L. (1996) · Science · doi:10.1126/science.274.5294.1926 · Cited by 5,659
Extrapolation, Interpolation, and Smoothing of Stationary Time Series: With Engineering Applications
Wiener, N. (1949) · The MIT Press eBooks · doi:10.7551/mitpress/2946.001.0001 · Cited by 4,826
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.415 · Cited by 4,388
An Introduction to Kolmogorov Complexity and Its Applications
Li, M., & Vitányi, P. (2019) · Texts in Computer Science · doi:10.1007/978-3-030-11298-1 · Cited by 3,681
Some informational aspects of visual perception.
Attneave, F. (1954) · Psychological Review · doi:10.1037/h0054663 · Cited by 2,890
Spikes: Exploring the Neural Code
Rieke, F. M., Warland, D., van Steveninck, R. R. D. R., & Bialek, W. (1996) · Medical Entomology and Zoology · Cited by 2,844
Self-citationPrediction and Entropy of Printed English
Shannon, C. E. (1951) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1951.tb01366.x · Cited by 2,686
The information bottleneck method
Tishby, N., Pereira, F. C., & Bialek, W. (2000) · arXiv · doi:10.48550/arxiv.physics/0004057 · Cited by 2,626 · cited 1×
“The problem of separating predictive from nonpredictive information is a special case of the problem discussed recently ( Tishby, Pereira, & Bialek, 1999; Bialek & Tishby, in preparation) : given some data x, how do we compress our description of x while preserving as much information as possible about some other vari…”
BackgroundSelf-citationThree approaches to the quantitative definition of information*
Kolmogorov, A. N. (1968) · International Journal of Computer Mathematics · doi:10.1080/00207166808803030 · Cited by 2,031
The analysis of observed chaotic data in physical systems
Abarbanel, H. D. I., Brown, R., Sidorowich, J. J., & Tsimring, L. S. (1993) · Reviews of Modern Physics · doi:10.1103/revmodphys.65.1331 · Cited by 1,848
A formal theory of inductive inference. Part II
Solomonoff, R. J. (1964) · Information and Control · doi:10.1016/s0019-9958(64)90131-7 · Cited by 1,758
Statistical learning of tone sequences by human infants and adults
Saffran, J. R., Johnson, E. K., Aslin, R. N., & Newport, E. L. (1999) · Cognition · doi:10.1016/s0010-0277(98)00075-4 · Cited by 1,429
Logothetis, N. K., & Sheinberg, D. L. (1996) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.19.030196.003045 · Cited by 1,351
A formal theory of inductive inference. Part I
Solomonoff, R. J. (1964) · Information and Control · doi:10.1016/s0019-9958(64)90223-2 · Cited by 1,192
Inferring statistical complexity
Crutchfield, J. P., & Young, K. (1989) · Physical Review Letters · doi:10.1103/physrevlett.63.105 · Cited by 1,002
1 A Statistical Measure of ComplexityPreprint
López‐Ruiz, R., Mancini, H., & Calbet, X. (1995) · Cited by 939
An Information Measure for Classification
Wallace, C. S., & Boulton, D. M. (1968) · The Computer Journal · doi:10.1093/comjnl/11.2.185 · Cited by 933
Stochastic Complexity and Modeling
Rissanen, J. (1986) · The Annals of Statistics · doi:10.1214/aos/1176350051 · Cited by 922
A Statistical Measure of Complexity
López-Ruiz, R., Mancini, H. L., & Calbet, X. (1995) · Physics Letters A · doi:10.1016/0375-9601(95)00867-5 · Cited by 920
Universal coding, information, prediction, and estimation
Rissanen, J. (1984) · IEEE Transactions on Information Theory · doi:10.1109/tit.1984.1056936 · Cited by 898
A Theory of Program Size Formally Identical to Information Theory
Chaitin, G. J. (1975) · Journal of the ACM · doi:10.1145/321892.321894 · Cited by 849
Neural computation of log likelihood in control of saccadic eye movements
Carpenter, R. H. S., & Williams, M. L. L. (1995) · Nature · doi:10.1038/377059a0 · Cited by 823
Fisher information and stochastic complexity
Rissanen, J. J. (1996) · IEEE Transactions on Information Theory · doi:10.1109/18.481776 · Cited by 814
Entropy and information in neural spike trains
Strong, S. P., Koberle, R., de Ruyter van Steveninck, R. R., & Bialek, W. (1998) · Physical Review Letters · doi:10.1103/physrevlett.80.197 · Cited by 791
Self-citationToward a quantitative theory of self-generated complexity
Grassberger, P. (1986) · International Journal of Theoretical Physics · doi:10.1007/bf00668821 · Cited by 694
Could information theory provide an ecological theory of sensory processing?
Atick, J. J. (2011) · Network: Computation in Neural Systems · doi:10.3109/0954898x.2011.638888 · Cited by 679
Adaptive Rescaling Maximizes Information Transmission
Brenner, N., Bialek, W., & de Ruyter van Steveninck, R. (2000) · Neuron · doi:10.1016/s0896-6273(00)81205-2 · Cited by 647
Self-citationComputation theory of cellular automata
Wolfram, S. (1984) · Communications in Mathematical Physics · doi:10.1007/bf01217347 · Cited by 625
Gasarch, W. (1997) · ACM SIGACT News · doi:10.1145/262301.1040343 · Cited by 583
Motion Integration and Postdiction in Visual Awareness
Eagleman, D. M., & Sejnowski, T. J. (2000) · Science · doi:10.1126/science.287.5460.2036 · Cited by 562
Adaptation of retinal processing to image contrast and spatial scale
Smirnakis, S. M., Berry, M. J., Warland, D. K., Bialek, W., & Meister, M. (1997) · Nature · doi:10.1038/386069a0 · Cited by 547
Self-citationStatistical mechanics of learning from examples
Seung, H. S., Sompolinsky, H., & Tishby, N. (1992) · Physical Review A · doi:10.1103/physreva.45.6056 · Cited by 532
Self-citationComplexity as thermodynamic depth
Lloyd, S., & Pagels, H. (1988) · Annals of Physics · doi:10.1016/0003-4916(88)90094-2 · Cited by 517
The structure and precision of retinal spike trains
Berry, M. J., Warland, D. K., & Meister, M. (1997) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.94.10.5411 · Cited by 500
Minimum complexity density estimation
Barron, A. R., & Cover, T. M. (1991) · IEEE Transactions on Information Theory · doi:10.1109/18.86996 · Cited by 497
Computational Mechanics: Pattern and Prediction, Structure and Simplicity
Shalizi, C. R., & Crutchfield, J. P. (2001) · Journal of Statistical Physics · doi:10.1023/a:1010388907793 · Cited by 471
Measures of statistical complexity: Why?
Feldman, D. P., & Crutchfield, J. P. (1998) · Physics Letters A · doi:10.1016/s0375-9601(97)00855-4 · Cited by 436
Information-theoretic asymptotics of Bayes methods
Clarke, B. S., & Barron, A. R. (1990) · IEEE Transactions on Information Theory · doi:10.1109/18.54897 · Cited by 431
Temporal Coding of Visual Information in the Thalamus
Reinagel, P., & Reid, R. C. (2000) · The Journal of Neuroscience · doi:10.1523/jneurosci.20-14-05392.2000 · Cited by 408
Regularities unseen, randomness observed: Levels of entropy convergence
Crutchfield, J. P., & Feldman, D. P. (2003) · Chaos: An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.1530990 · Cited by 368
Mutual Information, Fisher Information, and Population Coding
Brunel, N., & Nadal, J.-P. (1998) · Neural Computation · doi:10.1162/089976698300017115 · Cited by 350
Rissanen, J. (1987) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1987.tb01694.x · Cited by 339
Shiner, J. S., Davison, M., & Landsberg, P. T. (1999) · Physical Review E · doi:10.1103/physreve.59.1459 · Cited by 317
Information measures, effective complexity, and total information
Gell-Mann, M., & Lloyd, S. (1996) · Complexity · doi:10.1002/(sici)1099-0526(199609/10)2:1<44::aid-cplx10>3.0.co;2-x · Cited by 300
Minimum description length induction, Bayesianism, and Kolmogorov complexity
Vitanyi, P. M. B., & Li, M. (2000) · IEEE Transactions on Information Theory · doi:10.1109/18.825807 · Cited by 251
Balasubramanian, V. (1997) · Neural Computation · doi:10.1162/neco.1997.9.2.349 · Cited by 236
Time series prediction: Forecasting the future and understanding the past
Chatfield, C., & Weigend, A. S. (1994) · International Journal of Forecasting · doi:10.1016/0169-2070(94)90058-2 · Cited by 225
A convergent gambling estimate of the entropy of English
Cover, T., & King, R. (1978) · IEEE Transactions on Information Theory · doi:10.1109/tit.1978.1055912 · Cited by 220
Interpolation and extrapolation of stationary random sequences.
Kolmogorov, A. N., Doyle, W., & Selin, I. (1962) · Cited by 195
Entropy and Long-Range Correlations in Literary English
Ebeling, W., & Pöschel, T. (1993) · Europhysics Letters (EPL) · doi:10.1209/0295-5075/26/4/001 · Cited by 183
Physical complexity of symbolic sequences
Adami, C., & Cerf, N. J. (2000) · Physica D Nonlinear Phenomena · doi:10.1016/s0167-2789(99)00179-7 · Cited by 156
Distinguishing between random and nonrandom events.
Lopes, L. L., & Oden, G. C. (1987) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.13.3.392 · Cited by 155
Thermodynamic depth of causal states: Objective complexity via minimal representations
Crutchfield, J. P., & Shalizi, C. R. (1999) · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · doi:10.1103/physreve.59.275 · Cited by 144
Mutual information, metric entropy and cumulative relative entropy risk
Haussler, D. H., & Opper, M. (1997) · The Annals of Statistics · doi:10.1214/aos/1030741081 · Cited by 137
Physical complexity of symbolic sequences
Adami, C., & Cerf, N. J. (2000) · Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) · Cited by 137
Structural information in two-dimensional patterns: Entropy convergence and excess entropy
Feldman, D. P., & Crutchfield, J. P. (2003) · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · doi:10.1103/physreve.67.051104 · Cited by 128
Density estimation by stochastic complexity
Rissanen, J., Speed, T. P., & Yu, B. (1992) · IEEE Transactions on Information Theory · doi:10.1109/18.119689 · Cited by 114
Statistical complexity of simple one-dimensional spin systems
Crutchfield, J. P., & Feldman, D. P. (1997) · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · doi:10.1103/physreve.55.r1239 · Cited by 112
Field Theories for Learning Probability Distributions
Bialek, W., Callan, C. G., & Strong, S. P. (1996) · Physical Review Letters · doi:10.1103/physrevlett.77.4693 · Cited by 109
Self-citationBounds on the Sample Complexity of Bayesian Learning Using Information Theory and the VC Dimension
Haussler, D. H., Kearns, M. J., & Schapire, R. E. (1994) · Machine Learning · doi:10.1007/bf00993163 · Cited by 104
Information and Complexity Measures in Dynamical Systems
Grassberger, P. (1991) · NATO ASI series. Series B : Physics · doi:10.1007/978-1-4899-2305-9_2 · Cited by 99
Rigorous learning curve bounds from statistical mechanics
Haussler, D. H., Seung, H. S., Kearns, M. J., & Tishby, N. (1994) · Conference on Learning Theory · doi:10.1145/180139.181018 · Cited by 98
Self-citationThe Use of Simplicity in Induction
Kemeny, J. G. (1953) · The Philosophical Review · doi:10.2307/2182878 · Cited by 97
NEUROETHOLOGY OF SPATIAL LEARNING: The Birds and the Bees
Capaldi, E. A., Robinson, G. E., & Fahrbach, S. E. (1999) · Annual Review of Psychology · doi:10.1146/annurev.psych.50.1.651 · Cited by 80
On stochastic complexity and nonparametric density estimation
Hall, P. M., & Hannan, E. J. (1988) · Biometrika · doi:10.1093/biomet/75.4.705 · Cited by 75
Intelligence, guesswork, language
Barlow, H. B. (1983) · Nature · doi:10.1038/304207a0 · Cited by 69
Hilberg, W. (1990) · Frequenz · doi:10.1515/freq.1990.44.9-10.243 · Cited by 59
Opper, M. (1994) · Physical Review Letters · doi:10.1103/physrevlett.72.2113 · Cited by 56
Predictive InformationPreprint
Bialek, W., & Tishby, N. (1999) · arXiv (Cornell University) · doi:10.48550/arxiv.cond-mat/9902341 · Cited by 48
Self-citationRigorous Learning Curve Bounds from Statistical Mechanics
Haussler, D. H., Kearns, M. J., Seung, H. S., & Tishby, N. (1996) · Machine Learning · doi:10.1023/a:1026499208981 · Cited by 41
Self-citationShalizi, C. R., & Crutchfield, J. P. (2002) · Advances in Complex Systems · doi:10.1142/s0219525902000481 · Cited by 34
Worst-Case Bounds for the Logarithmic Loss of Predictors
Cesa‐Bianchi, N., & Lugosi, G. (2001) · Machine Learning · doi:10.1023/a:1010848128995 · Cited by 32
Occam factors and model independent Bayesian learning of continuous distributions
Nemenman, I. M., & Bialek, W. (2002) · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · doi:10.1103/physreve.65.026137 · Cited by 30
Self-citationWorst Case Prediction over Sequences under Log Loss
Opper, M., & Haussler, D. H. (1999) · The IMA volumes in mathematics and its applications · doi:10.1007/978-1-4612-1524-0_6 · Cited by 29
Guessing probability distributions from small samples
Pöschel, T., Ebeling, W., & Rosé, H. (1995) · Journal of Statistical Physics · doi:10.1007/bf02179880 · Cited by 26
Analysis of Data from Continuous Probability Distributions
Holy, T. E. (1997) · Physical Review Letters · doi:10.1103/physrevlett.79.3545 · Cited by 26
Field Theoretical Analysis of On-Line Learning of Probability Distributions
Aida, T. (1999) · Physical Review Letters · doi:10.1103/physrevlett.83.3554 · Cited by 16
Reparametrization Invariant Statistical Inference and Gravity
Periwal, V. (1997) · Physical Review Letters · doi:10.1103/physrevlett.78.4671 · Cited by 16
Information theory and learning: a physical approachPreprint
Nemenman, I. M. (2000) · arXiv (Cornell University) · doi:10.48550/arxiv.physics/0009032 · Cited by 15
Self-citation
Cited by
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120
Information-seeking, curiosity, and attention: computational and neural mechanisms
Gottlieb, J., Oudeyer, P.-Y., Lopes, M., & Baranes, A. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.09.001 · Cited by 784
Friston, K. (2013) · Journal of The Royal Society Interface · doi:10.1098/rsif.2013.0475 · Cited by 754
“Furthermore, we have shown elsewhere [11,41] that free energy minimization is consistent with information-theoretic formulations of sensory processing and behaviour [23,42,43].”
Active inference and epistemic value
Friston, K., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T., & Pezzulo, G. (2015) · Cognitive Neuroscience · doi:10.1080/17588928.2015.1020053 · Cited by 731
Machine learning and radiology
Wang, S., & Summers, R. M. (2012) · Medical Image Analysis · doi:10.1016/j.media.2012.02.005 · Cited by 697
Metadata
- Authors
- Bialek, W., Nemenman, I., & Tishby, N.
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2001-11-01
- Volume
- 13
- Issue
- 11
- Pages
- 2409-2463
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2128957129
- Cited by
- 504
- References
- 88
- FWCI
- 4.87
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.