On a measure of the information provided by an experiment
Open accessAbstract
A measure is introduced of the information provided by an experiment. The measure is derived from the work of Shannon [10] and involves the knowledge prior to performing the experiment, expressed through a prior probability distribution over the parameter space. The measure is used to compare some pairs of experiments without reference to prior distributions; this method of comparison is contrasted with the methods discussed by Blackwell. Finally, the measure is applied to provide a solution to some problems of experimental design, where the object of experimentation is not to reach decisions but rather to gain knowledge about the world.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This intrinsic value maximisation underwrites Bayesian experimental design, active learning, intrinsic motivation, and artificial curiosity.
“This underwrites Bayesian experimental design [134] and active learning in statistics [135], intrinsic motivation and artificial curiosity in machine learning and robotics [136, 137, 138, 139, 140].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [134]
Figure 7 shows that expected free energy contains terms that arise in various formulations of optimal behaviour, ranging from optimal Bayesian design to control as inference.
“Figure 7 shows that the expected free energy contains terms that arise in various formulations of optimal behaviour; ranging from optimal Bayesian design [134] through to control as inference [132, 127].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [134]
On a Bayesian reading, expected information gain is exactly the same quantity that underwrites the principles of optimal Bayesian design.
“On a Bayesian reading, the expected information gain is exactly the same quantity that underwrites the principles of optimal Bayesian design [134, 76, 163].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [134]
High entropy corresponds to high uncertainty.
“High entropy corresponds to high uncertainty23–25—so semantic entropy is one way to estimate semantic uncertainties.”
From Detecting hallucinations in large language models using semantic entropy (2024) · cites this paper as [25]
In order to reduce parameters' uncertainties one should choose an experiment that maximizes the mutual information between the parameter and the output.
“Therefore, in order to reduce parameters' uncertainties one should choose an experiment that maximizes the mutual information between and .”
From Maximizing the information content of experiments in systems biology (2013) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Probability and Information Theory with Applications to Radar
(1953) · Elsevier eBooks · doi:10.1016/c2013-0-05390-x · Cited by 1,287 · see citing works
The Basic Theorems of Information Theory
McMillan, B. (1953) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729028 · Cited by 431 · see citing works
On the Efficient Design of Statistical Investigations
Wald, A. (1943) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731454 · Cited by 257 · see citing works
Flagged first, then by citations
Research works printed but not linked: 11
- [1][1] G. A. BARNARD, "Simplified decision functions," Biometrika, Vol. 41 (1954), pp. 241251. -
- [2][2] DAvıd BLAckwELL, "Comparison of experiments," Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, University of California Press, 1951, pp. 93–102. -
- [3][3] DAviD BLAckwELL, "Equivalent comparisons of experiments," Ann. Math. Stat., Vol. 24 (1953), pp. 265–272. -
- [4][4] DAViD BLAcCKwELL AND M. A. GIRsHICK, Theory of Games and Statistical Decisions, John Wiley and Sons, New York, 1954. -
- [5][5] G. H. HARDY, J. E. LITTLEwooD, AND G. PóLYA, Inequalities, Cambridge University Press, 1952. -
- [6][6] S. KuLLBAck, "An application of information theory to multivariate analysis,"Ann. Math. Stat., Vol. 23 (1952), pp. 88–102. -
- [7][7] S. KuLLBAck, "Certain inequalities in information theory and the Cramér-Rao inequality," Ann. Math. Stat., Vol. 25 (1954), pp. 745–751. -
- [8][8] S. KuLLBACK AND R. A. LEIBLER, "Information and sufficiency," Ann. Math. Stat., Vol. 22 (1951), pp. 79–86. -
- [10][10] C. E. SHANNoN, "A mathematical theory of communication," Bell System Technical Journal, Vol. 27 (1948), pp. 379–423, 623–656. -
- [12][12] P. M. WoodwArd, "Theory of Radar Information" Proceedings of a Symposium on Information Theory, Ministry of Supply (London) (1950), pp. 108–113. -
- [15][15] R. A. FisíER, 'Statistical methods and scientific induction."J. Roy. Stat. Soc. (B), Vol. 17 (1955), pp. 69–78.
Cited by
Sorted by citations
Information-based objective functions for active data selection
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.4.590 · Cited by 1,221 · see citing works
Detecting hallucinations in large language models using semantic entropy
Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024) · Nature · doi:10.1038/s41586-024-07421-0 · Cited by 934 · see citing works
“High entropy corresponds to high uncertainty 23–25 —so semantic entropy is one way to estimate semantic uncertainties.”
Bayesian Active Learning for Classification and Preference Learning
Houlsby, N., Huszár, F., Ghahramani, Z., & Lengyel, M. (2011) · arXiv · doi:10.48550/arxiv.1112.5745 · Cited by 507 · see citing works
“Although the latter approach was proposed several decades ago [Lindley, 1956, Bernardo, 1979], it is not always straightforward to apply the criteria to complicated models such as nonparametric processes with infinite parameter spaces.”
BackgroundMethodSimulation-based optimal bayesian experimental design for nonlinear systems
Huan, X., & Marzouk, Y. M. (2013) · Journal of Computational Physics · doi:10.1016/j.jcp.2012.08.013 · Cited by 438 · see citing works
Maximum entropy sampling and optimal bayesian experimental design
Sebastiani, P., & Wynn, H. P. (2000) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00225 · Cited by 243 · see citing works
Active inference on discrete state-spaces: A synthesis
Da Costa, L., Parr, T., Sajid, N., Veselic, S., Neacsu, V., & Friston, K. (2020) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2020.102447 · Cited by 226 · see citing works
“Finally, from a statistical perspective, maximising intrinsic value (i.e., salience and novelty) corresponds to optimal Bayesian design [156] and machine learning derivatives, such as active learning [157].”
BackgroundMaximizing the information content of experiments in systems biology
Liepe, J., Filippi, S., Komorowski, M., & Stumpf, M. P. H. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002888 · Cited by 170 · see citing works
The free energy principle made simpler but not too simple
Friston, K., Da Costa, L., Sajid, N., Heins, C., Ueltzhöffer, K., Pavliotis, G. A., & Parr, T. (2023) · Physics Reports · doi:10.1016/j.physrep.2023.07.001 · Cited by 163 · see citing works
“7 shows that the expected free energy contains terms that arise in various formulations of optimal behaviour; ranging from optimal Bayesian design [134] through to control as inference [127,132].”
Planning to Explore via Self-Supervised World Models
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D. (2020) · arXiv · doi:10.48550/arxiv.2005.05960 · Cited by 129 · see citing works
Planning to be surprised: optimal bayesian exploration in dynamic environments
Sun, Y., Gomez, F., & Schmidhuber, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22887-2_5 · Cited by 134 · see citing works
“In particular, measuring learning progress using KL divergence dates back to the 50’s [2,4].”
BackgroundPerceptual awareness and active inference
Parr, T., Corcoran, A. W., Friston, K. J., & Hohwy, J. (2019) · Neuroscience of Consciousness · doi:10.1093/nc/niz012 · Cited by 91 · see citing works
Generating meaning: active inference and the scope and limits of passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.002 · Cited by 80 · see citing works
Generative models for active vision
Parr, T., Sajid, N., Da Costa, L., Mirza, M. B., & Friston, K. J. (2021) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2021.651432 · Cited by 46 · see citing works
Reward maximization through discrete active inference
Da Costa, L., Sajid, N., Parr, T., Friston, K., & Smith, R. (2023) · Neural Computation · doi:10.1162/neco_a_01574 · Cited by 35 · see citing works
Accelerating scientific progress through Bayesian adversarial collaboration
Corcoran, A. W., Hohwy, J., & Friston, K. J. (2023) · Neuron · doi:10.1016/j.neuron.2023.08.027 · Cited by 26 · see citing works
Cultivating creativity: predictive brains and the enlightened room problem
Constant, A., Friston, K. J., & Clark, A. (2024) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2022.0415 · Cited by 19 · see citing works
Generating Meaning: Active Inference and the Scope and Limits of Passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2023) · doi:10.31234/osf.io/8xgzv · Cited by 8 · see citing works
Decomposing information into copying versus transformation
Kolchinsky, A., & Corominas-Murtra, B. (2020) · Journal of The Royal Society Interface · doi:10.1098/rsif.2019.0623 · Cited by 5 · see citing works
- Authors
- Lindley, D. V.
- Corresponding author
- Lindley, D. V.
- Venue
- The Annals of Mathematical Statistics
- Publisher
- Institute of Mathematical Statistics
- Published
- 1956-12
- Volume
- 27
- Issue
- 4
- Pages
- 986-1005
- ISSN
- 0003-4851 (ISSN-L)
- OpenAlex
- W2076580309
- Cited by
- 1,509
- References
- 3
- Citations by year
- 2026: 65 · 2025: 66 · 2024: 68 · 2023: 71 · 2022: 62 · 2021: 77 · 2020: 70 · 2019: 74 · 2018: 62 · 2017: 56 · 2016: 33 · 2015: 45 · 2014: 41 · 2013: 43 · 2012: 46
- FWCI
- 4.90
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- The Annals of Mathematical Statistics (publishedVersion)
- Project Euclid (Cornell University) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- information measures, experimental design, prior distributions, experimental comparison, Shannon information, parameter space, knowledge improvement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
