Information Theory, Inference, and Learning Algorithms
Abstract
Fun and exciting textbook on the mathematics underpinning the most dynamic areas of modern science and engineering.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
On a Bayesian reading, expected information gain is exactly the same quantity that underwrites the principles of optimal Bayesian design.
“On a Bayesian reading, the expected information gain is exactly the same quantity that underwrites the principles of optimal Bayesian design [134, 76, 163].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [76]
Cited by
Sorted by citations
Fortunato, S. (2010) · Physics Reports · doi:10.1016/j.physrep.2009.11.002 · Cited by 11,293 · see citing works
A Review on Evaluation Metrics for Data Classification Evaluations
M, H., & M.N, S. (2015) · International Journal of Data Mining & Knowledge Management Process · doi:10.5121/ijdkp.2015.5201 · Cited by 2,693 · see citing works
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 2,720 · see citing works
On representing chemical environments
Bartók, A. P., Kondor, R., & Csányi, G. (2013) · Physical Review B · doi:10.1103/physrevb.87.184115 · Cited by 2,572 · see citing works
Probabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995 · see citing works
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779 · see citing works
Bayesian model selection for group studies
Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.03.025 · Cited by 1,423 · see citing works
MRtrix: Diffusion tractography in crossing fiber regions
Tournier, J., Calamante, F., & Connelly, A. (2012) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.22005 · Cited by 1,370 · see citing works
Towards Practical, High-Capacity, Low-Maintenance Information Storage in Synthesized DNA
Goldman, N., Bertone, P., Chen, S., Dessimoz, C., LeProust, E. M., Sipos, B., & Birney, E. (2013) · Nature · doi:10.1038/nature11875 · Cited by 1,220 · see citing works
Comparing dynamic causal models.
Penny, W. D., Stephan, K. E., Mechelli, A., & Friston, K. J. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.03.026 · Cited by 848 · see citing works
Mutation effects predicted from sequence co-variation
Hopf, T. A., Ingraham, J. B., Poelwijk, F. J., Schärfe, C. P. I., Springer, M., Sander, C., & Marks, D. S. (2017) · Nature Biotechnology · doi:10.1038/nbt.3769 · Cited by 775 · see citing works
Cortical rewiring and information storage
Chklovskii, D. B., Mel, B. W., & Svoboda, K. (2004) · Nature · doi:10.1038/nature03012 · Cited by 717 · see citing works
Whittington, J. C. R., Muller, T. H., Mark, S., Chen, G., Barry, C., Burgess, N., & Behrens, T. E. J. (2020) · Cell · doi:10.1016/j.cell.2020.10.024 · Cited by 660 · see citing works
Comparing families of dynamic causal models
Penny, W. D., Stephan, K. E., Daunizeau, J., Rosa, M. J., Friston, K. J., Schofield, T. M., & Leff, A. P. (2010) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000709 · Cited by 688 · see citing works
The communicative function of ambiguity in language
Piantadosi, S. T., Tily, H., & Gibson, E. (2012) · Cognition · doi:10.1016/j.cognition.2011.10.004 · Cited by 628 · see citing works
<scp>G</scp>aussian approximation potentials: A brief tutorial introduction
Bartók, A. P., & Csányi, G. (2015) · International Journal of Quantum Chemistry · doi:10.1002/qua.24927 · Cited by 617 · see citing works
Stegle, O., Parts, L., Durbin, R., & Winn, J. (2010) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000770 · Cited by 497 · see citing works
The rubber hand illusion: Sensitivity and reference frame for body ownership.
Costantini, M., & Haggard, P. (2007) · Consciousness and Cognition · doi:10.1016/j.concog.2007.01.001 · Cited by 489 · see citing works
Mapping anhedonia onto reinforcement learning: a behavioural meta-analysis
Huys, Q. J., Pizzagalli, D. A., Bogdan, R., & Dayan, P. (2013) · Biology of Mood & Anxiety Disorders · doi:10.1186/2045-5380-3-12 · Cited by 480 · see citing works
Huys, Q. J. M., Eshel, N., O'Nions, E., Sheridan, L., Dayan, P., & Roiser, J. P. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002410 · Cited by 439 · see citing works
Simulation-based optimal bayesian experimental design for nonlinear systems
Huan, X., & Marzouk, Y. M. (2013) · Journal of Computational Physics · doi:10.1016/j.jcp.2012.08.013 · Cited by 438 · see citing works
Innateness and culture in the evolution of language
Kirby, S., Dowman, M., & Griffiths, T. L. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0608222104 · Cited by 442 · see citing works
Rational integration of noisy evidence and prior semantic expectations in sentence interpretation
Gibson, E., Bergen, L., & Piantadosi, S. T. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1216438110 · Cited by 407 · see citing works
Robust Network Traffic Classification
Zhang, J., Chen, X., Xiang, Y., Zhou, W., & Wu, J. (2015) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2014.2320577 · Cited by 391 · see citing works
Metacognition: computation, biology and function
Fleming, S. M., Dolan, R. J., & Frith, C. D. (2012) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2012.0021 · Cited by 344 · see citing works
A generative modeling approach for benchmarking and training shallow quantum circuits
Benedetti, M., Garcia-Pintos, D., Perdomo, O., Leyton-Ortega, V., Nam, Y., & Perdomo-Ortiz, A. (2019) · npj Quantum Information · doi:10.1038/s41534-019-0157-8 · Cited by 379 · see citing works
Learning latent structure: carving nature at its joints
Gershman, S. J., & Niv, Y. (2010) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2010.02.008 · Cited by 326 · see citing works
Rational approximations to rational models: alternative algorithms for category learning
Sanborn, A. N., Griffiths, T. L., & Navarro, D. J. (2010) · Psychological Review · doi:10.1037/a0020511 · Cited by 369 · see citing works
Human brain lesion-deficit inference remapped
Mah, Y.-H., Husain, M., Rees, G., & Nachev, P. (2014) · Brain · doi:10.1093/brain/awu164 · Cited by 336 · see citing works
Curvature tension: Evidence for a closed Universe
Handley, W. (2021) · Physical Review D · doi:10.1103/physrevd.103.l041301 · Cited by 377 · see citing works
Dimiduk, D. M., Holm, E. A., & Niezgoda, S. R. (2018) · Integrating Materials and Manufacturing Innovation · doi:10.1007/s40192-018-0117-8 · Cited by 347 · see citing works
The anatomy of choice: active inference and agency
Friston, K., Schwartenbeck, P., FitzGerald, T., Moutoussis, M., Behrens, T., & Dolan, R. J. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00598 · Cited by 340 · see citing works
A ventral visual stream reading center independent of visual experience
Reich, L., Szwed, M., Cohen, L., & Amedi, A. (2011) · Current Biology · doi:10.1016/j.cub.2011.01.040 · Cited by 295 · see citing works
Comparing dynamic causal models using AIC, BIC and free energy
Penny, W. D. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.07.039 · Cited by 298 · see citing works
The anatomy of choice: dopamine and decision-making.
Friston, K., Schwartenbeck, P., FitzGerald, T., Moutoussis, M., Behrens, T., & Dolan, R. J. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0481 · Cited by 289 · see citing works
When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298 · see citing works
Differentiable learning of quantum circuit Born machines
Liu, J.-G., & Wang, L. (2018) · Physical Review A · doi:10.1103/physreva.98.062324 · Cited by 279 · see citing works
Smith, G. D. (2008) · Basic & Clinical Pharmacology & Toxicology · doi:10.1111/j.1742-7843.2007.00191.x · Cited by 257 · see citing works
Machine Learning Based Trust Computational Model for IoT Services
Jayasinghe, U., Lee, G. M., Um, T.-W., & Shi, Q. (2019) · IEEE Transactions on Sustainable Computing · doi:10.1109/tsusc.2018.2839623 · Cited by 256 · see citing works
Perspectives on the integration between first-principles and data-driven modeling
Bradley, W., Kim, J., Kilwein, Z., Blakely, L., Eydenberg, M., Jalvin, J., Laird, C., & Boukouvala, F. (2022) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2022.107898 · Cited by 231 · see citing works
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Bhatt, U., Antorán, J., Zhang, Y., Liao, Q. V., Sattigeri, P., Fogliato, R., Melançon, G., Krishnan, R., Stanley, J., Tickoo, O., et al. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462571 · Cited by 225 · see citing works
Probabilistic models in human sensorimotor control
Wolpert, D. M. (2007) · Human Movement Science · doi:10.1016/j.humov.2007.05.005 · Cited by 212 · see citing works
Data-driven clustering reveals a fundamental subdivision of the human cortex into two global systems
Golland, Y., Golland, P., Bentin, S., & Malach, R. (2008) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2007.10.003 · Cited by 190 · see citing works
Bayesian ranking of biochemical system models
Vyshemirsky, V., & Girolami, M. A. (2008) · Bioinformatics · doi:10.1093/bioinformatics/btm607 · Cited by 181 · see citing works
The free energy principle made simpler but not too simple
Friston, K., Da Costa, L., Sajid, N., Heins, C., Ueltzhöffer, K., Pavliotis, G. A., & Parr, T. (2023) · Physics Reports · doi:10.1016/j.physrep.2023.07.001 · Cited by 163 · see citing works
Consciousness as a state of matter
Tegmark, M. (2015) · Chaos, Solitons & Fractals · doi:10.1016/j.chaos.2015.03.014 · Cited by 183 · see citing works
Lawlor, D. A. (2013) · International Journal of Epidemiology · doi:10.1093/ije/dys209 · Cited by 114 · see citing works
Kanen, J. W., Ersche, K. D., Fineberg, N. A., Robbins, T. W., & Cardinal, R. N. (2019) · Psychopharmacology · doi:10.1007/s00213-019-05325-w · Cited by 101 · see citing works
Lise, S., Archambeau, C., Pontil, M., & Jones, D. T. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-365 · Cited by 107 · see citing works
Optimal and efficient decoding of concatenated quantum block codes
Poulin, D. (2006) · Physical Review A · doi:10.1103/physreva.74.052333 · Cited by 105 · see citing works
Learning and Adaptation in Cognitive Radios Using Neural Networks
Baldo, N., & Zorzi, M. (2008) · 2008 5th IEEE Consumer Communications and Networking Conference · doi:10.1109/ccnc08.2007.229 · Cited by 102 · see citing works
Exploration, Sampling, And Reconstruction of Free Energy Surfaces with Gaussian Process Regression
Mones, L., Bernstein, N., & Csányi, G. (2016) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.6b00553 · Cited by 91 · see citing works
LeDuc, R. D., Fellers, R. T., Early, B. P., Greer, J. B., Thomas, P. M., & Kelleher, N. L. (2014) · Journal of Proteome Research · doi:10.1021/pr401277r · Cited by 85 · see citing works
An Ontology-Based Interpretable Fuzzy Decision Support System for Diabetes Diagnosis
El-Sappagh, S., Alonso, J. M., Ali, F., Ali, A., Jang, J.-H., & Kwak, K.-S. (2018) · IEEE Access · doi:10.1109/access.2018.2852004 · Cited by 79 · see citing works
From simple innate biases to complex visual concepts
Ullman, S., Harari, D., & Dorfman, N. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1207690109 · Cited by 77 · see citing works
Balancing new against old information: the role of puzzlement surprise in learning
Faraji, M., Preuschoff, K., & Gerstner, W. (2018) · Neural Computation · doi:10.1162/neco_a_01025 · Cited by 68 · see citing works
Signal detection theory, uncertainty, and Poisson-like population codes
Ma, W. J. (2010) · Vision Research · doi:10.1016/j.visres.2010.08.035 · Cited by 60 · see citing works
Considerate approaches to constructing summary statistics for ABC model selection
Barnes, C. P., Filippi, S., Stumpf, M. P. H., & Thorne, T. (2012) · Statistics and Computing · doi:10.1007/s11222-012-9335-7 · Cited by 61 · see citing works
P300 amplitude variations, prior probabilities, and likelihoods: a Bayesian ERP study
Kopp, B., Seer, C., Lange, F., Kluytmans, A., Kolossa, A., Fingscheidt, T., & Hoijtink, H. (2016) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-016-0442-3 · Cited by 60 · see citing works
Machine Learning Approaches for Early DRG Classification and Resource Allocation
Gartner, D., Kolisch, R., Neill, D. B., & Padman, R. (2015) · INFORMS Journal on Computing · doi:10.1287/ijoc.2015.0655 · Cited by 58 · see citing works
Jiang, H., Dai, X., Gao, D. W., Zhang, J. J., Zhang, Y., & Muljadi, E. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2552229 · Cited by 56 · see citing works
Hilbert, M., Ahmed, S., Cho, J., Liu, B., & Luu, J. (2018) · Communication Methods and Measures · doi:10.1080/19312458.2018.1479843 · Cited by 49 · see citing works
Compression, The Fermi Paradox and Artificial Super-Intelligence
Bennett, M. T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-93758-4_5 · Cited by 31 · see citing works
Optimal Adaptive Power Control for Over-The-Air Federated Edge Learning Under Fading Channels
Yu, X., Xiao, B., Ni, W., & Wang, X. (2023) · IEEE Transactions on Communications · doi:10.1109/tcomm.2023.3283790 · Cited by 19 · see citing works
Large language models at work in China’s labor market
Chen, Q., Ge, J., Xie, H., Xu, X., & Yang, Y. (2025) · China Economic Review · doi:10.1016/j.chieco.2025.102413 · Cited by 17 · see citing works
The impact of quantitative optimization of hybridization conditions on gene expression analysis
Sykacek, P., Kreil, D. P., Meadows, L. A., Auburn, R. P., Fischer, B., Russell, S., & Micklem, G. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-73 · Cited by 13 · see citing works
Project Silica: Towards Sustainable Cloud Archival Storage in Glass
Anderson, P., Aranas, E., Assaf, Y., Behrendt, R., Black, R., Caballero, M., Cameron, P., Canakci, B., Chatzieleftheriou, A., Clarke, R., et al. (2025) · ACM Transactions on Storage · doi:10.1145/3708996 · Cited by 13 · see citing works
iVUN: interactive Visualization of Uncertain biochemical reaction Networks
Vehlow, C., Hasenauer, J., Kramer, A., Raue, A., Hug, S., Timmer, J., Radde, N., Theis, F. J., & Weiskopf, D. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-s19-s2 · Cited by 11 · see citing works
Applying Bayesian Network to Assess the Levels of Skills Mastering in Adaptive Dynamic OER-Systems
Nekhaev, I., Zhuykov, I., Manukyants, S., & Maslennikov, A. (2020) · Advances in Intelligent Systems and Computing · doi:10.1007/978-3-030-63322-6_94 · Cited by 3 · see citing works
Tsanas, A. (2025) · World Forum on Internet of Things · doi:10.1109/wf-iot64238.2025.11270790
- Authors
- MacKay, D.
- Corresponding author
- Djc MacKay
- Venue
- Kybernetes
- Publisher
- Emerald
- Published
- 2004-08-01
- Volume
- 33
- Issue
- 7
- Pages
- 1217-1218
- ISSN
- 0368-492X (ISSN-L)
- OpenAlex
- W1607198972
- Cited by
- 4,937
- References
- 0
- Citations by year
- 2026: 1 · 2025: 163 · 2024: 204 · 2023: 340 · 2022: 267 · 2021: 297 · 2020: 334 · 2019: 273 · 2018: 313 · 2017: 243 · 2016: 205 · 2015: 234 · 2014: 266 · 2013: 217 · 2012: 237
- Type
- book review
- Language
- en
- Open access
- closed
- Locations
- Kybernetes (publishedVersion)
- International Linear Collider (submittedVersion)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Error Correcting Code Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- information theory, statistical inference, learning algorithms, mathematical foundations of machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
