The Well-Calibrated Bayesian
Abstract
Suppose that a forecaster sequentially assigns probabilities to events. He is well calibrated if, for example, of those events to which he assigns a probability 30 percent, the long-run proportion that actually occurs turns out to be 30 percent. We prove a theorem to the effect that a coherent Bayesian expects to be well calibrated, and consider its destructive implications for the theory of coherence.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
An introduction to probability theory and its applications
(1958) · Journal of the Franklin Institute · doi:10.1016/0016-0032(58)90688-4 · Cited by 29,620 · see citing works
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1038/109132a0 · Cited by 7,837 · see citing works
An Introduction to Probability Theory and its Applications.
Kingmán, J. F. C., & Feller, W. (1972) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344620 · Cited by 5,297 · see citing works
Journal of the Royal Statistical Society.
(1921) · The Economic Journal · doi:10.1093/ej/31.121.133 · Cited by 1,503 · see citing works
Sampling and Bayes' Inference in Scientific Modelling and Robustness
Box, G. E. P. (1980) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2982063 · Cited by 905 · see citing works
Calibration of Probabilities: The State of the Art
Lichtenstein, S., Fischhoff, B., & Phillips, L. (1977) · Springer eBooks · doi:10.1007/978-94-010-1276-8_19 · Cited by 736 · see citing works
Two further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556 · see citing works
Studies in Subjective Probability.
Hill, B. M., Kyburg, H. E., & Smokler, H. E. (1965) · Journal of the American Statistical Association · doi:10.2307/2283162 · Cited by 549 · see citing works
Reliability of Subjective Probability Forecasts of Precipitation and Temperature
Murphy, A. H., & Winkler, R. L. (1977) · Applied Statistics · doi:10.2307/2346866 · Cited by 386 · see citing works
Combining Expert Judgments: A Bayesian Approach
Morris, P. A. (1977) · Management Science · doi:10.1287/mnsc.23.7.679 · Cited by 356 · see citing works
On the Reconciliation of Probability Assessments
Lindley, D. V., Tversky, A., & Brown, R. V. (1979) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2345078 · Cited by 340 · see citing works
Comparison of Discrimination Techniques Applied to a Complex Data Set of Head Injured Patients
Titterington, D. M., Murray, G. D., Murray, L., Spiegelhalter, D. J., Skene, A. M., Habbema, J. D. F., & Gelpke, G. J. (1981) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2981918 · Cited by 305 · see citing works
Morris, P. A. (1974) · Management Science · doi:10.1287/mnsc.20.9.1233 · Cited by 297 · see citing works
Coherence and the axioms of confirmation
Shimony, A. (1955) · Journal of Symbolic Logic · doi:10.2307/2268039 · Cited by 231 · see citing works
Fair bets and inductive probabilities
Kemeny, J. G. (1955) · Journal of Symbolic Logic · doi:10.2307/2268222 · Cited by 229 · see citing works
6. A Basic System of Inductive Logic Part II
Carnap, R. (1980) · University of California Press eBooks · doi:10.1525/9780520318328-002 · Cited by 168 · see citing works
Conditional Independence for Statistical Operations
Dawid, A. P. (1980) · The Annals of Statistics · doi:10.1214/aos/1176345011 · Cited by 166 · see citing works
Self-citationOn confirmation and rational betting
Lehman, R. S. (1955) · Journal of Symbolic Logic · doi:10.2307/2268221 · Cited by 110 · see citing works
Verification of Probabilistic Predictions: A Brief Review
Murphy, A. H., & Epstein, E. S. (1967) · Journal of applied meteorology · doi:10.1175/1520-0450(1967)006 · Cited by 69 · see citing works
Some Recent Advances in Statistics.
Morgan, G. W., Oliveira, J. T. D. F., & Epstein, B. (1983) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2987553 · Cited by 58 · see citing works
Bayesian approach to Statistics
(2008) · Encyclopedia of Epidemiology · doi:10.4135/9781412953948.n34 · Cited by 33 · see citing works
Independence and Calibration in Decision Analysis
Harrison, J. M. (1977) · Management Science · doi:10.1287/mnsc.24.3.320 · Cited by 33 · see citing works
Buehler, R. J. (1976) · The Annals of Statistics · doi:10.1214/aos/1176343641 · Cited by 24 · see citing works
Improving predictive distributions
DeGroot, M. H. (1980) · Trabajos de Estadistica y de Investigacion Operativa · doi:10.1007/bf02888361 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Prediction, learning, and games
Cesa-Bianchi, N., & Lugosi, G. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511546921 · Cited by 4,660 · see citing works
“It was Dawid [80] who introduced the problem to the literature of mathematical statistics – see also Dawid’s related work on calibration and “prequential” statistics [81, 82, 84].”
BackgroundThe weighing of evidence and the determinants of confidence
Griffin, D., & Tversky, A. (1992) · Cognitive Psychology · doi:10.1016/0010-0285(92)90013-r · Cited by 1,796 · see citing works
“This objection, however, overlooks the fact that a Bayesian expects to be calibrated (Dawid, 1982), hence the theory of subjective probability permits the comparison of confidence and accuracy.”
Probabilistic Forecasts, Calibration and Sharpness
Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2007) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2007.00587.x · Cited by 1,783 · see citing works
“Various researchers have studied calibration in the context of probability forecasts for sequences of binary events (DeGroot and Fienberg, 1982; Dawid, 1982, 1985a, b; Oakes, 1985; Schervish, 1985, 1989; Dawid and Vovk, 1999; Shafer and Vovk, 2001; Sandroni et al., 2003).”
BackgroundFeng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425 · see citing works
“empirical calibration curves [204] and log predictive probabilities.”
BackgroundForecast verification : a practitioner's guide in atmospheric science
Jolliffe, I., & Stephenson, D. (2011) · Wiley eBooks · doi:10.1002/9781119960003 · Cited by 1,457 · see citing works
Veitch, J., Raymond, V., Farr, B., Farr, W. M., Graff, P. B., Vitale, S., Aylott, B. E., Blackburn, K., Christensen, N. L., Coughlin, M. W., et al. (2015) · Physical review. D. Particles, fields, gravitation, and cosmology/Physical review. D. Particles and fields · doi:10.1103/physrevd.91.042003 · Cited by 966 · see citing works
Forecasting: theory and practice
Petropoulos, F., Apiletti, D., Assimakopoulos, V., Babai, M. Z., Barrow, D. K., Taieb, S. B., Bergmeir, C., Bessa, R., Bijak, J., Boylan, J., et al. (2020) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2021.11.001 · Cited by 630 · see citing works
bModelTest: Bayesian phylogenetic site model averaging and model comparison
Bouckaert, R., & Drummond, A. J. (2017) · BMC Evolutionary Biology · doi:10.1186/s12862-017-0890-6 · Cited by 903 · see citing works
“On average, one would expect the coverage to be 95% if simulations are drawn from the prior [25], so each entry in Table 1 has an expected value of 95, but can deviate due to small sample size.”
BackgroundIndividual differences in rational thought.
Stanovich, K. E., & West, R. F. (1998) · Journal of Experimental Psychology General · doi:10.1037/0096-3445.127.2.161 · Cited by 875 · see citing works
The Power of Ensembles for Active Learning in Image Classification
Beluch, W. H., Genewein, T., Nurnberger, A., & Kohler, J. M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00976 · Cited by 655 · see citing works
“Uncertainty calibration To assess calibration ([7]) quality we determine whether the expected fraction of correct classifications (as predicted by the model confidence, i.”
BackgroundThe Comparison and Evaluation of Forecasters
DeGroot, M. H., & Fienberg, S. E. (1983) · The Statistician · doi:10.2307/2987588 · Cited by 604 · see citing works
Calibration and probability judgements: Conceptual and methodological issues
Keren, G. (1991) · Acta Psychologica · doi:10.1016/0001-6918(91)90036-y · Cited by 575 · see citing works
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646 · see citing works
“Synopsis: Calibration is the process of ensuring that the proportion of positive predictions is equal to the proportion of positive examples [83].”
BackgroundRöver, C., Knapp, G., & Friede, T. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0091-1 · Cited by 527 · see citing works
“Simulations Since a (1−α) confidence interval is supposed to cover the true parameter value with probability (1−α), the calibration of such intervals may be checked by repeatedly generating random data based on known parameter values and then determining the empirical frequency with which true values are actually cove…”
BackgroundBayesian Inference of Sampled Ancestor Trees for Epidemiology and Fossil Calibration
Gavryushkina, A. “., Welch, D., Stadler, T., & Drummond, A. J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003919 · Cited by 365 · see citing works
“e, the median of the estimated medians, the median of the relative errors and so forth) and count the number of times when the true value belongs to the 95% HPD interval [43].”
BackgroundRizve, M. N., Duarte, K., Rawat, Y. S., & Shah, M. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.06329 · Cited by 476 · see citing works
“Since, calibration measures the discrepancy between the confidence level of a network’s individual predictions and its overall accuracy (Dawid, 1982; Degroot & Fienberg, 1983); for poorly calibrated networks, an incorrect prediction might have high confidence.”
BackgroundCalibrated Learning and Correlated Equilibrium
Foster, D. P., & Vohra, R. (1997) · Games and Economic Behavior · doi:10.1006/game.1997.0595 · Cited by 366 · see citing works
The philosophy of Bayes factors and the quantification of statistical evidence
Morey, R. D., Romeijn, J., & Rouder, J. N. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.11.001 · Cited by 307 · see citing works
“It may be thought that the context-sensitivity of evidence is more pronounced in Bayesian statistics, because a Bayesian inference is closed-minded about which hypotheses can be true: after the prior has been chosen, hypotheses with zero probability cannot enter the theory (cf. Dawid, 1982).”
BackgroundInherent Tradeoffs in the Fair Determination of Risk Scores
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603197 · Cited by 347 · see citing works
Bayesian Random-Effects Meta-Analysis Using the <b>bayesmeta</b> <i>R</i> Package
Röver, C. (2020) · Journal of Statistical Software · doi:10.18637/jss.v093.i06 · Cited by 310 · see citing works
Howson, C. (1995) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/46.1.1 · Cited by 289 · see citing works
Consistency and cautious fictitious play
Fudenberg, D., & Levine, D. K. (1995) · Journal of Economic Dynamics and Control · doi:10.1016/0165-1889(94)00819-4 · Cited by 292 · see citing works
An Empirical Bayes Approach to Mantel‐Haenszel DIF Analysis
Zwick, R., Thayer, D. T., & Lewis, C. (1999) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1999.tb00543.x · Cited by 285 · see citing works
Low‐Choy, S., O’Leary, R., & Mengersen, K. L. (2009) · Ecology · doi:10.1890/07-1886.1 · Cited by 283 · see citing works
Negro, M. D., & Schorfheide, F. (2012) · Handbook of economic forecasting · doi:10.1016/b978-0-444-53683-9.00002-5 · Cited by 278 · see citing works
The Logic of Scientific Discovery
Howson, C. (2000) · doi:10.1093/0198250371.003.0009 · Cited by 277 · see citing works
Beyond Subjective and Objective in Statistics
Gelman, A., & Hennig, C. M. (2017) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12276 · Cited by 257 · see citing works
“There is also some work on (partial) intersubjective agreement on prior specifications, e.g. Dawid (1982a), providing a rationale for consensus (virtue V2(c)).”
BackgroundMethodLittle, R. J. A. (2006) · The American Statistician · doi:10.1198/000313006x117837 · Cited by 240 · see citing works
“Many statisticians have advanced the calibrated Bayesian idea; some examples are Peers (1965), Welch (1965), and Dawid (1982). I myself have assessed the frequentist properties of Bayesian procedures in methodological work (e.g., Little 1988). But two seminal papers by leading proponents of this school, Box (1980) and…”
BackgroundInfluentialHowson, C. (2000) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · doi:10.1093/0198250371.001.0001 · Cited by 235 · see citing works
Scoring rules and the evaluation of probabilities
Winkler, R., Muñoz, J., Cervera, J. L., Bernardo, J., Blattenberger, G., Kadane, J., Lindley, D., Murphy, A. H., Oliver, R. M., & Insua, D. R. (1996) · Test · doi:10.1007/bf02562681 · Cited by 238 · see citing works
Combining probability forecasts
Ranjan, R., & Gneiting, T. (2010) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2009.00726.x · Cited by 279 · see citing works
The ‘Heuristics and Biases’ Bias in Expert Elicitation
Kynn, M. (2007) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2007.00499.x · Cited by 223 · see citing works
“For theoretical discussions see Dawid (1982) and Kadane and Lichtenstein in Kadane and Wolfson (1998).”
BackgroundModels of the Spiral-Down Effect in Revenue Management
Cooper, W. L., Homem‐de‐Mello, T., & Kleywegt, A. J. (2006) · Operations Research · doi:10.1287/opre.1060.0304 · Cited by 185 · see citing works
“Blackwell and Dubins (1962) introduced the concept of merging as a way to formulate that property mathematically, and Dawid (1982) proposed the idea of calibration, which means that the observed empirical distributions converge to the forecasted ones.”
BackgroundDoes label smoothing mitigate label noise?Preprint
Łukasik, M., Bhojanapalli, S., Menon, A. K., & Kumar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02819 · Cited by 191 · see citing works
“Model Calibration: To further tease out the impact of label smearing on model confidences, we ask: how do these techniques affect the calibration of the output probabilities? This measures how meaningful the model probabilities are in a frequentist sense (Dawid, 1982).”
BackgroundMethodRöver, C., Bender, R., Dias, S., Schmid, C. H., Schmidli, H., Sturtz, S., Weber, S., & Friede, T. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1475 · Cited by 192 · see citing works
Forecasting Performance of an Open Economy DSGE Model
Malin Adolfson, A., Lindé, J., & Villani, M. (2007) · Econometric Reviews · doi:10.1080/07474930701220543 · Cited by 168 · see citing works
“Interval Forecasts A forecast interval10 is said to be well calibrated if the long run relative frequency of realized observations included in the forecast interval equals the pre-specified coverage probability of the interval (Dawid, 1982).”
BackgroundMethodOptimal Decisions from Probabilistic Models: The Intersection-over-Union Case
Nowozin, S. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.77 · Cited by 163 · see citing works
“How can we improve the calibration of our posterior probabilities? If our probabilistic model class is sufficiently accurate then known results guarantee that we will eventually be well-calibrated [9].”
BackgroundNested sampling for physical scientists
Ashton, G., Bernstein, N., Büchner, J., Chen, X., Cśanyi, G., Fowlie, A., Feroz, F., Griffiths, M., Handley, W. J., Habeck, M., et al. (2022) · Nature Reviews Methods Primers · doi:10.1038/s43586-022-00121-x · Cited by 159 · see citing works
Improving Predictions using Ensemble Bayesian Model Averaging
Montgomery, J., Hollenbach, F. M., & Ward, M. (2012) · Political Analysis · doi:10.1093/pan/mps002 · Cited by 143 · see citing works
Frequentist and Subjectivist Perspectives on the Problems of Model Building in Economics
Poirier, D. J. (1988) · The Journal of Economic Perspectives · doi:10.1257/jep.2.1.121 · Cited by 158 · see citing works
Conditional Universal Consistency
Fudenberg, D., & Levine, D. K. (1999) · Games and Economic Behavior · doi:10.1006/game.1998.0705 · Cited by 154 · see citing works
A Theory of Higher Order Probabilities
Gaifman, H. (1988) · Springer eBooks · doi:10.1007/978-94-009-2863-3_11 · Cited by 151 · see citing works
Recalibrating software reliability models
Brocklehurst, S., Chan, P. Y., Bev Littlewood, A., & Snell, J. (1990) · IEEE Transactions on Software Engineering · doi:10.1109/32.54297 · Cited by 142 · see citing works
Semi-Supervised Learning under Class Distribution Mismatch
Chen, Y., Zhu, X., Li, W., & Gong, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5763 · Cited by 133 · see citing works
“Since qt reflects the agreement among the historic networks in a frequentist perspective (Dawid 1982), its maximal class probability indicates the best consensus a sample is assigned to a specific class.”
BackgroundFairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646 · see citing works
“Synopsis: Calibration is the process of ensuring that the proportion of positive predictions is equal to the proportion of positive examples [83].”
BackgroundSayer, A. M., paul, G. Y. M. J., Kolmonen, P., Lipponen, A., Luffarelli, M., Mielonen, T., Patadia, F., Popp, T., Povey, A. C., Stebel, K., et al. (2020) · Atmospheric measurement techniques · doi:10.5194/amt-13-373-2020 · Cited by 129 · see citing works
Reliable support: Measuring calibration of likelihood ratios
Ramos, D., & González-Rodríguez, J. (2013) · Forensic Science International · doi:10.1016/j.forsciint.2013.04.014 · Cited by 126 · see citing works
“Mainly because of the property described above, calibration has been dubbed reliability in the context of Bayesian probabilistic assessment [14,11].”
BackgroundCausality and Causal Modelling in the Social Sciences
Federica Russo, B. (2008) · Springer eBooks · doi:10.1007/978-1-4020-8817-9 · Cited by 111 · see citing works
Deconstructing Cross-Entropy for Probabilistic Binary Classifiers
Ramos, D., Franco-Pedroso, J., Lozano-Diez, A., & González-Rodríguez, J. (2018) · Entropy · doi:10.3390/e20030208 · Cited by 111 · see citing works
“However, it is not enough: probabilities must also present a good calibration [18,20].”
BackgroundMethodCalibration, Coherence, and Scoring Rules
Seidenfeld, T. (1985) · Philosophia Scientiæ · doi:10.1086/289244 · Cited by 74 · see citing works
“Dawid (1982), variations on the strong law of large numbers (for martingales) apply.”
BackgroundMethodInfluentialFudenberg, D., & Levine, D. K. (2009) · Annual Review of Economics · doi:10.1146/annurev.economics.050708.142930 · Cited by 103 · see citing works
Self-Calibrating Priors Do Not Exist
Oakes, D. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10478117 · Cited by 99 · see citing works
Calibration of Transformer-Based Models for Identifying Stress and Depression in Social Media
Ilias, L., Mouzakitis, S., & Askounis, D. (2023) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3283009 · Cited by 97 · see citing works
“In addition, the reliability of an ML model’s confidence in its predictions, denoted as calibration [15], [16], is critical for high-risk applications, such as deciding whether to trust a medical diagnosis prediction [17], [18], [19].”
BackgroundMosquera, C., Ferrer, L., Milone, D. H., Luna, D., & Ferrante, E. (2024) · European Radiology · doi:10.1007/s00330-024-10834-0 · Cited by 92 · see citing works
Röver, C., Bender, R., Dias, S., Schmid, C. H., Schmidli, H., Sturtz, S., Weber, S., & Friede, T. (2021) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 89 · see citing works
Coherent combination of experts' opinions
Dawid, A. P., DeGroot, M. H., Mortera, J., Cooke, R. M., French, S., Genest, C., Schervish, M. J., Lindley, D. V., McConway, K., & Winkler, R. L. (1995) · Test · doi:10.1007/bf02562628 · Cited by 87 · see citing works
Self-citationCalibration with Many Checking Rules
Sandroni, A., Rann Smorodinsky, B., & Vohra, R. (2003) · Mathematics of Operations Research · doi:10.1287/moor.28.1.141.14264 · Cited by 87 · see citing works
“The idea that calibration is a desirable property of probabilistic forecasts is due to Dawid (1982) .”
BackgroundAnimal Welfare and the Intensification of Animal Production
Fraser, D. J. (2008) · The international library of environmental, agricultural and food ethics/The International library of environmental, agricultural and food ethics · doi:10.1007/978-1-4020-8722-6_12 · Cited by 86 · see citing works
Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning
Zhang, J., Kailkhura, B., & Han, Y.-J. (2020) · Unpublished · doi:10.13140/rg.2.2.11390.13125 · Cited by 86 · see citing works
On Testing the Validity of Sequential Probability Forecasts
Seillier‐Moiseiwitsch, F., & Dawid, A. P. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10594328 · Cited by 84 · see citing works
Self-citationAdaptive temperature scaling for Robust calibration of deep neural networks
Balanya, S. A., Maroñas, J., & Ramos, D. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09505-4 · Cited by 83 · see citing works
“The extent to which the confidence outputs of a classifier can be interpreted as class probabilities is what is known as the calibration of a classifier [4, 5].”
BackgroundThe reproducible properties of correct forecasts
Sandroni, A. (2003) · International Journal of Game Theory · doi:10.1007/s001820300153 · Cited by 82 · see citing works
“When h ¼ 1 this is finite version of the calibration test of Dawid (1982) and Foster and Vohra (1998).”
BackgroundMethodProbabilistic preference learning with the Mallows rank modelPreprint
Vitelli, V., Sørensen, Ø., Crispino, M., Frigessi, A., & Arjas, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.7945 · Cited by 81 · see citing works
Calibrated Language Models Must Hallucinate
Kalai, A. T., & Vempala, S. S. (2024) · Proceedings of the 56th Annual ACM Symposium on Theory of Computing · doi:10.1145/3618260.3649777 · Cited by 91 · see citing works
“" The Good-Turing estimator was shown to be within | 𝑝 ( 𝑈 ) − (cid:100) 𝑀𝐹 | = ˜ 𝑂 ( √︁ 1 / )”
Limit theorems for a class of identically distributed random variables
Berti, P., Pratelli, L., & Rigo, P. (2004) · The Annals of Probability · doi:10.1214/009117904000000676 · Cited by 79 · see citing works
A benchmark on uncertainty quantification for deep learning prognostics
Basora, L., Viens, A., Chao, M. A., & Olivé, X. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110513 · Cited by 84 · see citing works
Ehud Lehrer, A. (2001) · Econometrica · doi:10.1111/1468-0262.00244 · Cited by 77 · see citing works
“The forecaster wins the game if µ passes the calibration test induced by D at ω∞, and looses otherwise.”
BackgroundMethodInfluentialBayesian reconstruction of transmission within outbreaks using genomic variants
Maio, N. D., Worby, C. J., Wilson, D. J., & Stoesser, N. E. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006117 · Cited by 74 · see citing works
On the Quantification of Model Uncertainty: A Bayesian Perspective
Kaplan, D. (2021) · Psychometrika · doi:10.1007/s11336-021-09754-5 · Cited by 51 · see citing works
“…not mutually exclusive, but we argue that prediction should take precedent to explanation simply due to the fact that it is hard to warrant explanations of psychological phenomena derived from a psychometric model if the model has difficulty predicting what has actually occurred (see Dawid 1982).”
BackgroundInfluentialCalibrated Bayes, for Statistics in General, and Missing Data in Particular
Little, R. J. A. (2011) · Statistical Science · doi:10.1214/10-sts318 · Cited by 75 · see citing works
“Many statisticians have advanced CB ideas (e.g., Peers, 1965; Welch, 1965; Dawid, 1982), but I was particularly influenced by seminal papers by Box (1980) and Rubin (1984).”
BackgroundEstimating Epidemic Incidence and Prevalence from Genomic Data
Vaughan, T. G., Leventhal, G. E., Rasmussen, D. A., Drummond, A. J., Welch, D., & Stadler, T. (2019) · Molecular Biology and Evolution · doi:10.1093/molbev/msz106 · Cited by 72 · see citing works
“Although agreement between simulated and subsequently inferred trajectories is encouraging, we use a well-calibrated approach (Dawid 1982) for a more robust quantitative validation of the inference algorithm.”
Separating the Brier Score into Calibration and Refinement Components: A Graphical Exposition
Blattenberger, G., & Lad, F. (1985) · The American Statistician · doi:10.1080/00031305.1985.10479382 · Cited by 92 · see citing works
“These general results (Pratt 1962; Dawid 1982) motivate the common definition that the probability assessments of a forecaster are well calibrated if the limiting frequency of occurrence of events assessed with probability pi equals pi.”
BackgroundAre Non-animal Systemic Safety Assessments Protective? A Toolbox and Workflow
Middleton, A., Reynolds, J., Cable, S., Baltazar, M. T., Li, H., Bevan, S. J., Carmichael, P. L., Dent, M. P., Hatherell, S., Houghton, J., et al. (2022) · Toxicological Sciences · doi:10.1093/toxsci/kfac068 · Cited by 72 · see citing works
“The distributions obtained from the model can be considered fit for purpose only if they can be demonstrated to be wellcalibrated (Dawid, 1982).”
BackgroundProbability, Causality and the Empirical World: A Bayes–de Finetti–Popper– Borel Synthesis
Dawid, A. P. (2004) · Statistical Science · doi:10.1214/088342304000000125 · Cited by 72 · see citing works
Self-citationEvaluating DSGE model forecasts of comovements
Herbst, E. P., & Schorfheide, F. (2012) · Journal of Econometrics · doi:10.1016/j.jeconom.2012.06.008 · Cited by 71 · see citing works
Random Choice and Private Information
Lu, J. (2016) · Econometrica · doi:10.3982/ecta12821 · Cited by 71 · see citing works
Calibrated structured prediction
Kuleshov, V., & Liang, P. (2015) · neural information processing systems · Cited by 72 · see citing works
Fukuda, S., Varshney, A., Fowler, B. J., Wang, S.-B., Narendran, S., Ambati, K., Yasuma, T., Magagnoli, J., Leung, H., Hirahara, S., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2022751118 · Cited by 65 · see citing works
“A Bayesian meta-analysis employing a random-effects normal– normal hierarchical model (42, 43) was performed.”
MethodAmbati, J., Magagnoli, J., Leung, H., Wang, S.-B., Andrews, C. A., Fu, D., Pandey, A., Sahu, S., Narendran, S., Hirahara, S., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-18528-z · Cited by 66 · see citing works
Blackwell Approachability and No-Regret Learning are Equivalent
Abernethy, J., Bartlett, P. L., & Hazan, E. (2011) · QUT ePrints (Queensland University of Technology) · Cited by 66 · see citing works
Integrated Objective Bayesian Estimation and Hypothesis Testing
Bernardo, J. M. (2011) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199694587.003.0001 · Cited by 65 · see citing works
“Note that these concerns are motivated by the sort of validation involved in the prequential approach, see Dawid (1982).”
Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603203 · Cited by 67 · see citing works
An informed reference prior for between‐study heterogeneity in meta‐analyses of binary outcomes
Pullenayegum, E. (2011) · Statistics in Medicine · doi:10.1002/sim.4326 · Cited by 64 · see citing works
“Senn [6] commented that one of the problems with a frequentist analysis of a small number of trials is that the estimate of between-trial variance is unlikely to be credible, and pointed out that this problem is not solved by using a vague prior distribution that incorporates implausible values for the between-study v…”
BackgroundIntroduction to large language models (LLMs) for dementia care and research
Treder, M. S., Lee, S., & Tsvetanov, K. A. (2024) · Frontiers in Dementia · doi:10.3389/frdem.2024.1385303 · Cited by 63 · see citing works
“As an illustrative example, if a calibrated weather forecasting model predicts 30% change for rain on a given day, the prediction should turn out to be correct in 30% of the cases (Dawid, 1982).”
Non-Bayesian Testing of a Stochastic Prediction
Dekel, E., & Feinberg, Y. (2006) · The Review of Economic Studies · doi:10.1111/j.1467-937x.2006.00401.x · Cited by 63 · see citing works
“The existing literature focused on calibration tests (Dawid (1982, 1985)).”
BackgroundMethodSet-valued approachability and online learning with partial monitoring
Mannor, S., Perchet, V., & Stoltz, G. (2014) · Journal of Machine Learning Research · Cited by 62 · see citing works
Diagnostics of prior-data agreement in applied Bayesian analysis
Bousquet, N. (2008) · Journal of Applied Statistics · doi:10.1080/02664760802192981 · Cited by 61 · see citing works
“Main references are De Finetti [26], Dawid [25], Hill [39], Lucas [42] and especially O’Hagan [43–45,48], Andrade and O’Hagan [3] and Angers [5].”
BackgroundFudenberg, D., & Levine, D. K. (1999) · Games and Economic Behavior · doi:10.1006/game.1999.0726 · Cited by 62 · see citing works
Modeling Probability Forecasts via Information Diversity
Satopää, V., Pemantle, R., & Ungar, L. (2015) · Journal of the American Statistical Association · doi:10.1080/01621459.2015.1100621 · Cited by 60 · see citing works
“introduced by DeGroot (1988) and later discussed by Dawid, DeGroot, and Mortera (1995).”
MethodDocument Understanding Dataset and Evaluation (DUDE)
Landeghem, J. V., Powalski, R., Tito, R., Jurkiewicz, D., Blaschko, M. B., Borchmann, Ł., Coustaty, M., Moens, S., Pietruszka, M., Ackaert, B., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01789 · Cited by 70 · see citing works
“Calibration requires that the probability a model assigns to its predictions equals their true likelihood of being correct [19, 20, 101].”
BackgroundToward a Reconciliation of the Bayesian and Frequentist Approaches to Point Estimation
Samaniego, F. J., & Reneau, D. M. (1994) · Journal of the American Statistical Association · doi:10.1080/01621459.1994.10476828 · Cited by 59 · see citing works
Probabilistic Forecasts, Calibration and Sharpness
Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2005) · doi:10.21236/ada454827 · Cited by 58 · see citing works
ConMatch: Semi-supervised Learning with Confidence-Guided Consistency Regularization
Kim, J., Min, Y., Kim, D.-H., Lee, G., Seo, J., Ryoo, K., & Kim, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20056-4_39 · Cited by 57 · see citing works
Busk, J., Jørgensen, P. B., Bhowmik, A., Schmidt, M. N., Winther, O., & Vegge, T. (2021) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ac3eb3 · Cited by 59 · see citing works
Learning Bayesian Networks with Mixed Variables
Bøttcher, S. G. (2001) · International Conference on Artificial Intelligence and Statistics · Cited by 56 · see citing works
Foundations of Decision Analysis: Along the Way
Fishburn, P. C. (1989) · Management Science · doi:10.1287/mnsc.35.4.387 · Cited by 56 · see citing works
Assessing Probability Assessors: Calibration and Refinement.
DeGroot, M. H., & Fienberg, S. E. (1981) · Defense Technical Information Center (DTIC) · Cited by 56 · see citing works
Bayesian Nonparametric Calibration and Combination of Predictive Distributions
Bassetti, F., Casarin, R., & Ravazzolo, F. (2017) · Journal of the American Statistical Association · doi:10.1080/01621459.2016.1273117 · Cited by 56 · see citing works
“See, for example, Theorem 1 by Diaconis and Ylvisaker (1985) or Altomare and Campiti (1994), p.”
BackgroundMethodInfluentialHowson, C. (1998) · Springer eBooks · doi:10.1007/978-94-017-1735-9_4 · Cited by 54 · see citing works
Calibrated Forecasting and Merging
Kalai, E., Ehud Lehrer, A., & Rann Smorodinsky, B. (1999) · Games and Economic Behavior · doi:10.1006/game.1998.0608 · Cited by 53 · see citing works
“Notions of calibration, originated in Dawid (1982) , check whether the observed empirical”
BackgroundMethodInfluential
- Authors
- Dawid, A. P.
- Affiliations
- University College London (GB)
- Corresponding author
- Dawid, A. P.
- ORCID
- Dawid, A. P.
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 1982-09
- Volume
- 77
- Issue
- 379
- Pages
- 605-610
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2023943903
- Cited by
- 625
- References
- 24
- Citations by year
- 2026: 22 · 2025: 26 · 2024: 45 · 2023: 51 · 2022: 27 · 2021: 31 · 2020: 39 · 2019: 18 · 2018: 19 · 2017: 15 · 2016: 13 · 2015: 18 · 2014: 23 · 2013: 18 · 2012: 17
- FWCI
- 6.76
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Keywords
- well calibration, Bayesian forecasting, coherence, subjective probability, Dutch book argument
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
