The Comparison and Evaluation of Forecasters
The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Morris H. DeGroot, Stephen E. Fienberg, The Comparison and Evaluation of Forecasters, Journal of the Royal Statistical Society. Series D (The Statistician), Vol. 32, No. 1/2, Proceedings of the 1982 I.O.S. Annual Conference on Practical Bayesian Statistics (Mar. - Jun., 1983), pp. 12-22
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DeGroot and Fienberg (1983) showed that these two terms make up proper scoring rules in general, and that probabilistic predictions should minimise refinement loss subject to calibration.”
From Classifier calibration: a survey on how to assess and improve predicted class probabilities (2023) · cites this paper as DeGroot and Fienberg (1983)
Citations
Cites
Discrete Multivariate Analysis: Theory and Practice.
Beniger, J. R., Bishop, Y. M. M., Feinberg, S. E., Holland, P. W., Light, R. J., & Mosteller, F. (1975) · Contemporary Sociology · doi:10.2307/2063625 · Cited by 5,108
Optimal Statistical Decisions.
Siegmund, D., & DeGroot, M. H. (1972) · Journal of the American Statistical Association · doi:10.2307/2284675 · Cited by 3,386
Self-citationThe Analysis of Cross-Classified Categorical Data
Drane, J. W. (1979) · Technometrics · doi:10.1080/00401706.1979.10489789 · Cited by 1,941
Self-citationElicitation of Personal Probabilities and Expectations
Savage, L. J. (1971) · Journal of the American Statistical Association · doi:10.1080/01621459.1971.10482346 · Cited by 1,226
Theory of Games and Statistical Decisions.
Vajda, S., Blackwell, D., & Girshick, M. A. (1955) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2342525 · Cited by 1,032
Blackwell, D. (1951) · Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability · doi:10.1525/9780520411586-009 · Cited by 1,024
Dawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.1080/01621459.1982.10477856 · Cited by 625
Reliability of Subjective Probability Forecasts of Precipitation and Temperature
Murphy, A. H., & Winkler, R. L. (1977) · Applied Statistics · doi:10.2307/2346866 · Cited by 386
On the Reconciliation of Probability Assessments
Lindley, D. V., Tversky, A., & Brown, R. V. (1979) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2345078 · Cited by 340
Admissible Probability Measurement Procedures
Shuford, E. H., Albert, A., & Massengill, H. E. (1966) · Psychometrika · doi:10.1007/bf02289503 · Cited by 257
Cited by
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
Predicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884
“On real problems where the true conditional probabilities are not known, model calibration can be visualized with reliability diagrams (DeGroot & Fienberg, 1982).”
BackgroundProbabilistic Forecasts, Calibration and Sharpness
Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2007) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2007.00587.x · Cited by 1,783
“Diebold and Mariano (1995) gave a thoughtful discussion of these issues, and we refer to their work for a comprehensive account of tests of predictive performance.”
BackgroundA survey of uncertainty in deep neural networks
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10562-9 · Cited by 1,406
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280
“DeGroot and Fienberg [7] show that the MSE can be separated into two components, one measuring calibration and the other measuring refinement.”
BackgroundMethodForecast Verification: A Practitioner's Guide in Atmospheric Science
(2011) · Wiley eBooks · doi:10.1002/9781119960003 · Cited by 1,267
Making and Evaluating Point Forecasts
Gneiting, T. (2011) · Journal of the American Statistical Association · doi:10.1198/jasa.2011.r10138 · Cited by 1,154
Zhu, Y., Zabaras, N., Koutsourelakis, P.-S., & Perdikaris, P. (2019) · Journal of Computational Physics · doi:10.1016/j.jcp.2019.05.024 · Cited by 1,030
“Here, the predictive uncertainty is calibrated using the reliability diagram [59].”
MethodObtaining well calibrated probabilities using Bayesian binning
Pakdaman Naeini, M., Cooper, G., & Hauskrecht, M. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9602 · Cited by 991
“Figure 1 shows a hypothetical example of a reliability curve (DeGroot and Fienberg 1983; Niculescu-Mizil and Caruana 2005), which displays the calibration performance of a prediction method.”
BackgroundCan You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02530 · Cited by 989
“We refer to DeGroot & Fienberg (1983) for the decomposition of Brier score into calibration and refinement for classification and to (Br¨ocker, 2009) for the general decomposition for any proper scoring rule.”
MethodAn experimental comparison of performance measures for classification
Ferri, C., Hernández-Orallo, J., & Modroiu, R. (2009) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.08.010 · Cited by 889
“It is usually a measure of the reliability of the prediction (DeGroot and Fienberg, 1982).”
BackgroundApplication-independent evaluation of speaker detection
Brümmer, N., & du Preez, J. (2006) · Computer Speech & Language · doi:10.1016/j.csl.2005.08.001 · Cited by 595
The spread of low-credibility content by social bots
Shao, C., Ciampaglia, G. L., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F. (2018) · RePEc: Research Papers in Economics · Cited by 574
Accuracy and the Laws of Credence
Pettigrew, R. (2016) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780198732716.001.0001 · Cited by 473
Arming the public with artificial intelligence to counter social bots
Yang, K., Varol, O., Davis, C. A., Ferrara, E., Flammini, A., & Menczer, F. (2019) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.115 · Cited by 419
“The figure also shows reliability diagrams for raw and calibrated scores (DeGroot and Fienberg, 1983).”
BackgroundPrediction of Germline Mutations and Cancer Risk in the Lynch Syndrome
Chen, S., Wang, W., Lee, S., Nafa, K., Lee, J., Romans, K., Watson, P., Gruber, S. B., Euhus, D., Kinzler, K. W., et al. (2006) · JAMA · doi:10.1001/jama.296.12.1479 · Cited by 384
Rizve, M. N., Duarte, K., Rawat, Y. S., & Shah, M. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.06329 · Cited by 706
“Since, calibration measures the discrepancy between the confidence level of a network’s individual predictions and its overall accuracy (Dawid, 1982; Degroot & Fienberg, 1983); for poorly calibrated networks, an incorrect prediction might have high confidence.”
BackgroundRegularizing Class-Wise Predictions via Self-Knowledge Distillation
Yun, S., Park, J., Lee, K., & Shin, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01389 · Cited by 321
Burgman, M. A., McBride, M., Ashton, R., Speirs-Bridge, A., Flander, L., Wintle, B., Fidler, F., Rumpff, L., & Twardy, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0022998 · Cited by 314
“Different approaches are appropriate depending on whether the purpose is evaluation of performance or aggregation of results [19], [30], [31], [32].”
BackgroundData mining in metric space: an empirical analysis of supervised learning performance criteria
Caruana, R., & Niculescu-Mizil, A. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014063 · Cited by 303
Contrastive Test-Time Adaptation
Chen, D., Wang, D., Darrell, T., & Ebrahimi, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00039 · Cited by 272
“calibration [16] and illustrate reliability diagrams [10, 35].”
Interpreting and Unifying Outlier Scores
Kriegel, H.-P., Kroger, P., Schubert, E., & Zimek, A. (2011) · Proceedings of the 2011 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972818.2 · Cited by 266
“, the forecast probabilities should be concentrated near 1 and near 0 (still being correlated with the actual occurrence of the forecast event) [13, 47].”
BackgroundScalia, G., Grambow, C. A., Pernici, B., Li, Y.-P., & Green, W. H. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00975 · Cited by 269
Decision Theory: Principles and Approaches
Parmigiani, G., & Inoue, L. Y. T. (2009) · Wiley Series in Probability and Statistics · doi:10.1002/9780470746684 · Cited by 266
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X., & Hu, C. (2023) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2023.110796 · Cited by 243
Classifier calibration: a survey on how to assess and improve predicted class probabilities
Silva Filho, T., Song, H., Perello-Nieto, M., Santos-Rodriguez, R., Kull, M., & Flach, P. (2023) · Machine Learning · doi:10.1007/s10994-023-06336-7 · Cited by 228
“DeGroot and Fienberg (1983) showed that these two terms make up proper scoring rules in general, and that probabilistic predictions should minimise refinement loss subject to calibration.”
BackgroundScoring rules and the evaluation of probabilities
Winkler, R. L., Muñoz, J., Cervera, J. L., Bernardo, J. M., Blattenberger, G., Kadane, J. B., Lindley, D. V., Murphy, A. H., Oliver, R. M., & Ríos-Insua, D. (1996) · Test · doi:10.1007/bf02562681 · Cited by 227
Combining Probability Forecasts
Ranjan, R., & Gneiting, T. (2010) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2009.00726.x · Cited by 224
“This is akin to the setting in DeGroot and Fienberg (1982, 1983) and Murphy and Winkler (1987) but considers an arbitrary number k of individual probability forecasts, each of which is a random variable, with full generality in the joint depen- dence structure.”
MethodCalibrating Deep Neural Networks using Focal LossPreprint
Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P. H. S., & Dokania, P. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09437 · Cited by 257
“Baselines Along with cross-entropy loss, we compare our method against the following baselines: a) MMCE (Maximum Mean Calibration Error) [16], a continuous and differentiable proxy for calibration error that is normally used as a regulariser alongside cross-entropy, b) Brier loss [1], the squared error between the pre…”
MethodBayesian Modeling of Uncertainty in Ensembles of Climate Models
Smith, R. L., Tebaldi, C., Nychka, D., & Mearns, L. O. (2009) · Journal of the American Statistical Association · doi:10.1198/jasa.2009.0007 · Cited by 225
“Uncertainties in climate change projections are broadly of three types (Meehl et al. 2007): (a) natural climate variability; (b) uncertainties in the responses to climate forcing factors, such as changes in atmospheric levels of greenhouse gases and sulfate aerosols; and (c) uncertainties in future emissions of greenh…”
BackgroundStar–galaxy classification using deep convolutional neural networks
Kim, E. J., & Brunner, R. J. (2017) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stw2672 · Cited by 221
Buja, A., Stuetzle, W., & Shen, Y. (2005) · Cited by 308
An Introduction to Application-Independent Evaluation of Speaker Recognition Systems
van Leeuwen, D. A., & Brümmer, N. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74200-5_19 · Cited by 179
“This is similar in spirit, but not in form, to the refinement/calibration decomposition which was introduced by De Groot two decades ago [15] and again recently examined for speaker detection in ref.”
Nascita, A., Montieri, A., Aceto, G., Ciuonzo, D., Persico, V., & Pescapé, A. (2021) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2021.3098157 · Cited by 177
Evaluating and Calibrating Uncertainty Prediction in Regression Tasks
Levi, D., Gispan, L., Giladi, N., & Fetaya, E. (2022) · Sensors · doi:10.3390/s22155540 · Cited by 169
“Calibration plots or Reliability diagrams provide a visual representation of uncertainty prediction calibration [3, 19] by plotting expected sample accuracy as a function of confidence.”
BackgroundEhm, W., Gneiting, T., Jordan, A., & Krüger, F. (2016) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12154 · Cited by 170
“Interest sometimes centres on decompositions of expected or empirical scores into uncertainty, resolution, and reliability components, as studied by DeGroot and Fienberg (1983), Bröcker (2009) and Bentzien and Friederichs (2014), among others.”
BackgroundInfluentialAlleviating naive Bayes attribute independence assumption by attribute weighting
Zaidi, N. A., Cerquides, J., Carman, M., & Webb, G. I. (2013) · DIGITAL.CSIC (Spanish National Research Council (CSIC)) · doi:10.13039/501100000923 · Cited by 167
Shouval, R., Labopin, M., Bondi, O., Mishan-Shamay, H., Shimoni, A., Ciceri, F., Esteve, J., Giebel, S., Gorin, N. C., Schmid, C., et al. (2015) · Journal of Clinical Oncology · doi:10.1200/jco.2014.59.1339 · Cited by 166
“The quality of the score after calibration was evaluated through a reliability diagram, which verified that in each score interval, defined according to the deciles, the mean score was consistent with the observed proportions of events.(22,23) The prediction model’s discriminationwas assessed using the area under the…”
MethodGonzalez-Rodriguez, J., Rose, P., Ramos, D., Toledano, D. T., & Ortega-Garcia, J. (2007) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/tasl.2007.902747 · Cited by 160
Obtaining Calibrated Probabilities from BoostingPreprint
Niculescu-Mizil, A., & Caruana, R. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.1403 · Cited by 183
Converting Output Scores from Outlier Detection Algorithms into Probability Estimates
Gao, J., & Tan, P. (2006) · Proceedings · doi:10.1109/icdm.2006.43 · Cited by 145
The BOSARIS Toolkit: Theory, Algorithms and Code for Surviving the New DCFPreprint
Brümmer, N., & de Villiers, E. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1304.2865 · Cited by 192
Bayesian stochastic optimization of reservoir operation using uncertain forecasts
Karamouz, M., & Vasiliadis, H. V. (1992) · Water Resources Research · doi:10.1029/92wr00103 · Cited by 141
Practical Deep Learning with Bayesian PrinciplesPreprint
Osawa, K., Swaroop, S., Jain, A., Eschenhagen, R., Turner, R. E., Yokota, R., & Khan, M. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02506 · Cited by 206
“We also show calibration curves [7] in Figures 1 and 14.”
The Economic Consequences of Data Privacy Regulation: Empirical Evidence from GDPRPreprint
Aridor, G., Che, Y., Nelson, W., & Salz, T. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3522845 · Cited by 137
The Right Tool for the Job: Matching Model and Instance Complexities
Schwartz, R., Stanovsky, G., Swayamdipta, S., Dodge, J., & Smith, N. A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.593 · Cited by 238
A Study of Parts-Based Object Class Detection Using Complete Graphs
Bergtholdt, M., Kappes, J. H., Schmidt, S., & Schnörr, C. (2009) · International Journal of Computer Vision · doi:10.1007/s11263-009-0209-1 · Cited by 142
“order-preserving—function is fitted by minimizing the mean-squared error to the empirical class membership probability using the pair-adjacent violators (PAV) algorithm (DeGroot and Fienberg 1982).”
MethodHemming, V., Walshe, T. V., Hanea, A. M., Fidler, F. M., & Burgman, M. A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0198468 · Cited by 139
Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation
Tan, S., Caruana, R., Hooker, G., & Lou, Y. (2018) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3278721.3278725 · Cited by 211
“ch we exploit in Section 2.3.3 to detect differences between the mimic and outcome models. 2.3.2 Calibrating model inputs. Calibration is the process of matching predicted and empirical probabilities [14, 28]. If a risk score is well-calibrated, the relationship between the risk score and empirical probabilities will…”
BackgroundInfluentialReliable support: measuring calibration of likelihood ratios
Ramos, D., & Gonzalez-Rodriguez, J. (2013) · Forensic Science International · doi:10.1016/j.forsciint.2013.04.014 · Cited by 118
“In order to measure the accuracy of a validation set of posterior probabilities from the forecaster, the following procedure has been proposed [11]: LC 1⁄4 1 N p X”
BackgroundMethodInfluentialLearning Patient-Specific Cancer Survival Distributions as a Sequence of Dependent Regressors
Lin, H.-C., Baracos, V. E., Greiner, R., & Yu, C.-N. J. (2011) · neural information processing systems · Cited by 125
Analyzing the quality and challenges of uncertainty estimations for brain tumor segmentation
Jungo, A., Balsiger, F., & Reyes, M. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00282 · Cited by 115
“Reliability diagrams (DeGroot and Fienberg, 1983) assess the quality of a model’s confidence.”
BackgroundOn calibration of language recognition scores
Brümmer, N., & van Leeuwen, D. A. (2006) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.1109/odyssey.2006.248106 · Cited by 118
Deconstructing cross-entropy for probabilistic binary classifiers
Ramos, D., Franco-Pedroso, J., Lozano-Diez, A., & Gonzalez-Rodriguez, J. (2018) · Entropy · doi:10.3390/e20030208 · Cited by 99
“However, it is not enough: probabilities must also present a good calibration [18,20].”
BackgroundMethodInfluentialCalibration, Coherence, and Scoring Rules
Seidenfeld, T. (1985) · Philosophy of Science · doi:10.1086/289244 · Cited by 107
Werner, C. P., Bedford, T., Cooke, R. M., Hanea, A. M., & Morales‐Nápoles, O. (2016) · European Journal of Operational Research · doi:10.1016/j.ejor.2016.10.018 · Cited by 127
Information‐Theoretical Assessment of the Performance of Likelihood Ratio Computation Methods
Ramos, D., González-Rodríguez, J., Zadora, G., & Aitken, C. G. G. (2013) · Journal of Forensic Sciences · doi:10.1111/1556-4029.12233 · Cited by 119
Properties and Benefits of Calibrated Classifiers
Cohen, I., & Goldszmidt, M. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30116-5_14 · Cited by 125
“Procedures for calibrating classifiers have been proposed in different contexts: In weather prediction tasks [1], in game theory [2, 3], and more recently in the context of pattern classification [4, 5].”
BackgroundMethodInfluentialDecomposition and graphical portrayal of the quantile score
Wahl, S., & Friederichs, P. (2013) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.2284 · Cited by 110
“1 Decomposition of proper scores Following Gneiting and Raftery (2007), the entropy e(Q) = S(Q,Q) is defined as the minimal achievable score under Q if S(.”
BackgroundMethodThe Analysis of Gene Expression Data: An Overview of Methods and Software
Parmigiani, G., Garrett, E. S., Irizarry, R. A., & Zeger, S. L. (2003) · Statistics for biology and health · doi:10.1007/0-387-21679-0_1 · Cited by 110
“Statistical validation of probabilistic models (DeGroot and Fienberg, 1983) has two goals: assessing calibration (that is, the correspondence between the fraction predicted and the fraction observed in the validation sample) and measuring refinement (that is, the ability of the model to discriminate between classes).”
BackgroundAn artificial intelligence system for predicting customer default in e-commerce
Vanneschi, L., Horn, D. M., Castelli, M., & Popovič, A. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.03.025 · Cited by 105
“Such a classifier is represented by a diagonal line in the reliability plot (Zadrozny and Elkan, 2002; DeGroot and Fienberg, 1983).”
MethodPosterior calibration and exploratory analysis for natural language processing models
Nguyen, K., & O'Connor, B. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1182 · Cited by 179
“5 Applying this factorization to the Brier score leads to the calibrationrefinement decomposition (DeGroot and Fienberg, 1983), in terms of expectations with respect to the prediction strength distribution P(q):”
MethodBayesian deep learning for error estimation in the analysis of anomalous diffusion
Seckler, H., & Metzler, R. (2022) · Nature Communications · doi:10.1038/s41467-022-34305-6 · Cited by 108
“To analyse the performance of the error prediction we use a reliability diagram [1, 2, 79] in figure 3b.”
Evaluating probabilities: asymmetric scoring rules
Winkler, R. (1994) · Management Science · doi:10.1287/mnsc.40.11.1395 · Cited by 102
Calibration of Option‐Based Probability Assessments in Agricultural Commodity Markets
Fackler, P. L., & King, R. P. (1990) · American Journal of Agricultural Economics · doi:10.2307/1243146 · Cited by 94
Allocating the weights in the linear opinion pool
Genest, C., & McConway, K. (1990) · Journal of Forecasting · doi:10.1002/for.3980090106 · Cited by 111
Measuring, refining and calibrating speaker and language information extracted from speech
Brümmer, N. (2010) · SUNScholar (Stellenbosch University) · Cited by 145
Werther, M., Odermatt, D., Simis, S. G. H., Gurlin, D., Lehmann, M. K., Kutser, T., Gupana, R. S., Varley, A., Hunter, P., Tyler, A. N., et al. (2022) · Remote Sensing of Environment · doi:10.1016/j.rse.2022.113295 · Cited by 96
Predicting systemic financial crises with recurrent neural networks
Tölö, E. (2020) · Journal of Financial Stability · doi:10.1016/j.jfs.2020.100746 · Cited by 122
Coherent combination of experts' opinions
Dawid, A. P., DeGroot, M. H., Mortera, J., Cooke, R. M., French, S., Genest, C., Schervish, M. J., Lindley, D. V., McConway, K., & Winkler, R. L. (1995) · Test · doi:10.1007/bf02562628 · Cited by 101
Self-citationLocal Temperature Scaling for Probability Calibration
Ding, Z., Han, X., Liu, P., & Niethammer, M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00681 · Cited by 147
“As it is difficult to work directly with this definition to assess miscalibration, we extend several visual and quantitative metrics [11, 53, 54, 56, 57], which have previously been proposed in the context of classification.”
BackgroundMethodAdaptive temperature scaling for Robust calibration of deep neural networks
Balanya, S. A., Maroñas, J., & Ramos, D. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09505-4 · Cited by 118
“The extent to which the confidence outputs of a classifier can be interpreted as class probabilities is what is known as the calibration of a classifier [4, 5].”
BackgroundCalibration and information in expert resolution; a classical approach
Cooke, R. M., Mendel, M. B., & Thijs, W. L. T. (1988) · Automatica · doi:10.1016/0005-1098(88)90011-8 · Cited by 83
Detection of Mental Health from Reddit via Deep Contextualized Representations
Jiang, Z., Levitan, S. I., Zomick, J., & Hirschberg, J. (2020) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2020.louhi-1.16 · Cited by 85
“…the random baseline (.125) indicates that, though under binary settings deep contextualized word representation is a strong feature, the model is not well calibrated, (DeGroot and Fienberg, 1983; Niculescu-Mizil and Caruana, 2005) as is often the case for modern deep networks (Guo et al., 2017).”
BackgroundCrimes analysis software: ‘pins in maps’, clustering and Bayes net prediction
Oatley, G., & Ewart, B. W. (2003) · Expert Systems with Applications · doi:10.1016/s0957-4174(03)00097-6 · Cited by 80
Evidential calibration of binary SVM classifiers
Xu, P., Davoine, F., Zha, H., & Denœux, T. (2016) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2015.05.002 · Cited by 83
“In this context, the performance of each base classifier is not of primary importance, instead the calibration step becomes the major issue [7, 3].”
BackgroundA benchmark on uncertainty quantification for deep learning prognostics
Basora, L., Viens, A., Chao, M. A., & Olivé, X. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110513 · Cited by 106
Spiegelhalter, D. J., Franklin, R. C. G., & Bull, K. (1990) · Machine intelligence and pattern recognition · doi:10.1016/b978-0-444-88738-2.50029-4 · Cited by 84
Reliability Theories of Justified Credence
Tang, W. H. (2016) · Mind · doi:10.1093/mind/fzv199 · Cited by 76
AlignMixup: Improving Representations By Interpolating Aligned Features
Venkataramanan, S., Kijak, E., Amsaleg, L., & Avrithis, Y. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.01858 · Cited by 81
“Calibration According to [13], calibration measures the discrepancy between the accuracy and confidence level of a network’s predictions.”
BackgroundEvaluating Attribution for Graph Neural Networks
Sánchez-Lengeling, B., Wei, J. N., Lee, B. K., Reif, E., Wang, P. Y., Qian, W. W., McCloskey, K., Colwell, L. J., & Wiltschko, A. B. (2020) · neural information processing systems · Cited by 202
Separating the Brier Score into Calibration and Refinement Components: A Graphical Exposition
Blattenberger, G., & Lad, F. (1985) · The American Statistician · doi:10.1080/00031305.1985.10479382 · Cited by 75
Forensic evaluation of the evidence using automatic speaker recognition systems
Ramos, D. (2007) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · Cited by 74
On the appropriateness of Platt scaling in classifier calibration
Böken, B. (2021) · Information Systems · doi:10.1016/j.is.2020.101641 · Cited by 70
The distribution of calibrated likelihood-ratios in speaker recognition
van Leeuwen, D. A., & Brümmer, N. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-406 · Cited by 73
“Since around 2004 [1,2] the concept of calibration in speaker recognition has been generalized to a range of operating points by using proper scoring rules [17] to evaluate probabilistic statements about whether a trial is a same-speaker (target) or different-speaker (non-target) trial.”
BackgroundA Log-Linear Model with Latent Features for Dyadic Prediction
Menon, A. K., & Elkan, C. (2010) · Proceedings · doi:10.1109/icdm.2010.148 · Cited by 78
“If we train using a proper loss function, then it is guaranteed that we obtain well-calibrated probability estimates [22].”
MethodThe Effect of Privacy Regulation on the Data Industry: Empirical Evidence from Gdpr
Aridor, G., Che, Y., & Salz, T. (2020) · The RAND Journal of Economics · doi:10.1111/1756-2171.12455 · Cited by 101
Automated face recognition in forensic science: Review and perspectives
Jacquet, M., & Champod, C. (2020) · Forensic Science International · doi:10.1016/j.forsciint.2019.110124 · Cited by 55
“Calibration has been described in general terms in [68] and addressed in the Bayesian framework subsequently (see [69] for example).”
BackgroundEvaluating probability forecasts for GDP declines using alternative methodologies
Lahiri, K., & Wang, J. G. (2012) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2012.07.004 · Cited by 64
“Even though calibration is a natural feature to have, it is resolution that makes the forecasts useful in practice, cf. DeGroot and Fienberg (1983) .”
Fusion of detection probabilities and comparison of multisensor systems
Krzysztofowicz, R., & Long, D. (1990) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/21.57281 · Cited by 63
An Axiomatic View of Statistical Privacy and Utility
Kifer, D., & Lin, B.-R. (2012) · Journal of Privacy and Confidentiality · doi:10.29012/jpc.v4i1.610 · Cited by 59
Bisaso, K. R., Karungi, S. A., Kiragga, A. N., Mukonzo, J., & Castelnuovo, B. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0659-x · Cited by 71
Document Understanding Dataset and Evaluation (DUDE)
Landeghem, J. V., Powalski, R., Tito, R., Jurkiewicz, D., Blaschko, M. B., Borchmann, Ł., Coustaty, M., Moens, S., Pietruszka, M., Ackaert, B., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01789 · Cited by 216
“Calibration requires that the probability a model assigns to its predictions equals their true likelihood of being correct [19, 20, 101].”
BackgroundThe role of the information set for forecasting—with applications to risk management
Holzmann, H., & Eulert, M. (2014) · The Annals of Applied Statistics · doi:10.1214/13-aoas709 · Cited by 87
“The second part of the corollary extends results by Bröcker (2009) and DeGroot and Fienberg (1983) from finite to general real state space.”
BackgroundImproving Uncertainty Estimation in Convolutional Neural Networks Using Inter-rater Agreement
Jensen, M. H., Jørgensen, D. R., Jalaboi, R., Hansen, M. E., & Olsen, M. A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32251-9_59 · Cited by 85
ConMatch: Semi-supervised Learning with Confidence-Guided Consistency Regularization
Kim, J., Min, Y., Kim, D.-H., Lee, G., Seo, J., Ryoo, K., & Kim, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20056-4_39 · Cited by 84
Learning from ExpertsPreprint
Valsecchi, I. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1123412 · Cited by 57
Ma, S., Wang, X., Lei, Y., Shi, C., Yin, M., & Ma, X. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642671 · Cited by 94
“One of the most widely used measurements is Reliability diagrams [14, 27, 58] (Figure 1), assessing the alignment between stated confidence and actual accuracy.”
Busk, J., Jørgensen, P. B., Bhowmik, A., Schmidt, M. N., Winther, O., & Vegge, T. (2021) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ac3eb3 · Cited by 84
“Calibration alone is not sufficient to ensure that individual uncertainty estimates are informative [4, 6, 19, 33].”
Classifying Variants of Undetermined Significance in BRCA2 with Protein Likelihood Ratios
Karchin, R., Agarwal, M. K., Šali, A., Couch, F. J., & Beattie, M. S. (2008) · Cancer Informatics · doi:10.4137/cin.s618 · Cited by 55
“The domainspecific sequences were used as input to the SAM-T2K iterative multiple sequence alignmentbuilding algorithm [29].”
Method
Metadata
- Authors
- DeGroot, M. H., & Fienberg, S. E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- The Statistician
- Publisher
- JSTOR
- Published
- 1983-03
- Volume
- 32
- Issue
- 1/2
- Pages
- 12
- ISSN
- 0039-0526 (ISSN-L)
- OpenAlex
- W1599263113
- Cited by
- 604
- References
- 18
- Citations by year
- 2026: 14 · 2025: 38 · 2024: 32 · 2023: 51 · 2022: 42 · 2021: 52 · 2020: 48 · 2019: 36 · 2018: 26 · 2017: 20 · 2016: 16 · 2015: 21 · 2014: 12 · 2013: 17 · 2012: 14
- FWCI
- 5.36
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the Royal Statistical Society Series D (The Statistician) (publishedVersion)
- Defense Technical Information Center (DTIC) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- forecast accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.