The Comparison and Evaluation of Forecasters
The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Morris H. DeGroot, Stephen E. Fienberg, The Comparison and Evaluation of Forecasters, Journal of the Royal Statistical Society. Series D (The Statistician), Vol. 32, No. 1/2, Proceedings of the 1982 I.O.S. Annual Conference on Practical Bayesian Statistics (Mar. - Jun., 1983), pp. 12-22
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DeGroot and Fienberg (1983) showed that these two terms make up proper scoring rules in general, and that probabilistic predictions should minimise refinement loss subject to calibration.”
From Classifier calibration: a survey on how to assess and improve predicted class probabilities (2023) · cites this paper as DeGroot and Fienberg (1983)
Citations
Cites
Discrete Multivariate Analysis: Theory and Practice.
Beniger, J. R., Bishop, Y. M. M., Feinberg, S. E., Holland, P. W., Light, R. J., & Mosteller, F. (1975) · Contemporary Sociology · doi:10.2307/2063625 · Cited by 5,108
Optimal Statistical Decisions.
Siegmund, D., & DeGroot, M. H. (1972) · Journal of the American Statistical Association · doi:10.2307/2284675 · Cited by 3,386
Self-citationThe Analysis of Cross-Classified Categorical Data
Drane, J. W. (1979) · Technometrics · doi:10.1080/00401706.1979.10489789 · Cited by 1,941
Self-citationElicitation of Personal Probabilities and Expectations
Savage, L. J. (1971) · Journal of the American Statistical Association · doi:10.1080/01621459.1971.10482346 · Cited by 1,226
Theory of Games and Statistical Decisions.
Vajda, S., Blackwell, D., & Girshick, M. A. (1955) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2342525 · Cited by 1,032
Blackwell, D. (1951) · Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability · doi:10.1525/9780520411586-009 · Cited by 1,024
Dawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.1080/01621459.1982.10477856 · Cited by 625
Reliability of Subjective Probability Forecasts of Precipitation and Temperature
Murphy, A. H., & Winkler, R. L. (1977) · Applied Statistics · doi:10.2307/2346866 · Cited by 386
On the Reconciliation of Probability Assessments
Lindley, D. V., Tversky, A., & Brown, R. V. (1979) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2345078 · Cited by 340
Admissible Probability Measurement Procedures
Shuford, E. H., Albert, A., & Massengill, H. E. (1966) · Psychometrika · doi:10.1007/bf02289503 · Cited by 257
Cited by
Items 301–400 of 604
Robust PAC m : Training Ensemble Models Under Misspecification and Outliers
Zecchin, M., Park, S., Simeone, O., Kountouris, M., & Gesbert, D. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3295168 · Cited by 18
“Additional results on calibration in the form of reliability diagrams [45] can be found in supplementary material.”
BackgroundObjective paper structure comparison: Assessing comparison algorithms
Berger, C. E. H., & Ramos, D. (2012) · Forensic Science International · doi:10.1016/j.forsciint.2012.07.018 · Cited by 10
“Moreover, the discriminating power of the scores at hand is given by the Calibrated ECE curve (solid curve in Figure 6), because, once calibrated, the loss of information of the scores in the experimental set is only due to their non-perfect discriminating power (a proof for this can be found in [22]).”
The ABC’s of Who Benefits from Working with AI: Ability, Beliefs, and Calibration
Caplin, A., Deming, D. J., Li, S., Martín, D., Marx, P., Weidmann, B., & Ye, K. J. (2024) · National Bureau of Economic Research · doi:10.3386/w33021 · Cited by 25
Average Calibration Error: A Differentiable Loss for Improved Reliability in Image Segmentation
Barfoot, T., García-Peraza-Herrera, L. C., Glocker, B., & Vercauteren, T. K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72114-4_14 · Cited by 9
“Reliability diagrams are the reference means of assessing and visualising the calibration performance of a predictor over a dataset and this has extensively been used for classification tasks [5].”
BackgroundTowards a Rigorous Calibration Assessment Framework: Advancements in Metrics, Methods, and Use
Famiglini, L., Campagner, A., & Cabitza, F. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230327 · Cited by 17
Studying the Impact of Augmentations on Medical Confidence Calibration
Rao, A., Lee, J., & Aalami, O. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00260 · Cited by 14
“Reliability Plotting allows for visualizing the calibration of neural networks in a qualitative manner [15, 42].”
BackgroundMethodA View of Margin Losses as Regularizers of Probability Estimates
Masnadi-Shirazi, H., & Vasconcelos, N. M. (2015) · eScholarship (California Digital Library) · Cited by 9
Binary Classifier Calibration Using an Ensemble of Linear Trend Estimation
Naeini, M. P., & Cooper, G. F. (2016) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611974348.30 · Cited by 11
“Refinement accounts for the usefulness of the probabilities by favoring those that are either close to 0 or 1 [9, 7] Naeini and Cooper Page 8”
BackgroundTasche, D. (2021) · Statistics · doi:10.1080/02331888.2021.2016767 · Cited by 12
“3 (iii) of Denneberg and Maaß (2006), we obtain the existence of a strictly increasing function φ : R → R such that P [A |H] = φ(T ).”
BackgroundInfluentialMedical decision support systems based on machine learning
Chi, C. (2009) · doi:10.17077/etd.o5gmwvxk · Cited by 11
“by a reliability diagram [28].”
BackgroundPAC Confidence Predictions for Deep Neural Network ClassifiersPreprint
Park, S., Li, S., Lee, I., & Bastani, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.00716 · Cited by 24
“We first introduce the definition of calibration (DeGroot & Fienberg, 1983; Zadrozny & Elkan, 2002; Park et al., 2020b)— i.e., what we mean for a predicted confidence to be “correct”.”
BackgroundMethodFairCal: Fairness Calibration for Face VerificationPreprint
Salvador, T., Cairns, S., Voleti, V., Marshall, N., & Oberman, A. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03761 · Cited by 25
“Another measure is the Brier score (BS↓) (DeGroot & Fienberg, 1983), which estimates the mean squared error between the correctness of prediction and the confidence score: BS(D) = 1 |D| ∑ (ci,yi)∈D (1ŷi=yi − ci) 2 (5) For all the above metrics (ECE, KS, BS), lower is better.”
BackgroundMethodDawid, A. P. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess2064 · Cited by 9
Seckler, H., & Metzler, R. (2024) · Journal of Physics Photonics · doi:10.1088/2515-7647/ad884c · Cited by 14
“…on the prediction of the full time series of α , we define the mean absolute error as well as the root mean squared error To visualise the performance of the error prediction we use reliability diagrams [112, 113], which illustrate the observed errors as a function of the predicted uncertainties.”
MethodBelief elicitation with multiple point predictions
Eyting, M., & Schmidt, P. (2021) · European Economic Review · doi:10.1016/j.euroecorev.2021.103700 · Cited by 9
“15The idea goes back to DeGroot and Fienberg (1983); Murphy and Winkler (1987). See Gneiting and Katzfuss (2014) for a recent review.”
BackgroundIshwaran, H., & Blackstone, E. H. (2024) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2024.07.033 · Cited by 9
Modeling machine learning: A cognitive economic approach
Caplin, A., Martin, D., & Marx, P. (2025) · Journal of Economic Theory · doi:10.1016/j.jet.2025.105970 · Cited by 21
Data mining techniques: Modern approaches to application in credit scoring
Volkova, E. S., Gisin, V. B., & Solov'ev, V. I. (2017) · Finance and Credit · doi:10.24891/fc.23.34.2044 · Cited by 8
The Box Size Confidence Bias Harms Your Object Detector
Gilg, J., Teepe, T., Herzog, F., & Rigoll, G. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00152 · Cited by 14
“Inspired by the earlier visualization through reliability diagrams [8], the nowadays widely used Expected Calibration Error (ECE) [26] still has many shortcomings exposed and modifications proposed [21, 30, 44] including an adaption to object detectors [22].”
MethodCalibrating Imbalanced Classifiers with Focal Loss: An Empirical Study
Wang, C., Balazs, J., Szarvas, G., Erñst, P., Poddar, L., & Danchenko, P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-industry.14 · Cited by 22
Hughes, G., & Topp, K. (2015) · Entropy · doi:10.3390/e17085450 · Cited by 8
“However, both scoring rules are “strictly proper” [6,7].”
BackgroundStatistics in Expert Resolution: A Theory of Weights for Combining Expert Opinion
Cooke, R. M. (1990) · Boston studies in the philosophy of science · doi:10.1007/978-94-009-0619-8_4 · Cited by 8
Calibrated Confidence Scoring for Biometric Identification
Gorodnichy, D. O., & Hoshino, R. (2010) · Cited by 8
Adaptive Label SmoothingPreprint
Krothapalli, U., & Abbott, A. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06432 · Cited by 17
Reconciling Individual Probability Forecasts✱
Roth, A., Tolbert, A. W., & Weinstein, S. A. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593980 · Cited by 32
Probabilistic assessment of postfire debris-flow inundation in response to forecast rainfall
Prescott, A. B., McGuire, L. A., Jun, K.-S., Barnhart, K. R., & Oakley, N. S. (2024) · Natural hazards and earth system sciences · doi:10.5194/nhess-24-2359-2024 · Cited by 10
“We borrowed distributions-oriented methodologies developed in the weather modeling community over the last several decades (e.g., Bröcker and Smith, 2007; DeGroot and Fienberg, 1983; Gneiting et al., 2007; Murphy, 1993; Murphy and Winkler, 1987) to graphically assess the calibration-refinement…”
MethodKim, J., Cha, J., Kim, T., Lee, H., Yu, H.-Y., & Suh, M. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108967 · Cited by 7
Adapting Ensemble‐Calibration Techniques to Probabilistic Solar‐Wind Forecasting
Edward-Inatimi, N. O., Owens, M. J., Barnard, L., Turner, H., Marsh, M. S., Gonzi, S., Lang, M. S., & Riley, P. (2024) · Space Weather · doi:10.1029/2024sw004164 · Cited by 10
“…is used to infer biases towards over/under confident predictions and can also distinguish between forecasts which are well calibrated but differ in their decisiveness; a model’s abil-273 ity to assign more extreme probabilities to events with high confidence (Dawid, 1982; DeGroot & Fienberg, 1983).”
Evaluation of geological hazards susceptibility along a key railway based on machine learning
Liang, J., Qi, W., Xu, C. B., Wang, P., Sun, J., Zhang, X., Xue, Z., Chen, J., Cui, Y., Pan, J., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-26496-x · Cited by 9
“Calibration curve or Reliability diagram is frequently used to assess model uncertainty 47 .”
Exchangeability, Prediction and Predictive Modeling in Bayesian Statistics
Fortini, S., & Petrone, S. (2025) · Statistical Science · doi:10.1214/24-sts965 · Cited by 7
“…more usual ‘inferential approach’ is centered on inference on parameters, here one focuses on observable quantities and prediction, evaluates models and priors based on their implications on prediction, and even deduce models and parameters from the predictive rule (e.g. [37], [64], [65], [26]).”
Near-instantaneous Atmospheric Retrievals and Model Comparison with FASTER
Lueber, A., Karchev, K., Fisher, C., Heim, M., Trotta, R., & Heng, K. (2025) · The Astrophysical Journal Letters · doi:10.3847/2041-8213/adc7aa · Cited by 12
“…one can test using a large number of simulated examples the reliability of the posterior probabilities, as well as the ability of the network – averaged over repeated draws from the parameters’ priors – to identify the correct model (DeGroot & Fienberg 1983, see also Karchev et al. 2023).”
Predicting the Incremental Benefits of Online Information Search for Heterogeneous Consumers
Wang, H. E., Guo, X., Zhang, M., Wei, Q., & Chen, G. (2015) · Decision Sciences · doi:10.1111/deci.12200 · Cited by 7
Cal-Net: Jointly Learning Classification and Calibration On Imbalanced Binary Classification Tasks
Datta, A., Flynn, N. R., & Swamidass, S. J. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534411 · Cited by 7
Confidence-Calibrated Human Activity Recognition
Roy, D., Girdzijauskas, Š., & Socolovschi, S. (2021) · Sensors · doi:10.3390/s21196566 · Cited by 9
“To visualize calibration reliability diagram is used [38].”
MethodMany Are Better Than One: Improving Probabilistic Estimates from Decision Trees
Chawla, N. V. (2006) · Lecture notes in computer science · doi:10.1007/11736790_4 · Cited by 8
Modeling physician variability to prioritize relevant medical record information
Tajgardoon, M., Cooper, G. F., King, A. J., Clermont, G., Hochheiser, H., Hauskrecht, M., Sittig, D. F., & Visweswaran, S. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooaa058 · Cited by 8
Foundations of Risk Analysis: A Knowledge and Decision-Oriented Perspective
Banks, D. L. (2005) · Journal of the American Statistical Association · doi:10.1198/jasa.2005.s16 · Cited by 7
“Also, the work on calibrating probability forecasters (cf. DeGroot and Fienberg 1983) is relevant, as is more detail on elicitation (cf. Kadane and Winkler 1988).”
BackgroundVandalism Detection in Wikipedia: a Bag-of-Words Classifier ApproachPreprint
Belani, A. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1001.0700 · Cited by 9
Abramowicz, M. (2007) · SSRN Electronic Journal · doi:10.2139/ssrn.976809 · Cited by 7
“Hanson’s market scoring rules are built on a statistics literature on “scoring rules,” of which DeGroot and Fienberg (1983) provide a useful overview.”
Are Graph Neural Networks Miscalibrated?Preprint
Teixeira, L. V., Jalaian, B. A., & Ribeiro, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.02296 · Cited by 26
“Throughout this paper, we employ the two common tools used in the literature to evaluate calibration: reliability diagrams (DeGroot & Fienberg, 1983; Niculescu-Mizil & Caruana, 2005) and the expected calibration error metric (Guo et al., 2017; Naeini et al., 2015).”
BackgroundMethodCalibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction CalibrationPreprint
Venkatesh, B., Thiagarajan, J. J., Thopalli, K., & Sattigeri, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03875 · Cited by 16
“Therefore, the predicted softmax probabilities can be used to evaluate the quality of the prediction uncertainties and thus the model reliability [24, 19, 25].”
Uncertainty Estimation and Calibration with Finite-State Probabilistic RNNsPreprint
Wang, C., Lawrence, C., & Niepert, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.12010 · Cited by 8
“Reliability Diagrams (DeGroot & Fienberg, 1983; Niculescu-Mizil & Caruana, 2005) visualise whether a model is over-or under-confident by grouping predictions into bins according to their prediction probability.”
Using a Bootstrap Approach to Rate the RatersPreprint
Guettler, A. (2005) · SSRN Electronic Journal · doi:10.2139/ssrn.499185 · Cited by 7
“The concept of refinement pertains to how spread out the default predictions of a rating system are (DEGROOT and FIENBERG 1983).”
InfluentialCalibration for Long-tailed Scene Graph Generation
Zhu, X., Xing, Y., Wang, R., Wang, Y., & Lan, X. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680818 · Cited by 7
Friesacher, H. R., Engkvist, O., Mervin, L., Moreau, Y., & Arany, Á. (2025) · Journal of Cheminformatics · doi:10.1186/s13321-025-00964-y · Cited by 11
“More advanced methods are available for molecular representations and model architectures, including graph neural networks for handling molecular graphs and language models for processing SMILES representations.”
Uncertainties in outcome modelling in radiation oncology
Dünger, L., Mäusel, E., Zwanenburg, A., & Löck, S. (2025) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2025.100774 · Cited by 14
“A reliability diagram is a calibration plot that shows how well the uncertainty estimate is calibrated [202 – 204] through plotting the expected confidence based on the uncertainty estimate against the observed confidence.”
Hammonds, S. K., Eftestøl, T. C., Kurz, K. D., Fernández-Quilez, Á., & Initiative, A. D. N. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01604-5 · Cited by 7
“Model calibration in terms of ECE has its origin in forecasting [35], but has been explored and adapted in recent times for wider use within machine learning for binary and multi-class settings [34, 36].”
Connolly, B., Cohen, K. B., Santel, D., Bayram, U., & Pestian, J. P. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1736-3 · Cited by 7
Yi, C., Chen, H., Zhang, Y., Xu, Y., Zhou, Y., & Cui, L. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680910 · Cited by 14
A novel severity calibration algorithm for defect detection by constructing maps
Li, Y., Fan, B., Ding, W., Zhang, W., & Yin, J. (2022) · Information Sciences · doi:10.1016/j.ins.2022.06.076 · Cited by 6
On confidence computation and calibration of deep support vector data description
Deng, X., & Jiang, X. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106646 · Cited by 7
Photo-zSNthesis: Converting Type Ia Supernova Lightcurves to Redshift Estimates via Deep Learning
Qu, H., & Sako, M. (2023) · The Astrophysical Journal · doi:10.3847/1538-4357/aceafa · Cited by 11
Bag of Tricks for In-Distribution Calibration of Pretrained Transformers
Kim, J., Na, D., Choi, S., & Lim, S. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.40 · Cited by 7
Multiple Model Evaluation Absent the Gold Standard Through Model Combination
Iversen, E. S., Parmigiani, G., & Chen, S. (2008) · Journal of the American Statistical Association · doi:10.1198/016214507000001012 · Cited by 6
“Model calibration, a measure of reliability in repeated forecasts, quantifies agreement between the forecast value and the frequency of the forecasted event as a function of the forecast value (DeGroot and Fienberg 1983).”
BackgroundInfluentialSmooth Receiver Operating Characteristics (smROC) Curves
Klement, W., Flach, P. A., Japkowicz, N., & Matwin, S. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23783-6_13 · Cited by 6
Abramovicz, M. (2012) · The Journal of Prediction Markets · doi:10.5750/jpm.v1i2.423 · Cited by 6
Uncertainty Estimation with a VAE-Classifier Hybrid Model
Lin, S., Clark, R., Trigoni, N., & Roberts, S. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747520 · Cited by 6
Clustering and Classification Methods for Gene Expression Data Analysis
Garrett-Mayer, E. S., & Parmigiani, G. (2004) · Collection of Biostatistics Research Archive · Cited by 6
RELIABILITY OF OPTIONS MARKETS FOR CROP REVENUE INSURANCE RATING
Buschena, D. E., & Ziegler, L. (1999) · RePEc: Research Papers in Economics · doi:10.22004/ag.econ.30806 · Cited by 6
Quality assessment of physiological signals during ambulatory measurements
Schumm, J. (2010) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-a-006253101 · Cited by 6
“This equation can be interpreted as follows: among those samples on which the prediction is s, the score will be (s−1)2 with the relative frequency p(g|s), and it will be s(2) with the relative frequency (1 − p(g|s)) [118].”
BackgroundSpline-Based Probability CalibrationPreprint
Lucena, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.07751 · Cited by 23
“In Figure 4.1(a), we plot a reliability diagram [3] to assess how well-calibrated the vote percentage is.”
InfluentialEvaluating Scalable Uncertainty Estimation Methods for DNN-Based Molecular Property PredictionPreprint
Scalia, G., Grambow, C. A., Pernici, B., Li, Y., & Green, W. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03127 · Cited by 6
Temporal Probability CalibrationPreprint
Leathart, T., & Polaczuk, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02644 · Cited by 10
“Accuracy and confidence are often compared visually in reliability diagrams, where they are plotted against each other (DeGroot & Fienberg, 1983).”
BackgroundOn the Convergence of Deep Learning with Differential Privacy.Preprint
Bu, Z., Wang, H., Long, Q., & Su, W. (2021) · arXiv (Cornell University) · Cited by 7
PAC Prediction Sets Under Covariate ShiftPreprint
Park, S., Dobriban, E., Lee, I., & Bastani, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09848 · Cited by 29
“…way to quantify uncertainty is calibrated prediction (e.g., Brier, 1950; Cox, 1958; Miller, 1962; Murphy, 1972; Lichtenstein et al., 1977; DeGroot & Fienberg, 1983; Guo et al., 2017, etc), which aims to ensure that among instances with a predicted confidence p , the model is correct a…”
Loureiro, J., Petrella, L., Gil, P. S., Rocha, T., Paredes, S., & Henriques, J. M. O. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2025.3577486 · Cited by 8
“This assessment is usually performed via calibration diagrams [24], [25], Platt scaling [26], isotonic regression [27], and beta-calibration [28].”
The Role of Teacher Calibration in Knowledge Distillation
Kim, S., Park, S., Lee, J., & Kwak, N. (2025) · IEEE Access · doi:10.1109/access.2025.3585106 · Cited by 12
“Originally, ECE was designed for binary classification [24] and has been adapted for multi-class problems by treating the problem with K classes as a set of binary predictions: one for the target label and the other for non-target labels.”
Cuff, P., Cover, T. M., Kumar, G. R., & Zhao, L. (2011) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2011.6033851 · Cited by 5
Bias-corrected score decomposition for generalized quantiles
Ehm, W., & Ovcharov, E. Y. (2017) · Biometrika · doi:10.1093/biomet/asx004 · Cited by 5
“Additional criteria include skill scores (Murphy & Winkler, 1987), calibration and sharpness measures (DeGroot & Fienberg, 1983; Gneiting et al., 2007), as well as decompositions of the average score into three components commonly referred to as reliability, resolution and uncertainty (Murphy &…”
BackgroundCalibrated simplex-mapping classification
Heese, R., Schmid, J., Walczak, M., & Bortz, M. (2023) · PLoS ONE · doi:10.1371/journal.pone.0279876 · Cited by 7
“For binary classification problems, it is also interesting to take a look at the calibration curves (or reliability diagrams) in addition to the scores (DeGroot and Fienberg, 1983; Niculescu-Mizil and Caruana, 2005).”
Data Mining Techniques: Modern Approaches to Application in Credit Scoring
Volkova, V. S., Gisin, V. B., & Solov'ev, V. I. (2017) · Digest Finance · doi:10.24891/df.22.4.400 · Cited by 4
From Simple to Complex: A Progressive Framework for Document-level Informative Argument Extraction
Huang, Q., Zhang, Y., & Zhao, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.408 · Cited by 16
Calibrating the Predictions for Top-N Recommendations
Sato, M. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688177 · Cited by 7
Optimizing the liquidity parameter of logarithmic market scoring rules prediction markets
Lekwijit, S., & Sutivong, D. (2018) · Journal of Modelling in Management · doi:10.1108/jm2-06-2017-0066 · Cited by 5
Meta-free few-shot learning via representation learning with weight averaging
Chen, K., & Lee, C.-G. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892722 · Cited by 6
“A reliability diagram can be used to check model calibration visually, which plots an identity function between prediction accuracy and confidence when the model is perfectly calibrated [62].”
MethodUncertainty-sensitive Activity Recognition: A Reliability Benchmark and the CARING Models
Roitberg, A., Haurilet, M., Martínez, M., & Stiefelhagen, R. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412540 · Cited by 7
“However, this problem has been addressed before in image classification [6], [8], person identification [1] and classical machine learning [3], [21], [23].”
BackgroundPerformance of Likelihood Ratio Methods
Zadora, G., Martyna, A., Ramos, D., & Aitken, C. G. G. (2013) · doi:10.1002/9781118763155.ch6 · Cited by 5
Theoretical guidelines for a partially informed forecast examiner
Tsyplakov, A. A. (2014) · Munich Personal RePEc Archive (Ludwig Maximilian University of Munich) · Cited by 15
Walsh-Hadamard Variational Inference for Bayesian Deep LearningPreprint
Rossi, S., Marmin, S., & Filippone, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11248 · Cited by 18
“Figure 6: Reliability diagram and expected calibration error (ECE) of VGG16, ALEXNET and RESNET with WHVI [9, 41, 37].”
Lust, J., & Condurache, A. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.09381 · Cited by 5
“It can be explained as a decomposition between calibration and refinement, for further explanation we refer to [14].”
BackgroundVerification of River Stage Forecasts
Welles, E. (2005) · UA Campus Repository (The University of Arizona) · Cited by 5
On Calibration of Mixup Training for Deep Neural Networks
Maroñas, J., Ramos, D., & Paredes, R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-73973-7_7 · Cited by 8
A Soil Moisture-Informed Seismic Landslide Model Using SMAP Satellite Data
Farahani, A., & Ghayoomi, M. (2025) · Remote Sensing · doi:10.3390/rs17152671 · Cited by 6
Xiang, K., Shi, D., & Zhu, X. (2026) · Frontiers in Plant Science · doi:10.3389/fpls.2026.1826962 · Cited by 10
“Calibration quality is assessed through reliability diagrams (DeGroot and Fienberg, 1983), expected calibration error (ECE) (Naeini et al., 2015), and the Brier score (Brier, 1950).”
Andrews, B., Wongchokprasitti, C., Visweswaran, S., Lakhani, C. M., Patel, C. J., & Cooper, G. F. (2022) · SSRN Electronic Journal · doi:10.1016/j.artmed.2023.102546 · Cited by 6
Choi, M.-S., Lim, H., & Choo, J. (2023) · doi:10.18653/v1/2023.findings-ijcnlp.4 · Cited by 6
Rabe, M., Meliadò, E. F., Marschner, S. N., Belka, C., Corradini, S., van den Berg, C. A. T., Landry, G., & Kurz, C. (2025) · Physics in Medicine and Biology · doi:10.1088/1361-6560/add640 · Cited by 6
“…(Degroot and Fienberg 1983) and quantified through the expected calibration error (ECE), a standard scalar metric in machine learning to evaluate the discrepancy between predicted confidence levels and actual accuracy (Degroot and Fienberg 1983, Naeini et al 2015, Guo et al 2017, Nixon et al 2019).”
Subnetwork ensembling and data augmentation: Effects on calibration
Demir, A. Ç., Caton, S. J., & Dondio, P. (2023) · Expert Systems · doi:10.1111/exsy.13252 · Cited by 4
“Calibration is a notion which measures how a model's predictions match the empirical frequency of the true probabilities (Degroot & Fienberg, 1983).”
BackgroundPerception and Semantic Aware Regularization for Sequential Confidence Calibration
Peng, Z., Luo, Y., Chen, T., Xu, K., & Huang, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01027 · Cited by 6
“Evaluation Metrics : We adopt the widely used expected calibration error (ECE) [38], adaptive ECE (ACE) [39], maximum calibration error (MCE) [17], and reliability diagram [9] as calibration metrics.”
MethodCalibrating Bayesian decoders of neural spiking activity
Wei, G., Mansouri, Z. T., Wang, X., & Stevenson, I. H. (2024) · Journal of Neuroscience · doi:10.1523/jneurosci.2158-23.2024 · Cited by 3
Cross-calibration of categorical variables: An evaluation of the genetic algorithm approach
M’Hallah, R., & Aboukhamseen, S. M. (2018) · Applied Soft Computing · doi:10.1016/j.asoc.2018.10.009 · Cited by 4
Adapting to Label Shift with Bias-Corrected Calibration
Shrikumar, A., Alexandari, A. M., & Kundaje, A. B. (2019) · Cited by 4
Confidence Calibration for Incremental Learning
Kang, D., Jo, Y., Nam, Y., & Choi, J. (2020) · IEEE Access · doi:10.1109/access.2020.3007234 · Cited by 5
“In Figure 2, we visualize the overconfidence by the Reliability diagram [47] and the ECE of three models (refer to the caption for details).”
MethodTo protect or not to protect: Bayes decisions with forecasts
Krzysztofowicz, R., & Long, D. (1990) · European Journal of Operational Research · doi:10.1016/0377-2217(90)90243-5 · Cited by 4
Probleme des Qualitätsvergleichs von Kreditausfallprognosen
Krämer, W., & Bücker, M. (2011) · AStA Wirtschafts- und Sozialstatistisches Archiv · doi:10.1007/s11943-011-0096-0 · Cited by 4
ACCURACY UNCOMPOSED: AGAINST CALIBRATIONISM
Levinstein, B. A. (2017) · Episteme · doi:10.1017/epi.2016.50 · Cited by 4
“DeGroot and Fienberg (1982, 1983) prove that all such measures in fact obey the axiom of DECOMPOSITION.”
BackgroundProbabilistic Object Detection with an Ensemble of Experts
Miller, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-65414-6_5 · Cited by 4
An Application of Operational-Subjective Statistical Methods to Rational Expectations
Blattenberger, G., & Lad, F. (1988) · Journal of Business and Economic Statistics · doi:10.1080/07350015.1988.10509690 · Cited by 4
Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical DomainPreprint
Mimori, T., Sasada, K., Matsui, H., & Sato, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.01659 · Cited by 9
Metadata
- Authors
- DeGroot, M. H., & Fienberg, S. E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- The Statistician
- Publisher
- JSTOR
- Published
- 1983-03
- Volume
- 32
- Issue
- 1/2
- Pages
- 12
- ISSN
- 0039-0526 (ISSN-L)
- OpenAlex
- W1599263113
- Cited by
- 604
- References
- 18
- Citations by year
- 2026: 14 · 2025: 38 · 2024: 32 · 2023: 51 · 2022: 42 · 2021: 52 · 2020: 48 · 2019: 36 · 2018: 26 · 2017: 20 · 2016: 16 · 2015: 21 · 2014: 12 · 2013: 17 · 2012: 14
- FWCI
- 5.36
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the Royal Statistical Society Series D (The Statistician) (publishedVersion)
- Defense Technical Information Center (DTIC) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- forecast accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.