The Comparison and Evaluation of Forecasters
The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Morris H. DeGroot, Stephen E. Fienberg, The Comparison and Evaluation of Forecasters, Journal of the Royal Statistical Society. Series D (The Statistician), Vol. 32, No. 1/2, Proceedings of the 1982 I.O.S. Annual Conference on Practical Bayesian Statistics (Mar. - Jun., 1983), pp. 12-22
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DeGroot and Fienberg (1983) showed that these two terms make up proper scoring rules in general, and that probabilistic predictions should minimise refinement loss subject to calibration.”
From Classifier calibration: a survey on how to assess and improve predicted class probabilities (2023) · cites this paper as DeGroot and Fienberg (1983)
Citations
Cites
Discrete Multivariate Analysis: Theory and Practice.
Beniger, J. R., Bishop, Y. M. M., Feinberg, S. E., Holland, P. W., Light, R. J., & Mosteller, F. (1975) · Contemporary Sociology · doi:10.2307/2063625 · Cited by 5,108
Optimal Statistical Decisions.
Siegmund, D., & DeGroot, M. H. (1972) · Journal of the American Statistical Association · doi:10.2307/2284675 · Cited by 3,386
Self-citationThe Analysis of Cross-Classified Categorical Data
Drane, J. W. (1979) · Technometrics · doi:10.1080/00401706.1979.10489789 · Cited by 1,941
Self-citationElicitation of Personal Probabilities and Expectations
Savage, L. J. (1971) · Journal of the American Statistical Association · doi:10.1080/01621459.1971.10482346 · Cited by 1,226
Theory of Games and Statistical Decisions.
Vajda, S., Blackwell, D., & Girshick, M. A. (1955) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2342525 · Cited by 1,032
Blackwell, D. (1951) · Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability · doi:10.1525/9780520411586-009 · Cited by 1,024
Dawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.1080/01621459.1982.10477856 · Cited by 625
Reliability of Subjective Probability Forecasts of Precipitation and Temperature
Murphy, A. H., & Winkler, R. L. (1977) · Applied Statistics · doi:10.2307/2346866 · Cited by 386
On the Reconciliation of Probability Assessments
Lindley, D. V., Tversky, A., & Brown, R. V. (1979) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2345078 · Cited by 340
Admissible Probability Measurement Procedures
Shuford, E. H., Albert, A., & Massengill, H. E. (1966) · Psychometrika · doi:10.1007/bf02289503 · Cited by 257
Cited by
Items 201–300 of 604
Cross-entropy analysis of the information in forensic speaker recognition
Ramos, D., & González-Rodríguez, J. (2008) · Cited by 24
Probabilistic classifiers with high-dimensional data
Kim, K. I., & Simon, R. (2011) · Biostatistics · doi:10.1093/biostatistics/kxq069 · Cited by 23
On the Calibration and Uncertainty of Neural Learning to Rank Models for Conversational Search
Penha, G., & Hauff, C. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.12 · Cited by 43
Bayesian Neural Networks With Maximum Mean Discrepancy Regularization
Pomponi, J., Scardapane, S., & Uncini, A. (2020) · Neurocomputing · doi:10.1016/j.neucom.2021.01.090 · Cited by 25
“To visually show the calibration of these models we used the reliability diagrams [31], [29].”
BackgroundMethodMartyna, A., Sjåstad, K.-E., Zadora, G., & Ramos, D. (2012) · Talanta · doi:10.1016/j.talanta.2012.11.079 · Cited by 23
Stochastic Model of Seasonal Runoff Forecasts
Krzysztofowicz, R., & Watada, L. M. (1986) · Water Resources Research · doi:10.1029/wr022i003p00296 · Cited by 23
A Close Look into the Calibration of Pre-trained Language Models
Chen, Y., Yuan, L., Cui, G., Liu, Z., & Ji, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.75 · Cited by 22
Maity, D., & Ciezobka, J. (2018) · Journal of Petroleum Science and Engineering · doi:10.1016/j.petrol.2018.09.062 · Cited by 22
Pereira, T., Cardoso, S., Guerreiro, M., de Mendonça, A., & Madeira, S. C. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103350 · Cited by 28
“However, several studies [7–10] have shown that some of these probability estimates are not well-calibrated, i.e., they do not reflect the empirical probabilities as the number of predictions goes to infinity [11].”
BackgroundEvaluating Density Forecasting Models
Carney, M., & Cunningham, P. (2006) · Arrow@dit (Dublin Institute of Technology) · Cited by 22
PAC Confidence Sets for Deep Neural Networks via Calibrated PredictionPreprint
Park, S., Bastani, O., Matni, N., & Lee, I. (2019) · International Conference on Learning Representations · doi:10.48550/arxiv.2001.00106 · Cited by 84
“Finally, we build on a long line of work on calibrated prediction , which aims to construct “cali-brated” probabilities (Murphy, 1972; DeGroot & Fienberg, 1983; Platt, 1999; Zadrozny & Elkan, 2001; 2002; Naeini et al., 2015; Kuleshov & Liang, 2015).”
MethodForensic Automatic Speaker Classification in the “Coming Paradigm Shift”
González-Rodríguez, J., & Ramos, D. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74200-5_11 · Cited by 23
“Highly discriminant (or refined [ 13 ]) systems may lead to wrong posterior probabilities if they do not elicit reliable (or calibrated ) confidences [13,14,15].”
BackgroundInfluentialA Characterization for the Spherical Scoring Rule
Jose, V. R. R. (2007) · Theory and Decision · doi:10.1007/s11238-007-9067-x · Cited by 21
“In addition, scores generated from this mechanism can also provide feedback for assessors to help them improve themselves ( De Groot and Fienberg, 1983; Schervish, 1989; Johnstone, 2007).”
BackgroundCalibrate: Interactive Analysis of Probabilistic Model Output
Xenopoulos, P., Rulff, J., Nonato, L. G., Barr, B., & Silva, C. T. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209489 · Cited by 25
“Reliability diagrams are a popular method to assess model calibration [12, 34].”
MethodDynamic Correlation Learning and Regularization for Multi-Label Confidence Calibration
Chen, T., Wang, W., Pu, T., Qin, J., Yang, Z., Liu, J., & Lin, L. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3448248 · Cited by 32
“We use the traditional metrics like adaptive calibration error (ACE) [22], expected calibrated error (ECE) [23], maximum calibration error (MCE) [20] and reliability diagram [24] for performance evaluation.”
MethodA Descriptive Bayesian Approach to Modeling and Calibrating Drivers' En Route Diversion Behavior
Xiong, C., & Zhang, L. (2013) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2013.2270974 · Cited by 20
The Curious Case of Convex Neural Networks
Sivaprasad, S., Singh, A., Manwani, N., & Gandhi, V. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86486-6_45 · Cited by 41
“These diagrams show expected sample accuracy as a function of confidence [9].”
BackgroundMethodTwitter Language Identification Of Similar Languages And Dialects Without Ground Truth
Williams, J. A., & Dagli, C. K. (2017) · Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial) · doi:10.18653/v1/w17-1209 · Cited by 21
Calibrated Prediction with Covariate Shift via Unsupervised Domain AdaptationPreprint
Park, S., Bastani, O., Weimer, J. E., & Lee, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.00343 · Cited by 26
“A forecaster f : for all k ∈ Y and t ∈ [0 , 1] (DeGroot and Fienberg, 1983; Zadrozny and Elkan, 2002).”
BackgroundDawid, A. P. (2005) · Wiley eBooks · doi:10.1002/0471667196.ess2064.pub2 · Cited by 22
Binary Classifier Calibration Using a Bayesian Non-Parametric Approach
Naeini, M. P., Cooper, G. F., & Hauskrecht, M. (2015) · Proceedings of the 2015 SIAM International Conference on Data Mining · doi:10.1137/1.9781611974010.24 · Cited by 19
“Figure 1 shows a hypothetical example of a reliability curve [3, 9], which displays the calibration performance of a prediction method.”
BackgroundA deep analysis on high‐resolution dermoscopic image classification
Pollastri, F., Lara, M. P., Maroñas, J., Bolelli, F., Paredes, R., Ramos, D., & Grana, C. (2021) · IET Computer Vision · doi:10.1049/cvi2.12048 · Cited by 22
“In general, this is achieved by models both able to separate between classes (a property known as discrimination or refinement [46]), and able to assign correct probabilities based on how the data is distributed (calibration).”
Local scale invariance and robustness of proper scoring rules
Bolin, D., & Wallin, J. (2022) · Statistical Science · doi:10.1214/22-sts864 · Cited by 36
“decomposed into a calibration component and a refinement component (Murphy, 1972; DeGroot and Fienberg, 1983).”
BackgroundWang, Z., Luo, Y., Chen, Z., Wang, S., & Huang, Z. H. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611808 · Cited by 31
A Primer on Bayesian Neural Networks: Review and Debates
Arbel, J., Pitas, K., Vladimirova, M., & Fortuin, V. (2026) · Statistical Science · doi:10.1214/24-sts969 · Cited by 90
Matching of catalogues by probabilistic pattern classification
Rohde, D., Gallagher, M. R., Drinkwater, M. J., & Pimbblet, K. A. (2006) · Monthly Notices of the Royal Astronomical Society · doi:10.1111/j.1365-2966.2006.10304.x · Cited by 20
Binary classifier calibration using an ensemble of piecewise linear regression models
Naeini, M. P., & Cooper, G. F. (2017) · Knowledge and Information Systems · doi:10.1007/s10115-017-1133-2 · Cited by 22
“Refinement accounts for the usefulness of the probabilities by favoring those that are either close to 0 or 1 [6,7].”
BackgroundMethodEvaluation of Probabilistic Forecasts: Proper Scoring Rules and MomentsPreprint
Tsyplakov, A. A. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2236605 · Cited by 22
“Similar results for discrete outcome can be found in DeGroot and Fienberg (1983) and Bröcker (2009).”
Characterizing Sources of Uncertainty to Proxy Calibration and Disambiguate Annotator and Data BiasPreprint
Picard, R. W., & Ghandeharioun, A. (2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 21
Classifying stars, galaxies, and AGNs in CLAUDS + HSC-SSP using gradient boosted decision trees
Golob, A., Sawicki, M., Goulding, A. D., & Coupon, J. (2021) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stab719 · Cited by 21
On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations
Schwartz, R., & Stanovsky, G. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.168 · Cited by 35
Bayesian Active Meta-Learning for Reliable and Efficient AI-Based Demodulation
Cohen, K. M., Park, S., Simeone, O., & Shamai, S. S. (2022) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3220035 · Cited by 30
“Two standard means of quantifying the extent to which the perfect calibration is satisfied are reliability diagrams [88] and”
BackgroundVrugt, J. A. (2024) · Water Resources Research · doi:10.1029/2023wr036710 · Cited by 31
Probability Calibration for Knowledge Graph Embedding ModelsPreprint
Tabacof, P., & Costabello, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.10000 · Cited by 36
Distribution-free binary classification: prediction sets, confidence intervals and calibrationPreprint
Gupta, C., Podkopaev, A., & Ramdas, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10564 · Cited by 106
A Close Look into the Calibration of Pre-trained Language Models
Chen, Y., Yuan, L., Cui, G., Liu, Z., & Ji, H. (2022) · arXiv · doi:10.48550/arxiv.2211.00151 · Cited by 18
On the Calibration of Large Language Models and Alignment
Zhu, C., Xu, B., Wang, Q., Zhang, Y., & Mao, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.654 · Cited by 109
Confidence Calibration for Deep Renal Biopsy Immunofluorescence Image Classification
Pollastri, F., Maroñas, J., Bolelli, F., Ligabue, G., Paredes, R., Magistroni, R., & Grana, C. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412685 · Cited by 23
“In theory, to achieve optimal accuracy and calibration we need to train the model by minimizing a proper scoring rule [39], [35].”
BackgroundTran, L. Q., Bonti, A., Chi, L., Abdelrazek, M., & Chen, Y. P. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117393 · Cited by 16
On (assessing) the fairness of risk score models
Petersen, E., Ganz, M., Holm, S., & Feragen, A. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594045 · Cited by 36
“Many metrics have been proposed for assessing calibration performance, including the expected calibration error (ECE) [70], adaptive calibration error (ACE) [71], and metrics based on proper scoring rule decompositions [9, 17, 18, 69].”
BackgroundBayesian Reanalysis of the Challenger O‐Ring Data
Maranzano, C. J., & Krzysztofowicz, R. (2008) · Risk Analysis · doi:10.1111/j.1539-6924.2008.01081.x · Cited by 17
Navidi, S., Bajarunas, K., Chao, M. A., & Hu, C. K. (2025) · eTransportation · doi:10.1016/j.etran.2025.100432 · Cited by 18
Calibration and Uncertainty in Neural Time-to-Event Modeling
Chapfuwa, P., Tao, C., Li, C., Khan, I., Chandross, K. J., Pencina, M. J., Carin, L., & Henao, R. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3029631 · Cited by 16
“…of a predictive model that characterizes the statistical consistency of the predictions relative to the distribution of the observations on a population level, has been studied in forecasting [27], Bayesian analysis [28], and in machine learning, for classification [29] and regression [30] problems.”
BackgroundXu, M., Zhou, Y., Jin, C., de Groot, M., Alexander, D. C., Oxtoby, N. P., & Jacob, J. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3273158 · Cited by 19
Label fusion and training methods for reliable representation of inter-rater uncertainty
Lemay, A., Gros, C., Karthik, E. N., & Cohen‐Adad, J. (2023) · The Journal of Machine Learning for Biomedical Imaging · doi:10.59275/j.melba.2022-db5c · Cited by 37
Eugene: Towards Deep Intelligence as a Service
Yao, S., Jayarajah, K., Misra, A., Abdelzaher, T., Hao, Y., Zhao, Y., Piao, A., Shao, H., Liu, D., Liu, S., et al. (2019) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2019.00162 · Cited by 22
“The calibration of confidence can be visually represented by the reliability diagram [12].”
BackgroundMaros, M. E., Cho, C. G., Junge, A. G., Kämpgen, B., Saase, V., Siegel, F. P., Trinkmann, F., Ganslandt, T., Groden, C., & Wenz, H. (2021) · Scientific Reports · doi:10.1038/s41598-021-85016-9 · Cited by 19
“Hence, the probability output of an ideally calibrated ML classifier would lie on the diagonal line 87,89 .”
BackgroundMethodInfluentialBAYESIAN PROCESSOR OF OUTPUT: A NEW TECHNIQUE FOR PROBABILISTIC WEATHER FORECASTING
Krzysztofowicz, R. (2004) · Cited by 16
PD-Validation: Experience from Banking Practice
Rauhmeier, R. (2011) · Springer eBooks · doi:10.1007/978-3-642-16114-8_15 · Cited by 16
Testing for Calibration Discrepancy of Reported Likelihood Ratios in Forensic Science
Hannig, J., & Iyer, H. K. (2021) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12747 · Cited by 18
“A key publication discussing the accuracy of probability assessments is by DeGroot and Fienberg (1983) where the authors consider the empirical evaluation of probability assessments using the context of weather forecasters for illustrative purposes.”
BackgroundA machine learning approach to detect early signs of startup success
Thirupathi, A. N., Alhanai, T., & Ghassemi, M. M. (2021) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3490354.3494374 · Cited by 15
“Finally, in Figure 2, we illustrate the statistical calibration of the XGBoost model using a Reliability Diagram [7] and a Brier Score of 0.”
MethodInfluentialEvaluating probabilistic classifiers: The triptych
Dimitriadis, T., Gneiting, T., Jordan, A. I., & Vogel, P. (2023) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2023.09.007 · Cited by 29
Sub-Ensembles for Fast Uncertainty Estimation in Neural Networks
Valdenegro-Toro, M. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00445 · Cited by 18
Measuring and Modeling Uncertainty Degree for Monocular Depth Estimation
Xiang, M., Zhang, J., Barnes, N. M., & Dai, Y. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3362357 · Cited by 16
“Expected Calibration Error (ECE) [12] is a bin-based strategy to measure the expectation difference between model confidence and model accuracy.”
BackgroundInfluentialA Machine Learning Approach for Non-Invasive Diagnosis of Metabolic Syndrome
Datta, S., Schraplau, A., Freitas da Cruz, H., Philipp Sachs, J., Mayer, F., & Bottinger, E. (2019) · 2019 IEEE 19th International Conference on Bioinformatics and Bioengineering (BIBE) · doi:10.1109/bibe.2019.00175 · Cited by 15
“There are different ways to check for model calibration, calibration plot usually being the preferred approach [27], [29].”
BackgroundRamos-Castro, D., González-Rodríguez, J., Montero-Asenjo, A., & Ortega-García, J. (2006) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.1109/odyssey.2006.248090 · Cited by 15
“Moreover, (cid:5)(cid:6)(cid:6)(cid:7) includes two different measures: i) the loss in accuracy because of the discrimination capabilities of the system (i. e., a bad refinement [19]) and ii) a penalty to (cid:2)(cid”
BackgroundForecast sufficiency characteristic: Construction and application
Krzysztofowicz, R., & Long, D. (1991) · International Journal of Forecasting · doi:10.1016/0169-2070(91)90031-p · Cited by 15
Comparison Methods for Stochastic Models and Risks
Kiessler, P. C. (2005) · Journal of the American Statistical Association · doi:10.1198/jasa.2005.s17 · Cited by 15
“Also, the work on calibrating probability forecasters (cf. DeGroot and Fienberg 1983) is relevant, as is more detail on elicitation (cf. Kadane and Winkler 1988).”
BackgroundGilg, J., Teepe, T., Herzog, F., Wolters, P., & Rigoll, G. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00478 · Cited by 23
“The result can be visualized in a reliability diagram [8] or averaged to produce the detection ECE-metric [13,21]: The standard ECE was shown to have numerous pathologies.”
MethodPost hoc calibration of medical segmentation models
Rousseau, A.-J., Becker, T., Appeltans, S., Blaschko, M. B., & Valkenborg, D. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-06587-0 · Cited by 28
“The MSE decomposes into a calibration term equal to the CE 2 error and a refinement term [39].”
Non-parametric Bayesian Isotonic Calibration: Fighting Over-Confidence in Binary Classification
Allikivi, M.-L., & Kull, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-46147-8_7 · Cited by 15
“Proper losses are minimized if the calibration method achieves perfectly calibrated probabilities, due to the decomposition into calibration loss and refinement loss [3, 9].”
MethodCalibration, sharpness and the weighting of experts in a linear opinion pool
Hora, S. C., & Kardeş, E. (2015) · Annals of Operations Research · doi:10.1007/s10479-015-1846-0 · Cited by 13
The CCB-ID approach to tree species mapping with airborne imaging spectroscopy
Anderson, C. B. (2018) · PeerJ · doi:10.7717/peerj.5666 · Cited by 16
Variance estimation for Brier Score decomposition
Siegert, S. (2013) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.2228 · Cited by 14
“In fact, any such deviance from p will increase his expected Brier Score, which makes the Brier Score a strictly proper scoring rule (DeGroot and Fienberg, 1983).”
BackgroundClemen, R. T. (1985) · Journal of Forecasting · doi:10.1002/for.3980040403 · Cited by 14
Gold, E. (2014) · White Rose eTheses Online (University of Leeds, The University of Sheffield, University of York) · Cited by 15
Lad, F. (1984) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/35.3.213 · Cited by 14
Auditing Black-Box Models Using Transparent Model Distillation With Side InformationPreprint
Tan, S., Caruana, R., Hooker, G., & Lou, Y. (2017) · arXiv (Cornell University) · Cited by 14
Joe, B., Park, Y.-H., Hamm, J., Shin, I., & Lee, J. (2022) · JMIR Medical Informatics · doi:10.2196/38440 · Cited by 20
“Figure 10 shows the calibration curves [33] that represent the reliability of the prediction probabilities of the input model.”
BackgroundMethodStoppa, F., Bhattacharyya, S., de Austri, R. R., Vreeswijk, P. M., Caron, S., Zaharijaš, G., Bloemen, S., Principe, G., Malyshev, D. V., Vodeb, V., et al. (2023) · Astronomy and Astrophysics · doi:10.1051/0004-6361/202347576 · Cited by 19
Bayesian Reanalysis of the Challenger O-Ring Data
Maranzano, C. J., & Krzysztofowicz, R. (2008) · Risk Analysis · doi:10.1111/j.1365-2966.2008.01081.x · Cited by 13
Managing Behaviour Trust in Grids Using Statistical Methods of Quality Assurance
Papalilo, E., & Freisleben, B. (2007) · International Conference on Information Assurance and Security (IAS) · doi:10.1109/ias.2007.51 · Cited by 13
“This is due to a different calibration of each procedure, as defined in [7].”
A large-scale active learning system for topical categorization on the web
Rajan, S., Yankov, D., Gaffney, S. J., & Ratnaparkhi, A. (2010) · International World Wide Web Conference (WWW) · doi:10.1145/1772690.1772771 · Cited by 13
“Not extrapolated to population Recall Figure 1: Comparison of linear SVM, MaxEnt and stochastic gradient descent on a .ve fold split of the Sensitivity Class 2 training set (cf. Section 5). techniques, however, produce a non-sparse representation which is a signi.cant concern as the feature space that we work with con…”
BackgroundMethodInfluentialKull, M., Perello-Nieto, M., Kängsepp, M., Filho, T. S., Song, H., & Flach, P. (2019) · arXiv · doi:10.48550/arxiv.1910.12656 · Cited by 206
“to visualise calibration (reliability) errors; and [6] discussing proper scoring rules for forecaster evaluation and the decomposition of these loss measures into calibration and refinement losses.”
BackgroundDecision criteria, data fusion and prediction calibration: a Bayesian approach
Krzysztofowicz, R. (2010) · Hydrological Sciences Journal · doi:10.1080/02626667.2010.505894 · Cited by 12
“The notion of calibration of probabilistic predictions is an established concept in Bayesian theory (DeGroot & Fienberg, 1982 , 1983 ; Vardeman & Meeden, 1983 ).”
BackgroundUser decisions, and how these could guide developments in probabilistic forecasting
Rodwell, M. J., Hammond, J., Thornton, S., & Richardson, D. S. (2020) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.3845 · Cited by 12
“The third line in Equation (32) shows the ‘Reliability–Refinement’ decomposition (Degroot and Fienberg, 1983; Rodwell et al. , 2018) of the Brier Score.”
BackgroundMBCT: Tree-Based Feature-Aware Binning for Individual Uncertainty Calibration
Huang, S., Wang, Y., Mou, L., Zhang, H., Zhu, H., Yu, C., & Zheng, B. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512096 · Cited by 24
“A large number of tasks not only concern the classification accuracy, but also expect that the output of the machine learning model reflects the true frequency of an event, such as medical diagnosis [10], meteorological forecasting [3, 5, 7], and computation advertising [21, 22].”
BackgroundRoitberg, A., Peng, K., Schneider, D., Yang, K., Koulakis, M., Martínez, M., & Stiefelhagen, R. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3196410 · Cited by 14
Context-Aware Selective Label Smoothing for Calibrating Sequence Recognition Model
Huang, S., Luo, Y., Zhuang, Z., Yu, J.-G., He, M., & Wang, Y. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475618 · Cited by 12
“Reliability Diagram [5] The reliability diagram is the plot of 𝐴𝑐𝑐 ( 𝐵 ) versus 𝐶𝑜𝑛𝑓 ( 𝐵 ) .”
The Value of a Probability Forecast from Portfolio Theory
Johnstone, D. J. (2007) · Theory and Decision · doi:10.1007/s11238-006-9023-1 · Cited by 12
Building reliable metaclassifiers for text learning
Carbonell, J., Lafferty, J., & Bennett, P. N. (2006) · Cited by 11
Ghosh, S., Liao, Q. V., Ramamurthy, K. N., Navrátil, J., Sattigeri, P., Varshney, K. R., & Zhang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01410 · Cited by 39
“We also include additional diagnostic tools such as reliability diagrams (DeGroot and Fienberg, 1983) and risk-vs-rejection rate curves (El-Yaniv et al., 2010) .”
MethodAre reported likelihood ratios well calibrated?
Hannig, J., Riman, S., Iyer, H. K., & Vallone, P. M. (2019) · Forensic science international. Genetics supplement series · doi:10.1016/j.fsigss.2019.10.094 · Cited by 12
Datta, A., & Swamidass, S. J. (2022) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0010877400003116 · Cited by 13
Comparing Sequential Forecasters
Choe, Y. J., & Ramdas, A. (2023) · Operations Research · doi:10.1287/opre.2021.0792 · Cited by 32
On the Convergence and Calibration of Deep Learning with Differential Privacy
Bu, Z., Wang, H., Dai, Z., & Long, Q. (2021) · Transactions on machine learning research · doi:10.48550/arxiv.2106.07830 · Cited by 51
“In Figure 6, the reliability diagram [20, 43] displays the accuracy as a function of confidence.”
Fernández, A., García, S., Galar, M., Prati, R. C., Krawczyk, B., & Herrera, F. (2018) · Springer eBooks · doi:10.1007/978-3-319-98074-4_3 · Cited by 14
Santis, E. D., Arnò, F., & Rizzi, A. (2022) · Soft Computing · doi:10.1007/s00500-022-07194-6 · Cited by 12
“Specifically, the calibration procedure is experimented on two UCI datasets (Dua and Graff 2019) (ABALONE, ADULT) (Asuncion and Newman 2007) and on a protein contact network datasets for predicting the function of specific proteins (De Santis et al. 2018a).”
MethodA Unifying Theory of Distance from Calibration
Błasiok, J., Gopalan, P., Hu, L., & Nakkiran, P. (2023) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/3564246.3585182 · Cited by 97
Modular Model-Based Bayesian Learning for Uncertainty-Aware and Reliable Deep MIMO Receivers
Raviv, T., Park, S., Simeone, O., & Shlezinger, N. (2023) · IEEE International Conference on Communications workshops · doi:10.1109/iccworkshops57953.2023.10283687 · Cited by 15
“We evaluate two standard measures: (i) reliability diagrams [29] and (ii) expected calibration error (ECE) [10].”
MethodOn Calibration of Scene-Text Recognition Models
Slossberg, R., Anschel, O., Markovitz, A., Litman, R., Aberdam, A., Tsiper, S., Mazor, S., Wu, J., & Manmatha, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25069-9_18 · Cited by 14
Wei, J., Yao, L., & Meng, Q. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.02.013 · Cited by 13
Evaluating probability forecasts in terms of refinement and strictly proper scoring rules
Krämer, W. (2006) · Journal of Forecasting · doi:10.1002/for.976 · Cited by 11
“DeGroot and Fienberg (1983) introduce the refinement ordering among well calibrated probability forecasters.”
BackgroundFeature Weighted Minimum Distance Classifier with Multi-class Confidence Estimation
Rudrapatna, M., & Sowmya, A. (2006) · Lecture notes in computer science · doi:10.1007/11941439_29 · Cited by 11
“Reliability diagrams [26] can be used in binary class problems.”
BackgroundOn the Ordering of Probability ForecastsPreprint
Krämer, W. (2002) · RePEc: Research Papers in Economics · doi:10.17877/de290r-12198 · Cited by 11
“whenever q(aj) > 0. Such forecasters are called “well calibrated” ( DeGroot and Fienberg, 1983 ).”
BackgroundInfluentialUnsupervised Calibration under Covariate ShiftPreprint
Pampari, A., & Ermon, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16405 · Cited by 24
“Reliability diagrams are visual representation of model calibration (DeGroot & Fienberg, 1983; Niculescu-Mizil & Caruana, 2005) as shown in Figure.”
MethodKoller, C., Kauermann, G., & Zhu, X. X. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3336357 · Cited by 21
“It can be visualized in a so-called reliability diagram [61], [62], which displays the accuracy versus the corresponding confidence in a single plot, averaged over predefined intervals.”
BackgroundInterplay between Bayesian neural networks and deep learning: A survey
Chen, Y., Yu, S. S., Li, Z., Eshraghian, J. K., & Lim, C. P. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114438 · Cited by 12
Benchmarking Scalable Predictive Uncertainty in Text Classification
Landeghem, J. V., Blaschko, M. B., Anckaert, B., & Moens, M. (2022) · IEEE Access · doi:10.1109/access.2022.3168734 · Cited by 22
“Definition 1 (Perfect Calibration [86], [87]): Calibration is a property of an empirical estimator f , which states that on finite-sample data it converges to a solution where the scoring function reflects the probability v of being correct.”
BackgroundPACE: Learning Effective Task Decomposition for Human-in-the-loop Healthcare Delivery
Zheng, K., Chen, G., Herschel, M., Ngiam, K. Y., Ooi, B. C., & Gao, J. (2021) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3448016.3457281 · Cited by 22
Metadata
- Authors
- DeGroot, M. H., & Fienberg, S. E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- The Statistician
- Publisher
- JSTOR
- Published
- 1983-03
- Volume
- 32
- Issue
- 1/2
- Pages
- 12
- ISSN
- 0039-0526 (ISSN-L)
- OpenAlex
- W1599263113
- Cited by
- 604
- References
- 18
- Citations by year
- 2026: 14 · 2025: 38 · 2024: 32 · 2023: 51 · 2022: 42 · 2021: 52 · 2020: 48 · 2019: 36 · 2018: 26 · 2017: 20 · 2016: 16 · 2015: 21 · 2014: 12 · 2013: 17 · 2012: 14
- FWCI
- 5.36
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the Royal Statistical Society Series D (The Statistician) (publishedVersion)
- Defense Technical Information Center (DTIC) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- forecast accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.