The Comparison and Evaluation of Forecasters
The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Morris H. DeGroot, Stephen E. Fienberg, The Comparison and Evaluation of Forecasters, Journal of the Royal Statistical Society. Series D (The Statistician), Vol. 32, No. 1/2, Proceedings of the 1982 I.O.S. Annual Conference on Practical Bayesian Statistics (Mar. - Jun., 1983), pp. 12-22
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DeGroot and Fienberg (1983) showed that these two terms make up proper scoring rules in general, and that probabilistic predictions should minimise refinement loss subject to calibration.”
From Classifier calibration: a survey on how to assess and improve predicted class probabilities (2023) · cites this paper as DeGroot and Fienberg (1983)
Citations
Cites
Discrete Multivariate Analysis: Theory and Practice.
Beniger, J. R., Bishop, Y. M. M., Feinberg, S. E., Holland, P. W., Light, R. J., & Mosteller, F. (1975) · Contemporary Sociology · doi:10.2307/2063625 · Cited by 5,108
Optimal Statistical Decisions.
Siegmund, D., & DeGroot, M. H. (1972) · Journal of the American Statistical Association · doi:10.2307/2284675 · Cited by 3,386
Self-citationThe Analysis of Cross-Classified Categorical Data
Drane, J. W. (1979) · Technometrics · doi:10.1080/00401706.1979.10489789 · Cited by 1,941
Self-citationElicitation of Personal Probabilities and Expectations
Savage, L. J. (1971) · Journal of the American Statistical Association · doi:10.1080/01621459.1971.10482346 · Cited by 1,226
Theory of Games and Statistical Decisions.
Vajda, S., Blackwell, D., & Girshick, M. A. (1955) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2342525 · Cited by 1,032
Blackwell, D. (1951) · Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability · doi:10.1525/9780520411586-009 · Cited by 1,024
Dawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.1080/01621459.1982.10477856 · Cited by 625
Reliability of Subjective Probability Forecasts of Precipitation and Temperature
Murphy, A. H., & Winkler, R. L. (1977) · Applied Statistics · doi:10.2307/2346866 · Cited by 386
On the Reconciliation of Probability Assessments
Lindley, D. V., Tversky, A., & Brown, R. V. (1979) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2345078 · Cited by 340
Admissible Probability Measurement Procedures
Shuford, E. H., Albert, A., & Massengill, H. E. (1966) · Psychometrika · doi:10.1007/bf02289503 · Cited by 257
Cited by
Items 401–500 of 604
Valsecchi, I., & Valsecchi, I. (2008) · AgEcon Search (University of Minnesota, USA) · doi:10.22004/ag.econ.36756 · Cited by 4
On the Dark Side of Calibration for Modern Neural Networks.Preprint
Singh, A. K., Bay, A., Sengupta, B., & Mirabile, A. (2021) · arXiv (Cornell University) · Cited by 7
Micro𝕊plit: Semantic Unmixing of Fluorescent Microscopy DataPreprint
Ashesh, A., Carrara, F., Zubarev, I., Galinova, V., Croft, M., Pezzotti, M., Gong, D., Casagrande, F., Colombo, E., Giussani, S., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.02.10.637323 · Cited by 4
Strategic bidding in pay-as-bid power reserve markets: A machine learning approach
Koechlin, G., Bovera, F., & Secchi, P. (2025) · Energy Economics · doi:10.1016/j.eneco.2025.108780 · Cited by 3
From Robustness to Improved Generalization and Calibration in Pre-trained Language Models
Jukić, J., & Šnajder, J. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00739 · Cited by 8
11th German Conference on Chemoinformatics (GCC 2015): Fulda, Germany. 8–10 November 2015
Fechner, U., de Graaf, C., Torda, A. E., Güssregen, S., Evers, A., Matter, H., Heßler, G., Richmond, N. J., Schmidtke, P., Segler, M., et al. (2016) · Journal of Cheminformatics · doi:10.1186/s13321-016-0119-5 · Cited by 3
Comparing the accuracy of default predictions in the rating industry for different sets of obligors
Krämer, W., & Neumärker, S. (2016) · Economics Letters · doi:10.1016/j.econlet.2016.05.021 · Cited by 3
“3 For the special case where A and B are rating the same set of debtors, DeGroot and Fienberg (1983) suggest the concept of refinement to discriminate between the two.”
BackgroundDecision Making under Uncertainty with Convolutional Deep Gaussian Processes
Jain, D. C., Anumasa, S., & Srijith, P. K. (2020) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3371158.3371383 · Cited by 3
“Reliability diagram [5, 22] graphically represent the calibration confidence of the model.”
MethodUncertainty in Predictive Process Monitoring
Portolani, P., Brusaferri, A., Ballarino, A., & Matteucci, M. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-08974-9_44 · Cited by 3
Dawkins, H., & Nejadgholi, I. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.59 · Cited by 3
“Typically, the error is composed of a sum of observed errors across the certainty landscape, visualized using a reliability diagram (DeGroot and Fienberg, 1983; Niculescu-Mizil and Caruana, 2005).”
MethodGerevini, A., Maroldi, R., Olivato, M., Putelli, L., & Serina, I. (2023) · IEEE Access · doi:10.1109/access.2023.3296260 · Cited by 3
“The most important technique for making a machine learning model a probabilistic classifier [41], [42] is calibration [11], [43], [44].”
BackgroundMethodFarkhari, H., Viana, J., Kahvazadeh, S., Sebastião, P. J. A., Jiménez, V. P. G., & Dinis, R. (2024) · IEEE Access · doi:10.1109/access.2024.3504729 · Cited by 6
Generic Conditions for Forecast Dominance
Krüger, F., & Ziegel, J. F. (2020) · doi:10.6084/m9.figshare.11968584 · Cited by 3
Improving the Reliability of Decision Tree and Naive Bayes Learners
Lindsay, D. G., & Cox, S. S. E. (2005) · Proceedings · doi:10.1109/icdm.2004.10037 · Cited by 3
“The problem of defining effective scoring rules or loss functions for evaluating probability forecasts has been considered in depth [ 4 ].”
BackgroundMethodInfluentialComparative Forecast Evaluation: Graphical Gaussian Models and Sufficiency Relations
Callies, U. (2000) · Monthly Weather Review · doi:10.1175/1520-0493(2000)128<1912:cfeggm>2.0.co;2 · Cited by 3
Obtaining Accurate Probabilities Using Classifier Calibration.
Naeini, M. P. (2016) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 4
Carriquiry, A. L., Reid, N. M., & Slavković, A. (2019) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-030718-105334 · Cited by 4
“But while I did see some advocacy of the subjectivist approach, (DeGroot & Fienberg 1983, Fienberg & Kadane 1983, Fienberg & Schervish 1986), I also saw his promotion of objective priors (Tiao & Fienberg 1969, his earliest Bayesian publication), his appeal to frequentist risk functions for…”
Improving Neural Network Classification Training
Rimer, M. E. (2007) · ScholarsArchive (Brigham Young University) · Cited by 3
Obtaining Accurate Probabilistic Causal Inference by Post-Processing CalibrationPreprint
Jabbari, F., Naeini, M. P., & Cooper, G. F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.08626 · Cited by 6
“Informally, we say that probabilities are well-calibrated if events predicted to occur with probability p do occur about p fraction of the time, for all p [4].”
BackgroundMethodSeeing in the dark with recurrent convolutional neural networksPreprint
Hartmann, T. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.08537 · Cited by 3
“We determine the models calibration using reliability plots [6, 23]; these show the calculated fraction of positives for a range of predicted probability intervals.”
MethodEsuli, A., Moreo, A., & Sebastiani, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.11459 · Cited by 3
“2Posterior probabilities Pr(c |d ) are said to be well calibrated when lim|S |→∞ |{d∈c | Pr(c |d )=x }| |{d∈S | Pr(c |d )=x }| = x [9].”
Calibration tests beyond classification
Widmann, D., Lindsten, F., & Zachariah, D. (2022) · arXiv · doi:10.48550/arxiv.2210.13355 · Cited by 3
“One such statistical guarantee is calibration, which has been studied extensively in metereological and statistical literature (DeGroot & Fienberg, 1983; Murphy & Winkler, 1977).”
BackgroundLearning Augmentation Distributions using Transformed Risk Minimization
Chatzipantazis, E., Pertigkiozoglou, S., Daniilidis, K., & Dobriban, E. (2021) · Transactions on machine learning research · doi:10.48550/arxiv.2111.08190 · Cited by 15
“As discussed in (Cox, 1958; DeGroot & Fienberg, 1983; Gneiting et al., 2007; Guo et al., 2017), given a random datapoint ( X, Y ) drawn from the data distribution D , a model h is perfectly calibrated if A commonly used metric for quantifying the calibration of a model is the Expected Calibration…”
MethodKermut: Composite kernel regression for protein variant effectsPreprint
Groth, P. M., Kerrn, M. H., Olsen, L., Salomon, J., & Boomsma, W. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.05.28.596219 · Cited by 14
“First, we perform a confidence interval-based calibration analysis [41], resulting in calibration curves which in the classification setting are known as reliability diagrams [68].”
MethodH-Calibration: Rethinking Classifier Recalibration With Probabilistic Error-Bounded Objective
Huang, W., Cao, G., Xia, J., Chen, J., Wang, H., & Zhang, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3582796 · Cited by 8
“…in various fields, including healthcare diagnosis [5], [23], [36], [37], [38], [39], [40], [41], [42], [43], meteorological forecasting [2], [3], [44], [45], [46], economics analysis [47], [48], [49], natural language processing [50], [51], [52], fairness studies [53], [54], [55] and many…”
On assessing the relative performance of default predictions
Krämer, W. (2017) · Journal of Forecasting · doi:10.1002/for.2479 · Cited by 2
Uncertainty Estimation in Instance Segmentation with Star-convex Shapes
Siddiqui, Q. M. K., Starke, S., & Steinbach, P. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00145 · Cited by 3
“If a model is perfectly calibrated, the expected accuracy and certainty score should be equal for each bin [3, 29].”
BackgroundScoring rules and performance, new analysis of expert judgment data
Nane, G. F., & Cooke, R. M. (2024) · Futures & Foresight Science · doi:10.1002/ffo2.189 · Cited by 2
Liang, J., Liu, D., Feng, L., & Huang, K. (2025) · Remote Sensing · doi:10.3390/rs17101747 · Cited by 3
Plasma state monitoring and disruption characterization using multimodal VAEs
Poels, Y., Pau, A., Donner, C., Romanelli, G., Sauter, O., Venturini, C., Menkovski, V., the TCV Team, & the WPTE Team (2025) · Nuclear Fusion · doi:10.1088/1741-4326/adf121 · Cited by 2
“We denote the real disruption rate as D rate and describe the deviation to D risk using the expected calibration error (ECE) [73, 74]: with N denoting the total number of samples.”
LEARNING PATIENT-SPECIFIC MODELS FROM CLINICAL DATA
Visweswaran, S. (2007) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 2
George, E. I. (2013) · CHANCE · doi:10.1080/09332480.2013.868751 · Cited by 2
Residual Autocorrelation in Probabilistic Model-Based Diagnostics
Schwall, M. L., & Gerdes, J. C. (2005) · doi:10.1115/imece2005-82165 · Cited by 2
Wang, H., Guo, X., Wei, Q., & Chen, G. (2011) · Journal of the Association for Information Systems · Cited by 2
Koźniewski, M., Cypko, M. A., & Drużdżel, M. J. (2016) · Advances in computer science research · Cited by 2
Genetic algorithms for cross-calibration of categorical data
Aboukhamseen, S. M., & M’Hallah, R. (2017) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1493599080 · Cited by 2
“Specifically, the GA fitness construct, which is based on methods developed in probability forecasting theory (DeGroot and Fienberg, 1983; Lichtenstein et al., 1982; Gneiting and Katzfuss, 2014), ensures that the generated forecasts are valid and that they are the best among all forecasts in their…”
MethodOn Probability Calibration of Recurrent Text Recognition Network
Zhu, X., Wang, J., Hong, Z., Guo, J., & Xiao, J. Z. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-36718-3_36 · Cited by 2
Active Bayesian Assessment of Black-Box Classifiers
Ji, D., Logan, R. L., Smyth, P., & Steyvers, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.16968 · Cited by 20
“Reliability diagrams are a widely used tool for visually diagnosing model calibration (DeGroot and Fienberg 1983; Niculescu-Mizil and Caruana 2005).”
Model Calibration in Dense Classification with Adaptive Label Perturbation
Liu, J., Ye, C., Wang, S., Cui, R., Zhang, J., Zhang, K., & Barnes, N. M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00114 · Cited by 5
“Brier loss [4, 10], confidence penalty [52] and focal loss [45, 17].”
BackgroundBoldness-Recalibration for Binary Event Predictions
Guthrie, A. P., & Franck, C. T. (2024) · The American Statistician · doi:10.1080/00031305.2024.2339266 · Cited by 3
“Probability calibration is well studied within the fields of statistics, meteorology, psychology, machine learning, and others (Bross 1953; Murphy and Winkler 1977; Dawid 1982; DeGroot and Fienberg 1983; Gonzalez and Wu 1999; Guo et al. 2017).”
Rethinking Evaluation Metric for Probability Estimation Models Using Esports Data
Choi, E., Kim, J.-Y., & Lee, W.-K. (2023) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom60117.2023.00374 · Cited by 3
“In recent studies, several approaches such as the reliability diagram [15], Expected Calibration Error ( ECE ), and Maximum Calibration Error ( MCE ) [8] have been proposed to measure the calibration performance of a model.”
BackgroundCalibrating Bayesian Learning via Regularization, Confidence Minimization, and Selective Inference
Huang, J., Park, S., & Simeone, O. (2025) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2025.3629292 · Cited by 2
“The extent to which condition (3) is satisfied is typically evaluated via the reliability diagram [36] and the expected calibration error (ECE) [25].”
Skill Scores and modified Lorenz domination in default forecasts
Krämer, W., & Neumärker, S. (2019) · Economics Letters · doi:10.1016/j.econlet.2019.05.006 · Cited by 2
Refinement revisited with connections to Bayes error, conditional entropy and calibrated classifiersPreprint
Masnadi-Shirazi, H. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1303.2517 · Cited by 2
Evaluation of Probabilities: A Level Playing Field?
Winkler, R. L. (1999) · Springer eBooks · doi:10.1007/978-1-4615-5089-1_9 · Cited by 2
Calibration and refinement for classification trees
Fienberg, S. E., & Kim, S. (1998) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(97)00195-x · Cited by 2
Self-citationBAYESIAN PROCESSOR OF OUTPUT FOR PROBABILISTIC FORECASTING OF PRECIPITATION OCCURRENCE
Maranzano, C. J., & Krzysztofowicz, R. (2004) · Cited by 2
Using Kullback-Leibler Divergence to Analyze the Performance of Collaborative Positioning
Nounagnon, J. (2016) · VTechWorks (Virginia Tech) · Cited by 2
Reliable Semi-Supervised Learning when Labels are Missing at RandomPreprint
Liu, X., Zachariah, D., Wågberg, J., & Schön, T. B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.10947 · Cited by 6
Spiegelhalter, D. J., Franklin, R. C. G., & Bull, K. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1304.1529 · Cited by 4
“There has been extensive work on decompositions of scoring rules into terms that reflect different aspects of an assessor's skill, see, for example, Yates {1982) and DeGroot and Fienberg (1983 ).”
BackgroundInformation-theoretical comparison of evidence evaluation methods for score-based biometric systems
Ramos, D., González-Rodríguez, J., & Fiérrez, J. (2008) · Biblos-e Archivo (Universidad Autónoma de Madrid) · Cited by 2
On the Calibration and Uncertainty of Neural Learning to Rank ModelsPreprint
Penha, G., & Hauff, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.04356 · Cited by 2
“It measures the difference between the observed reliability curve (DeGroot and Fienberg, 1983) and the ideal one4.”
BackgroundAstashkina, E., Momot, R., & Salikhov, M. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3260826 · Cited by 3
Die Bewertung und der Vergleich von Kreditausfall-Prognosen
Krämer, W. (2003) · Credit and Capital Markets – Kredit und Kapital · doi:10.3790/ccm.36.3.395 · Cited by 2
Improved Predictive Uncertainty using Corruption-based Calibration.Preprint
Salvador, T., Voleti, V., Iannantuono, A., & Oberman, A. M. (2021) · arXiv (Cornell University) · Cited by 7
Specialists Outperform Generalists in Ensemble ClassificationPreprint
Meyen, S., Göppert, F., Alber, H., Ulrike Von Luxburg, A., & Franz, V. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.04381 · Cited by 3
“We build on the setting of probability elicitation (DeGroot and Fienberg, 1983, Masnadi-Shirazi, 2013) and ask the question: Which possible ensemble accuracies can be achieved by a set of individual classifiers with known accuracies, and which kind of individual classifiers produce the best- and…”
BackgroundMethodOn Deep Neural Network Calibration by Regularization and its Impact on RefinementPreprint
Singh, A. K., Bay, A., Sengupta, B., & Mirabile, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09385 · Cited by 4
Creating Well Calibrated and Refined Win Prediction ModelsPreprint
Brewer, G., Demediuk, S., Drachen, A., Block, F., & Jackson, T. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4054211 · Cited by 3
Modeling Machine Learning: A Cognitive Economic Approach
Caplin, A., Martín, D., & Marx, P. (2022) · National Bureau of Economic Research · doi:10.3386/w30600 · Cited by 5
Tirhekar, H. M., Yadav, P., & Bajaj, C. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.12.732003 · Cited by 2
“Third, are posterior probabilities reliable under patient-level refitting, or do they express mathematical confidence without empirical stability [26, 27, 28, 29]?”
A Cheat Sheet for Bayesian Prediction
Clarke, B. S., & Yao, Y. (2025) · Statistical Science · doi:10.1214/24-sts951 · Cited by 3
On Calibration in Multi-Distribution Learning
Verma, R., Fischer, V., & Nalisnick, E. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732060 · Cited by 5
Salience, legitimacy, and credibility of two end-to-end forensic single cell probabilistic systems
Bhembe, Q., Lun, D. S., Duffy, K. R., & Grgicak, C. M. (2025) · Forensic Science International Genetics · doi:10.1016/j.fsigen.2025.103369 · Cited by 1
Human-centered method estimating safe width of inland waterways
Медведєв, Є., Muzylyov, D., & Montewka, J. (2026) · Ocean Engineering · doi:10.1016/j.oceaneng.2026.124448 · Cited by 1
Žnidarič, L., Juričić, Đ., Morel, B., & Gradišar, Ž. (2026) · Applied Energy · doi:10.1016/j.apenergy.2026.127889 · Cited by 1
Calibrating Margin-Based Classifier Scores into Polychotomous Probabilities
Gebel, M., & Weihs, C. (2008) · Studies in classification, data analysis, and knowledge organization · doi:10.1007/978-3-540-78246-9_4 · Cited by 1
Qualitätsvergleiche bei Kreditausfallprognosen
Krämer, W. (2012) · Physica-Verlag eBooks · doi:10.1007/978-3-7908-2797-2_23 · Cited by 1
Multimodal classification with deformable part-based models for urban cartography
Randrianarivo, H., Saux, B. L., & Ferecatu, M. (2014) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss.2014.6946392 · Cited by 1
“Calibration The calibration of the classifier is evaluated with the reliability diagram of [11].”
MethodRainer, T. H., Peter Sollich, A., Piotrowski, T., Coolen, A. C. C., Cheng, H. W. B., & Graham, C. A. (2012) · Emergency Medicine Journal · doi:10.1136/emermed-2011-200996 · Cited by 1
Density estimation-based document categorization using von Mises-Fisher kernels
Skabar, A. A., & Memon, S. R. (2010) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2010.5596595 · Cited by 1
Bayesian decision-makers reaching consensus using expert information
Garisch, I. (2009) · Journal for New Generation Sciences · Cited by 1
On using focal loss for neural network calibration
Mukhoti, J., Kulharia, Sanyal, A., Golodetz, S., Torr, P. H. S., & Dokania, P. (2020) · Oxford University Research Archive (ORA) (University of Oxford) · Cited by 1
A study on the calibration of fingerprint classifiers
Peralta, D., Tang, L., Lippeveld, M., & Saeys, Y. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671708 · Cited by 1
“The error on the calibration is the expected difference between the classifier confidence and the true probability, the Expected Calibration Error (ECE) [22]: (3) This quantity can be approximated using a histogram binning approach.”
MethodPartial Orderings of Default Predictions
Krämer, W., & Posch, P. N. (2019) · Studies in classification, data analysis, and knowledge organization · doi:10.1007/978-3-030-25147-5_12 · Cited by 1
Privacy-Preserving Inference on the Ratio of Two Gaussians Using Sums
Miao, J., & Li, Y. P. (2022) · Journal of Data Science · doi:10.6339/22-jds1050 · Cited by 1
Design and Evaluation of Object Classifiers for Probabilistic Decision-Making in Autonomous Systems
Ullah, H., Fan, W., & Wongpiromsarn, T. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812171 · Cited by 1
NCTV: Neural Clamping Toolkit and Visualization for Neural Network Calibration
Hsiung, L., Tang, Y.-C., Chen, P., & Ho, T.-Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.27074 · Cited by 1
You can have your ensemble and run it too – Deep Ensembles Spread Over Time
Meding, I., Bodin, A., Tonderski, A., Johnander, J., Petersson, C., & Svensson, L. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00434 · Cited by 2
“Figure 5 illustrates how various augmentations affect accuracy, Brier reliability [31], [32], and mean entropy of models in sequence settings at different intensity levels.”
BackgroundImproving GNN Calibration with Discriminative Ability: Insights and Strategies
Fang, Y., Li, X., Chen, Q., & Wang, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i11.29082 · Cited by 1
Injury Probability Prediction Modeling Using Decision-Tree-Based Machine Learning Models
Miyazaki, T., Takahashi, K., MIYAZAKI, Y., Kitamura, K., & Sato, F. (2024) · International Journal of Automotive Engineering · doi:10.20485/jsaeijae.15.2_66 · Cited by 1
Evaluating and Improving Probability Assessment in an Ambiguous, Sequential Environment
Goodie, A. S., Meisel, M. K., Ceren, R., Hall, D. B., & Doshi, P. (2015) · Current Psychology · doi:10.1007/s12144-015-9335-9 · Cited by 1
“The Brier score is widely utilized to incentivize probability estimates (DeGroot and Fienberg 1983).”
MethodTrinary tools for continuously valued binary classifiers
Gleicher, M. L., Yu, X., & Chen, Y. (2022) · Visual Informatics · doi:10.1016/j.visinf.2022.04.002 · Cited by 1
“Reliability or calibration curves are a standard approach for assessing these [22, 23, 41].”
BackgroundODIN TS: A Tool for the Black-Box Evaluation of Time Series Analytics
Zangrando, N., Torres, R. N., Milani, F., & Fraternali, P. (2022) · International Conference on Time Series and Forecasting (ITISE) · doi:10.3390/engproc2022018004 · Cited by 1
Visualization for Machine Learning
Xenopoulos, P., Nonato, L. G., & Silva, C. T. (2022) · SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI) · doi:10.1109/sibgrapi55357.2022.9991759 · Cited by 1
“Reliability diagrams are the most common way to analyze model calibration and have foundations in statistical forecasting literature from the 60s and 70s [9]–[12].”
BackgroundWilliams, J. R., & Pettigrew, R. (2023) · The British Journal for the Philosophy of Science · doi:10.1086/725097 · Cited by 1
Machine learning in health informatics: making better use of domain experts
Brodley, C. E., & Wallace, B. (2012) · Cited by 1
Model Integration in Data Mining: From Local to Global Decisions
Sanjuán, A. B. (2012) · RiuNet (Universitat Politècnica de València) · doi:10.4995/thesis/10251/16964 · Cited by 1
On Calibrated Predictions for Auction Selection MechanismsPreprint
McMahan, H. B., & Muralidharan, O. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1211.3955 · Cited by 1
“Calibration is particularly important when comparing predictors, since two sets of calibrated predictions can be fairly evaluated by how concentrated they are on observed outcomes [7, 12, 11].”
BackgroundComparative regression analysis coupled with rational verification for forecasting night cooling
Michaelides, S. C. (1991) · Applied Stochastic Models and Data Analysis · doi:10.1002/asm.3150070303 · Cited by 1
Statistischer Qualitätsvergleich von Kreditausfallprognosen
Bücker, M., & Krämer, W. (2009) · Technische Universität Dortmund Eldorado (Technische Universität Dortmund) · doi:10.17877/de290r-14434 · Cited by 1
Exploratory analysis with imprecise queries
Qarabaqi, B. (2015) · Digital Repository Service (Northeastern University) · doi:10.17760/d20197911 · Cited by 1
Masnadi-Shirazi, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.02578 · Cited by 1
“(23) Finally, the loss is said to be proper and the predictor calibrated (DeGroot and Fienberg, 1983; Platt, 2000; Niculescu-Mizil and Caruana, 2005; Gneiting and Raftery, 2007).”
BackgroundMethodInfluentialCombining Forecasts Using Ensemble LearningPreprint
Masnadi-Shirazi, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02430 · Cited by 1
“This means that if the distribution of the forecasters predictions PH(η) is more concentrated around 0 and 1, thus increasing the PH(η)J(η) term, then SRefinement will increase and the forecaster is said to be more refined (DeGroot and Fienberg, 1983).”
BackgroundMethodInfluentialGroup calibration is a byproduct of unconstrained learning.Preprint
Liu, L. T., Simchowitz, M., & Hardt, M. (2018) · arXiv (Cornell University) · Cited by 3
Calibrated Top-1 Uncertainty estimates for classification by score based modelsPreprint
Oberman, A. M., Finlay, C., Iannantuono, A., & Salvador, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.09215 · Cited by 3
AlignMix: Improving representations by interpolating aligned featuresPreprint
Venkataramanan, S., Ewa Kijak, C., Amsaleg, L., & Avrithis, Y. (2021) · arXiv (Cornell University) · Cited by 1
Cooke, R. M. (1985) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)60220-9 · Cited by 1
Dawid, A. P. (2014) · Wiley eBooks · doi:10.1002/9781118445112.stat02978 · Cited by 2
PD-Validation — Experience from Banking Practice
Rauhmeier, R. (2006) · Springer eBooks · doi:10.1007/3-540-33087-9_14 · Cited by 1
The Promise of Diversity: Distribution-based Hydrologic Model Evaluation and DiagnosticsPreprint
Vrugt, J. A. (2023) · doi:10.22541/essoar.168056799.99219343/v1 · Cited by 1
Metadata
- Authors
- DeGroot, M. H., & Fienberg, S. E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- The Statistician
- Publisher
- JSTOR
- Published
- 1983-03
- Volume
- 32
- Issue
- 1/2
- Pages
- 12
- ISSN
- 0039-0526 (ISSN-L)
- OpenAlex
- W1599263113
- Cited by
- 604
- References
- 18
- Citations by year
- 2026: 14 · 2025: 38 · 2024: 32 · 2023: 51 · 2022: 42 · 2021: 52 · 2020: 48 · 2019: 36 · 2018: 26 · 2017: 20 · 2016: 16 · 2015: 21 · 2014: 12 · 2013: 17 · 2012: 14
- FWCI
- 5.36
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the Royal Statistical Society Series D (The Statistician) (publishedVersion)
- Defense Technical Information Center (DTIC) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- forecast accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.