The Comparison and Evaluation of Forecasters
The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Morris H. DeGroot, Stephen E. Fienberg, The Comparison and Evaluation of Forecasters, Journal of the Royal Statistical Society. Series D (The Statistician), Vol. 32, No. 1/2, Proceedings of the 1982 I.O.S. Annual Conference on Practical Bayesian Statistics (Mar. - Jun., 1983), pp. 12-22
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DeGroot and Fienberg (1983) showed that these two terms make up proper scoring rules in general, and that probabilistic predictions should minimise refinement loss subject to calibration.”
From Classifier calibration: a survey on how to assess and improve predicted class probabilities (2023) · cites this paper as DeGroot and Fienberg (1983)
Citations
Cites
Discrete Multivariate Analysis: Theory and Practice.
Beniger, J. R., Bishop, Y. M. M., Feinberg, S. E., Holland, P. W., Light, R. J., & Mosteller, F. (1975) · Contemporary Sociology · doi:10.2307/2063625 · Cited by 5,108
Optimal Statistical Decisions.
Siegmund, D., & DeGroot, M. H. (1972) · Journal of the American Statistical Association · doi:10.2307/2284675 · Cited by 3,386
Self-citationThe Analysis of Cross-Classified Categorical Data
Drane, J. W. (1979) · Technometrics · doi:10.1080/00401706.1979.10489789 · Cited by 1,941
Self-citationElicitation of Personal Probabilities and Expectations
Savage, L. J. (1971) · Journal of the American Statistical Association · doi:10.1080/01621459.1971.10482346 · Cited by 1,226
Theory of Games and Statistical Decisions.
Vajda, S., Blackwell, D., & Girshick, M. A. (1955) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2342525 · Cited by 1,032
Blackwell, D. (1951) · Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability · doi:10.1525/9780520411586-009 · Cited by 1,024
Dawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.1080/01621459.1982.10477856 · Cited by 625
Reliability of Subjective Probability Forecasts of Precipitation and Temperature
Murphy, A. H., & Winkler, R. L. (1977) · Applied Statistics · doi:10.2307/2346866 · Cited by 386
On the Reconciliation of Probability Assessments
Lindley, D. V., Tversky, A., & Brown, R. V. (1979) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2345078 · Cited by 340
Admissible Probability Measurement Procedures
Shuford, E. H., Albert, A., & Massengill, H. E. (1966) · Psychometrika · doi:10.1007/bf02289503 · Cited by 257
Cited by
Items 101–200 of 604
Ma, S., Wang, X., Lei, Y., Shi, C., Yin, M., & Ma, X. (2024) · arXiv · doi:10.48550/arxiv.2403.09552 · Cited by 56
Classifying Variants of Undetermined Significance in BRCA2 with Protein Likelihood Ratios
Karchin, R., Agarwal, M. K., Šali, A., Couch, F. J., & Beattie, M. S. (2008) · Cancer Informatics · doi:10.4137/cin.s618 · Cited by 55
“The domainspecific sequences were used as input to the SAM-T2K iterative multiple sequence alignmentbuilding algorithm [29].”
MethodImplications of non-stationarity on predictive modeling using EHRs
Jung, K., & Shah, N. H. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.10.006 · Cited by 63
Towards increased trustworthiness of deep learning segmentation methods on cardiac MRI
Sander, J., de Vos, B. D., Wolterink, J. M., & Išgum, I. (2019) · SPIE Medical Imaging: Image Processing · doi:10.1117/12.2511699 · Cited by 63
Building cost functions minimizing to some summary statistics
Saerens, M. (2000) · IEEE Transactions on Neural Networks · doi:10.1109/72.883416 · Cited by 57
Calibrating and Combining Precipitation Probability Forecasts
Clemen, R. T., & Winkler, R. L. (1987) · Springer eBooks · doi:10.1007/978-1-4613-1885-9_10 · Cited by 55
On Calibration and Out-of-domain GeneralizationPreprint
Wald, Y., Feder, A., Greenfeld, D., & Shalit, U. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.10395 · Cited by 99
“We will assess model calibration by the Expected Calibration Error (ECE) of the calibration curve [7], which is the average deviation between model accuracy and model confidence.”
MethodDeGroot, M. H., & Mortera, J. (1991) · Management Science · doi:10.1287/mnsc.37.5.546 · Cited by 56
Self-citationDawid, A. P. (2015) · Synthese · doi:10.1007/s11229-015-0953-4 · Cited by 52
“These two forecasters, while both well-calibrated, perform very differently in terms of their ability to discriminate between different days—Forecaster B is more refined (DeGroot and Fienberg 1983) than Forecaster A.”
A Survey on Learning to Reject
Zhang, X.-Y., Xie, G.-S., Li, X., Mei, T., & Liu, C. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3238024 · Cited by 86
“Reliability diagrams [50], [200] are proposed as a visual representation of confidence calibration, which plots expected accuracy as a function of confidence, as shown in Fig.”
MethodOn using expert opinion in ecological analyses: a frequentist approach
Lele, S. R., & Allen, K. L. (2006) · Environmetrics · doi:10.1002/env.786 · Cited by 56
“(2) The papers by French (1980), Dawid (1982), DeGroot and Fienberg (1982) among others do explicitly try to study the concept of calibration of an expert.”
BackgroundKessler, M. R. (2002) · The Botanical Review · doi:10.1663/0006-8101(2002)068[0100:epaeco]2.0.co;2 · Cited by 50
Hematoma Expansion Context Guided Intracranial Hemorrhage Segmentation and Uncertainty Estimation
Li, X., Luo, G., Wang, W., Wang, K., Gao, Y., & Li, S. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3103850 · Cited by 63
“Specifically, we applied reliability diagrams [37], [38] as a visual assessment tool and further calculated Expected Calibration Error (ECE) [39] as a summary statistic of calibration.”
MethodProbabilistic Novelty Detection With Support Vector Machines
Clifton, L., Clifton, D. A., Zhang, Y., Watkinson, P., Tarassenko, L., & Yin, H. (2014) · IEEE Transactions on Reliability · doi:10.1109/tr.2014.2315911 · Cited by 50
“For different data sets, the suitability of a calibration method needs to be validated using a reliability diagram, which visualises the calibration of a classifier [30].”
MethodFlow-Dependent Reliability: A Path to More Skillful Ensemble Forecasts
Rodwell, M. J., Richardson, D. S., Parsons, D. B., & Wernli, H. (2018) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-17-0027.1 · Cited by 62
“(SB1) shows the “reliability– refinement” decomposition of the Brier score (DeGroot and Fienberg 1983).”
BackgroundLevin, S., Harley, E. T., Fackler, J. C., Lehmann, C. U., Custer, J. W., France, D. J., & Zeger, S. L. (2012) · Critical Care Medicine · doi:10.1097/ccm.0b013e31825bc399 · Cited by 50
“The RPS has been demonstrated to be an effective measure for evaluating the quality of probabilistic fore-casts (31).”
BackgroundZadora, G., & Ramos, D. (2010) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2010.03.007 · Cited by 48
Modeling Hierarchical Uncertainty for Multimodal Emotion Recognition in Conversation
Chen, F., Shao, J., Zhu, A., Ouyang, D., Liu, X., & Shen, H. T. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3185119 · Cited by 57
“We further employ a reliability diagram [61], [62] for visualization.”
MethodCypko, M. A., Stoehr, M., Koźniewski, M., Drużdżel, M. J., Dietz, A., Berliner, L., & Lemke, H. U. (2017) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-017-1531-7 · Cited by 49
“Acalibration curve [5] expresses amodel’s ability to produce precise probabilities.”
BackgroundUsing asymmetric distributions to improve text classifier probability estimates
Bennett, P. N. (2003) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/860435.860457 · Cited by 52
“Lindley et. al [19] (cid:2)rst proposed the idea of recalibrating classi(cid:2)ers, and DeGroot & Fienberg [5, 6] gave the now accepted standard formalization for the problem of assessing calibration initiated by others [4, 24].”
BackgroundEffective Ways to Build and Evaluate Individual Survival DistributionsPreprint
Haider, H., Hoehn, B., Davis, S. L., & Greiner, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.11347 · Cited by 171
“The Brier score (Brier and Allen, 1951) is a commonly used metric that measures both calibration and discrimination (Murphy, 1972, 1973; DeGroot and Fienberg, 1983).”
MethodExplicit Regularisation in Gaussian Noise InjectionsPreprint
Camuto, A., Willetts, M., Şimşekli, U., Roberts, S., & Holmes, C. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07368 · Cited by 80
“Given a network’s prediction ŷ(x) with confidence p̂(x) for a point x, perfect calibration consists of being as likely to be correct as you are confident: p(ŷ = y|p̂ = r) = r, 8r 2 [0, 1] (Dawid, 1982; DeGroot and Fienberg, 1983).”
BackgroundCalibration and the Expert Problem
French, S. (1986) · Management Science · doi:10.1287/mnsc.32.3.315 · Cited by 48
Agbavor, F., & Liang, H. (2022) · Brain Sciences · doi:10.3390/brainsci13010028 · Cited by 47
“The plot leads to the reliability diagram that provides a visualization of how well-calibrated a model is [32].”
BackgroundPrediction Market Performance and Market Liquidity: A Comparison of Automated Market Makers
Slamka, C., Skiera, B., & Spann, M. (2012) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2012.2191618 · Cited by 47
Huo, D., Senie, R. T., Daly, M. B., Buys, S. S., Cummings, S. A., Ogutha, J. O., Hope, K. T., & Olopade, O. I. (2009) · Journal of Clinical Oncology · doi:10.1200/jco.2008.17.5869 · Cited by 49
Modelling Probabilistic Agent Opinion
West, M., & Crosse, J. (1992) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1992.tb01882.x · Cited by 46
“Of the various notions of calibration, the empirical concept of frequency calibration (e.g. DeGroot and Fienberg (1983) and French (1986)) is perhaps the most widely discussed.”
Outlier-based detection of unusual patient-management actions: An ICU study
Hauskrecht, M., Batal, I., Hong, C., Nguyen, Q., Cooper, G. F., Visweswaran, S., & Clermont, G. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.10.002 · Cited by 45
Likelihood Ratio Calibration in a Transparent and Testable Forensic Speaker Recognition Framework
Ramos-Castro, D., González-Rodríguez, J., & Ortega-García, J. (2006) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.1109/odyssey.2006.248088 · Cited by 45
“First, the only confidence value which optimize a strictly proper scoring rule is the actual probability of occurrence of the hypothesis [19].”
BackgroundInfluentialWalker, K. D., Catalano, P. J., Hammitt, J. K., & Evans, J. S. (2003) · Journal of Exposure Science & Environmental Epidemiology · doi:10.1038/sj.jea.7500253 · Cited by 46
“Scoring Rules Several different forms of scoring rules have been proposed (Brier, 1950; Murphy, 1972, 1973; Matheson and Winkler, 1976; DeGroot and Fienberg, 1983) but all essentially give a mathematical measure of the distance between the predicted probabilities and observed frequencies.”
BackgroundCao, S., Liu, A., & Huang, C.-M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637318 · Cited by 41
“In interaction paradigms where the user provides an initial decision before receiving the AI suggestion, the “switch” metric is commonly used.”
BackgroundThe Importance of Calibration for Estimating Proportions from Annotations
Card, D., & Smith, N. A. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1148 · Cited by 62
Guillon, H., Byrne, C. F., Lane, B., Sandoval-Solís, S., & Pasternack, G. B. (2020) · Water Resources Research · doi:10.1029/2019wr026691 · Cited by 45
“Such calibration corrects the potential distortion of the posterior probabilities when compared to empirical probabilities and improves model performance (DeGroot & Fienberg, 1983; Niculescu-Mizil & Caruana, 2005; Zadrozny, 2002).”
Generic Conditions for Forecast Dominance
Krüger, F. P., & Ziegel, J. (2018) · Journal of business & economic statistics · doi:10.1080/07350015.2020.1741376 · Cited by 48
Kull, M., & Flach, P. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-23528-8_5 · Cited by 43
“The decomposition of Brier score into calibration and refinement loss (which were back then called reliability and resolution) was introduced by Murphy [8] and was generalised for proper scoring rules by DeGroot and Fienberg [5].”
BackgroundMethodInfluentialBinary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models
Naeini, M. P., & Cooper, G. F. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0047 · Cited by 46
“Figure 1 illustrates the binary calibration problem using a reliability curve [5], [17].”
MethodEstimating Expected Calibration Errors
Posocco, N., & Bonnefoy, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86380-7_12 · Cited by 85
“Mirroring the procedure used to compute the reliability diagram, the empirical Expected Calibration Error ( ECE ) was designed [3], and later has been proven to be an estimator for the natural theoretical notion of calibration error [4].”
Quality measures based calibration with duration and noise dependency for speaker recognition
Mandasari, M. I., Saeidi, R., & van Leeuwen, D. A. (2015) · Speech Communication · doi:10.1016/j.specom.2015.05.009 · Cited by 42
“It is a proper scoring rule [50], and is similar to a cross-entropy measure.”
MethodScreening probability forecasts: contrasts between choosing and combining
Clemen, R. T., Murphy, A. H., & Winkler, R. L. (1995) · International Journal of Forecasting · doi:10.1016/0169-2070(94)02007-c · Cited by 41
Detecting inappropriate access to electronic health records using collaborative filtering
Menon, A. K., Jiang, X., Kim, J., Vaidya, J. S., & Lucila Ohno-machado, A. (2013) · Machine Learning · doi:10.1007/s10994-013-5376-1 · Cited by 41
“A model for which the curve is a diagonal is said to be calibrated (DeGroot and Fienberg 1983).”
BackgroundA Pragmatist’s Guide to Epistemic Utility
Levinstein, B. A. (2017) · Philosophy of Science · doi:10.1086/693444 · Cited by 40
“See DeGroot and Fienberg (1982, 1983) and DeGroot and Eriksson (1985) for a detailed characterization of when one credence function is more accurate than another on every strictly proper rule. measure of epistemic utility.”
BackgroundBiedermann, A., Garbolino, P., & Taroni, F. (2013) · Science & Justice · doi:10.1016/j.scijus.2013.01.003 · Cited by 41
“values [62].”
Lin, T.-E., & Xu, H. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.104979 · Cited by 45
“For probability calibration, we use Expected Calibration Error [28] (ECE) to evaluate the effectiveness of temperature scaling.”
MethodA decision support system for cost-effective diagnosis
Chi, C., Street, W. N., & Katz, D. A. (2010) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2010.08.001 · Cited by 38
The Parimutuel Kelly Probability Scoring Rule
Johnstone, D. J. (2007) · Decision Analysis · doi:10.1287/deca.1070.0091 · Cited by 38
“In finance theory and investments, expected log utility maximization is known as log optimal investment and was introduced by Hakansson (1971), Roll (1973), Markowitz (1976), and Rubinstein (1976).”
BackgroundReliability for degrees of belief
Dunn, J. (2014) · Philosophical Studies · doi:10.1007/s11098-014-0380-2 · Cited by 38
“An example of a proper scoring rule is the Brier Score9 (Brier 1950): Brier Score: ½cðPÞ ¼ n is given a score of ðP nÞ2.”
BackgroundMethodInfluentialWeakly supervised joint whole-slide segmentation and classification in prostate cancer
Pati, P., Jaume, G., Ayadi, Z., Thandiackal, K., Bozorgtabar, B., Gabrani, M., & Goksel, O. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102915 · Cited by 37
Toward Explainable Fashion Recommendation
Tangseng, P., & Okatani, T. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093367 · Cited by 42
“Figure 5 shows the reliability diagrams [39,40] before and after the calibration.”
BackgroundRamos, D., & Zadora, G. (2011) · Analytica Chimica Acta · doi:10.1016/j.aca.2011.05.029 · Cited by 36
Camporeale, E., Cash, M. D., Singer, H. J., Balch, C. C., Huang, Z., & Tóth, G. (2020) · Journal of Geophysical Research Space Physics · doi:10.1029/2019ja027684 · Cited by 35
“One way to visualize the relationship between predicted probabilities and observed frequency is through a reliability diagram (DeGroot & Fienberg, 1983).”
MethodThe Effect of Privacy Regulation on the Data Industry: Empirical Evidence from GDPR
Aridor, G., Che, Y., & Salz, T. (2020) · National Bureau of Economic Research · doi:10.3386/w26900 · Cited by 47
Mars Image Content Classification: Three Years of NASA Deployment and Recent Advances
Wagstaff, K. L., Lu, S., Dunkel, E. R., Grimes, K., Zhao, B., Cai, J., Cole, S. B., Doran, G. B., Francis, R., Lee, J., et al. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17784 · Cited by 37
Copula Based Classifier Fusion Under Statistical Dependence
Özdemir, O., Allen, T. G., Choi, S., Wimalajeewa, T., & Varshney, P. K. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2774300 · Cited by 34
“We show by simulations as well as by applying on real data that our classifier fusion approach performs better (in terms of both Brier Score [23] and classification accuracy) than existing approaches.”
BackgroundMethodInfluentialFederated Generalized Bayesian Learning via Distributed Stein Variational Gradient Descent
Kassab, R., & Simeone, O. (2022) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3168490 · Cited by 59
Estimating and Evaluating Confidence for Forensic Speaker Recognition
Campbell, W. M., Reynolds, D. A., Campbell, J. P., & Brady, K. J. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2005.1415214 · Cited by 34
“The problem has been studied in statistics [11, 12], meteorology [13], and in speech processing [4, 14].”
BackgroundOn comparing the accuracy of default predictions in the rating industryPreprint
Krämer, W., & Güttler, A. (2006) · SSRN Electronic Journal · doi:10.2139/ssrn.450600 · Cited by 76
“Such forecasters are called ”well calibrated” (DeGroot and Fienberg 1983).”
Calibration of deep probabilistic models with decoupled bayesian neural networks
Maroñas, J., Paredes, R., & Ramos, D. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.04.103 · Cited by 41
Understanding Scores in Forensic Speaker Recognition
Campbell, W. M., Brady, K. J., Campbell, J. P., Granville, R., & Reynolds, D. A. (2006) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.1109/odyssey.2006.248091 · Cited by 33
“DeGroot introduces the idea of calibration and refinement in [6].”
BackgroundInfluentialDeMaria, M., & Kaplan, J. (1999) · Weather and Forecasting · doi:10.1175/1520-0434(1999)014<0326:auship>2.0.co;2 · Cited by 350
A Bayesian view of assessing uncertainty and comparing expert opinion
DeGroot, M. H. (1988) · Journal of Statistical Planning and Inference · doi:10.1016/0378-3758(88)90094-8 · Cited by 29
Self-citationCalibration, Sufficiency, and Domination Considerations for Bayesian Probability Assessors
Vardeman, S. B., & Meeden, G. D. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477025 · Cited by 31
Error awareness and recovery in conversational spoken language interfaces
Rudnicky, A. I., Rosenfeld, R., & Bohus, D. (2007) · Cited by 33
Calibrated lazy associative classification
Veloso, A., Meira, W., Gonçalves, M., Almeida, H. M., & Zaki, M. (2010) · Information Sciences · doi:10.1016/j.ins.2010.03.007 · Cited by 30
“These diagrams are built as follows [7].”
Inducing Models of Black-Box Storage Arrays
Kelly, T., Cohen, I. L., Goldszmidt, M., & Keeton, K. (2004) · Cited by 30
Evaluating probability forecasts
Lai, T. L., Gross, S. T., & Shen, D. B. (2011) · The Annals of Statistics · doi:10.1214/11-aos902 · Cited by 39
“Note that Dawid’s well-calibration theorem (3.6) involves the subjective probability measure Π. DeGroot and Fienberg (1983) have noted that well-calibrated forecasts need not reflect the forecaster’s “honest subjective probabilities,” that is, need not satisfy Dawid’s coherence criterion p̂i = πi.”
BackgroundInfluentialKrzysztofowicz, R. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478387 · Cited by 30
Bayesian machine learning applied in a brain-computer interface for disabled users
Hoffmann, U. (2007) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.5075/epfl-thesis-3924 · Cited by 37
“A tool that allows to measure calibration as well as refinement is the so-called Brier score (Brier, 1950; DeGroot and Fienberg, 1983).”
Ozdemir, O., Woodward, B., & Berlin, A. A. (2017) · arXiv · doi:10.48550/arxiv.1712.00497 · Cited by 30
“We also show Brier scores [12] (weighted due to heavy class imbalance) of test predictions in Table 1 showing that the Bayesian CNN with Uncertainty Fusion provides better performance than the 1-channel Bayesian CNN in terms of Brier score, and that ensembling the two methods provides the best performance.”
MethodTransferring Inductive Biases through Knowledge Distillation
Abnar, S., Dehghani, M., & Zuidema, W. (2020) · arXiv · doi:10.48550/arxiv.2006.00555 · Cited by 30
“acies are perfectly correlated with the confidence values. The Expected Calibration Error (ECE) is computed as the distance between the calibration curve of the model and the perfect calibration curve [5]. In Figure 3b, we plot the ECE [13] of the models in the classification setup, with the filled circles on the black d…”
MethodRobust Bayesian Learning for Reliable Wireless AI: Framework and Applications
Zecchin, M., Park, S., Simeone, O., Kountouris, M., & Gesbert, D. (2023) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2023.3261300 · Cited by 44
“We adopt the standard reliability diagrams [59] and the expected calibration error as diagnostic tools for the calibration performance [10].”
MethodInformation-theoretical comparison of likelihood ratio methods of forensic evidence evaluation
Ramos, D., González-Rodríguez, J., Zadora, G., Zięba‐Palus, J., & Aitken, C. G. G. (2007) · International Conference on Information Assurance and Security (IAS) · doi:10.1109/ias.2007.63 · Cited by 29
“This is due to a different calibration of each procedure, as defined in [7].”
Why Evaluating Uncertainty Visualization is Error Prone
Hullman, J. R. (2016) · doi:10.1145/2993901.2993919 · Cited by 41
“A rational agent’s willingness to accept betting odds given a utility function that expresses the value of making a decision, conditional on the true values of the quantities in question, is used to derive probabilities [51, 15].”
BackgroundOn comparing the accuracy of default predictions in the rating industry
Krämer, W., & Güttler, A. (2007) · Empirical Economics · doi:10.1007/s00181-007-0125-6 · Cited by 29
Using a Bootstrap Approach to Rate the Raters
Güttler, A. (2005) · Financial markets and portfolio management · doi:10.1007/s11408-005-4693-1 · Cited by 29
Padhy, S., Nado, Z., Ren, J., Liu, J. Z., Snoek, J., & Lakshminarayanan, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.05134 · Cited by 48
“To further understand the interplay of behaviors between one-vs-all normalizations and distance-based logits, we plot reliability diagrams (DeGroot & Fienberg, 1983; Niculescu-Mizil & Caruana, 2005) which show accuracy as a function of confidence in the form of binned histograms of confi-dences over…”
Pan, F., Ao, X., Tang, P., Lu, M., Liu, D., Xiao, L., & He, Q. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380154 · Cited by 39
“In many previous studies [6, 10, 23], the calibration error is formulated by partitioning the predictions into bins and summing up the errors over the bins.”
BackgroundMethodCharnock, T., Perreault-Levasseur, L., & Lanusse, F. (2022) · WORLD SCIENTIFIC eBooks · doi:10.1142/9789811234026_0018 · Cited by 55
Jointly Minimizing the Expected Costs of Review for Responsiveness and Privilege in E-Discovery
Oard, D. W., Sebastiani, F., & Vinjumur, J. K. (2019) · ACM Transactions on Information Systems · doi:10.1145/3268928 · Cited by 18
“Posterior probabilities Pr(c |d ) are said to be well calibrated when, given a sample S drawn from some population, lim|S |→∞ |{d∈c | Pr(c |d )=x }| |{d∈S | Pr(c |d )=x }| = x [10].”
BackgroundOdisho, A. Y., Park, B., Altieri, N., DeNero, J., Cooperberg, M. R., Carroll, P. R., & Yu, B. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooaa029 · Cited by 31
“02 for both classification and extraction tasks.(27) We find that for most classifications fields, the model had expected calibration scores less than 0.”
BackgroundOn Calibrating Semantic Segmentation Models: Analyses and An Algorithm
Wang, D., Gong, B., & Wang, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02265 · Cited by 54
“Reliability diagrams [5] of visualized ECE for Segmenter [39].”
BackgroundEconomic Interpretation of Probabilities Estimated by Maximum Likelihood or Score
Johnstone, D. J. (2010) · Management Science · doi:10.1287/mnsc.1100.1272 · Cited by 27
Bipartite ranking: a risk-theoretic perspective
Menon, A. K., & Williamson, R. C. (2016) · ANU Open Research (Australian National University) · Cited by 27
With All Due Respect: The Macro-Epistemology of Disagreement
Levinstein, B. A. (2015) · University of Michigan Library Repository · Cited by 26
Esuli, A., Moreo, A., & Sebastiani, F. (2019) · ACM Transactions on Information Systems · doi:10.1145/3326065 · Cited by 25
“2Posterior probabilities Pr(c |d ) are said to be well calibrated when lim|S |→∞ |{d∈c | Pr(c |d )=x }| |{d∈S | Pr(c |d )=x }| = x [9].”
Stable Discovery of Interpretable Subgroups via Calibration in Causal Studies
Dwivedi, R., Yan, T., Park, B., Wei, M., Horgan, K., Madigan, D., & Yu, B. (2020) · International Statistical Review · doi:10.1111/insr.12427 · Cited by 47
“The concept of calibration has a long history (Dawid, 1982; DeGroot & Fienberg, 1983), and it has also been referred to as validity (Miller, 1962) or reliability (Murphy, 1973).”
BackgroundMethodInfluentialCalibration of Machine Learning Models
Bella, A., Ferri, C., Hernández‐Orallo, J., & Ramírez-Quintana, M. J. (2011) · IGI Global eBooks · doi:10.4018/978-1-60960-818-7.ch104 · Cited by 26
Ji, B., Jung, H., Yoon, J., Kim, K., & Shin, Y. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00515 · Cited by 28
“In Figure 1, we visualize the baseline TS concept and the two proposed approaches, BTS and ABTS, using a reliability diagram [3] [1].”
MethodThe PAV algorithm optimizes binary proper scoring rulesPreprint
Brümmer, N., & du Preez, J. A. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1304.2331 · Cited by 36
“See for example [13], where the form ∫ 1”
BackgroundAn Experimental Investigation of Calibration Techniques for Imbalanced Data
Huang, L., Zhao, J., Zhu, B., Chen, H., & Broucke, S. V. (2020) · IEEE Access · doi:10.1109/access.2020.3008150 · Cited by 29
On Focal Loss for Class-Posterior Probability Estimation: A Theoretical Perspective
Charoenphakdee, N., Vongkulbhisal, J., Chairatanakul, N., & Sugiyama, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00516 · Cited by 47
“In that literature, overconfidence was used to describe the empirical performance of modern neural networks [13, 35], where a classifier outputs an average confidence score higher than its average accuracy for a set of data points.”
MethodHypothesis Disparity Regularized Mutual Information Maximization
Lao, Q., Jiang, X., & Havaei, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.17003 · Cited by 32
Bayesian continual learning via spiking neural networks
Skatchkovsky, N., Jang, H., & Simeone, O. (2022) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2022.1037976 · Cited by 43
Incentive contrats and strictly proper scoring rules
Clemen, R. T. (2002) · Test · doi:10.1007/bf02595735 · Cited by 25
“Second, a strictly proper scoring rule has the property that a “more knowledgeable” expert must have a higher expected score ( DeGroot and Fienberg (1982, 1983) ; Winkler (1977, 1986)).”
BackgroundThe maximum entropy approach and probabilistic IR models
Greiff, W. R., & Ponte, J. M. (2000) · ACM Transactions on Information Systems · doi:10.1145/352595.352597 · Cited by 25
Experts and Non-ExpertsPreprint
Valsecchi, I. (2007) · SSRN Electronic Journal · doi:10.2139/ssrn.1001904 · Cited by 24
Source-Free Adaptation to Measurement Shift via Bottom-Up Feature RestorationPreprint
Eastwood, C., Mason, I., Williams, C. K. I., & Schölkopf, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05446 · Cited by 66
“This section shows reliability diagrams (DeGroot & Fienberg, 1983; Niculescu-Mizil & Caruana, 2005) and confidence histograms (Zadrozny & Elkan, 2001): (i) over all EMNIST - DA shifts (see Figure 10); (ii) a severe EMNIST - DA shift (see Figure 11); and (iii) a mild shift EMNIST - DA shift (see…”
BackgroundNon-Parametric Calibration for Classification
Wenger, J., Kjellström, H., & Triebel, R. (2019) · elib (German Aerospace Center) · doi:10.48550/arxiv.1906.04933 · Cited by 114
“Originally developed in the context of forecasting (Murphy, 1973, DeGroot and Fienberg, 1983), uncertainty calibration has seen an increased interest in recent years (Naeini et al., 2015, Guo et al., 2017, Vaicenavicius et al., 2019), partly because of the popularity of CNNs which generally lack an…”
BackgroundEsuli, A., Molinari, A., & Sebastiani, F. (2020) · ACM Transactions on Information Systems · doi:10.1145/3433164 · Cited by 39
“Interestingly, the fact that SLD could not reduce RE is reminiscent of an observation by DeGroot and Fienberg [7]: We then study the question of when an observer can use a forecaster’s predictions to obtain a better score than the forecaster himself, and show that such an improvement can be achieved by the observer es…”
BackgroundResultInfluentialStop Measuring Calibration When Humans Disagree
Baan, J., Aziz, W. F., Plank, B., & Fernández, R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.124 · Cited by 109
Optimal Reporting of Predictions
Bayarri, M. J., & DeGroot, M. H. (1989) · Journal of the American Statistical Association · doi:10.1080/01621459.1989.10478758 · Cited by 24
Self-citation
Metadata
- Authors
- DeGroot, M. H., & Fienberg, S. E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- The Statistician
- Publisher
- JSTOR
- Published
- 1983-03
- Volume
- 32
- Issue
- 1/2
- Pages
- 12
- ISSN
- 0039-0526 (ISSN-L)
- OpenAlex
- W1599263113
- Cited by
- 604
- References
- 18
- Citations by year
- 2026: 14 · 2025: 38 · 2024: 32 · 2023: 51 · 2022: 42 · 2021: 52 · 2020: 48 · 2019: 36 · 2018: 26 · 2017: 20 · 2016: 16 · 2015: 21 · 2014: 12 · 2013: 17 · 2012: 14
- FWCI
- 5.36
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the Royal Statistical Society Series D (The Statistician) (publishedVersion)
- Defense Technical Information Center (DTIC) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- forecast accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.