On the evaluation of probability judgments: Calibration, resolution, and monotonicity.
Abstract
Judgments of probability are commonly evaluated by two criteria: calibration, namely, the correspondence between stated confidence and rate of occurrence, and resolution, namely, the ability to distinguish between events that do and do not occur. Two representations of probability judgments are contrasted: the designated form that presupposes a particular coding of outcomes (e.g., rain vs. no rain) and the inclusive form that incorporates all events and their complements. It is shown that the indices of calibration and resolution derived from these representations measure different characteristics of judgment. Calibration is distinguished from two types of overconfidence: specific and generic. An ordinal measure of performance is proposed and compared to the standard measures in forecasts of recession and in both numerical and verbal assessments of general knowledge. Much research on judgment under uncertainty has focused on the comparison of probability judgments with the corresponding relative frequency of occurrence. In a typical study, judges are presented with a series of prediction or knowledge problems and asked to assess the probability of the events in question. Judgments of probability or confidence are used both in research (Lichtenstein, Fischhoff, & Phillips, 1982; Wallsten & Budescu, 1983) and in practice. For example, weather forecasters often report the probability of rain (Murphy & Daan, 1985), and economists are sometimes called upon to estimate the chances of recession (Zarnowitz & Lambros, 1987). The two main criteria used to evaluate such judgments are calibration and resolution. A judge is said to be calibrated if his or her probability judgments match the corresponding relative frequency of occurrence. More specifically, consider all events to which the judge assigns a probability p; the judge is calibrated if the proportion of events in that class that actually occur equals p. Calibration is a desirable property, especially for communication, but it does not ensure informativeness. A judge can be properly calibrated and entirely noninformative if, for example, he or she predicts the sex of each newborn with
Cites (4)
- “Clinical versus Actuarial Judgment.”Dawes, Robyn M., Faust, David, Meehl, Paul E. · 1989 · Science
- Calibration of probabilities: the state of the art to 1980Lichtenstein, Sarah, Fischhoff, Baruch, Phillips, Lawrence D. · 1982 · Judgment under Uncertainty
- A comparison of current measures of the accuracy of feeling-of-knowing predictionsNelson, Thomas O. · 1984 · Psychological Bulletin
- Verification of forecasts expressed in terms of probabilityBRIER, GLENN W. · 1950 · Monthly Weather Review
Cited by (23)
- Heuristics and Biases: The Psychology of Intuitive JudgmentBottom, William P., Gilovich, Thomas, Griffin, Dale, Kahneman, Daniel · 2004 · The Academy of Management Review
- Representativeness revisited: Attribute substitution in intuitive judgment.Kahneman, Daniel, Frederick, Shane · 2002 · Heuristics and Biases
- The Affect HeuristicSlovic, Paul, Finucane, Melissa, Peters, Ellen, MacGregor, Donald G. · 2002 · Heuristics and Biases
- Heuristics and Biases: The Psychology of Intuitive Judgment2002 · Cambridge University Press eBooks
- The weighing of evidence and the determinants of confidenceGriffin, Dale, Tversky, Amos · 1992 · Cognitive Psychology
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: On the evaluation of probability judgments: Calibration, resolution, and monotonicity. https://app.cochat.ai/papers/10.1037//0033-2909.114.1.162
- The paper
- Liberman, V., & Tversky, A. (1993). On the evaluation of probability judgments: Calibration, resolution, and monotonicity. Psychological Bulletin. https://doi.org/10.1037//0033-2909.114.1.162
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.