Scoring Rules and Decision Analysis Education
Abstract
Experiential learning is perhaps the most effective way to teach. One example is the scoring procedure used for exams in some decision analysis programs. Under this grading scheme, students take a multiple-choice exam, but rather than simply marking which answer they think is correct, they must assign a probability to each possible answer. The exam is then scored with a special scoring rule, under which students' best strategy is to avoid guessing and instead assign their true beliefs. Such a scoring function is known as a strictly proper scoring rule. In this paper, we discuss several different scoring rules and demonstrate how their use in testing situations provides insights for both students and instructors.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Similar scoring rules are being used to grade students in several university courses on decision analysis outside the research setting.
“Outside of the research setting, similar scoring rules are being used to grade students in several university courses on decision analysis”
From Equity and Artificial Intelligence in Education: Will "AIEd" Amplify or Alleviate Inequities in Education? (2021) · cites this paper as Bickel, 2010
Cites
Dawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.1080/01621459.1982.10477856 · Cited by 625 · cited 1× · see citing works
“Perhaps more important for Bayesians is the concept of strict coherence , or Cromwell's rule (Dawid 1982, Lindley 1982), which states that a probability of zero should not be assigned to any possibility.”
Scoring Rules and the Evaluation of Probability Assessors
Winkler, R. L. (1969) · Journal of the American Statistical Association · doi:10.1080/01621459.1969.10501037 · Cited by 271 · see citing works
Some Comparisons Among Quadratic, Spherical, and Logarithmic Scoring Rules
Bickel, J. E. (2007) · Decision Analysis · doi:10.1287/deca.1070.0089 · Cited by 142 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
On the appropriateness of Platt scaling in classifier calibration
Böken, B. (2021) · Information Systems · doi:10.1016/j.is.2020.101641 · Cited by 70 · see citing works
Holstein, K., & Doroudi, S. (2021) · arXiv · doi:10.48550/arxiv.2104.12920 · Cited by 29 · see citing works
- Authors
- Bickel, J. E.
- Affiliations
- The University of Texas at Austin (US)
- Corresponding author
- Bickel, J. Eric
- ORCID
- Bickel, J. Eric
- Venue
- Decision Analysis
- Published
- 2010-12
- Volume
- 7
- Issue
- 4
- Pages
- 346-357
- ISSN
- 1545-8490 (ISSN-L)
- 1545-8504
- OpenAlex
- W2123564302
- Cited by
- 31
- References
- 3
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 1 · 2023: 1 · 2022: 2 · 2020: 4 · 2019: 1 · 2018: 3 · 2016: 2 · 2015: 2 · 2014: 1 · 2013: 4 · 2012: 2
- FWCI
- 2.21
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Decision Analysis (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Social Sciences › Education › Student Assessment and Feedback
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Keywords
- scoring rules, strictly proper scoring rules, multiple-choice tests, probability elicitation, experiential learning, test scoring, belief elicitation, guessing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
