The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
A penny for your thoughts: a survey of methods for eliciting beliefs
Open accessAbstract
Incentivized methods for eliciting subjective probabilities in economic experiments present the subject with risky choices that encourage truthful reporting. We discuss the most prominent elicitation methods and their underlying assumptions, provide theoretical comparisons and give a new justification for the quadratic scoring rule. On the empirical side, we survey the performance of these elicitation methods in actual experiments, considering also practical issues of implementation such as order effects, hedging, and different ways of presenting probabilities and payment schemes to experimental subjects. We end with a discussion of the trade-offs involved in using incentives for belief elicitation and some guidelines for implementation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Support theory: a nonextensional representation of subjective probability
Tversky, A., & Koehler, D. J. (1994) · Psychological Review · doi:10.1037/0033-295x.101.4.547 · Cited by 1,167
Good, I. J. (1952) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1952.tb00104.x · Cited by 976
Interpreting the predictions of prediction markets
Manski, C. F. (2006) · Economics Letters · doi:10.1016/j.econlet.2006.01.004 · Cited by 330
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 1×
“This rule is based on the Brier score (Brier, 1950), which is the sum of the squared errors of the reported probabilities.”
Research works printed but not linked: 96
- Allen, F. (1987). Discovering personal probabilities when utility functions are unknown. Management Science, 33(4), 542–544. · DOI
- Andersen, S., Fountain, J., Harrison, G. W. and Rutström, E. E. (2014). Estimating Subjective Probabilities. Journal of Risk and Uncertainty, 48(3), 207–220. · DOI
- Armantier, O., & Treich, N. (2009). Subjective probabilities in games: A solution to the overbidding puzzle. International Economic Review, 50(4), 1079–1102. · DOI
- Armantier, O., & Treich, N. (2013). Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging. European Economic Review, 62, 17–40. · DOI
- Beach, L., & Phillips, L. (1967). Subjective probabilities inferred from estimates and bets. Journal of Experimental Psychology, 75(3), 354–359. · doi:10.1037/h0025061
- Beach, L. R., & Wise, J. A. (1969). Subjective probability revision and subsequent decisions. Journal of Experimental Psychology, 81(3), 561–565. · doi:10.1037/h0027919
- Becker, G., DeGroot, M., & Marschak, J. (1964). Measuring utility by a single-response sequential method. Behavioral Science, 9(3), 226. · DOI · Paper page
- Berg, J. E., Rietz, T. A., & Dickhaut, J. W. (2008). On the performance of the lottery procedure for controlling risk preferences. Handbook of Experimental Economics Results, 1, 1087–1097. · doi:10.1016/s1574-0722(07)00115-1
- Blanco, M., Engelmann, D., Koch, A. K., & Normann, H.-T. (2010). Belief elicitation in experiments: Is there a hedging problem? Experimental Economics, 13(4), 412–438. · DOI · Paper page
- Cervera, J. L., & Muñoz, J. (1996). Proper scoring rules for fractiles. In J. O. Berger, A. P. Dawid, & A. F. M. Smith (Eds.), Bayesian statistics 5 (pp. 513–519). Oxford: Oxford University Press. · doi:10.1093/oso/9780198523567.003.0029
- Croson, R. T. A. (2000). Thinking like a game theorist: Factors affecting the frequency of equilibrium play. Journal of Economic Behavior & Organization, 41(3), 299–314. · DOI · Paper page
- Dawes, R. M., McTavish, J., & Shaklee, H. (1977). Behavior, communication, and assumptions about other people’s behavior in a commons dilemma situation. Journal of Personality and Social Psychology, 35(1), 1. · doi:10.1037/0022-3514.35.1.1 · Paper page
- De Finetti, B. (1965). Methods for discriminating levels of partial knowledge concerning a test item. British Journal of Mathematical and Statistical Psychology, 18(1), 87–123. · DOI · Paper page
- De Finetti, B. (1970). Logical foundations and measurement of subjective probability. Acta Psychologica, 34, 129–145. · DOI
- De Finetti, B. (1974). Theory of probability (Vol. 1). New York: Wiley.
- Delavande, A., Giné, X., & McKenzie, D. (2011). Measuring subjective expectations in developing countries: A critical review and new evidence. Journal of Development Economics, 94(2), 151–163. · DOI · Paper page
- Ducharme, W., & Donnell, M. (1973). Intrasubject comparison of four response modes for “subjective probability” assessment. Organizational Behavior and Human Performance, 10, 108–117. · doi:10.1016/0030-5073(73)90007-x
- Engelmann, D., & Strobel, M. (2000). The false consensus effect disappears if representative information and monetary incentives are given. Experimental Economics, 260(2000), 241–260. · doi:10.1023/a:1011472501737
- Erev, I., Bornstein, G., & Wallsten, T. (1993). The negative effect of probability assessments on decision quality. Organizational Behavior and Human Decision Processes, 55, 78–94. · DOI
- Fischer, G. W. (1982). Scoring-rule feedback and the overconfidence syndrome in subjective probability forecasting. Organizational Behavior and Human Performance, 29(3), 352–369. · DOI
- Fountain, J., & Harrison, G. W. (2011). What do prediction markets predict? Applied Economics Letters, 18(3), 267–272. · DOI
- Friedman, D. (1983). Effective scoring rules for probabilistic forecasts. Management Science, 29(4), 447–454. · DOI
- Gächter, S., & Renner, E. (2010). The effects of (incentivized) belief elicitation in public goods experiments. Experimental Economics, 13(3), 364–377. · DOI · Paper page
- Garthwaite, P. H., Kadane, J. B., & O’Hagan, A. (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470), 680–701. · DOI · Paper page
- Gigerenzer, G., & Hoffrage, U. (1995). How to improve bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684–704. · doi:10.1037/0033-295x.102.4.684 · Paper page
- Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. · DOI · Paper page
- Guerra, G., & Zizzo, D. J. (2004). Trust responsiveness and beliefs. Journal of Economic Behavior & Organization, 55(1), 25–30. · DOI
- Hao, L., & Houser, D. (2012). Belief elicitation in the presence of novice participants: An experimental study. Journal of Risk and Uncertainty, 2, 161–180. · doi:10.1007/s11166-011-9133-1
- Harrison, G. W., Martínez-Correa, J., & Swarthout, J. (2014). Eliciting subjective probabilities with binary lotteries. Journal of Economic Behavior and Organization, 101, 128–140. · DOI
- Harrison, G. W., Martinez-Correa, J., & Swarthout, T. (2013b). Inducing risk neutral preferences with binary lotteries: A reconsideration. Journal of Economic Behavior and Organization, 94, 145–159. · DOI
- Harrison, G. W., & Rutström, E. E. (2009). Expected utility theory and prospect theory: One wedding and a decent funeral. Experimental Economics, 12(2), 133–158. · DOI · Paper page
- Heinemann, F., Nagel, R., & Ockenfels, P. (2009). Measuring strategic uncertainty in coordination games. Review of Economic Studies, 76, 181–221. · DOI · Paper page
- Holt, C. (2006). Markets, games and strategic behavior. Boston: Pearson/Addison-Wesley.
- Holt, C. A., & Laury, S. (2002). Risk aversion and incentive effects. The American Economic Review, 92(5), 1644. · DOI · Paper page
- Hossain, T., & Okui, R. (2013). The binarized scoring rule. The Review of Economic Studies, 80(3), 984–1001. · DOI · Paper page
- Huck, S., & Weizsäcker, G. (2002). Do players correctly estimate what others do?: Evidence of conservatism in beliefs. Journal of Economic Behavior & Organization, 47(1), 71–85. · doi:10.1016/s0167-2681(01)00170-6
- Hurley, T., & Shogren, J. (2005). An experimental comparison of induced and elicited beliefs. Journal of Risk and Uncertainty, 30(2), 169–188. · DOI
- Jaffray, J., & Karni, E. (1999). Elicitation of subjective probabilities when the initial endowment is unobservable. Journal of Risk and Uncertainty, 8, 5–20. · doi:10.1023/a:1007706009628
- Jenkinson, D. (2005). The elicitation of probabilities—A review of the statistical literature. Manuscript, University of Sheffield.
- Jensen, F. A., & Peterson, C. R. (1973). Psychological effects of proper scoring rules. Organizational Behavior and Human Performance, 9(2), 307–317. · DOI
- Jose, V. R. R., & Winkler, R. L. (2009). Evaluating quantile assessments. Operations Research, 57(5), 1287–1297. · DOI
- Kadane, J., & Winkler, R. (1988). Separating probability elicitation from utilities. Journal of the American Statistical Association, 83(402), 357–363. · doi:10.1080/01621459.1988.10478605
- Karni, E. (2009). A mechanism for eliciting probabilities. Econometrica, 77(2), 603–606. · DOI · Paper page
- Karni, E., & Safra, Z. (1995). The impossibility of experimental elicitation of subjective probabilities. Theory and Decision, 38, 313–320. · DOI
- Koessler, F., Noussair, C., & Ziegelmeyer, A. (2012). Information aggregation and belief elicitation in experimental parimutuel betting markets. Journal of Economic Behavior & Organization, 83(2), 195–208. · DOI
- Kothiyal, A., Spinu, V., & Wakker, P. (2011). Comonotonic proper scoring rules to measure ambiguity and subjective beliefs. Journal of Multi-Criteria Decision Analysis, 101, 101–113.
- Lambert, N., Pennock, D., & Shoham, Y. (2008). Eliciting properties of probability distributions: The highlights. ACM SIGecom Exchanges, 7(3), 1–5. · DOI
- Lindley, D. V., Tversky, A., & Brown, R. V. (1979). On the reconciliation of probability assessments. Journal of the Royal Statistical Society. Series A (General), 142(2), 146. · DOI · Paper page
- Lipkus, I., Samsa, G., & Rimer, B. (2001). General performance on a numeracy scale among highly educated samples. Medical Decision Making, 21, 37–44. · DOI · Paper page
- Machina, M. J., & Schmeidler, D. (1992). A more robust definition of subjective probability. Econometrica, 60(4), 745–780. · DOI
- Manski, C. (2002). Identification of decision rules in experiments on simple games of proposal and response. European Economic Review, 46, 880–891. · DOI
- Manski, C. (2004). Measuring expectations. Econometrica, 72(5), 1329–1376. · DOI · Paper page
- Matheson, J., & Winkler, R. (1976). Scoring rules for continuous probability distributions. Management Science, 22(10), 1087–1096. · DOI · Paper page
- McCarthy, J. (1956). Measures of the value of information. Proceedings of the National Academy of Sciences of the United States of America, 42(9), 654–655. · DOI · Paper page
- McKelvey, R., & Page, T. (1990). Public and private information: An experimental study of information pooling. Econometrica, 58(6), 1321–1339. · DOI · Paper page
- Miller, N., Resnick, P., & Zeckhauser, R. (2005). Eliciting informative feedback: The peer-prediction method. Management Science, 51(9), 1359–1373. · DOI · Paper page
- Nyarko, Y., & Schotter, A. (2002). An experimental study of belief learning using elicited beliefs. Econometrica, 70(3), 971–1005. · DOI · Paper page
- Offerman, T., & Palley, A. B. (2013). Lossed in Translation : An Off-the-Shelf Method to Recover Probabilistic Beliefs from Loss-Averse Agents. Manuscript, University of Amsterdam. · doi:10.2139/ssrn.2353500
- Offerman, T., Sonnemans, J., & Schram, A. (1996). Value orientations, expectations and voluntary contributions in public goods. The Economic Journal, 106(437), 817–845. · DOI · Paper page
- Offerman, T., Sonnemans, J., Van de Kuilen, G., & Wakker, P. P. (2009). A truth serum for non-bayesians. Review of Economic Studies, 76(4), 1461–1489. · DOI · Paper page
- Palfrey, T. R., & Wang, S. W. (2009). On eliciting beliefs in strategic games. Journal of Economic Behavior & Organization, 71(2), 98–109. · DOI
- Phillips, L. D., & Edwards, W. (1966). Conservatism in a simple probability inference task. Journal of Experimental Psychology, 72(3), 346–354. · DOI · Paper page
- Prelec, D. (2004). A bayesian truth serum for subjective data. Science, 306, 462–466. · DOI · Paper page
- Price, P. (1998). Effects of a relative-frequency elicitation question on likelihood judgment accuracy: The case of external correspondence. Organizational Behavior and Human Decision Processes, 76(3), 277–297. · DOI
- Ramsey, F. (1926). Truth and Probability. In R. B. Braithwaite (ed.), The foundations of mathematics and other logical essays (pp. 156–198). New York (1931): Harcourt.
- Roby, T. B. (1964). Belief states: A preliminary empirical study. Technical Documentary Report, Decision Sciences Laboratory. · DOI
- Rutström, E. E., & Wilcox, N. T. (2009). Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test. Games and Economic Behavior, 67(2), 616–632. · DOI · Paper page
- Savage, L. J. (1954). The foundation of statistics. New York: Wiley.
- Savage, L. J. (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336), 783–801. · DOI · Paper page
- Schervish, M. (1989). A general method for comparing probability assessors. The Annals of Statistics, 17(4), 1856–1879. · DOI · Paper page
- Schlag, K. H. and Tremewan, J. (2014). Simple belief elicitation. SSRN Working paper 2449224. · DOI
- Schlag, K. H., & van der Weele, J. J. (2013). Eliciting probabilities, means, medians, variances and covariances without assuming risk neutrality. Theoretical Economics Letters, 03(1), 38–42. · doi:10.4236/tel.2013.31006
- Schmalensee, R. (1976). An experimental study of expectation formation. Econometrica, 44(1), 17–41. · DOI
- Schotter, A., & Trevino, I. (2014). Belief elicitation in the laboratory. Annual Review of Economics, 6, 103–128. · DOI · Paper page
- Schum, D. A., Goldstein, I. L., Howell, W. C., & Southard, J. F. (1967). Subjective probability revisions under several cost-payoff arrangements. Organizational Behavior and Human Performance, 2, 84–104. · doi:10.1016/0030-5073(67)90012-8
- Selten, R. (1998). Axiomatic characterization of the quadratic scoring rule. Experimental Economics, 62, 43–62. · doi:10.1023/a:1009957816843 · Paper page
- Selten, R., Sadrieh, A., & Abbink, K. (1999). Money does not induce risk neutral behavior, but binary lotteries do even worse. Theory and Decision, 46, 211–249. · DOI
- Shuford, E., Albert, A., & Massengill, H. E. (1966). Admissible probability measurement procedures. Psychometrika, 31(2), 125–145. · DOI · Paper page
- Smith, A. (2013). Estimating the causal effect of beliefs on contributions in repeated public good games. Experimental Economics, 16(3), 414–425. · DOI
- Smith, C. (1961). Consistency in statistical inference and decision. Journal of the Royal Statistical Society. Series B, 23(1), 1–37. · DOI · Paper page
- Staël von Holstein, C.-A. S. (1970). Measurement of subjective probability. Acta Psychologica, 34, 146–159. · DOI
- Toda, M. (1951). Measurement of intuitive probability by a method of game. Japanese Journal of Psychology, 22, 29–40. · doi:10.4992/jjpsy.22.29
- Trautmann, S. T. and van de Kuilen, G. (2014). Belief elicitation: A horse race among truth serums. The Economic Journal. doi: 10.1111/ecoj.12160 . · DOI
- Vlek, C. (1973a). The fair betting game as an admissible procedure for assessment of subjective probabilities. British Journal of Mathematical and Statistical Psychology, 26(1), 18–30. · doi:10.1111/j.2044-8317.1973.tb00503.x
- Vlek, C. A. J. C. (1973b). Coherence of human judgment in a limited probabilistic environment. Organizational Behavior and Human Performance, 9(460–481), 460–481. · doi:10.1016/0030-5073(73)90065-2
- Wang, H., Dash, D., & Druzdzel, M. J. (2002). A method for evaluating elicitation schemes for probabilistic models. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 32(1), 38–43. · DOI
- Wang, S. W. (2011). Incentive effects: The case of belief elicitation from individuals in groups. Economics Letters, 111(1), 30–33. · DOI
- Whitcomb, K. M., Önkal, D., Benson, P. G., & Curley, S. P. (1993). An evaluation of the reliability of probability judgments across response modes and over time. Journal of Behavioral Decision Making, 6(4), 283–296. · DOI
- Winkler, R. (1996). Scoring rules and the evaluation of probabilities. Test, 5(1), 1–60. · DOI · Paper page
- Winkler, R., & Murphy, A. (1968). ”Good” probability assessors. Journal of Applied Meteorology, 7, 751. · DOI · Paper page
- Winkler, R., & Murphy, A. (1970). Nonlinear utility and the probability score. Journal of Applied Meteorology, 9, 143–148. · doi:10.1175/1520-0450(1970)009<0143:nuatps>2.0.co;2
- Winkler, R., & Murphy, A. (1979). The use of probabilities in forecasts of maximum and minimum temperatures. The Meteorological Magazine, 108(1288), 317–329.
- Wolfers, J. and Zitzewitz, E. (2006). Interpreting prediction market prices as probabilities. NBER Working Paper 12200. · DOI
- Yaniv, I., & Foster, D. (1995). Graininess of judgement under uncertainty: An accuracy-informativeness trade-off. Journal of Experimental Psychology: General, 124(4), 424–432. · DOI · Paper page
- Yaniv, I., & Foster, D. (1997). Precision and accuracy of judgmental estimation. Journal of Behavioral Decision Making, 10, 21–32. · DOI · Paper page
- Zizzo, D. J. (2009). Experimenter demand effects in economic experiments. Experimental Economics, 13(1), 75–98. · DOI · Paper page
Cited by
Köbis, N., & Mossink, L. D. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106553 · Cited by 406
Hossain, T., & Okui, R. (2013) · The Review of Economic Studies · doi:10.1093/restud/rdt006 · Cited by 296
Schwardmann, P., & van der Weele, J. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0666-7 · Cited by 215
Mixing Methods: A Bayesian Approach
HUMPHREYS, M., & JACOBS, A. M. (2015) · American Political Science Review · doi:10.1017/s0003055415000453 · Cited by 198
Fooled twice: People cannot detect deepfakes but think they can
Köbis, N. C., Doležalová, B., & Soraperra, I. (2021) · iScience · doi:10.1016/j.isci.2021.103364 · Cited by 193
Metadata
- Authors
- Schlag, K. H., Tremewan, J., & van der Weele, J. J.
- Venue
- Experimental Economics
- Publisher
- Cambridge University Press (CUP)
- Published
- 2015-09
- Volume
- 18
- Issue
- 3
- Pages
- 457-490
- ISSN
- 1386-4157 (ISSN-L)
- 1573-6938
- OpenAlex
- W1883255247
- Cited by
- 224
- References
- 4
- FWCI
- 19.94
- Type
- journal article
- Language
- en
- Open access
- bronze
- Licence
- www.cambridge.org/core/terms (unspecified, 3482 d)
- www.springer.com/tdm (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A penny for your thoughts: a survey of methods for eliciting beliefs. https://app.cochat.ai/papers/10.1007/s10683-014-9416-x
- The paper
- Schlag, K. H., Tremewan, J., & van der Weele, J. J. (2015). A penny for your thoughts: a survey of methods for eliciting beliefs. Experimental Economics. https://doi.org/10.1007/s10683-014-9416-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.