Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 224

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

A penny for your thoughts: a survey of methods for eliciting beliefs

Open access
Schlag, Karl H., Tremewan, James, van der Weele, Joël J. · 2015 · Experimental Economics

Abstract

Incentivized methods for eliciting subjective probabilities in economic experiments present the subject with risky choices that encourage truthful reporting. We discuss the most prominent elicitation methods and their underlying assumptions, provide theoretical comparisons and give a new justification for the quadratic scoring rule. On the empirical side, we survey the performance of these elicitation methods in actual experiments, considering also practical issues of implementation such as order effects, hedging, and different ways of presenting probabilities and payment schemes to experimental subjects. We end with a discussion of the trade-offs involved in using incentives for belief elicitation and some guidelines for implementation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10683-014-9416-x

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 4 references

Research works printed but not linked: 96

  1. Allen, F. (1987). Discovering personal probabilities when utility functions are unknown. Management Science, 33(4), 542–544. · DOI
  2. Andersen, S., Fountain, J., Harrison, G. W. and Rutström, E. E. (2014). Estimating Subjective Probabilities. Journal of Risk and Uncertainty, 48(3), 207–220. · DOI
  3. Armantier, O., & Treich, N. (2009). Subjective probabilities in games: A solution to the overbidding puzzle. International Economic Review, 50(4), 1079–1102. · DOI
  4. Armantier, O., & Treich, N. (2013). Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging. European Economic Review, 62, 17–40. · DOI
  5. Beach, L., & Phillips, L. (1967). Subjective probabilities inferred from estimates and bets. Journal of Experimental Psychology, 75(3), 354–359. · doi:10.1037/h0025061
  6. Beach, L. R., & Wise, J. A. (1969). Subjective probability revision and subsequent decisions. Journal of Experimental Psychology, 81(3), 561–565. · doi:10.1037/h0027919
  7. Becker, G., DeGroot, M., & Marschak, J. (1964). Measuring utility by a single-response sequential method. Behavioral Science, 9(3), 226. · DOI · Paper page
  8. Berg, J. E., Rietz, T. A., & Dickhaut, J. W. (2008). On the performance of the lottery procedure for controlling risk preferences. Handbook of Experimental Economics Results, 1, 1087–1097. · doi:10.1016/s1574-0722(07)00115-1
  9. Blanco, M., Engelmann, D., Koch, A. K., & Normann, H.-T. (2010). Belief elicitation in experiments: Is there a hedging problem? Experimental Economics, 13(4), 412–438. · DOI · Paper page
  10. Cervera, J. L., & Muñoz, J. (1996). Proper scoring rules for fractiles. In J. O. Berger, A. P. Dawid, & A. F. M. Smith (Eds.), Bayesian statistics 5 (pp. 513–519). Oxford: Oxford University Press. · doi:10.1093/oso/9780198523567.003.0029
  11. Croson, R. T. A. (2000). Thinking like a game theorist: Factors affecting the frequency of equilibrium play. Journal of Economic Behavior & Organization, 41(3), 299–314. · DOI · Paper page
  12. Dawes, R. M., McTavish, J., & Shaklee, H. (1977). Behavior, communication, and assumptions about other people’s behavior in a commons dilemma situation. Journal of Personality and Social Psychology, 35(1), 1. · doi:10.1037/0022-3514.35.1.1 · Paper page
  13. De Finetti, B. (1965). Methods for discriminating levels of partial knowledge concerning a test item. British Journal of Mathematical and Statistical Psychology, 18(1), 87–123. · DOI · Paper page
  14. De Finetti, B. (1970). Logical foundations and measurement of subjective probability. Acta Psychologica, 34, 129–145. · DOI
  15. De Finetti, B. (1974). Theory of probability (Vol. 1). New York: Wiley.
  16. Delavande, A., Giné, X., & McKenzie, D. (2011). Measuring subjective expectations in developing countries: A critical review and new evidence. Journal of Development Economics, 94(2), 151–163. · DOI · Paper page
  17. Ducharme, W., & Donnell, M. (1973). Intrasubject comparison of four response modes for “subjective probability” assessment. Organizational Behavior and Human Performance, 10, 108–117. · doi:10.1016/0030-5073(73)90007-x
  18. Engelmann, D., & Strobel, M. (2000). The false consensus effect disappears if representative information and monetary incentives are given. Experimental Economics, 260(2000), 241–260. · doi:10.1023/a:1011472501737
  19. Erev, I., Bornstein, G., & Wallsten, T. (1993). The negative effect of probability assessments on decision quality. Organizational Behavior and Human Decision Processes, 55, 78–94. · DOI
  20. Fischer, G. W. (1982). Scoring-rule feedback and the overconfidence syndrome in subjective probability forecasting. Organizational Behavior and Human Performance, 29(3), 352–369. · DOI
  21. Fountain, J., & Harrison, G. W. (2011). What do prediction markets predict? Applied Economics Letters, 18(3), 267–272. · DOI
  22. Friedman, D. (1983). Effective scoring rules for probabilistic forecasts. Management Science, 29(4), 447–454. · DOI
  23. Gächter, S., & Renner, E. (2010). The effects of (incentivized) belief elicitation in public goods experiments. Experimental Economics, 13(3), 364–377. · DOI · Paper page
  24. Garthwaite, P. H., Kadane, J. B., & O’Hagan, A. (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470), 680–701. · DOI · Paper page
  25. Gigerenzer, G., & Hoffrage, U. (1995). How to improve bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684–704. · doi:10.1037/0033-295x.102.4.684 · Paper page
  26. Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. · DOI · Paper page
  27. Guerra, G., & Zizzo, D. J. (2004). Trust responsiveness and beliefs. Journal of Economic Behavior & Organization, 55(1), 25–30. · DOI
  28. Hao, L., & Houser, D. (2012). Belief elicitation in the presence of novice participants: An experimental study. Journal of Risk and Uncertainty, 2, 161–180. · doi:10.1007/s11166-011-9133-1
  29. Harrison, G. W., Martínez-Correa, J., & Swarthout, J. (2014). Eliciting subjective probabilities with binary lotteries. Journal of Economic Behavior and Organization, 101, 128–140. · DOI
  30. Harrison, G. W., Martinez-Correa, J., & Swarthout, T. (2013b). Inducing risk neutral preferences with binary lotteries: A reconsideration. Journal of Economic Behavior and Organization, 94, 145–159. · DOI
  31. Harrison, G. W., & Rutström, E. E. (2009). Expected utility theory and prospect theory: One wedding and a decent funeral. Experimental Economics, 12(2), 133–158. · DOI · Paper page
  32. Heinemann, F., Nagel, R., & Ockenfels, P. (2009). Measuring strategic uncertainty in coordination games. Review of Economic Studies, 76, 181–221. · DOI · Paper page
  33. Holt, C. (2006). Markets, games and strategic behavior. Boston: Pearson/Addison-Wesley.
  34. Holt, C. A., & Laury, S. (2002). Risk aversion and incentive effects. The American Economic Review, 92(5), 1644. · DOI · Paper page
  35. Hossain, T., & Okui, R. (2013). The binarized scoring rule. The Review of Economic Studies, 80(3), 984–1001. · DOI · Paper page
  36. Huck, S., & Weizsäcker, G. (2002). Do players correctly estimate what others do?: Evidence of conservatism in beliefs. Journal of Economic Behavior & Organization, 47(1), 71–85. · doi:10.1016/s0167-2681(01)00170-6
  37. Hurley, T., & Shogren, J. (2005). An experimental comparison of induced and elicited beliefs. Journal of Risk and Uncertainty, 30(2), 169–188. · DOI
  38. Jaffray, J., & Karni, E. (1999). Elicitation of subjective probabilities when the initial endowment is unobservable. Journal of Risk and Uncertainty, 8, 5–20. · doi:10.1023/a:1007706009628
  39. Jenkinson, D. (2005). The elicitation of probabilities—A review of the statistical literature. Manuscript, University of Sheffield.
  40. Jensen, F. A., & Peterson, C. R. (1973). Psychological effects of proper scoring rules. Organizational Behavior and Human Performance, 9(2), 307–317. · DOI
  41. Jose, V. R. R., & Winkler, R. L. (2009). Evaluating quantile assessments. Operations Research, 57(5), 1287–1297. · DOI
  42. Kadane, J., & Winkler, R. (1988). Separating probability elicitation from utilities. Journal of the American Statistical Association, 83(402), 357–363. · doi:10.1080/01621459.1988.10478605
  43. Karni, E. (2009). A mechanism for eliciting probabilities. Econometrica, 77(2), 603–606. · DOI · Paper page
  44. Karni, E., & Safra, Z. (1995). The impossibility of experimental elicitation of subjective probabilities. Theory and Decision, 38, 313–320. · DOI
  45. Koessler, F., Noussair, C., & Ziegelmeyer, A. (2012). Information aggregation and belief elicitation in experimental parimutuel betting markets. Journal of Economic Behavior & Organization, 83(2), 195–208. · DOI
  46. Kothiyal, A., Spinu, V., & Wakker, P. (2011). Comonotonic proper scoring rules to measure ambiguity and subjective beliefs. Journal of Multi-Criteria Decision Analysis, 101, 101–113.
  47. Lambert, N., Pennock, D., & Shoham, Y. (2008). Eliciting properties of probability distributions: The highlights. ACM SIGecom Exchanges, 7(3), 1–5. · DOI
  48. Lindley, D. V., Tversky, A., & Brown, R. V. (1979). On the reconciliation of probability assessments. Journal of the Royal Statistical Society. Series A (General), 142(2), 146. · DOI · Paper page
  49. Lipkus, I., Samsa, G., & Rimer, B. (2001). General performance on a numeracy scale among highly educated samples. Medical Decision Making, 21, 37–44. · DOI · Paper page
  50. Machina, M. J., & Schmeidler, D. (1992). A more robust definition of subjective probability. Econometrica, 60(4), 745–780. · DOI
  51. Manski, C. (2002). Identification of decision rules in experiments on simple games of proposal and response. European Economic Review, 46, 880–891. · DOI
  52. Manski, C. (2004). Measuring expectations. Econometrica, 72(5), 1329–1376. · DOI · Paper page
  53. Matheson, J., & Winkler, R. (1976). Scoring rules for continuous probability distributions. Management Science, 22(10), 1087–1096. · DOI · Paper page
  54. McCarthy, J. (1956). Measures of the value of information. Proceedings of the National Academy of Sciences of the United States of America, 42(9), 654–655. · DOI · Paper page
  55. McKelvey, R., & Page, T. (1990). Public and private information: An experimental study of information pooling. Econometrica, 58(6), 1321–1339. · DOI · Paper page
  56. Miller, N., Resnick, P., & Zeckhauser, R. (2005). Eliciting informative feedback: The peer-prediction method. Management Science, 51(9), 1359–1373. · DOI · Paper page
  57. Nyarko, Y., & Schotter, A. (2002). An experimental study of belief learning using elicited beliefs. Econometrica, 70(3), 971–1005. · DOI · Paper page
  58. Offerman, T., & Palley, A. B. (2013). Lossed in Translation : An Off-the-Shelf Method to Recover Probabilistic Beliefs from Loss-Averse Agents. Manuscript, University of Amsterdam. · doi:10.2139/ssrn.2353500
  59. Offerman, T., Sonnemans, J., & Schram, A. (1996). Value orientations, expectations and voluntary contributions in public goods. The Economic Journal, 106(437), 817–845. · DOI · Paper page
  60. Offerman, T., Sonnemans, J., Van de Kuilen, G., & Wakker, P. P. (2009). A truth serum for non-bayesians. Review of Economic Studies, 76(4), 1461–1489. · DOI · Paper page
  61. Palfrey, T. R., & Wang, S. W. (2009). On eliciting beliefs in strategic games. Journal of Economic Behavior & Organization, 71(2), 98–109. · DOI
  62. Phillips, L. D., & Edwards, W. (1966). Conservatism in a simple probability inference task. Journal of Experimental Psychology, 72(3), 346–354. · DOI · Paper page
  63. Prelec, D. (2004). A bayesian truth serum for subjective data. Science, 306, 462–466. · DOI · Paper page
  64. Price, P. (1998). Effects of a relative-frequency elicitation question on likelihood judgment accuracy: The case of external correspondence. Organizational Behavior and Human Decision Processes, 76(3), 277–297. · DOI
  65. Ramsey, F. (1926). Truth and Probability. In R. B. Braithwaite (ed.), The foundations of mathematics and other logical essays (pp. 156–198). New York (1931): Harcourt.
  66. Roby, T. B. (1964). Belief states: A preliminary empirical study. Technical Documentary Report, Decision Sciences Laboratory. · DOI
  67. Rutström, E. E., & Wilcox, N. T. (2009). Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test. Games and Economic Behavior, 67(2), 616–632. · DOI · Paper page
  68. Savage, L. J. (1954). The foundation of statistics. New York: Wiley.
  69. Savage, L. J. (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336), 783–801. · DOI · Paper page
  70. Schervish, M. (1989). A general method for comparing probability assessors. The Annals of Statistics, 17(4), 1856–1879. · DOI · Paper page
  71. Schlag, K. H. and Tremewan, J. (2014). Simple belief elicitation. SSRN Working paper 2449224. · DOI
  72. Schlag, K. H., & van der Weele, J. J. (2013). Eliciting probabilities, means, medians, variances and covariances without assuming risk neutrality. Theoretical Economics Letters, 03(1), 38–42. · doi:10.4236/tel.2013.31006
  73. Schmalensee, R. (1976). An experimental study of expectation formation. Econometrica, 44(1), 17–41. · DOI
  74. Schotter, A., & Trevino, I. (2014). Belief elicitation in the laboratory. Annual Review of Economics, 6, 103–128. · DOI · Paper page
  75. Schum, D. A., Goldstein, I. L., Howell, W. C., & Southard, J. F. (1967). Subjective probability revisions under several cost-payoff arrangements. Organizational Behavior and Human Performance, 2, 84–104. · doi:10.1016/0030-5073(67)90012-8
  76. Selten, R. (1998). Axiomatic characterization of the quadratic scoring rule. Experimental Economics, 62, 43–62. · doi:10.1023/a:1009957816843 · Paper page
  77. Selten, R., Sadrieh, A., & Abbink, K. (1999). Money does not induce risk neutral behavior, but binary lotteries do even worse. Theory and Decision, 46, 211–249. · DOI
  78. Shuford, E., Albert, A., & Massengill, H. E. (1966). Admissible probability measurement procedures. Psychometrika, 31(2), 125–145. · DOI · Paper page
  79. Smith, A. (2013). Estimating the causal effect of beliefs on contributions in repeated public good games. Experimental Economics, 16(3), 414–425. · DOI
  80. Smith, C. (1961). Consistency in statistical inference and decision. Journal of the Royal Statistical Society. Series B, 23(1), 1–37. · DOI · Paper page
  81. Staël von Holstein, C.-A. S. (1970). Measurement of subjective probability. Acta Psychologica, 34, 146–159. · DOI
  82. Toda, M. (1951). Measurement of intuitive probability by a method of game. Japanese Journal of Psychology, 22, 29–40. · doi:10.4992/jjpsy.22.29
  83. Trautmann, S. T. and van de Kuilen, G. (2014). Belief elicitation: A horse race among truth serums. The Economic Journal. doi: 10.1111/ecoj.12160 . · DOI
  84. Vlek, C. (1973a). The fair betting game as an admissible procedure for assessment of subjective probabilities. British Journal of Mathematical and Statistical Psychology, 26(1), 18–30. · doi:10.1111/j.2044-8317.1973.tb00503.x
  85. Vlek, C. A. J. C. (1973b). Coherence of human judgment in a limited probabilistic environment. Organizational Behavior and Human Performance, 9(460–481), 460–481. · doi:10.1016/0030-5073(73)90065-2
  86. Wang, H., Dash, D., & Druzdzel, M. J. (2002). A method for evaluating elicitation schemes for probabilistic models. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 32(1), 38–43. · DOI
  87. Wang, S. W. (2011). Incentive effects: The case of belief elicitation from individuals in groups. Economics Letters, 111(1), 30–33. · DOI
  88. Whitcomb, K. M., Önkal, D., Benson, P. G., & Curley, S. P. (1993). An evaluation of the reliability of probability judgments across response modes and over time. Journal of Behavioral Decision Making, 6(4), 283–296. · DOI
  89. Winkler, R. (1996). Scoring rules and the evaluation of probabilities. Test, 5(1), 1–60. · DOI · Paper page
  90. Winkler, R., & Murphy, A. (1968). ”Good” probability assessors. Journal of Applied Meteorology, 7, 751. · DOI · Paper page
  91. Winkler, R., & Murphy, A. (1970). Nonlinear utility and the probability score. Journal of Applied Meteorology, 9, 143–148. · doi:10.1175/1520-0450(1970)009<0143:nuatps>2.0.co;2
  92. Winkler, R., & Murphy, A. (1979). The use of probabilities in forecasts of maximum and minimum temperatures. The Meteorological Magazine, 108(1288), 317–329.
  93. Wolfers, J. and Zitzewitz, E. (2006). Interpreting prediction market prices as probabilities. NBER Working Paper 12200. · DOI
  94. Yaniv, I., & Foster, D. (1995). Graininess of judgement under uncertainty: An accuracy-informativeness trade-off. Journal of Experimental Psychology: General, 124(4), 424–432. · DOI · Paper page
  95. Yaniv, I., & Foster, D. (1997). Precision and accuracy of judgmental estimation. Journal of Behavioral Decision Making, 10, 21–32. · DOI · Paper page
  96. Zizzo, D. J. (2009). Experimenter demand effects in economic experiments. Experimental Economics, 13(1), 75–98. · DOI · Paper page

Cited by

See all 224

Metadata

Authors
Schlag, K. H., Tremewan, J., & van der Weele, J. J.
Venue
Experimental Economics
Published
2015-09
Volume
18
Issue
3
Pages
457-490
ISSN
  • 1386-4157 (ISSN-L)
  • 1573-6938
OpenAlex
W1883255247
Cited by
224
References
4
FWCI
19.94
Type
journal article
Language
en
Open access
bronze

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A penny for your thoughts: a survey of methods for eliciting beliefs. https://app.cochat.ai/papers/10.1007/s10683-014-9416-x
The paper
Schlag, K. H., Tremewan, J., & van der Weele, J. J. (2015). A penny for your thoughts: a survey of methods for eliciting beliefs. Experimental Economics. https://doi.org/10.1007/s10683-014-9416-x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.