Optimal design of experiments to identify latent behavioral types
Open accessThe DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
Abstract Bayesian optimal experiments that maximize the information gained from collected data are critical to efficiently identify behavioral models. We extend a seminal method for designing Bayesian optimal experiments by introducing two computational improvements that make the procedure tractable: (1) a search algorithm from artificial intelligence that efficiently explores the space of possible design parameters, and (2) a sampling procedure which evaluates each design parameter combination more efficiently. We apply our procedure to a game of imperfect information to evaluate and quantify the computational improvements. We then collect data across five different experimental designs to compare the ability of the optimal experimental design to discriminate among competing behavioral models against the experimental designs chosen by a “wisdom of experts” prediction experiment. We find that data from the experiment suggested by the optimal design approach requires significantly less data to distinguish behavioral models (i.e., test hypotheses) than data from the experiment suggested by experts. Substantively, we find that reinforcement learning best explains human decision-making in the imperfect information game and that behavior is not adequately described by the Bayesian Nash equilibrium. Our procedure is general and computationally efficient and can be applied to dynamically optimize online experiments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“On a Bayesian reading, the expected information gain is exactly the same quantity that underwrites the principles of optimal Bayesian design [134, 76, 163].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [163]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On Information and Sufficiency
Kullback, S., & Leibler, R. A. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729694 · Cited by 20,193
Network Externalities, Competition, and Compatibility
Katz, M. L. (1985) · American Economic Review · Cited by 8,582
Colman (2022) · doi:10.1073/pnas · Cited by 4,370
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 2×
“We recruited participants from the online labor market Amazon Mechanical Turk (AMT) (Horton et al. 2011; Mason and Suri 2012; Paolacci et al. 2010) and ran our experiments on the nodeGame platform for group experimentation online (Balietti 2017), according to the experimental procedure outlined…”
Behavioral Game Theory: Experiments in Strategic Interaction
Camerer, C. F. (2003) · Cited by 3,722
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Games with Incomplete Information Played by “Bayesian” Players, I–III: Part I. The Basic Model
Harsanyi, J. C. (2004) · Management Science · doi:10.1287/mnsc.1040.0270 · Cited by 2,127
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,530
Experience-weighted Attraction Learning in Normal Form Games
Camerer, C., & Hua Ho, T. (1999) · Econometrica · doi:10.1111/1468-0262.00054 · Cited by 1,487
Controlled experiments on the web: survey and practical guide
Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-008-0114-1 · Cited by 690
Research works printed but not linked: 11
- [56] Eytan Bakshy, Lili Dworkin, Brian Karrer, Konstantin Kashin, Benjamin Letham, Ashwin Murthy, and Shaun Singh. 'AE: A domain-agnostic platform for adaptive experimentation'. In: Conference on Neural Information Processing Systems (2018), pp. 1-8. url : eytan . github.io/papers/ae\_workshop.pdf . -
- [14] Paul A. David. 'Clio and the economics of QWERTY'. In: American Economic Review 75.2 (1985), pp. 332-337. url : jstor.org/stable/1805621 . -
- [31] Nando De Freitas, Alex J. Smola, and Masrour Zoghi. 'Exponential regret bounds for Gaussian process bandits with deterministic observations'. In: Proceedings of the 29th International Coference on International Conference on Machine Learning . ICML '12. 2012, pp. 955-962. isbn : 9781450312851. doi : 10.5555/3042573.3042697 . - · doi:10.5555/3042573.3042697
- [40] Ido Erev and Alvin E. Roth. 'Predicting how people play games: Reinforcement learning in games with unique strategy equilibrium'. In: American Economic Review 88.4 (1998), pp. 848881. url : jstor.org/stable/117009 . -
- [18] Drew Fudenberg and Jean Tirole. Game theory . MIT Press, 1991. url : mitpress.mit.edu/ books/game-theory . -
- [2] Theodore P. Hill. 'A statistical derivation of the significant-digit law'. In: Statistical Science 10.4 (1995), pp. 354-363. doi : 10.2307/2246134 . - · doi:10.2307/2246134
- [54] Ron Kohavi and Stefan Thomke. 'The surprising power of online experiments'. In: Harvard Business Review 95.5 (2017), pp. 2-9. url : hbr.org/2017/09/the-surprising-power-ofonline-experiments . -
- [7] Benjamin Letham, Brian Karrer, Guilherme Ottoni, and Eytan Bakshy. 'Constrained Bayesian optimization with noisy experiments'. In: arXiv (2017), pp. 1-20. url : arxiv.org/abs/1706. 07094 . -
- [26] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning . The MIT Press, 2005. isbn : 026218253X. url : gaussianprocess.org/gpml/ . - · DOI · Paper page
- [30] Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias Seeger. 'Gaussian process optimization in the bandit setting: No regret and experimental design'. In: Proceedings of the 27th International Conference on International Conference on Machine Learning . ICML '10. 2010, pp. 1015-1022. doi : 10.5555/3104322.3104451 . - · doi:10.5555/3104322.3104451
- [10] Stephanie W. Wang, Michelle Filiba, and Colin F. Camerer. 'Dynamically optimized sequential experimentation (DOSE) for estimating economic preference parameters'. In: working paper (2010), pp. 1-41. url : https://bit.ly/39iPvkw . -
Cited by
The free energy principle made simpler but not too simple
Friston, K., Da Costa, L., Sajid, N., Heins, C., Ueltzhöffer, K., Pavliotis, G. A., & Parr, T. (2023) · Physics Reports · doi:10.1016/j.physrep.2023.07.001 · Cited by 163
“On a Bayesian reading, the expected information gain is exactly the same quantity that underwrites the principles of optimal Bayesian design [76,134,163].”
Epistemic Communities under Active Inference
Albarracin, M., Demekas, D., Ramstead, M. J. D., & Heins, C. (2022) · Entropy · doi:10.3390/e24040476 · Cited by 60
Metadata
- Authors
- Balietti, S., Klein, B., & Riedl, C.
- Affiliations
- University of Mannheim (DE)
- Northeastern University (US)
- Corresponding author
- Riedl, Christoph
- Venue
- Experimental Economics
- Publisher
- Cambridge University Press (CUP)
- Published
- 2021-09
- Volume
- 24
- Issue
- 3
- Pages
- 772-799
- ISSN
- 1386-4157 (ISSN-L)
- 1573-6938
- OpenAlex
- W3048086061
- Cited by
- 18
- References
- 48
- Citations by year
- 2026: 4 · 2024: 3 · 2023: 3 · 2022: 5 · 2021: 2
- FWCI
- 1.75
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.cambridge.org/core/terms (unspecified, 1290 d)
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Experimental Economics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- MADOC (University of Mannheim) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Office of Naval Research (N00014-16-1-3005, N00014-17-1-2542)
- National Defense Science and Engineering Graduate (NA)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
- Keywords
- optimal experimental design, imperfect information games, search algorithms, sampling efficiency, reinforcement learning, Bayesian Nash equilibrium, human decision-making, online experiments
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Optimal design of experiments to identify latent behavioral types. https://app.cochat.ai/papers/10.1007/s10683-020-09680-w
- The paper
- Balietti, S., Klein, B., & Riedl, C. (2021). Optimal design of experiments to identify latent behavioral types. Experimental Economics. https://doi.org/10.1007/s10683-020-09680-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.