CoChat

Optimal design of experiments to identify latent behavioral types

Open access
Balietti, Stefano, Klein, Brennan, Riedl, Christoph · 2021 · Experimental Economics

The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.

Abstract

Abstract Bayesian optimal experiments that maximize the information gained from collected data are critical to efficiently identify behavioral models. We extend a seminal method for designing Bayesian optimal experiments by introducing two computational improvements that make the procedure tractable: (1) a search algorithm from artificial intelligence that efficiently explores the space of possible design parameters, and (2) a sampling procedure which evaluates each design parameter combination more efficiently. We apply our procedure to a game of imperfect information to evaluate and quantify the computational improvements. We then collect data across five different experimental designs to compare the ability of the optimal experimental design to discriminate among competing behavioral models against the experimental designs chosen by a “wisdom of experts” prediction experiment. We find that data from the experiment suggested by the optimal design approach requires significantly less data to distinguish behavioral models (i.e., test hypotheses) than data from the experiment suggested by experts. Substantively, we find that reinforcement learning best explains human decision-making in the imperfect information game and that behavior is not adequately described by the Bayesian Nash equilibrium. Our procedure is general and computationally efficient and can be applied to dynamically optimize online experiments.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10683-020-09680-w

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 48 references

Research works printed but not linked: 11

  1. [56] Eytan Bakshy, Lili Dworkin, Brian Karrer, Konstantin Kashin, Benjamin Letham, Ashwin Murthy, and Shaun Singh. 'AE: A domain-agnostic platform for adaptive experimentation'. In: Conference on Neural Information Processing Systems (2018), pp. 1-8. url : eytan . github.io/papers/ae\_workshop.pdf . -
  2. [14] Paul A. David. 'Clio and the economics of QWERTY'. In: American Economic Review 75.2 (1985), pp. 332-337. url : jstor.org/stable/1805621 . -
  3. [31] Nando De Freitas, Alex J. Smola, and Masrour Zoghi. 'Exponential regret bounds for Gaussian process bandits with deterministic observations'. In: Proceedings of the 29th International Coference on International Conference on Machine Learning . ICML '12. 2012, pp. 955-962. isbn : 9781450312851. doi : 10.5555/3042573.3042697 . - · doi:10.5555/3042573.3042697
  4. [40] Ido Erev and Alvin E. Roth. 'Predicting how people play games: Reinforcement learning in games with unique strategy equilibrium'. In: American Economic Review 88.4 (1998), pp. 848881. url : jstor.org/stable/117009 . -
  5. [18] Drew Fudenberg and Jean Tirole. Game theory . MIT Press, 1991. url : mitpress.mit.edu/ books/game-theory . -
  6. [2] Theodore P. Hill. 'A statistical derivation of the significant-digit law'. In: Statistical Science 10.4 (1995), pp. 354-363. doi : 10.2307/2246134 . - · doi:10.2307/2246134
  7. [54] Ron Kohavi and Stefan Thomke. 'The surprising power of online experiments'. In: Harvard Business Review 95.5 (2017), pp. 2-9. url : hbr.org/2017/09/the-surprising-power-ofonline-experiments . -
  8. [7] Benjamin Letham, Brian Karrer, Guilherme Ottoni, and Eytan Bakshy. 'Constrained Bayesian optimization with noisy experiments'. In: arXiv (2017), pp. 1-20. url : arxiv.org/abs/1706. 07094 . -
  9. [26] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning . The MIT Press, 2005. isbn : 026218253X. url : gaussianprocess.org/gpml/ . - · DOI · Paper page
  10. [30] Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias Seeger. 'Gaussian process optimization in the bandit setting: No regret and experimental design'. In: Proceedings of the 27th International Conference on International Conference on Machine Learning . ICML '10. 2010, pp. 1015-1022. doi : 10.5555/3104322.3104451 . - · doi:10.5555/3104322.3104451
  11. [10] Stephanie W. Wang, Michelle Filiba, and Colin F. Camerer. 'Dynamically optimized sequential experimentation (DOSE) for estimating economic preference parameters'. In: working paper (2010), pp. 1-41. url : https://bit.ly/39iPvkw . -

Cited by

See all 18

Metadata

Authors
Balietti, S., Klein, B., & Riedl, C.
Affiliations
  • University of Mannheim (DE)
  • Northeastern University (US)
Corresponding author
Riedl, Christoph
Venue
Experimental Economics
Published
2021-09
Volume
24
Issue
3
Pages
772-799
ISSN
  • 1386-4157 (ISSN-L)
  • 1573-6938
OpenAlex
W3048086061
Cited by
18
References
48
Citations by year
2026: 4 · 2024: 3 · 2023: 3 · 2022: 5 · 2021: 2
FWCI
1.75
Type
journal article
Language
en
Open access
Locations
  • Experimental Economics (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • MADOC (University of Mannheim) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Office of Naval Research (N00014-16-1-3005, N00014-17-1-2542)
  • National Defense Science and Engineering Graduate (NA)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
Keywords
optimal experimental design, imperfect information games, search algorithms, sampling efficiency, reinforcement learning, Bayesian Nash equilibrium, human decision-making, online experiments

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Optimal design of experiments to identify latent behavioral types. https://app.cochat.ai/papers/10.1007/s10683-020-09680-w
The paper
Balietti, S., Klein, B., & Riedl, C. (2021). Optimal design of experiments to identify latent behavioral types. Experimental Economics. https://doi.org/10.1007/s10683-020-09680-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.