CoChat
Sign Up
Cited by 23Experimental Behavioral Economics StudiesOptimal Experimental Design MethodsGame Theory and Voting Systems

So you want to run an experiment, now what? Some Simple Rules of Thumb for Optimal Experimental Design

Open access
List, John, Sadoff, Sally, Wagner, Mathis·2010·National Bureau of Economic Research
DOI 10.3386/w15701
Open this paper in CoChat — free account

Abstract

Experimental economics represents a strong growth industry.In the past several decades the method has expanded beyond intellectual curiosity, now meriting consideration alongside the other more traditional empirical approaches used in economics.Accompanying this growth is an influx of new experimenters who are in need of straightforward direction to make their designs more powerful.This study provides several simple rules of thumb that researchers can apply to improve the efficiency of their experimental designs.We buttress these points by including empirical examples from the literature.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3386/w15701

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • When designing a study using the t-test, it is ideal to have the same sample size for the experimental and control groups.

    “When designing a study using the t-test, it is ideal to have the same sample size for the experimental and control groups [5].”

    From More about the basic assumptions of t-test: normality and sample size (2019) · cites this paper as [5]

    The Introduction says equal allocation between treatment and control is optimal for a continuous outcome only when their outcome variances are expected to be equal. Section 3.1 discusses this in the setting of a t-test and says allocation should otherwise reflect the groups’ standard deviations. The claim omits that condition and presents equal sample sizes as generally ideal. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • When the variances of the two groups to be compared are similar, the smallest sample size is acquired when those of the two groups are the same.

    “When the variances of the two groups to be compared are similar, the smallest sample size is acquired when those of the two groups are the same [5].”

    From More about the basic assumptions of t-test: normality and sample size (2019) · cites this paper as [5]

    The Introduction says equal allocation is optimal when the groups’ outcome variances are expected to be equal. Section 3.1 says that when variances are merely similar, equal allocation causes little loss of efficiency—not that it necessarily yields the smallest total sample size. Page numbers are unavailable in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The number of independent questions n required to achieve a Type I error rate alpha and Type II error rate beta with a given Minimum Detectable Effect delta is given by the paper's sample-size formula.

    “Then the number of independent questions n required to achieve a Type I error rate α and Type II error rate β with a given Minimum Detectable Effect δ is”

    From Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (2024) · cites this paper as [14]

    Section 3.1 derives a sample-size formula using the Type I error rate, Type II error rate, and minimum detectable effect, assuming independently drawn treatment and control observations. It gives n as observations per treatment cell, not as a number of independent questions; applying the formula to questions requires assumptions the cited passage does not establish. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 22 references couldn’t be matched to a known paper.

Research works printed but not linked: 9

  1. Camerer, Colin F. and Robin M. Hogarth (1999). "The E¤ects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework," Journal of Risk and Uncertainty, 7-42.
  2. Cochran, W.G. and Cox, G.M. (1950). Experimental Designs. New York, NY: J. Wiley & Sons, Inc.
  3. El-Gamal M. and Palfrey T. (1996). "Economical Experiments: Bayesian E¢ cient Experimental Designs,"International Journal of Game Theory, 25: 495-517.
  4. Goodman S.N. and Berlin J.A. (1994). "The use of predicted con…dence intervals when planning experiments and the misuse of power when interpreting results," An- nals of Internal Medicine; 121:200-6.
  5. Greenwald, A.G. (1976). Within-Subjects Design: To Use or not to Use. Psychological Bulletin, 83, 314-320.
  6. Kish, L. (1965). Survey sampling. New York: Wiley.
  7. Levitt, Steven D. and John A. List (2007). "What do Laboratory Experiments Mea- suring Social Preferences tell us about the Real World," Journal of Economic Per- spectives, 21 (2): 153-174.
  8. Schuirmann, D. J. (1987), "A Comparison of the Two One-sided Tests Procedure and the Power Approach for Assessing the Equivalence of Bioavailability," Journal of Pharmacokinetics and Biopharmaceutics, 15, 657-680.
  9. Wilcox, R. R. (1996). Statistics for the Social Sciences. San Diego, CA: Academic Press.

Cited by

Sorted by citations

Related

Authors
List, J., Sadoff, S., & Wagner, M.
Affiliations
  • National Bureau of Economic Research (US)
  • University of Chicago (US)
  • Boston College (US)
ORCID
List, John
Venue
National Bureau of Economic Research
Publisher
National Bureau of Economic Research
Published
2010-01
DOI
10.3386/w15701
OpenAlex
W2122271870
Cited by
22
References
22
Citations by year
2022: 1 · 2020: 1 · 2019: 2 · 2018: 1 · 2017: 1 · 2016: 2 · 2013: 3 · 2012: 1
Type
report
Language
en
Open access
Locations
  • National Bureau of Economic Research (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Topics
  • Social Sciences › Safety Research › Experimental Behavioral Economics Studies
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Game Theory and Voting Systems
Keywords
experimental design, optimal experimental design, rules of thumb, statistical power, experimental economics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review