The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Multiple hypothesis testing in experimental economics
Abstract
The analysis of data from experiments in economics routinely involves testing multiple null hypotheses simultaneously. These different null hypotheses arise naturally in this setting for at least three different reasons: when there are multiple outcomes of interest and it is desired to determine on which of these outcomes a treatment has an effect; when the effect of a treatment may be heterogeneous in that it varies across subgroups defined by observed characteristics and it is desired to determine for which of these subgroups a treatment has an effect; and finally when there are multiple treatments of interest and it is desired to determine which treatments have an effect relative to either the control or relative to each of the other treatments. In this paper, we provide a bootstrap-based procedure for testing these null hypotheses simultaneously using experimental data in which simple random sampling is used to assign treatment status to units. Using the general results in Romano and Wolf (Ann Stat 38:598–633, 2010), we show under weak assumptions that our procedure (1) asymptotically controls the familywise error rate—the probability of one or more false rejections—and (2) is asymptotically balanced in that the marginal probability of rejecting any true null hypothesis is approximately equal in large samples. Importantly, by incorporating information about dependence ignored in classical multiple testing procedures, such as the Bonferroni and Holm corrections, our procedure has much greater ability to detect truly false null hypotheses. In the presence of multiple treatments, we additionally show how to exploit logical restrictions across null hypotheses to further improve power. We illustrate our methodology by revisiting the study by Karlan and List (Am Econ Rev 97(5):1774–1793, 2007) of why people give to charitable causes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Testing Statistical Hypotheses
(2021) · Wiley Series in Probability and Statistics · doi:10.1002/9781119243830.ch12 · Cited by 5,106
A sharper Bonferroni procedure for multiple tests of significance
HOCHBERG, Y. (1988) · Biometrika · doi:10.1093/biomet/75.4.800 · Cited by 4,829
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 3×
“We consider four subgroups contained in Karlan and List (2007): red county in a red state, blue county in a red state, red county in a blue state, and blue county in a blue state.”
Scandinavian Journal of Statistics
Rosén, B. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess2351.pub2 · Cited by 3,595
Do Women Shy Away From Competition? Do Men Compete Too Much?
Niederle, M., & Vesterlund, L. (2007) · The Quarterly Journal of Economics · doi:10.1162/qjec.122.3.1067 · Cited by 3,186
Experimental Analysis of Neighborhood Effects
Kling, J. R., Liebman, J. B., & Katz, L. F. (2007) · Econometrica · doi:10.1111/j.1468-0262.2007.00733.x · Cited by 2,719
Anderson, M. L. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000841 · Cited by 2,410 · cited 2×
“(2007), who use a more restrictive resampling-based multiple testing procedure due to Westfall and Young (1993) and Anderson (2008), Heckman et al.”
Performance in competitive environments: gender differences
Gneezy, U., Niederle, M., & Rustichini, A. (2003) · The Quarterly Journal of Economics · doi:10.1162/00335530360698496 · Cited by 1,997
Cited by
Misperceived Social Norms: Women Working Outside the Home in Saudi Arabia
Bursztyn, L., González, A. L., & Yanagizawa-Drott, D. (2020) · American Economic Review · doi:10.1257/aer.20180975 · Cited by 596
The Romano–Wolf multiple-hypothesis correction in Stata
Clarke, D., Romano, J. P., & Wolf, M. (2020) · The Stata Journal: Promoting communications on statistics and Stata · doi:10.1177/1536867x20976314 · Cited by 261
HEATH, D., RINGGENBERG, M. C., SAMADI, M., & WERNER, I. M. (2023) · The Journal of Finance · doi:10.1111/jofi.13250 · Cited by 133
Can Solar Lanterns Improve Youth Academic Performance? Experimental Evidence from Bangladesh
Kudo, Y., Shonchoy, A. S., & Takahashi, K. (2017) · doi:10.1596/1813-9450-7954 · Cited by 14
Metadata
- Authors
- List, J. A., Shaikh, A. M., & Xu, Y.
- Venue
- Experimental Economics
- Publisher
- Cambridge University Press (CUP)
- Published
- 2019-12
- Volume
- 22
- Issue
- 4
- Pages
- 773-793
- ISSN
- 1386-4157 (ISSN-L)
- 1573-6938
- OpenAlex
- W2281690356
- Cited by
- 463
- References
- 52
- FWCI
- 41.45
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multiple hypothesis testing in experimental economics. https://app.cochat.ai/papers/10.1007/s10683-018-09597-5
- The paper
- List, J. A., Shaikh, A. M., & Xu, Y. (2019). Multiple hypothesis testing in experimental economics. Experimental Economics. https://doi.org/10.1007/s10683-018-09597-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.