The Experimentalist Looks Within: Toward an Understanding of Within-Subject Experimental Designs
Open accessAbstract
The traditional approach in experimental economics is to use a between-subject design: the analyst places each unit in treatment or control simultaneously and recovers treatment effects via differencing conditional expectations. Within-subject designs represent a significant departure from this method, as the same unit is observed in both treatment and control conditions sequentially. While many might consider the choice straightforward (always opt for a between-subject design), given the distinct benefits of within-subject designs, I argue that researchers should meticulously weigh the advantages and disadvantages of each design type. In doing so, I propose a categorization for within-subject designs based on the plausibility of recovering an internally valid estimate. In one instance, which I denote as stealth designs, the analyst should unequivocally choose a within-subject design rather than a between-subject design. Institutional subscribers to the NBER working paper series, and residents of developing countries may download this paper without additional charge at www.nber.org.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
One limitation of contingent valuation (CV) studies is that stated willingness to pay (WTP) may overstate true WTP, as some respondents may feel social pressure to indicate support for policies, even if they would not do so in an actual decision-making context.
“One limitation of contingent valuation (CV) studies is that stated willingness to pay (WTP) may overstate true WTP, as some respondents may feel social pressure to indicate support for policies, even if they would not do so in an actual decision-making context.(Diamond and Hausman, 1994;List, 2025;Bursztyn et al., 2025).”
From What is the Best Response? Examining the Impact of Police and Their Alternatives (2025) · cites this paper as List, 2025
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 25
- Bellemare, Charles, Luc Bissonnette, and Sabine Kröger. 2014. "Statistical Power of Within and Between-Subjects Designs in Economic Experiments." SSRN Electronic Journal, January. https://doi.org/10.2139/ssrn.2529895. · DOI
- ---2016. "Simulating Power of Economic Experiments: The powerBBK Package." Journal of the Economic Science Association 2 (2): 157-68. https://doi.org/10.1007/s40881-016- 0028-4. · DOI
- Biesanz, Jeremy C., Stephen G. West, and Oi-Man Kwok. 2003. "Personality Over Time: Methodological Approaches to the Study of Short-Term and Long-Term Development and Change." Journal of Personality 71 (6): 905-42. https://doi.org/10.1111/1467- 6494.7106002. · DOI
- Charness, Gary, Uri Gneezy, and Michael A. Kuhn. 2012. "Experimental Methods: Between- Subject and within-Subject Design." Journal of Economic Behavior & Organization 81 (1): 1-8. https://doi.org/10.1016/j.jebo.2011.08.009. · DOI · Paper page
- Frederick, S.F. and Fischhoff, B., 1998. Scope (in) sensitivity in elicited valuations. Risk Decision and Policy, 3(2), pp.109-123.
- Gneezy, Uri. 2005. "Deception: The Role of Consequences." American Economic Review 95 (1): 384-94. https://doi.org/10.1257/0002828053828662. · DOI
- Greenwald, A.G., 1976. Within-subjects designs: To use or not to use?. Psychological Bulletin, 83(2), p.314.
- Grice, G. Robert. 1966. "Dependence of empirical laws upon the source of experimental variation." Psychological Bulletin, 66 (1966), pp. 488-498
- Hsee, C.K., 1996. The evaluability hypothesis: An explanation for preference reversals between joint and separate evaluations of alternatives. Organizational behavior and human decision processes, 67(3), pp.247-257.
- Isaac, R.M. and Walker, J.M., 1988a. Group size effects in public goods provision: The voluntary contributions mechanism. The Quarterly Journal of Economics, 103(1), pp.179-199.
- ---1988b. Communication and free-riding behavior: The voluntary contribution mechanism. Economic inquiry, 26(4), pp.585-608.
- Levitt, Steven D., and John A. List. 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?" Journal of Economic Perspectives 21 (2): 153-74. https://doi.org/10.1257/jep.21.2.153. · DOI · Paper page
- Levitt, Steven D., and John A. List. 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?" Journal of Economic Perspectives 21 (2): 153-74. https://doi.org/10.1257/jep.21.2.153. · DOI · Paper page
- ---2009. "Field Experiments in Economics: The Past, the Present, and the Future." European Economic Review 53 (1): 1-18. https://doi.org/10.1016/j.euroecorev.2008.12.001. · DOI
- List, John A. 2004. "The Nature and Extent of Discrimination in the Marketplace: Evidence from the Field*." The Quarterly Journal of Economics 119 (1): 49-89. https://doi.org/10.1162/003355304772839524. · DOI
- ---2002. Preference reversals of a different kind: The "more is less" phenomenon. American Economic Review, 92(5), pp.1636-1643.
- ---2024. Optimally generate policy-based evidence before scaling. Nature, 626(7999), pp.491-499.
- ---2025. Experimental Economics: Theory and Practice. The University of Chicago Press.
- Neyman, J. 1923 trans. 1990. "On the application of probability theory to agricultural experiments. Essay on principles. Section 9." Statistical Science 5 (4): 465-472. https://www.jstor.org/stable/2245382.
- Poulton, E.C., 1973. Unwanted range effects from using within-subject experimental designs. Psychological Bulletin, 80(2), p.113.
- Rodemeier, Matthias. 2023. "Willingness to Pay for Carbon Mitigation: Field Evidence from the Market for Carbon Offsets." SSRN Scholarly Paper. Rochester, NY. Available at SSRN: https://papers.ssrn.com/abstract=4360822.
- Rosenthal, R., 1976. Experimenter effects in behavioral research. (2nd ed.), Wiley, New York
- Rubin, Donald B. 1974. "Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies." Journal of Educational Psychology 66 (5): 688-701. https://doi.org/10.1037/h0037350. · DOI · Paper page
- ---. 1975. "Bayesian Inference for Causality: The Importance of Randomization." The Proceedings of the Social Statistics Section of the American Statistical Association, 233- 239.
- ---. 1978. "Bayesian Inference for Causal Effects: The Role of Randomization." The Annals of Statistics 6 (1): 34-58. https://doi.org/10.1214/aos/1176344064. · DOI · Paper page
Cited by
Sorted by citations
What is the Best Response? Examining the Impact of Police and Their Alternatives
Ba, B., Chen, P., Cheng, T., Eies, M., & Holz, J. (2025) · National Bureau of Economic Research · doi:10.3386/w34344 · Cited by 2 · see citing works
- Authors
- List, J.
- Affiliations
- University of Chicago (US)
- Corresponding author
- List, John
- ORCID
- List, John
- Venue
- Review of Behavioral Economics
- Publisher
- National Bureau of Economic Research
- Published
- 2025-02
- Pages
- 1-21
- ISSN
- 2326-6198 (ISSN-L)
- OpenAlex
- W4407312719
- Cited by
- 12
- References
- 0
- Citations by year
- 2026: 3 · 2025: 9
- FWCI
- 8.22
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Review of Behavioral Economics (publishedVersion)
- National Bureau of Economic Research (publishedVersion)
- SSRN Electronic Journal (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Safety Research › Experimental Behavioral Economics Studies
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Keywords
- within-subjects design, between-subjects design, experimental design, treatment effects, internal validity, stealth design, design decisions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
