The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
Testing the Waters: Behavior across Participant Pools
Abstract
We leverage a large-scale incentivized survey eliciting behaviors from (almost) an entire undergraduate university student population, a representative sample of the US population, and Amazon Mechanical Turk (MTurk) to address concerns about the external validity of experiments with student participants. Behavior in the student population offers bounds on behaviors in other populations, and correlations between behaviors are similar across samples. Furthermore, non-student samples exhibit higher levels of noise. Adding historical lab participation data, we find a small set of attributes over which lab participants differ from non-lab participants. An additional set of lab experiments shows no evidence of observer effects. (JEL C83, D90, D91)
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Measuring individual differences in implicit cognition: the implicit association test
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.6.1464 · Cited by 9,210
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Understanding and using the Implicit Association Test: I. An improved scoring algorithm.
Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.85.2.197 · Cited by 5,815
Gender Differences in Preferences
Croson, R., & Gneezy, U. (2009) · Journal of Economic Literature · doi:10.1257/jel.47.2.448 · Cited by 5,694
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Orne, M. T. (1962) · American Psychologist · doi:10.1037/h0043424 · Cited by 4,041
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
INDIVIDUAL RISK ATTITUDES: MEASUREMENT, DETERMINANTS, AND BEHAVIORAL CONSEQUENCES
Dohmen, T., Falk, A., Huffman, D., Sunde, U., Schupp, J., & Wagner, G. G. (2011) · Journal of the European Economic Association · doi:10.1111/j.1542-4774.2011.01015.x · Cited by 3,896
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Gender differences in risk taking: a meta-analysis
Byrnes, J. P., Miller, D. C., & Schafer, W. D. (1999) · Psychological Bulletin · doi:10.1037/0033-2909.125.3.367 · Cited by 3,723
Do Women Shy Away From Competition? Do Men Compete Too Much?
Niederle, M., & Vesterlund, L. (2007) · The Quarterly Journal of Economics · doi:10.1162/qjec.122.3.1067 · Cited by 3,186
What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?
Levitt, S. D., & List, J. A. (2007) · Journal of Economic Perspectives · doi:10.1257/jep.21.2.153 · Cited by 2,668
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
The Human Problems of an Industrial Civilization
(1934) · Nature · doi:10.1038/134201b0 · Cited by 1,923
cocor: A Comprehensive Solution for the Statistical Comparison of Correlations
Diedenhofen, B., & Musch, J. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121945 · Cited by 1,919
The American Journal of Sociology
F., C. B. (1939) · American Journal of Psychiatry · doi:10.1176/ajp.96.1.252 · Cited by 1,886
Deception: The Role of Consequences
Gneezy, U. (2005) · American Economic Review · doi:10.1257/0002828053828662 · Cited by 1,729
Preferences, property rights, and anonymity in bargaining games.
Hoffman, E., McCabe, K., Shachat, K., & Smith, V. (1994) · Games and Economic Behavior · doi:10.1006/game.1994.1056 · Cited by 1,674
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,529
Are Risk Aversion and Impatience Related to Cognitive Ability?
Dohmen, T., Falk, A., Huffman, D., & Sunde, U. (2010) · American Economic Review · doi:10.1257/aer.100.3.1238 · Cited by 1,332
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314 · cited 2×
“In the parlance of the literature concerned with the reliability of social science findings: the result we find in the CCS sample does not replicate in the MTurk and Representative samples (Camerer et al., 2016; Open Science Collaboration, 2015).”
BackgroundResultAn Experiment on Risk Taking and Evaluation Periods
Gneezy, U., & Potters, J. (1997) · The Quarterly Journal of Economics · doi:10.1162/003355397555217 · Cited by 1,288
Lab Experiments Are a Major Source of Knowledge in the Social Sciences
Falk, A., & Heckman, J. J. (2009) · Science · doi:10.1126/science.1168244 · Cited by 1,033
Chapter 113 Men, Women and Risk Aversion: Experimental Evidence
Eckel, C. C., & Grossman, P. J. (2008) · Handbook of Experimental Economics Results · doi:10.1016/s1574-0722(07)00113-8 · Cited by 924
Huff, C., & Tingley, D. (2015) · Research & Politics · doi:10.1177/2053168015604648 · Cited by 903
Experimental methods: Eliciting risk preferences
Charness, G., Gneezy, U., & Imas, A. (2013) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2012.12.023 · Cited by 791
Erat, S., & Gneezy, U. (2012) · Management Science · doi:10.1287/mnsc.1110.1449 · Cited by 698
Men, Women and Risk Aversion: Experimental Evidence
Eckel, C. C., & Grossman, P. J. (2008) · Cited by 662
Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515
Dictator game giving: Rules of fairness versus acts of kindness.
Bolton, G. E., Katok, E., & Zwick, R. (1998) · International Journal of Game Theory · doi:10.1007/s001820050072 · Cited by 480
(1992) · American Journal of Sociology · doi:10.1086/230046 · Cited by 406
What Motivates Effort? Evidence and Expert Forecasts
DellaVigna, S., & Pope, D. (2018) · The Review of Economic Studies · doi:10.1093/restud/rdx033 · Cited by 392
Overconfidence in Political Behavior
Ortoleva, P., & Snowberg, E. (2015) · American Economic Review · doi:10.1257/aer.20130921 · Cited by 355
Self-citationExperimenting with Measurement Error: Techniques with Applications to the Caltech Cohort Study
Gillen, B., Snowberg, E., & Yariv, L. (2019) · Journal of Political Economy · doi:10.1086/701681 · Cited by 350
Self-citationCamerer, C. F. (2015) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195328325.003.0016 · Cited by 344
Conducting interactive experiments online
Arechar, A. A., Gächter, S., & Molleman, L. (2018) · Experimental Economics · doi:10.1007/s10683-017-9527-2 · Cited by 340
Levitt, S. D., & List, J. A. (2011) · American Economic Journal: Applied Economics · doi:10.1257/app.3.1.224 · Cited by 297
Viewpoint: On the generalizability of lab behaviour to the field
Levitt, S. D., & List, J. A. (2007) · Canadian Journal of Economics/Revue canadienne d'économique · doi:10.1111/j.1365-2966.2007.00412.x · Cited by 289
Artificiality: The tension between internal and external validity in economic experiments
Schram, A. (2005) · Journal of Economic Methodology · doi:10.1080/13501780500086081 · Cited by 260
Experimental subjects are not different
Exadaktylos, F., Espín, A. M., & Brañas-Garza, P. (2013) · Scientific Reports · doi:10.1038/srep01213 · Cited by 257
DO LAB EXPERIMENTS MISREPRESENT SOCIAL PREFERENCES? THE CASE OF SELF-SELECTED STUDENT SAMPLES
Falk, A., Meier, S., & Zehnder, C. (2013) · Journal of the European Economic Association · doi:10.1111/jeea.12019 · Cited by 243
Levitt, S. D., & List, J. A. (2008) · Science · doi:10.1126/science.1153640 · Cited by 226
Laboratory Experiments: Professionals Versus Students
Fréchette, G. R. (2015) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195328325.003.0019 · Cited by 214
Monopsony in Online Labor Markets
Dube, A., Jacobs, J., Naidu, S., & Suri, S. (2020) · American Economic Review: Insights · doi:10.1257/aeri.20180150 · Cited by 209
On the Relationship between Cognitive Ability and Risk Preference
Dohmen, T., Falk, A., Huffman, D., & Sunde, U. (2018) · Journal of Economic Perspectives · doi:10.1257/jep.32.2.115 · Cited by 201
Experiments in economics: External validity and the robustness of phenomena
Guala, F., & Mittone, L. (2005) · Journal of Economic Methodology · doi:10.1080/13501780500342906 · Cited by 192
ON THE EXTERNAL VALIDITY OF LABORATORY TAX COMPLIANCE EXPERIMENTS
Alm, J., Bloomquist, K. M., & McKee, M. (2015) · Economic Inquiry · doi:10.1111/ecin.12196 · Cited by 186
Peer effects on worker output in the laboratory generalize to the field
Herbst, D., & Mas, A. (2015) · Science · doi:10.1126/science.aac9555 · Cited by 185
Common Concerns with MTurk as a Participant Pool
Hauser, D., Paolacci, G., & Chandler, J. (2019) · Handbook of Research Methods in Consumer Psychology · doi:10.4324/9781351137713-17 · Cited by 171
Risk attitudes, randomization to treatment, and self-selection into experiments
Harrison, G. W., Lau, M. I., & Elisabet Rutström, E. (2009) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2008.02.011 · Cited by 168
A comprehensive comparison of students and non-students in classic experimental games
Belot, M. V. K., Duch, R. M., & Miller, L. M. (2015) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2015.02.007 · Cited by 165
The External Validity of Laboratory Experiments: The Misleading Emphasis on Quantitative Effects
Kessler, J. B., & Lise Vesterlund, C. (2015) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195328325.003.0020 · Cited by 149
Social Preferences in the Lab: A Comparison of Students and a Representative Population
Cappelen, A. W., Nygaard, K., Sørensen, E. Ø., & Tungodden, B. (2015) · Scandinavian Journal of Economics · doi:10.1111/sjoe.12114 · Cited by 132
Social Preferences in the Online Laboratory: A Randomized Experiment
Hergueux, J., & Jacquemet, N. (2012) · Experimental Economics · doi:10.1007/s10683-014-9400-5 · Cited by 115
Arechar, A. A., Kraft‐Todd, G., & Rand, D. G. (2017) · Journal of the Economic Science Association · doi:10.1007/s40881-017-0035-0 · Cited by 103
Comparing Corruption in the Laboratory and in the Field in Burkina Faso and in Canada
Armantier, O., & Boly, A. (2012) · The Economic Journal · doi:10.1111/ecoj.12019 · Cited by 98
Is there selection bias in laboratory experiments? The case of social and risk preferences
Cleave, B. L., Nikiforakis, N., & Slonim, R. L. (2012) · Experimental Economics · doi:10.1007/s10683-012-9342-8 · Cited by 89
7. Experimental Economics across Subject Populations
Fréchette, G. R. (2016) · Princeton University Press eBooks · doi:10.1515/9781400883172-008 · Cited by 74
Anonymity and the voluntary provision of public goods
Laury, S. K., Walker, J. M., & Williams, A. W. (1995) · Journal of Economic Behavior & Organization · doi:10.1016/0167-2681(94)00073-n · Cited by 69
Roth, A. E. (2020) · Princeton University Press eBooks · doi:10.2307/j.ctvzsmff5.8 · Cited by 62
(Dis)advantages of student subjects: What is your research question?
Gächter, S. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x10000099 · Cited by 50
Gender differences in giving in the Dictator Game: the role of reluctant altruism
Klinowski, D. (2018) · Journal of the Economic Science Association · doi:10.1007/s40881-018-0058-1 · Cited by 14
Cooperation stability: A representative sample in the labPreprint
Fosgaard, T. R. (2018) · RePEc: Research Papers in Economics · Cited by 6
Cited by
A Single-Item Visual Analogue Scale (VAS) Measure for Assessing Depression among College Students
Huang, Z., Kohler, I. V., & Kämpfen, F. (2020) · Community Mental Health Journal · doi:10.1007/s10597-019-00469-7 · Cited by 51
Looming Large or Seeming Small? Attitudes Towards Losses in a Representative Sample
Chapman, J., Snowberg, E., Wang, S. W., & Camerer, C. (2025) · Review of Economic Studies · doi:10.1093/restud/rdae093 · Cited by 10
Self-citationKang, B., & Wu, S. (2023) · Journal of Experimental Criminology · doi:10.1007/s11292-022-09512-2 · Cited by 6
Metadata
- Authors
- Snowberg, E., & Yariv, L.
- Venue
- American Economic Review
- Publisher
- American Economic Association
- Published
- 2021-02-01
- Volume
- 111
- Issue
- 2
- Pages
- 687-719
- ISSN
- 0002-8282 (ISSN-L)
- OpenAlex
- W2887346961
- Cited by
- 273
- References
- 65
- FWCI
- 58.39
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.