The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions.
Abstract
The authors find that experimental studies using online samples (e.g., MTurk) often violate the assumption of random assignment, because participant attrition-quitting a study before completing it and getting paid-is not only prevalent, but also varies systemically across experimental conditions. Using standard social psychology paradigms (e.g., ego-depletion, construal level), they observed attrition rates ranging from 30% to 50% (Study 1). The authors show that failing to attend to attrition rates in online panels has grave consequences. By introducing experimental confounds, unattended attrition misled them to draw mind-boggling yet false conclusions: that recalling a few happy events is considerably more effortful than recalling many happy events, and that imagining applying eyeliner leads to weight loss (Study 2). In addition, attrition rate misled them to draw a logical yet false conclusion: that explaining one's view on gun rights decreases progun sentiment (Study 3). The authors offer a partial remedy (Study 4) and call for minimizing and reporting experimental attrition in studies conducted on the Web. (PsycINFO Database Record
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,530
Ease of retrieval as information: another look at the availability heuristic
Schwarz, N., Bless, H., Strack, F., Klumpp, G., Rittenauer-Schatka, H., & Simons, A. (1991) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.61.2.195 · Cited by 1,412
Who are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004
Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.105 · Cited by 924
Reips, U.-D. (2000) · Psychological Experiments on the Internet · doi:10.1016/b978-012099980-4/50005-8 · Cited by 460
Political Extremism Is Supported by an Illusion of Understanding
Fernbach, P. M., Rogers, T., Fox, C. R., & Sloman, S. A. (2013) · Psychological Science · doi:10.1177/0956797612464058 · Cited by 396
A Brief History of Web Experimenting
Musch, J., & Reips, U.-D. (2000) · Psychological Experiments on the Internet · doi:10.1016/b978-012099980-4/50004-6 · Cited by 229
Ego depletion and its paradoxical effects on ethical decision making
Yam, K. C., Chen, X.-P., & Reynolds, S. J. (2014) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2014.03.008 · Cited by 136
May, F. B., & Monga, A. (2013) · Journal of Consumer Research · doi:10.1086/673384 · Cited by 119
Self-Interest, Symbolic Politics, and Public Attitudes Toward Gun Control
Wolpert, R. M., & Gimpel, J. G. (1998) · Political Behavior · doi:10.1023/a:1024814624070 · Cited by 112
Determinants of the starting rate and the completion rate in online panel studies 1
Göritz, A. S. (2014) · Wiley eBooks · doi:10.1002/9781118763520.ch7 · Cited by 92
The great escape: The role of self-esteem and self-related cognition in terror management
Wisman, A., Heflick, N., & Goldenberg, J. L. (2015) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.05.006 · Cited by 65
Henderson, M. D. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.03.001 · Cited by 64
Regulatory Focus in Predictions About Others
Woltin, K., & Yzerbyt, V. Y. (2015) · Personality and Social Psychology Bulletin · doi:10.1177/0146167214566188 · Cited by 21
Mediated Models for the Analysis of Confounded Variables and Self-Selected Samples
Birnbaum, M. H., & Mellers, B. A. (1989) · Journal of Educational Statistics · doi:10.3102/10769986014002146 · Cited by 19
Cited by
MTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2021) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 984
How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
Montgomery, J. M., Nyhan, B., & Torres, M. (2018) · American Journal of Political Science · doi:10.1111/ajps.12357 · Cited by 800
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 767
Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments
Thomas, K. A., & Clifford, S. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.038 · Cited by 625
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 588
Metadata
- Authors
- Zhou, H., & Fishbach, A.
- Venue
- Journal of Personality and Social Psychology
- Published
- 2016
- Volume
- 111
- Issue
- 4
- Pages
- 493-504
- ISSN
- 0022-3514 (ISSN-L)
- 1939-1315
- OpenAlex
- W2426479609
- Cited by
- 459
- References
- 20
- FWCI
- 65.69
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.