The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Data collection from online platforms, such as Amazon’s Mechanical Turk (MTurk), has become popular in clinical research. However, there are also concerns about the representativeness and the quality of these data for clinical studies. The present work explores these issues in the specific case of major depression. Analyses of two large data sets gathered from MTurk (Sample 1: N = 2,692; Sample 2: N = 2,354) revealed two major findings: First, failing to screen for inattentive and fake respondents inflates the rates of major depression artificially and significantly (by 18.5%–27.5%). Second, after cleaning the data sets, depression in MTurk is still 1.6 to 3.6 times higher than general population estimates. Approximately half of this difference can be attributed to differences in the composition of MTurk samples and the general population (i.e., sociodemographics, health, and physical activity lifestyle). Several explanations for the other half are proposed, and practical data-quality tools are provided.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PHQ-9: Validity of a brief depression severity measure
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2001.016009606.x · Cited by 44,910
A Brief Measure for Assessing Generalized Anxiety Disorder: The GAD-7
Spitzer, R. L., Kroenke, K., Williams, J. B. W., & Löwe, B. (2006) · Archives of Internal Medicine · doi:10.1001/archinte.166.10.1092 · Cited by 31,765
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Spitzer, R. L. (1999) · JAMA · doi:10.1001/jama.282.18.1737 · Cited by 9,802
Beck, A. T., Steer, R. A., & Brown, G. (2011) · PsycTESTS Dataset · doi:10.1037/t00742-000 · Cited by 7,535
UCLA loneliness scale (version 3): reliability, validity, and factor structure
Russell, D. W. (1996) · Journal of Personality Assessment · doi:10.1207/s15327752jpa6601_2 · Cited by 5,800
Nolen-Hoeksema, S., Wisco, B. E., & Lyubomirsky, S. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00088.x · Cited by 5,172
Nolen-Hoeksema, S., & Morrow, J. (1991) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.61.1.115 · Cited by 3,276
Sensitive questions in surveys.
Tourangeau, R., & Yan, T. (2007) · Psychological Bulletin · doi:10.1037/0033-2909.133.5.859 · Cited by 3,057 · cited 1×
“The stigma could inhibit depressed individuals from sharing personal information in face-to-face interactions (de Leeuw, 1992; Tourangeau & Yan, 2007), which is the standard method of data collection in the NHANES.”
MethodKessler, R. C., Petukhova, M., Sampson, N. A., Zaslavsky, A. M., & Wittchen, H. (2012) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1359 · Cited by 2,947
Cacioppo, J. T., Hughes, M. E., Waite, L. J., Hawkley, L. C., & Thisted, R. A. (2006) · Psychology and Aging · doi:10.1037/0882-7974.21.1.140 · Cited by 2,380 · cited 1×
“It is possible that individuals who work for online data collection forums for extended periods during a day are at increased risk of developing feelings of loneliness, which is a well-documented risk factor for developing depressive symptoms (Cacioppo et al., 2006).”
BackgroundData collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Developing a measure of loneliness
Russell, D., Peplau, L. A., & Ferguson, M. L. (1978) · Journal of Personality Assessment · doi:10.1207/s15327752jpa4203_11 · Cited by 2,089
Twenge, J. M., Joiner, T. E., Rogers, M. L., & Martin, G. N. (2018) · Clinical Psychological Science · doi:10.1177/2167702617723376 · Cited by 1,705
Elhai, J. D., Dvorak, R. D., Levine, J. C., & Hall, B. J. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.08.030 · Cited by 1,689 · cited 1×
“Likewise, in a recent systematic review, Elhai, Dvorak, Levine, and Hall (2017) reported that problematic smartphone use is consistently associated with depressive symptoms (Elhai et al., 2017).”
BackgroundPrevalence and correlates of depression, anxiety, and suicidality among university students
Eisenberg, D., Gollust, S. E., Golberstein, E., & Hefner, J. L. (2007) · American Journal of Orthopsychiatry · doi:10.1037/0002-9432.77.4.534 · Cited by 1,592
Methods for the detection of carelessly invalid responses in survey data
Curran, P. G. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.07.006 · Cited by 1,356
Validation of PHQ-2 and PHQ-9 to Screen for Major Depression in the Primary Care Population
Arroll, B., Goodyear-Smith, F., Crengle, S., Gunn, J., Kerse, N., Fishman, T., Falloon, K., & Hatcher, S. (2010) · The Annals of Family Medicine · doi:10.1370/afm.1139 · Cited by 1,279 · cited 2×
“(2018) administered a two-item screening tool for depression (Patient Health Questionnaire–2, or PHQ-2; Arroll et al., 2010) to a sample of 591 young MTurk workers (age < 50).”
MethodUsing Mechanical Turk to Study Clinical Populations
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013) · Clinical Psychological Science · doi:10.1177/2167702612469015 · Cited by 1,139 · cited 3×
“Previous studies reported that approximately 50% of MTurk users suffer from clinical levels of social anxiety (Arditte et al., 2016; Shapiro et al., 2013).”
BackgroundMethodResultConducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
LOWE, B. (2004) · Journal of Affective Disorders · doi:10.1016/s0165-0327(02)00237-9 · Cited by 1,060
Fried, E. I., & Nesse, R. M. (2015) · Journal of Affective Disorders · doi:10.1016/j.jad.2014.10.010 · Cited by 1,043
Ruminative coping with depressed mood following loss.
Nolen-Hoeksema, S., Parker, L. E., & Larson, J. (1994) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.67.1.92 · Cited by 932
Beesdo, K., Bittner, A., Pine, D. S., Stein, M. B., Höfler, M., Lieb, R., & Wittchen, H.-U. (2007) · Archives of General Psychiatry · doi:10.1001/archpsyc.64.8.903 · Cited by 731 · cited 1×
“One such subgroup may be individuals who suffer from social anxiety, which is a dominant risk factor for major depression (odds ratio = 2.9; Kessler, Stang, Wittchen, Stein, & Walters, 1999) and often precedes the onset of the depressive episode (Beesdo et al., 2007).”
BackgroundInfluentialThe 52 symptoms of major depression: lack of content overlap among seven common depression scales
Fried, E. I. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.10.019 · Cited by 649 · cited 1×
“This difference in diagnoses is partially because different self-report questionnaires may result in different clusters of symptoms (Fried, 2017) and fail to encompass the heterogenic psychosocial nature of depression (Fried & Nesse, 2015).”
BackgroundSartorius, N., Üstün, T. B., Lecrubier, Y., & Wittchen, H.-U. (1996) · British Journal of Psychiatry · doi:10.1192/s0007125000298395 · Cited by 541
Mini-SPIN: A brief screening assessment for generalized social anxiety disorder
Connor, K. M., Kobak, K. A., Churchill, L. E., Katzelnick, D., & Davidson, J. R. T. (2001) · Depression and Anxiety · doi:10.1002/da.1055 · Cited by 537
The shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506 · cited 1×
“(2019). The Turker blues: Hidden factors behind increased depression rates among Amazon’s Mechanical Turkers.”
BackgroundImpact of major depression on chronic medical illness
Katon, W., & Ciechanowski, P. (2002) · Journal of Psychosomatic Research · doi:10.1016/s0022-3999(02)00313-6 · Cited by 485 · cited 1×
“Even depression, which is considered a rather common mental condition, is still relatively rare in random samples recruited from the general population (Katon & Ciechanowski, 2002), and using MTurk can facilitate recruitment efforts and provide researchers with easy access to peripheral populations.”
BackgroundKESSLER, R. C., STANG, P., WITTCHEN, H.-U., STEIN, M., & WALTERS, E. E. (1999) · Psychological Medicine · doi:10.1017/s0033291799008375 · Cited by 434
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304
Schoofs, H., Hermans, D., & Raes, F. (2010) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-010-9182-9 · Cited by 261
Social anxiety and internet use – a meta-analysis: what do we know? What are we missing?
Prizant-Passal, S., Shechner, T., & Aderka, I. M. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.04.003 · Cited by 227 · cited 1×
“Social anxiety has been found to be associated with preferences of computermediated over face-to-face interactions (Lee & Stapinski, 2012; Prizant-Passal, Shechner, and Aderka, 2016).”
Background
Cited by
Data Collection via Online Platforms: Challenges and Recommendations for Future Research
Newman, A., Bavik, Y. L., Mount, M., & Shao, B. (2021) · Applied Psychology · doi:10.1111/apps.12302 · Cited by 306
Kujawa, A., Green, H., Compas, B. E., Dickey, L., & Pegg, S. (2020) · Depression and Anxiety · doi:10.1002/da.23109 · Cited by 278
Park, C. L., Finkelstein-Fox, L., Russell, B. S., Fendrich, M., Hutchison, M., & Becker, J. (2021) · American Psychologist · doi:10.1037/amp0000813 · Cited by 127
Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup
Lu, L., Neale, N., Line, N. D., & Bonn, M. (2022) · Cornell Hospitality Quarterly · doi:10.1177/19389655211025475 · Cited by 94
“Workers with suspicious IP addresses are also much less likely to pass attention checks (e.g., Ophir et al., 2020; S. M. Smith et al., 2016).”
BackgroundMethodWorker Health and Well-Being in the Gig Economy: A Proposed Framework and Research Agenda
Keith, M. G., Harms, P. D., & Long, A. C. (2020) · Entrepreneurial and Small Business Stressors, Experienced Stress, and Well-Being · doi:10.1108/s1479-355520200000018002 · Cited by 72
Metadata
- Authors
- Ophir, Y., Sisso, I., Asterhan, C. S. C., Tikochinski, R., & Reichart, R.
- Venue
- Clinical Psychological Science
- Publisher
- SAGE Publications
- Published
- 2019-10-02
- Volume
- 8
- Issue
- 1
- Pages
- 65-83
- ISSN
- 2167-7026 (ISSN-L)
- 2167-7034
- OpenAlex
- W2978801451
- Cited by
- 126
- References
- 33
- FWCI
- 20.42
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.