The DOI matches its Crossref record; 20 of the 23 references it cites were found in a registry or an index.
A Convenient Solution: Using MTurk To Sample From Hard-To-Reach Populations
Abstract
We agree with Landers and Behrend's (2015) proposition that Amazon's Mechanical Turk (MTurk) may provide great opportunities for organizational research samples. However, some groups are characteristically difficult to recruit because they are stigmatized or socially disenfranchised (Birman, 2005; Miller, Forte, Wilson, & Greene, 2006; Sullivan & Cain, 2004; see Campbell, Adams, & Patterson, 2008, for a review). These groups may include individuals who have not previously been the focus of much organizational research, such as those of low socioeconomic status; individuals with disabilities; lesbian, gay, bisexual, or transgender (LGBT) individuals; or victims of workplace harassment. As Landers and Behrend (2015) point out, there is an overrepresentation of research using “Western, educated, industrialized, rich, and democratic” participants. It is important to extend research beyond these samples to examine workplace phenomena that are specific to special populations. We contribute to this argument by noting the particular usefulness that MTurk can provide for sampling from hard-to-reach populations, which we characterize as groups that are in the numerical minority in terms of nationwide representation. To clarify, we focus our discussion on populations that are traditionally hard to reach in the context of contemporary organizational research within the United States.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Understanding the small business sector
(4462813), D. S. (1995) · Long Range Planning · doi:10.1016/0024-6301(95)90953-2 · Cited by 4,212
Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.93 · Cited by 2,932 · cited 3×
“…when using data collected from MTurk, such as incorporating manipulation checks, establishing thresholds formissing data, ensuring that there is enough variability in responses, deleting responses with identical IP addresses, and matching key demographic characteristics (Gosling et al., 2004).”
BackgroundMethodInfluentialTurner, C. F., Ku, L. C., Rogers, S., Lindberg, L., Pleck, J. H., & Sonenstein, F. L. (1998) · Science · doi:10.1126/science.280.5365.867 · Cited by 2,048
Landers, R. N., & Behrend, T. S. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.13 · Cited by 714
Gonzalez, J. A., & Denisi, A. (2009) · Journal of Organizational Behavior · doi:10.1002/job.498 · Cited by 429
Mckay, P. F., Avery, D. R., & Morris, M. A. (2009) · Personnel Psychology · doi:10.1111/j.1744-6570.2009.01157.x · Cited by 244
Trans-parency in the workplace: How the experiences of transsexual employees can be improved
Law, C. L., Martinez, L. R., Ruggs, E. N., Hebl, M. R., & Akers, E. (2011) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2011.03.018 · Cited by 165
Sullivan, C. M., & Cain, D. (2004) · Journal of Interpersonal Violence · doi:10.1177/0886260504263249 · Cited by 137
Feitosa, J., Joseph, D. L., & Newman, D. A. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.11.017 · Cited by 121
Substance use behaviors among college students with same-sex and opposite-sex experience
Eisenberg, M. E., & Wechsler, H. (2003) · Addictive Behaviors · doi:10.1016/s0306-4603(01)00286-6 · Cited by 118
Commonalities in grief responding across bereavement and non-bereavement losses
Papa, A., Lancaster, N. G., & Kahler, J. (2014) · Journal of Affective Disorders · doi:10.1016/j.jad.2014.03.018 · Cited by 113
Assessment of suicide risk by computer‐delivered self‐rating questionnaire: preliminary findings
Levine, S., Ancill, R. J., & Roberts, A. P. (1989) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1989.tb01330.x · Cited by 112
The Relationship Between Self‐Efficacy Beliefs and Inventoried Vocational Interests
Lenox, R. A., & Subich, L. M. (1994) · The Career Development Quarterly · doi:10.1002/j.2161-0045.1994.tb00514.x · Cited by 107
Innovative sampling and participant recruitment in sexuality research
McCormack, M. (2014) · Journal of Social and Personal Relationships · doi:10.1177/0265407514522889 · Cited by 101
Ethical Issues in Research With Immigrants and Refugees
Birman, D. (2006) · Sage eBooks · doi:10.4135/9781412986168.n9 · Cited by 95
Protecting Sexual Minority Youth from Research Risks: Conflicting Perspectives
Miller, R. L., Forte, D., Wilson, B. D. M., & Greene, G. J. (2006) · American Journal of Community Psychology · doi:10.1007/s10464-006-9053-4 · Cited by 36
Methodological Challenges of Collecting Evaluation Data From Traumatized Clients/Consumers
Campbell, R., Adams, A. E., & Patterson, D. (2008) · American Journal of Evaluation · doi:10.1177/1098214008320736 · Cited by 24
Tenenbaum, R. Z., Byrne, C. J., & Dahling, J. J. (2013) · Journal of Career Assessment · doi:10.1177/1069072713493981 · Cited by 23
Resource Effects on In-Group Boundary Formation With Regard to Sexual Identity
Vaughn, A. A., Cronan, S. B., & Beavers, A. J. (2014) · Social Psychological and Personality Science · doi:10.1177/1948550614559604 · Cited by 6
Research works printed but not linked: 3
- Birman, 2005, The handbook of ethical research with ethnocultural populations and communities, 155
- Locke, 1995, Method of psychological assessment, self-disclosure, and experiential differences: A study of computer, questionnaire, and interview assessment formats, Journal of Social Behavior and Personality, 10, 255
- Skitka, 2005, The social net: The social psychology of the Internet, 1
Cited by
Sorted by citations
A Tale of Two Sample Sources: Do Results from Online Panel Data and Conventional Data Converge?
Walter, S. L., Seibert, S. E., Goering, D. D., & O’Boyle, E. H. (2018) · Journal of Business and Psychology · doi:10.1007/s10869-018-9552-y · Cited by 593
Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017) · Journal of Business and Psychology · doi:10.1007/s10869-016-9458-5 · Cited by 474
The Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J., & Cho, T. S. (2018) · Journal of Management · doi:10.1177/0149206318811569 · Cited by 368
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319
Data Collection via Online Platforms: Challenges and Recommendations for Future Research
Newman, A., Bavik, Y. L., Mount, M. P., & Shao, B. (2020) · Applied Psychology · doi:10.1111/apps.12302 · Cited by 360
Metadata
- Authors
- Smith, N. A., Sabat, I. E., Martinez, L. R., Weaver, K., & Xu, S.
- Affiliations
- Pennsylvania State University (US)
- George Mason University (US)
- Corresponding author
- Smith, Nicholas A.
- Venue
- Industrial and Organizational Psychology
- Publisher
- Cambridge University Press (CUP)
- Published
- 2015-06
- Volume
- 8
- Issue
- 2
- Pages
- 220-228
- ISSN
- 1754-9426 (ISSN-L)
- 1754-9434
- OpenAlex
- W2424520822
- Cited by
- 190
- References
- 23
- Citations by year
- 2026: 8 · 2025: 13 · 2024: 17 · 2023: 24 · 2022: 24 · 2021: 35 · 2020: 16 · 2019: 23 · 2018: 15 · 2017: 10 · 2016: 4
- FWCI
- 14.55
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Industrial and Organizational Psychology (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › Employer Branding and e-HRM
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- Amazon Mechanical Turk, hard-to-reach populations, online recruitment, stigmatized groups, low socioeconomic status, people with disabilities, LGBT people, workplace harassment, underrepresented populations in research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review