The DOI matches its Crossref record; 20 of the 23 references it cites were found in a registry or an index.
A Convenient Solution: Using MTurk To Sample From Hard-To-Reach Populations
Abstract
We agree with Landers and Behrend's (2015) proposition that Amazon's Mechanical Turk (MTurk) may provide great opportunities for organizational research samples. However, some groups are characteristically difficult to recruit because they are stigmatized or socially disenfranchised (Birman, 2005; Miller, Forte, Wilson, & Greene, 2006; Sullivan & Cain, 2004; see Campbell, Adams, & Patterson, 2008, for a review). These groups may include individuals who have not previously been the focus of much organizational research, such as those of low socioeconomic status; individuals with disabilities; lesbian, gay, bisexual, or transgender (LGBT) individuals; or victims of workplace harassment. As Landers and Behrend (2015) point out, there is an overrepresentation of research using “Western, educated, industrialized, rich, and democratic” participants. It is important to extend research beyond these samples to examine workplace phenomena that are specific to special populations. We contribute to this argument by noting the particular usefulness that MTurk can provide for sampling from hard-to-reach populations, which we characterize as groups that are in the numerical minority in terms of nationwide representation. To clarify, we focus our discussion on populations that are traditionally hard to reach in the context of contemporary organizational research within the United States.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Understanding the small business sector
(4462813), D. S. (1995) · Long Range Planning · doi:10.1016/0024-6301(95)90953-2 · Cited by 4,212
Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.93 · Cited by 2,932 · cited 3×
“…when using data collected from MTurk, such as incorporating manipulation checks, establishing thresholds formissing data, ensuring that there is enough variability in responses, deleting responses with identical IP addresses, and matching key demographic characteristics (Gosling et al., 2004).”
BackgroundMethodInfluentialTurner, C. F., Ku, L. C., Rogers, S., Lindberg, L., Pleck, J. H., & Sonenstein, F. L. (1998) · Science · doi:10.1126/science.280.5365.867 · Cited by 2,048
Landers, R. N., & Behrend, T. S. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.13 · Cited by 714
Gonzalez, J. A., & Denisi, A. (2009) · Journal of Organizational Behavior · doi:10.1002/job.498 · Cited by 429
Mckay, P. F., Avery, D. R., & Morris, M. A. (2009) · Personnel Psychology · doi:10.1111/j.1744-6570.2009.01157.x · Cited by 244
Trans-parency in the workplace: How the experiences of transsexual employees can be improved
Law, C. L., Martinez, L. R., Ruggs, E. N., Hebl, M. R., & Akers, E. (2011) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2011.03.018 · Cited by 165
Sullivan, C. M., & Cain, D. (2004) · Journal of Interpersonal Violence · doi:10.1177/0886260504263249 · Cited by 137
Feitosa, J., Joseph, D. L., & Newman, D. A. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.11.017 · Cited by 121
Research works printed but not linked: 3
- Birman, 2005, The handbook of ethical research with ethnocultural populations and communities, 155
- Locke, 1995, Method of psychological assessment, self-disclosure, and experiential differences: A study of computer, questionnaire, and interview assessment formats, Journal of Social Behavior and Personality, 10, 255
- Skitka, 2005, The social net: The social psychology of the Internet, 1
Cited by
Sorted by citations
Items 101–190 of 190
Investigating the role of artificial intelligence in the US criminal justice system
Vo, A., & Plachkinova, M. (2023) · Journal of Information Communication and Ethics in Society · doi:10.1108/jices-11-2022-0101 · Cited by 14
Lefler, E. K., Tabler, J., Abu-Ramadan, T. M., Stevens, A. E., Serrano, J. W., Shelton, C. R., & Hartung, C. M. (2023) · Psychological Reports · doi:10.1177/00332941231199959 · Cited by 12
Lu, X., Wang, W., & Xu, H. (2020) · Public Management Review · doi:10.1080/14719037.2020.1864014 · Cited by 10
An Empirical Study on Type Annotations
Ore, J., Detweiler, C., & Elbaum, S. G. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3439775 · Cited by 10
Said, M., Soria, J., & Stathopoulos, A. (2023) · Transportation Research Record · doi:10.1177/03611981221147520 · Cited by 6
Generalizability of Effect Sizes Within Aviation Research: More Samples Are Needed
Mehta, R., Rice, S. C., Winter, S. R., Spence, T. B., Edwards, M., & Candelaria-Oquendo, K. (2019) · International Journal of Aviation Aeronautics and Aerospace · doi:10.15394/ijaaa.2019.1404 · Cited by 10
Hussain, U., & Cunningham, G. B. (2023) · Cogent Social Sciences · doi:10.1080/23311886.2023.2244839 · Cited by 11
An Examination of Gain- and Loss-Framed Messaging on Smart Home Security Training Programs
Plachkinova, M., & Menard, P. (2019) · Information Systems Frontiers · doi:10.1007/s10796-019-09970-6 · Cited by 9
Shortland, N. D., McGarry, P., THOMPSON, L. A., Stevens, C., & Alison, L. J. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.674694 · Cited by 9
The Sins of the Parents Are to Be Laid Upon the Children: Biased Humans, Biased Data, Biased Models
Osborne, M. R., Omrani, A., & Dehghani, M. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916231180099 · Cited by 9
Cristofaro, V. D., Pellegrini, V., Livi, S., & van Zomeren, M. (2022) · European Journal of Social Psychology · doi:10.1002/ejsp.2853 · Cited by 8
The Relationship of Employment Scarcity and Perceived Threat With Ageist and Sexist Attitudes
Ospina, J. H., Cleveland, J. N., & Gibbons, A. M. (2019) · Work Aging and Retirement · doi:10.1093/workar/waz003 · Cited by 8
Status Threat, Social Concerns, and Conservative Media: A Look at White America and the Alt-Right
Isom, D. A., Boehme, H. M., Mikell, T. C., Chicoine, S., & Renner, M. (2021) · Societies · doi:10.3390/soc11030072 · Cited by 8
Who spoils the barrel? Negative spillover effect on competitor brands during food crises
Seo, S., & Jang, S. S. (2021) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2021.103047 · Cited by 8
Predictors of Health-Related Quality of Life Among African American Men
Wippold, G. M., & Frary, S. G. (2021) · Journal of Racial and Ethnic Health Disparities · doi:10.1007/s40615-021-01151-z · Cited by 7
Diamond-Welch, B. K., Hellwege, J. M., & Mann, O. (2018) · Journal of Interpersonal Violence · doi:10.1177/0886260518792241 · Cited by 7
Minority stress and sexual minorities of colour: the mediating role of mastery
Marshall, R. A., Vargas, E. A., & Mahalingam, R. (2021) · Psychology and Sexuality · doi:10.1080/19419899.2021.1903535 · Cited by 8
Kim, A. C. H., Du, J., & Andrew, D. P. S. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.769930 · Cited by 8
Networks of ADHD and SCT Symptoms in Emerging Adulthood
Lee, C. A., Goh, P. K., Shelton, C. R., & Hartung, C. M. (2021) · Journal of Attention Disorders · doi:10.1177/10870547211045743 · Cited by 7
Martinez, L. R., Smith, N. A., Snoeyink, M. J., Noone, B. M., & Shockley, A. (2022) · Journal of Community Psychology · doi:10.1002/jcop.22849 · Cited by 7
Said, M., Soria, J., & Stathopoulos, A. (2021) · arXiv.org · doi:10.48550/arxiv.2108.02324 · Cited by 4
Factors Affecting Autistic Students’ School Motivation
Lebenhagen, C., & Dynia, J. M. (2024) · Education Sciences · doi:10.3390/educsci14050527 · Cited by 6
Merkuž, A., & Mihelič, K. K. (2023) · SAGE Open · doi:10.1177/21582440231155152 · Cited by 6
Peng, Y., Xu, X. V., Ma, J., & Zhang, W. (2020) · Occupational Health Science · doi:10.1007/s41542-020-00074-1 · Cited by 7
Core Beliefs Disruption in the Context of an Election: Implications for Subjective Well-Being
Zhu, X., & Neupert, S. D. (2021) · Psychological Reports · doi:10.1177/00332941211021347 · Cited by 6
Xu, S. T., Martinez, L. R., & Hoof, H. B. V. (2021) · Cornell Hospitality Quarterly · doi:10.1177/19389655211022661 · Cited by 6
Livingston, T. N., & Rerick, P. O. (2023) · Sexes · doi:10.3390/sexes4030022 · Cited by 5
Abrams, A. L., Reavy, R., & Linden‐Carmichael, A. N. (2022) · Substance Use & Misuse · doi:10.1080/10826084.2022.2120362 · Cited by 5
Breaking the Chains of Red Tape: Social Media’s Strategic Role in Fostering Innovation
Hijal‐Moghrabi, I., & Suh, J. (2025) · Public Performance & Management Review · doi:10.1080/15309576.2025.2510982 · Cited by 5
A comparative assessment of household power failure coping strategies in three American cities
Andresen, A. X., Kurtz, L. C., Chakalian, P. M., Hondula, D. M., Meerow, S., & Gall, M. (2024) · Energy Research & Social Science · doi:10.1016/j.erss.2024.103573 · Cited by 4
Au, C. H., Ho, K. K. W., Law, K. M. Y., & Chiu, D. K. W. (2024) · Electronic Markets · doi:10.1007/s12525-024-00721-3 · Cited by 6
Rivera, Z., Simon, V. A., & Partridge, T. (2022) · The Journal of Sex Research · doi:10.1080/00224499.2022.2091100 · Cited by 4
Wang, X., Peng, Y., Xu, X. V., Arenare, E., & Zhang, W. (2023) · Occupational Health Science · doi:10.1007/s41542-023-00152-0 · Cited by 4
DeGhetto, K., Russell, Z. A., & McAllister, C. P. (2023) · Society and Business Review · doi:10.1108/sbr-10-2022-0262 · Cited by 4
LGBTQ access to generalized youth development programs
Murray, K. (2023) · Journal of Agricultural Education · doi:10.5032/jae.v64i4.2464 · Cited by 4
López‐Aybar, L., Gonzales, L., & Dhillon, A. (2024) · Journal of Community Psychology · doi:10.1002/jcop.23132 · Cited by 5
Ringler, N. (2021) · Current Psychology · doi:10.1007/s12144-021-01853-x · Cited by 4
Empirical Methods in the Social Science for Researching Socially Interactive Agents
Pütten, A. M. R.-V. D., & Abrams, A. M. H. (2021) · ACM eBooks · doi:10.1145/3477322.3477325 · Cited by 4
Reinvesting in equity crowdfunding: the case of digital workers
Makki, R. F. A., & Almazor, E. V. H. (2022) · Journal of Economics Finance and Administrative Science · doi:10.1108/jefas-07-2021-0116 · Cited by 4
Self-Sexualization in Relation to Sexual Harassment and Body Shame
Choi, D., Kang, J. M., Kim, J. H., & Lee, H. K. (2023) · Sexuality & Culture · doi:10.1007/s12119-023-10096-w · Cited by 4
Li, Y., & Samp, J. A. (2022) · Journal of Social and Personal Relationships · doi:10.1177/02654075221098420 · Cited by 3
Examining the reasons why individuals want and do not want a career in policing
Morrow, W., Vickovic, S., & Whitehouse, E. (2025) · Criminal justice studies · doi:10.1080/1478601x.2025.2531760 · Cited by 2
Being What you Eat: the Impact of Workplace Cafeteria Food on Employee Attitudes
Smith, N. A., Martinez, L. R., & Gettle, L. S. (2020) · Occupational Health Science · doi:10.1007/s41542-020-00069-y · Cited by 5
Sexual and Gender Minorities’ Positive and Negative Experiences with Law Enforcement
Goodman, J. A., Israel, T., Avellar, T. R., Delucio, K., Harkness, A., & Bettergarcia, J. N. (2022) · Journal of Police and Criminal Psychology · doi:10.1007/s11896-022-09496-9 · Cited by 6
Problem Drinking, Perceived Burden, Depression, and Suicide Ideation: A Conceptual Model
Nichols, Z. C., Lee, U., Mills, D. J., & Comiskey, G. E. (2022) · Alcoholism Treatment Quarterly · doi:10.1080/07347324.2022.2149373 · Cited by 3
Can Stablecoins Foster Cryptocurrencies Adoption?
Au, C. H., Hsu, W. S., Shieh, P.-H., & Yue, L. (2023) · Journal of Computer Information Systems · doi:10.1080/08874417.2023.2214103 · Cited by 7
Jia, R., Steelman, Z., & Jia, H. H. (2023) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2023.673 · Cited by 3
Jia, R., Reich, B. H., & Jia, H. H. (2016) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2016.06.001 · Cited by 3
Unmasking survey fraud: investigating data quality issues in an MTurk sample
Maline, M. N., & Polonijo, A. N. (2025) · International Journal of Social Research Methodology · doi:10.1080/13645579.2025.2601994 · Cited by 2
White “Victim” Ideology and Online Aggression: A Look at Gender, Extremism, and the Dark Triad
Isom, D. A., & Hubbard, K. (2024) · Crime & Delinquency · doi:10.1177/00111287241264240 · Cited by 2
Exploring the Gender Roles and Parenting Attitudes of LGBTQ Parents: Implications for Counselors
Speciale, M., Lamar, M. R., & Erickson, A. J. (2024) · Journal of LGBTQ Issues in Counseling · doi:10.1080/26924951.2024.2355876 · Cited by 2
Tollini, C., & Diamond-Welch, B. K. (2021) · Sexuality & Culture · doi:10.1007/s12119-021-09872-3 · Cited by 2
Donahue, J. M., Simone, M., Scharmer, C., Anderson, L. M., & Anderson, D. A. (2020) · Eating and Weight Disorders - Studies on Anorexia Bulimia and Obesity · doi:10.1007/s40519-020-01035-x · Cited by 2
Sojo, V. E., & Wheeler, M. A. (2024) · Cambridge University Press eBooks · doi:10.1017/9781009000796.007 · Cited by 2
Online Prescriptions to HIV Pre-Exposure Prophylaxis and Pre-Exposure Prophylaxis Uptake
Wille, L., & Parent, M. C. (2020) · AIDS Patient Care and STDs · doi:10.1089/apc.2020.0196 · Cited by 2
Milner, J. S., Ammar, J., Crouch, J. L., & Wagner, M. F. (2024) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2024.106829 · Cited by 1
Drivers of Perceived Value of Cryptocurrency: Comparing Stablecoins and Non-Stable Cryptocurrency
Li, G., Au, C. H., Ho, K. K. W., & Law, K. M. Y. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-74437-2_7 · Cited by 3
Doing and undoing gender: examining nonbinary and cisgender expression at work in the U.S.
Hernandez, T. R., Nault, E., Sabat, I. E., & Kostecki, T. P. (2025) · The International Journal of Human Resource Management · doi:10.1080/09585192.2024.2449531 · Cited by 1
Hadaschik, J., van Sintemaartensdijk, I., Ruiter, R. A. C., Stel, M., & Massar, K. (2025) · Evolutionary Psychological Science · doi:10.1007/s40806-024-00406-2 · Cited by 1
Li, S., & Sanusi, E. S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0327894 · Cited by 1
Detecting anger-provoking events with smartphones: a naturalistic driving study
Wang, Y., Liang Ma, A., & Zhang, W. (2025) · Ergonomics · doi:10.1080/00140139.2025.2597507 · Cited by 1
The dark side of leave: how voluntary leave shapes preferences for male and female supervisors
Motro, D., Pittarello, A., Nolan, K. P., Shahani‐Denning, C., & Lenaghan, J. A. (2022) · Journal of Managerial Psychology · doi:10.1108/jmp-05-2021-0267 · Cited by 1
Rosales, R., Zelaya, D. G., Hevey, B., Moreno, O., Chavez, S. J., Figuereo, V. J., Colby, S. M., & DeBlaere, C. (2025) · Journal of Racial and Ethnic Health Disparities · doi:10.1007/s40615-025-02426-5 · Cited by 1
Financial Attitudes, Behaviours, and the Disposition EffectPreprint
Wierzbitzki, M., Seidens, S., & Rudolf, M. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3339102 · Cited by 1
Burleigh, T. J., & Rubel, A. N. (2018) · doi:10.31234/osf.io/st2k5 · Cited by 1
Self-sexualization in Relation to Sexual Harassment and Body ShamePreprint
Choi, D., Kang, J. M., Kim, J., & Lee, H. K. (2022) · Research Square · doi:10.21203/rs.3.rs-2330732/v1 · Cited by 1
Blankenship, B. T. (2019) · Deep Blue (University of Michigan)
"Hey, Are You Busy Right Now?" Stressor Appraisals of Interruptions to Workflow
Wheatley, C. L., & Black, K. J. (2020) · UTC Scholar (University of Tennessee at Chattanooga)
Said, M., Zajdela, E. R., & Stathopoulos, A. (2021) · arXiv (Cornell University)
The gene of an early onset progressive cataract (cerulean cataract) maps to 17q24
Armitage, M. M., Ferrell, R. E., & Kivlin, J. D. (1994) · The American Journal of Human Genetics
Using Amazon Mturk to Reach Public Assistance RecipientsPreprint
Hamilton, L. K., & Wright, A. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3997870
The Role of Anxiety and Fear in Sexual Orientation Self-Disclosure
Li, Y., & Samp, J. A. (2023) · Western Journal of Communication · doi:10.1080/10570314.2023.2268587
Concerns, Worries, and Fears During Pregnancy in Active-Duty United States Air Force Families
Milner, J. S., Crouch, J. L., Ammar, J., Jensen, A. P., Travis, W. J., & Wagner, M. F. (2024) · The Journal of Perinatal Education · doi:10.1891/jpe-2023-0005
MTURK: Utility in Gerontological Behavioral Health Research
Pudur, R., & Mpofu, E. (2024) · Springer eBooks · doi:10.1007/978-981-92-1288-0_135
A Multidimensional Approach to Measuring the Risk Tolerance of Households in ChinaPreprint
Lyons, A. C., Liu, F., & Fang, E. S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5064190
Online recruitment for interview participants impacted by the criminal legal systemPreprint
Chen, E. Y., Lageson, S. E., Caliman, C. R., & Adams, E. B. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5248425
Online recruitment for interview participants impacted by the criminal legal systemPreprint
Lageson, S. E., Caliman, C. R., Chen, E. Y., & Adams, E. B. (2025) · CrimRxiv · doi:10.21428/cb6ab371.2fe99c1b
Examining Gender Differences in Cyber Deviance During the Covid-19 Pandemic
Nodeland, B., Rabbani, M. G., Hong, G., & Saber, M. (2025) · Deviant Behavior · doi:10.1080/01639625.2025.2555966
Reynolds, A., MacLeod, C., & Grafton, B. (2025) · Behaviour Research and Therapy · doi:10.1016/j.brat.2025.104909
George, S. S., Amin, A., & Morgan, S. J. (2026) · Behavioral Sciences & the Law · doi:10.1002/bsl.70038
Robo-Advisor Adoption and Influences of Innovation Attributes, Trust, and Image
Mohamed, N. (2026) · FinTech · doi:10.3390/fintech5010011
Improving Mental Health by Challenging the Societal Roots of Ageism
Levy, B. R., Provolo, N., Levy, S., & Slade, M. D. (2026) · GeroPsych · doi:10.1024/1662-9647/a000365
Ortega, A. X., Camacho, E., & García, J. J. (2026) · Current Psychology · doi:10.1007/s12144-026-09625-1
Reynolds, A., MacLeod, C., & Grafton, B. (2026) · Behaviour Research and Therapy · doi:10.1016/j.brat.2026.105105
The service variety paradox in crypto-exchanges: A stimulus-organism-response (SOR) perspective
Au, C. H., Shieh, P.-H., Nurbaev, V., Law, K. M. Y., & Ho, K. K. W. (2026) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2026.103114
Using selection tests to hire autistic employees: An exploration of methods
Whelpley, C. E., Fezzey, T. N. A., Batchelor, J. H., & Weekley, J. A. (2026) · PLoS ONE · doi:10.1371/journal.pone.0356539
Mitsouridis, M. A., Genitsaris, E., Gkalimanis, P., Naveau, M., Petrík, V., & Mansard, N. (2026) · Open Research Europe · doi:10.12688/openreseurope.25208.1
Metadata
- Authors
- Smith, N. A., Sabat, I. E., Martinez, L. R., Weaver, K., & Xu, S.
- Affiliations
- Pennsylvania State University (US)
- George Mason University (US)
- Corresponding author
- Smith, Nicholas A.
- Venue
- Industrial and Organizational Psychology
- Publisher
- Cambridge University Press (CUP)
- Published
- 2015-06
- Volume
- 8
- Issue
- 2
- Pages
- 220-228
- ISSN
- 1754-9426 (ISSN-L)
- 1754-9434
- OpenAlex
- W2424520822
- Cited by
- 190
- References
- 23
- Citations by year
- 2026: 8 · 2025: 13 · 2024: 17 · 2023: 24 · 2022: 24 · 2021: 35 · 2020: 16 · 2019: 23 · 2018: 15 · 2017: 10 · 2016: 4
- FWCI
- 14.55
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Industrial and Organizational Psychology (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › Employer Branding and e-HRM
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- Amazon Mechanical Turk, hard-to-reach populations, online recruitment, stigmatized groups, low socioeconomic status, people with disabilities, LGBT people, workplace harassment, underrepresented populations in research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review