The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
The Opportunities and Limitations of Using Mechanical Turk (MTURK) in Public Administration and Management Scholarship
Open accessAbstract
Other social science fields are increasingly conducting research using Amazon’s Mechanical Turk (MTurk)—an online crowdsourcing platform—but how might MTurk be useful to public administration and management research? This article provides an introduction of the platform and considers both the opportunities and limitations for using MTurk in public administration and management scholarship. We find that MTurk might be relevant for examining particular types of research questions. We identify five areas where MTurk data may complement and enhance public administration and management research: (1) exploratory analyses and survey construction; (2) measurement refinement of latent constructs; (3) experiments; (4) longitudinal research and data collection; and (5) collection of data from citizens. The article emphasizes how a key requisite conditions both the applicability of MTurk data and the validity of MTurk-based findings: the researcher must understand the boundaries and potential of the platform, since the issues related to representativeness, participation, and data quality are non-trivial.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.5.879 · Cited by 82,194
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 3×
“Such actions may dilute the reliability of MTurk data and increase the chance of false negatives (Button et al. 2013).”
BackgroundInfluentialEvaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · cited 4×
“Scholars suggest a number of reasons why MTurk has emerged as a popular platform for social science research (Buhrmester et al. 2011; Mason and Suri 2012; Paolacci and Chandler 2014; Paolacci, Chandler, and Ipeirotis 2010).”
BackgroundInfluentialWhat Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?
Levitt, S. D., & List, J. A. (2007) · Journal of Economic Perspectives · doi:10.1257/jep.21.2.153 · Cited by 2,668
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · cited 1×
“Researchers note that some Turkers might not pay sufficient attention to the tasks they are performing or are simply clicking through surveys and experiments as quickly as possible (Goodman, Cryder, and Cheema 2013).”
BackgroundInside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Sears, D. O. (1986) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.51.3.515 · Cited by 2,119
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 7×
“…do not replace the high-quality field experiments (Avellaneda 2013; Bellé 2013; 2014; James 2011) in terms of external and ecological validity, MTurk provides a useful platform for survey experiments with high levels of internal validity similar to that of laboratory studies (Casler et al. 2013).”
BackgroundMethodInfluentialEvaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951 · cited 1×
“Examples include political science (Arceneaux 2012; Berinsky, Huber, and Lenz 2012; Berinsky, Margolis, and Sances 2014; Dowling and Wichowsky 2015; Healy and Lenz 2014; Thibodeau et al. 2015), economics and behavioral science (Chandler and Kapelner 2013; Chandler, Mueller, and Paolacci 2014;…”
BackgroundHuff, C., & Tingley, D. (2015) · Research & Politics · doi:10.1177/2053168015604648 · Cited by 903
How (Not) to Solve the Problem: An Evaluation of Scholarly Responses to Common Source Bias
Favero, N., & Bullock, J. B. (2015) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muu020 · Cited by 372
Bellé, N. (2014) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/mut033 · Cited by 332 · cited 2×
“While experiments on MTurk do not replace the high-quality field experiments (Avellaneda 2013; Bellé 2013; 2014; James 2011) in terms of external and ecological validity, MTurk provides a useful platform for survey experiments with high levels of internal validity similar to that of laboratory…”
BackgroundMethodShifting Administrative Burden to the State: The Case of Medicaid Take‐Up
Herd, P., DeLeire, T., Harvey, H., & Moynihan, D. P. (2013) · Public Administration Review · doi:10.1111/puar.12114 · Cited by 263 · cited 1×
“…Osborne 2008; Riccucci et al. 2016), citizen engagement (Reddel and Woolcock 2004; Smith, Santucci, Xu, Cox, and Henderson 2012), and administrative burden (Herd et al. 2013; Moynihan, Herd, and Harvey 2014) might all be studied using different types of experimental designs on the MTurk platform.”
BackgroundUnconscious Bias in Citizens’ Evaluations of Public Sector Performance
Marvel, J. D. (2015) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muu053 · Cited by 252
A convenient solution: Using MTurk to sample from hard-to-reach populations
Smith, N. A., Sabat, I. E., Martinez, L. R., Weaver, K., & Xu, S. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.29 · Cited by 190
Interpreting performance information: Motivated reasoning or unbiased comprehension.
Baekgaard, M., & Serritzlew, S. (2016) · Public Administration Review · doi:10.1111/puar.12406 · Cited by 188
Cited by
MTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2021) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 984
The shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506
“In fact, MTurk samples generally provide higher quality data than student samples, community samples, and even some high-quality national samples (Hauser and Schwarz, 2015; Mullinix et al ., 2015; Thomas and Clifford, 2017; Anson, 2018).”
BackgroundThe Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J. P., & Cho, T. S. (2019) · Journal of Management · doi:10.1177/0149206318811569 · Cited by 338
Data Collection via Online Platforms: Challenges and Recommendations for Future Research
Newman, A., Bavik, Y. L., Mount, M., & Shao, B. (2021) · Applied Psychology · doi:10.1111/apps.12302 · Cited by 306
Ahn, M. J., & Chen, Y.-C. (2022) · Government Information Quarterly · doi:10.1016/j.giq.2021.101664 · Cited by 305
Metadata
- Authors
- Stritch, J. M., Pedersen, M. J., & Taggart, G.
- Venue
- International Public Management Journal
- Publisher
- Informa UK Limited
- Published
- 2017-01-19
- Volume
- 20
- Issue
- 3
- Pages
- 489-511
- ISSN
- 1096-7494 (ISSN-L)
- 1559-3169
- OpenAlex
- W2573660563
- Cited by
- 181
- References
- 20
- FWCI
- 21.37
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.