The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
Data Collection via Online Platforms: Challenges and Recommendations for Future Research
Open accessAbstract
Online platforms such as Amazon's Mechanical Turk (MTurk) are increasingly used by researchers to collect survey and experimental data. Yet, such platforms often represent a tumultuous terrain for both researchers and reviewers. Researchers have to navigate the complexities of obtaining representative samples from online participant cohorts, ensuring data quality, ethically incentivizing participant engagement, and maintaining transparency. Reviewers, on the other hand, have to navigate the complexities of evaluating the efficacy of such data collection and execution efforts in answering important research questions. In order to provide clarity to these issues, this article provides researchers and reviewers with a series of recommendations for effectively executing and evaluating data collection via online platforms, respectively.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Identifying careless responses in survey data.
Meade, A. W., & Craig, S. B. (2012) · Psychological Methods · doi:10.1037/a0028085 · Cited by 3,865
Prolific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences
Litman, L., Robinson, J., & Abberbock, T. (2017) · Behavior Research Methods · doi:10.3758/s13428-016-0727-z · Cited by 1,982
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662
Methods for the detection of carelessly invalid responses in survey data
Curran, P. G. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.07.006 · Cited by 1,356
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902
Microfoundations of Organizational Paradox: The Problem Is How We Think about the Problem
Miron-Spektor, E., Ingram, A., Keller, J., Smith, W. K., & Lewis, M. W. (2018) · Academy of Management Journal · doi:10.5465/amj.2016.0594 · Cited by 805
How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
Montgomery, J. M., Nyhan, B., & Torres, M. (2018) · American Journal of Political Science · doi:10.1111/ajps.12357 · Cited by 800
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 767
The demographic and political composition of Mechanical Turk samples
Levay, K. E., Freese, J., & Druckman, J. N. (2016) · Sage Open · doi:10.1177/2158244016636433 · Cited by 676
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
A Tale of Two Sample Sources: Do Results from Online Panel Data and Conventional Data Converge?
Walter, S. L., Seibert, S. E., Goering, D., & O’Boyle, E. H. (2019) · Journal of Business and Psychology · doi:10.1007/s10869-018-9552-y · Cited by 544
The shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506
Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017) · Journal of Business and Psychology · doi:10.1007/s10869-016-9458-5 · Cited by 474
What You See Is What You Get? Enhancing Methodological Transparency in Management Research
Aguinis, H., Ramani, R. S., & Alabduljader, N. (2018) · Academy of Management Annals · doi:10.5465/annals.2016.0011 · Cited by 463
Zhou, H., & Fishbach, A. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000056 · Cited by 459
Research Methods for Organizational Studies
Schwab, D. P. (2013) · Psychology Press eBooks · doi:10.4324/9781410611284 · Cited by 455
Insufficient effort responding: Examining an insidious confound in survey data.
Huang, J. L., Liu, M., & Bowling, N. A. (2015) · Journal of Applied Psychology · doi:10.1037/a0038510 · Cited by 416
Smith, S. M., Roster, C. A., Golden, L. L., & Albaum, G. S. (2016) · Journal of Business Research · doi:10.1016/j.jbusres.2015.12.002 · Cited by 415
The Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J. P., & Cho, T. S. (2019) · Journal of Management · doi:10.1177/0149206318811569 · Cited by 338
MTurk Character Misrepresentation: Assessment and Solutions
Sharpe Wessling, K., Huber, J., & Netzer, O. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx053 · Cited by 330
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304
Lowry, P. B., D’Arcy, J., Hammer, B., & Moody, G. D. (2016) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2016.06.002 · Cited by 301
Chandler, J. J., & Paolacci, G. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617698203 · Cited by 277
A Note on Dropping Experimental Subjects who Fail a Manipulation Check
Aronow, P. M., Baron, J., & Pinson, L. (2019) · Political Analysis · doi:10.1017/pan.2019.5 · Cited by 277
The Crowd is a Collaborative Network
Gray, M. L., Suri, S., Ali, S. S., & Kulkarni, D. (2016) · Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing · doi:10.1145/2818048.2819942 · Cited by 235
Who Are the Turkers? A Characterization of MTurk Workers Using the Personality Assessment Inventory
McCredie, M. N., & Morey, L. C. (2019) · Assessment · doi:10.1177/1073191118760709 · Cited by 234
Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019) · PLOS ONE · doi:10.1371/journal.pone.0226394 · Cited by 205
Psychological research in the internet age: The quality of web-based data
Ramsey, S. R., Thompson, K. L., McKenzie, M., & Rosenbaum, A. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.12.049 · Cited by 203
A convenient solution: Using MTurk to sample from hard-to-reach populations
Smith, N. A., Sabat, I. E., Martinez, L. R., Weaver, K., & Xu, S. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.29 · Cited by 190
Dennis, S. A., Goodson, B. M., & Pearson, C. A. (2020) · Behavioral Research in Accounting · doi:10.2308/bria-18-044 · Cited by 186
Using Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185
The Role of MTurk in Education Research: Advantages, Issues, and Future Directions
Follmer, D. J., Sperling, R. A., & Suen, H. K. (2017) · Educational Researcher · doi:10.3102/0013189x17725519 · Cited by 184
Stritch, J. M., Pedersen, M. J., & Taggart, G. (2017) · International Public Management Journal · doi:10.1080/10967494.2016.1276493 · Cited by 181
Using Crowdsourcing Websites for Sociological Research: The Case of Amazon Mechanical Turk
Shank, D. B. (2016) · The American Sociologist · doi:10.1007/s12108-015-9266-9 · Cited by 176
Common Concerns with MTurk as a Participant Pool
Hauser, D., Paolacci, G., & Chandler, J. (2019) · Handbook of Research Methods in Consumer Psychology · doi:10.4324/9781351137713-17 · Cited by 171
Necka, E. A., Cacioppo, S., Norman, G. J., & Cacioppo, J. T. (2016) · PLOS ONE · doi:10.1371/journal.pone.0157732 · Cited by 161
Applying Social Psychology to Prevent Careless Responding during Online Surveys
Ward, M. K., & Meade, A. W. (2017) · Applied Psychology · doi:10.1111/apps.12118 · Cited by 144
Using MTurk to Distribute a Survey or Experiment: Methodological Considerations
Hunt, N. C., & Scheetz, A. M. (2018) · Journal of Information Systems · doi:10.2308/isys-52021 · Cited by 144
The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers
Ophir, Y., Sisso, I., Asterhan, C. S. C., Tikochinski, R., & Reichart, R. (2019) · Clinical Psychological Science · doi:10.1177/2167702619865973 · Cited by 133
Lovett, M., Bajaba, S. M., Lovett, M., & Simmering, M. J. (2017) · Applied Psychology · doi:10.1111/apps.12124 · Cited by 125
Mechanical Turk upends social sciences
Bohannon, J. (2016) · Science · doi:10.1126/science.352.6291.1263 · Cited by 121
Using Nonnaive Participants Can Reduce Effect SizesPreprint
Chandler, J., Paolacci, G., Péer, E., Mueller, P. A., & Ratliff, K. A. (2015) · RePEc: Research Papers in Economics · Cited by 118
Clifford, S., & Jerit, J. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfw030 · Cited by 108
Gleibs, I. H. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0789-y · Cited by 104
Instructional Manipulation Checks: A longitudinal analysis with implications for MTurk
Paas, L. J., Dolničar, S., & Karlsson, L. (2018) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2018.01.003 · Cited by 93
Actionable recommendations for narrowing the science-practice gap in open science
Aguinis, H., Banks, G. C., Rogelberg, S. G., & Cascio, W. F. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.02.007 · Cited by 90
How serious is the ‘carelessness’ problem on Mechanical Turk?
Aruguete, M. S., Huynh, H. P., Browne, B. L., Jurs, B. S., Flint, E., & McCutcheon, L. E. (2019) · International Journal of Social Research Methodology · doi:10.1080/13645579.2018.1563966 · Cited by 89
A panel for lemons? Positivity bias, reputation systems and data quality on MTurk
Matherly, T. (2019) · European Journal of Marketing · doi:10.1108/ejm-07-2017-0491 · Cited by 78
Gender differences in response to competition with same-gender coworkers: A relational perspective.
Lee, S. Y., Kesebir, S., & Pillutla, M. M. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000051 · Cited by 74
Pittman, M., & Sheehan, K. B. (2016) · Journal of Media Ethics · doi:10.1080/23736992.2016.1228811 · Cited by 73
Shao, B., Cardona, P., Ng, I., & Trau, R. (2017) · Asia Pacific Journal of Management · doi:10.1007/s10490-017-9512-5 · Cited by 58
Self-citationOwens, J., & Hawkins, E. M. (2018) · Journal of Information Systems · doi:10.2308/isys-52036 · Cited by 53
Kan, I. P., & Drummey, A. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.02.005 · Cited by 52
Take it to the top: Imagined interactions with leaders elevates organizational identification
Meleady, R., & Crisp, R. J. (2017) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.01.008 · Cited by 47
Siegel, J. T., Navarro, M. A., & Thomson, A. L. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.08.020 · Cited by 45
Reliability of MTurk Data From Masters and Workers
Rouse, S. V. (2019) · Journal of Individual Differences · doi:10.1027/1614-0001/a000300 · Cited by 44
Leader power, power stability, and interpersonal trust
Mooijman, M., DIJK, W. W. V., van Dijk, E., & Ellemers, N. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2019.03.009 · Cited by 41
Stone, A. A., Walentynowicz, M., Schneider, S., Junghaenel, D. U., & Wen, C. K. F. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2018.12.042 · Cited by 38
Ethical issues in psychological research on the Internet.
Buchanan, T., & Williams, J. E. (2010) · American Psychological Association eBooks · doi:10.1037/12076-016 · Cited by 36
More evidence of participant misrepresentation on Mturk and investigating who misrepresents
MacInnis, C. C., Boss, H. C. D., & Bourdage, J. (2020) · Personality and Individual Differences · doi:10.1016/j.paid.2019.109603 · Cited by 34
Turkers of the World Unite: Multilevel In-Group Bias Among Crowdworkers on Amazon Mechanical Turk
Almaatouq, A., Krafft, P. M., Dunham, Y. C., Rand, D. G., & Pentland, A. (2019) · Social Psychological and Personality Science · doi:10.1177/1948550619837002 · Cited by 27
Siegel, J. T., & Navarro, M. (2019) · Journal of Applied Social Psychology · doi:10.1111/jasp.12580 · Cited by 21
DeVoe, S. E., & House, J. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.11.004 · Cited by 12
Muise, D., & Pan, J. (2018) · Asian Journal of Communication · doi:10.1080/01292986.2018.1453850 · Cited by 12
Cited by
Mental health and academic experiences among U.S. college students during the COVID-19 pandemic
Roberts, M. E., Bell, E. A., & Meyer, J. L. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1166960 · Cited by 29
Bredemeier, K., Church, L. D., Bounoua, N., Feler, B., & Spielberg, J. M. (2023) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2022.102660 · Cited by 18
Can crowdsourcing platforms be useful for educational research?
Wang, K. D., Chen, Z., & Wieman, C. (2024) · Proceedings of the 14th Learning Analytics and Knowledge Conference · doi:10.1145/3636555.3636897 · Cited by 9
Metadata
- Authors
- Newman, A., Bavik, Y. L., Mount, M., & Shao, B.
- Venue
- Applied Psychology
- Publisher
- Wiley
- Published
- 2021-01-15
- Volume
- 70
- Issue
- 3
- Pages
- 1380-1402
- ISSN
- 0269-994X (ISSN-L)
- 1464-0597
- OpenAlex
- W3107213376
- Cited by
- 306
- References
- 74
- FWCI
- 26.40
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.