Experiences surveying the crowd: reflections on methods, participation, and reliability
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Crowdsourcing services such as Amazon's Mechanical Turk (MTurk) provide new venues for recruiting participants and conducting studies; hundreds of surveys may be offered to workers at any given time. We reflect on the results of six related studies we performed on MTurk over a two year period. The studies used a combination of open-ended questions and structured hypothetical statements about story-like scenarios to engage the efforts of 1252 participants. We describe the method used in the studies and reflect on what we have learned about identified best practices. We analyze the aggregated data to profile the types of Turkers who take surveys and examine how the characteristics of the surveys may influence data reliability. The results point to the value of participant engagement, identify potential changes in MTurk as a study venue, and highlight how communication among Turkers influences the data that researchers collect.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · cited 2×
“Inexpensive, educated, and relatively reliable (especially as researchers identify best practices for using these platforms [2,6,15,17]), Turkers provide us with a convenient and diverse pool of prospective study participants [17,24].”
BackgroundRemix: Making art and commerce thrive in the hybrid economy
Lessig, L. (2008) · doi:10.5040/9781849662505 · Cited by 1,081
Who are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004
The public domain: enclosing the commons of the mind
(2009) · Choice Reviews Online · doi:10.5860/choice.46-6473 · Cited by 496
Shepherding the crowd yields better work
Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145355 · Cited by 334 · cited 1×
“Assumptions must be examined carefully when researchers build in various quality mechanisms, including comprehension questions (which, as we saw in the Subreddit, may irritate or confuse respondents) [17], data cleaning methods [32], and feedback [5].”
Background
Cited by
Work experiences on MTurk: Job satisfaction, turnover, and information sharing
Brawley, A. M., & Pury, C. L. S. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.08.031 · Cited by 188
“One recommended practice is to pilot a HIT on MTurk and set pay based on the observed HIT submission time, but our evidence supports other researchers' caution to Requesters in using HIT or survey completion time alone to determine pay (cf. Marshall & Shipman, 2013).”
BackgroundMethodMessages about brilliance undermine women's interest in educational and professional opportunities
Bian, L., Leslie, S.-J., Murphy, M. C., & Cimpian, A. (2018) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.11.006 · Cited by 181
“Ipeirotis, 2010; Marshall & Shipman, 2013 Next, participants were shown the student characteristics that were“ most frequently mentioned during their interviews by the professors for the new major, which varied by condition.”
Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments
Hube, C., Fetahu, B., & Gadiraju, U. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300637 · Cited by 129
“We also included two attention check questions to filter out inattentive workers [39].”
Internet surveillance, regulation, and chilling effects online: a comparative case study
Penney, J. W. (2017) · Internet Policy Review · doi:10.14763/2017.2.692 · Cited by 122
Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
He, G., Kuiper, L., & Gadiraju, U. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581025 · Cited by 107
Metadata
- Authors
- Marshall, C. C., & Shipman, F. M.
- Venue
- Proceedings of the 5th Annual ACM Web Science Conference
- Publisher
- ACM
- Published
- 2013-05-02
- Pages
- 234-243
- OpenAlex
- W2146436037
- Cited by
- 63
- References
- 5
- FWCI
- 4.95
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.