Creativity on Paid Crowdsourcing Platforms
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Crowdsourcing platforms are increasingly being harnessed for creative work. The platforms' potential for creative work is clearly identified, but the workers' perspectives on such work have not been extensively documented. In this paper, we uncover what the workers have to say about creative work on paid crowdsourcing platforms. Through a quantitative and qualitative analysis of a questionnaire launched on two different crowdsourcing platforms, our results revealed clear differences between the workers on the platforms in both preferences and prior experience with creative work. We identify common pitfalls with creative work on crowdsourcing platforms, provide recommendations for requesters of creative work, and discuss the meaning of our findings within the broader scope of creativity-oriented research. To the best of our knowledge, we contribute the first extensive worker-oriented study of creative work on paid crowdsourcing platforms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
The standard definition of creativity
Runco, M. A., & Jaeger, G. J. (2012) · Creativity Research Journal · doi:10.1080/10400419.2012.650092 · Cited by 3,895 · cited 1×
“, in this case divergent thinking) has become a staple in the toolbox of many researchers [61].”
BackgroundGuilford, J. P. (1950) · American Psychologist · doi:10.1037/h0063487 · Cited by 3,588
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · cited 1×
“Prolific is a crowdsourcing platform targeted towards academic studies [56].”
BackgroundBeyond Big and Little: The Four C Model of Creativity
Kaufman, J. C., & Beghetto, R. A. (2009) · Review of General Psychology · doi:10.1037/a0013688 · Cited by 2,669
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · cited 2×
“Similar qualification criteria are typically being used in academic studies to improve the chance of receiving high quality responses [57].”
BackgroundMcDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
Turkopticon: interrupting worker invisibility in amazon mechanical turk
Irani, L. C., & Silberman, M. S. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470742 · Cited by 798 · cited 3×
“ed research. We look into worker nonnaïveté in regard to the most commonly used creativity tests and touch on a number of issues previously reported in the literature, such as underpayment of workers [33,65] and worker motivation [36,45]. As a result of our studies, we profile workers specifically concerning creative wo…”
BackgroundQuantifying the Creativity Support of Digital Tools through the Creativity Support Index
Cherry, E., & Latulipe, C. (2014) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2617588 · Cited by 498 · cited 2×
“For instance, the Creativity Support Index (CSI) [7, 11] aims to measure how well a CST supports creativity and the Experience of Creativity Questionnaire (ECQ) [50] measures experiential and existential aspects of artistic creativity.”
BackgroundMapping the Landscape of Creativity Support Tools in HCI
Frich, J., MacDonald Vermeulen, L., Remy, C., Biskjaer, M. M., & Dalsgaard, P. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300619 · Cited by 410 · cited 1×
“This increased interest within HCI has resulted in the development of a plethora of creativity support tools (CSTs) that aim to augment the creativity of groups and individuals [18].”
BackgroundThe average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 1×
“an “emerging tragedy of the commons” [66]. A large portion of the tasks are completed by a relatively small pool of active professional workers who have been exposed to many types of different tasks [8,15,66,67]. A high number of crowdsourcing tasks may be carried out by the most active “professional Turkers” [27] or…”
BackgroundUsing Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185 · cited 1×
“A worker’s prior exposure to commonly used manipulations and measures may negatively impact the validity of the results obtained [8, 9, 26].”
BackgroundDistributed analogical idea generation: inventing with crowds
Yu, L., Kittur, A., & Kraut, R. E. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557371 · Cited by 60
Cited by
Comparing attentional disengagement between prolific and MTurk samples
Albert, D. A., & Smilek, D. (2023) · Scientific Reports · doi:10.1038/s41598-023-46048-5 · Cited by 166
“Like in previous studies, we also found significant differences in demographic characteristics, namely education and employment in E1 and age and employment in E2, between the MTurk and Prolific samples 40,70,71 .”
ResultDing, Z., Srinivasan, A., Macneil, S., & Chan, J. (2023) · Creativity and Cognition · doi:10.1145/3591196.3593516 · Cited by 35
“We recruited participants from Prolific 6 as Pro-lific participants typically exhibit greater care in completing tasks compared to Amazon MTurk participants [74].”
BackgroundTowards a Reflection in Creative Experience Questionnaire
Ford, C., & Bryan-Kinns, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581077 · Cited by 15
“We use Prolific instead of alternatives because it is academic-focused and its participants might show more interest in creativity-related work [66].”
Metadata
- Authors
- Oppenlaender, J., Milland, K., Visuri, A., Ipeirotis, P., & Hosio, S.
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-14
- OpenAlex
- W3002599298
- Cited by
- 35
- References
- 16
- FWCI
- 6.54
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.