The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
Abstract
Understanding perception is critical to effective visualization design. With its low cost and scalability, crowdsourcing presents an attractive option for evaluating the large design space of visualizations; however, it first requires validation. In this paper, we assess the viability of Amazon's Mechanical Turk as a platform for graphical perception experiments. We replicate previous studies of spatial encoding and luminance contrast and compare our results. We also conduct new experiments on rectangular area perception (as in treemaps or cartograms) and on chart size and gridline spacing. Our results demonstrate that crowdsourced perception experiments are viable and contribute new insights for visualization design. Lastly, we report cost and performance data from our experiments and distill recommendations for the design of crowdsourced studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2013) · Information Visualization · doi:10.1016/b978-0-12-381464-7.00018-1 · Cited by 3,742
The Processing of Information and Structure
Royer, F. L., & Garner, W. R. (1974) · The American Journal of Psychology · doi:10.2307/1421985 · Cited by 2,117
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Cleveland, W. S., & McGill, R. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478080 · Cited by 1,788
Cleveland, W. S. (1993) · Cited by 1,698
Automating the design of graphical presentations of relational information
Mackinlay, J. (1986) · ACM Transactions on Graphics · doi:10.1145/22949.22950 · Cited by 1,610
Tree visualization with tree-maps: 2-d space-filling approach
Shneiderman, B. (1992) · ACM Transactions on Graphics · doi:10.1145/102377.115768 · Cited by 1,328
How Maps WorkRepresentation, Visualization, and Design
MacEachren, A. M. (1995) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,095
Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.105 · Cited by 924
ManyEyes: a Site for Visualization at Internet Scale
Viegas, F. B., Wattenberg, M., van Ham, F., Kriss, J., & McKeon, M. (2007) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.70577 · Cited by 720
Cartography: Thematic Map Design
Dent, B. D. (1990) · Cited by 673
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2009) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1600150.1600175 · Cited by 520
Using Social Psychology to Motivate Contributions to Online Communities
Ling, K., Beenen, G., Ludford, P., Wang, X., Chang, K., Li, X., Cosley, D., Frankowski, D., Terveen, L., Rashid, A. M., et al. (2005) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2005.tb00273.x · Cited by 493
Show Me: Automatic Presentation for Visual Analysis
Mackinlay, J., Hanrahan, P., & Stolte, C. (2007) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.70594 · Cited by 480
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1809400.1809422 · Cited by 463
Practical guide to controlled experiments on the web: listen to your customers not to the hippo
Kohavi, R., Henne, R. M., & Sommerfield, D. (2007) · Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1281192.1281295 · Cited by 377
Using social psychology to motivate contributions to online communities
Beenen, G., Ling, K., Wang, X., Chang, K., Frankowski, D., Resnick, P., & Kraut, R. E. (2004) · Proceedings of the 2004 ACM conference on Computer supported cooperative work · doi:10.1145/1031607.1031642 · Cited by 320
An Information-Processing Analysis of Graph Perception
Simkin, D., & Hastie, R. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478448 · Cited by 309
Heer, J., Kong, N., & Agrawala, M. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518897 · Cited by 288
Self-citationVariables on Scatterplots Look More Highly Correlated When the Scales Are Increased
Cleveland, W. S., Diaconis, P., & McGill, R. (1982) · Science · doi:10.1126/science.216.4550.1138 · Cited by 173
WATTENBERG, M. P. (1999) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/632716.632834 · Cited by 166
Displaying proportions and percentages
Spence, I., & Lewandowsky, S. (1991) · Applied Cognitive Psychology · doi:10.1002/acp.2350050106 · Cited by 162
The Shaping of Information by Visual Metaphors
Ziemkiewicz, C., & Kosara, R. (2008) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2008.171 · Cited by 132
Web‐Based Experiments for the Study of Collective Social Dynamics in Cultural Markets
Salganik, M., & Watts, D. J. (2009) · Topics in Cognitive Science · doi:10.1111/j.1756-8765.2009.01030.x · Cited by 117
Selective Incentives and Generalized Information Exchange
Cheshire, C. (2007) · Social Psychology Quarterly · doi:10.1177/019027250707000109 · Cited by 98
Discriminating Strata in Scatterplots
Lewandowsky, S., & Spence, I. (1989) · Journal of the American Statistical Association · doi:10.1080/01621459.1989.10478821 · Cited by 82
The Impact of Graph Slope on Rate of Change Judgments in Corporate Reports
BEATTIE, V. A., & Jones, M. J. (2002) · Abacus · doi:10.1111/1467-6281.00104 · Cited by 68
Perception of elementary graphical elements in tabletop and multi-surface environments
Wigdor, D., Shen, C., Forlines, C., & Balakrishnan, R. (2007) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/1240624.1240701 · Cited by 66
Mackay, W. E., Appert, C., Beaudouin-Lafon, M., Chapuis, O., Du, Y., Fekete, J., & Guiard, Y. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240840 · Cited by 56
How well do line drawings depict shape?
Cole, F. H., Sanik, K., DeCarlo, D., Finkelstein, A., Funkhouser, T. A., Rusinkiewicz, S. M., & Singh, M. (2009) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1576246.1531334 · Cited by 49
The Visual Separability of Plotting Symbols in Scatterplots
Tremmel, L. T. (1995) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1995.10474669 · Cited by 25
White rooms and morphing don't mix
Reilly, D. F., & Inkpen, K. M. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240640 · Cited by 19
Cited by
UpSet: Visualization of Intersecting Sets
Lex, A., Gehlenborg, N., Strobelt, H., Vuillemot, R., & Pfister, H. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346248 · Cited by 2,323
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Towards an integrated crowdsourcing definition
Estellés-Arolas, E., & González-Ladrón-de-Guevara, F. (2012) · Journal of Information Science · doi:10.1177/0165551512437638 · Cited by 1,858
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
(2021) · doi:10.1145/3411764 · Cited by 1,240
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
Metadata
- Authors
- Heer, J., & Bostock, M.
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2010-04-10
- Pages
- 203-212
- OpenAlex
- W2117470435
- Cited by
- 816
- References
- 32
- FWCI
- 38.21
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.