The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
Abstract
Understanding perception is critical to effective visualization design. With its low cost and scalability, crowdsourcing presents an attractive option for evaluating the large design space of visualizations; however, it first requires validation. In this paper, we assess the viability of Amazon's Mechanical Turk as a platform for graphical perception experiments. We replicate previous studies of spatial encoding and luminance contrast and compare our results. We also conduct new experiments on rectangular area perception (as in treemaps or cartograms) and on chart size and gridline spacing. Our results demonstrate that crowdsourced perception experiments are viable and contribute new insights for visualization design. Lastly, we report cost and performance data from our experiments and distill recommendations for the design of crowdsourced studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2013) · Information Visualization · doi:10.1016/b978-0-12-381464-7.00018-1 · Cited by 3,742
The Processing of Information and Structure
Royer, F. L., & Garner, W. R. (1974) · The American Journal of Psychology · doi:10.2307/1421985 · Cited by 2,117
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Cleveland, W. S., & McGill, R. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478080 · Cited by 1,788
Cleveland, W. S. (1993) · Cited by 1,698
Automating the design of graphical presentations of relational information
Mackinlay, J. (1986) · ACM Transactions on Graphics · doi:10.1145/22949.22950 · Cited by 1,610
Tree visualization with tree-maps: 2-d space-filling approach
Shneiderman, B. (1992) · ACM Transactions on Graphics · doi:10.1145/102377.115768 · Cited by 1,328
How Maps WorkRepresentation, Visualization, and Design
MacEachren, A. M. (1995) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,095
Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.105 · Cited by 924
ManyEyes: a Site for Visualization at Internet Scale
Viegas, F. B., Wattenberg, M., van Ham, F., Kriss, J., & McKeon, M. (2007) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.70577 · Cited by 720
Cited by
UpSet: Visualization of Intersecting Sets
Lex, A., Gehlenborg, N., Strobelt, H., Vuillemot, R., & Pfister, H. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346248 · Cited by 2,323
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Towards an integrated crowdsourcing definition
Estellés-Arolas, E., & González-Ladrón-de-Guevara, F. (2012) · Journal of Information Science · doi:10.1177/0165551512437638 · Cited by 1,858
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
(2021) · doi:10.1145/3411764 · Cited by 1,240
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
MTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2021) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 984
Soylent: a word processor with a crowd inside
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866078 · Cited by 686
“Heer and Bostock explored Mechanical Turk as a testbed for graphical perception experiments, finding reliable results when they implemented basic measures like qualification tests [ 10 ].”
BackgroundEmpirical Studies in Information Visualization: Seven Scenarios
Lam, H., Bertini, E., Isenberg, P., Plaisant, C., & Carpendale, S. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.279 · Cited by 623
Evaluation on crowdsourcing research: Current status and future direction
Zhao, Y., & Zhu, Q. (2014) · Information Systems Frontiers · doi:10.1007/s10796-012-9350-4 · Cited by 566
Shape-changing interfaces: a review of the design space and open research questions
Rasmussen, M. K., Pedersen, E. W., Petersen, M. G., & Hornbæk, K. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207781 · Cited by 445
CrowdForge: crowdsourcing complex work
Kittur, A., Smus, B., & Kraut, R. (2011) · CHI '11 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1979742.1979902 · Cited by 434
A Survey of Crowdsourcing Systems
Yuen, M.-C., King, I., & Leung, K.-S. (2011) · 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing · doi:10.1109/passat/socialcom.2011.203 · Cited by 392
The Science of Visual Data Communication: What Works
Franconeri, S. L., Padilla, L. M., Shah, P., Zacks, J. M., & Hullman, J. (2021) · Psychological Science in the Public Interest · doi:10.1177/15291006211051956 · Cited by 360
Shepherding the crowd yields better work
Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145355 · Cited by 334
Modeling the efficacy of persuasive strategies for different gamer types in serious games for health
Orji, R., Vassileva, J., & Mandryk, R. L. (2014) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-014-9149-8 · Cited by 319
Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco
Moritz, D., Wang, C., Nelson, G. L., Lin, H., Smith, A. M., Howe, B., & Heer, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865240 · Cited by 317
Self-citationData visualization literacy: Definitions, conceptual frameworks, exercises, and assessments
Börner, K., Bueckle, A., & Ginda, M. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1807180116 · Cited by 316
Bartneck, C., Duenser, A., Moltchanova, E., & Zawieska, K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121595 · Cited by 317
“The average time to complete a task is higher in AMT studies compared to laboratory setting which indicates a possible problem of controlling the participants’ environment [5], although there are no set rules on how long a researcher should give subjects to complete tasks [30].”
BackgroundMethodResultInfluentialDaniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., & Allahbakhsh, M. (2019) · ACM Computing Surveys · doi:10.1145/3148148 · Cited by 313
CrowdForge: crowdsourcing complex work
Kittur, A., Smus, B., Khamkar, S., & Kraut, R. E. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047202 · Cited by 309
imMens: Real‐time Visual Querying of Big Data
Liu, Z., Jiang, B., & Heer, J. (2013) · Computer Graphics Forum · doi:10.1111/cgf.12129 · Cited by 308
Self-citationTurKit: human computation algorithms on mechanical turk
Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866040 · Cited by 307
A Principled Way of Assessing Visualization Literacy
Boy, J., Rensink, R. A., Bertini, E., & Fekete, J.-D. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346984 · Cited by 278
ReVision: automated classification, analysis and redesign of chart images
Savva, M., Kong, N., Chhajta, A., Fei-Fei, L., Agrawala, M., & Heer, J. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047247 · Cited by 264
Self-citationRanking Visualizations of Correlation Using Weber's Law
Harrison, L., Yang, F., Franconeri, S., & Chang, R. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346979 · Cited by 212
Hullman, J., Resnick, P., & Adar, E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0142444 · Cited by 194
“AMT has been shown to be an 330 effective platform for conducting graphical perception studies [24], and has been used in the 331 information visualization community for testing how non-statisticians interpret visualizations of 332 uncertainty [11].”
MethodWang, Q., Li, B., & Singh, P. V. (2018) · Information Systems Research · doi:10.1287/isre.2017.0735 · Cited by 162
Using fNIRS brain sensing to evaluate information visualization interfaces
Peck, E. M. M., Yuksel, B. F., Ottley, A., Jacob, R. J. K., & Chang, R. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470723 · Cited by 113
Mitra, T., Hutto, C. J., & Gilbert, E. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702553 · Cited by 102
Working Together Apart: Collaboration over the Internet
Olson, J. S., & Olson, G. M. (2014) · Synthesis Lectures on Human-Centered Informatics · doi:10.2200/s00542ed1v01y201310hci020 · Cited by 95
The Attraction Effect in Information Visualization
Dimara, E., Bezerianos, A., & Dragicevic, P. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598594 · Cited by 89
Utility of human-computer interactions: toward a science of preference measurement
Toomim, M., Kriplean, T., Pörtner, C., & Landay, J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979277 · Cited by 64
Early and Repeated Exposure to Examples Improves Creative Work
Kulkarni, C., Dow, S. P., & Klemmer, S. R. (2014) · Design Thinking Research · doi:10.1007/978-3-319-01303-9_4 · Cited by 61
Complementing Text Entry Evaluations with a Composition Task
Vertanen, K., & Kristensson, P. O. (2014) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2555691 · Cited by 62
Turkers, Scholars, "Arafat" and "Peace": Cultural Communities and Algorithmic Gold Standards
Sen, S., Giesel, M. E., Gold, R., Hillmann, B., Lesicko, M., Naden, S., Russell, J., Wang, Z. (., & Hecht, B. (2015) · Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing · doi:10.1145/2675133.2675285 · Cited by 47
A Large Inclusive Study of Human Listening Rates
Bragg, D., Bennett, C., Reinecke, K., & Ladner, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174018 · Cited by 43
Metadata
- Authors
- Heer, J., & Bostock, M.
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2010-04-10
- Pages
- 203-212
- OpenAlex
- W2117470435
- Cited by
- 816
- References
- 32
- FWCI
- 38.21
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.