The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.
Empirical Studies in Information Visualization: Seven Scenarios
Abstract
We take a new, scenario-based look at evaluation in information visualization. Our seven scenarios, evaluating visual data analysis and reasoning, evaluating user performance, evaluating user experience, evaluating environments and work practices, evaluating communication through visualization, evaluating visualization algorithms, and evaluating collaborative data analysis were derived through an extensive literature review of over 800 visualization publications. These scenarios distinguish different study goals and types of research questions and are illustrated through example studies. Through this broad survey and the distillation of these scenarios, we make two contributions. One, we encapsulate the current practices in the information visualization research community and, two, we provide a different approach to reaching decisions about what might be the most effective evaluation of a given information visualization. Scenarios can be used to choose appropriate research questions and goals and the provided examples can be consulted for guidance on how to design one's own study.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1998) · Sage eBooks · Cited by 38,631
Research Design: Qualitative, Quantitative, and Mixed Methods Approaches
Creswell, J. W. (1994) · Cited by 35,473
Research design: qualitative, quantitative, and mixed methods approaches
Caldas, M. P. (2003) · Revista de Administração Contemporânea · doi:10.1590/s1415-65552003000100015 · Cited by 18,745
Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory
Corbin, J., & Strauss, A. (2008) · SAGE · doi:10.4135/9781452230153 · Cited by 12,136
Designing the user interface: Strategies for effective human-computer interaction
(1993) · Applied Ergonomics · doi:10.1016/0003-6870(93)90471-k · Cited by 5,587
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Illuminating the Path: The Research and Development Agenda for Visual Analytics
Thomas, J. J., & Cook, K. A. (2005) · Cited by 1,770
Designing the user interface: strategies for effective human-computer interaction
Briggs, J. S. (1987) · Information and Software Technology · doi:10.1016/0950-5849(87)90040-1 · Cited by 1,146
Why CSCW applications fail: problems in the design and evaluationof organizational interfaces
Grudin, J. (1988) · Proceedings of the 1988 ACM conference on Computer-supported cooperative work - CSCW '88 · doi:10.1145/62266.62273 · Cited by 961
A Nested Model for Visualization Design and Validation
Munzner, T. (2009) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2009.111 · Cited by 880
The state of the art in automating usability evaluation of user interfaces
Ivory, M. Y., & Hearst, M. A. (2001) · ACM Computing Surveys · doi:10.1145/503112.503114 · Cited by 875
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
Heer, J., & Bostock, M. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753357 · Cited by 816
ManyEyes: a Site for Visualization at Internet Scale
Viegas, F. B., Wattenberg, M., van Ham, F., Kriss, J., & McKeon, M. (2007) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.70577 · Cited by 720
Group Cognition: Computer Support for Building Collaborative Knowledge
Stahl, G. (2006) · The MIT Press eBooks · doi:10.7551/mitpress/3372.001.0001 · Cited by 694
Usability evaluation considered harmful (some of the time)
Greenberg, S., & Buxton, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357074 · Cited by 598
The challenge of information visualization evaluation
Plaisant, C. (2004) · Proceedings of the working conference on Advanced visual interfaces · doi:10.1145/989863.989880 · Cited by 590
Self-citationVizster: Visualizing Online Social Networks
Heer, J., & Boyd, D. (2006) · Proceedings of the 2005 IEEE Symposium on Information Visualization (INFOVIS'05) · doi:10.1109/infovis.2005.39 · Cited by 542
Data mountain: using spatial memory for document management
Robertson, G., Czerwinski, M., Larson, K., Robbins, D. C., Thiel, D., & van Dantzich, M. (1998) · Proceedings of the 11th annual ACM symposium on User interface software and technology · doi:10.1145/288392.288596 · Cited by 497
Show Me: Automatic Presentation for Visual Analysis
Mackinlay, J., Hanrahan, P., & Stolte, C. (2007) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.70594 · Cited by 480
Extracting usability information from user interface events
Hilbert, D. M., & Redmiles, D. F. (2000) · ACM Computing Surveys · doi:10.1145/371578.371593 · Cited by 477
Animated Transitions in Statistical Data Graphics
Heer, J., & Robertson, G. (2007) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.70539 · Cited by 438
Ordered and quantum treemaps: Making effective use of 2D space to display hierarchies
Bederson, B. B., Shneiderman, B., & Wattenberg, M. (2002) · ACM Transactions on Graphics · doi:10.1145/571647.571649 · Cited by 424
Casual Information Visualization: Depictions of Data in Everyday Life
Pousman, Z., Stasko, J., & Mateas, M. (2007) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.70541 · Cited by 421
Shneiderman, B., & Plaisant, C. (2006) · Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization · doi:10.1145/1168149.1168158 · Cited by 398
Self-citationEvaluating Information Visualizations
Carpendale, S. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-70956-5_2 · Cited by 364
Self-citationSpaceTree: supporting exploration in large node link tree, design evolution and empirical evaluation
Plaisant, C., Grosjean, J., & Bederson, B. B. (2003) · IEEE Symposium on Information Visualization, 2002. INFOVIS 2002. · doi:10.1109/infvis.2002.1173148 · Cited by 313
Self-citationPinelle, D., Gutwin, C., & Greenberg, S. (2003) · ACM Transactions on Computer-Human Interaction · doi:10.1145/966930.966932 · Cited by 310
METHODOLOGY MATTERS: DOING RESEARCH IN THE BEHAVIORAL and SOCIAL SCIENCES
MCGRATH, J. E. (1995) · Readings in Human–Computer Interaction · doi:10.1016/b978-0-08-051574-8.50019-4 · Cited by 296
Scented Widgets: Improving Navigation Cues with Embedded Visualizations
Willett, W., Heer, J., & Agrawala, M. (2007) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.70589 · Cited by 276
How Information Visualization Novices Construct Visualizations
Grammel, L., Tory, M., & Storey, M. (2010) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2010.164 · Cited by 264
Voyagers and voyeurs: supporting asynchronous collaborative information visualization
Heer, J., Viégas, F. B., & Wattenberg, M. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240781 · Cited by 261
Clutter Reduction in Multi-Dimensional Data Visualization Using Dimension Reordering
Peng, W., Ward, M. O., & Rundensteiner, E. A. (2004) · Cited by 251
Gutwin, C., & Greenberg, S. (2002) · Proceedings IEEE 9th International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET ICE 2000) · doi:10.1109/enabl.2000.883711 · Cited by 225
Evaluating computer-supported cooperative work: models and frameworks
Neale, D. C., Carroll, J. M., & Rosson, M. B. (2004) · Proceedings of the 2004 ACM conference on Computer supported cooperative work · doi:10.1145/1031607.1031626 · Cited by 224
Evaluating visualizations: do expert reviews work?
Tory, M., & Moller, T. (2005) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2005.102 · Cited by 216
Contextual Inquiry: A Participatory Technique for System Design
Karen, H., & Sandra, J. (2017) · Participatory Design · doi:10.1201/9780203744338-9 · Cited by 190
Groupware walkthrough: adding context to groupware usability evaluation
Pinelle, D., & Gutwin, C. A. (2002) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/503376.503458 · Cited by 185
Vizster: visualizing online social networks
Heer, J., & Boyd, D. (2005) · IEEE Symposium on Information Visualization, 2005. INFOVIS 2005. · doi:10.1109/infvis.2005.1532126 · Cited by 182
Heuristic Evaluation of Groupware Based on the Mechanics of Collaboration
Baker, K., Greenberg, S., & Gutwin, C. A. (2001) · Lecture notes in computer science · doi:10.1007/3-540-45348-2_14 · Cited by 180
From Mice to Men - 24 Years of Evaluation in CHI
Barkhuus, L., & Rode, J. A. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.2180963 · Cited by 180
Clutter Reduction in Multi-Dimensional Data Visualization Using Dimension Reordering
Peng, W., Ward, M. O., & Rundensteiner, E. A. (2005) · IEEE Symposium on Information Visualization · doi:10.1109/infvis.2004.15 · Cited by 164
EMDialog: Bringing Information Visualization into the Museum
Hinrichs, U., Schmidt, H., & Carpendale, S. (2008) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2008.127 · Cited by 162
Self-citationHeuristics for information visualization evaluation
Zuk, T., Schlesier, L., Isenberg, P., Hancock, M., & Carpendale, S. (2006) · doi:10.1145/1168149.1168162 · Cited by 159
Self-citationEmpirical studies of information visualization: a meta-analysis
Chen, C., & Yu, Y. (2000) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2000.0422 · Cited by 151
An Insight-Based Longitudinal Study of Visual Analytics
Saraiya, P., North, C. L., Lam, V., & Duca, K. (2006) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2006.85 · Cited by 146
An explorative analysis of user evaluation studies in information visualisation
Ellis, G. P., & Dix, A. J. (2006) · doi:10.1145/1168149.1168152 · Cited by 137
Reading of electronic documents
Hornbæk, K., & Frøkjær, E. (2001) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365118 · Cited by 131
Situated evaluation for cooperative systems
Twidale, M. B., Randall, D. W., & Bentley, R. (1994) · doi:10.1145/192844.193066 · Cited by 127
Turning pictures into numbers: extracting and generating information from complex visualizations
Trafton, J. G., Kirschenbaum, S. S., Tsui, T. L., Miyamoto, R. T., Ballas, J. A., & Raymond, P. (2000) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2000.0419 · Cited by 127
An exploratory study of visual information analysis
Isenberg, P., Tang, A., & Carpendale, S. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357245 · Cited by 126
Self-citationA review of groupware evaluations
Pinelle, D., & Gutwin, C. A. (2002) · IEEE International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE) · doi:10.1109/enabl.2000.883709 · Cited by 117
Czerwinski, M. P., van Dantzich, M., Robertson, G. G., & Hoffman, H. G. (1999) · International Conference on Human-Computer Interaction · Cited by 117
An exploratory study of co-located collaborative visual analytics around a tabletop display
Isenberg, P., Fisher, D., Morris, M. R., Inkpen, K. M., & Czerwinski, M. P. (2010) · IEEE Symposium on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2010.5652880 · Cited by 113
Self-citationGrounded evaluation of information visualizations
Isenberg, P., Zuk, T., Collins, C. M., & Carpendale, S. (2008) · doi:10.1145/1377966.1377974 · Cited by 113
Self-citationAn Evaluation of Microarray Visualization Tools for Biological Insight
Saraiya, P., North, C. L., & Duca, K. (2005) · IEEE Symposium on Information Visualization · doi:10.1109/infvis.2004.5 · Cited by 113
Symon, G. (2004) · SAGE Publications Ltd eBooks · doi:10.4135/9781446280119.n9 · Cited by 112
Between aesthetics and utility: designing ambient information visualizations
Skog, T., Ljungblad, S., & Holmquist, L. E. (2004) · IEEE Symposium on Information Visualization · doi:10.1109/infvis.2003.1249031 · Cited by 108
Uncovering Clusters in Crowded Parallel Coordinates Visualizations
Artero, A. O., de Oliveira, M. C. F., & Levkowitz, H. (2005) · IEEE Symposium on Information Visualization · doi:10.1109/infvis.2004.68 · Cited by 107
Seo, J., & Shneiderman, B. (2006) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2006.50 · Cited by 104
Wongsuphasawat, K., & Shneiderman, B. (2009) · IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2009.5332595 · Cited by 103
Filtering and Brushing with Motion
Bartram, L., & Ware, C. (2002) · Information Visualization · doi:10.1057/palgrave.ivs.9500005 · Cited by 96
The Perceptual Scalability of Visualization
Yost, B., & North, C. L. (2006) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2006.184 · Cited by 93
Eccles, R., Kapler, T., Harper, R., et al. (2008) · Information Visualization · doi:10.1057/palgrave.ivs.9500173 · Cited by 91
Collaborative geographic visualization: enabling shared understanding of environmental processes
Brewer, I., MacEachren, A. M., Abdo, H. A., Gundrum, J., & Otto, G. H. (2002) · IEEE Symposium on Information Visualization · doi:10.1109/infvis.2000.885102 · Cited by 82
Information visualization evaluation in large companies: Challenges, experiences and recommendations
Sedlmair, M., Isenberg, P., Baur, D., & Butz, A. M. (2011) · Information Visualization · doi:10.1177/1473871611413099 · Cited by 72
Self-citationSpatialization Design: Comparing Points and Landscapes
Tory, M. K., Sprague, D., Wu, F., So, W. Y., & Munzner, T. (2007) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.70596 · Cited by 63
Perer, A., & Shneiderman, B. (2009) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2009.44 · Cited by 60
Alonso, D. L., Rose, A., Catherine Plaisant, B., & Norman, K. L. (1998) · Behaviour and Information Technology · doi:10.1080/014492998119328 · Cited by 59
Drury, J. L., & Williams, M. G. (2003) · IEEE International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE) · doi:10.1109/enabl.2002.1029981 · Cited by 56
Kulyk, O., Kosara, R., Urquiza‐Fuentes, J., & Wassink, I. (2007) · Springer eBooks · doi:10.1007/978-3-540-71949-6_2 · Cited by 54
Visualising Changes in Fund Manager Holdings in Two and a Half-Dimensions
Dwyer, T., & Gallagher, D. R. (2004) · Information Visualization · doi:10.1057/palgrave.ivs.9500081 · Cited by 53
Adapting the Locales Framework for Heuristic Evaluation of Groupware
Greenberg, S., Fitzpatrick, G., Gutwin, C. A., & Kaplan, S. M. (2000) · AJIS. Australasian journal of information systems/AJIS. Australian journal of information systems/Australian journal of information systems · doi:10.3127/ajis.v7i2.267 · Cited by 42
Sedig, K., Rowhani, S., Morey, J., & Liang, H. (2003) · Information Visualization · doi:10.1057/palgrave.ivs.9500047 · Cited by 37
The Effects of Collaboration and System Transparency on CIVE Usage: An Empirical Study and Model
Mark, G. J., & Kobsa, A. (2005) · PRESENCE Virtual and Augmented Reality · doi:10.1162/1054746053890279 · Cited by 37
Qualitative analysis of visualization
Tory, M. K., & Staub‐French, S. (2008) · doi:10.1145/1377966.1377975 · Cited by 34
Density Displays for Data Stream Monitoring
Hao, M., Keim, D. A., Umeshwar Dayal, B., Oelke, D., & Tremblay, C. (2008) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2008.01222.x · Cited by 26
BELIV'08: Beyond time and errors
Bertini, E., Perer, A., Catherine Plaisant, B., & Santucci, G. (2008) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1358628.1358955 · Cited by 24
Self-citationArtifacts of the Presence Era: Using Information Visualization to Create an Evocative Souvenir
Viégas, F. B., Perry, E., Howe, E., & Donath, J. (2005) · IEEE Symposium on Information Visualization · doi:10.1109/infvis.2004.8 · Cited by 23
Visualizing Roles of Variables in Program Animation
Sajaniemi, J., & Kuittinen, M. (2004) · Information Visualization · doi:10.1057/palgrave.ivs.9500075 · Cited by 22
Increasing the utility of quantitative empirical studies for meta-analysis
Lam, H., & Munzner, T. (2008) · doi:10.1145/1377966.1377969 · Cited by 17
Self-citation
Cited by
Science Mapping: A Systematic Review of the Literature
Chen, C. (2017) · Journal of Data and Information Science · doi:10.1515/jdis-2017-0006 · Cited by 1,925
Design Study Methodology: Reflections from the Trenches and the Stacks
Sedlmair, M., Meyer, M., & Munzner, T. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.213 · Cited by 826
A Multi-Level Typology of Abstract Visualization Tasks
Brehmer, M., & Munzner, T. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.124 · Cited by 747
Opportunities and Challenges for Data Physicalization
Jansen, Y., Dragicevic, P., Isenberg, P., Alexander, J., Karnik, A., Kildal, J., Subramanian, S., & Hornbæk, K. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702180 · Cited by 571
Self-citationA survey on information visualization: recent advances and challenges
Liu, S., Cui, W., Wu, Y., & Liu, M. (2014) · The Visual Computer · doi:10.1007/s00371-013-0892-3 · Cited by 373
Metadata
- Authors
- Lam, H., Bertini, E., Isenberg, P., Plaisant, C., & Carpendale, S.
- Venue
- IEEE Transactions on Visualization and Computer Graphics
- Published
- 2012-09
- Volume
- 18
- Issue
- 9
- Pages
- 1520-1536
- ISSN
- 1077-2626 (ISSN-L)
- OpenAlex
- W2058203255
- Cited by
- 623
- References
- 80
- FWCI
- 31.74
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.