The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Design Study Methodology: Reflections from the Trenches and the Stacks
Abstract
Design studies are an increasingly popular form of problem-driven visualization research, yet there is little guidance available about how to do them effectively. In this paper we reflect on our combined experience of conducting twenty-one design studies, as well as reading and reviewing many more, and on an extensive literature review of other field work methods and methodologies. Based on this foundation we provide definitions, propose a methodological framework, and provide practical guidance for conducting design studies. We define a design study as a project in which visualization researchers analyze a specific real-world problem faced by domain experts, design a visualization system that supports solving this problem, validate the design, and reflect about lessons learned in order to refine visualization design guidelines. We characterize two axes - a task clarity axis from fuzzy to crisp and an information location axis from the domain expert's head to the computer - and use these axes to reason about design study contributions, their suitability, and uniqueness from other approaches. The proposed methodological framework consists of 9 stages: learn, winnow, cast, discover, design, implement, deploy, reflect, and write. For each stage we provide practical guidance and outline potential pitfalls. We also conducted an extensive literature survey of related methodological approaches that involve a significant amount of qualitative field work, and compare design study methodology to that of ethnography, grounded theory, and action research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Constructing Grounded Theory. A Practical Guide Through Qualitative Analysis
(2006) · QMiP Bulletin · doi:10.53841/bpsqmip.2006.1.2.36 · Cited by 11,828
Kruger, J., & Dunning, D. (1999) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.77.6.1121 · Cited by 7,731
Handbook of Qualitative Research
(1994) · Journal of Environmental Psychology · doi:10.1016/s0272-4944(05)80230-0 · Cited by 5,715
Ericsson (1993) · doi:10.1037/0033-295x.87.3.215 · Cited by 4,539
The eyes have it: a task by data type taxonomy for information visualizations
Shneiderman, B. (2002) · Proceedings 1996 IEEE Symposium on Visual Languages · doi:10.1109/vl.1996.545307 · Cited by 4,535
(2013) · Information Visualization · doi:10.1016/b978-0-12-381464-7.00018-1 · Cited by 3,742
Contextual Design: Defining Customer-Centered Systems
Beyer, H. R., & Holtzblatt, K. (1997) · Cited by 2,748
Research methods in Anthropology: Qualitative and quantitative approaches
van Arsdale, P. W. (1996) · Evaluation Practice · doi:10.1016/s0886-1633(96)90044-6 · Cited by 2,526
Interaction Design: Beyond Human-Computer Interaction
Rogers, Y., Sharp, H., & Preece, J. J. (2002) · Open Research Online - ORO (The Open University) · Cited by 1,843
Illuminating the Path: The Research and Development Agenda for Visual Analytics
Thomas, J. J., & Cook, K. A. (2005) · Cited by 1,770
Technology probes: inspiring design for and with families
Hutchinson, H., Mackay, W., Westerlund, B., Bederson, B. B., Druin, A., Plaisant, C., Beaudouin-Lafon, M., Conversy, S., Evans, H., Hansen, H., et al. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642616 · Cited by 1,471
Iterative and incremental developments. a brief history
Larman, C., & Basili, V. R. (2003) · Computer · doi:10.1109/mc.2003.1204375 · Cited by 1,164
A Nested Model for Visualization Design and Validation
Munzner, T. (2009) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2009.111 · Cited by 880
Self-citationDourish, P. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124855 · Cited by 860
Observing the User Experience: A Practitioner's Guide to User Research
Goodman, E., Kuniavsky, M., & Moed, A. (2003) · Cited by 791
A technique for drawing directed graphs
Gansner, E. R., Koutsofios, E., North, S. C., & Vo, K.-P. (1993) · IEEE Transactions on Software Engineering · doi:10.1109/32.221135 · Cited by 784
Guidelines for using multiple views in information visualization
Wang Baldonado, M. Q., Woodruff, A., & Kuchinsky, A. (2000) · Proceedings of the working conference on Advanced visual interfaces · doi:10.1145/345513.345271 · Cited by 747
Empirical Studies in Information Visualization: Seven Scenarios
Lam, H., Bertini, E., Isenberg, P., Plaisant, C., & Carpendale, S. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.279 · Cited by 623
Usability evaluation considered harmful (some of the time)
Greenberg, S., & Buxton, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357074 · Cited by 598
The challenge of information visualization evaluation
Plaisant, C. (2004) · Proceedings of the working conference on Advanced visual interfaces · doi:10.1145/989863.989880 · Cited by 590
Moving out from the control room: ethnography in system design
Hughes, J., King, V., Rodden, T., & Andersen, H. (1994) · Proceedings of the 1994 ACM conference on Computer supported cooperative work - CSCW '94 · doi:10.1145/192844.193065 · Cited by 526
Bias, R. G. (2005) · Cost-Justifying Usability · doi:10.1016/b978-012095811-5/50022-5 · Cited by 519
Rapid ethnography: time deepening strategies for HCI field research
Millen, D. R. (2000) · Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques · doi:10.1145/347642.347763 · Cited by 499
The relationship of action research to human-computer interaction
Hayes, G. R. (2011) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1993060.1993065 · Cited by 452
Agile Software Development: The Cooperative Game
Cockburn, A., Cunningham, W., van Bennekum, A., Fowler, M., & Thomas, D. A. (2006) · Cited by 437
Human factors in visualization research
Tory, M., & Moller, T. (2004) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2004.1260759 · Cited by 420
Shneiderman, B., & Plaisant, C. (2006) · Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization · doi:10.1145/1168149.1168158 · Cited by 398
New theoretical approaches for human‐computer interaction
Rogers, Y. (2004) · Annual Review of Information Science and Technology · doi:10.1002/aris.1440380103 · Cited by 386
Parallel prototyping leads to better design results, more divergence, and increased self-efficacy
Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1879831.1879836 · Cited by 367
Evaluating Information Visualizations
Carpendale, S. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-70956-5_2 · Cited by 364
An Insight-Based Methodology for Evaluating Bioinformatics Visualizations
Saraiya, P., North, C., & Duca, K. (2005) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2005.53 · Cited by 343
Low-level components of analytic activity in information visualization
Amar, R., Eagan, J., & Stasko, J. (2005) · IEEE Symposium on Information Visualization, 2005. INFOVIS 2005. · doi:10.1109/infvis.2005.1532136 · Cited by 343
Into the wild: challenges and opportunities for field trial methods
Brown, B., Reeves, S., & Sherwood, S. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979185 · Cited by 257
Evaluating visualizations: do expert reviews work?
Tory, M., & Moller, T. (2005) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2005.102 · Cited by 216
Participatory IT Design: Designing for Business and Workplace Realities
Bødker, K., Kensing, F., & Simonsen, J. (2004) · The MIT Press eBooks · doi:10.7551/mitpress/5249.001.0001 · Cited by 215
Ethnographic Field Methods and Their Relation to Design
Blomberg, J., Giacomi, J., Mosher, A., & Swenton-Wall, P. (2017) · Participatory Design · doi:10.1201/9780203744338-7 · Cited by 201
Rogers, Y., Connelly, K., Tedesco, L., Hazlewood, W., Kurtz, A., Hall, R. E., Hursey, J., & Toscos, T. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74853-3_20 · Cited by 199
How Not to Lie with Visualization
Rogowitz, B. E., Treinish, L. A., & Bryson, S. (1996) · Computers in Physics · doi:10.1063/1.4822401 · Cited by 195
Contextual Inquiry: A Participatory Technique for System Design
Karen, H., & Sandra, J. (2017) · Participatory Design · doi:10.1201/9780203744338-9 · Cited by 190
Design principles for visual communication
Agrawala, M., Li, W., & Berthouzoz, F. (2011) · Communications of the ACM · doi:10.1145/1924421.1924439 · Cited by 169
Evaluation of Artery Visualizations for Heart Disease Diagnosis
Borkin, M., Gajos, K., Peters, A., Mitsouras, D., Melchionna, S., Rybicki, F., Feldman, C., & Pfister, H. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.192 · Cited by 164
Reflexivity in digital anthropology
Rode, J. A. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978961 · Cited by 162
MizBee: A Multiscale Synteny Browser
Meyer, M., Munzner, T., & Pfister, H. (2009) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2009.167 · Cited by 159
Self-citationHeuristics for information visualization evaluation
Zuk, T., Schlesier, L., Isenberg, P., Hancock, M., & Carpendale, S. (2006) · doi:10.1145/1168149.1168162 · Cited by 159
Ethnography considered harmful
Crabtree, A., Rodden, T. A., Tolmie, P., & Button, G. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518835 · Cited by 149
Wagstaff, K. (2012) · arXiv · doi:10.48550/arxiv.1206.4656 · Cited by 148
Knowledge Precepts for Design and Evaluation of Information Visualizations
Amar, R., & Stasko, J. (2005) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2005.63 · Cited by 147
An Insight-Based Longitudinal Study of Visual Analytics
Saraiya, P., North, C. L., Lam, V., & Duca, K. (2006) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2006.85 · Cited by 146
Imagery paradigms: How vulnerable are they to experimenters' expectations?
Intons-Peterson, M. J. (1983) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.9.3.394 · Cited by 125
Hughes, J. A., O’Brien, J., Rodden, T. A., Rouncefield, M., & Blythin, S. (1997) · Designing Interactive Systems Conference · doi:10.1145/263552.263598 · Cited by 125
van Wijk, J. J. (2006) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2006.80 · Cited by 125
Lloyd, D., & Dykes, J. A. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.209 · Cited by 123
Cerebral: Visualizing Multiple Experimental Conditions on a Graph with Biological Context
Barsky, A., Munzner, T., Gardy, J. L., & Kincaid, R. H. (2008) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2008.117 · Cited by 115
Self-citationGrounded evaluation of information visualizations
Isenberg, P., Zuk, T., Collins, C. M., & Carpendale, S. (2008) · doi:10.1145/1377966.1377974 · Cited by 113
A very short, fairly interesting and reasonably cheap book about qualitative research
Georgiou, V. (2011) · Action Learning Research and Practice · doi:10.1080/14767333.2010.518383 · Cited by 108
Responsibilities and implications: further thoughts on ethnography and design
Dourish, P. (2007) · Conference on Designing for User Experience · doi:10.1145/1389908.1389941 · Cited by 96
Booshehrian, M., Möller, T. B., Peterman, R. M., & Munzner, T. (2012) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2012.03116.x · Cited by 93
Self-citationConfessions from a grounded theory PhD
Furniss, D. J., BLANDFORD, A. E., & Curzon, P. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978960 · Cited by 88
Interactive visualization of large graphs and networks
Hanrahan, P., & Munzner, T. (2000) · Cited by 84
Self-citationNEW THEORETICAL APPROACHES FOR HCI
Rogers, Y. (2004) · Cited by 83
Pathline: A Tool For Comparative Functional Genomics
Meyer, M., Wong, B., Styczynski, M. P., Munzner, T., & Pfister, H. (2010) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2009.01710.x · Cited by 76
Self-citationInformation visualization evaluation in large companies: Challenges, experiences and recommendations
Sedlmair, M., Isenberg, P., Baur, D., & Butz, A. M. (2011) · Information Visualization · doi:10.1177/1473871611413099 · Cited by 72
Self-citationMulteeSum: A Tool for Comparative Spatial and Temporal Gene Expression Data
Meyer, M., Munzner, T., DePace, A. H., & Pfister, H. (2010) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2010.137 · Cited by 63
Self-citationSession Viewer: Visual Exploratory Analysis of Web Session Logs
Lam, H., Russell, D., Tang, D., & Munzner, T. (2007) · 2007 IEEE Symposium on Visual Analytics Science and Technology · doi:10.1109/vast.2007.4389008 · Cited by 62
Self-citationInteractive Coordinated Multiple-View Visualization of Biomechanical Motion Data
Keefe, D. F., Ewert, M., Ribarsky, W., & Chang, R. (2009) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2009.152 · Cited by 62
Glaserian and Straussian grounded theory: similar or completely different?
van Niekerk, J. C., & Roode, J. D. (2009) · Proceedings of the 2009 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists · doi:10.1145/1632149.1632163 · Cited by 55
The Streams of Our Lives: Visualizing Listening Histories in Context
Baur, D., Seiffert, F., Sedlmair, M., & Boring, S. (2010) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2010.206 · Cited by 47
Self-citationWhat Does the User Want to See? What do the Data Want to Be?
Pretorius, A. J., & van Wijk, J. J. (2009) · Information Visualization · doi:10.1057/ivs.2009.13 · Cited by 45
Sedlmair, M., Isenberg, P., Baur, D., Mauerer, M., Pigorsch, C., & Butz, A. M. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979194 · Cited by 43
Self-citationTheoretical analysis and theory creation
Dix, A. J. (2008) · Cambridge University Press eBooks · doi:10.1017/cbo9780511814570.010 · Cited by 24
Reflections on QuestVis: A Visualization System for an Environmental Sustainability Model
Munzner, T., Barsky, A., & Williams, M. (2011) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/dfu.vol2.sciviz.2011.240 · Cited by 17
Self-citation
Cited by
A Multi-Level Typology of Abstract Visualization Tasks
Brehmer, M., & Munzner, T. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.124 · Cited by 747
Self-citationA survey on information visualization: recent advances and challenges
Liu, S., Cui, W., Wu, Y., & Liu, M. (2014) · The Visual Computer · doi:10.1007/s00371-013-0892-3 · Cited by 373
A Systematic Review on the Practice of Evaluating Visualization
Isenberg, T., Isenberg, P., Chen, J., Sedlmair, M., & Moller, T. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.126 · Cited by 323
Self-citationKwon, B. C., Choi, M.-J., Kim, J. T., Choi, E., Kim, Y. B., Kwon, S., Sun, J., & Choo, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865027 · Cited by 294
Temporal Event Sequence Simplification
Monroe, M., Lan, R., Lee, H., Plaisant, C., & Shneiderman, B. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.200 · Cited by 266
Metadata
- Authors
- Sedlmair, M., Meyer, M., & Munzner, T.
- Venue
- IEEE Transactions on Visualization and Computer Graphics
- Published
- 2012-12
- Volume
- 18
- Issue
- 12
- Pages
- 2431-2440
- ISSN
- 1077-2626 (ISSN-L)
- OpenAlex
- W1970569592
- Cited by
- 826
- References
- 71
- FWCI
- 33.44
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.