The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Design Study Methodology: Reflections from the Trenches and the Stacks
Abstract
Design studies are an increasingly popular form of problem-driven visualization research, yet there is little guidance available about how to do them effectively. In this paper we reflect on our combined experience of conducting twenty-one design studies, as well as reading and reviewing many more, and on an extensive literature review of other field work methods and methodologies. Based on this foundation we provide definitions, propose a methodological framework, and provide practical guidance for conducting design studies. We define a design study as a project in which visualization researchers analyze a specific real-world problem faced by domain experts, design a visualization system that supports solving this problem, validate the design, and reflect about lessons learned in order to refine visualization design guidelines. We characterize two axes - a task clarity axis from fuzzy to crisp and an information location axis from the domain expert's head to the computer - and use these axes to reason about design study contributions, their suitability, and uniqueness from other approaches. The proposed methodological framework consists of 9 stages: learn, winnow, cast, discover, design, implement, deploy, reflect, and write. For each stage we provide practical guidance and outline potential pitfalls. We also conducted an extensive literature survey of related methodological approaches that involve a significant amount of qualitative field work, and compare design study methodology to that of ethnography, grounded theory, and action research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Constructing Grounded Theory. A Practical Guide Through Qualitative Analysis
(2006) · QMiP Bulletin · doi:10.53841/bpsqmip.2006.1.2.36 · Cited by 11,828
Kruger, J., & Dunning, D. (1999) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.77.6.1121 · Cited by 7,731
Handbook of Qualitative Research
(1994) · Journal of Environmental Psychology · doi:10.1016/s0272-4944(05)80230-0 · Cited by 5,715
Ericsson (1993) · doi:10.1037/0033-295x.87.3.215 · Cited by 4,539
The eyes have it: a task by data type taxonomy for information visualizations
Shneiderman, B. (2002) · Proceedings 1996 IEEE Symposium on Visual Languages · doi:10.1109/vl.1996.545307 · Cited by 4,535
(2013) · Information Visualization · doi:10.1016/b978-0-12-381464-7.00018-1 · Cited by 3,742
Contextual Design: Defining Customer-Centered Systems
Beyer, H. R., & Holtzblatt, K. (1997) · Cited by 2,748
Research methods in Anthropology: Qualitative and quantitative approaches
van Arsdale, P. W. (1996) · Evaluation Practice · doi:10.1016/s0886-1633(96)90044-6 · Cited by 2,526
Interaction Design: Beyond Human-Computer Interaction
Rogers, Y., Sharp, H., & Preece, J. J. (2002) · Open Research Online - ORO (The Open University) · Cited by 1,843
Illuminating the Path: The Research and Development Agenda for Visual Analytics
Thomas, J. J., & Cook, K. A. (2005) · Cited by 1,770
Cited by
A Multi-Level Typology of Abstract Visualization Tasks
Brehmer, M., & Munzner, T. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.124 · Cited by 747
Self-citationA survey on information visualization: recent advances and challenges
Liu, S., Cui, W., Wu, Y., & Liu, M. (2014) · The Visual Computer · doi:10.1007/s00371-013-0892-3 · Cited by 373
A Systematic Review on the Practice of Evaluating Visualization
Isenberg, T., Isenberg, P., Chen, J., Sedlmair, M., & Moller, T. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.126 · Cited by 323
Self-citationKwon, B. C., Choi, M.-J., Kim, J. T., Choi, E., Kim, Y. B., Kwon, S., Sun, J., & Choo, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865027 · Cited by 294
“We initially generated research questions of our target users’ potential interest (led by medical experts), derived visual analytics tasks (led by deep learning and visual analytics experts), and then further evaluated them by closely following a design study methodology [70].”
MethodTemporal Event Sequence Simplification
Monroe, M., Lan, R., Lee, H., Plaisant, C., & Shneiderman, B. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.200 · Cited by 266
Degas, A., Islam, M. R., Hurter, C., Barua, S., Rahman, H., Poudel, M., Ruscio, D., Ahmed, M. U., Begum, S., Rahman, M. A., et al. (2022) · Applied Sciences · doi:10.3390/app12031295 · Cited by 168
Extending the Nested Model for User-Centric XAI: A Design Study on GNN-based Drug Repurposing
Wang, Q., Huang, K., Chandak, P., Zitnik, M., & Gehlenborg, N. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209435 · Cited by 82
How Does Automation Shape the Process of Narrative Visualization: A Survey of Tools
Chen, Q., Cao, S., Wang, J., & Cao, N. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3261320 · Cited by 71
“For example, some researchers have proposed visualization design process frameworks [149], [150], but do not explain what scenarios and how to use these frameworks for visualization design.”
BackgroundA Critical Reflection on Visualization Research: Where Do Decision Making Tasks Hide?
Dimara, E., & Stasko, J. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114813 · Cited by 60
Bernard, J., Sessler, D., Kohlhammer, J., & Ruddle, R. A. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2803829 · Cited by 55
Chishtie, J., Bielska, I. A., Barrera, A., Marchand, J.-S., Imran, M., Tirmizi, S. F. A., Turcotte, L. A., Munce, S., Shepherd, J., Senthinathan, A., et al. (2022) · Journal of Medical Internet Research · doi:10.2196/27534 · Cited by 49
Berendt, B., & Preibusch, S. (2014) · Artificial Intelligence and Law · doi:10.1007/s10506-013-9152-0 · Cited by 44
Troubling Collaboration: Matters of Care for Visualization Design Study
Akbaba, D., Lange, D., Correll, M., Lex, A., & Meyer, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581168 · Cited by 40
Self-citationExploring Discrimination: A User-centric Evaluation of Discrimination-Aware Data Mining
Berendt, B., & Preibusch, S. (2012) · 2012 IEEE 12th International Conference on Data Mining Workshops · doi:10.1109/icdmw.2012.109 · Cited by 24
“Pertinent methodology comes from design studies and visual data mining [20].”
MethodFunding the Frontier: Visualizing the Broad Impact of Science and Science Funding
Wang, Y., Qian, Y., Qi, X., Yin, Y., Dang, S., Qian, Z., Jones, B. F., Cao, N., & Wang, D. (2025) · arXiv · doi:10.48550/arxiv.2509.16323
Metadata
- Authors
- Sedlmair, M., Meyer, M., & Munzner, T.
- Venue
- IEEE Transactions on Visualization and Computer Graphics
- Published
- 2012-12
- Volume
- 18
- Issue
- 12
- Pages
- 2431-2440
- ISSN
- 1077-2626 (ISSN-L)
- OpenAlex
- W1970569592
- Cited by
- 826
- References
- 71
- FWCI
- 33.44
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.