Evaluation in participatory design: a literature survey
Abstract
This paper focuses on evaluation in Participatory Design (PD), and especially upon how the central aims of mutual learning, empowerment, democracy and workplace quality have been assessed. We surveyed all Participatory Design Conference papers (1990-2014) and papers from special journal issues on PD, focusing on systematic, explicit evaluations. The survey resulted in 143 papers of which 66 were deemed relevant. Of these, 17 papers deal with evaluation of the above mentioned aims. Based on evaluation theory, we propose seven key questions through which to characterize evaluations in PD and analyze the 17 papers. Our analysis reveals that formal evaluations of PD's aims are rare; generally lack details on methods; are researcher- and not participant-led, and that a corpus of work around evaluations needs to be developed. We suggest more explicit, systematic evaluations of PD's central aims to enhance accountability, learning and knowledge building, and to strengthen PD internally and externally.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Each of the aforementioned methods and meanings of PD has different implications on the politics of design, the definition of participation, and the goals of the participatory process.
“Each of the aforementioned methods and meanings of PD has different implications on the politics of design, the definition of participation, and the goals of the participatory process”
From An introduction to radical participatory design: decolonising participatory design processes (2022) · cites this paper as Bossen, Dindler & Iversen 2016
Today, there is still no organic consensus on how to evaluate participatory processes.
“Today, there is still no organic consensus on how to evaluate participatory processes”
From An introduction to radical participatory design: decolonising participatory design processes (2022) · cites this paper as Bossen et al. 2016
Formal evaluations of participatory processes are uncommon, often lack details, and are not done in the same participatory way as the participatory project that is being evaluated.
“In fact, formal evaluations are uncommon, often lack details, and are not done in the same participatory way as the, hopefully, participatory project that is being evaluated”
From An introduction to radical participatory design: decolonising participatory design processes (2022) · cites this paper as Bossen et al. 2016
Cites
(2014) · SAGE Publications Ltd eBooks · doi:10.4135/9781446294406.n148 · Cited by 7,272 · see citing works
Utilization-Focused Evaluation
(2018) · The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation · doi:10.4135/9781506326139.n729 · Cited by 2,315 · see citing works
Routledge International Handbook of Participatory Design
(2012) · doi:10.4324/9780203108543 · Cited by 1,606 · see citing works
Participatory Design: Issues and Concerns
Kensing, F., & Blomberg, J. (1998) · Computer Supported Cooperative Work (CSCW) · doi:10.1023/a:1008689307411 · Cited by 1,094 · see citing works
Work‐Oriented Design of Computer Artifacts
Hakken, D. (1989) · Anthropology of Work Review · doi:10.1525/awr.1989.10.4.14 · Cited by 717 · see citing works
The changing paradigms of project management
Pollack, J. (2006) · International Journal of Project Management · doi:10.1016/j.ijproman.2006.08.002 · Cited by 349 · see citing works
In pursuit of rigour and accountability in participatory design
Frauenberger, C., Good, J., Fitzpatrick, G., & Iversen, O. S. (2014) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2014.09.004 · Cited by 267 · see citing works
Self-citationDahler‐Larsen, P. (2011) · Stanford University Press eBooks · doi:10.11126/stanford/9780804776929.001.0001 · Cited by 250 · see citing works
Participatory design in community computing contexts
Merkel, C., Xiao, L., Farooq, U., et al. (2004) · doi:10.1145/1011870.1011872 · Cited by 142 · see citing works
User gains and PD aims: assessment from a participatory design project
Bossen, C., Dindler, C., & Iversen, O. S. (2010) · Proceedings of the 11th Biennial Participatory Design Conference · doi:10.1145/1900441.1900461 · Cited by 105 · see citing works
Self-citationIntroducing participatory design in museums
Taxén, G. (2004) · doi:10.1145/1011870.1011894 · Cited by 99 · see citing works
Shapiro, D. (2005) · doi:10.1145/1094562.1094567 · Cited by 83 · see citing works
MUST: A Method for Participatory Design
Kensing, F., Simonsen, J., & Bødker, K. (1998) · Human-Computer Interaction · doi:10.1207/s15327051hci1302_3 · Cited by 74 · see citing works
Assessing participative systems design: Some conclusions from an exploratory study
Hirschheim, R. (1983) · Information & Management · doi:10.1016/0378-7206(83)90040-x · Cited by 73 · see citing works
Bossen, C., Dindler, C., & Iversen, O. S. (2012) · doi:10.1145/2347635.2347641 · Cited by 73 · see citing works
Self-citationKapuire, G. K., Winschiers‐Theophilus, H., & Blake, E. (2014) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2014.10.004 · Cited by 65 · see citing works
Ethics: engagement, representation and politics-in-action
Trigg, R., & Ishimaru, K. (2012) · Routledge eBooks · doi:10.4324/9780203108543-11 · Cited by 64 · see citing works
Program Evaluation: A Historical Overview
Madaus, G. F., & Stufflebeam, D. L. (2005) · Kluwer Academic Publishers eBooks · doi:10.1007/0-306-47559-6_1 · Cited by 62 · see citing works
Modeling the Knowledge Perspective of IT Projects
Reich, B. H., Gemino, A., & Sauer, C. (2008) · Project Management Journal · doi:10.1002/pmj.20056 · Cited by 51 · see citing works
Participative design and the challenges of large-scale systems: extending the iterative PD approachPreprint
Simonsen, J., & Hertzum, M. (2008) · doi:10.5555/1795234.1795236 · Cited by 40 · see citing works
How much participation is enough?
Whittle, J. (2014) · doi:10.1145/2661435.2661445 · Cited by 34 · see citing works
Gerrard, V., & Sosa, R. (2014) · doi:10.1145/2661435.2661451 · Cited by 34 · see citing works
Instant card technique: how and why to apply in user-centered design
Beck, E., Obrist, M., Bernhaupt, R., et al. (2008) · Participatory Design Conference · doi:10.5555/1795234.1795261 · Cited by 33 · see citing works
Design decisions and the sharing of power in PD
Bratteteig, T., & Wagner, I. (2014) · Participatory Design Conference · doi:10.1145/2662155.2662192 · Cited by 31 · see citing works
Effects-driven IT development: an instrument for supporting sustained participatory design
Hertzum, M., & Simonsen, J. (2010) · Participatory Design Conference · doi:10.1145/1900441.1900451 · Cited by 25 · see citing works
Participants' view on personal gains and PD process
Garde, J. A., & van der Voort, M. C. (2014) · Participatory Design Conference · doi:10.1145/2662155.2662194 · Cited by 19 · see citing works
Hornecker, E., Eden, H., & Scharff, E. (2002) · Participatory Design Conference · Cited by 12 · see citing works
Participatory Design Projects: A Retrospective Look
Clement, A., Griffiths, M., & van den Besselaar, P. (1992) · Participatory Design Conference · Cited by 12 · see citing works
Real-use evaluation of effects
Simonsen, J., & Hertzum, M. (2012) · doi:10.1145/2348144.2348166 · Cited by 6 · see citing works
Kiniery, P., & Ward, H. F. (1940) · The American Catholic Sociological Review · doi:10.2307/3706771 · Cited by 6 · see citing works
Participatory design of public library e-services
Costantino, T., LeMay, S., Vizard, L., Moore, H., Renton, D., Gornall, S., & Strang, I. (2014) · Participatory Design Conference · doi:10.1145/2662155.2662232 · Cited by 5 · see citing works
Influencing System Quality by Using Decision Diaries in Prototyping Projects
Braa, K. (1992) · Participatory Design Conference · Cited by 5 · see citing works
Voices in Design: The dynamics of participatory information system design
Timpka, T., & Sjöberg, C. M. (1994) · Participatory Design Conference · Cited by 5 · see citing works
Hallberg, N., Pilemalm, S., Jägare, S., et al. (2000) · Participatory Design Conference · Cited by 4 · see citing works
Hänninen, R., & Blom, J. (2012) · doi:10.1145/2348144.2348151 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Iversen, O. S., Smith, R. C., & Dindler, C. (2017) · Proceedings of the 2017 Conference on Interaction Design and Children · doi:10.1145/3078072.3079725 · Cited by 306 · see citing works
“These have surfaced recently in retrospective studies assessing what people gain from PD projects [7,8] and in frameworks suggesting how such outcomes might be evaluated [16].”
BackgroundSelf-citationAn introduction to radical participatory design: decolonising participatory design processes
Udoewa, V. (2022) · Design Science · doi:10.1017/dsj.2022.24 · Cited by 77 · see citing works
“In fact, formal evaluations are uncommon, often lack details, and are not done in the same participatory way as the, hopefully, participatory project that is being evaluated (Bossen et al. 2016).”
Background
- Authors
- Bossen, C., Dindler, C., & Iversen, O. S.
- Affiliations
- Aarhus University (DK)
- Venue
- Proceedings of the 14th Participatory Design Conference: Full papers - Volume 1
- Publisher
- ACM
- Published
- 2016-08-15
- Pages
- 151-160
- OpenAlex
- W2495526613
- Cited by
- 112
- References
- 35
- Citations by year
- 2026: 17 · 2025: 22 · 2024: 16 · 2023: 12 · 2022: 15 · 2021: 5 · 2020: 5 · 2019: 8 · 2018: 7 · 2017: 5
- FWCI
- 4.55
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Participatory Design Conference (publishedVersion)
- Scopus (Elsevier) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- participatory design, evaluation, mutual learning, empowerment, workplace democracy, workplace quality, evaluation theory, accountability, knowledge construction, assessment methods
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
