CoChat
Sign Up
Cited by 48Usability and User Interface DesignInnovative Human-Technology InteractionPersona Design and Applications

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Evaluation of Prototypes and the Problem of Possible Futures

Open access
Salovaara, Antti, Oulasvirta, Antti, Jacucci, Giulio·2017·Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3025453.3025658
Open this paper in CoChat — free accountOpen PDF

Abstract

There is a blind spot in HCI's evaluation methodology: we rarely consider the implications of the fact that a prototype can never be fully evaluated in a study. A prototype under study exists firmly in the present world, in the circumstances created in the study, but its real context of use is a partially unknown future state of affairs. This present-future gap is implicit in any evaluation of prototypes, be they usability tests, controlled experiments, or field trials. A carelessly designed evaluation may inadvertently evaluate the wrong futures, contexts, or user groups, thereby leading to false conclusions and expensive design failures. The essay analyses evaluation methodology from this perspective, illuminating how to mitigate the present-future gap.

Overview

Integrity signals

Registry

Crossref · 10.1145/3025453.3025658

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 71

  1. 10.1147/sj.384.0508 · DOI
  2. 10.1109/MC.2016.229 · DOI
  3. Liam Bannon . 1991. From Human Factors to Human Actors: The Role of Psychology and Human-Computer Interaction Studies in System Design . In Design at Work: Cooperative Design of Computer Systems, Joan M . Greenbaum and Morten Kyng (Eds.). Lawrence Erlbaum , Hillsdale, NJ , Chapter 2, 25--44. Liam Bannon. 1991. From Human Factors to Human Actors: The Role of Psychology and Human-Computer Interaction Studies in System Design. In Design at Work: Cooperative Design of Computer Systems, Joan M. Greenbaum and Morten Kyng (Eds.). Lawrence Erlbaum, Hillsdale, NJ, Chapter 2, 25--44.
  4. Isabelle Baszanger and Nicolas Dodier . 2004 . Ethnography: Relating the Part to the Whole. In Qualitative Research: Theory, Methods and Practice (2 ed.), David Silverman (Ed.). Sage Publications, London, UK , Chapter 2 , 9 -- 34 . Isabelle Baszanger and Nicolas Dodier. 2004. Ethnography: Relating the Part to the Whole. In Qualitative Research: Theory, Methods and Practice (2 ed.), David Silverman (Ed.). Sage Publications, London, UK, Chapter 2, 9--34.
  5. 10.1145/2207676.2208347 · DOI
  6. 10.1145/2037373.2037396 · DOI
  7. Wiebe E. Bijker Thomas P. Hughes and Trevor J. Pinch (Eds.). 1987. The Social Construction of Technological Systems: New Directions in the Sociology and History of Technology. The MIT Press Cambridge MA. Wiebe E. Bijker Thomas P. Hughes and Trevor J. Pinch (Eds.). 1987. The Social Construction of Technological Systems: New Directions in the Sociology and History of Technology. The MIT Press Cambridge MA.
  8. 10.1145/1240624.1240789 · DOI
  9. 10.1145/2317956.2318014 · DOI
  10. 10.5555/1289765.1289768 · DOI
  11. 10.1007/978-1-4471-0665-4_1 · DOI
  12. 10.1145/1978942.1979185 · DOI
  13. 10.1145/2858036.2858498 · DOI · Paper page
  14. 10.1145/347642.347652 · DOI
  15. 10.1145/67449.67452 · DOI
  16. Thomas D. Cook and Donald Thomas Campbell . 1979 . Quasi-Experimentation: Design and Analysis Issues for Field Settings. Houghton Mifflin, New York, NY. Thomas D. Cook and Donald Thomas Campbell. 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Houghton Mifflin, New York, NY.
  17. Ron Eglash , Jennifer L. Croissant , Giovanna Di Chiro, and Rayvon Fouché (Eds.) . 2004 . Appropriating Technology : Vernacular Science and Social Power. University of Minnesota Press , Minneapolis, MN. Ron Eglash, Jennifer L. Croissant, Giovanna Di Chiro, and Rayvon Fouché (Eds.). 2004. Appropriating Technology: Vernacular Science and Social Power. University of Minnesota Press, Minneapolis, MN.
  18. 10.1145/291224.291235 · DOI
  19. 10.1145/2207676.2208538 · DOI
  20. 10.1016/j.intcom.2011.03.005 · DOI
  21. Steve Harrison , Deborah Tatar , and Phoebe Sengers . 2007 . The Three Paradigms of HCI . In Proceedings of alt.chi 2007. Steve Harrison, Deborah Tatar, and Phoebe Sengers. 2007. The Three Paradigms of HCI. In Proceedings of alt.chi 2007.
  22. 10.1561/1100000043 · DOI
  23. 10.1145/2470654.2481312 · DOI
  24. Edwin Hutchins . 1995. Cognition in the Wild . The MIT Press , Cambridge, MA . Edwin Hutchins. 1995. Cognition in the Wild. The MIT Press, Cambridge, MA.
  25. 10.1145/642611.642616 · DOI · Paper page
  26. J. Hyndman and George Athanasopoulos . 2013 . Forecasting: Principles and Practice. OTexts , Melbourne, Australia . J. Hyndman and George Athanasopoulos. 2013. Forecasting: Principles and Practice. OTexts, Melbourne, Australia.
  27. 10.1007/s007790200031 · DOI
  28. 10.1145/347642.347715 · DOI
  29. 10.1145/1240624.1240817 · DOI
  30. 10.1145/2371574.2371586 · DOI
  31. 10.1145/2628363.2628398 · DOI
  32. 10.1007/978-3-540-28637-0_6 · DOI
  33. 10.1145/1028014.1028060 · DOI
  34. 10.1145/2556288.2557111 · DOI
  35. Morten Kyng and Joan Greenbaum (Eds.). 1991 . Design at Work: Cooperative Design of Computer Systems . Lawrence Erlbaum , Hillsdale, NJ . Morten Kyng and Joan Greenbaum (Eds.). 1991. Design at Work: Cooperative Design of Computer Systems. Lawrence Erlbaum, Hillsdale, NJ.
  36. 10.1145/2858036.2858446 · DOI
  37. Daniel MacKenzie and Judy Wajcman (Eds.). 1998. The Social Shaping of Technology (2 ed.). Open University Press Buckingham UK. Daniel MacKenzie and Judy Wajcman (Eds.). 1998. The Social Shaping of Technology (2 ed.). Open University Press Buckingham UK.
  38. 10.1016/j.intcom.2006.05.006 · DOI
  39. 10.1145/347642.347763 · DOI
  40. Bonnie A. Nardi . 1993. A Small Matter of Programming: Perspectives on End User Computing . The MIT Press , Cambridge, MA . Bonnie A. Nardi. 1993. A Small Matter of Programming: Perspectives on End User Computing. The MIT Press, Cambridge, MA.
  41. 10.4135/9781452229560.n11 · DOI
  42. Ulric Neisser . 1976. Cognition and Reality: Principles and Implications of Cognitive Psychology . W. H. Freeman and Company , San Francisco, CA . Ulric Neisser. 1976. Cognition and Reality: Principles and Implications of Cognitive Psychology. W. H. Freeman and Company, San Francisco, CA.
  43. Jakob Nielsen . 1993. Usability Engineering . Academic Press , Boston, MA . Jakob Nielsen. 1993. Usability Engineering. Academic Press, Boston, MA.
  44. 10.1080/07421222.1996.11518138 · DOI
  45. 10.1109/MCG.2008.75 · DOI
  46. 10.1145/2317956.2318008 · DOI
  47. Nelly Oudshoorn and Trevor Pinch (Eds.). 2003. How Users Matter: The Co-Construction of Users and Technology. The MIT Press Cambridge MA. Nelly Oudshoorn and Trevor Pinch (Eds.). 2003. How Users Matter: The Co-Construction of Users and Technology. The MIT Press Cambridge MA. · DOI · Paper page
  48. 10.1007/s00779-008-0209-0 · DOI
  49. 10.1145/2858036.2858283 · DOI
  50. 10.1007/s00779-003-0238-7 · DOI
  51. 10.1145/2207676.2208278 · DOI
  52. 10.1162/DESI_a_00395 · DOI
  53. 10.1007/978-3-540-74853-3_20 · DOI · Paper page
  54. Jeffrey Rubin and Dana Chisnell . 2008. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests (2 ed.) . Wiley Publishing, Indianapolis , IN. Jeffrey Rubin and Dana Chisnell. 2008. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests (2 ed.). Wiley Publishing, Indianapolis, IN.
  55. 10.1007/11555261_58 · DOI
  56. 10.1002/for.3980100602 · DOI
  57. Donald A. Schön . 1983. The Reflective Practitioner: How Professionals Think in Action . Basic Books , New York, NY . Donald A. Schön. 1983. The Reflective Practitioner: How Professionals Think in Action. Basic Books, New York, NY.
  58. Douglas Schuler and Aki Namioka (Eds.). 1993 . Participatory Design: Principles and Practices . Lawrence Erlbaum, Hillsdale , NJ. Douglas Schuler and Aki Namioka (Eds.). 1993. Participatory Design: Principles and Practices. Lawrence Erlbaum, Hillsdale, NJ.
  59. Herbert A. Simon . 1969. The Sciences of the Artificial . The MIT Press , Cambridge, MA . Herbert A. Simon. 1969. The Sciences of the Artificial. The MIT Press, Cambridge, MA.
  60. 10.1080/07370020903586696 · DOI
  61. Lucy A. Suchman . 1987. Plans and Situated Actions: The Problem of Human-Machine Communication . Cambridge University Press, Cambridge , UK. Lucy A. Suchman. 1987. Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge University Press, Cambridge, UK.
  62. Eric von Hippel . 1988. The Sources of Innovation . Oxford University Press, New York , NY. Eric von Hippel. 1988. The Sources of Innovation. Oxford University Press, New York, NY.
  63. Eric von Hippel . 2005. Democratizing Innovation . The MIT Press , Cambridge, MA . Eric von Hippel. 2005. Democratizing Innovation. The MIT Press, Cambridge, MA.
  64. 10.1007/978-1-4419-1153-7_1140 · DOI
  65. 10.1038/scientificamerican0991-94 · DOI · Paper page
  66. William Foote Whyte (Ed.). 1990. Participatory Action Research. Sage Publications Thousand Oaks CA. William Foote Whyte (Ed.). 1990. Participatory Action Research. Sage Publications Thousand Oaks CA.
  67. Terry Winograd , John Bennett , Laura De Young , and Bradley Hartfield (Eds.). 1996 . Bringing Design to Software. ACM Press , New York, NY. Terry Winograd, John Bennett, Laura De Young, and Bradley Hartfield (Eds.). 1996. Bringing Design to Software. ACM Press, New York, NY.
  68. Terry Winograd and Fernando Flores . 1986. Understanding Computers and Cognition: A New Foundation for Design . Addison-Wesley, Reading , MA. Terry Winograd and Fernando Flores. 1986. Understanding Computers and Cognition: A New Foundation for Design. Addison-Wesley, Reading, MA.
  69. 10.1145/1518701.1518874 · DOI
  70. 10.1145/1240624.1240704 · DOI
  71. 10.1145/1858171.1858228 · DOI

Cited by

Sorted by citations

Metadata

Authors
Salovaara, A., Oulasvirta, A., & Jacucci, G.
Affiliations
  • University of Helsinki (FI)
  • Aalto University (FI)
ORCID
Venue
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2017-05-02
Pages
2064-2077
DOI
10.1145/3025453.3025658
OpenAlex
W2610788932
Cited by
48
References
2
Citations by year
2026: 4 · 2025: 7 · 2024: 5 · 2023: 7 · 2022: 7 · 2021: 3 · 2020: 3 · 2019: 6 · 2018: 5 · 2017: 1
FWCI
3.10
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion)
  • Aaltodoc (Aalto University) (submittedVersion, other-oa)
  • Työväentutkimus Vuosikirja (submittedVersion, other-oa)
Funders
  • H2020 European Research Council (637991)
  • Suomen Akatemia (259281,298879)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
  • Physical Sciences › Computer Science › Human-Computer Interaction › Persona Design and Applications
Keywords
prototype testing, context of use, future scenarios, usability testing, field trials, controlled experiments, design failure, evaluation methodology, human-computer interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review