Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience
Abstract
This paper reviews how empirical research on User Experience (UX) is conducted. It integrates products, dimensions of experience, and methodologies across a systematically selected sample of 51 publications from 2005-2009, reporting a total of 66 empirical studies. Results show a shift in the products and use contexts that are studied, from work towards leisure, from controlled tasks towards open use situations, and from desktop computing towards consumer products and art. Context of use and anticipated use, often named key factors of UX, are rarely researched. Emotions, enjoyment and aesthetics are the most frequently assessed dimensions. The methodologies used are mostly qualitative, and known from traditional usability studies, though constructive methods with unclear validity are being developed and used. Many studies use self-developed questionnaires without providing items or statistical validations. We discuss underexplored research questions and potential improvements of UX research.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In HCI and user experience studies, scale development procedures are not commonly used.
“Specifically, in HCI and user experience studies, scale development procedures are not commonly used (Bargas-Avila & Hornbæk, 2011) which highlights the necessity to introduce the benefits of applying validated scales to the HCI research community”
From Development and Validation of a Basic Psychological Needs Scale for Technology Use (2024) · cites this paper as Bargas-Avila & Hornbæk, 2011
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826 · see citing works
Agarwal, R., & Karahanna, E. (2000) · MIS Quarterly · doi:10.2307/3250951 · Cited by 4,726 · see citing works
What are emotions? And how can they be measured?
Scherer, K. R. (2005) · Social Science Information · doi:10.1177/0539018405058216 · Cited by 4,264 · see citing works
Tractinsky, N., Katz, A. S., & Ikar, D. (2000) · Interacting with Computers · doi:10.1016/s0953-5438(00)00031-x · Cited by 1,376 · see citing works
The Thing and I: Understanding the Relationship Between User and Product
Hassenzahl, M. (2003) · Human-Computer Interaction Series · doi:10.1007/1-4020-2967-5_4 · Cited by 1,097 · see citing works
Handbook of Emotion Elicitation and Assessment
(2007) · doi:10.1093/oso/9780195169157.001.0001 · Cited by 348 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
On the HORYZON: Moderated online social therapy for long-term recovery in first episode psychosis
Alvarez-Jimenez, M., Bendall, S., Lederman, R., Wadley, G., Chinnery, G., Vargas, S., Larkin, M., Killackey, E., McGorry, P. D., & Gleeson, J. F. (2013) · Schizophrenia Research · doi:10.1016/j.schres.2012.10.009 · Cited by 224 · see citing works
Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study
Hämäläinen, P., Tavast, M., & Kunnari, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580688 · Cited by 248 · see citing works
Usability, user experience and accessibility: towards an integrative model
Sauer, J., Sonderegger, A., & Schmutz, S. (2020) · Ergonomics · doi:10.1080/00140139.2020.1774080 · Cited by 195 · see citing works
The State of Speech in HCI: Trends, Themes and Challenges
Clark, L., Doyle, P., Garaialde, D., Gilmartin, E., Schlögl, S., Edlund, J., Aylett, M., Cabral, J., Munteanu, C., Edwards, J., et al. (2019) · Interacting with Computers · doi:10.1093/iwc/iwz016 · Cited by 193 · see citing works
“Papers were selected using adapted PRISMA guidelines, similar to the adapted QUORUM procedures in previous reviews (Bargas-Avila & Hornbæk, 2011; Mekler, Bopp, Tuch, & Opwis, 2014).”
MethodAlvarez-Jimenez, M., Gleeson, J. F., Bendall, S., Penn, D. L., Yung, A. R., Ryan, R. M., Eleftheriadis, D., D'Alfonso, S., Rice, S., Miles, C., et al. (2018) · Schizophrenia Research · doi:10.1016/j.schres.2018.07.022 · Cited by 118 · see citing works
Alvarez-Jimenez, M., Rice, S., D'Alfonso, S., Leicester, S., Bendall, S., Pryor, I., Russon, P., McEnery, C., Santesteban-Echarri, O., Da Costa, G., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/17155 · Cited by 106 · see citing works
Exploring perceptions of web accessibility: a survey approach
Yesilada, Y., Brajnik, G., Vigo, M., & Harper, S. (2015) · Behaviour & Information Technology · doi:10.1080/0144929x.2013.848238 · Cited by 99 · see citing works
The Usability Construct: A Dead End?
Tractinsky, N. (2018) · Human–Computer Interaction · doi:10.1080/07370024.2017.1298038 · Cited by 87 · see citing works
Measuring Voice UX Quantitatively: A Rapid Review
Seaborn, K., & Urakami, J. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451712 · Cited by 55 · see citing works
Prejudices, memories, expectations and confidence influence experienced accessibility on the Web
Aizpurua, A., Arrue, M., & Vigo, M. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.04.035 · Cited by 50 · see citing works
“With the aim of bridging the HCI and UX research, Bargas-Avila and Hornbæk (2011) conducted a systematic survey on how previous UX studies address issues like situation, context of use, dimensions, when is UX assessed and how UX data is collected.”
MethodDevelopment and Validation of a Basic Psychological Needs Scale for Technology Use
Moradbakhti, L., Leichtmann, B., & Mara, M. (2024) · Psychological Test Adaptation and Development · doi:10.1027/2698-1866/a000062 · Cited by 41 · see citing works
Older adults and hackathons: a qualitative study
Kopeć, W., Balcerzak, B., Nielek, R., Kowalik, G., Wierzbicki, A., & Casati, F. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9565-6 · Cited by 39 · see citing works
Culturally sensitive user interface design: a case study with German and Vietnamese users
Lachner, F., Nguyen, M.-A., & Butz, A. (2018) · Proceedings of the Second African Conference for Human Computer Interaction: Thriving Communities · doi:10.1145/3283458.3283459 · Cited by 19 · see citing works
Love it or hate it!: interactivity and user types
Hart, J., Sutcliffe, A. G., & De Angeli, A. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2481282 · Cited by 15 · see citing works
- Authors
- Bargas-Avila, J. A., & Hornbæk, K.
- Affiliations
- University of Basel (CH)
- University of Copenhagen (DK)
- ORCID
- Hornbæk, Kasper
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2011-05-07
- Pages
- 2689-2698
- OpenAlex
- W2020639717
- Cited by
- 660
- References
- 7
- Citations by year
- 2026: 31 · 2025: 69 · 2024: 57 · 2023: 37 · 2022: 38 · 2021: 39 · 2020: 39 · 2019: 42 · 2018: 50 · 2017: 49 · 2016: 41 · 2015: 52 · 2014: 38 · 2013: 41 · 2012: 28
- FWCI
- 26.94
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Persona Design and Applications
- Keywords
- user experience research, context of use, emotions, enjoyment, aesthetics, consumer goods, usability testing, questionnaire validation, research methodology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
