A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form
Open accessAbstract
User engagement (UE) and its measurement have been of increasing interest in human-computer interaction (HCI). The User Engagement Scale (UES) is one tool developed to measure UE, and has been used in a variety of digital domains. The original UES consisted of 31-items and purported to measure six dimensions of engagement: aesthetic appeal, focused attention, novelty, perceived usability, felt involvement, and endurability. A recent synthesis of the literature questioned the original six-factors. Further, the ways in which the UES has been implemented in studies suggests there may be a need for a briefer version of the questionnaire and more effective documentation to guide its use and analysis. This research investigated and verified a four-factor structure of the UES and proposed a Short Form (SF). We employed contemporary statistical tools that were unavailable during the UES’ development to re-analyze the original data, consisting of 427 and 779 valid responses across two studies, and examined new data ( N =344) gathered as part of a three-year digital library project. In this paper we detail our analyses, present a revised long and short form (SF) version of the UES, and offer guidance for researchers interested in adopting the UES and UES-SF in their own studies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The UES-SF is a 12-item self-report measure assessing determinants of adherence, with four factors: perceived usability, aesthetic appeal, focused attention, and reward.
“The User Engagement Scale–Short Form (UES-SF) [36] is a self-report measure comprised of 12 items rated on a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree).”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
School Engagement: Potential of the Concept, State of the Evidence
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004) · Review of Educational Research · doi:10.3102/00346543074001059 · Cited by 12,581 · see citing works
Dunn, T. J., Baguley, T. S., & Brunsden, V. (2013) · British Journal of Psychology · doi:10.1111/bjop.12046 · Cited by 3,758 · see citing works
mirt : A Multidimensional Item Response Theory Package for the R Environment
Chalmers, R. P. (2012) · Journal of Statistical Software · doi:10.18637/jss.v048.i06 · Cited by 2,715 · see citing works
The Rediscovery of Bifactor Measurement Models
Reise, S. P. (2012) · Multivariate Behavioral Research · doi:10.1080/00273171.2012.715555 · Cited by 2,112 · see citing works
What is user engagement? A conceptual framework for defining user engagement with technology
O'Brien, H. L., & Toms, E. G. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20801 · Cited by 1,397 · see citing works
Self-citationA Basis for Scaling Qualitative Data
Guttman, L. (1944) · American Sociological Review · doi:10.2307/2086306 · Cited by 1,333 · see citing works
The development and evaluation of a survey to measure user engagement
O'Brien, H. L., & Toms, E. G. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21229 · Cited by 794 · see citing works
Self-citationAudience engagement in multimedia presentations
Webster, J., & Ho, H. (1997) · ACM SIGMIS Database the DATABASE for Advances in Information Systems · doi:10.1145/264701.264706 · Cited by 382 · see citing works
Measuring engagement in video game-based environments: Investigation of the User Engagement Scale
Wiebe, E., Lamb, A., Frankosky, M. H., & Sharek, D. J. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2013.12.001 · Cited by 309 · see citing works
Vakkari, P. (2001) · Journal of Documentation · doi:10.1108/eum0000000007075 · Cited by 294 · see citing works
Webster, J., & Ahuja, J. S. (2006) · MIS Quarterly · doi:10.2307/25148744 · Cited by 250 · see citing works
Oh, J., & Sundar, S. (2015) · Journal of Communication · doi:10.1111/jcom.12147 · Cited by 224 · see citing works
Rowe, D. W., Sibert, J. L., & Irwin, D. (1998) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/274644.274709 · Cited by 192 · see citing works
Lalmas, M., O’Brien, H., & Yom-Tov, E. (2015) · Synthesis Lectures on Information Concepts, Retrieval, and Services · doi:10.1007/978-3-031-02289-0 · Cited by 192 · see citing works
Self-citationBartlett, F. C. (1947) · Nature · doi:10.1038/159005a0 · Cited by 189 · see citing works
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Atwood, M. E., Karat, C.-M., Lund, A. M., Coutaz, J., Karat, J., USA, C. A. C. O. H. F. A. C. S. 1. L. A. C., Association for Computing Machinery-Digital Library, ACM Special Interest Group on Computer-Human Interaction, & SIGs, A. (1998) · Cited by 186 · see citing works
Examining the generalizability of the User Engagement Scale (UES) in exploratory search
O'Brien, H. L., & Toms, E. G. (2012) · Information Processing & Management · doi:10.1016/j.ipm.2012.08.005 · Cited by 180 · see citing works
Self-citationThe Analysis and interpretation of multivariate data for social scientists
(2002) · Choice Reviews Online · doi:10.5860/choice.40-0338 · Cited by 174 · see citing works
User engagement in online N ews: Under the scope of sentiment, interest, affect, and gaze
Arapakis, I., Lalmas, M., Cambazoglu, B. B., Marcos, M., & Jose, J. M. (2014) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23096 · Cited by 131 · see citing works
Theoretical Perspectives on User Engagement
O’Brien, H. (2016) · Springer eBooks · doi:10.1007/978-3-319-27446-1_1 · Cited by 127 · see citing works
Self-citationDimensions of Reading Motivation: Development of an Adult Reading Motivation Scale
Schutte, N. S., & Malouff, J. M. (2007) · Reading Psychology · doi:10.1080/02702710701568991 · Cited by 125 · see citing works
A systematic review of interactive information retrieval evaluation studies, 1967–2006
Kelly, D., & Sugimoto, C. R. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22799 · Cited by 109 · see citing works
O'Brien, H., & Cairns, P. A. (2016) · Springer eBooks · doi:10.1007/978-3-319-27446-1 · Cited by 100 · see citing works
Self-citationMixed‐methods approach to measuring user experience in online news interactions
O'Brien, H. L., & Lebow, M. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22871 · Cited by 95 · see citing works
Self-citationDesigning for User Engagement: Aesthetic and Attractive User Interfaces
Sutcliffe, A. (2009) · Synthesis lectures on human-centered informatics · doi:10.2200/s00210ed1v01y200910hci005 · Cited by 91 · see citing works
Designing for User Engagment: Aesthetic and Attractive User Interfaces
Sutcliffe, A. (2009) · CERN Document Server (European Organization for Nuclear Research) · Cited by 86 · see citing works
An empirical evaluation of the User Engagement Scale (UES) in online news environments
O'Brien, H. L., & Cairns, P. A. (2015) · Information Processing & Management · doi:10.1016/j.ipm.2015.03.003 · Cited by 85 · see citing works
Self-citationExperimental IR Meets Multilinguality, Multimodality, and Interaction
Mothe, J., Savoy, J., Kamps, J., et al. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24027-5 · Cited by 75 · see citing works
Information Access Evaluation. Multilinguality, Multimodality, and Interaction
Kanoulas, E., Lupu, M., Clough, P., et al. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11382-1 · Cited by 74 · see citing works
Taub, M., Mudrick, N. V., Azevedo, R., Millar, G. C., Rowe, J., & Lester, J. C. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.01.038 · Cited by 72 · see citing works
Task complexity, vertical display and user interaction in aggregated search
Arguello, J., Wu, W., Kelly, D., & Edwards, A. (2012) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2348283.2348343 · Cited by 70 · see citing works
Questionnaire mode effects in interactive information retrieval experiments
Kelly, D., Harper, D. J., & Landau, B. J. (2007) · Information Processing & Management · doi:10.1016/j.ipm.2007.02.007 · Cited by 56 · see citing works
Building a Common Framework for IIR Evaluation
Hall, M. M., & Toms, E. G. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40802-1_3 · Cited by 32 · see citing works
Self-citationA Commentary on Short Questionnaires for Assessing Usability
Cairns, P. A. (2013) · Interacting with Computers · doi:10.1093/iwc/iwt019 · Cited by 32 · see citing works
Self-citationIs there a universal instrument for measuring interactive information retrieval?
O'Brien, H. L., & Toms, E. G. (2010) · doi:10.1145/1840784.1840835 · Cited by 30 · see citing works
Self-citationOverview of the CLEF 2016 Social Book Search Lab
Koolen, M., Bogers, T., Gäde, M., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-44564-9_29 · Cited by 29 · see citing works
Proceedings of the third symposium on Information interaction in context
Belkin, N. J., & Kelly, D. (2010) · doi:10.1145/1840784 · Cited by 28 · see citing works
Overview of the CLEF 2015 Social Book Search Lab
Koolen, M., Bogers, T., Gäde, M., et al. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24027-5_51 · Cited by 23 · see citing works
Translating Theory into Methodological Practice
O'Brien, H. (2016) · Why Engagement Matters · doi:10.1007/978-3-319-27446-1_2 · Cited by 23 · see citing works
Self-citationAntecedents and learning outcomes of online news engagement
O'Brien, H. L. (2017) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23854 · Cited by 21 · see citing works
Self-citationBellot, P., Bogers, T., Geva, S., et al. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11382-1_19 · Cited by 21 · see citing works
Measuring user engagement attributes in social networking application
Banhawi, F., & Ali, N. M. (2011) · International Conference on Semantic Technology and Information Retrieval (STAIR) · doi:10.1109/stair.2011.5995805 · Cited by 19 · see citing works
O'Brien, H. L., & McCay‐Peet, L. (2017) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3020165.3020167 · Cited by 13 · see citing works
Self-citationO’Brien, H., & Cairns, P. A. (2016) · Springer eBooks · Cited by 11 · see citing works
Self-citationO'Brien, H. L., & McKay, J. (2016) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.2016.14505301150 · Cited by 11 · see citing works
Self-citationMultimodal Affect Modeling in Task-Oriented Tutorial Dialogue.
Grafsgaard, J. F. (2014) · NCSU Libraries Repository (North Carolina State University Libraries) · Cited by 9 · see citing works
Selecting measures for research investigations.
Kazdin, A. E. (2015) · American Psychological Association eBooks · doi:10.1037/14805-013 · Cited by 6 · see citing works
An Exploration of Cursor tracking DataPreprint
Warnock, D., & Lalmas, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.00317 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- Devellis, R.F. , 2003. Scale Development: Theory and Applications (Applied Social Re- search Methods), second ed. Sage Publications, Inc .
- Edwards, A. , 2015. Engaged or Frustrated? Disambiguating Engagement and Frustration in Search. University of North Caroline at Chapel Hill Ph.D. thesis .
- Embretson, S.E. , Reise, S.P. , 2013. Item Response Theory. Psychology Press .
- Jacques, R.D. , 1996. The Nature of Engagement and its Role in Hypermedia Evaluation and Design. South Bank University Ph.D. thesis .
- Krueger, R.F. , Kling, K.C. , 2000. Self-report. In: Kazdin, A.E. (Ed.), Encyclopedia of psy- chology, 7. Oxford University Press, pp. 220-224 .
- O 'Brien, H.L. , 2008. Defining and Measuring Engagement in User Experiences with Tech- nology. Dalhousie University Ph.D. thesis .
- Tabachnick, B. , Fidell, L. , 2013. Using Multivariate Statistics (6th International Edition.
- Pearson, Boston, [Mass.] . (cover). ed.).
Cited by
Sorted by citations
Enhancing user engagement: The role of gamification in mobile apps
Bitrián, P., Buil, I., & Catalán, S. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2021.04.028 · Cited by 442 · see citing works
Gabrielli, S., Rizzi, S., Bassi, G., Carbone, S., Maimone, R., Marchesoni, M., & Forti, S. (2021) · JMIR mHealth and uHealth · doi:10.2196/27965 · Cited by 125 · see citing works
“The main descriptive analyses (ie, mean, standard deviation, and frequencies) were performed in order to assess the demographic characteristics of the overall sample (ie, age and gender), as well as the GAD-7 and the PSS-10 scores (ie, anxiety and stress symptoms, respectively), the total score and the five subscale s…”
BackgroundInfluentialMiller, S., Gilbert, S., Virani, V., & Wicks, P. (2020) · JMIR Human Factors · doi:10.2196/19713 · Cited by 117 · see citing works
“Similarly, an independent study by an external academic group unrelated to Ada sought to understand the applicability of a multidimensional short form User Engagement Scale [27] in mobile health apps, using the Ada symptom checker as an example [28].”
BackgroundExplainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers
Ghai, B., Liao, Q. V., Zhang, Y., Bellamy, R., & Mueller, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432934 · Cited by 106 · see citing works
Mclaughlin, M., Delaney, T., Hall, A., Byaruhanga, J., Mackie, P., Grady, A., Reilly, K., Campbell, E., Sutherland, R., Wiggers, J., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/23180 · Cited by 105 · see citing works
Human-centered evaluation of explainable AI applications: a systematic review
Kim, J., Maathuis, H., & Sent, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1456486 · Cited by 130 · see citing works
Does gamification engage users in online shopping?
García-Jurado, A., Torres-Jiménez, M., Leal-Rodríguez, A. L., & Castro-González, P. (2021) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2021.101076 · Cited by 81 · see citing works
Recommendations for implementing gamification for mental health and wellbeing
Cheng, V. W. S. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.586379 · Cited by 74 · see citing works
Czok, V., Krug, M., Müller, S., Huwer, J., & Weitzel, H. (2023) · Sustainability · doi:10.3390/su152115313 · Cited by 44 · see citing works
Jung, J.-Y., Qiu, S., Bozzon, A., & Gadiraju, U. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517653 · Cited by 43 · see citing works
Lukic, Y. X., Teepe, G. W., Fleisch, E., & Kowatsch, T. (2022) · JMIR Serious Games · doi:10.2196/39186 · Cited by 32 · see citing works
Radomski, A. D., Bagnell, A., Curtis, S., Hartling, L., & Newton, A. S. (2020) · JMIR Mental Health · doi:10.2196/15795 · Cited by 30 · see citing works
“Use of factor analysis [90] or multiple regression [91] could help to illuminate how different constructs of user experience relate to one another and to intervention use and how the constructs change over the course of treatment.”
Beccaluva, E. A., Catania, F., Arosio, F., & Garzotto, F. (2024) · Human–Computer Interaction · doi:10.1080/07370024.2023.2242837 · Cited by 24 · see citing works
Chen, C., Lee, S., Jang, E., & Sundar, S. S. (2024) · Proceedings of the Second International Symposium on Trustworthy Autonomous Systems · doi:10.1145/3686038.3686060 · Cited by 21 · see citing works
Wei, C. Z., Kim, Y.-H., & Kuzminykh, A. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3597139 · Cited by 14 · see citing works
Towards a Reflection in Creative Experience Questionnaire
Ford, C., & Bryan-Kinns, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581077 · Cited by 15 · see citing works
The Effects of Gamification on Learners’ Engagement According to Their Gamification User Types
dos Santos, L. F., Oliveira, W., Corrêa de Lima, A., de Castro Junior, A. A., & Hamari, J. (2026) · Technology, Knowledge and Learning · doi:10.1007/s10758-025-09866-2 · Cited by 11 · see citing works
Evaluasi User Engagement dan User Experience pada Aplikasi Satu Sehat
Halim, F., & Suwandy, R. (2023) · Journal of Information System Research (JOSH) · doi:10.47065/josh.v5i1.4346 · Cited by 2 · see citing works
Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation
Ko, J., Choi, J., Morales, C., Kim, D., & Ko, M. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790929 · Cited by 1 · see citing works
Hou, Y., Wang, S., Su, Z., Bi, L., Ye, X., & Zhang, Y. (2026) · Multimodal Technologies and Interaction · doi:10.3390/mti10090096
- Authors
- O’Brien, H. L., Cairns, P., & Hall, M.
- Affiliations
- University of York (GB)
- Martin Luther University Halle-Wittenberg (DE)
- Venue
- International Journal of Human-Computer Studies
- Publisher
- Elsevier BV
- Published
- 2018-04
- Volume
- 112
- Pages
- 28-39
- ISSN
- 1071-5819 (ISSN-L)
- OpenAlex
- W2782898451
- Cited by
- 1,014
- References
- 49
- Citations by year
- 2026: 186 · 2025: 203 · 2024: 140 · 2023: 135 · 2022: 116 · 2021: 93 · 2020: 79 · 2019: 43 · 2018: 19
- FWCI
- 58.28
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Journal of Human-Computer Studies (publishedVersion, cc-by-nc-nd)
- Open Research Online (The Open University) (acceptedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- cIRcle (University of British Columbia) (submittedVersion, cc-by-nc-nd)
- Open Collections
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Social Sciences › Psychology › Developmental and Educational Psychology › Flow Experience in Various Fields
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- user engagement, short-form questionnaire, factor structure, scale validation, digital libraries, aesthetic appeal, focused attention, novelty, perceived usability, questionnaire design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
