The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
The development and evaluation of a survey to measure user engagement
Abstract
Facilitating engaging user experiences is essential in the design of interactive systems. To accomplish this, it is necessary to understand the composition of this construct and how to evaluate it. Building on previous work that posited a theory of engagement and identified a core set of attributes that operationalized this construct, we constructed and evaluated a multidimensional scale to measure user engagement. In this paper we describe the development of the scale, as well as two large‐scale studies (N=440 and N=802) that were undertaken to assess its reliability and validity in online shopping environments. In the first we used Reliability Analysis and Exploratory Factor Analysis to identify six attributes of engagement: Perceived Usability, Aesthetics, Focused Attention, Felt Involvement, Novelty, and Endurability. In the second we tested the validity of and relationships among those attributes using Structural Equation Modeling. The result of this research is a multidimensional scale that may be used to test the engagement of software applications. In addition, findings indicate that attributes of engagement are highly intertwined, a complex interplay of user‐system interaction variables. Notably, Perceived Usability played a mediating role in the relationship between Endurability and Novelty, Aesthetics, Felt Involvement, and Focused Attention.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 80,032
Development and validation of brief measures of positive and negative affect: The PANAS scales.
Watson, D., Clark, L. A., & Tellegen, A. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.6.1063 · Cited by 32,433
User Acceptance of Computer Technology: A Comparison of Two Theoretical Models
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989) · Management Science · doi:10.1287/mnsc.35.8.982 · Cited by 25,709
(1991) · Cited by 15,554
Information Systems Success: The Quest for the Dependent Variable
DeLone, W. H., & McLean, E. R. (1992) · Information Systems Research · doi:10.1287/isre.3.1.60 · Cited by 10,697
What is coefficient alpha? An examination of theory and applications
Cortina, J. M. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.1.98 · Cited by 8,810
Measuring Presence in Virtual Environments: A Presence Questionnaire
Witmer, B. G., & Singer, M. J. (1998) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474698565686 · Cited by 6,371
Work and/or Fun: Measuring Hedonic and Utilitarian Shopping Value
Babin, B. J., Darden, W. R., & Griffin, M. (1994) · Journal of Consumer Research · doi:10.1086/209376 · Cited by 6,020
Usability Evaluation In Industry
(1996) · CRC Press eBooks · doi:10.1201/9781498710411 · Cited by 4,354
Patterns of cognitive appraisal in emotion.
Smith, C. A., & Ellsworth, P. C. (1985) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.48.4.813 · Cited by 4,045
Research works printed but not linked: 19
- Bojko A., 2006, Using eye tracking to compate web page designs: A case study, Journal of Usability Studies, 3, 112
- 10.1145/604471.604532 · doi:10.1145/604471.604532
- Chapman P.(1997).Models of Engagement: Intrinsically Motivated Interaction with Multimedia Learning Software. Unpublished master's thesis University of Waterloo Waterloo Ontario Canada.
- Clip Culture: Internet video, 2006, The Economist, 379, 73
- Feedback. (1995). The Penguin dictionary of psychology. Retrieved 29 April 2005 from xreferplus.http://www.xreferplus.com/entry/150004FinneranZhang
- Haywood N., 2005, People and Computers XIX — The Bigger Picture, Proceedings of HCI 2005
- Hebb D.O., 1972, Textbook of psychology
- Jacques R.D.(1996).The nature of engagement and its role in hypermedia evaluation and design. Unpublished doctoral dissertation South Bank University London.
- Laurel B., 1993, Computers as Theatre
- Madden M., 2007, Online Video
- Matlin M.W., 1994, Cognition
- Nunnally J.C., 1978, Pyschometric theory
- Pine B.J., 1999, The experience economy
- Quesenbery W., 2003, Content and complexity: Information design in technical communications, 81
- Schmidt K.E. Bauerly M. Liu Y. &Sridharan S.(2003).Web page aesthetics and performance: A survey and an experimental study. Paper presented at the International Conference on Industrial Engineering Las Vegas NV.
- Seah M.‐L. &Cairns P.(2008). From immersion to addiction in video games. In England D. & Beale R. (Eds.) Proceedings of Human Computer Interaction Conference Vol. 1 (HCI 2008) (pp. 55–63). Bedfordshire UK: British Computer Society. Retrieved October 8 2009 fromhttp://www.users.cs.york.ac.uk/∼pcairns/papers/Seah_Cairns_HCI2008.pdf
- Singleton R.A., 1993, Approaches to social science research
- 10.1037/0022-3514.48.4.813 · doi:10.1037/0022-3514.48.4.813 · Paper page
- Vick R.M. &Ikehara C.S.(2002). Methodological issues of real time data acquisition from multiple sources of physiological data. In Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS‐ 36). Los Alamitos CA: IEEE. Retrieved September 17 2009 fromhttp://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1174295&isnumber=26341 · doi:10.1109/hicss.2003.1174295
Cited by
Perski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,193
O’Brien, H. L., Cairns, P., & Hall, M. (2018) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2018.01.004 · Cited by 1,014
Self-citationKim, Y. H., Kim, D. J., & Wachter, K. (2013) · Decision Support Systems · doi:10.1016/j.dss.2013.07.002 · Cited by 769
The Faces of Engagement: Automatic Recognition of Student Engagementfrom Facial Expressions
Whitehill, J., Serpell, Z., Lin, Y.-C., Foster, A., & Movellan, J. R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2316163 · Cited by 677
Toward Meaningful Engagement: A Framework for Design and Research of Gamified Information Systems1
Liu, D., Santhanam, R., & Webster, J. (2017) · MIS Quarterly · doi:10.25300/misq/2017/41.4.01 · Cited by 521
Enhancing user engagement: The role of gamification in mobile apps
Bitrián, P., Buil, I., & Catalán, S. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2021.04.028 · Cited by 442
Carolan, S., Harris, P. R., & Cavanagh, K. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7583 · Cited by 385
“Other limitations specific to this review include the use in the positive deviance analysis of intervention completion and intervention group attrition as proxy measures of intervention engagement and adherence; the number of times that a participant logs in to an intervention or the number of modules that they comple…”
Short, C. E., DeSmet, A., Woods, C., Williams, S. L., Maher, C., Middelweerd, A., Müller, A. M., Wark, P. A., Vandelanotte, C., Poppe, L., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9397 · Cited by 400
“Both Perski et al and Yardley et al extend previous models [6,9-11] by considering the interaction between usage and psychological processes.”
BackgroundMethodInfluentialEngagement detection in online learning: a review
Dewan, M. A. A., Murshed, M., & Lin, F. (2019) · Smart Learning Environments · doi:10.1186/s40561-018-0080-z · Cited by 350
The Development and Validation of the Game User Experience Satisfaction Scale (GUESS)
Phan, M. H., Keebler, J. R., & Chaparro, B. S. (2016) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816669646 · Cited by 342
O’Brien, H. L. (2010) · Interacting with Computers · doi:10.1016/j.intcom.2010.04.001 · Cited by 317
Self-citationChandra, S., Shirish, A., & Srivastava, S. C. (2022) · Journal of Management Information Systems · doi:10.1080/07421222.2022.2127441 · Cited by 315
Current challenges and visions in music recommender systems research
Schedl, M., Zamani, H., Chen, C.-W., Deldjoo, Y., & Elahi, M. (2018) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-018-0154-2 · Cited by 315
Measuring engagement in video game-based environments: Investigation of the User Engagement Scale
Wiebe, E. N., Lamb, A., Hardy, M., & Sharek, D. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2013.12.001 · Cited by 306
This Logo Moves Me: Dynamic Imagery from Static Images
Cian, L., Krishna, A., & Elder, R. S. (2014) · Journal of Marketing Research · doi:10.1509/jmr.13.0023 · Cited by 303
Högberg, J., Hamari, J., & Wästlund, E. (2019) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-019-09223-w · Cited by 299
Consumer Grade EEG Measuring Sensors as Research Tools: A Review
Sawangjai, P., Hompoonsup, S., Leelaarporn, P., Kongwudhikunakorn, S., & Wilaiprasitporn, T. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2962874 · Cited by 267
Lehmann, J., Lalmas, M., Yom-Tov, E., & Dupret, G. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-31454-4_14 · Cited by 261
The Audience-Oriented Editor: Making sense of the audience in the newsroom
Ferrer-Conill, R., & Tandoc, E. C. (2018) · Digital Journalism · doi:10.1080/21670811.2018.1440972 · Cited by 257
What Makes Live Events Engaging on Facebook Live, Periscope, and Snapchat
Haimson, O. L., & Tang, J. C. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025642 · Cited by 223
Cole-Lewis, H., Ezeanochie, N., & Turgiss, J. (2019) · JMIR Formative Research · doi:10.2196/14052 · Cited by 155
Szinay, D., Cameron, R., Naughton, F., Whitty, J. A., Brown, J., & Jones, A. (2021) · Journal of Medical Internet Research · doi:10.2196/32365 · Cited by 106
Clinically Meaningful Use of Mental Health Apps and its Effects on Depression: Mixed Methods Study
Zhang, R., Nicholas, J., Knapp, A. A., Graham, A. K., Gray, E., Kwasny, M. J., Reddy, M., & Mohr, D. C. (2019) · Journal of Medical Internet Research · doi:10.2196/15644 · Cited by 91
Paetzel, M., Perugia, G., & Castellano, G. (2020) · Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3319502.3374786 · Cited by 99
“71) [43], which was completed for the robot and the game.”
BackgroundDoes gamification engage users in online shopping?
García-Jurado, A., Torres-Jiménez, M., Leal-Rodríguez, A. L., & Castro-González, P. (2021) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2021.101076 · Cited by 81
Vaezipour, A., Campbell, J., Theodoros, D., & Russell, T. (2020) · JMIR mhealth and uhealth · doi:10.2196/18858 · Cited by 65
Wong, V., Smith, A. J., Hawkins, N. J., Kumar, R. K., Young, N., Kyaw, M., & Velan, G. M. (2015) · Academic Radiology · doi:10.1016/j.acra.2015.07.002 · Cited by 55
Applicability of the User Engagement Scale to Mobile Health: A Survey-Based Quantitative Study
(2019) · doi:10.2196/13244 · Cited by 54
Kocielnik, R., Langevin, R., George, J. S., Akenaga, S., Wang, A., Jones, D. P., Argyle, A., Fockele, C., Anderson, L., Hsieh, D. T., et al. (2021) · CUI 2021 - 3rd Conference on Conversational User Interfaces · doi:10.1145/3469595.3469599 · Cited by 37
“Participants evaluated both interfaces in terms of workload (NASA TLX [41]), engagement (adapted from O’Brian’s engagement [37], e.”
MethodBeccaluva, E. A., Catania, F., Arosio, F., & Garzotto, F. (2024) · Human–Computer Interaction · doi:10.1080/07370024.2023.2242837 · Cited by 24
Social presence and use of Internet-delivered interventions: a multi-method approach
Crutzen, R., Cyr, D., Larios, H., Ruiter, R. A. C., & de Vries, N. K. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057067 · Cited by 12
Consumer Engagement in the Digital Era: Its Nature, Drivers, and Outcomes
Weitzl, W., & Einwiller, S. (2018) · The Handbook of Communication Engagement · doi:10.1002/9781119167600.ch31 · Cited by 14
Towards Analyzing and Predicting the Experience of Live Performances with Wearable Sensing
Gedik, E., Cabrera-Quiros, L., Martella, C., Englebienne, G., & Hung, H. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2875987 · Cited by 10
“For measuring “enjoyment”, we adapted and selected questions presented in [51]; for “immersion”, we selected involvement questions from the Igroup Presence Questionnaire [52]; for “recommendation” we used items from O’Brien’s questionnaire [53].”
MethodLee, C., Lin, T., Pfister, H., & Zhu-Tian, C. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456332 · Cited by 7
Metadata
- Authors
- O'Brien, H. L., & Toms, E. G.
- Venue
- Journal of the American Society for Information Science and Technology
- Publisher
- Wiley
- Published
- 2009-12-09
- Volume
- 61
- Issue
- 1
- Pages
- 50-69
- ISSN
- 1532-2882 (ISSN-L)
- 1532-2890
- OpenAlex
- W4255441728
- Cited by
- 790
- References
- 73
- FWCI
- 8.95
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.