The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
The development and evaluation of a survey to measure user engagement
Abstract
Facilitating engaging user experiences is essential in the design of interactive systems. To accomplish this, it is necessary to understand the composition of this construct and how to evaluate it. Building on previous work that posited a theory of engagement and identified a core set of attributes that operationalized this construct, we constructed and evaluated a multidimensional scale to measure user engagement. In this paper we describe the development of the scale, as well as two large‐scale studies (N=440 and N=802) that were undertaken to assess its reliability and validity in online shopping environments. In the first we used Reliability Analysis and Exploratory Factor Analysis to identify six attributes of engagement: Perceived Usability, Aesthetics, Focused Attention, Felt Involvement, Novelty, and Endurability. In the second we tested the validity of and relationships among those attributes using Structural Equation Modeling. The result of this research is a multidimensional scale that may be used to test the engagement of software applications. In addition, findings indicate that attributes of engagement are highly intertwined, a complex interplay of user‐system interaction variables. Notably, Perceived Usability played a mediating role in the relationship between Endurability and Novelty, Aesthetics, Felt Involvement, and Focused Attention.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 80,031
Development and validation of brief measures of positive and negative affect: The PANAS scales.
Watson, D., Clark, L. A., & Tellegen, A. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.6.1063 · Cited by 32,433
User Acceptance of Computer Technology: A Comparison of Two Theoretical Models
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989) · Management Science · doi:10.1287/mnsc.35.8.982 · Cited by 25,709
(1991) · Cited by 15,554
Information Systems Success: The Quest for the Dependent Variable
DeLone, W. H., & McLean, E. R. (1992) · Information Systems Research · doi:10.1287/isre.3.1.60 · Cited by 10,697
What is coefficient alpha? An examination of theory and applications
Cortina, J. M. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.1.98 · Cited by 8,810
Measuring Presence in Virtual Environments: A Presence Questionnaire
Witmer, B. G., & Singer, M. J. (1998) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474698565686 · Cited by 6,371
Work and/or Fun: Measuring Hedonic and Utilitarian Shopping Value
Babin, B. J., Darden, W. R., & Griffin, M. (1994) · Journal of Consumer Research · doi:10.1086/209376 · Cited by 6,020
Usability Evaluation In Industry
(1996) · CRC Press eBooks · doi:10.1201/9781498710411 · Cited by 4,354
Patterns of cognitive appraisal in emotion.
Smith, C. A., & Ellsworth, P. C. (1985) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.48.4.813 · Cited by 4,045
Hedonic and utilitarian motivations for online retail shopping behavior
Childers, T. L., Carr, C. L., Peck, J., & Carson, S. (2001) · Journal of Retailing · doi:10.1016/s0022-4359(01)00056-2 · Cited by 3,219
Measuring the Customer Experience in Online Environments: A Structural Modeling Approach
Novak, T. P., Hoffman, D. L., & Yung, Y.-F. (2000) · Marketing Science · doi:10.1287/mksc.19.1.22.15184 · Cited by 3,189
Perceived Usefulness, Ease of Use, and Usage of Information Technology: A Replication
Adams, D. A., Nelson, R. R., & Todd, P. A. (1992) · MIS Quarterly · doi:10.2307/249577 · Cited by 3,084
Using LISREL for Structural Equation Modeling: A Researcher′s Guide
Kelloway, E. K. (1998) · Cited by 2,792
A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys
Cook, C., Heath, F., & Thompson, R. L. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970934 · Cited by 2,396
Arnold, M. J., & Reynolds, K. E. (2003) · Journal of Retailing · doi:10.1016/s0022-4359(03)00007-1 · Cited by 2,373
Mathwick, C., Malhotra, N., & Rigdon, E. (2001) · Journal of Retailing · doi:10.1016/s0022-4359(00)00045-2 · Cited by 2,158
User Centered System Design: New Perspectives on Human-Computer Interaction
Anderson, N. S., Norman, D. A., & Draper, S. W. (1988) · The American Journal of Psychology · doi:10.2307/1422802 · Cited by 1,840
The Relative Importance of Perceived Ease of Use in IS Adoption: A Study of E-Commerce Adoption
Drexel University, Gefen, D., Straub, D., & Georgia State University (2000) · Journal of the Association for Information Systems · doi:10.17705/1jais.00008 · Cited by 1,833
The Multivariate Social Scientist
Hutcheson, G. (1999) · SAGE Publications Ltd eBooks · doi:10.4135/9780857028075 · Cited by 1,728
Guay, F., Vallerand, R. J., & Blanchard, C. (2000) · Motivation and Emotion · doi:10.1023/a:1005614228250 · Cited by 1,579
What is user engagement? A conceptual framework for defining user engagement with technology
O'Brien, H. L., & Toms, E. G. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20801 · Cited by 1,397
Self-citationTractinsky, N., Katz, A. S., & Ikar, D. (2000) · Interacting with Computers · doi:10.1016/s0953-5438(00)00031-x · Cited by 1,376
Assessing dimensions of perceived visual aesthetics of web sites
Lavie, T., & Tractinsky, N. (2004) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2003.09.002 · Cited by 1,335
Student engagement in high school classrooms from the perspective of flow theory
Shernoff, D. J., Csikszentmihalyi, M., Shneider, B., & Shernoff, E. S. (2003) · School Psychology Quarterly · doi:10.1521/scpq.18.2.158.21860 · Cited by 1,237
The dimensionality and correlates of flow in human-computer interactions
Webster, J., Trevino, L. K., & Ryan, L. (1993) · Computers in Human Behavior · doi:10.1016/0747-5632(93)90032-n · Cited by 1,102
Attention web designers: You have 50 milliseconds to make a good first impression!
Lindgaard, G., Fernandes, G., Dudek, C., & Brown, J. (2006) · Behaviour & Information Technology · doi:10.1080/01449290500330448 · Cited by 1,025
Rieber, L. P. (1996) · Educational Technology Research and Development · doi:10.1007/bf02300540 · Cited by 999
Understanding experience in interactive systems
Forlizzi, J., & Battarbee, K. (2004) · Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques · doi:10.1145/1013115.1013152 · Cited by 925
Choi, D., & Kim, J. (2004) · CyberPsychology & Behavior · doi:10.1089/109493104322820066 · Cited by 683
Measuring Epistemic Curiosity and Its Diversive and Specific Components
Litman, J. A., & Spielberger, C. D. (2003) · Journal of Personality Assessment · doi:10.1207/s15327752jpa8001_16 · Cited by 677
Developing a scale to measure the interactivity of websites
Liu, Y. (2003) · Journal of Advertising Research · doi:10.2501/jar-43-2-207-216 · Cited by 659
Play, Flow, and the Online Search Experience
Mathwick, C., & Rigdon, E. (2004) · Journal of Consumer Research · doi:10.1086/422111 · Cited by 614
Extrinsic versus intrinsic motivations for consumers to shop on-line
Shang, R.-A., Chen, Y.-C., & Shen, L. (2005) · Information & Management · doi:10.1016/j.im.2004.01.009 · Cited by 518
Constructing Effective Questionnaires
Peterson, R. (2000) · Sage eBooks · doi:10.4135/9781483349022 · Cited by 499
(2001) · Choice Reviews Online · doi:10.5860/choice.38-5345 · Cited by 490
Satisfiers and dissatisfiers: A two-factor model for website design and evaluation
Zhang, P., & von Dran, G. M. (2000) · Journal of the American Society for Information Science · doi:10.1002/1097-4571(2000)9999:9999<::aid-asi1039>3.0.co;2-o · Cited by 488
Aesthetics and apparent usability: empirically assessing cultural and methodological issues
Tractinsky, N. (1997) · Proceedings of the ACM SIGCHI Conference on Human factors in computing systems · doi:10.1145/258549.258626 · Cited by 482
Aesthetics and preferences of web pages
Schenkman, B. N., & Jönsson, F. U. (2000) · Behaviour & Information Technology · doi:10.1080/014492900750000063 · Cited by 475
Designing website attributes to induce experiential encounters
Huang, M.-H. (2003) · Computers in Human Behavior · doi:10.1016/s0747-5632(02)00080-8 · Cited by 468
A grounded theory of the flow experiences of Web users
Pace, S. (2004) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2003.08.005 · Cited by 399
Audience engagement in multimedia presentations
Webster, J., & Ho, H. (1997) · ACM SIGMIS Database: the DATABASE for Advances in Information Systems · doi:10.1145/264701.264706 · Cited by 380
On the Meaning of Leisure: An Investigation of Some Determinants of the Subjective Experience
Unger, L. S., & Kernan, J. B. (1983) · Journal of Consumer Research · doi:10.1086/208932 · Cited by 350
Common Method Variance and Specification Errors: A Practical Approach to Detection
Kline, T. J. B., Sulsky, L. M., & Rever-Moriyama, S. D. (2000) · The Journal of Psychology · doi:10.1080/00223980009598225 · Cited by 280
The experience of flow in computer-mediated and in face-to-face groups
Ghani, J. A., Supnick, R. M., & Rooney, P. S. (1991) · Journal of the Association for Information Systems · Cited by 268
Webster, J., & Ahuja, J. S. (2006) · MIS Quarterly · doi:10.2307/25148744 · Cited by 249
Measuring one component of dispositional self-regulation: attention control in goal pursuit
Luszczynska, A., Diehl, M., Gutiérrez-Doña, B., Kuusinen, P., & Schwarzer, R. (2004) · Personality and Individual Differences · doi:10.1016/j.paid.2003.09.026 · Cited by 177
The pleasure principle: immersion, engagement, flow
Douglas, Y., & Hargadon, A. (2000) · Proceedings of the eleventh ACM on Hypertext and hypermedia · doi:10.1145/336296.336354 · Cited by 175
The Aesthetic Point of View: Selected Essays
Beardsley, M. C., Wreen, M. J., & Callen, D. M. (1983) · Poetics Today · doi:10.2307/1772303 · Cited by 163
The role of choice in reader engagement.
Schraw, G. J., Flowerday, T., & Reisetter, M. (1998) · Journal of Educational Psychology · doi:10.1037/0022-0663.90.4.705 · Cited by 133
Theory and models for creating engaging and immersive ecommerce Websites
Jennings, M. M. (2000) · ACM SIGCPR Conference on Computer Personnel Research (SIGCPR) · doi:10.1145/333334.333358 · Cited by 122
Engagement as a Design Concept for Multimedia
Jacques, R. D., Preece, J. J., & Carey, T. (1995) · Canadian Journal of Learning and Technology · doi:10.21432/t2vg77 · Cited by 117
Engagement in multimedia training systems
Chapman, P., Selvarajah, S., & Webster, J. (2003) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.1999.772808 · Cited by 110
Overbeeke, K., Djajadiningrat, T., Hummels, C., Wensveen, S. S. A. G., & Prens, J. (2003) · Human-computer interaction series · doi:10.1007/1-4020-2967-5_2 · Cited by 109
Pirhonen, A., Saariluoma, P. O., Roast, C., & Isomäki, H. (2005) · Springer eBooks · doi:10.1007/b138650 · Cited by 93
Park, S. E., Choi, D., & Kim, J. (2004) · Interacting with Computers · doi:10.1016/j.intcom.2003.07.001 · Cited by 92
Measuring user involvement: a diffusion of innovation perspective
Kappelman, L. A. (1995) · ACM SIGMIS Database the DATABASE for Advances in Information Systems · doi:10.1145/217278.217286 · Cited by 89
Critical review of state-trait curiosity test development
Boyle, G. J. (1983) · Motivation and Emotion · doi:10.1007/bf00991647 · Cited by 81
The value of the novel in designing for experience
Wright, P., & McCarthy, J. (2005) · Springer eBooks · doi:10.1007/1-84628-089-3_2 · Cited by 75
An engaging multimedia design model
Said, N. S. (2004) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/1017833.1017873 · Cited by 72
The Experience of Flow in Interacting With a Hypermedia Learning Environment
Konradt, U., & Sulz, K. (2001) · Journal of educational multimedia and hypermedia · Cited by 64
Feedback and driver situation awareness (SA): A comparison of SA measures and contexts
Walker, G. H., Stanton, N. A., & Young, M. S. (2008) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2008.01.003 · Cited by 59
Understanding affect in design: an outline conceptual framework
Aboulafia, A., & Bannon, L. J. (2004) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922031000086708 · Cited by 50
Information and Emotion: The Emergent Affective Paradigm in Information Behavior Research and Theory
(2008) · Library Management · doi:10.1108/01435120810917585 · Cited by 41
EFFECTS OF CURIOSITY ON SOCIALIZATION-RELATED LEARNING AND JOB PERFORMANCE IN ADULTS
Reio, T. G. (1997) · VTechWorks (Virginia Tech) · Cited by 39
Designing Engaging Experiences with Children and Artists
Hull, R., & Reid, J. (2003) · Human-computer interaction series · doi:10.1007/1-4020-2967-5_17 · Cited by 17
Transcendent experience in the use of computer-based media
Laarni, J., Ravaja, N., Kallinen, K., & Saari, T. (2004) · Nordic Conference on Human-Computer Interaction · doi:10.1145/1028014.1028082 · Cited by 16
Skelly, T., Fries, K., Linnett, B., Nass, C., & Reeves, B. B. (1994) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/259963.260452 · Cited by 16
Objectively evaluating entertainment technology
Mandryk, R. L. (2004) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/985921.985977 · Cited by 15
Adventurers versus nit-pickers on affective computing
Lindgaard, G. (2004) · Interacting with Computers · doi:10.1016/j.intcom.2004.06.006 · Cited by 15
Does collaborative hypertext support better engagement in learning of the basics in informatics?
Makkonen, P. (1997) · ACM SIGCSE Bulletin · doi:10.1145/268809.268857 · Cited by 7
Context and leadership in the remote environment
Kelley, E. (2005) · Saint Mary's University Institutional Repository (Saint Mary's University) · Cited by 2
(2005)
Research works printed but not linked: 19
- Bojko A., 2006, Using eye tracking to compate web page designs: A case study, Journal of Usability Studies, 3, 112
- 10.1145/604471.604532 · doi:10.1145/604471.604532
- Chapman P.(1997).Models of Engagement: Intrinsically Motivated Interaction with Multimedia Learning Software. Unpublished master's thesis University of Waterloo Waterloo Ontario Canada.
- Clip Culture: Internet video, 2006, The Economist, 379, 73
- Feedback. (1995). The Penguin dictionary of psychology. Retrieved 29 April 2005 from xreferplus.http://www.xreferplus.com/entry/150004FinneranZhang
- Haywood N., 2005, People and Computers XIX — The Bigger Picture, Proceedings of HCI 2005
- Hebb D.O., 1972, Textbook of psychology
- Jacques R.D.(1996).The nature of engagement and its role in hypermedia evaluation and design. Unpublished doctoral dissertation South Bank University London.
- Laurel B., 1993, Computers as Theatre
- Madden M., 2007, Online Video
- Matlin M.W., 1994, Cognition
- Nunnally J.C., 1978, Pyschometric theory
- Pine B.J., 1999, The experience economy
- Quesenbery W., 2003, Content and complexity: Information design in technical communications, 81
- Schmidt K.E. Bauerly M. Liu Y. &Sridharan S.(2003).Web page aesthetics and performance: A survey and an experimental study. Paper presented at the International Conference on Industrial Engineering Las Vegas NV.
- Seah M.‐L. &Cairns P.(2008). From immersion to addiction in video games. In England D. & Beale R. (Eds.) Proceedings of Human Computer Interaction Conference Vol. 1 (HCI 2008) (pp. 55–63). Bedfordshire UK: British Computer Society. Retrieved October 8 2009 fromhttp://www.users.cs.york.ac.uk/∼pcairns/papers/Seah_Cairns_HCI2008.pdf
- Singleton R.A., 1993, Approaches to social science research
- 10.1037/0022-3514.48.4.813 · doi:10.1037/0022-3514.48.4.813 · Paper page
- Vick R.M. &Ikehara C.S.(2002). Methodological issues of real time data acquisition from multiple sources of physiological data. In Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS‐ 36). Los Alamitos CA: IEEE. Retrieved September 17 2009 fromhttp://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1174295&isnumber=26341 · doi:10.1109/hicss.2003.1174295
Cited by
Perski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,193
O’Brien, H. L., Cairns, P., & Hall, M. (2018) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2018.01.004 · Cited by 1,014
Self-citationKim, Y. H., Kim, D. J., & Wachter, K. (2013) · Decision Support Systems · doi:10.1016/j.dss.2013.07.002 · Cited by 769
The Faces of Engagement: Automatic Recognition of Student Engagementfrom Facial Expressions
Whitehill, J., Serpell, Z., Lin, Y.-C., Foster, A., & Movellan, J. R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2316163 · Cited by 677
Toward Meaningful Engagement: A Framework for Design and Research of Gamified Information Systems1
Liu, D., Santhanam, R., & Webster, J. (2017) · MIS Quarterly · doi:10.25300/misq/2017/41.4.01 · Cited by 521
Metadata
- Authors
- O'Brien, H. L., & Toms, E. G.
- Venue
- Journal of the American Society for Information Science and Technology
- Publisher
- Wiley
- Published
- 2009-12-09
- Volume
- 61
- Issue
- 1
- Pages
- 50-69
- ISSN
- 1532-2882 (ISSN-L)
- 1532-2890
- OpenAlex
- W4255441728
- Cited by
- 790
- References
- 73
- FWCI
- 8.95
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.