Measuring Game Engagement: Multiple Methods and Construct Complexity
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Background. Engagement has been identified as a crucial component of learning in games research. However, the conceptualization and operationalization of engagement vary widely in the literature. Many valuable approaches illuminate ways in which presence, flow, arousal, participation, and other concepts constitute or contribute to engagement. However, few studies examine multiple conceptualizations of engagement in the same project. Method. This article discusses the results of two experiments that measure engagement in five different ways: survey self-report, content analyses of player videos, electro-dermal activity, mouse movements, and game click logs. We examine the relationships among these measures and assess how they are affected by the technical characteristics of a 30-minute, custom-built, educational game: use of a customized character, level of narrative complexity, and level of art complexity. Results. We found that the five measures of engagement correlated in limited ways, and that they revealed substantially different relationships with game characteristics. We conclude that engagement as a construct is more complex than is captured in any of these measures individually and that using multiple methods to assess engagement can illuminate aspects of engagement not detectable by a single method of measurement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
School Engagement: Potential of the Concept, State of the Evidence
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004) · Review of Educational Research · doi:10.3102/00346543074001059 · Cited by 12,581 · cited 1×
“As a multi-dimensional or meta-construct (Appleton, 2008; Fredricks, Blumenfeld, & Paris, 2004), we include self-reports, physiological, and behavioral measures of engagement and, as described next, manipulate game characteristics to determine how measures interrelate.”
What Video Games Have to Teach Us about Learning and Literacy
(2004) · Education + Training · doi:10.1108/et.2004.00446dae.002 · Cited by 6,674
What video games have to teach us about learning and literacy
Gee, J. P. (2003) · Computers in Entertainment · doi:10.1145/950566.950595 · Cited by 5,735 · cited 2×
“Other research suggests that strong feelings of presence in games can also emerge when characters are a fixed element of the game setting, rather than customized by the player (Gee, 2003; Juul, 2005; Shaw, 2010).”
BackgroundThe role of transportation in the persuasiveness of public narratives.
Green, M. C., & Brock, T. C. (2000) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.79.5.701 · Cited by 4,861
Student engagement with school: Critical conceptual and methodological issues of the construct
Appleton, J. J., Christenson, S. L., & Furlong, M. J. (2008) · Psychology in the Schools · doi:10.1002/pits.20303 · Cited by 1,956 · cited 1×
“As a multi-dimensional or meta-construct (Appleton, 2008; Fredricks, Blumenfeld, & Paris, 2004), we include self-reports, physiological, and behavioral measures of engagement and, as described next, manipulate game characteristics to determine how measures interrelate.”
What is user engagement? A conceptual framework for defining user engagement with technology
O'Brien, H. L., & Toms, E. G. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20801 · Cited by 1,397
Measuring Narrative Engagement
Busselle, R., & Bilandzic, H. (2009) · Media Psychology · doi:10.1080/15213260903287259 · Cited by 1,088
Dinh, H. Q., Walker, N., Hodges, L. F., Song, C., & Kobayashi, A. (2003) · Proceedings IEEE Virtual Reality (Cat. No. 99CB36316) · doi:10.1109/vr.1999.756955 · Cited by 566 · cited 2×
“Other research suggests that in some cases, level of visual abstraction has no effect on a sense of presence (Dinh, Walker, Song, Kobayashi, & Hodges, 1999; Zimmons & Panter, 2003).”
Background
Cited by
Perski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,193
Short, C. E., DeSmet, A., Woods, C., Williams, S. L., Maher, C., Middelweerd, A., Müller, A. M., Wark, P. A., Vandelanotte, C., Poppe, L., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9397 · Cited by 400
“engagement can illuminate aspects of engagement not detectable by a single method of measurement” [124].”
BackgroundMethodResultInfluentialAdaptive scaffolding and engagement in digital game-based learning
Chen, C.-H., Law, V., & Huang, K. (2023) · Educational technology research and development · doi:10.1007/s11423-023-10244-x · Cited by 43
Melcer, E. F., Grasse, K. M., Ryan, J., Junius, N., Kreminski, M., Squinkifer, D., Hill, B., & Wardrip-Fruin, N. (2020) · International Conference on the Foundations of Digital Games · doi:10.1145/3402942.3403005 · Cited by 24
“The TPI is an instrument that has been validated for use with games [28] and measuring game engagement [30].”
MethodGrasse, K. M., Melcer, E. F., Kreminski, M., Junius, N., Ryan, J., & Wardrip-Fruin, N. (2022) · International Series on Computer Entertainment and Media Technology · doi:10.1007/978-3-030-81538-7_12 · Cited by 11
“The TPI has been validated for use with games [Lombard et al., 2011] and measuring game engage- ment [Martey et al., 2014].”
Method
Metadata
- Authors
- Martey, R. M., Kenski, K., Folkestad, J., Feldman, L., Gordis, E., Shaw, A., Stromer-Galley, J., Clegg, B., Zhang, H., Kaufman, N., Rabkin, A. N., Shaikh, S., & Strzalkowski, T.
- Venue
- Simulation & Gaming
- Publisher
- SAGE Publications
- Published
- 2014-08
- Volume
- 45
- Issue
- 4-5
- Pages
- 528-547
- ISSN
- 1046-8781 (ISSN-L)
- 1552-826X
- OpenAlex
- W2112422747
- Cited by
- 57
- References
- 8
- FWCI
- 6.89
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.