Using touchscreen interaction data to predict cognitive workload
The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Although a great number of today’s learning applications run on devices with an interactive screen, the high-resolution interaction data which these devices provide have not been used for workload-adaptive systems yet. This paper aims at exploring the potential of using touch sensor data to predict user states in learning scenarios. For this purpose, we collected touch interaction patterns of children solving math tasks on a multi-touch device. 30 fourth-grade students from a primary school participated in the study. Based on these data, we investigate how machine learning methods can be applied to predict cognitive workload associated with tasks of varying difficulty. Our results show that interaction patterns from a touchscreen can be used to significantly improve automatic prediction of high levels of cognitive workload (average classification accuracy of 90.67% between the easiest and most difficult tasks). Analyzing an extensive set of features, we discuss which characteristics are most likely to be of high value for future implementations. We furthermore elaborate on design choices made for the used multiple choice interface and discuss critical factors that should be considered for future touch-based learning interfaces.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Building Intelligent Interactive Tutors : student-centered strategies for revolutionizing e-learning
Woolf, B. P. (2009) · HAL · Cited by 1,374
Combining SVMs with Various Feature Selection Strategies
Chen, Y.-W., & Lin, C.-J. (2008) · Studies in Fuzziness and Soft Computing · doi:10.1007/978-3-540-35488-8_13 · Cited by 980 · cited 1×
“For feature selection and generation of the final individual classifiers, we opted for an approach suggested by Chen et al. [3].”
MethodCognitive arithmetic: Evidence for retrieval and decision processes in mental addition.
Ashcraft, M. H., & Battaglia, J. (1978) · Journal of Experimental Psychology: Human Learning & Memory · doi:10.1037//0278-7393.4.5.527 · Cited by 350 · cited 1×
“Both, the so called problem size as well as the carry effect have been identified to robustly increase response latencies and/or error rates when solving multi-digit addition problems (e.g., [1, 10, 11]).”
BackgroundD’Mello, S. K., & Graesser, A. (2010) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-010-9074-4 · Cited by 323
The neural substrate of arithmetic operations and procedure complexity
Kong, J., Wang, C., Kwong, K., Vangel, M., Chua, E., & Gollub, R. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2004.09.011 · Cited by 215 · cited 1×
“The carry effect has furthermore been repeatedly identified to be the source of increased cognitive workload (e.g, [8, 9]).”
BackgroundWhat Does Touch Tell Us about Emotions in Touchscreen-Based Gameplay?
Gao, Y., Bianchi-Berthouze, N., & Meng, H. (2012) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2395131.2395138 · Cited by 170 · cited 3×
“CCS Concepts •Human-centered computing → User models; Touch screens; Empirical studies in HCI; Keywords Multi-Touch, Machine Learning, Adaptive Learning Systems”
BackgroundMethodInfluentialGerjets, P. H., Walter, C., Rosenstiel, W., Bogdan, M., & Zander, T. O. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00385 · Cited by 160 · cited 2×
“We present the results of a user study with 30 fourthgraders from a primary school and discuss how our findings might influence the design of future adaptive systems that utilize multi-touch interaction data.”
BackgroundSelf-citationToward Measuring and Maintaining the Zone of Proximal Development in Adaptive Instructional Systems
Murray, T., & Arroyo, I. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47987-2_75 · Cited by 122 · cited 1×
“In order to maintain the so called zone of proximal development, cognitive and affective factors have to be taken into account [16].”
BackgroundImbo, I., Vandierendonck, A., & De Rammelaere, S. (2007) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470210600762447 · Cited by 107
AdELE: A Framework for Adaptive E-Learning through Eye Tracking
García-Barrios, V. M., Gütl, C., Preis, A., Andrews, K., Pivec, M., Mödritscher, F., & Trummer, C. (2004) · Cited by 91
Saneiro, M., Santos, O. C., Salmeron-Majadas, S., & Boticario, J. G. (2014) · The Scientific World Journal · doi:10.1155/2014/484873 · Cited by 91 · cited 1×
“Audio and video recordings have also been analyzed to assess affective and/or cognitive states (e.g., [4, 12, 19, 23]).”
BackgroundAffective State Level Recognition in Naturalistic Facial and Vocal Expressions
Meng, H., & Bianchi‐Berthouze, N. (2013) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2013.2253768 · Cited by 80 · cited 1×
“The idea is to find a form of representation and difficulty level that neither over- or underwhelms the learner [21], such that he or she remains attentive and is thus able to process and memorize the presented information.”
BackgroundKlein, E., Moeller, K., Dressel, K., Domahs, F., Wood, G. M. D. O., Willmes, K., & Nuerk, H. (2010) · Acta Psychologica · doi:10.1016/j.actpsy.2010.06.002 · Cited by 65 · cited 1×
“The question always is ”How difficult did you find the last exercises?” with the answers being ”easy”, ”relatively easy”, ”relatively hard” and ”hard”.”
Salmerón-Majadas, S., Santos, O. C., & Boticario, J. G. (2014) · Procedia Computer Science · doi:10.1016/j.procs.2014.08.151 · Cited by 61
Working memory demands of exact and approximate addition
Kalaman, D. A., & Lefevre, J.-A. (2007) · European Journal of Cognitive Psychology · doi:10.1080/09541440600713445 · Cited by 53 · cited 1×
“Moeller et al. used eye tracking data to show that the increased load can be scribed both to additional unit-based calculations on the summands to evaluate whether a carry-over is needed or not and additional processing of the decade digit of the sum [15].”
BackgroundTowards Automatic Cognitive Load Measurement from Speech Analysis
Yin, B., & Chen, F. (2007) · Springer eBooks · doi:10.1007/978-3-540-73105-4_111 · Cited by 51 · cited 1×
“Audio and video recordings have also been analyzed to assess affective and/or cognitive states (e.g., [4, 12, 19, 23]).”
BackgroundThree processes underlying the carry effect in addition – Evidence from eye tracking
Moeller, K., Klein, E., & Nuerk, H. (2011) · British Journal of Psychology · doi:10.1111/j.2044-8295.2011.02034.x · Cited by 49 · cited 1×
“used eye tracking data to show that the increased load can be scribed both to additional unit-based calculations on the summands to evaluate whether a carry-over is needed or not and additional processing of the decade digit of the sum [15].”
MethodHuman-Computer Interaction. Interaction Design and Usability
Domingo, M. G., Almirall, M., & Pera, E. M. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73105-4 · Cited by 41
Intelligent Tutoring Systems 92: Montreal, Canada, 10–12 June
Soldato, T. D. (1992) · AI Communications · doi:10.3233/aic-1992-5411 · Cited by 33
User identification using raw sensor data from typing on interactive displays
Mock, P., Edelmann, J., Schilling, A., & Rosenstiel, W. (2014) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/2557500.2557503 · Cited by 14 · cited 3×
“[13] applied raw optical sensor data as the main set of features to identify users in typing scenarios.”
BackgroundMethodInfluentialSelf-citationEdelmann, J., Mock, P., Schilling, A., Gerjets, P. H., Rosenstiel, W., & Straßer, W. (2012) · ACM International Conference on Interactive Tabletops and Surfaces (ITS) · doi:10.1145/2396636.2396662 · Cited by 13
Self-citationSensor Technology for Learning Support
Specht, M. M. (2014) · Cited by 7
Mock, P., Jaszkowic, J., Edelmann, J., Kammerer, Y., Schilling, A., & Rosenstiel, W. (2015) · Australian Conference on Computer-Human Interaction (OzCHI) · doi:10.1145/2838739.2838744 · Cited by 2 · cited 1×
“For classification, we chose SVM classifiers due to convincing prior results with similar touch sensor data [14].”
MethodSelf-citation
Research works printed but not linked: 3
- 10.1037/0278-7393.4.5.527 · doi:10.1037/0278-7393.4.5.527 · Paper page
- Samsung. SUR40. http: //www.samsung.com/uk/b/p/LH40SFWTGC/EN , 2012 . Accessed : 2016-09-08. Samsung. SUR40. http: //www.samsung.com/uk/b/p/LH40SFWTGC/EN, 2012. Accessed: 2016-09-08.
- L. Vgotsky . Mind in society: The development of higher mental processes , 1978 . L. Vgotsky. Mind in society: The development of higher mental processes, 1978.
Cited by
A Survey on Measuring Cognitive Workload in Human-Computer Interaction
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W. (2023) · ACM Computing Surveys · doi:10.1145/3582272 · Cited by 304
“Similarly, touch input [39] – especially finger trajectories [173] – were used to measure cognitive workload.”
Multimodal data capabilities for learning: What can multimodal data tell us about learning?
Sharma, K., & Giannakos, M. (2020) · British Journal of Educational Technology · doi:10.1111/bjet.12993 · Cited by 237
“( 2018 ) used skeleton positions and kinematics features to predict participants’ recall; Andrade, Danish, and ( 2016 ) used interaction logs from a touchscreen and the hand movements to predict the cognitive workload of the students.”
BackgroundMethodInfluentialNam, H., Seol, K., Lee, J., Cho, H., & Jung, S. W. (2021) · Sensors · doi:10.3390/s21144776 · Cited by 82
Cognitive load and self-regulation: Attempts to build a bridge
Boekaerts, M. (2017) · Learning and Instruction · doi:10.1016/j.learninstruc.2017.07.001 · Cited by 73
“Gerjets and colleagues collected interaction patterns of primary school children while they solved math problems on a multi-touch table and then used machine learning methods on these high-resolution interactions (see Mock et al., 2016).”
BackgroundTian, F., Fan, X., Fan, J., Zhu, Y., Gao, J., Wang, D., Bi, X., & Wang, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300313 · Cited by 38
“The mean directional change in degrees of each point on the actual path [38].”
Background
Metadata
- Authors
- Mock, P., Gerjets, P., Tibus, M., Trautwein, U., Möller, K., & Rosenstiel, W.
- Affiliations
- Leibniz-Institut für Wissensmedien (DE)
- University of Tübingen (DE)
- Venue
- Proceedings of the 18th ACM International Conference on Multimodal Interaction
- Publisher
- ACM
- Published
- 2016-10-31
- Pages
- 349-356
- OpenAlex
- W2547090472
- Cited by
- 21
- References
- 23
- Citations by year
- 2025: 2 · 2023: 4 · 2022: 3 · 2021: 3 · 2020: 3 · 2019: 3 · 2018: 1 · 2017: 2
- FWCI
- 1.08
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Keywords
- multi-touch devices, task difficulty, fourth grade, interaction patterns
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.