Using touchscreen interaction data to predict cognitive workload
The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Although a great number of today’s learning applications run on devices with an interactive screen, the high-resolution interaction data which these devices provide have not been used for workload-adaptive systems yet. This paper aims at exploring the potential of using touch sensor data to predict user states in learning scenarios. For this purpose, we collected touch interaction patterns of children solving math tasks on a multi-touch device. 30 fourth-grade students from a primary school participated in the study. Based on these data, we investigate how machine learning methods can be applied to predict cognitive workload associated with tasks of varying difficulty. Our results show that interaction patterns from a touchscreen can be used to significantly improve automatic prediction of high levels of cognitive workload (average classification accuracy of 90.67% between the easiest and most difficult tasks). Analyzing an extensive set of features, we discuss which characteristics are most likely to be of high value for future implementations. We furthermore elaborate on design choices made for the used multiple choice interface and discuss critical factors that should be considered for future touch-based learning interfaces.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Building Intelligent Interactive Tutors : student-centered strategies for revolutionizing e-learning
Woolf, B. P. (2009) · HAL · Cited by 1,405
Combining SVMs with Various Feature Selection Strategies
Chen, Y.-W., & Lin, C.-J. (2008) · Studies in Fuzziness and Soft Computing · doi:10.1007/978-3-540-35488-8_13 · Cited by 980 · cited 1×
“For feature selection and generation of the final individual classifiers, we opted for an approach suggested by Chen et al. [3].”
MethodCognitive arithmetic: Evidence for retrieval and decision processes in mental addition.
Ashcraft, M. H., & Battaglia, J. (1978) · Journal of Experimental Psychology: Human Learning & Memory · doi:10.1037//0278-7393.4.5.527 · Cited by 350 · cited 1×
“Both, the so called problem size as well as the carry effect have been identified to robustly increase response latencies and/or error rates when solving multi-digit addition problems (e.g., [1, 10, 11]).”
BackgroundD’Mello, S. K., & Graesser, A. (2010) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-010-9074-4 · Cited by 323
The neural substrate of arithmetic operations and procedure complexity
Kong, J., Wang, C., Kwong, K., Vangel, M., Chua, E., & Gollub, R. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2004.09.011 · Cited by 215 · cited 1×
“The carry effect has furthermore been repeatedly identified to be the source of increased cognitive workload (e.g, [8, 9]).”
BackgroundWhat Does Touch Tell Us about Emotions in Touchscreen-Based Gameplay?
Gao, Y., Bianchi-Berthouze, N., & Meng, H. (2012) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2395131.2395138 · Cited by 170 · cited 3×
“CCS Concepts •Human-centered computing → User models; Touch screens; Empirical studies in HCI; Keywords Multi-Touch, Machine Learning, Adaptive Learning Systems”
BackgroundMethodInfluentialGerjets, P. H., Walter, C., Rosenstiel, W., Bogdan, M., & Zander, T. O. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00385 · Cited by 163 · cited 2×
“We present the results of a user study with 30 fourthgraders from a primary school and discuss how our findings might influence the design of future adaptive systems that utilize multi-touch interaction data.”
BackgroundSelf-citationToward Measuring and Maintaining the Zone of Proximal Development in Adaptive Instructional Systems
Murray, T., & Arroyo, I. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47987-2_75 · Cited by 122 · cited 1×
“In order to maintain the so called zone of proximal development, cognitive and affective factors have to be taken into account [16].”
BackgroundImbo, I., Vandierendonck, A., & De Rammelaere, S. (2007) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470210600762447 · Cited by 107
AdELE: A Framework for Adaptive E-Learning through Eye Tracking
García-Barrios, V. M., Gütl, C., Preis, A., Andrews, K., Pivec, M., Mödritscher, F., & Trummer, C. (2004) · Cited by 91
Research works printed but not linked: 3
- 10.1037/0278-7393.4.5.527 · doi:10.1037/0278-7393.4.5.527 · Paper page
- Samsung. SUR40. http: //www.samsung.com/uk/b/p/LH40SFWTGC/EN , 2012 . Accessed : 2016-09-08. Samsung. SUR40. http: //www.samsung.com/uk/b/p/LH40SFWTGC/EN, 2012. Accessed: 2016-09-08.
- L. Vgotsky . Mind in society: The development of higher mental processes , 1978 . L. Vgotsky. Mind in society: The development of higher mental processes, 1978.
Cited by
A Survey on Measuring Cognitive Workload in Human-Computer Interaction
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W. (2023) · ACM Computing Surveys · doi:10.1145/3582272 · Cited by 304
“Similarly, touch input [39] – especially finger trajectories [173] – were used to measure cognitive workload.”
Multimodal data capabilities for learning: What can multimodal data tell us about learning?
Sharma, K., & Giannakos, M. (2020) · British Journal of Educational Technology · doi:10.1111/bjet.12993 · Cited by 237
“( 2018 ) used skeleton positions and kinematics features to predict participants’ recall; Andrade, Danish, and ( 2016 ) used interaction logs from a touchscreen and the hand movements to predict the cognitive workload of the students.”
BackgroundMethodInfluentialNam, H., Seol, K., Lee, J., Cho, H., & Jung, S. W. (2021) · Sensors · doi:10.3390/s21144776 · Cited by 82
Cognitive load and self-regulation: Attempts to build a bridge
Boekaerts, M. (2017) · Learning and Instruction · doi:10.1016/j.learninstruc.2017.07.001 · Cited by 74
“Gerjets and colleagues collected interaction patterns of primary school children while they solved math problems on a multi-touch table and then used machine learning methods on these high-resolution interactions (see Mock et al., 2016).”
BackgroundTian, F., Fan, X., Fan, J., Zhu, Y., Gao, J., Wang, D., Bi, X., & Wang, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300313 · Cited by 38
“The mean directional change in degrees of each point on the actual path [38].”
BackgroundInvestigating Multi-Modal Measures for Cognitive Load Detection in E-Learning
Herbig, N., Düwel, T., Helali, M., Eckhart, L., Schuck, P., Choudhury, S., & Krüger, A. (2020) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3340631.3394861 · Cited by 40
“Instead of EEG data, Mock et al. [36] use touchscreen interactions to classify CL for children solving math problems.”
Assessing workload in using electromyography (EMG)-based prostheses
Park, J., Berman, J., Dodson, A., Liu, Y., Armstrong, M., Huang, H., KABER, D. B., Ruiz, J., & Zahabi, M. (2023) · Ergonomics · doi:10.1080/00140139.2023.2221413 · Cited by 23
“…method (Skaramagkas et al. 2021), recursive feature elimination (RFE) (Raufi 2019b), k-fold cross validation (CV) (Momeni et al. 2019), grid search (Mock et al. 2016) and random search (Skaramagkas et al. 2021), previous studies did not clearly state their feature selection meth-ods (Hillege et…”
MethodLi, Q., Luximon, Y., Zhang, J., & Song, Y. (2023) · British Journal of Educational Technology · doi:10.1111/bjet.13394 · Cited by 26
Motion-Based Educational Games: Using Multi-Modal Data to Predict Player’s Performance
Lee‐Cultura, S., Sharma, K., Papavlasopoulou, S., & Giannakos, M. N. (2020) · IEEE Conference on Games (CoG) · doi:10.1109/cog47356.2020.9231892 · Cited by 20
“ment systems [40], to adaptive systems [32], to collaborative systems [24].”
BackgroundSchwan, S., Gussmann, M., Gerjets, P. H., Drecoll, A., & Feiber, A. A. (2019) · Museum Management and Curatorship · doi:10.1080/09647775.2019.1666422 · Cited by 16
“, processing in working memory) can additionally be decomposed by using interaction protocols for hands-on or digital exhibits, again either by human observers (Van Schijndel, Franse, and Raijmakers 2010) or by logfiles recording digital usage patterns (Mock et al. 2016).”
BackgroundMethodSelf-citationPredicting ADHD Risk from Touch Interaction Data
Mock, P., Tibus, M., Ehlis, A., Baayen, R. H., & Gerjets, P. H. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3242986 · Cited by 12
“In sum, the above results confirm that our previous results for classification of task difficulties [20] hold for a larger population of students.”
MethodResultInfluentialSelf-citationGerjets, P. H. (2017) · Springer eBooks · doi:10.1007/978-3-319-49077-9_4 · Cited by 11
Self-citationExploiting Wearable Technologies to Measure and Predict Students’ Effort
Moissa, B., Bonnin, G., & Boyer, A. E. (2019) · Springer eBooks · doi:10.1007/978-3-319-64301-4_19 · Cited by 8
An Event-Triggered Low-Cost Tactile Perception System for Social Robot’s Whole Body Interaction
Lin, S., Su, J., Song, S., & Zhang, J. (2021) · IEEE Access · doi:10.1109/access.2021.3053117 · Cited by 8
“There is also research on the use of smartphone screens to collect interactive data [35], and to predict cognitive states [36].”
BackgroundLeveraging Initial Cognitive Load to Predict User Response to Complex Visual Tasks
Albaghli, R., Jandali, Y., Almahmid, S., & Attar, N. (2022) · Proceedings of the World Congress on Electrical Engineering and Computer Systems and Science · doi:10.11159/mhci22.103 · Cited by 6
“A study by Mock et al. explored the potential of using touch sensor data to predict user states in learning scenarios [26].”
Li, X., Jia, L., Sun, Q., Guo, G., & Li, W. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129317 · Cited by 4
Rhim, J., & Gweon, G. (2022) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3491140.3528311 · Cited by 2
“In another study, several features including length, speed, variance from the direct path, curvature, distance of trajectories are employed to automatically classify a student’s level of cognitive workload [9].”
The Potential of Cognitive Circles to Measure Mental Load
Méndez, G. G., Galárraga, L. A., Quijije, R., & Nacenta, M. A. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747716 · Cited by 2
“Of particular interest to us are the works of Ruiz et al. [4, 35–37], who analyzed pen strokes in a range of tasks to infer the load that they impose, Luria and Rosenblum [27] and Yu et al. [49, 50], who focus on the handwriting signal, and Mock et al. [30], who analyze touch signals.”
Rhim, J., kim, J., & Gweon, G. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4102940
Using Geometric Features of Drag-and-Drop Trajectories to Understand Students’ Learning
Rhim, J., Kim, J. L., & Gweon, G. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581143
“tasks [52], cognitive workload [47], and level of achievement in a given subject [53].”
BackgroundDelprino, F. (2023) · CINECA IRIS Institutial Research Information System (University of Genoa) · doi:10.15167/delprino-federica_phd2023-05-25
Byrne, G. (2022) · Arrow - TU Dublin (Technological University Dublin) · doi:10.21427/37jg-2r46 · Cited by 1
Maximizing students' engagement through effort-based recommendations
Moissa, B. (2021) · theses.fr (ABES) · doi:10.70675/2d6de5b4z1dfcz4480za4e2z4998f0da76ba
Byrne, G. (2026) · Arrow - TU Dublin (Technological University Dublin) · doi:10.21427/cfhb-3b73
Metadata
- Authors
- Mock, P., Gerjets, P., Tibus, M., Trautwein, U., Möller, K., & Rosenstiel, W.
- Affiliations
- Leibniz-Institut für Wissensmedien (DE)
- University of Tübingen (DE)
- Venue
- Proceedings of the 18th ACM International Conference on Multimodal Interaction
- Publisher
- ACM
- Published
- 2016-10-31
- Pages
- 349-356
- OpenAlex
- W2547090472
- Cited by
- 21
- References
- 23
- Citations by year
- 2025: 2 · 2023: 4 · 2022: 3 · 2021: 3 · 2020: 3 · 2019: 3 · 2018: 1 · 2017: 2
- FWCI
- 1.08
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Keywords
- multi-touch devices, task difficulty, fourth grade, interaction patterns
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.