Using pen input features as indices of cognitive load
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Multimodal interfaces are known to be useful in map-based applications, and in complex, time-pressure based tasks. Cognitive load variations in such tasks have been found to impact multimodal behaviour. For example, users become more multimodal and tend towards semantic complementarity as cognitive load increases. The richness of multimodal data means that systems could monitor particular input features to detect experienced load variations. In this paper, we present our attempt to induce controlled levels of load and solicit natural speech and pen-gesture inputs. In particular, we analyse for these features in the pen gesture modality. Our experimental design relies on a map-based Wizard of Oz, using a tablet PC. This paper details analysis of pen-gesture interaction across subjects, and presents suggestive trends of increases in the degree of degeneration of pen-gestures in some subjects, and possible trends in gesture kinematics, when cognitive load increases.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1994) · Psychological Review · doi:10.1037/0033-295x.101.2.343 · Cited by 17,161
Cognitive Load Theory and Instructional Design: Recent Developments
Paas, F., Renkl, A., & Sweller, J. (2003) · Educational Psychologist · doi:10.1207/s15326985ep3801_1 · Cited by 3,041
Cognitive Load Measurement as a Means to Advance Cognitive Load Theory
Paas, F., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. M. (2003) · Educational Psychologist · doi:10.1207/s15326985ep3801_8 · Cited by 2,527 · cited 2×
“They are more suited to discreet sequential tasks, as seen in traditional learning environments, rather than continuous interaction with an interface [5].”
BackgroundSpecifying gestures by example
Rubine, D. (1991) · Proceedings of the 18th annual conference on Computer graphics and interactive techniques · doi:10.1145/122718.122753 · Cited by 778 · cited 2×
“The result is the number of standard deviations a pen-gesture is away from the mean of its classification [6].”
BackgroundMethodWhen do we interact multimodally?: cognitive load and multimodal communication patterns
Oviatt, S., Coulston, R., & Lunsford, R. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027957 · Cited by 285 · cited 2×
“For example, increased use of multimodal input for more complex tasks is suggested to be a form of selfmanagement of cognitive load by using more than one modal channel to communicate [4].”
BackgroundInterpreting Symptoms of Cognitive Load in Speech Input
Berthold, A., & Jameson, A. (1999) · Courses and lectures · doi:10.1007/978-3-7091-2490-1_23 · Cited by 102 · cited 1×
“Significant variations in levels of disfluency across a number of modality channels, including gesture and speech, have also been shown in users experiencing low versus high cognitive load [2].”
BackgroundExamining the redundancy of multimodal input
Ruiz, N., Taib, R., & Chen, F. (2006) · Australian Conference on Computer-Human Interaction (OzCHI) · doi:10.1145/1228175.1228254 · Cited by 24 · cited 1×
“Another example of behaviour exhibited at high levels of load is a reduction in the degree of semantic redundancy between input modalities, where users tend to be more semantically concise when tasks become more difficult [7].”
BackgroundSelf-citation
Research works printed but not linked: 1
- Baddeley , A. , Working Memory. Science , 1992 . 255: 556 -- 559 . Baddeley, A., Working Memory. Science, 1992. 255: 556--559.
Cited by
Integrating cognitive load theory and concepts of human–computer interaction
Hollender, N., Hofmann, C., Deneke, M., & Schmitz, B. (2010) · Computers in Human Behavior · doi:10.1016/j.chb.2010.05.031 · Cited by 328
A Survey on Measuring Cognitive Workload in Human-Computer Interaction
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W. (2023) · ACM Computing Surveys · doi:10.1145/3582272 · Cited by 304
“During handwriting with a stylus, the velocity [211, 260], frequency [210], trajectory duration [209], and pressure of strokes [260] were studied as indicators for cognitive workload.”
Multimodal behavior and interaction as indicators of cognitive load
Chen, F., Ruiz, N., Choi, E., Epps, J., Khawaja, M. A., Taib, R., Yin, B., & Wang, Y. (2012) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2395123.2395127 · Cited by 141
Self-citationEstimating cognitive workload using wavelet entropy-based features during an arithmetic task
Zarjam, P., Epps, J., Chen, F., & Lovell, N. H. (2013) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2013.08.021 · Cited by 141
Self-citationBeyond Subjective Self-Rating: EEG Signal Classification of Cognitive Workload
Zarjam, P., Epps, J., & Lovell, N. H. (2015) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2015.2441960 · Cited by 140
Detecting Users’ Cognitive Load by Galvanic Skin Response with Affective Interference
Nourbakhsh, N., Chen, F., Wang, Y., & Calvo, R. A. (2017) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2960413 · Cited by 105
“Speech [18, 46], pen input [34, 47], and eye movements [8, 10] are examples of behaviors that have been studied for mental load assessment.”
BackgroundSelf-citationGSR and Blink Features for Cognitive Load Classification
Nourbakhsh, N., Wang, Y., & Chen, F. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40483-2_11 · Cited by 98
“Speech [10], pen input [21] and eye movements [5] are instances of behavioral signals used in cognitive load measurement.”
BackgroundSelf-citationLooking at Mental Effort Appraisals through a Metacognitive Lens: Are they Biased?
Scheiter, K., Ackerman, R., & Hoogerheide, V. (2020) · Educational Psychology Review · doi:10.1007/s10648-020-09555-9 · Cited by 63
“Nevertheless, research has shown that phasic EDA can be a reliable indicator of cognitive load (e.g., Ruiz et al. 2007).”
BackgroundADABase: A Multimodal Dataset for Cognitive Load Estimation
Oppelt, M. P., Foltyn, A., Deuschel, J., Lang, N. R., Holzer, N., Eskofier, B. M., & Yang, S. H. (2022) · Sensors · doi:10.3390/s23010340 · Cited by 64
“Behavioral Measurements such as linguistic [21], speech [22–24], device inputs like pen [25,26] or computer-mouse movements [27] are commonly used in various applications, while metrics computed from behaviors such as lane keeping, steering wheel activity, mean vehicle speed or the number of initiated speed changes ar…”
BackgroundCognitive load evaluation of handwriting using stroke-level features
Yu, K., Epps, J., & Chen, F. (2011) · Proceedings of the 16th international conference on Intelligent user interfaces · doi:10.1145/1943403.1943481 · Cited by 45
“In previous research, velocity and shape information [12] were analyzed for limited pen gestures.”
BackgroundSelf-citationCapturing self‐regulated learning processes in virtual reality: Causal sequencing of multimodal data
Sobocinski, M., Dever, D., Wiedbusch, M., Mubarak, F., Azevedo, R., & Järvelä, S. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13393 · Cited by 41
Stuijfzand, B. G., van der Schaaf, M. F., Kirschner, F., Ravesloot, C. J., van der Gijp, A., & Vincken, K. L. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.04.015 · Cited by 32
“…body of research using indirect and direct objective measures to measure cognitive load such as EEG (Antonenko et al., 2010), skin response (Nourbakhsh et al., 2013), speech (Khawaja, Chen, & Marcus, 2014), eye movements (Chen & Epps, 2013) and human-computer interaction (Ruiz et al., 2011, 2007).”
BackgroundCognitive Load in Voice Therapy Carry-Over Exercises
Iwarsson, J., Morris, D. J., & Balling, L. W. (2016) · Journal of Speech Language and Hearing Research · doi:10.1044/2016_jslhr-s-15-0235 · Cited by 30
“such as finger pressure on a mouse (Ikehara & Crosby, 2005), pen-input patterns (Taib et al., 2007), eye movements (Chen et al.”
BackgroundMethodSuryani, M., Sensuse, D. I., Santoso, H. B., Aji, R. F., Setiawan, H., Suryono, R. R., & Kautsarina, K. (2024) · Cognition Technology & Work · doi:10.1007/s10111-024-00772-8 · Cited by 39
“Intrinsic cognitive load relates to the inherent complexity (Ruiz et al. 2007).”
Digital ink and differentiated subjective ratings for cognitive load measurement in middle childhood
Altmeyer, K., Barz, M., Lauer, L., Peschel, M., Sonntag, D., Brünken, R., & Malone, S. (2023) · British Journal of Educational Psychology · doi:10.1111/bjep.12595 · Cited by 17
“Since high working memory load is assumed to interfere with cognitive and psycho-motor tasks, high cognitive load is expected to affect completion time in written tasks (Ruiz et al., 2007).”
Characterizing gesture knowledge transfer across multiple contexts of use
Vatavu, R.-D. (2017) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-017-0247-x · Cited by 12
“Various factors affect gesture articulation [84,89,90,95,96]”
BackgroundFreeform pen-input as evidence of cognitive load and expertise
Ruiz, N., Taib, R., & Chen, F. (2011) · Proceedings of the 13th international conference on multimodal interfaces · doi:10.1145/2070481.2070511 · Cited by 11
“speech signals [9], linguistic or grammatical features [10], or changes in the quality of interactive digital pen input [11].”
BackgroundSelf-citationCognitive skills learning: pen input patterns in computer-based athlete training
Ruiz, N., Feng, Q. Q., Taib, R., Handke, T., & Chen, F. (2010) · International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction · doi:10.1145/1891903.1891955 · Cited by 10
“When cognitive load is high, for example, the quality of the pen input trajectory may show signs of degradation, in comparison to the quality of the trajectory when load is low [7].”
BackgroundSelf-citationDrag and drop the apple: the semantic weight of words and images in touch-based interaction
Aslan, I., Murer, M., Fuchsberger, V., Fugard, A. J. B., & Tscheligi, M. (2013) · International Conference on Tangible, Embedded, and Embodied Interaction (TEI) · doi:10.1145/2460625.2460650 · Cited by 11
Wu, Z., Lin, T., & Tang, N. (2016) · International Journal of Technology and Human Interaction · doi:10.4018/ijthi.2016070102 · Cited by 8
Analysis of effect of Meditation on Cognitive Load using Higher Order Crossing Features
Jadhav, N., Manthalkar, R. R., & Joshi, Y. (2017) · doi:10.2991/iccasp-16.2017.65 · Cited by 8
Automatic cognitive load evaluation using writing features: An exploratory study
Lin, T., Xie, T., Chen, Y., & Tang, N. (2013) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2013.02.002 · Cited by 11
Modelling students' effort using behavioral data
Moissa, B., Bonnin, G., Castagnos, S., & Boyer, A. E. (2019) · HAL · Cited by 7
Workload on your fingertips: the influence of workload on touch-based drag and drop
Aslan, I., Murer, M., Fuchsberger, V., Fugard, A. J. B., & Tscheligi, M. (2013) · ACM International Conference on Interactive Tabletops and Surfaces (ITS) · doi:10.1145/2512349.2514918 · Cited by 6
“While there is prior work in handwriting, which suggests that it is possible to infer a user’s cognitive state from measures in complex pen gestures [4, 5, 6] (e.”
BackgroundDesigning Cognition-Adaptive Human–Computer Interface for Mission-Critical Systems
Shi, Y., Choi, E. H. C., Taib, R., & Chen, F. (2009) · Springer eBooks · doi:10.1007/b137171_12 · Cited by 7
“One interface allows traffic controllers to request information and issue commands using speech and handwriting on tablet monitor, facilitating fast incident data input, and eliminating tedious form filling and paper scratch pad [ 12 ].”
BackgroundSelf-citationExploiting Wearable Technologies to Measure and Predict Students’ Effort
Moissa, B., Bonnin, G., & Boyer, A. E. (2019) · Springer eBooks · doi:10.1007/978-3-319-64301-4_19 · Cited by 8
Teaching Athletes Cognitive Skills: Detecting Cognitive Load in Speech Input
Ruiz, N., Liu, G., Yin, B., Farrow, D., & Chen, F. (2010) · Electronic workshops in computing · doi:10.14236/ewic/hci2010.61 · Cited by 5
“…this, monitoring the cognitive load experienced by each learner as they learn is a key element for the development of adaptive educational user interfaces, in which the pace, content and presentation format of instruction can be dynamically adjusted to the learner’s current working memory state.”
BackgroundSelf-citationLin, T., Li, X., Wu, Z., & Tang, N. (2013) · International Journal of Technology and Human Interaction · doi:10.4018/jthi.2013070106 · Cited by 5
Effects of Information Widgets on Time Perception during Mentally Demanding Tasks
Li, Z., Shi, D., Gao, Q., Chen, Y., Wang, N., & Ren, X. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713270 · Cited by 4
“According to Warnock et al. and Ruiz et al., this approach demonstrated the power of controlled pressure to optimize user engagement and task performance in complex, high-demand environments [91, 114].”
Factors affecting nursing staff use of nursing information systems in residential aged care homes
Galani, M. (2015) · Research Online (University of Wollongong) · doi:10.13140/rg.2.1.4364.6969 · Cited by 5
Assessing recovery from cognitive load through pen input
Luo, L., & Taib, R. (2013) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2468356.2468597 · Cited by 2
“It can capture variations in performance implicitly without interrupting the task, and the data is available for analysis once the current gesture is finished [6].”
BackgroundMethodSelf-citationThe Potential of Cognitive Circles to Measure Mental Load
Méndez, G. G., Galárraga, L. A., Quijije, R., & Nacenta, M. A. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747716 · Cited by 2
Using High-Frequency Interaction Events to Automatically Classify Cognitive Load
Lin, T., Wu, Z., & Chen, Y. (2015) · Advances in human and social aspects of technology book series · doi:10.4018/978-1-4666-8450-8.ch010
Wu, Z., Lin, T., & Tang, N. (2020) · IGI Global eBooks · doi:10.4018/978-1-7998-2460-2.ch072
Metadata
- Authors
- Ruiz, N., Taib, R., Shi, Y. (., Choi, E., & Chen, F.
- Affiliations
- UNSW Sydney (AU)
- Data61 (AU)
- Venue
- Proceedings of the 9th international conference on Multimodal interfaces
- Publisher
- ACM
- Published
- 2007-11-12
- Pages
- 315-318
- OpenAlex
- W2017617574
- Cited by
- 34
- References
- 7
- Citations by year
- 2025: 2 · 2024: 1 · 2023: 3 · 2022: 1 · 2020: 2 · 2019: 2 · 2017: 3 · 2016: 3 · 2015: 3 · 2013: 7 · 2012: 1
- FWCI
- 1.18
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Keywords
- cognitive load, pen-based input, gesture kinematics, multimodal interfaces, map-based interface, Wizard of Oz, tablets, speech input, time pressure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.