The DOI matches its DataCite record; 14 of the 15 references it cites were found in a registry or an index, and 1 are not checked yet; none of the 33 sources its claims cite is checked for support yet.
Attention, Action, and Memory: How Multi-modal Interfaces and Cognitive Load Alter Information Retention
Open accessAbstract
Each year, multi-modal interaction continues to grow within both industry and academia. However, researchers have yet to fully explore the impact of multi-modal systems on learning and memory retention. This research investigates how combining gaze-based controls with gesture navigation affects information retention when compared to standard track-pad usage. A total of twelve participants read four textual articles through two different user interfaces which included a track-pad and a multi-modal interface that tracked eye movements and hand gestures for scrolling, zooming, and revealing content. Participants underwent two assessment sessions that measured their information retention immediately and after a twenty-four hour period along with the NASA-TLX workload evaluation and the System Usability Scale assessment. The initial analysis indicates that multi-modal interaction produces similar targeted information retention to traditional track-pad usage, but this neutral effect comes with higher cognitive workload demands and seems to deteriorate with long-term retention. The research results provide new knowledge about how multi-modal systems affect cognitive engagement while providing design recommendations for future educational and assistive technologies that require effective memory performance.
Checked 2026-11-18 · 1 reference not checked yet
Overview
CoChat Summary
TL;DR
In a within-subjects study with 12 participants, a gaze+gesture reading interface matched track-pad use for immediate information retention but showed significantly lower 24-hour delayed retention, alongside higher cognitive workload and moderate usability. The authors conclude that multi-modal interaction can impose extra cognitive burden that may hamper deeper learning despite short-term parity.
Key findings
- Immediate retention was slightly higher with track-pad (M = 7.88, SD = 0.71) than gaze+gesture (M = 7.38, SD = 1.00), but the paired-samples t-test was not significant, t(11) = -2.03, p = 0.067.
- Delayed retention was significantly higher with track-pad (M = 7.25, SD = 1.41) than gaze+gesture (M = 6.42, SD = 1.06), t(11) = -2.87, p = 0.015; Wilcoxon signed-rank W = 0.0, p = 0.011.
- Reading times were similar: gaze+gesture averaged 277.98 seconds (~4.63 min) per article versus 281.04 seconds (~4.68 min) for track-pad.
- For gaze+gesture, NASA-TLX subscales were Mental Demand M = 4.00 (SD = 1.21), Physical Demand M = 3.25 (SD = 1.91), Temporal Demand M = 3.92 (SD = 1.88), Performance M = 4.08 (SD = 0.90), Effort M = 3.92 (SD = 1.56), and Frustration M = 3.17 (SD = 1.11); SUS averaged 62.29 (SD = 14.87), below the 68 benchmark. Bottom retention performers made roughly 30% more gestures than top performers.
Methods
- Within-subjects user study with 12 participants (M = 8, F = 4; age 18–28; 10 right-handed, 2 left-handed; normal or corrected-to-normal vision; P5 owned a Meta Quest3; P9 reported ADHD). Each participant completed both gaze+gesture and track-pad conditions across four articles, with system condition and article randomized.
- Procedure: up to 10 minutes of training on the gaze+gesture system; maximum 8 minutes per article (articles rated ~4 minutes); after each article, a retention quiz with 3 multiple-choice, 3 true/false, and 3 short-answer questions; after all reading, SUS, NASA-TLX, and a semi-structured questionnaire; next day, four delayed quizzes for the same articles.
- System and analysis: custom gaze tracking via Beam Eye Tracker SDK using a webcam (stated 1.5° margin of error) and gesture recognition via OpenCV with MediaPipe Hands at 30 FPS for scroll and zoom gestures; analysis used paired-samples t-tests, Wilcoxon signed-rank tests, descriptive statistics, and qualitative theme aggregation.
Limitations
- Stated in the paper: small sample size (N = 12) limits generalizability; participants likely had far more lifetime track-pad experience, which could skew results toward the traditional interface; limited training may not capture the multi-modal learning curve; and the study examined article reading only, so problem-solving or creative tasks may differ.
- Evident from the text: recruitment was mostly via university posters and participants were aged 18–28, narrowing demographic and educational range; with only 12 participants, individual differences such as P9’s ADHD diagnosis or P5’s prior VR gesture/gaze experience may disproportionately influence results.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
0 Valid · 0 Not valid · 20 Not checked · 56 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants required roughly the same reading time with both systems, averaging 277.98 seconds (about 4.63 minutes) per article for gaze+gesture and 281.04 seconds (about 4.68 minutes) for trackpad.”
§ Attention, Action, and Memory: How Multi-modal Interfaces and Cognitive Load Alter Information Retention > Gaze+Gesture
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although multi-modal gaze+gesture interaction achieves information retention on par with the trackpad interface's immediate recall”
- Also stated in: § 7 Conclusion · hedged
“Although the immediate recall scores of the two interfaces were roughly comparable”
§ Attention, Action, and Memory: How Multi-modal Interfaces and Cognitive Load Alter Information Retention > 6 Discussions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 7 Conclusion · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average SUS score for the gaze+gesture interface was 62.29 (SD = 14.87) out of 100.”
§ Attention, Action, and Memory: How Multi-modal Interfaces and Cognitive Load Alter Information Retention > M
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Problems with backward navigation, as well as discomfort with the zoom gesture, were also frequently noted.”
§ Attention, Action, and Memory: How Multi-modal Interfaces and Cognitive Load Alter Information Retention > Retention and Understanding.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“zoom gestures were usually awkward or difficult to activate with certainty”
§ Attention, Action, and Memory: How Multi-modal Interfaces and Cognitive Load Alter Information Retention > Mental Effort.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
21 more claims
Create a free account to see all 76 claims and every source verdict.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 31
- The quote only partly states this claim. 3
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentt(11) = -2.03reported p = .067 · recomputed p = .067
“t (11) = -2.03,p = .067”
- Consistentt(11) = -2.87reported p = .015 · recomputed p = .015
“t (11) = -2.87,p = .015”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Self-check results appear here once an independent check has confirmed them.
Citations
Cites
Development and Validation of a Surgical Workload Measure: The Surgery Task Load Index (SURG‐TLX)
Wilson, M. R., Poolton, J. M., Malhotra, N., Ngo, K., Bright, E., & Masters, R. S. W. (2011) · World Journal of Surgery · doi:10.1007/s00268-011-1141-4 · Cited by 414
Determining what individual SUS scores mean
(2009) · Journal of Usability Studies archive · doi:10.5555/2835587.2835589 · Cited by 412
Gaze + pinch interaction in virtual reality
Pfeuffer, K., Mayer, B., Mardanbegi, D., & Gellersen, H. (2017) · Proceedings of the 5th Symposium on Spatial User Interaction · doi:10.1145/3131277.3132180 · Cited by 331
A Survey on Measuring Cognitive Workload in Human-Computer Interaction
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W. (2023) · ACM Computing Surveys · doi:10.1145/3582272 · Cited by 304
Human Mental Workload: A Survey and a Novel Inclusive Definition
Longo, L., Wickens, C. D., Hancock, G., & Hancock, P. A. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.883321 · Cited by 222
Chatterjee, I., Xiao, R., & Harrison, C. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2820752 · Cited by 134
A survey of multimodal learning: Methods, applications, and future
Yuan, Y., Li, Z., & Zhao, B. (2025) · ACM Computing Surveys · doi:10.1145/3713070 · Cited by 90
Naik, R., Kogkas, A., Ashrafian, H., Mylonas, G. P., & Darzi, A. W. (2022) · Journal of Surgical Research · doi:10.1016/j.jss.2022.07.010 · Cited by 48
THE IMPORTANCE OF HUMAN MENTAL WORKLOAD IN WEB DESIGN
Longo, L., Rusconi, F., Noce, L., & Barrett, S. (2012) · International Conference on Web Information Systems and Technologies · doi:10.5220/0003960204030409 · Cited by 41
Kopp, S., & Bergmann, K. (2017) · ACM eBooks · doi:10.1145/3015783.3015791 · Cited by 33
Research works printed but not linked: 5
- G. Bradski. 2000. The OpenCV Library. Dr. Dobb's Journal of Software Tools (2000).
- Martyna Kaczorowska. 2023. Measurement and Analysis of Cognitive Workload on the Basis of Eye-Tracking. Ph. D. Dissertation. Polish-Japanese Academy of Information Technology. https://pjait.edu.pl/en PhD Dissertation.
- Alexandra Papoutsaki, Patsorn Sangkloy, James Laskey, Nediyana Daskalova, Jeff Huang, and James Hays. 2016. Webgazer: scalable webcam eye tracking using user interactions. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (New York, New York, USA) (IJCAI'16). AAAI Press, 3839-3845.
- Stefan Schuss, Philipp Schäfer, Marc Schweigert, and Albrecht Schmidt. 2024. Heart Rate Dynamics for Cognitive Load Estimation in a Driving Simulation Task. Behavior Research Methods (2024). doi:10.3758/s13428-023-02289-6 · doi:10.3758/s13428-023-02289-6
- John Sweller, Paul Ayres, and Slava Kalyuga. 2011. Cognitive Load Theory. Springer. https://link.springer.com/book/10.1007/978-1-4419-8126-1 · doi:10.1007/978-1-4419-8126-1
Cited by
The eye-tracking accuracy of Apple Vision Pro and its impact on perceived usability
Huang, Z., Zhu, G., Duan, X., Li, Y., Wang, R., Zhang, X., & Wang, Z. (2026) · Virtual Reality · doi:10.1007/s10055-026-01442-y · Cited by 15
A Lexical Learning Case Study of Multimodality Discourse Analysis
Lai, J., & Xue, R. (2026) · Contemporary Education and Teaching Research · doi:10.61360/bonicetr262019790301
Raposo, D., Neves, J., Silva, R., Castilho, L. C., Dias, R., Fernandes, A. M., & Neves, L. (2026) · Springer series in design and innovation · doi:10.1007/978-3-032-40958-4
Digital Reading Experience for Users with ADHD: How Can Design Contribute?
Duarte, C., & Reis, M. (2026) · Springer series in design and innovation · doi:10.1007/978-3-032-40958-4_7
Metadata
- Authors
- Elgohary, O., & Zhu-Tien
- Affiliations
- University of Minnesota (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2509.05898
- OpenAlex
- W4414756152
- Cited by
- 4
- References
- 14
- Citations by year
- 2026: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Keywords
- multimodal interfaces, information retention, long-term memory, gaze-based interaction, mental workload, cognitive load, eye tracking, usability, educational technology, assistive technology, reading
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Attention, Action, and Memory: How Multi-modal Interfaces and Cognitive Load Alter Information Retention. https://app.cochat.ai/papers/10.48550/arxiv.2509.05898
- The paper
- Elgohary, O., & Zhu-Tien. (2025). Attention, Action, and Memory: How Multi-modal Interfaces and Cognitive Load Alter Information Retention. arXiv. https://doi.org/10.48550/arxiv.2509.05898
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.