Modeling Touch-based Menu Selection Performance of Blind Users via Reinforcement Learning
Open accessAbstract
Although menu selection has been extensively studied in HCI, most existing studies have focused on sighted users, leaving blind users’ menu selection under-studied. In this paper, we propose a computational model that can simulate blind users’ menu selection performance and strategies, including the way they use techniques like swiping, gliding, and direct touch. We assume that selection behavior emerges as an adaptation to the user’s memory of item positions based on experience and feedback from the screen reader. A key aspect of our model is a model of long-term memory, predicting how a user recalls and forgets item position based on previous menu selections. We compare simulation results predicted by our model against data obtained in an empirical study with ten blind users. The model correctly simulated the effect of the menu length and menu arrangement on selection time, the action composition, and the menu selection strategy of the users.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Assistive technology research for improving computer application usability has largely focused on web and mobile applications.
“However, these works have largely centered on the web [6, 12, 15, 16, 43] and mobile applications [11, 22, 27–29].”
From Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models (2024) · cites this paper as [28]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · see citing works
Data Analysis Using Regression and Multilevel/Hierarchical Models
Hilbe, J. (2009) · Journal of Statistical Software · doi:10.18637/jss.v030.b03 · Cited by 8,128 · see citing works
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582 · see citing works
Gajos, K. Z., & Weld, D. S. (2004) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/964442.964461 · Cited by 431 · see citing works
Kane, S. K., Bigham, J. P., & Wobbrock, J. O. (2008) · Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/1414471.1414487 · Cited by 374 · see citing works
A predictive model of menu performance
Cockburn, A., Gutwin, C., & Greenberg, S. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240723 · Cited by 306 · cited 3× · see citing works
“Modeling target selection behavior on menus and estimation of selection time with sighted users have been well-studied [2, 5, 6, 13, 15].”
BackgroundMethodModel-based evaluation of expert cell phone menu interaction
Amant, R. S., Horton, T. E., & Ritter, F. E. (2007) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1229855.1229856 · Cited by 180 · see citing works
Khan, A., & Khusro, S. (2021) · Universal Access in the Information Society · doi:10.1007/s10209-020-00733-8 · Cited by 177 · see citing works
Using strokes as command shortcuts
Appert, C., & Zhai, S. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519052 · Cited by 163 · see citing works
Computational Rationality as a Theory of Interaction
Oulasvirta, A., Jokinen, J., & Howes, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517739 · Cited by 128 · see citing works
Self-citationSenjam, S. S., Manna, S., & Bascarán, C. (2021) · Clinical Optometry · doi:10.2147/opto.s336361 · Cited by 110 · see citing works
The Emergence of Interactive Behavior
Chen, X., Bailly, G., Brumby, D. P., Oulasvirta, A., & Howes, A. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702483 · Cited by 105 · see citing works
Self-citationTouchscreen Typing As Optimal Supervisory Control
Jokinen, J., Acharya, A., Uzair, M., Jiang, X., & Oulasvirta, A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445483 · Cited by 104 · see citing works
Self-citationUnderstanding palm-based imaginary interfaces
Gustafson, S. G., Rabe, B., & Baudisch, P. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466114 · Cited by 94 · see citing works
Bailly, G., Oulasvirta, A., Kötzing, T., & Hoppe, S. (2013) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2501988.2502024 · Cited by 90 · see citing works
Self-citationFrom Tapping to Touching: Making Touch Screens Accessible to Blind Users
Guerreiro, T., Lagoá, P., Nicolau, H., Gonalves, D., & Jorge, J. A. (2008) · IEEE Multimedia · doi:10.1109/mmul.2008.88 · Cited by 89 · see citing works
Accessible Touchscreen Technology for People with Visual Impairments: A Survey
Grussenmeyer, W., & Folmer, E. (2017) · ACM Transactions on Accessible Computing · doi:10.1145/3022701 · Cited by 86 · see citing works
Adaptive feature guidance: Modelling visual search with graphical layouts
Jokinen, J., Wang, Z., Sarcar, S., Oulasvirta, A., & Ren, X. (2019) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2019.102376 · Cited by 86 · see citing works
Self-citationPredicting Mid-Air Interaction Movements and Fatigue Using Deep Reinforcement Learning
Cheema, N., Frey‐Law, L., Naderi, K., Lehtinen, J., Slusallek, P., & Hämäläinen, P. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376701 · Cited by 71 · see citing works
Model of visual search and selection time in linear menus
Bailly, G., Oulasvirta, A., Brumby, D. P., & Howes, A. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557093 · Cited by 69 · see citing works
Self-citationA system for fast, full-text entry for small electronic devices
Nesbat, S. B. (2003) · International Conference on Multimodal Interfaces · doi:10.1145/958432.958437 · Cited by 66 · see citing works
Ability-Based Optimization of Touchscreen Interactions
Sarcar, S., Jokinen, J., Oulasvirta, A., Wang, Z., Silpasuwanchai, C., & Ren, X. (2018) · IEEE Pervasive Computing · doi:10.1109/mprv.2018.011591058 · Cited by 60 · see citing works
Self-citationInferring Cognitive Models from Data using Approximate Bayesian Computation
Kangasrääsiö, A., Athukorala, K., Howes, A., Corander, J., Samuel Kaski, A., & Oulasvirta, A. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025576 · Cited by 56 · see citing works
Self-citationSmartphone usage among people living with severe visual impairment and blindness
Abraham, C. H., Boadi-Kusi, B., Morny, E. K. A., & Agyekum, P. (2022) · Assistive Technology · doi:10.1080/10400435.2021.1907485 · Cited by 49 · see citing works
A Cognitive Model of How People Make Decisions Through Interaction with Visual Displays
Chen, X., Starke, S. D., Baber, C., & Howes, A. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025596 · Cited by 47 · see citing works
The Emergence of Zipf's Law: Spontaneous Encoding Optimization by Users of a Command Language
Ellis, S. R., & Hitchcock, R. J. (1986) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1986.4308973 · Cited by 41 · see citing works
Smartphone Usage by Expert Blind Users
Jain, M., Diwakar, N., & Swaminathan, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445074 · Cited by 41 · cited 1× · see citing works
“, selecting items on a menu) is one of the most basic and common tasks that blind users perform on smartphones [30, 40].”
BackgroundThe effect of time-based cost of error in target-directed pointing tasks
Banović, N., Grossman, T., & Fitzmaurice, G. W. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466181 · Cited by 40 · see citing works
An Adaptive Model of Gaze-based Selection
Chen, X., Acharya, A., Oulasvirta, A., & Howes, A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445177 · Cited by 40 · see citing works
Self-citationYalla, P., & Walker, B. N. (2008) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1414471.1414492 · Cited by 37 · see citing works
Brumby, D. P., Davies, S. C. E., Janssen, C. P., & Grace, J. J. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979009 · Cited by 34 · see citing works
Evaluating Fitts’ law on vibrating touch-screen to improve visual data accessibility for blind users
Lahib, M. E., Tekli, J., & Issa, Y. B. (2018) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2018.01.005 · Cited by 30 · see citing works
A Large-Scale Study of iPhone App Launch Behaviour
Morrison, A., Xiong, X., Higgs, M., Bell, M., & Chalmers, M. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173918 · Cited by 27 · see citing works
The adaptation of visual search to utility, ecology and design
Tseng, Y., & Howes, A. (2015) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2015.03.005 · Cited by 24 · see citing works
Effects of Frequency Distribution on Linear Menu Performance
Liu, W., Bailly, G., & Howes, A. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025707 · Cited by 23 · see citing works
Lelis, S., & Howes, A. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979278 · Cited by 21 · see citing works
Trappenberg, T. P. (2019) · Oxford University Press eBooks · doi:10.1093/oso/9780198828044.003.0010 · Cited by 19 · see citing works
Using Bayes' Theorem for Command Input: Principle, Models, and Applications
Zhu, S., Kim, Y., Zheng, J., Luo, J. Y., Qin, R., Wang, L., Fan, X., Tian, F., & Bi, X. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376771 · Cited by 18 · see citing works
Self-citationLi, Z., Zhao, M., Das, D., ZHAO, H., Ma, Y., Liu, W., Beaudouin-Lafon, M., Wang, F., Ramakrishnan, IV, & Bi, X. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517472 · Cited by 10 · see citing works
Self-citationShukri, S. R. M., & Howes, A. (2019) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2019.06.004 · Cited by 8 · see citing works
Modeling Gliding-based Target Selection for Blind Touchscreen Users
Ko, Y.-J., Putkonen, A., Aydin, A. S., Feiz, S., Wang, Y., Ashok, V., Ramakrishnan, IV, Oulasvirta, A., & Bi, X. (2021) · Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction · doi:10.1145/3447526.3472022 · Cited by 3 · see citing works
Self-citationProbabilistic Formulation of the Take The Best HeuristicPreprint
Peltola, T., Jokinen, J., & Kaski, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00572
Flagged first, then by citations
Research works printed but not linked: 20
- [6]6. Gilles Bailly, Antti Oulasvirta, Timo Kötzing, and Sabrina Hoppe. 2013. Menuopti- mizer: Interactive optimization of menu systems. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, St. Andrews, UK, 331-342.
- [8]8. Richard E Bellman and Lotf Asker Zadeh. 1970. Decision-making in a fuzzy environment. Management science 17, 4 (1970), B-141.
- [16]16. Alistair DN Edwards. 1989. Modelling blind users' interactions with an auditory computer interface. International journal of man-Machine studies 30, 5 (1989), 575-589.
- [19]19. Paul M Fitts. 1954. The information capacity of the human motor system in controlling the amplitude of movement. Journal of experimental psychology 47, 6 (1954), 381.
- [20]20. Krzysztof Gajos and Daniel S Weld. 2004. SUPPLE: automatically generating user interfaces. In Proceedings of the 9th international conference on Intelligent user interfaces. ACM, Funchal, Portugal, 93-100.
- [22]22. Samuel J Gershman, Eric J Horvitz, and Joshua B Tenenbaum. 2015. Computa- tional rationality: A converging paradigm for intelligence in brains, minds, and machines. Science 349, 6245 (2015), 273-278.
- [29]29. Andrew Howes, Richard L Lewis, and Alonso Vera. 2009. Rational adaptation un- der task and processing constraints: implications for testing theories of cognition and action. Psychological review 116, 4 (2009), 717.
- [31]31. Edwin T Jaynes. 1957. Information theory and statistical mechanics. Physical review 106, 4 (1957), 620.
- [33]33. Jussi PP Jokinen and Tuomo Kujala. 2021. Modelling Drivers' Adaptation to Assistance Systems. In 13th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, Leeds, UK, 12-19.
- [34]34. Jussi PP Jokinen, Tuomo Kujala, and Antti Oulasvirta. 2021. Multitasking in driving as optimal adaptation under uncertainty. Human factors 63, 8 (2021), 1324-1341.
- [36]36. Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. 1998. Plan- ning and acting in partially observable stochastic domains. Artifcial intelligence 101, 1-2 (1998), 99-134.
- [37]37. Shaun K. Kane, Jefrey P. Bigham, and Jacob O. Wobbrock. 2008. Slide Rule: Making Mobile Touch Screens Accessible to Blind People Using Multi-Touch Interaction Techniques. In Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility (Halifax, Nova Scotia, Canada) (Assets '08). Association for Computing Machinery, New York, NY, USA, 73-80.
- [43]43. Richard L Lewis, Andrew Howes, and Satinder Singh. 2014. Computational ratio- nality: Linking mechanism and behavior through bounded utility maximization. Topics in cognitive science 6, 2 (2014), 279-311.
- [44]44. Frank Chun Yat Li, David Dearman, and Khai N Truong. 2010. Leveraging proprioception to make mobile phones more accessible to users with visual impairments. In Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility. ACM, Orlando, FL, USA, 187-194.
- [47]47. Wanyu Liu, Gilles Bailly, and Andrew Howes. 2017. Efects of frequency distri- bution on linear menu performance. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, Denver, CO, USA, 1307-1312.
- [48]48. I Scott MacKenzie. 1992. Fitts' law as a research and design tool in human- computer interaction. Human-computer interaction 7, 1 (1992), 91-139.
- [54]54. Rajesh PN Rao. 2010. Decision making under uncertainty: a neural model based on partially observable markov decision processes. Frontiers in computational neuroscience 4 (2010), 146.
- [55]55. Stuart J Russell and Devika Subramanian. 1994. Provably bounded-optimal agents. Journal of Artifcial Intelligence Research 2 (1994), 575-609.
- [59]59. Benjamin L Somberg. 1986. A comparison of rule-based and positionally constant arrangements of computer menu items. ACM SIGCHI Bulletin 17, SI (1986), 255- 260.
- [60]60. Matthijs TJ Spaan. 2012. Partially observable Markov decision processes. In Reinforcement Learning. Springer, 387-414.
Cited by
Sorted by citations
CRTypist: Simulating Touchscreen Typing Behavior via Computational Rationality
Shi, D., Zhu, Y., Jokinen, J., Acharya, A., Putkonen, A., Zhai, S., & Oulasvirta, A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642918 · Cited by 25 · see citing works
Self-citationEnabling Uniform Computer Interaction Experience for Blind Users through Large Language Models
Kodandaram, S. R., Uckun, U., Bi, X., Ramakrishnan, IV, & Ashok, V. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675605 · Cited by 24 · see citing works
Self-citationChartist: Task-driven Eye Movement Control for Chart Reading
Shi, D., Wang, Y., Bai, Y., Bulling, A., & Oulasvirta, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713128 · Cited by 15 · see citing works
Self-citationWang, Y., Srinivasan, A. R., Jokinen, J. P. P., Oulasvirta, A., & Markkula, G. (2025) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2024.104963 · Cited by 11 · see citing works
Self-citationVidmanov, D. A., & Alfimtsev, A. (2024) · Multimodal Technologies and Interaction · doi:10.3390/mti8040026 · Cited by 13 · see citing works
Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays
Bai, Y., Ikkala, A., Oulasvirta, A., Zhao, S., Wang, L. J., Yang, P., & Xu, P. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642540 · Cited by 7 · see citing works
Self-citationWang, Y., Srinivasan, A. R., Lee, Y. M., & Markkula, G. M. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3581693 · Cited by 5 · see citing works
Explaining crowdworker behaviour through computational rationality
Hedderich, M. A., & Oulasvirta, A. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2329616 · Cited by 4 · see citing works
Self-citationUnderstanding the Effects of Visual Impairment on Visual Search
Aljasem, D., & Howes, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60884-1_25 · Cited by 1 · see citing works
Spatial Variability Models for Straight Pen-stroking Tasks with Temporal Constraints
Yamanaka, S., & Miyashita, H. (2025) · Journal of Information Processing · doi:10.2197/ipsjjip.33.91
Computational Modeling of Non-Visual Vibrotactile Touchscreen Exploration
Hwang, S., Moss, R. J., Fan, D., & Follmer, S. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719851
Fleig, A., Fischer, F., Klar, M., Ebel, P., Bachinski, M., Kristensson, P. O., Murray‐Smith, R., & Oulasvirta, A. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746058.3758473
Self-citationAdaptive Bounded-Rationality Modeling of Early-Stage Takeover in Shared-Control Driving
Sun, J., Jiang, X., Zhao, X., Wang, J., Hang, P., & Li, Z. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790701 · Cited by 1 · see citing works
Self-citationZhang, H., Xiao, Y., Zheng, Y., Huang, J., Yan, X., Hu, X., Li, N., Tu, H., & Tian, F. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791672
Offline Reinforcement Learning for Adaptive Support in AI-Assisted Decision-Making
Buçinca, Z., Swaroop, S., Paluch, A. E., Murphy, S. A., & Gajos, K. Z. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3815418
Beyond Correctness: A Design Vocabulary for Perceived Response Dynamics in Affective Adaptation
Luganga, A. H., Keighrey, C., & Murray, N. (2026) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3788851.3805013
Integrating Multi-Source Feedback in Computational Design
Fernandes, F. E., Langerak, T., Keränen, M., Shi, D., Alipova, A., & Oulasvirta, A. (2026) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3848025
Self-citationFocalFlow: Facilitating Decision-Making in Multi-Step Mobile Tasks for Blind and Low Vision People
Xie, J., Wu, Z., Wang, Z., Li, Z., Franz, R. L., & Fan, M. (2026) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3831982 · Cited by 1 · see citing works
Self-citation
- Authors
- Li, Z., Ko, Y.-J., Putkonen, A., Feiz, S., Ashok, V., Ramakrishnan, IV, Oulasvirta, A., & Bi, X.
- Affiliations
- Stony Brook University (US)
- Aalto University (FI)
- Old Dominion University (US)
- ORCID
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-18
- OpenAlex
- W4366588290
- Cited by
- 18
- References
- 39
- Citations by year
- 2026: 6 · 2025: 5 · 2024: 7
- FWCI
- 9.26
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Aaltodoc (Aalto University) (submittedVersion, other-oa)
- Funders
- DOD award (AL210015)
- NIH award (R01EY030085)
- NSF award (2113485)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Keywords
- blind users, menu selection, touch interaction, screen readers, long-term memory, reinforcement learning, computational modeling, selection strategies, accessibility, selection time
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
