Older Adults’ Think-Aloud Verbalizations and Speech Features for Identifying User Experience Problems
The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
Subtle patterns in users’ think-aloud (TA) verbalizations and speech features are shown to be telltale signs of User Experience (UX) problems. However, such patterns were uncovered among young adults. Whether such patterns apply for older adults remains unknown. We conducted TA usability testing with older adults using physical and digital products. We analyzed their verbalizations, extracted speech features, identified UX problems, and uncovered the patterns that indicate UX problems. Our results show that when older adults encounter problems, their verbalizations tend to include observations (remarks), negations, question words and words with negative sentiments; and their voices tend to include high loudness, high pitch and high speech rate. We compare these subtle patterns with those of young adults uncovered in recent studies and discuss the implications of these patterns for the design of Human-AI collaborative UX analysis tools to better pinpoint UX problems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“older users tend to verbalize question words, like 'what', 'why', and 'how', when encountering an interaction problem [17].”
From "Where is history": Toward Designing a Voice Assistant to help Older Adults locate Interface Features quickly (2023) · cites this paper as [17]
In the citing paper
“Similar to a previous study, older users sometimes used question words, like”
From "Where is history": Toward Designing a Voice Assistant to help Older Adults locate Interface Features quickly (2023) · cites this paper as [17]
In the citing paper
“A prior study looking into older users' think-aloud (TA) verbalization and speech features found that when encountering an interaction problem, older users tend to verbalize question words, like 'what', 'why', and 'how' [17].”
From "Where is history": Toward Designing a Voice Assistant to help Older Adults locate Interface Features quickly (2023) · cites this paper as [17]
In the citing paper
“These types of articulations have also been reported in the literature when older users encountered interaction problems [17,69].”
From "Where is history": Toward Designing a Voice Assistant to help Older Adults locate Interface Features quickly (2023) · cites this paper as [17]
Citations
Cites
(1993) · Cited by 9,879
Praat: Doing Phonetics by Computer
(2011) · Ear & Hearing · doi:10.1097/aud.0b013e31821473f7 · Cited by 7,863
Protocol Analysis: Verbal Reports as Data
Venkatesan, M., Ericsson, K. A., & Simon, H. A. (1986) · Journal of Marketing Research · doi:10.2307/3151491 · Cited by 4,347
NLTK: the Natural Language Toolkit
Loper, E., & Bird, S. (2002) · Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics - · doi:10.3115/1118108.1118117 · Cited by 3,218
The Usability Engineering Lifecycle
NIELSEN, J. (1993) · Usability Engineering · doi:10.1016/b978-0-08-052029-2.50007-3 · Cited by 2,657
The handbook of aging and cognition
Craik, F. I. M., & Salthouse, T. A. (1992) · doi:10.4324/9781410601988 · Cited by 2,003
Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests
Rubin, J. N. (1994) · Cited by 1,740
Thinking aloud: reconciling theory and practice
Boren, T., & Ramey, J. (2000) · IEEE Transactions on Professional Communication · doi:10.1109/47.867942 · Cited by 808
Honest Signals: How They Shape Our World
Pentland, A. (2008) · Cited by 568
Salthouse, T. A. (1985) · Cited by 538
Handbook of usability testing: How to plan, design, and conduct effective tests
Shaw, D. (1996) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199603)47:3<258::aid-asi18>3.3.co;2-l · Cited by 455
The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods
Hertzum, M., & Jacobsen, N. E. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_05 · Cited by 419
The Handbook of Aging and Cognition
(2011) · Psychology Press eBooks · doi:10.4324/9780203837665 · Cited by 397
Paraverbal indicators of deception: a meta‐analytic synthesis
Sporer, S. L., & Schwandt, B. (2006) · Applied Cognitive Psychology · doi:10.1002/acp.1190 · Cited by 369
What happened to remote usability testing?: an empirical study of three methods
Andreasen, M. S., Nielsen, H. V., Schrøder, S. O., & Stage, J. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240838 · Cited by 177
In a Nervous Voice: Acoustic Analysis and Perception of Anxiety in Social Phobics’ Speech
Laukka, P., Linnman, C., Åhs, F., Pissiota, A., Frans, Ö., Faria, V., Michelgård, Å., Appel, L., Fredrikson, M. G., & Furmark, T. (2008) · Journal of Nonverbal Behavior · doi:10.1007/s10919-008-0055-9 · Cited by 172
Analysis in practical usability evaluation: a survey study
Følstad, A., Law, E., & Hornbæk, K. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208365 · Cited by 135
Handbook of usability testing: How to plan, design, and conduct effective tests
Shaw, D. (1996) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199603)47:3<258::aid-asi18>3.0.co;2-# · Cited by 134
Rethinking Thinking Aloud: A Comparison of Three Think-Aloud Protocols
Alhadreti, O., & Mayhew, P. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173618 · Cited by 133
Cooke, L. (2010) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.2010.2052859 · Cited by 128
Exploring Think-Alouds in Usability Testing: An International Survey
McDonald, S., Edwards, H. M., & Zhao, T. (2012) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.2011.2182569 · Cited by 122
An Empirical Comparison of Lab and Remote Usability Testing of Web Sites
Tullis, T., Fleischman, S., McNulty, M., Cianchette, C., & Bergel, M. (2002) · Cited by 120
"Why would anybody do this?": Understanding Older Adults' Motivations and Challenges in Crowd Work
Brewer, R., Morris, M. R., & Piper, A. M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858198 · Cited by 111
Light, L. L., & Salthouse, T. A. (1987) · The American Journal of Psychology · doi:10.2307/1422647 · Cited by 107
Let your users do the testing: a comparison of three remote asynchronous usability testing methods
Bruun, A., Gull, P., Hofmeister, L., & Stage, J. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518948 · Cited by 85
Automatic detection of usability smells in web applications
Grigera, J., Garrido, A., Rivero, J. M., & Rossi, G. H. (2016) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2016.09.009 · Cited by 83
Synchronous remote usability testing: a new approach facilitated by virtual worlds
Madathil, K. C., & Greenstein, J. S. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979267 · Cited by 78
Older Adult Experience of Online Diagnosis: Results From a Scenario-Based Think-Aloud Protocol
Luger, T. M., Houston, T. K., & Suls, J. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2924 · Cited by 73
Concurrent Think-Aloud Verbalizations and Usability Problems
Fan, M., Lin, J., Chung, C., & Truong, K. N. (2019) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3325281 · Cited by 72
Self-citationLang, M. K. (2002) · it - Information Technology · doi:10.1524/itit.2002.44.1.003 · Cited by 72
Hertzum, M., Borlund, P., & Kristoffersen, K. B. (2015) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2015.1065691 · Cited by 69
Usability Testing by Older Adults of a Computer-Mediated Health Communication Program
Lin, C. A., Neafsey, P. J., & Strickler, Z. (2009) · Journal of Health Communication · doi:10.1080/10810730802659095 · Cited by 66
Elling, S., Lentz, L., & de Jong, M. (2012) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.2012.2206190 · Cited by 48
Automatic Detection of Usability Problem Encounters in Think-aloud Sessions
Fan, M., Li, Y., & Truong, K. N. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3385732 · Cited by 44
Self-citationFan, M., Wu, K., Zhao, J., Li, Y., Wei, W., & Truong, K. N. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934797 · Cited by 43
Self-citationZhao, T., & McDonald, S. (2010) · Nordic Conference on Human-Computer Interaction · doi:10.1145/1868914.1868979 · Cited by 37
Older adults' online health information seeking behavior
Huang, M., Hansen, D., & Xie, B. (2012) · Proceedings of the 2012 iConference · doi:10.1145/2132176.2132220 · Cited by 32 · cited 2×
“TAs were used to study older adults’ thought process to understand their experiences with different software tools (e.g., [6, 9, 27, 30, 32]).”
MethodAlhadreti, O., & Mayhew, P. J. (2017) · UEA Digital Repository (University of East Anglia) · Cited by 32
Chung, J., Chaudhuri, S., Le, T., Chi, N., Thompson, H. J., & Demiris, G. (2015) · Educational Gerontology · doi:10.1080/03601277.2015.1071586 · Cited by 11
Investigation of Indirect Oral Operation Method for Think Aloud Usability Testing
Hori, M., Kihara, Y., & Kato, T. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21753-1_5 · Cited by 7
Olmsted-Hawala, E. L., & Holland, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-20892-3_46 · Cited by 4
Research works printed but not linked: 4
- 10.5555/3190862.3190865 · doi:10.5555/3190862.3190865
- 10.5555/3532708.3532711 · doi:10.5555/3532708.3532711
- Erica Olmsted-Hawala and Jennifer Romano Bergstrom. 2012. Think-aloud protocols: Does age make a difference. Proceedings of Society for Technical Communication (STC) Summit, Chicago, IL (2012).
- Tingting Zhao, Sharon McDonald, and Helen M Edwards. 2014. The impact of two different think-aloud instructions in a usability test: a case of just following orders?Behaviour & Information Technology 33, 2 (2014), 163–183.
Cited by
Jin, X., & Fan, M. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3544820 · Cited by 50
Self-citationJin, X., Kuang, E., & Fan, M. (2021) · Designing Interactive Systems Conference · doi:10.1145/3461778.3462127 · Cited by 43
Self-citationCoUX: Collaborative Visual Analysis of Think-Aloud Usability Test Videos for Digital Interfaces
Soure, E. J., Kuang, E., Fan, M., & Zhao, J. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114822 · Cited by 43
Self-citationuxSense: Supporting User Experience Analysis with Visualization and Computer Vision
Batch, A., Ji, Y., Fan, M., Zhao, J., & Elmqvist, N. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3241581 · Cited by 38
Self-citationLiu, C., Su, M., Xiang, Y., Huang, Y., Yang, Y., Zhang, K., & Fan, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714228 · Cited by 36
Self-citation
Metadata
- Authors
- Fan, M., Zhao, Q., & Tibdewal, V.
- Affiliations
- Rochester Institute of Technology (US)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-13
- OpenAlex
- W3161683830
- Cited by
- 29
- References
- 40
- Citations by year
- 2026: 3 · 2025: 9 · 2024: 3 · 2023: 5 · 2022: 5 · 2021: 4
- FWCI
- 3.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Health Sciences › Medicine › Geriatrics and Gerontology › Technology Use by Older Adults
- Keywords
- speech features, older adults, usability testing, negative sentiment, speech rate, pitch, loudness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.