Word Clarity as a Metric in Sampling Keyboard Test Sets
The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
Test sets play an essential role in evaluating text entry techniques. In this paper, we argue that in addition to the widely adopted metric of bigram representativeness and memorability, word clarity should also be considered as a metric when creating test sets from the target dataset. Word clarity quantifies the extent to which a word is likely to confuse with other words on a keyboard. We formally define word clarity, derive equations calculating it, and both theoretically and empirically show that word clarity has a significant effect on text entry performance: it can yield up to 26.4% difference in error rate, and 25% difference in input speed. We later propose a Pareto optimization method for sampling test sets with different sizes, which optimizes the word clarity and bigram representativeness, and memorability of the test set. The obtained test sets are published on the Internet.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In one evaluation study, the dataset proposed in [66] was used. One evaluation study selected words from the Open American National Corpus (ANC), and one evaluation study used Pinyin, a phonetic spelling system in Roman characters for inputting Chinese characters. Finally, one evaluation study did not provide details about the used phrase set.”
From A Review of Design and Evaluation Practices in Mobile Text Entry for Visually Impaired and Blind Persons (2023) · cites this paper as [66]
Citations
Cites
The Kolmogorov-Smirnov Test for Goodness of Fit
Massey, F. J. (1951) · Journal of the American Statistical Association · doi:10.1080/01621459.1951.10500769 · Cited by 7,210
The information capacity of the human motor system in controlling the amplitude of movement.
Fitts, P. M. (1954) · Journal of Experimental Psychology · doi:10.1037/h0055392 · Cited by 6,466 · cited 1×
“Researchers usually calculate the movement efficiency (MT ) as the sum of Fitts’ law [12] movement time (Ti j) between all pairs of letters (bigrams) weighted by bigram probabilities Pi j calculated from a language corpus:”
BackgroundThe information capacity of the human motor system in controlling the amplitude of movement.
Fitts, P. M. (1992) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.121.3.262 · Cited by 6,278
The Kolmogorov-Smirnov Test for Goodness of Fit
Massey, F. J. (1951) · Journal of the American Statistical Association · doi:10.2307/2280095 · Cited by 3,690 · cited 1×
“clarity of Ŝ and that of U based on the Kolmogorov-Smirnov test (K-S test) [28].”
MethodPhrase sets for evaluating text entry techniques
MacKenzie, I. S., & Soukoreff, R. W. (2003) · CHI '03 extended abstracts on Human factors in computing systems - CHI '03 · doi:10.1145/765891.765971 · Cited by 825 · cited 10×
“Optimization Metrics When sampling phrase sets for text entry experiments, memorability and representativeness are two metrics that have been widely adopted by researchers [23, 25, 33, 39].”
BackgroundMethodInfluentialKlimt, B., & Yang, Y. (2004) · Conference on Email and Anti-Spam · Cited by 575
Metrics for text entry research: an evaluation of MSD and KSPC, and a new unified error metric
Soukoreff, R. W., & MacKenzie, I. S. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642632 · Cited by 398 · cited 2×
“Consistent with existing works, subjects tended to fix most of the errors and left few in the final submitted string [24, 38, 42].”
MethodResultFat Finger Worries: How Older and Younger Users Physically Interact with PDAs
Siek, K. A., Rogers, Y., & Connelly, K. H. (2005) · Lecture Notes in Computer Science · doi:10.1007/11555261_24 · Cited by 337 · cited 1×
“INTRODUCTION Due to the “fat-finger problem” [35] and lack of tactile feedback, text entry has long been a challenge for touchscreen devices [2].”
BackgroundPredicting text entry speed on mobile phones
Silfverberg, M., MacKenzie, I. S., & Korhonen, P. (2000) · Proceedings of the SIGCHI conference on Human Factors in Computing Systems · doi:10.1145/332040.332044 · Cited by 305 · cited 1×
“Meanwhile, bigram probability has been extensively used for both predicting the text rate [8, 10, 26, 36] and for designing representative phrase sets [31, 39].”
MethodPatterns of entry and correction in large vocabulary continuous speech recognition systems
Karat, C.-M., Halverson, C., Horn, D., & Karat, J. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303160 · Cited by 275 · cited 1×
“western novel [18], Linux operating system [17], and news [43]).”
Background
Research works printed but not linked: 1
- Paul Nation and Robert Waring . 1997. Vocabulary size, text coverage and word lists. Vocabulary: Description, acquisition and pedagogy 14 ( 1997 ), 6--19. Paul Nation and Robert Waring. 1997. Vocabulary size, text coverage and word lists. Vocabulary: Description, acquisition and pedagogy 14 (1997), 6--19.
Cited by
Observations on Typing from 136 Million Keystrokes
Dhakal, V., Feit, A. M., Kristensson, P. O., & Oulasvirta, A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174220 · Cited by 226
“[42] proposed word clarity as a metric for sampling test sets for touch screen keyboards.”
BackgroundWrisText: One-handed Text Entry on Smartwatch using Wrist Gestures
Gong, J., Xu, Z., Guo, Q., Seyed, T., Chen, X. '., Bi, X., & Yang, X.-D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173755 · Cited by 91
“In each session, participants typed 18 phrases with each hand posture, the same across all participants, randomly picked from [60].”
Self-citationBiTipText: Bimanual Eyes-Free Text Entry on a Fingertip Keyboard
Xu, Z., Chen, W., Zhao, D., Luo, J., Wu, T.-Y., Gong, J., Yin, S., Zhai, J., & Yang, X.-D. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376306 · Cited by 52
“We took a simplified approach similar to the one used in previous work for word clarity on a soft keyboard [39], where 𝑥 and 𝑦 coordinates were assumed to be independent to each other, 𝑠 (cid:3036) and the center of the touch point distribution were also assumed to be in the center of the target…”
MethodVelociWatch: Designing and Evaluating a Virtual Keyboard for the Input of Challenging Text
Vertanen, K., Gaines, D., Fletcher, C., Stanage, A. M., Watling, R., & Kristensson, P. O. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300821 · Cited by 42
“Yi et al. [32] proposed a word clarity metric based on geometric pattern matching [16] for sampling phrases containing words that might be confused due to their geometric proximity.”
BackgroundThe Impact of Word, Multiple Word, and Sentence Input on Virtual Keyboard Decoding Performance
Vertanen, K., Fletcher, C., Gaines, D., Gould, J., & Kristensson, P. O. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174200 · Cited by 39
“Later research has investigated further ways of sampling representative phrases based on criteria such as word clarity [36].”
BackgroundLiang, C., Hsia, C., Yu, C., Yan, Y., Wang, Y., & Shi, Y. (2022) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3569463 · Cited by 36
“Yi et al. [55, 56] investigated the touch point distribution and showed the feasibility to apply the ordinary QWERTY keyboard on a smartwatch touchscreen.”
BackgroundSelf-citationGlanceWriter: Writing Text by Glancing Over Letters with Gaze
Cui, W., Liu, R., Li, Z., Wang, Y., Wang, A., Zhao, X., Rashidian, S., Baig, F., Ramakrishnan, I. V., Wang, F., et al. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581269 · Cited by 41
“We randomly selected 20 phrases from a subset of the MacKenzie and Soukoref phrase set [31, 52].”
MethodSelf-citationi'sFree: Eyes-Free Gesture Typing via a Touch-Enabled Remote Control
Zhu, S., Zheng, J., Zhai, S., & Bi, X. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300678 · Cited by 40
“The phrases were selected from a subset of the MacKenzie and Soukoreff phrase set [15, 29].”
MethodSelf-citationOptimal-T9: An Optimized T9-like Keyboard for Small Touchscreen Devices
Qin, R., Zhu, S., Lin, Y., Ko, Y.-J., & Bi, X. (2018) · ACM International Conference on Interactive Surfaces and Spaces (ISS) · doi:10.1145/3279778.3279786 · Cited by 27
“The phrases were randomly selected from a subset of the Mackenzie and Soukoreff phrase set [27], using the selection procedure proposed by [36].”
MethodSelf-citationAccessible Gesture Typing for Non-Visual Text Entry on Smartphones
Billah, S. M., Ko, Y.-J., Ashok, V., Bi, X., & Ramakrishnan, IV (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300606 · Cited by 21
“These phrases were randomly selected from the T20 phrase set specified in [28].”
MethodSelf-citationBackSwipe: Back-of-device Word-Gesture Interaction on Smartphones
Cui, W., Zhu, S., Li, Z., Xu, Z., Yang, X.-D., Ramakrishnan, I. V., & Bi, X. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445081 · Cited by 20
“The testing phrases were also selected from a subset of the MacKenzie and Soukoreff phrase set [39, 57].”
MethodSelf-citationKomninos, A., Stefanis, V., & Garofalakis, J. (2023) · Multimodal Technologies and Interaction · doi:10.3390/mti7020022 · Cited by 16
Shi, W., Yu, C., Fan, S., Wang, F., Wang, T., Yi, X., Bi, X., & Shi, Y. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300747 · Cited by 16
“The phrase set for English input was T-40 set from [41], phrases for Chinese input were those with suitable length parsed from [13].”
MethodSelf-citationMultilingual phrase sampling for text entry evaluations
Franco-Salvador, M., & Leiva, L. A. (2018) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2018.01.006 · Cited by 15
“But evaluations focused e.g. on novel soft keyboards could include more design dimensions, such as word clarity [67] or the keyboard layout.”
BackgroundPalmBoard: Leveraging Implicit Touch Pressure in Statistical Decoding for Indirect Text Entry
Yi, X., Wang, C., Bi, X., & Shi, Y. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376441 · Cited by 15
Self-citationLearning to type with mobile keyboards: Findings with a randomized keyboard
Jiang, X., Jokinen, J., Oulasvirta, A., & Ren, X. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106992 · Cited by 18
“…or two fingers) (Azenkot & Zhai, 2012, pp. 251 – 260; Buschek et al., 2018; Jiang et al., 2020; Nicolau & Jorge, 2012), the size of the device (Yi et al., 2017) and the virtual keys (Kim et al., 2013), and the conditions of typing tasks (copy typing or memorized typing) (Varcholik et al.,…”
BackgroundHiPad: Text entry for Head-Mounted Displays Using Circular Touchpad
Jiang, H., & Weng, D. (2020) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr46266.2020.1581236395562 · Cited by 69
“The distribution of letters of HiPad can be optimized by combining the word clarity [17, 58] with speed predicted by the Fitts-Digraph model [59, 60] in the future.”
MethodTowards Flexible and Robust User Interface Adaptations With Multiple Objectives
Johns, C. A., Belo, J. M. E., Feit, A. M., Klokmose, C. N., & Pfeuffer, K. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586183.3606799 · Cited by 19
“[5] A priori* Offline Text entry [6] A priori* Offline Text entry [14] A priori* Offline Text entry [24] A priori* Offline Text entry [40] A priori* Offline Text entry [44] A priori* Offline Text entry [45] A priori* Offline Text entry [50] A priori* Offline Text entry”
MethodWhy Is Gesture Typing Promising for Older Adults?
Lin, Y., Zhu, S., Ko, Y.-J., Cui, W., & Bi, X. (2018) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3234695.3236350 · Cited by 11
Self-citationAccessible Gesture Typing on Smartphones for People with Low Vision
Zhang, D., Li, Z., Ashok, V. G., Seiple, W., Ramakrishnan, I. V., & Bi, X. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676447 · Cited by 8
Self-citationTypeBoard: Identifying Unintentional Touch on Pressure-Sensitive Touchscreen Keyboards
Gu, Y., Yu, C., Chen, X., Li, Z., & Shi, Y. (2021) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3472749.3474770 · Cited by 5
“We did not include transcription as a task as other text entry studies do [38, 46, 54].”
BackgroundMethodSelf-citationEnhanced Touchable Projector-depth System with Deep Hand Pose EstimationPreprint
Chai, Z., & Shilkrot, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.11090 · Cited by 3
“The same 10 volunteers were used to evaluate the keyboard by typing 20 samples from a small but representative dataset of English [22].”
MethodLLM Powered Text Entry Decoding and Flexible Typing on Smartphones
Ma, Y., Zhang, D., Ramakrishnan, I. V., & Bi, X. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714314 · Cited by 1
Self-citationEnabling Auto-Correction on Soft Braille Keyboard
Zhang, D., Ma, Y., Dausch, G., Seiple, W., Gu, D., Ramakrishnan, I. V., & Bi, X. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747699 · Cited by 1
Self-citationGenerating Representative Phrase Sets for Text Entry Experiments by GA-Based Text Corpora Sampling
Ljubić, S., & Salkanović, A. (2023) · Mathematics · doi:10.3390/math11112550 · Cited by 1
“[19] MacKenzie and Soukoreff Combined analysis of word clarity, digram frequency, and memorability Relative entropy between the target phrase set and the source corpus (KLD) Four sets with 20, 40, 80, and 160 phrases; see [19] for details”
BackgroundMethodA Systematic Review and Meta-analysis of Text Entry Studies
Iarygina, O., & Mottelson, A. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720117 · Cited by 6
Metadata
- Authors
- Yi, X., Yu, C., Shi, W., Bi, X., & Shi, Y.
- Affiliations
- Tsinghua University (CN)
- Stony Brook University (US)
- Venue
- Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2017-05-02
- Pages
- 4216-4228
- OpenAlex
- W2611531483
- Cited by
- 26
- References
- 40
- Citations by year
- 2025: 3 · 2024: 1 · 2023: 4 · 2022: 1 · 2021: 3 · 2020: 3 · 2019: 4 · 2018: 7
- FWCI
- 2.89
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- Natural Science Foundation of China (61272230,61572276)
- National Key Research and Development Plan (2016YFB1001200)
- Tsinghua University Research Funding (20151080408)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › User Authentication and Security Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Keywords
- text entry, memorability, input speed, multi-objective optimization, word confusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.