Word Clarity as a Metric in Sampling Keyboard Test Sets
The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
Test sets play an essential role in evaluating text entry techniques. In this paper, we argue that in addition to the widely adopted metric of bigram representativeness and memorability, word clarity should also be considered as a metric when creating test sets from the target dataset. Word clarity quantifies the extent to which a word is likely to confuse with other words on a keyboard. We formally define word clarity, derive equations calculating it, and both theoretically and empirically show that word clarity has a significant effect on text entry performance: it can yield up to 26.4% difference in error rate, and 25% difference in input speed. We later propose a Pareto optimization method for sampling test sets with different sizes, which optimizes the word clarity and bigram representativeness, and memorability of the test set. The obtained test sets are published on the Internet.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In one evaluation study, the dataset proposed in [66] was used. One evaluation study selected words from the Open American National Corpus (ANC), and one evaluation study used Pinyin, a phonetic spelling system in Roman characters for inputting Chinese characters. Finally, one evaluation study did not provide details about the used phrase set.”
From A Review of Design and Evaluation Practices in Mobile Text Entry for Visually Impaired and Blind Persons (2023) · cites this paper as [66]
Citations
Cites
The Kolmogorov-Smirnov Test for Goodness of Fit
Massey, F. J. (1951) · Journal of the American Statistical Association · doi:10.1080/01621459.1951.10500769 · Cited by 7,210
The information capacity of the human motor system in controlling the amplitude of movement.
Fitts, P. M. (1954) · Journal of Experimental Psychology · doi:10.1037/h0055392 · Cited by 6,466 · cited 1×
“Researchers usually calculate the movement efficiency (MT ) as the sum of Fitts’ law [12] movement time (Ti j) between all pairs of letters (bigrams) weighted by bigram probabilities Pi j calculated from a language corpus:”
BackgroundThe information capacity of the human motor system in controlling the amplitude of movement.
Fitts, P. M. (1992) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.121.3.262 · Cited by 6,278
The Kolmogorov-Smirnov Test for Goodness of Fit
Massey, F. J. (1951) · Journal of the American Statistical Association · doi:10.2307/2280095 · Cited by 3,690 · cited 1×
“clarity of Ŝ and that of U based on the Kolmogorov-Smirnov test (K-S test) [28].”
MethodPhrase sets for evaluating text entry techniques
MacKenzie, I. S., & Soukoreff, R. W. (2003) · CHI '03 extended abstracts on Human factors in computing systems - CHI '03 · doi:10.1145/765891.765971 · Cited by 825 · cited 10×
“Optimization Metrics When sampling phrase sets for text entry experiments, memorability and representativeness are two metrics that have been widely adopted by researchers [23, 25, 33, 39].”
BackgroundMethodInfluentialKlimt, B., & Yang, Y. (2004) · Conference on Email and Anti-Spam · Cited by 575
Metrics for text entry research: an evaluation of MSD and KSPC, and a new unified error metric
Soukoreff, R. W., & MacKenzie, I. S. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642632 · Cited by 398 · cited 2×
“Consistent with existing works, subjects tended to fix most of the errors and left few in the final submitted string [24, 38, 42].”
MethodResultFat Finger Worries: How Older and Younger Users Physically Interact with PDAs
Siek, K. A., Rogers, Y., & Connelly, K. H. (2005) · Lecture Notes in Computer Science · doi:10.1007/11555261_24 · Cited by 337 · cited 1×
“INTRODUCTION Due to the “fat-finger problem” [35] and lack of tactile feedback, text entry has long been a challenge for touchscreen devices [2].”
BackgroundPredicting text entry speed on mobile phones
Silfverberg, M., MacKenzie, I. S., & Korhonen, P. (2000) · Proceedings of the SIGCHI conference on Human Factors in Computing Systems · doi:10.1145/332040.332044 · Cited by 305 · cited 1×
“Meanwhile, bigram probability has been extensively used for both predicting the text rate [8, 10, 26, 36] and for designing representative phrase sets [31, 39].”
MethodPatterns of entry and correction in large vocabulary continuous speech recognition systems
Karat, C.-M., Halverson, C., Horn, D., & Karat, J. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303160 · Cited by 275 · cited 1×
“western novel [18], Linux operating system [17], and news [43]).”
BackgroundFFitts law: modeling finger touch with fitts' law
Bi, X., Li, Y., & Zhai, S. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466180 · Cited by 235
Self-citationText entry using soft keyboards
Mackenzie, I. S., Zhang, S. X., & Soukoreff, R. W. (1999) · Behaviour & Information Technology · doi:10.1080/014492999118995 · Cited by 218 · cited 1×
“Meanwhile, bigram probability has been extensively used for both predicting the text rate [8, 10, 26, 36] and for designing representative phrase sets [31, 39].”
MethodTouch behavior with different postures on soft smartphone keyboards
Azenkot, S., & Zhai, S. (2012) · Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services · doi:10.1145/2371574.2371612 · Cited by 214 · cited 3×
“Generally, this speed is consistent with the index-finger text entry speed on smart phones [4].”
BackgroundGoel, M., Findlater, L., & Wobbrock, J. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208662 · Cited by 203 · cited 2×
“tablet [11, 21], phone [14, 41], and smartwatch [40]).”
MethodAnalyzing the input stream for character- level errors in unconstrained text entry evaluations
Wobbrock, J. O., & Myers, B. A. (2006) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1188816.1188819 · Cited by 200 · cited 1×
“Consistent with existing works, subjects tended to fix most of the errors and left few in the final submitted string [24, 38, 42].”
ResultA versatile dataset for text entry evaluations based on genuine mobile emails
Vertanen, K., & Kristensson, P. O. (2011) · Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services · doi:10.1145/2037373.2037418 · Cited by 177 · cited 9×
“Optimization Metrics When sampling phrase sets for text entry experiments, memorability and representativeness are two metrics that have been widely adopted by researchers [23, 25, 33, 39].”
BackgroundMethodResultInfluentialLanguage modeling for soft keyboards
Goodman, J., Venolia, G., Steury, K., & Parker, C. (2002) · Proceedings of the 7th international conference on Intelligent user interfaces · doi:10.1145/502716.502753 · Cited by 161 · cited 1×
“[16], which calculates the likelihood of a word W within a pre-defined dictionary given an input sequence I as:”
BackgroundVulture: a mid-air word-gesture keyboard
Markussen, A., Jakobsen, M. R., & Hornbæk, K. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2556964 · Cited by 156 · cited 1×
“smart gesture keyboard [27, 32] and pointing on large wall display [34]).”
BackgroundA character-level error analysis technique for evaluating text entry methods
MacKenzie, I. S., & Soukoreff, R. W. (2002) · Proceedings of the second Nordic conference on Human-computer interaction · doi:10.1145/572020.572056 · Cited by 150 · cited 1×
“Consistent with existing works, subjects tended to fix most of the errors and left few in the final submitted string [24, 38, 42].”
ResultReyal, S., Zhai, S., & Kristensson, P. O. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702597 · Cited by 135 · cited 1×
“smart gesture keyboard [27, 32] and pointing on large wall display [34]).”
BackgroundVertanen, K., Memmi, H., Emge, J., Reyal, S., & Kristensson, P. O. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702135 · Cited by 133 · cited 3×
“proposed VelociTap [40], a decoder that incorporates a probabilistic keyboard model, a character language model and a word language model.”
BackgroundEffects of Language Modeling and its Personalization on Touchscreen Typing Performance
Fowler, A., Partridge, K., Chelba, C., Bi, X., Ouyang, T., & Zhai, S. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702503 · Cited by 127 · cited 4×
“We modelled the distribution of touch location using 2D Gaussian distribution [13].”
BackgroundMethodInfluentialSelf-citationQuasi-qwerty soft keyboard optimization
Bi, X., Smith, B. A., & Zhai, S. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753367 · Cited by 109 · cited 1×
“Meanwhile, bigram probability has been extensively used for both predicting the text rate [8, 10, 26, 36] and for designing representative phrase sets [31, 39].”
MethodSelf-citationPersonalized input: improving ten-finger touchscreen typing through automatic adaptation
Findlater, L., & Wobbrock, J. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208520 · Cited by 106 · cited 2×
“[11] adapted the underlying key-press classification models to improve the ten-finger typing performance.”
MethodPerformance comparisons of phrase sets and presentation styles for text entry evaluations
Kristensson, P. O., & Vertanen, K. (2012) · Proceedings of the 2012 ACM international conference on Intelligent User Interfaces · doi:10.1145/2166966.2166972 · Cited by 102 · cited 2×
“The tradeoff between internal validity and external validity in text entry experiments has been extensively discussed by many researchers [20, 23, 25, 33, 39].”
BackgroundDunlop, M., & Levine, J. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208659 · Cited by 99 · cited 5×
“to optimize both keyboard layouts [10, 37] and keyboard algorithms [7].”
BackgroundMethodInfluentialText Entry on Tiny QWERTY Soft Keyboards
Leiva, L. A., Sahami, A., Catala, A., Henze, N., & Schmidt, A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702388 · Cited by 96 · cited 1×
“These sizes approximate the size of keyboards on smartphones [4, 6] and smartwatches [22].”
BackgroundContextType: using hand posture information to improve mobile touch screen text entry
Goel, M., Jansen, A., Mandel, T., Patel, S. N., & Wobbrock, J. O. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2481386 · Cited by 96 · cited 2×
“proposed ContextType [15], which combines users’ posture-specific touch pattern information with a language model to classify users’ touch events as pressed keys.”
BackgroundRelaxing stylus typing precision by geometric pattern matching
Kristensson, P.-O., & Zhai, S. (2005) · Proceedings of the 10th international conference on Intelligent user interfaces · doi:10.1145/1040830.1040867 · Cited by 88 · cited 6×
“In literature, the concept of word clarity has been noticed by some researchers [10, 21, 37].”
BackgroundMethodInfluentialDevice independent text input: a rationale and an example
Isokoski, P., & Raisamo, R. (2000) · Proceedings of the working conference on Advanced visual interfaces · doi:10.1145/345513.345262 · Cited by 88 · cited 1×
“western novel [18], Linux operating system [17], and news [43]).”
BackgroundUncertain text entry on mobile devices
Weir, D., Pohl, H., Rogers, S., Vertanen, K., & Kristensson, P. O. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557412 · Cited by 85 · cited 2×
“[41] combined pressure information and Gaussian Progress regression within a probabilistic decoder.”
BackgroundMovement model, hits distribution and learning in virtual keyboarding
Zhai, S., Sue, A., & Accot, J. (2002) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/503376.503381 · Cited by 77 · cited 1×
“western novel [18], Linux operating system [17], and news [43]).”
BackgroundBoth complete and correct?: multi-objective optimization of touchscreen keyboard
Bi, X., Ouyang, T., & Zhai, S. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557414 · Cited by 59 · cited 2×
“to optimize both keyboard layouts [10, 37] and keyboard algorithms [7].”
MethodSelf-citationMid-air text input techniques for very large wall displays
Shoemaker, G. B. D., Findlater, L., Dawson, J. Q., & Booth, K. S. (2009) · Cited by 52
Touch typing using thumbs: understanding the effect of mobility and hand posture
Nicolau, H., & Jorge, J. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208661 · Cited by 50 · cited 1×
“There are two reasons for this: 1) Literature [9, 30] shows that the frequencies of insertion/omission errors are only 1% when entering text on phones.”
BackgroundOptimizing Touchscreen Keyboards for Gesture Typing
Smith, B. A., Bi, X., & Zhai, S. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702357 · Cited by 48 · cited 10×
“[37] defined gesture clarity as the pairwise Manhattan distance between sampled points on each gesture.”
BackgroundMethodResultInfluentialSelf-citationTexting while walking: an evaluation of mini-qwerty text input while on-the-go
Clawson, J., Starner, T. E., Kohlsdorf, D., Quigley, D. P., & Gilliland, S. M. (2014) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/2628363.2628408 · Cited by 38
Octopus: evaluating touchscreen keyboard correction and recognition algorithms via
Bi, X., Azenkot, S., Partridge, K., & Zhai, S. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470732 · Cited by 36
Self-citationSampling representative phrase sets for text entry experiments: a procedure and public resource
Paek, T., & Hsu, B.-J. (. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979304 · Cited by 30 · cited 9×
“[31], and calculate the representativeness as:”
BackgroundMethodInfluentialLeiva, L. A., & Sanchis-Trilles, G. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557024 · Cited by 16
A systematic comparison of 3 phrase sampling methods for text entry experiments in 10 languages
Sanchis-Trilles, G., & Leiva, L. A. (2014) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/2628363.2634229 · Cited by 7
Research works printed but not linked: 1
- Paul Nation and Robert Waring . 1997. Vocabulary size, text coverage and word lists. Vocabulary: Description, acquisition and pedagogy 14 ( 1997 ), 6--19. Paul Nation and Robert Waring. 1997. Vocabulary size, text coverage and word lists. Vocabulary: Description, acquisition and pedagogy 14 (1997), 6--19.
Cited by
Observations on Typing from 136 Million Keystrokes
Dhakal, V., Feit, A. M., Kristensson, P. O., & Oulasvirta, A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174220 · Cited by 226
“[42] proposed word clarity as a metric for sampling test sets for touch screen keyboards.”
BackgroundWrisText: One-handed Text Entry on Smartwatch using Wrist Gestures
Gong, J., Xu, Z., Guo, Q., Seyed, T., Chen, X. '., Bi, X., & Yang, X.-D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173755 · Cited by 91
“In each session, participants typed 18 phrases with each hand posture, the same across all participants, randomly picked from [60].”
Self-citationBiTipText: Bimanual Eyes-Free Text Entry on a Fingertip Keyboard
Xu, Z., Chen, W., Zhao, D., Luo, J., Wu, T.-Y., Gong, J., Yin, S., Zhai, J., & Yang, X.-D. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376306 · Cited by 52
“We took a simplified approach similar to the one used in previous work for word clarity on a soft keyboard [39], where 𝑥 and 𝑦 coordinates were assumed to be independent to each other, 𝑠 (cid:3036) and the center of the touch point distribution were also assumed to be in the center of the target…”
MethodVelociWatch: Designing and Evaluating a Virtual Keyboard for the Input of Challenging Text
Vertanen, K., Gaines, D., Fletcher, C., Stanage, A. M., Watling, R., & Kristensson, P. O. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300821 · Cited by 42
“Yi et al. [32] proposed a word clarity metric based on geometric pattern matching [16] for sampling phrases containing words that might be confused due to their geometric proximity.”
BackgroundThe Impact of Word, Multiple Word, and Sentence Input on Virtual Keyboard Decoding Performance
Vertanen, K., Fletcher, C., Gaines, D., Gould, J., & Kristensson, P. O. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174200 · Cited by 39
“Later research has investigated further ways of sampling representative phrases based on criteria such as word clarity [36].”
Background
Metadata
- Authors
- Yi, X., Yu, C., Shi, W., Bi, X., & Shi, Y.
- Affiliations
- Tsinghua University (CN)
- Stony Brook University (US)
- Venue
- Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2017-05-02
- Pages
- 4216-4228
- OpenAlex
- W2611531483
- Cited by
- 26
- References
- 40
- Citations by year
- 2025: 3 · 2024: 1 · 2023: 4 · 2022: 1 · 2021: 3 · 2020: 3 · 2019: 4 · 2018: 7
- FWCI
- 2.89
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- Natural Science Foundation of China (61272230,61572276)
- National Key Research and Development Plan (2016YFB1001200)
- Tsinghua University Research Funding (20151080408)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › User Authentication and Security Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Keywords
- text entry, memorability, input speed, multi-objective optimization, word confusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.