Complementing text entry evaluations with a composition task
The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
A common methodology for evaluating text entry methods is to ask participants to transcribe a predefined set of memorable sentences or phrases. In this article, we explore if we can complement the conventional transcription task with a more externally valid composition task. In a series of large-scale crowdsourced experiments, we found that participants could consistently and rapidly invent high quality and creative compositions with only modest reductions in entry rates. Based on our series of experiments, we provide a best-practice procedure for using composition tasks in text entry evaluations. This includes a judging protocol which can be performed either by the experimenters or by crowdsourced workers on a microtask market. We evaluated our composition task procedure using a text entry method unfamiliar to participants. Our empirical results show that the composition task can serve as a valid complementary text entry evaluation method.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The main task in all evaluation studies was a transcription task”
From A Review of Design and Evaluation Practices in Mobile Text Entry for Visually Impaired and Blind Persons (2023) · cites this paper as [64]
In the citing paper
“Delving into the details of how the lab studies were performed, we noted that all involved a form of transcription tasks, which have been termed as the de facto method for text entry studies [64,97], even though its validity has been criticized previously and alternatives have been proposed, such as the image description task or the composition task [64,98,99].”
From A Review of Design and Evaluation Practices in Mobile Text Entry for Visually Impaired and Blind Persons (2023) · cites this paper as [64]
Citations
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
The Enron Corpus: A New Dataset for Email Classification Research
Klimt, B., & Yang, Y. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30115-8_22 · Cited by 1,157
Phrase sets for evaluating text entry techniques
MacKenzie, I. S., & Soukoreff, R. W. (2003) · CHI '03 extended abstracts on Human factors in computing systems - CHI '03 · doi:10.1145/765891.765971 · Cited by 825
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
Heer, J., & Bostock, M. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753357 · Cited by 816
Text Entry for Mobile Computing: Models and Methods, Theory and Practice
MacKenzie, I. S., & Soukoreff, R. W. (2002) · Human-Computer Interaction · doi:10.1207/s15327051hci172&3_2 · Cited by 571
The design and evaluation of a high-performance soft keyboard
MacKenzie, I. S., & Zhang, S. X. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.302983 · Cited by 422
Metrics for text entry research: an evaluation of MSD and KSPC, and a new unified error metric
Soukoreff, R. W., & MacKenzie, I. S. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642632 · Cited by 398
Fast hands-free writing by gaze direction
Ward, D. J., & MacKay, D. J. C. (2002) · Nature · doi:10.1038/418838a · Cited by 329
Patterns of entry and correction in large vocabulary continuous speech recognition systems
Karat, C.-M., Halverson, C., Horn, D., & Karat, J. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303160 · Cited by 275
Fast Hands-free Writing by Gaze Direction
Ward, D. J., & Mackay, D. (2002) · PubMed · doi:10.48550/arxiv.cs/0204030 · Cited by 214
A versatile dataset for text entry evaluations based on genuine mobile emails
Vertanen, K., & Kristensson, P. O. (2011) · Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services · doi:10.1145/2037373.2037418 · Cited by 177
Self-citationThe potential of dwell-free eye-typing for fast assistive gaze communication
Kristensson, P. O., & Vertanen, K. (2012) · Proceedings of the Symposium on Eye Tracking Research and Applications · doi:10.1145/2168556.2168605 · Cited by 142
Self-citationAn empirical study of typing rates on mini-QWERTY keyboards
Clarkson, E., Clawson, J., Lyons, K., & Starner, T. (2005) · CHI '05 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1056808.1056898 · Cited by 136
Performance comparisons of phrase sets and presentation styles for text entry evaluations
Kristensson, P. O., & Vertanen, K. (2012) · Proceedings of the 2012 ACM international conference on Intelligent User Interfaces · doi:10.1145/2166966.2166972 · Cited by 102
Self-citationParakeet: a continuous speech recognition system for mobile touch-screen devices
Vertanen, K., & Kristensson, P. O. (2009) · Proceedings of the 14th international conference on Intelligent user interfaces · doi:10.1145/1502650.1502685 · Cited by 102
Self-citationDevice independent text input: a rationale and an example
Isokoski, P., & Raisamo, R. (2000) · Proceedings of the working conference on Advanced visual interfaces · doi:10.1145/345513.345262 · Cited by 88
Human–computer interaction: A stable discipline, a nascent science, and the growth of the long tail
Dix, A. (2010) · Interacting with Computers · doi:10.1016/j.intcom.2009.11.007 · Cited by 86
Vertanen, K., & Kristensson, P. O. (2011) · Cambridge University Engineering Department Publications Database · Cited by 80
Self-citationIn search of effective text input interfaces for off the desktop computing
Zhai, S., Kristensson, P.-O., & Smith, B. A. (2005) · Interacting with Computers · doi:10.1016/j.intcom.2003.12.007 · Cited by 76
Self-citationFive Challenges for Intelligent Text Entry Methods
Kristensson, P. O., & Jameson, A. (2009) · AI Magazine · doi:10.1609/aimag.v30i4.2269 · Cited by 75
Self-citationDiscrete and Continuous Shape Writing for Text Entry and Control
Kristensson, P. O. (2007) · DiVA (Linkoping University) · Cited by 72
Self-citationGraffiti vs. unistrokes: an empirical comparison
Castellucci, S. J., & MacKenzie, I. S. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357106 · Cited by 69
Wobbrock, J. O., Chau, D. H., & Myers, B. A. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240728 · Cited by 50
Effect of foreign language on text transcription performance: Finns writing English
Isokoski, P., & Linden, T. (2004) · Nordic Conference on Human-Computer Interaction · doi:10.1145/1028014.1028032 · Cited by 34
Experimental Evaluations of the Twiddler One-Handed Chording Mobile Keyboard
Lyons, K., Starner, T., & Gane, B. (2006) · Human–Computer Interaction · doi:10.1207/s15327051hci2104_1 · Cited by 34
Sampling representative phrase sets for text entry experiments: a procedure and public resource
Paek, T., & Hsu, B.-J. (. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979304 · Cited by 30
Children's phrase set for text input method evaluations
Kano, A., Read, J. C., & Dix, A. (2006) · Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles · doi:10.1145/1182475.1182534 · Cited by 29
Text entry performance of state of the art unconstrained handwriting recognition
Kristensson, P. O., & Denby, L. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518788 · Cited by 26
Self-citationFast and Flexible Selection with a Single Switch
Broderick, T., & Mackay, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0007481 · Cited by 23
Pickup Usability Dominates: A Brief History of Mobile Text Entry Research and Adoption
Dunlop, M. D., & Masters, M. M. (2009) · International Journal of Mobile Human Computer Interaction · doi:10.4018/jmhci.2009010103 · Cited by 19
Tagliavia, F. M. (2013) · Re-visiones · Cited by 1
Research works printed but not linked: 2
- Vertanen Keith, 2011, Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing. ACL, 700--711
- 10.5555/646079 · doi:10.5555/646079
Cited by
How do People Type on Mobile Devices?: Observations from a Study with 37,000 Volunteers
Palin, K., Feit, A. M., Kim, S., Kristensson, P. O., & Oulasvirta, A. (2019) · Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services · doi:10.1145/3338286.3340120 · Cited by 190
“It is also possible to use alternative methods, such as a composition task [38] or even object-based methods, such as instructing users to annotate an image [12].”
MethodSelf-citationPerformance Envelopes of Virtual Keyboard Text Input Strategies in Virtual Reality
Dudley, J. J., Benko, H., Wigdor, D., & Kristensson, P. O. (2019) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar.2019.00027 · Cited by 149
“Vertanen and Kristensson [30] found that entry rates dropped by between approximately 15 and 35% depending on the nature of the composition task.”
BackgroundSelf-citationReyal, S., Zhai, S., & Kristensson, P. O. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702597 · Cited by 135
“Complementary data might be gained by also using a composition task [23], which is able to more accurately model actual text entry activities.”
MethodSelf-citationVertanen, K., Memmi, H., Emge, J., Reyal, S., & Kristensson, P. O. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702135 · Cited by 133
“It is also possible to use composition in text entry studies [20].”
Self-citationBuschek, D., Zürn, M., & Eiband, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445372 · Cited by 108
Metadata
- Authors
- Vertanen, K., & Kristensson, P. O.
- Affiliations
- Montana Technological University (US)
- University of St Andrews (GB)
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2014-02
- Volume
- 21
- Issue
- 2
- Pages
- 1-33
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W2054689358
- Cited by
- 62
- References
- 31
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 6 · 2023: 7 · 2022: 4 · 2021: 6 · 2020: 6 · 2019: 4 · 2018: 4 · 2017: 5 · 2016: 7 · 2015: 3 · 2014: 5
- FWCI
- 12.92
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/H027408/1)
- Scottish Informatics and Computer Science Alliance
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Physical Sciences › Computer Science › Information Systems › User Authentication and Security Systems
- Keywords
- generalizability, crowdsourced experiments, microtasks, entry rate
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.