Automatically identifying targets users interact with during real world tasks
The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Information about the location and size of the targets that users interact with in real world settings can enable new innovations in human performance assessment and soft-ware usability analysis. Accessibility APIs provide some information about the size and location of targets. How-ever this information is incomplete because it does not sup-port all targets found in modern interfaces and the reported sizes can be inaccurate. These accessibility APIs access the size and location of targets through low-level hooks to the operating system or an application. We have developed an alternative solution for target identification that leverages visual affordances in the interface, and the visual cues produced as users interact with targets. We have used our novel target identification technique in a hybrid solution that combines machine learning, computer vision, and accessibility API data to find the size and location of targets users select with 89% accuracy. Our hybrid approach is superior to the performance of the accessibility API alone: in our dataset of 1355 targets covering 8 popular applications, only 74% of the targets were correctly identified by the API alone.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Data Mining: Practical Machine Learning Tools and Techniques
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,437
(1992) · Cited by 23,431
Witten, I. H. (2008) · Cited by 9,143
The information capacity of the human motor system in controlling the amplitude of movement.
Fitts, P. M. (1954) · Journal of Experimental Psychology · doi:10.1037/h0055392 · Cited by 6,466
The information capacity of the human motor system in controlling the amplitude of movement.
Fitts, P. M. (1992) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.121.3.262 · Cited by 6,278
Computer Vision: A Modern Approach
Forsyth, D. A., & Ponce, J. (2002) · Cited by 3,739
International Journal of Human-Computer Studies
Sdol, E. (2009) · Cited by 1,095
Accuracy measures for evaluating computer pointing devices
MacKenzie, I. S., Kauppinen, T., & Silfverberg, M. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365028 · Cited by 448
More than dotting the i's --- foundations for crossing-based interfaces
Accot, J., & Zhai, S. (2002) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/503376.503390 · Cited by 372
Sikuli: using GUI screenshots for search and automation
Yeh, T., Chang, T.-H., & Miller, R. C. (2009) · Proceedings of the 22nd annual ACM symposium on User interface software and technology · doi:10.1145/1622176.1622213 · Cited by 346
Research works printed but not linked: 5
- Microsoft Office Outlook 2003 Step By Step , Microsoft Press , 2004. Microsoft Office Outlook 2003 Step By Step, Microsoft Press, 2004.
- Microsoft Office Powerpoint 2003 Step By Step , Microsoft Press , 2004. Microsoft Office Powerpoint 2003 Step By Step, Microsoft Press, 2004.
- Microsoft Office Word 2003 Step By Step , Microsoft Press , 2004. Microsoft Office Word 2003 Step By Step, Microsoft Press, 2004.
- Mitchell , T. , ( 1997 ), Machine Learning , McGraw-Hill . Mitchell, T., (1997), Machine Learning, McGraw-Hill.
- Windows XP Step By Step , 2 nd edition, Microsoft Press , 2005 . Windows XP Step By Step, 2nd edition, Microsoft Press, 2005.
Cited by
Ability-Based Design: Concept, Principles and Examples
Wobbrock, J. O., Kane, S. K., Gajos, K. Z., Harada, S., & Froehlich, J. (2011) · ACM Transactions on Accessible Computing · doi:10.1145/1952383.1952384 · Cited by 567
SeeingVR: A Set of Tools to Make Virtual Reality More Accessible to People with Low Vision
Zhao, Y., Cutrell, E., Holz, C., Morris, M. R., Ofek, E., & Wilson, A. D. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300341 · Cited by 207
Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure
Dixon, M., & Fogarty, J. A. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753554 · Cited by 150
Designing Inclusive Interfaces Through User Modeling and Simulation
Biswas, P., Robinson, P., & Langdon, P. (2011) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2011.565718 · Cited by 79
Patina: dynamic heatmaps for visualizing application usage
Matejka, J., Grossman, T., & Fitzmaurice, G. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466442 · Cited by 67
Metadata
- Authors
- Hurst, A., Hudson, S. E., & Mankoff, J.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 15th international conference on Intelligent user interfaces
- Publisher
- ACM
- Published
- 2010-02-07
- Pages
- 11-20
- OpenAlex
- W2078091018
- Semantic Scholar
- 5192ed4c6f2b9dbe3dce239d9a9ea8ebbca06df3
- Cited by
- 50
- References
- 23
- Citations by year
- 2022: 1 · 2021: 1 · 2020: 3 · 2019: 6 · 2018: 1 · 2017: 3 · 2016: 3 · 2015: 8 · 2014: 4 · 2013: 2 · 2012: 6
- FWCI
- 4.89
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Keywords
- target identification, visual affordances, human performance assessment, usability analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Automatically identifying targets users interact with during real world tasks. https://app.cochat.ai/papers/10.1145/1719970.1719973
- The paper
- Hurst, A., Hudson, S. E., & Mankoff, J. (2010). Automatically identifying targets users interact with during real world tasks. Proceedings of the 15th international conference on Intelligent user interfaces. https://doi.org/10.1145/1719970.1719973
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.