The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Automatically identifying targets users interact with during real world tasks
Abstract
Information about the location and size of the targets that users interact with in real world settings can enable new innovations in human performance assessment and soft-ware usability analysis. Accessibility APIs provide some information about the size and location of targets. How-ever this information is incomplete because it does not sup-port all targets found in modern interfaces and the reported sizes can be inaccurate. These accessibility APIs access the size and location of targets through low-level hooks to the operating system or an application. We have developed an alternative solution for target identification that leverages visual affordances in the interface, and the visual cues produced as users interact with targets. We have used our novel target identification technique in a hybrid solution that combines machine learning, computer vision, and accessibility API data to find the size and location of targets users select with 89% accuracy. Our hybrid approach is superior to the performance of the accessibility API alone: in our dataset of 1355 targets covering 8 popular applications, only 74% of the targets were correctly identified by the API alone.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Data Mining: Practical Machine Learning Tools and Techniques
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,437
(1992) · Cited by 23,431
Witten, I. H. (2008) · Cited by 9,143
The information capacity of the human motor system in controlling the amplitude of movement.
Fitts, P. M. (1954) · Journal of Experimental Psychology · doi:10.1037/h0055392 · Cited by 6,466
The information capacity of the human motor system in controlling the amplitude of movement.
Fitts, P. M. (1992) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.121.3.262 · Cited by 6,278
Computer Vision: A Modern Approach
Forsyth, D. A., & Ponce, J. (2002) · Cited by 3,739
International Journal of Human-Computer Studies
Sdol, E. (2009) · Cited by 1,095
Accuracy measures for evaluating computer pointing devices
MacKenzie, I. S., Kauppinen, T., & Silfverberg, M. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365028 · Cited by 448
More than dotting the i's --- foundations for crossing-based interfaces
Accot, J., & Zhai, S. (2002) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/503376.503390 · Cited by 372
Sikuli: using GUI screenshots for search and automation
Yeh, T., Chang, T.-H., & Miller, R. C. (2009) · Proceedings of the 22nd annual ACM symposium on User interface software and technology · doi:10.1145/1622176.1622213 · Cited by 346
Research works printed but not linked: 5
- Microsoft Office Outlook 2003 Step By Step , Microsoft Press , 2004. Microsoft Office Outlook 2003 Step By Step, Microsoft Press, 2004.
- Microsoft Office Powerpoint 2003 Step By Step , Microsoft Press , 2004. Microsoft Office Powerpoint 2003 Step By Step, Microsoft Press, 2004.
- Microsoft Office Word 2003 Step By Step , Microsoft Press , 2004. Microsoft Office Word 2003 Step By Step, Microsoft Press, 2004.
- Mitchell , T. , ( 1997 ), Machine Learning , McGraw-Hill . Mitchell, T., (1997), Machine Learning, McGraw-Hill.
- Windows XP Step By Step , 2 nd edition, Microsoft Press , 2005 . Windows XP Step By Step, 2nd edition, Microsoft Press, 2005.
Cited by
Ability-Based Design: Concept, Principles and Examples
Wobbrock, J. O., Kane, S. K., Gajos, K. Z., Harada, S., & Froehlich, J. (2011) · ACM Transactions on Accessible Computing · doi:10.1145/1952383.1952384 · Cited by 567
SeeingVR: A Set of Tools to Make Virtual Reality More Accessible to People with Low Vision
Zhao, Y., Cutrell, E., Holz, C., Morris, M. R., Ofek, E., & Wilson, A. D. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300341 · Cited by 207
Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure
Dixon, M., & Fogarty, J. A. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753554 · Cited by 150
Designing Inclusive Interfaces Through User Modeling and Simulation
Biswas, P., Robinson, P., & Langdon, P. (2011) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2011.565718 · Cited by 79
Patina: dynamic heatmaps for visualizing application usage
Matejka, J., Grossman, T., & Fitzmaurice, G. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466442 · Cited by 67
Waken: reverse engineering usage information and interface structure from software videos
Banović, N., Grossman, T., Matejka, J., & Fitzmaurice, G. W. (2012) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2380116.2380129 · Cited by 68
Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility
Zhang, X., Ross, A. S., Caspi, A., Fogarty, J., & Wobbrock, J. O. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025846 · Cited by 59
“The problem is analogous to developer failure to provide alternative text for images on the web [6] or to implement accessibility guidelines in desktop applications [28].”
BackgroundMethodEvans, A. C., & Wobbrock, J. O. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208338 · Cited by 56
Content and hierarchy in pixel-based methods for reverse engineering interface structure
Dixon, M., Leventhal, D., & Fogarty, J. A. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979086 · Cited by 58
ActionNet: Vision-Based Workflow Action Recognition From Programming Screencasts
Zhao, D., Xing, Z., Chen, C., Xia, X., & Li, G. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00049 · Cited by 58
Associating the visual representation of user interfaces with their internal structures and metadata
Chang, T.-H., YEH, T. P., & Miller, R. C. (2011) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2047196.2047228 · Cited by 49
Biswas, P., & Langdon, P. (2012) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2011.636294 · Cited by 46
Zhong, Y., Weber, A., Burkhardt, C., Weaver, P., & Bigham, J. P. (2015) · Proceedings of the 12th International Web for All Conference · doi:10.1145/2745555.2747277 · Cited by 42
Creating contextual help for GUIs using screenshots
YEH, T. P., Chang, T.-H., Xie, B., Walsh, G., Watkins, I., Wongsuphasawat, K., Huang, M., Davis, L. S., & Bederson, B. B. (2011) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2047196.2047214 · Cited by 42
VASTA: a vision and language-assisted smartphone task automation system
Sereshkeh, A. R., Leung, G., Perumal, K., Phillips, C., Zhang, M., Fazly, A., & Mohomed, I. (2020) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3377325.3377515 · Cited by 44
RePlay: Contextually Presenting Learning Videos Across Software Applications
Fraser, C. A., Ngoon, T. J., Dontcheva, M., & Klemmer, S. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300527 · Cited by 49
Extracting and analyzing time-series HCI data from screen-captured task videos
Bao, L., Li, J., Xing, Z., Wang, X., Xia, X., & Zhou, B. (2016) · Empirical Software Engineering · doi:10.1007/s10664-015-9417-1 · Cited by 44
StateLens: A Reverse Engineering Solution for Making Existing Dynamic Touchscreens Accessible
Guo, A., Kong, J., Rivera, M., Xu, F. F., & Bigham, J. P. (2019) · Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3332165.3347873 · Cited by 35
“Hurst et al. [22] combine a number of useful computer vision techniques with mouse information to automatically identify clickable targets in the interface.”
BackgroundDixon, M., Fogarty, J. A., & Wobbrock, J. O. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208734 · Cited by 34
InterWeave: Presenting Search Suggestions in Context Scaffolds Information Search and Synthesis
Palani, S., Zhou, Y., Zhu, S., & Dow, S. P. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545696 · Cited by 33
Inference of development activities from interaction with uninstrumented applications
Bao, L., Xing, Z., Xia, X., Lo, D., & Hassan, A. E. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9547-8 · Cited by 25
Prefab layers and prefab annotations: extensible pixel-based interpretation of graphical interfaces
Dixon, M., Nied, A. C., & Fogarty, J. A. (2014) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2642918.2647412 · Cited by 24
Accurate measurements of pointing performance from in situ observations
Gajos, K. Z., Reinecke, K., & Herrmann, C. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208733 · Cited by 22
Reverse engineering time-series interaction data from screen-captured videos
Bao, L., Li, J., Xing, Z., Wang, X., & Zhou, B. (2015) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2015.7081850 · Cited by 19
ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints
Jiang, Y., Stuerzlinger, W., & Lutteroth, C. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445043 · Cited by 18
Help, It Looks Confusing: GUI Task Automation Through Demonstration and Follow-up Questions
Intharah, T., Turmukhambetov, D., & Brostow, G. (2017) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3025171.3025176 · Cited by 19
UIMarks: quick graphical interaction with specific targets
Chapuis, O., & Roussel, N. (2010) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1866029.1866057 · Cited by 18
ReMap: Lowering the Barrier to Help-Seeking with Multimodal Search
Fraser, C. A., Markel, J. M., Basa, N. J., Dontcheva, M., & Klemmer, S. (2020) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3379337.3415592 · Cited by 19
scvRipper: Video Scraping Tool for Modeling Developers' Behavior Using Interaction Data
Bao, L., Li, J., Xing, Z., Wang, X., & Zhou, B. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.220 · Cited by 18
Tracking and Analyzing Cross-Cutting Activities in Developers' Daily Work (N)
Bao, L., Xing, Z., Wang, X., & Zhou, B. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.43 · Cited by 15
Pixel-based methods for widget state and style in a runtime implementation of sliding widgets
Dixon, M., Laput, G., & Fogarty, J. A. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2556979 · Cited by 14
Billah, S. M., Porter, D. E., & Ramakrishnan, I. V. (2016) · European Conference on Computer Systems · doi:10.1145/2901318.2901335 · Cited by 12
Using graphical representation of user interfaces as visual references
Chang, T.-H. (2011) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2046396.2046411 · Cited by 10
Automatic Adjustment of Keyboard Settings Can Enhance Typing
Koester, H. H., & Mankowski, J. (2015) · Assistive Technology · doi:10.1080/10400435.2014.997899 · Cited by 10
HILC: Domain-Independent PbD System Via Computer Vision and Follow-Up Questions
Intharah, T., Turmukhambetov, D., & Brostow, G. (2019) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3234508 · Cited by 11
Automatic Adjustment of Mouse Settings to Improve Pointing Performance
Koester, H. H., & Mankowski, J. (2014) · Assistive Technology · doi:10.1080/10400435.2013.862583 · Cited by 8
Biswas, P., & Langdon, P. (2011) · Journal of Assistive Technologies · doi:10.1108/17549451111173460 · Cited by 8
Intelligently Creating Contextual Tutorials for GUI Applications
Guo, L., Lu, T., Yang, J., Zhou, X., Ding, X., & Gu, N. (2015) · IEEE International Conference on Ubiquitous Intelligence and Computing, Autonomic and Trusted Computing, and Scalable Computing and Communications (UIC-ATC-ScalCom) · doi:10.1109/uic-atc-scalcom-cbdcom-iop.2015.50 · Cited by 6
Intharah, T., Firman, M., & Brostow, G. (2018) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3170427.3188532 · Cited by 5
Evaluation of the Android Accessibility API Recognition Rate Towards a Better User Experience
Pichiliani, M. C., & Hirata, C. M. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-20678-3_33 · Cited by 5
Investigating the Accessibility of Program Selection Menus of a Digital TV Interface
Biswas, P., & Langdon, P. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21619-0_52 · Cited by 4
Paquette, I., Kwan, C., & Betke, M. (2011) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/2141622.2141625 · Cited by 4
Evans, A. C., & Wobbrock, J. O. (2011) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1979742.1979871 · Cited by 4
Rataplan: Resilient Automation of User Interface Actions with Multi-modal Proxies
Veuskens, T., Luyten, K., & Ramakers, R. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3397329 · Cited by 5
Kwan, C., Paquette, I., Magee, J., & Betke, M. (2013) · Universal Access in the Information Society · doi:10.1007/s10209-013-0295-2 · Cited by 3
Help, It Looks Confusing: GUI Task Automation Through Demonstration and Follow-up QuestionsPreprint
Intharah, T., Turmukhambetov, D., & Brostow, G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03906 · Cited by 2
Inclusive User Modelling and Simulation
Biswas, P., & Langdon, P. (2012) · CRC Press eBooks · doi:10.1201/b13729-14 · Cited by 1
Inclusive User Modeling and Simulation
Biswas, P., & Langdon, P. (2015) · Human-computer interaction series · doi:10.1007/978-1-4471-6708-2_5
FiTouch: Adapting graphical user interface for touch interaction
Liu, L., Zhang, J., & Yin, X. X. (2014) · IEEE International Workshop on Haptic Audio visual Environments and Games · doi:10.1109/have.2014.6954337
Runtime Repair and Enhancement of Mobile App Accessibility
Zhang, X. (2019) · ResearchWorks at the University of Washington (University of Washington)
Metadata
- Authors
- Hurst, A., Hudson, S. E., & Mankoff, J.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 15th international conference on Intelligent user interfaces
- Publisher
- ACM
- Published
- 2010-02-07
- Pages
- 11-20
- OpenAlex
- W2078091018
- Semantic Scholar
- 5192ed4c6f2b9dbe3dce239d9a9ea8ebbca06df3
- Cited by
- 50
- References
- 23
- Citations by year
- 2022: 1 · 2021: 1 · 2020: 3 · 2019: 6 · 2018: 1 · 2017: 3 · 2016: 3 · 2015: 8 · 2014: 4 · 2013: 2 · 2012: 6
- FWCI
- 4.89
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Keywords
- target identification, visual affordances, human performance assessment, usability analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.