The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Extracting usability information from user interface events
Open accessAbstract
Modern window-based user interface systems generate user interface events as natural products of their normal operation. Because such events can be automatically captured and because they indicate user behavior with respect to an application's user interface, they have long been regarded as a potentially fruitful source of information regarding application usage and usability. However, because user interface events are typically voluminos and rich in detail, automated support is generally required to extract information at a level of abstraction that is useful to investigators interested in analyzing application usage or evaluating usability. This survey examines computer-aided techniques used by HCI practitioners and researchers to extract usability-related information from user interface events. A framework is presented to help HCI practitioners and researchers categorize and compare the approaches that have been, or might fruitfully be, applied to this problem. Because many of the techniques in the research literature have not been evaluated in practice, this survey provides a conceptual evaluation to help identify some of the relative merits and drawbacks of the various classes of approaches. Ideas for future research in this area are also presented. This survey addresses the following questions: How might user interface events be used in evaluating usability? How are user interface events related to other forms of usability data? What are the key challenges faced by investigators wishing to exploit this data? What approaches have been brought to bear on this problem and how do they compare to one another? What are some of the important open research questions in this area?
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1993) · Cited by 9,879
Nielsen, J. (1994) · Conference companion on Human factors in computing systems - CHI '94 · doi:10.1145/259963.260531 · Cited by 2,759
The Usability Engineering Lifecycle
NIELSEN, J. (1993) · Usability Engineering · doi:10.1016/b978-0-08-052029-2.50007-3 · Cited by 2,657
Nielsen, J. (1995) · Conference companion on Human factors in computing systems - CHI '95 · doi:10.1145/223355.223730 · Cited by 2,620
Handbook of Human-Computer Interaction
(1988) · Elsevier eBooks · doi:10.1016/c2009-0-12113-x · Cited by 2,058
(2003) · CRC Press eBooks · doi:10.1201/9780367804787 · Cited by 1,227
A linear space algorithm for computing maximal common subsequences
Hirschberg, D. S. (1975) · Communications of the ACM · doi:10.1145/360825.360861 · Cited by 1,093
Watch what I do: programming by demonstration
Cypher, A., Halbert, D. C., Kurlander, D., Lieberman, H. A., Maulsby, D., Myers, B. A., & Turransky, A. (1993) · Medical Entomology and Zoology · Cited by 1,027
Cognitive walkthroughs: a method for theory-based evaluation of user interfaces
Polson, P. G., Lewis, C., Rieman, J., & Wharton, C. (1992) · International Journal of Man-Machine Studies · doi:10.1016/0020-7373(92)90039-n · Cited by 797
(1964) · Elsevier eBooks · doi:10.1016/c2013-0-12392-6 · Cited by 735
The GOMS family of user interface analysis techniques: comparison and contrast
John, B. E., & Kieras, D. E. (1996) · ACM Transactions on Computer-Human Interaction · doi:10.1145/235833.236054 · Cited by 687
FITTS, P. M. (1964) · Categories of Human Learning · doi:10.1016/b978-1-4832-3145-7.50016-9 · Cited by 657
Moran, T. P. (1981) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(81)80022-3 · Cited by 522
Melton, A. W. (1964) · Defense Technical Information Center (DTIC) · Cited by 501
Readings in human-computer interaction : toward the year 2000
Baecker, R. M. (1995) · Cited by 446
Aho, A. V., Kernighan, B. W., & Weinberger, P. J. (1988) · CERN Document Server (European Organization for Nuclear Research) · Cited by 405
Using GOMS for user interface design and evaluation: which technique?
John, B. E., & Kieras, D. E. (1996) · ACM Transactions on Computer-Human Interaction · doi:10.1145/235833.236050 · Cited by 396
Abbott, A. (1990) · Organization Science · doi:10.1287/orsc.1.4.375 · Cited by 382
Analyzing sequential categorical data on dyadic interaction: A comment on Gottman.
Allison, P. D., & Liker, J. K. (1982) · Psychological Bulletin · doi:10.1037//0033-2909.91.2.393 · Cited by 365
Task-Action Grammars: A Model of the Mental Representation of Task Languages
Payne, S., & Green, T. R. G. (1986) · Human-Computer Interaction · doi:10.1207/s15327051hci0202_1 · Cited by 350
EAGER: programming repetitive tasks by example
Cypher, A. (1991) · Proceedings of the SIGCHI conference on Human factors in computing systems Reaching through technology - CHI '91 · doi:10.1145/108844.108850 · Cited by 277 · cited 2×
“r Work in the area of programming by demonstration [Cypher 1993] and plan recognition and assisted completion [Cypher 1991] also addresses problems involved in inferring user intent based on lower level interactions.”
BackgroundSequential Analysis: A Guide for Behavorial Researchers
Gottman, J. M., & Roy, A. K. (1990) · Medical Entomology and Zoology · Cited by 242
Grammatical Models of Organizational Processes
Pentland, B. T. (1995) · Organization Science · doi:10.1287/orsc.6.5.541 · Cited by 233
Exploratory Sequential Data Analysis: Foundations
Sanderson, P., & Fisher, C. (1994) · Human-Computer Interaction · doi:10.1207/s15327051hci0903&4_2 · Cited by 229
Utility and usability: research issues and development contexts
Grudin, J. (1992) · Interacting with Computers · doi:10.1016/0953-5438(92)90005-z · Cited by 220
GEM: a generalized event monitoring language for distributed systems
Mansouri-Samani, M., & Sloman, M. S. (1997) · Distributed Systems Engineering · doi:10.1088/0967-1846/4/2/004 · Cited by 214
Automating process discovery through event-data analysis
Cook, J. E., & Wolf, A. L. (1995) · Proceedings of the 17th international conference on Software engineering - ICSE '95 · doi:10.1145/225014.225021 · Cited by 212
Sequential Analysis: A Guide for Behavorial Researchers
Gottman, J. M., & Roy, A. K. (1990) · Cambridge University Press eBooks · doi:10.1017/cbo9780511529696 · Cited by 209
Agrawal, R., Mehta, M., Shafer, J. C., Srikant, R., Arning, A., & Bollinger, T. (n.d.) · Cited by 200
Remote evaluation: the network as an extension of the usability laboratory
Hartson, H. R., Castillo, J. C., Kelso, J., & Neale, W. C. (1996) · Proceedings of the SIGCHI conference on Human factors in computing systems common ground - CHI '96 · doi:10.1145/238386.238511 · Cited by 195
Human-computer interaction: toward the year 2000
Baecker, R. M., Grudin, J. T., Buxton, W. A. S., & Greenberg, S. (1995) · Morgan Kaufmann Publishers Inc. eBooks · Cited by 191
Automatic Support for Usability Evaluation
Lecerof, A., & Paternò, F. (1998) · IEEE Trans. Software Eng. · doi:10.1109/32.729686 · Cited by 180
MacSHAPA and the enterprise of exploratory sequential data analysis (ESDA)
Sanderson, P., Scott, J., Johnston, T., Mainzer, J., Watanabe, L., & James, J. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1077 · Cited by 163
Debugging heterogeneous distributed systems using event-based models of behavior
Bates, P. C. (1995) · ACM Transactions on Computer Systems · doi:10.1145/200912.200913 · Cited by 153
Evaluating user-computer interaction: a framework
Sweeney, M., Maguire, M. C., & Shackel, B. (1993) · International Journal of Man-Machine Studies · doi:10.1006/imms.1993.1032 · Cited by 144
Yeast: a general purpose event-action system
Krishnamurthy, B., & Rosenblum, D. S. (1995) · IEEE Transactions on Software Engineering · doi:10.1109/32.469456 · Cited by 133
A framework for human factors evaluation
Whitefield, A., Wilson, F. E., & Dowell, J. D. (1991) · Behaviour and Information Technology · doi:10.1080/01449299108924272 · Cited by 121
A comparison of usability techniques for evaluating design
Doubleday, A., Ryan, M. B., Springett, M., & Sutcliffe, A. (1997) · Designing Interactive Systems Conference · doi:10.1145/263552.263583 · Cited by 119
A virtual protocol model for computer-human interaction
Nielsen, J. (1986) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(86)80028-1 · Cited by 105
Olson, G., Herbsleb, J., & Reuter, H. (1994) · Human-Computer Interaction · doi:10.1207/s15327051hci0903&4_6 · Cited by 104
Lag sequential analysis: Robust statistical methods.
Faraone, S. V., & Dorfman, D. D. (1987) · Psychological Bulletin · doi:10.1037/0033-2909.101.2.312 · Cited by 88
Computer analysis of user interfaces based on repetition in transcripts of user sessions
Siochi, A. C., & Ehrich, R. W. (1991) · ACM Transactions on Information Systems · doi:10.1145/119311.119312 · Cited by 81
A study in software process data capture and analysis
Wolf, A. L., & Rosenblum, D. S. (2002) · doi:10.1109/spcon.1993.236817 · Cited by 78
An Approach to Large-Scale Collection of Application Usage Data Over the Internet
Hilbert, D. M., & Redmiles, D. F. (1997) · eScholarship (California Digital Library) · Cited by 74
Self-citationLayered protocols for computer-human dialogue. I: Principles
Taylor, M. M. (1988) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(88)80036-1 · Cited by 72
Kay, J., & Thomas, R. C. (1995) · Communications of the ACM · doi:10.1145/213859.214799 · Cited by 68
Knowledge-based evaluation as design support for graphical user interfaces
Löwgren, J., & Nordqvist, T. (1992) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/142750.142786 · Cited by 65
A study of computer-supported user interface evaluation using maximal repeating pattern analysis
Siochi, A. C., & Hix, D. (1991) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/108844.108926 · Cited by 45
Interface usage measurements in a user interface management system
Olsen, D. R., & Halversen, B. W. (1988) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/62402.62420 · Cited by 40
Are we overlooking some usability testing methods? A comparison of lab, beta, and forum tests
Smilowitz, E. D., DARNELL, M. J., & Benson, A. E. (1994) · Behaviour and Information Technology · doi:10.1080/01449299408914597 · Cited by 38
Playback: A method for evaluating the usability of software and its documentation
Neal, A. S., & Simons, R. M. (1983) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/800045.801585 · Cited by 34
Analyzing and Visualizing Log Files: A Computational Science of Usability
Guzdial, M., Santos, P. J., Badre, A., Hudson, S. E., & Gray, M. H. (1994) · Cited by 28
Designing and using integrated data collection and analysis tools: challenges and considerations
HOIEM, D. E., & Sullivan, K. (1994) · Behaviour and Information Technology · doi:10.1080/01449299408914595 · Cited by 25
Software for the usability lab
Weiler, P. (1993) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/169059.169076 · Cited by 22
Automatic chunk detection in human-computer interaction
Santos, P. J., & Badre, A. N. (1994) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/192309.192327 · Cited by 20
Understanding the applicability of sequential data analysis techniques for analysing usability data
Cuomo, D. L. (1994) · Behaviour and Information Technology · doi:10.1080/01449299408914596 · Cited by 19
Layered protocols for computer-human dialogue. II: some practical issues
Taylor, M. M. (1988) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(88)80037-3 · Cited by 19
A toolkit for appraising the long-term usability of a text editor
Cook, R., Kay, J., Ryan, G., & Thomas, R. C. (1995) · Software Quality Journal · doi:10.1007/bf00402716 · Cited by 17
Extending programming by demonstration with hierarchical event histories
Kosbie, D. S., & Myers, B. A. (1994) · Lecture notes in computer science · doi:10.1007/3-540-58648-2_32 · Cited by 14
Chang, E. J., & Dillon, T. S. (1997) · Springer eBooks · doi:10.1007/978-0-387-35175-9_13 · Cited by 13
Badre, A. N., Guzdial, M., Hudson, S. E., et al. (1995) · Software Quality Journal · doi:10.1007/bf00402714 · Cited by 13
Characterizing Process Change Using Log File Data
Guzdial, M., Walton, C., Konemann, M., & Soloway, E. M. (1993) · SMARTech Repository (Georgia Institute of Technology) · Cited by 12
Proceedings of the workshop on Advanced visual interfaces - AVI '94
(1994) · doi:10.1145/192309 · Cited by 12
Cited by
Reconceptualizing System Usage: An Approach and Empirical Test
Burton-Jones, A., & Straub, D. W. (2006) · Information Systems Research · doi:10.1287/isre.1060.0096 · Cited by 1,040
Empirical Studies in Information Visualization: Seven Scenarios
Lam, H., Bertini, E., Isenberg, P., Plaisant, C., & Carpendale, S. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.279 · Cited by 623
From Use to Effective Use: A Representation Theory Perspective
Burton-Jones, A., & Grange, C. (2013) · Information Systems Research · doi:10.1287/isre.1120.0444 · Cited by 487
Reflective physical prototyping through integrated design, test, and analysis
Hartmann, B., Klemmer, S. R., Bernstein, M., Abdulla, L., Burr, B., Robinson-Mosher, A., & Gee, J. (2006) · Proceedings of the 19th annual ACM symposium on User interface software and technology · doi:10.1145/1166253.1166300 · Cited by 302
What would other programmers do: suggesting solutions to error messages
Hartmann, B., MacDougall, D., Brandt, J., & Klemmer, S. R. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753478 · Cited by 280
Metadata
- Authors
- Hilbert, D. M., & Redmiles, D. F.
- Venue
- ACM Computing Surveys
- Published
- 2000-12
- Volume
- 32
- Issue
- 4
- Pages
- 384-421
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2120973494
- Cited by
- 477
- References
- 63
- FWCI
- 24.62
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.