The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
A Survey of Usability Evaluation in Virtual Environments: Classification and Comparison of Methods
Abstract
Virtual environments (VEs) are a relatively new type of human–computer interface in which users perceive and act in a three-dimensional world. The designers of such systems cannot rely solely on design guidelines for traditional two-dimensional interfaces, so usability evaluation is crucial for VEs. This paper presents an overview of VE usability evaluation to organize and critically analyze diverse work from this field. First, we discuss some of the issues that differentiate VE usability evaluation from evaluation of traditional user interfaces such as GUIs. We also present a review of some VE evaluation methods currently in use, and discuss a simple classification space for VE usability evaluation methods. This classification space provides a structured means for comparing evaluation methods according to three key characteristics: involvement of representative users, context of evaluation, and types of results produced. Finally, to illustrate these concepts, we compare two existing evaluation approaches: testbed evaluation (Bowman, Johnson, & Hodges, 1999) and sequential evaluation (Gabbard, Hix, & Swan, 1999).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1993) · Cited by 9,879
Measuring Presence in Virtual Environments: A Presence Questionnaire
Witmer, B. G., & Singer, M. J. (1998) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474698565686 · Cited by 6,371
Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness
Kennedy, R. S., Lane, N. E., Berbaum, K. S., & Lilienthal, M. G. (1993) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0303_3 · Cited by 5,457
Nielsen, J. (1995) · Conference companion on Human factors in computing systems - CHI '95 · doi:10.1145/223355.223730 · Cited by 2,620
(1966) · Social Science Education Consortium eBooks · Cited by 2,483
The CAVE: audio visual experience automatic virtual environment
Cruz-Neira, C., Sandin, D. J., DeFanti, T. A., Kenyon, R. V., & Hart, J. C. (1992) · Communications of the ACM · doi:10.1145/129888.129892 · Cited by 1,772
Using Presence Questionnaires in Reality
Usoh, M., Catena, E., Arman, S., & Slater, M. (2000) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474600566989 · Cited by 980
Hix, D., & Hartson, H. R. (1993) · John Wiley & Sons, Inc. eBooks · Cited by 895
Self-citationMeasuring Presence: A Response to the Witmer and Singer Presence Questionnaire
Slater, M. (1999) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474699566477 · Cited by 855 · cited 2×
“Following are some examples: • Many VEs attempt to produce a sense of presence in the user; that is, a feeling of actually being in the virtual world rather than the physical one [Witmer & Singer, 1998; Slater 1999; Usoh et al., 2000].”
BackgroundUser and task analysis for interface design
Hackos, J. T., & Redish, J. C. (1998) · CERN Document Server (European Organization for Nuclear Research) · Cited by 815
Cognitive walkthroughs: a method for theory-based evaluation of user interfaces
Polson, P. G., Lewis, C., Rieman, J., & Wharton, C. (1992) · International Journal of Man-Machine Studies · doi:10.1016/0020-7373(92)90039-n · Cited by 797
Perspectives of Curriculum Evaluation
Walbesser, H. H., Tyler, R. W., Gagné, R. M., Scriven, M., & Gagne, R. M. (1968) · American Educational Research Journal · doi:10.2307/1161709 · Cited by 792
Bowman, D. A., & Hodges, L. F. (1997) · Proceedings of the 1997 symposium on Interactive 3D graphics - SI3D '97 · doi:10.1145/253284.253301 · Cited by 714
Self-citationTaking steps: the influence of a walking technique on presence in virtual reality
Slater, M., Usoh, M., & Steed, A. (1995) · ACM Transactions on Computer-Human Interaction · doi:10.1145/210079.210084 · Cited by 713
The Science of Virtual Reality and Virtual Environments
Kalawsky, R. (1993) · Cumulative Index of Computer Aided Architectural Design · Cited by 470
Wayfinding strategies and behaviors in large virtual worlds
Darken, R. P., & Sibert, J. L. (1996) · Proceedings of the SIGCHI conference on Human factors in computing systems common ground - CHI '96 · doi:10.1145/238386.238459 · Cited by 354
User-centered design and evaluation of virtual environments
Gabbard, J. L., Hix, D., & Swan, J. E. (1999) · IEEE Computer Graphics and Applications · doi:10.1109/38.799740 · Cited by 320
Self-citationBOWMAN, D. A., & HODGES, L. F. (1999) · Journal of Visual Languages & Computing · doi:10.1006/jvlc.1998.0111 · Cited by 318
Self-citationDuration and Exposure to Virtual Environments: Sickness Curves During and Across Sessions
Kennedy, R. S., Stanney, K. M., & Dunlap, W. P. (2000) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474600566952 · Cited by 284 · cited 2×
“Another important goal is to identify usability problems early in the development life cycle so that they may be addressed, and the redesign iteratively rened and evaluated (Nielsen & Mack, 1994). In a heuristic evaluation, VE usability experts compare elements of the user interaction design to guidelines or heuristi…”
BackgroundMethodTestbed evaluation of virtual environment interaction techniques
Bowman, D. A., Johnson, D. B., & Hodges, L. F. (1999) · Proceedings of the ACM symposium on Virtual reality software and technology · doi:10.1145/323663.323667 · Cited by 223
Self-citationA framework and testbed for studying manipulation techniques for immersive VR
Poupyrev, I., Weghorst, S., Billinghurst, M., & Ichikawa, T. (1997) · Proceedings of the ACM symposium on Virtual reality software and technology · doi:10.1145/261135.261141 · Cited by 178
User-centered design and evaluation of a real-time battlefield visualization virtual environment
Hix, D., Swan, J. E., Gabbard, J. L., McGee, M., Durbin, J., & King, T. (2003) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr.1999.756939 · Cited by 146
Self-citationDesigning Virtual Environments for Usability
Kaur, K. (1997) · Springer eBooks · doi:10.1007/978-0-387-35175-9_112 · Cited by 119
Virtual reality: an overview of User-related Design Issues
Mills, S., & Noyes, J. (1999) · Interacting with Computers · doi:10.1016/s0953-5438(98)00057-5 · Cited by 99
Usability Inspection Methods: Executive Summary
Mack, R. L., & Nielsen, J. (1995) · Elsevier eBooks · doi:10.1016/b978-0-08-051574-8.50020-0 · Cited by 70
Snow, M. P., & Williges, R. C. (1998) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872098779591395 · Cited by 69
Interacting with virtual environments: an evaluation of a model of interaction
Kaur, K., Maiden, N., & Sutcliffe, A. (1999) · Interacting with Computers · doi:10.1016/s0953-5438(98)00059-9 · Cited by 49
A study of computer-supported user interface evaluation using maximal repeating pattern analysis
Siochi, A. C., & Hix, D. (1991) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/108844.108926 · Cited by 45
Self-citationThe design space of input devices
Card, S. K., Mackinlay, J. D., & Robertson, G. G. (1992) · ACM eBooks · doi:10.1145/146022.146058 · Cited by 41
Experiences with the evaluation of CVE applications
Steed, A. J., & Tromp, J. G. (1998) · Cited by 28
CLIMATE: A framework for developing holistic requirements analysis in Virtual Environments
Conkar, T., Noyes, J., & Kimble, C. (1999) · Interacting with Computers · doi:10.1016/s0953-5438(98)00058-7 · Cited by 10
Research works printed but not linked: 1
- Bowman D., 2002, Mahwah NJ: Lawrence Erlbaum Associates., 277
Cited by
A survey of 3D object selection techniques for virtual environments
Argelaguet, F., & Andujar, C. (2013) · Computers & Graphics · doi:10.1016/j.cag.2012.12.003 · Cited by 521
Heuristic evaluation of virtual reality applications
Sutcliffe, A., & Gault, B. (2004) · Interacting with Computers · doi:10.1016/j.intcom.2004.05.001 · Cited by 289
Usability, user experience and accessibility: towards an integrative model
Sauer, J., Sonderegger, A., & Schmutz, S. (2020) · Ergonomics · doi:10.1080/00140139.2020.1774080 · Cited by 195
MacQuarrie, A., & Steed, A. (2017) · 2017 IEEE Virtual Reality (VR) · doi:10.1109/vr.2017.7892230 · Cited by 142
Wronikowska, M. W., Malycha, J., Morgan, L. J., Westgate, V., Petrinic, T., Young, J. D., & Watkinson, P. J. (2021) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.13582 · Cited by 59
Metadata
- Authors
- Bowman, D. A., Gabbard, J. L., & Hix, D.
- Venue
- Presence: Teleoperators and Virtual Environments
- Publisher
- MIT Press - Journals
- Published
- 2002-08
- Volume
- 11
- Issue
- 4
- Pages
- 404-424
- ISSN
- 1054-7460 (ISSN-L)
- OpenAlex
- W2112940138
- Cited by
- 375
- References
- 31
- FWCI
- 16.14
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.