Experiences with an AR evaluation test bed: Presence, performance, and physiological measurement
Abstract
This paper discusses an experiment carried out in an AR test bed called “the pit”. Inspired by the well-known VR acrophobia study of Meehan et al. [18], the experimental goals were to explore whether VR presence instruments were useful in AR (and to modify them where appropriate), to compare additional measures to these well-researched techniques, and to determine if findings from VR evaluations can be transferred to AR. An experimental protocol appropriate for AR was developed. The initial experimental findings concern varying immersion factors (frame rate) and their effect on feelings of presence, user performance and behavior. Unlike the VR study, which found differing frame rates to affect presence measures, there were few differences in the five frame rate modes in our study as measured by the qualitative and quantitative instruments, which included physiological responses, a custom presence questionnaire, task performance, and user behavior. The AR presence questionnaire indicated users experienced a high feeling of presence in all frame rate modes. Behavior, performance, and interview results indicated the participants felt anxiety in the pit environment. However, the physiological data did not reflect this anxiety due to factors of user experience and experiment design. Efforts to develop a useful AR test bed and to identify results from a large data set has produced a body of knowledge related to AR evaluation that can inform others seeking to create AR experiments.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The results from this study indicate that the frame rate have a smaller effect in VR applications with regards to improving the end-user's QoE.
“The results from this study indicate that the frame rate have a smaller effect in VR applications with regards to improving the end-user's QoE.”
From QoE Management of Multimedia Streaming Services in Future Networks: A Tutorial and Survey (2020) · cites this paper as [321]
Cites
Measuring Presence in Virtual Environments: A Presence Questionnaire
Witmer, B. G., & Singer, M. J. (1998) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474698565686 · Cited by 6,371 · see citing works
Using Presence Questionnaires in Reality
Usoh, M., Catena, E., Arman, S., & Slater, M. (2000) · PRESENCE Virtual and Augmented Reality · doi:10.1162/105474600566989 · Cited by 985 · see citing works
Psychophysiology -- Human Behavior and Physiological Response
McGuigan, F. J., & Andreassi, J. L. (1981) · The American Journal of Psychology · doi:10.2307/1422751 · Cited by 976 · see citing works
Walking > walking-in-place > flying, in virtual environments
Usoh, M., Arthur, K. W., Whitton, M. C., Bastos, R., Steed, A. J., Slater, M., & Brooks, F. P. (1999) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/311535.311589 · Cited by 924 · see citing works
Measuring Presence: A Response to the Witmer and Singer Presence Questionnaire
Slater, M. (1999) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474699566477 · Cited by 855 · see citing works
Slater, M., Usoh, M., & Steed, A. J. (1995) · ACM Transactions on Computer-Human Interaction · doi:10.1145/210079.210084 · Cited by 714 · see citing works
Physiological measures of presence in stressful virtual environments
Meehan, M., Insko, B. E., Whitton, M. C., & Brooks, F. P. (2002) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/566570.566630 · Cited by 479 · see citing works
Thompson, W. B., Willemsen, P., Gooch, A. A., Creem-Regehr, S. H., Loomis, J. M., & Beall, A. C. (2004) · PRESENCE Virtual and Augmented Reality · doi:10.1162/1054746042545292 · Cited by 411 · see citing works
Visual Realism Enhances Realistic Response in an Immersive Virtual Environment
Slater, M., Khanna, P., Mortensen, J., & Yu, I. (2009) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2009.55 · Cited by 320 · see citing works
Visual Realism Enhances Realistic Response in an Immersive Virtual Environment - Part 2
Yu, I., Mortensen, J., Khanna, P., Spanlang, B., & Slater, M. (2012) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2012.121 · Cited by 241 · see citing works
Measurement of presence and its consequences in virtual environments
NICHOLS, S., HALDANE, C., & WILSON, J. R. (2000) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0343 · Cited by 237 · see citing works
A survey of evaluation techniques used in augmented reality studies
Duenser, A., Grasset, R., & Billinghurst, M. (2008) · ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia) · doi:10.1145/1508044.1508049 · Cited by 235 · see citing works
How the eye measures reality and virtual reality
Cutting, J. E. (1997) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03200563 · Cited by 227 · see citing works
Comparison of self-report and overt-behavioral procedures for assessing acrophobia
Cohen, D. C. (1977) · Behavior Therapy · doi:10.1016/s0005-7894(77)80116-0 · Cited by 216 · see citing works
Anxiety Increases the Feeling of Presence in Virtual Reality
Bouchard, S., St-Jacques, J., Robillard, G., & Renaud, P. (2008) · PRESENCE Virtual and Augmented Reality · doi:10.1162/pres.17.4.376 · Cited by 205 · see citing works
Like bees around the hive: a comparative study of a mobile augmented reality map
Morrison, A., Oulasvirta, A., Peltonen, P., Lemmela, S., Jacucci, G., Reitmayr, G., Näsänen, J., & Juustila, A. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518991 · Cited by 199 · see citing works
Physiological indicators for the evaluation of co-located collaborative play
Mandryk, R. L., & Inkpen, K. M. (2004) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1031607.1031625 · Cited by 190 · see citing works
Tolerance of temporal delay in virtual environments
Allison, R. S., Harris, L. R., Jenkin, M. R. M., Jasiobedzka, U., & Zacher, J. E. (2002) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr.2001.913793 · Cited by 165 · see citing works
Mania, K., Adelstein, B. D., Ellis, S. R., & Hill, M. I. (2004) · Symposium on Applied Perception in Graphics and Visualization (APGV) · doi:10.1145/1012551.1012559 · Cited by 136 · see citing works
The influence of rendering quality on presence and task performance in a virtual environment
Zimmons, P., & Panter, A. T. (2003) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr.2003.1191170 · Cited by 127 · see citing works
Comparison of the Levels of Presence and Anxiety in an Acrophobic Environment Viewed via HMD or CAVE
Juan, M. ‐., & Pérez, D. (2009) · PRESENCE Virtual and Augmented Reality · doi:10.1162/pres.18.3.232 · Cited by 110 · see citing works
On the Role of Presence in Mixed Reality
Wagner, I., Broll, W., Jacucci, G., et al. (2009) · PRESENCE Virtual and Augmented Reality · doi:10.1162/pres.18.4.249 · Cited by 78 · see citing works
Physiological Reaction as an Objective Measure of Presence in Virtual Environments
Meehan, M., & Brooks, F. P. (2001) · Cited by 74 · see citing works
Putting the Virtual into Reality: Assessing Object-Presence with Projection-Augmented Models
Stevens, B., Jerrams-Smith, J., Heathcote, D., & Callear, D. (2002) · PRESENCE Virtual and Augmented Reality · doi:10.1162/105474602317343677 · Cited by 60 · see citing works
Tang, A., Biocca, F. A., & Lim, L. (2004) · Cited by 56 · see citing works
Presence and the Aura of Meaningful Places
MacIntyre, B., Bolter, J. D., & Gandy, M. (2004) · Cited by 54 · see citing works
Self-citationJuan, M. ‐., Baños, R., Botella, C., Pérez, D., Alcañíz, M., & Monserrat, C. (2006) · PRESENCE Virtual and Augmented Reality · doi:10.1162/pres.15.4.393 · Cited by 49 · see citing works
Cognitive presence as a unified concept of virtual reality effectiveness
Nuñez, D., & Blake, E. (2001) · doi:10.1145/513867.513892 · Cited by 41 · see citing works
Brogni, A., Vinayagamoorthy, V., Steed, A. J., & Slater, M. (2006) · ACM Symposium on Virtual Reality Software and Technology (VRST) · doi:10.1145/1180495.1180572 · Cited by 33 · see citing works
Psychophysiological indicators of the impact of media quality on users
Wilson, G. M. (2001) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/634067.634125 · Cited by 29 · see citing works
Fischer, J., Bartz, D., & Straβer, W. (2005) · ACM Symposium on Virtual Reality Software and Technology (VRST) · doi:10.1145/1101616.1101649 · Cited by 27 · see citing works
IJsselsteijn, W. A., Oosting, W. W., Vogels, I. M. L. C., et al. (2008) · PRESENCE Virtual and Augmented Reality · doi:10.1162/pres.17.3.269 · Cited by 23 · see citing works
Evaluating online control of goal‐directed arm movement while standing in virtual visual environment
Martin, O., Julian, B., Boissieux, L., et al. (2003) · The Journal of Visualization and Computer Animation · doi:10.1002/vis.322 · Cited by 19 · see citing works
Effects of heightened sensory feedback to presence and arousal in virtual driving simulators
Lee, M., & Kim, G. J. (2009) · ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry (VRCAI) · doi:10.1145/1670252.1670285 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
How Immersive Is Enough? A Meta-Analysis of the Effect of Immersive Technology on User Presence
Cummings, J. J., & Bailenson, J. N. (2016) · Media Psychology · doi:10.1080/15213269.2015.1015740 · Cited by 1,749 · see citing works
QoE Management of Multimedia Streaming Services in Future Networks: A Tutorial and Survey
Barakabitze, A. A., Barman, N., Ahmad, A., Zadtootaghaj, S., Sun, L., Martini, M. G., & Atzori, L. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2958784 · Cited by 239 · see citing works
A Survey of Emerging Concepts and Challenges for QoE Management of Multimedia Services
Skorin-Kapov, L., Varela, M., Hoßfeld, T., & Chen, K.-T. (2018) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3176648 · Cited by 116 · see citing works
- Authors
- Gandy, M., Catrambone, R., MacIntyre, B., Alvarez, C., Eiriksdottir, E., Hilimire, M., Davidson, B., & McLaughlin, A. C.
- Affiliations
- Georgia Institute of Technology (US)
- North Carolina State University (US)
- Venue
- 2010 IEEE International Symposium on Mixed and Augmented Reality
- Publisher
- IEEE
- Published
- 2010-10
- Pages
- 127-136
- ISSN
- 1554-7868 (ISSN-L)
- OpenAlex
- W2132851697
- Cited by
- 54
- References
- 34
- Citations by year
- 2025: 8 · 2024: 8 · 2023: 3 · 2022: 3 · 2021: 2 · 2020: 6 · 2019: 2 · 2018: 4 · 2016: 2 · 2015: 2 · 2014: 2 · 2013: 5 · 2012: 4
- FWCI
- 2.94
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Virtual Reality Applications and Impacts
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Augmented Reality Applications
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Keywords
- augmented reality, presence, frame rate, immersion, physiological measures, user performance, user behavior, anxiety, presence questionnaire
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
