Towards an efficient methodology for evaluation of quality of experience in Augmented Reality
Abstract
The goal of this paper is to survey existing quality assessment methodologies for Augmented Reality (AR) visualization and to introduce a methodology for subjective quality assessment. Methodologies to assess the quality of AR systems have existed since these technologies appeared. The existing methodologies typically take an approach from the fields they are used in, such as ergonomics, usability, psychophysics or ethnography. Each field utilizes different methods, looking at different aspects of AR quality such as physical limitations, tracking loss or jitter, perceptual issues or feedback issues, just to name a few. AR systems are complex experiences, involving a mix of user interaction, visual perception, audio, haptic or other types of multimodal interactions as well. This paper focuses on the quality assessment of AR visualization, with a special interest on applications for neuronavigation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
QoE measurements has also been addressed in the context of AR systems because of their applications in various fields such as emergency response training and medical surgery.
“QoE measurements has also been addressed in the context of AR systems [289], [290] because of their applications in various fields such as emergency response training and medical surgery.”
From QoE Management of Multimedia Streaming Services in Future Networks: A Tutorial and Survey (2020) · cites this paper as [289]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Azuma, R. T. (1997) · Presence: Teleoperators and Virtual Environments · doi:10.1162/pres.1997.6.4.355 · Cited by 10,121 · see citing works
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,533 · see citing works
A Taxonomy of Mixed Reality Visual Displays
Milgram, P., & Kishino, F. (1994) · IEICE Transactions on Information and Systems · Cited by 4,614 · see citing works
Spatial Augmented Reality: Merging Real and Virtual Worlds
Bimber, O., & Raskar, R. (2005) · Cited by 706 · see citing works
Comparative effectiveness of augmented reality in object assembly
Tang, A., Owen, C. B., Biocca, F. A., & Mou, W. (2003) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642626 · Cited by 619 · see citing works
Perceptual issues in augmented reality revisited
Kruijff, E., Swan, J. E., & Feiner, S. (2010) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar.2010.5643530 · Cited by 497 · see citing works
Ergonomic criteria for evaluating the ergonomic quality of interactive systems
Scapin, D. L., & Bastien, J. M. C. (1997) · Behaviour and Information Technology · doi:10.1080/014492997119806 · Cited by 350 · see citing works
Shuhaiber, J. H. (2004) · Archives of Surgery · doi:10.1001/archsurg.139.2.170 · Cited by 324 · see citing works
Perceptual issues in augmented reality
Drascic, D., & Milgram, P. (1996) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.237425 · Cited by 312 · see citing works
Advanced Medical Displays: A Literature Review of Augmented Reality
Sielhorst, T., Marco Feuerstein, A., & Navab, N. (2008) · Journal of Display Technology · doi:10.1109/jdt.2008.2001575 · Cited by 309 · see citing works
What is going on in augmented reality simulation in laparoscopic surgery?
Botden, S. M. B. I., & Jakimowicz, J. J. (2008) · Surgical Endoscopy · doi:10.1007/s00464-008-0144-1 · Cited by 190 · see citing works
Comparative effectiveness of augmented reality in object assembly
Tang, A., Owen, C., Biocca, F. A., & Mou, W. (2003) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/642625.642626 · Cited by 164 · see citing works
A taxonomy of quality of service and Quality of Experience of multimodal human-machine interaction
Moller, S., Engelbrecht, K.-P., Kühnel, C., Wechsung, I., & Weiss, B. (2009) · International Workshop on Quality of Multimedia Experience · doi:10.1109/qomex.2009.5246986 · Cited by 136 · see citing works
Perceptual-based quality assessment for audio–visual services: A survey
You, J., Reiter, U., Hannuksela, M. M., Gabbouj, M., & Perkis, A. (2010) · Signal Processing Image Communication · doi:10.1016/j.image.2010.02.002 · Cited by 124 · see citing works
Self-citationExplosion Diagrams in Augmented Reality
Kalkofen, D., Tatzgern, M., & Schmalstieg, D. (2009) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr.2009.4811001 · Cited by 42 · see citing works
Augmented Reality for Minimally Invasive Surgery: Overview and Some Recent Advances
Lamata, P., Ali, W., Cano, A. M., Cornellà, J., Declerck, J. M., Ole, J., Freudenthal, A., Furtado, H., Kalkofen, D., Nærum, E., et al. (2010) · InTech eBooks · doi:10.5772/7128 · Cited by 41 · see citing works
Importance masks for revealing occluded objects in augmented reality
Méndez, E., & Schmalstieg, D. (2009) · ACM Symposium on Virtual Reality Software and Technology (VRST) · doi:10.1145/1643928.1643988 · Cited by 29 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
QoE Management of Multimedia Streaming Services in Future Networks: A Tutorial and Survey
Barakabitze, A. A., Barman, N., Ahmad, A., Zadtootaghaj, S., Sun, L., Martini, M. G., & Atzori, L. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2958784 · Cited by 239 · see citing works
A Survey of Emerging Concepts and Challenges for QoE Management of Multimedia Services
Skorin-Kapov, L., Varela, M., Hoßfeld, T., & Chen, K.-T. (2018) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3176648 · Cited by 116 · see citing works
Brunnstrom, K. E., Dima, E., Qureshi, T. N., Johanson, M., Andersson, M., & Sjöström, M. (2020) · Signal Processing Image Communication · doi:10.1016/j.image.2020.116005 · Cited by 101 · see citing works
Keighrey, C., Flynn, R., Murray, S., & Murray, N. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2982046 · Cited by 78 · see citing works
Keighrey, C., Flynn, R., Murray, S., & Murray, N. (2017) · International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2017.7965656 · Cited by 51 · see citing works
Designing and Evaluating a Mesh Simplification Algorithm for Virtual Reality
Bahirat, K., Lai, C., McMahan, R. P., & Prabhakaran, B. (2018) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3209661 · Cited by 40 · see citing works
Keighrey, C., Flynn, R., Murray, S., Brennan, S., & Murray, N. (2017) · Thematic Workshops of ACM Multimedia · doi:10.1145/3126686.3126747 · Cited by 32 · see citing works
How Passive Image Viewers Became Active Multimedia Users
Redi, J., Zhu, Y., de Ridder, H., & Heynderickx, I. E. J. (2014) · Springer eBooks · doi:10.1007/978-3-319-10368-6_2 · Cited by 26 · see citing works
Quality of Experience for a Virtual Reality simulator
Brunnstrom, K. E., Sjöström, M., Imran, M., Pettersson, M., & Johanson, M. (2018) · Electronic Imaging · doi:10.2352/issn.2470-1173.2018.14.hvei-526 · Cited by 23 · see citing works
Visual quality beyond artifact visibility
Redi, J. (2013) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2006353 · Cited by 15 · see citing works
Dima, E., Brunnstrom, K. E., Sjöström, M., Andersson, M., Edlund, J., Johanson, M., & Qureshi, T. N. (2020) · Quality and User Experience · doi:10.1007/s41233-020-0031-7 · Cited by 11 · see citing works
A comprehensive review of quality of experience for emerging video services
Chen, W., Lan, F., Wei, H., Zhao, T., Liu, W., & Xu, Y. (2024) · Signal Processing Image Communication · doi:10.1016/j.image.2024.117176 · Cited by 10 · see citing works
Feature comparison and analysis for new challenging research fields of image quality assessment
Tang, L., Sun, K., Bi, J., & Wang, J. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2018.12.002 · Cited by 7 · see citing works
A-me and BrainCloud: Art-Science Interrogations of Localization in Neuroscience
Puig, J., Carusi, A., Cassinelli, Á., Pinel, P., & Hoel, A. S. (2016) · Leonardo · doi:10.1162/leon_a_01382 · Cited by 5 · see citing works
Self-citationEmerging Applications and Services in Future 5G Networks
Barakabitze, A. A., & Hines, A. (2022) · Wiley eBooks · doi:10.1002/9781119800828.ch8 · Cited by 4 · see citing works
Objective Metrics Definition for QoE Assessment for Extended Reality Applications
Ferrarotti, A., & Carli, M. (2023) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3573381.3596503 · Cited by 3 · see citing works
Puig, J., Perkis, A., Hoel, A. S., & Cassinelli, Á. (2013) · SIGGRAPH Asia Art Gallery · doi:10.1145/2542256.2542264 · Cited by 1 · see citing works
Self-citation
- Authors
- Puig, J., Perkis, A., Lindseth, F., & Ebrahimi, T.
- Affiliations
- Norwegian University of Science and Technology (NO)
- Venue
- 2012 Fourth International Workshop on Quality of Multimedia Experience
- Publisher
- IEEE
- Published
- 2012-07
- Volume
- 6
- Pages
- 188-193
- OpenAlex
- W1993105719
- Cited by
- 17
- References
- 17
- Citations by year
- 2024: 1 · 2023: 1 · 2022: 1 · 2020: 3 · 2019: 1 · 2018: 4 · 2017: 2 · 2016: 1 · 2014: 1 · 2013: 2
- FWCI
- 0.68
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Quality of Multimedia Experience (QoMEX) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Augmented Reality Applications
- Physical Sciences › Computer Science › Human-Computer Interaction › Virtual Reality Applications and Impacts
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Keywords
- augmented reality, quality of experience, subjective quality assessment, neuronavigation, usability evaluation, human factors, tracking jitter, multimodal interaction, visual perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
