A Call for Auditable Virtual, Augmented and Mixed Reality
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
XR (Virtual, Augmented and Mixed Reality) technologies are growing in prominence. However, they are increasingly being used in sectors and in situations that can result in harms. As such, this paper argues the need for auditability to become a key consideration of XR systems. Auditability entails capturing information of a system’s operation to enable oversight, inspection or investigation. Things can and will go wrong, and information that helps unpack situations of failure or harm, and that enables accountability and recourse, will be crucial to XR’s adoption and acceptance. In drawing attention to the urgent need for auditability, we illustrate some risks associated with XR technology and their audit implications, and present some initial findings from a survey with developers indicating the current ‘haphazard’ approach towards such concerns. We also highlight some challenges and considerations of XR audit in practice, as well as areas of future work for taking this important area of research forward.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, understanding ADM as a sociotechnical process, we argue that there is another form of opacityunwitting opacity, where those responsible for designing, developing, deploying, and using systems simply don't think to record relevant organisational aspects of ADM processes (perhaps unaware of their relevance for meaningful accountability [12]).”
From Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems (2021) · cites this paper as [12]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Toward Low-Latency and Ultra-Reliable Virtual Reality
Elbamby, M. S., Perfecto, C., Bennis, M., & Doppler, K. (2018) · IEEE Network · doi:10.1109/mnet.2018.1700268 · Cited by 564
Khor, W. S., Baker, B., Amin, K., Chan, A., Patel, K., & Wong, J. (2016) · Annals of Translational Medicine · doi:10.21037/atm.2016.12.23 · Cited by 398 · cited 1×
“cedures [1, 18], in construction [41, 45], energy [15, 21], manu-”
BackgroundMixed reality in education, entertainment, and training
Hughes, C. E., Stapleton, C. B., Hughes, D. E., & Smith, E. M. (2005) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2005.139 · Cited by 356 · cited 1×
“in a range of sectors, including in entertainment [17, 30, 31, 40],”
BackgroundAugmented reality applications in manufacturing: a survey
Ong, S. K., Yuan, M. L., & Nee, A. Y. C. (2008) · International Journal of Production Research · doi:10.1080/00207540601064773 · Cited by 343 · cited 2×
“Their use is envisaged in a range of sectors, including in entertainment [17, 30, 31, 40], education [6, 13], health and fitness [1, 2, 20], construction [41, 45], retail [8, 28], and manufacturing [23, 26], to name but a few.”
BackgroundTwenty Security Considerations for Cloud-Supported Internet of Things
Singh, J., Pasquier, T., Bacon, J., Ko, H., & Eyers, D. (2016) · IEEE Internet of Things Journal · doi:10.1109/jiot.2015.2460333 · Cited by 335 · cited 1×
“The tighter and more integrated the coupling of the physical and digital aspects, the greater the risk of significant, real world consequences [38].”
BackgroundSelf-citationVirtual and Augmented Reality Applications in Manufacturing
Nee, A. Y. C., & Ong, S. K. (2013) · IFAC Proceedings Volumes · doi:10.3182/20130619-3-ru-3018.00637 · Cited by 298 · cited 2×
“Their use is envisaged in a range of sectors, including in entertainment [17, 30, 31, 40], education [6, 13], health and fitness [1, 2, 20], construction [41, 45], retail [8, 28], and manufacturing [23, 26], to name but a few.”
BackgroundPizzi, G., Scarpi, D., Pichierri, M., & Vannucci, V. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.02.008 · Cited by 269 · cited 1×
“Their use is envisaged in a range of sectors, including in entertainment [17, 30, 31, 40], education [6, 13], health and fitness [1, 2, 20], construction [41, 45], retail [8, 28], and manufacturing [23, 26], to name but a few.”
BackgroundWang, X., Truijens, M., Hou, L., Wang, Y., & Zhou, Y. (2014) · Automation in Construction · doi:10.1016/j.autcon.2013.12.003 · Cited by 242 · cited 2×
“education [6, 13], health and fitness [1, 2, 20], construction [41, 45],”
BackgroundReaching for objects in VR displays: lag and frame rate
Ware, C., & Balakrishnan, R. (1994) · ACM Transactions on Computer-Human Interaction · doi:10.1145/198425.198426 · Cited by 231
Harassment in Social Virtual Reality: Challenges for Platform Governance
Blackwell, L., Ellison, N., Elliott-Deflo, N., & Schwartz, R. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359202 · Cited by 218 · cited 1×
“It follows that XR can exacerbate online harms – there is already evidence that bullying and harassment in XR contexts can affect users more severely than ‘traditional’ online interactions [5].”
BackgroundSecurity and Privacy Approaches in Mixed Reality: A Literature Survey
De Guzman, J. A., Thilakarathna, K., & Seneviratne, A. (2020) · ACM Computing Surveys · doi:10.1145/3359626 · Cited by 208
Al Janabi, H. F., Aydin, A., Palaneer, S., Macchione, N., Al-Jabir, A., Khan, M. S., Dasgupta, P., & Ahmed, K. (2020) · Surgical Endoscopy · doi:10.1007/s00464-019-06862-3 · Cited by 167
Evaluating label placement for augmented reality view management
Azuma, R. T., & Furmanski, C. (2003) · International Symposium on Mixed and Augmented Reality · doi:10.5555/946248.946790 · Cited by 130
An augmented reality application for improving shopping experience in large retail stores
Cruz, E., Orts‐Escolano, S., Gomez‐Donoso, F., Rizo-Maestre, C., Rangel, J. C., Mora, H., & Cazorla, M. (2018) · Virtual Reality · doi:10.1007/s10055-018-0338-3 · Cited by 126 · cited 1×
“retail [8, 28], and manufacturing [23, 26], to name but a few.”
BackgroundAugmented Reality in Medical Education and Training
Herron, J. (2016) · Journal of Electronic Resources in Medical Libraries · doi:10.1080/15424065.2016.1175987 · Cited by 123 · cited 1×
“Their use is envisaged in a range of sectors, including in entertainment [17, 30, 31, 40], education [6, 13], health and fitness [1, 2, 20], construction [41, 45], retail [8, 28], and manufacturing [23, 26], to name but a few.”
BackgroundThe real-life dangers of augmented reality
Sabelman, E. E., & Lam, R. (2015) · IEEE Spectrum · doi:10.1109/mspec.2015.7131695 · Cited by 111 · cited 1×
“There is therefore potential for critical information in the physical environment to be hidden from view of the operator, or otherwise inappropriately displayed [3, 19, 34].”
BackgroundDecision Provenance: Harnessing Data Flow for Accountable Systems
Singh, J., Cobbe, J., & Norval, C. (2019) · IEEE Access · doi:10.1109/access.2018.2887201 · Cited by 110
Self-citationSecuring Augmented Reality Output
Lebeck, K., Ruth, K., Kohno, T., & Roesner, F. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.13 · Cited by 105 · cited 1×
“There is therefore potential for critical information in the physical environment to be hidden from view of the operator, or otherwise inappropriately displayed [3, 19, 34].”
BackgroundDetection of large-scale concrete columns for automated bridge inspection
Zhu, Z., Paal, S. G., & Brilakis, I. (2010) · Automation in Construction · doi:10.1016/j.autcon.2010.07.016 · Cited by 101 · cited 2×
“Their use is envisaged in a range of sectors, including in entertainment [17, 30, 31, 40], education [6, 13], health and fitness [1, 2, 20], construction [41, 45], retail [8, 28], and manufacturing [23, 26], to name but a few.”
BackgroundEncyclopedia of Computer Graphics and Games
Lee, N. (2024) · Springer eBooks · doi:10.1007/978-3-031-23161-2 · Cited by 92
Yang, S. C. (2012) · Journal of Educational Computing Research · doi:10.2190/ec.47.3.a · Cited by 71 · cited 1×
“This is explicitly recognised, for example, by XR equipment vendors who provide warnings about such risks [16, 22, 25], and the numerous reports of incidents where XR systems have led to harm [4, 11, 32, 44].”
BackgroundAustin, P. D., & Siddall, P. J. (2021) · The Journal of Spinal Cord Medicine · doi:10.1080/10790268.2019.1575554 · Cited by 69 · cited 1×
“Their use is envisaged in a range of sectors, including in entertainment [17, 30, 31, 40], education [6, 13], health and fitness [1, 2, 20], construction [41, 45], retail [8, 28], and manufacturing [23, 26], to name but a few.”
BackgroundSport, M. (2019) · Cited by 64
Hou, L., Wang, Y., Wang, X., Maynard, N., Cameron, I., Zhang, S., & Maynard, Y. (2014) · Proceedings of the IEEE · doi:10.1109/jproc.2013.2295327 · Cited by 61 · cited 1×
“…more common as adoption grows, especially given that XR applications are increasingly being deployed in ‘high-stakes’ sectors and situations, such as to assist with surgical procedures [1, 18], in construction [41, 45], energy [15, 21], manufacturing [23, 26], and in military contexts [14, 33].”
BackgroundBarbieri, S., Vettore, G., Pietrantonio, V., Snenghi, R., Tredese, A., Bergamini, M., Previato, S., Stefanati, A., Gaudio, R. M., & Feltracco, P. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6596 · Cited by 54 · cited 2×
“Serious injuries [32], road accidents [4, 12] and deaths [11] have already been attributed to XR.”
BackgroundAugmented Reality Entertainment: Taking Gaming Out of the Box
Itzstein, S. V., Billinghurst, M., Smith, R., & Thomas, B. H. (2017) · Encyclopedia of Computer Graphics and Games · doi:10.1007/978-3-319-08234-9_81-1 · Cited by 37 · cited 1×
“Their use is envisaged in a range of sectors, including in entertainment [17, 30, 31, 40], education [6, 13], health and fitness [1, 2, 20], construction [41, 45], retail [8, 28], and manufacturing [23, 26], to name but a few.”
BackgroundBreaking the Barriers to True Augmented RealityPreprint
Sandor, C., Fuchs, M., Cassinelli, Á., Li, H., Newcombe, R. A., Yamamoto, G., & Feiner, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.05471 · Cited by 36 · cited 1×
“Misdirection is also a potential issue [35].”
Faccio, M., & McConnell, J. J. (2017) · Journal of Risk & Insurance · doi:10.1111/jori.12301 · Cited by 36
Xu, W., Liang, H., Chen, Y., Li, X., & Yu, K. (2020) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr46266.2020.00039 · Cited by 22
Death by Pokémon GO: The Economic and Human Cost of Using Apps While Driving
Faccio, M., & McConnell, J. J. (2018) · National Bureau of Economic Research · doi:10.3386/w24308 · Cited by 18 · cited 2×
“This is explicitly recognised, for example, by XR equipment vendors who provide warnings about such risks [16, 22, 25], and the numerous reports of incidents where XR systems have led to harm [4, 11, 32, 44].”
BackgroundTowards an accountable Internet of Things: a call for reviewability
Norval, C., Cobbe, J., & Singh, J. (2021) · Institution of Engineering and Technology eBooks · doi:10.1049/pbse014e_ch5 · Cited by 18
Self-citationEntertainment Computing - ICEC 2016
Wallner, G., Kriglstein, S., Hlavacs, H., Malaka, R., Lugmayr, A. R., & Yang, H.-S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46100-7 · Cited by 10
Virtual Reality In Entertainment: The State Of The Industry
Powell, W. A., Garner, T. A., Shapiro, S., & Paul, B. (2017) · Cited by 8
Planking or the “Lying-Down Game:” Two Case Reports
Barbieri, S., Feltracco, P., OMIZZOLO, L., Snenghi, R., Mazloum, R. E., Vettore, G., Bergamini, M., Stefanati, A., Donato, D., Ferronato, C., et al. (2017) · Interactive Journal of Medical Research · doi:10.2196/ijmr.6568 · Cited by 6
Xu, W., Liang, H., Chen, Y., et al. (2020) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr46266.2020.1581268453415 · Cited by 5 · cited 1×
“1), a virtual boundary issue [43], or that a gesture failed to be recognised.”
BackgroundAugmented reality game related injuries
Richards, K. G., Wong, K. Y., & Khan, M. S. (2017) · European Journal of Molecular & Clinical Medicine · doi:10.1016/j.nhccr.2017.10.019 · Cited by 4
Research works printed but not linked: 2
- Li X., Auto-Hierarchical Data Algorithm: Focus on Increasing Users’ Motivation and Duration In Virtual Reality. In 2020 5th IEEE International Conference on Big Data Analytics (ICBDA). 150–153
- Raaen Kjetil, 2015, Entertainment Computing - ICEC
Cited by
Lee, L.-H., Braud, T., Zhou, P., Wang, L., Xu, D., Lin, Z., Kumar, A., Bermejo, C., & Hui, P. (2021) · arXiv · doi:10.48550/arxiv.2110.05352 · Cited by 1,088
“Apart from the data collection, stakes are even higher in the metaverse, since unintended consequences could cause not only psychological damage, but also physical harm [709].”
BackgroundReviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems
Cobbe, J., Lee, M. S. A., & Singh, J. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445921 · Cited by 150
“Technical toolkits may also play a role in facilitating the logging and information to be extracted from technical components in a common manner [12].”
BackgroundSelf-citationThe realm of metaverse: A survey
Prakash, J., V, Vijay, S. A. A., & Peatchimuthu, J. (2023) · Computer Animation and Virtual Worlds · doi:10.1002/cav.2150 · Cited by 70
“222 For instance, the user’s XR mobile headsets may project digital overlays that convey crucial information, such as manholes or the view ahead, which could result in potentially fatal mishaps.”
BackgroundLee, L., & Lin, Z. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3677063 · Cited by 8
“…and ethically-informed designs, is challenging, with several key practical constraints: computational powers and performance on the mobile devices, tools for capturing of user (mis-)behaviours with high levels of data integrity [1], as well as privacy issues raised by capturing such user data [16].”
Effing, R., & Hinz, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70804-6_1 · Cited by 6
Metadata
- Authors
- Cloete, R., Norval, C., & Singh, J.
- Affiliations
- University of Cambridge (GB)
- Venue
- 26th ACM Symposium on Virtual Reality Software and Technology
- Publisher
- ACM
- Published
- 2020-11
- Pages
- 1-6
- OpenAlex
- W3097493778
- Semantic Scholar
- 11295fce38b33d4dce81a0733c3fc64c0ec3c233
- Cited by
- 8
- References
- 35
- Citations by year
- 2025: 1 · 2024: 2 · 2023: 1 · 2021: 4
- FWCI
- 0.48
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Symposium on Virtual Reality Software and Technology (VRST) (publishedVersion, cc-by)
- Funders
- EPSRC (EP/R033501/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Human-Computer Interaction › Virtual Reality Applications and Impacts
- Keywords
- auditability, extended reality, virtual reality, augmented reality, mixed reality, accountability, developer practices, system logging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.