The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Augmented Reality for Robotics: A Review
Open accessAbstract
Augmented reality (AR) is used to enhance the perception of the real world by integrating virtual objects to an image sequence acquired from various camera technologies. Numerous AR applications in robotics have been developed in recent years. The aim of this paper is to provide an overview of AR research in robotics during the five year period from 2015 to 2019. We classified these works in terms of application areas into four categories: (1) Medical robotics: Robot-Assisted surgery (RAS), prosthetics, rehabilitation, and training systems; (2) Motion planning and control: trajectory generation, robot programming, simulation, and manipulation; (3) Human-robot interaction (HRI): teleoperation, collaborative interfaces, wearable robots, haptic interfaces, brain-computer interfaces (BCIs), and gaming; (4) Multi-agent systems: use of visual feedback to remotely control drones, robot swarms, and robots with shared workspace. Recent developments in AR technology are discussed followed by the challenges met in AR due to issues of camera localization, environment mapping, and registration. We explore AR applications in terms of how AR was integrated and which improvements it introduced to corresponding fields of robotics. In addition, we summarize the major limitations of the presented applications in each category. Finally, we conclude our review with future directions of AR research in robotics. The survey covers over 100 research works published over the last five years.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Azuma, R. T. (1997) · Presence: Teleoperators and Virtual Environments · doi:10.1162/pres.1997.6.4.355 · Cited by 10,121
A Taxonomy of Mixed Reality Visual Displays
Milgram, P., & Kishino, F. (1994) · IEICE Transactions on Information and Systems · Cited by 4,905
Simultaneous localization and mapping: part I
Durrant-Whyte, H., & Bailey, T. (2006) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2006.1638022 · Cited by 4,199
Recent advances in augmented reality
Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & MacIntyre, B. (2001) · IEEE Computer Graphics and Applications · doi:10.1109/38.963459 · Cited by 4,148
Simultaneous localization and mapping (SLAM): part II
Bailey, T., & Durrant-Whyte, H. (2006) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2006.1678144 · Cited by 2,554
Marker tracking and HMD calibration for a video-based augmented reality conferencing system
Kato, H., & Billinghurst, M. (2003) · Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99) · doi:10.1109/iwar.1999.803809 · Cited by 2,194
A head-mounted three dimensional display
Sutherland, I. E. (1968) · Proceedings of the December 9-11, 1968, fall joint computer conference, part I on - AFIPS '68 (Fall, part I) · doi:10.1145/1476589.1476686 · Cited by 2,065
Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR
Zhou, F., Duh, H. B.-L., & Billinghurst, M. (2008) · 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2008.4637362 · Cited by 1,121
Visual simultaneous localization and mapping: a survey
Fuentes-Pacheco, J., Ruiz-Ascencio, J., & Rendón-Mancha, J. M. (2015) · Artificial Intelligence Review · doi:10.1007/s10462-012-9365-8 · Cited by 883
A Survey of Research on Cloud Robotics and Automation
Kehoe, B., Patil, S., Abbeel, P., & Goldberg, K. (2015) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2014.2376492 · Cited by 833
Human–Robot Interaction: Status and Challenges
Sheridan, T. B. (2016) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816644364 · Cited by 786
Augmented reality applications in design and manufacturing
Nee, A. Y. C., Ong, S. K., Chryssolouris, G., & Mourtzis, D. (2012) · CIRP Annals · doi:10.1016/j.cirp.2012.05.010 · Cited by 762
Pose estimation for augmented reality: a hands-on survey
Marchand, E., Uchiyama, H., & Spindler, F. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2513408 · Cited by 746 · cited 1×
“Another survey presenting an overview of pose estimation algorithms used in AR was presented in [12].”
Simulation in the design and operation of manufacturing systems: state of the art and new trends
Mourtzis, D. (2020) · International Journal of Production Research · doi:10.1080/00207543.2019.1636321 · Cited by 727
Human-machine-interaction in the industry 4.0 era
Gorecky, D., Schmitt, M., Loskyll, M., & Zuhlke, D. (2014) · 2014 12th IEEE International Conference on Industrial Informatics (INDIN) · doi:10.1109/indin.2014.6945523 · Cited by 724
Virtual fixtures: Perceptual tools for telerobotic manipulation
Rosenberg, L. B. (1993) · Proceedings of IEEE Virtual Reality Annual International Symposium · doi:10.1109/vrais.1993.380795 · Cited by 578
Motion tracking: no silver bullet, but a respectable arsenal
Welch, G., & Foxlin, E. (2002) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2002.1046626 · Cited by 535
A comprehensive survey of augmented reality assembly research
Wang, X., Ong, S. K., & Nee, A. Y. C. (2016) · Advances in Manufacturing · doi:10.1007/s40436-015-0131-4 · Cited by 476
Mobile Augmented Reality Survey: From Where We Are to Where We Go
Chatzopoulos, D., Bermejo, C., Huang, Z., & Hui, P. (2017) · IEEE Access · doi:10.1109/access.2017.2698164 · Cited by 442
Improving static and dynamic registration in an optical see-through HMD
Azuma, R., & Bishop, G. (1994) · Proceedings of the 21st annual conference on Computer graphics and interactive techniques - SIGGRAPH '94 · doi:10.1145/192161.192199 · Cited by 390
Augmented reality in laparoscopic surgical oncology
Nicolau, S., Soler, L., Mutter, D., & Marescaux, J. (2011) · Surgical Oncology · doi:10.1016/j.suronc.2011.07.002 · Cited by 367
First experiences with the da Vinci™ operating robot in thoracic surgery☆
Bodner, J., Wykypiel, H., Wetscher, G., & Schmid, T. (2004) · European Journal of Cardio-Thoracic Surgery · doi:10.1016/j.ejcts.2004.02.001 · Cited by 319
Diana, M., & Marescaux, J. (2015) · British Journal of Surgery · doi:10.1002/bjs.9711 · Cited by 318
The status of augmented reality in laparoscopic surgery as of 2016
Bernhardt, S., Nicolau, S. A., Soler, L., & Doignon, C. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.01.007 · Cited by 314
Touch-less interactive augmented reality game on vision-based wearable device
Lv, Z., Halawani, A., Feng, S., ur Réhman, S., & Li, H. (2015) · Personal and Ubiquitous Computing · doi:10.1007/s00779-015-0844-1 · Cited by 295
A history of robots: from science fiction to surgical robotics
Hockstein, N. G., Gourin, C. G., Faust, R. A., & Terris, D. J. (2007) · Journal of Robotic Surgery · doi:10.1007/s11701-007-0021-2 · Cited by 271
Communicating Robot Motion Intent with Augmented Reality
Walker, M., Hedayati, H., Lee, J., & Szafir, D. (2018) · Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3171221.3171253 · Cited by 219
Augmented Reality Technology Using Microsoft HoloLens in Anatomic Pathology
Hanna, M. G., Ahmed, I., Nine, J., Prajapati, S., & Pantanowitz, L. (2018) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2017-0189-oa · Cited by 213
Augmented Reality Application to Support Remote Maintenance as a Service in the Robotics Industry
Mourtzis, D., Zogopoulos, V., & Vlachou, E. (2017) · Procedia CIRP · doi:10.1016/j.procir.2017.03.154 · Cited by 201
Visual marker detection and decoding in AR systems: a comparative study
Zhang, X., Fronz, S., & Navab, N. (2002) · International Symposium on Mixed and Augmented Reality · doi:10.5555/850976.854955 · Cited by 189
Augmented reality-assisted robot programming system for industrial applications
Ong, S. K., Yew, A. W. W., Thanigaivel, N. K., & Nee, A. Y. C. (2020) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2019.101820 · Cited by 176
Improving Collocated Robot Teleoperation with Augmented Reality
Hedayati, H., Walker, M., & Szafir, D. (2018) · Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3171221.3171251 · Cited by 165
Robot Programming Through Augmented Trajectories in Augmented Reality
Quintero, C. P., Li, S., Pan, M. K. X. J., Chan, W. P., der Loos, H. F. M. V., & Croft, E. A. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593700 · Cited by 161
A Review of Augmented Reality in Robotic-Assisted Surgery
Qian, L., Wu, J. Y., DiMaio, S. P., Navab, N., & Kazanzides, P. (2020) · IEEE Transactions on Medical Robotics and Bionics · doi:10.1109/tmrb.2019.2957061 · Cited by 159
Towards BCI-Based Interfaces for Augmented Reality: Feasibility, Design and Evaluation
Si-Mohammed, H., Petit, J., Jeunet, C., Argelaguet, F., Spindler, F., Évain, A., Roussel, N., Casiez, G., & Lécuyer, A. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2873737 · Cited by 151
Mourtzis, D., Zogopoulos, V., & Xanthi, F. (2019) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-019-03941-6 · Cited by 145
Robot Teleoperation with Augmented Reality Virtual Surrogates
Walker, M. E., Hedayati, H., & Szafir, D. J. (2019) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2019.8673306 · Cited by 140
Virtual and Augmented Reality in Oncologic Liver Surgery
Quero, G., Lapergola, A., Soler, L., Shahbaz, M., Hostettler, A., Collins, T., Marescaux, J., Mutter, D., Diana, M., & Pessaux, P. (2018) · Surgical Oncology Clinics of North America · doi:10.1016/j.soc.2018.08.002 · Cited by 115
Porpiglia, F., Checcucci, E., Amparore, D., Riccardo Autorino, B., Piana, A., Bellin, A., Piazzolla, P., Massa, F., Bollito, E., Gned, D., et al. (2018) · British Journal of Urology · doi:10.1111/bju.14549 · Cited by 105
Brizzi, F., Peppoloni, L., Graziano, A., Stefano, E. D., Avizzano, C. A., & Ruffaldi, E. (2017) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2017.2782490 · Cited by 97
Chowriappa, A. J., Raza, S. J., Fazili, A. A., Field, E. P., Malito, C., Samarasekera, D., Shi, Y. Y., Ahmed, K., Wilding, G. E., Kaouk, J. H., et al. (2014) · British Journal of Urology · doi:10.1111/bju.12704 · Cited by 96
Haptic and visual augmented reality interface for programming welding robots
Ni, D., Yew, A. W. W., Ong, S. K., & Nee, A. Y. C. (2017) · Advances in Manufacturing · doi:10.1007/s40436-017-0184-7 · Cited by 90
Augmented-reality integrated robotics in neurosurgery: are we there yet?
Madhavan, K., Kolcun, J. P. G., Chieng, L. O., & Wang, M. Y. (2017) · Neurosurgical FOCUS · doi:10.3171/2017.2.focus177 · Cited by 87
Augmented reality experiments with industrial robot in industry 4.0 environment
Malý, I., Sedlacek, D., & Leitão, P. (2016) · IEEE International Conference on Industrial Informatics (INDIN) · doi:10.1109/indin.2016.7819154 · Cited by 85
Augmented reality assisted surgery: a urologic training tool
Hakky, T., Dickey, R., Srikishen, N., Lipshultz, L. I., Spiess, P., & Carrión, R. (2015) · Asian Journal of Andrology · doi:10.4103/1008-682x.166436 · Cited by 83
Gurevich, P., Lanir, J., & Cohen, B. (2015) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-015-9232-7 · Cited by 81
ARK: Augmented Reality for Kilobots
Reina, A., Cope, A. J., Nikolaidis, E., Marshall, J. A. R., & Sabo, C. M. (2017) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2017.2700059 · Cited by 80
Zeng, H., Wang, Y., Wu, C., Song, A., Liu, J., Ji, P., Bao-guo, X., Zhu, L., Li, H., & Wen, P. (2017) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2017.00060 · Cited by 75
Behzadan, A. H., Timm, B. W., & Kamat, V. R. (2008) · Advanced Engineering Informatics · doi:10.1016/j.aei.2007.08.005 · Cited by 73
Sophisticated collective foraging with minimalist agents: a swarm robotics test
Talamali, M. S., Bose, T., Haire, M., Xu, X., Marshall, J. A. R., & Reina, A. (2020) · Swarm Intelligence · doi:10.1007/s11721-019-00176-9 · Cited by 70
Immersive Augmented Reality Environment for the Teleoperation of Maintenance Robots
Yew, A. W. W., Ong, S. K., & Nee, A. Y. C. (2017) · Procedia CIRP · doi:10.1016/j.procir.2016.11.183 · Cited by 70
Development of a stereo video see-through HMD for AR systems
Takagi, A., Yamazaki, S., Saito, Y., & Taniguchi, N. (2002) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/isar.2000.880925 · Cited by 69
Interactive Robot Knowledge Patching Using Augmented Reality
Liu, H., Zhang, Y., Si, W., Xie, X., Zhu, Y., & Zhu, S. C. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8462837 · Cited by 68
Stadler, S., Kain, K., Giuliani, M., Mirnig, N., Stollnberger, G., & Tscheligi, M. (2016) · IEEE RO-MAN · doi:10.1109/roman.2016.7745108 · Cited by 63
The use of virtual fixtures to enhance operator performance in time delayed teleoperation.
Rosenberg, L. B. (1993) · PsycEXTRA Dataset · doi:10.1037/e477812004-001 · Cited by 58
Pai, Y. S., Yap, H. J., Dawal, S. Z. M., Ramesh, S. S. R., & Phoon, S. Y. (2016) · Scientific Reports · doi:10.1038/srep27380 · Cited by 53
Kilogrid: a novel experimental environment for the Kilobot robot
Valentini, G., Antoun, A., Trabattoni, M., Wiandt, B., Tamura, Y., Hocquard, E., Trianni, V., & Dorigo, M. (2018) · Swarm Intelligence · doi:10.1007/s11721-018-0155-z · Cited by 52
Humans Can Integrate Augmented Reality Feedback in Their Sensorimotor Control of a Robotic Hand
Clemente, F., Dosen, S. S., Lonini, L., Marković, M., Farina, D., & Cipriani, C. (2016) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2016.2611998 · Cited by 51
Zhou, C., Zhu, M. B., Shi, Y., Lin, L., Chai, G., Zhang, Y., & Xie, L. (2017) · Aesthetic Plastic Surgery · doi:10.1007/s00266-017-0900-5 · Cited by 50
Meli, L., Pacchierotti, C., Salvietti, G., Chinello, F., Maisto, M., Luca, A. D., & Prattichizzo, D. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2864354 · Cited by 49
Quality-Sensitive Foraging by a Robot Swarm Through Virtual Pheromone Trails
Llenas, A. F., Talamali, M. S., Xu, X., Marshall, J. A. R., & Reina, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00533-7_11 · Cited by 47
Simulating Kilobots Within ARGoS: Models and Experimental Validation
Pinciroli, C., Talamali, M. S., Reina, A., Marshall, J. A. R., & Trianni, V. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00533-7_14 · Cited by 36
Navab, N., Fellow, M., Hennersperger, C., Frisch, B., & Fürst, B. (2016) · Medical Image Analysis · doi:10.1016/j.media.2016.06.021 · Cited by 27
Augmented reality for robots: Virtual sensing technology applied to a swarm of e-pucks
Reina, A., Salvaro, M., Francesca, G., Garattoni, L., Pinciroli, C., Dorigo, M., & Birattari, M. (2015) · NASA/ESA Conference on Adaptive Hardware and Systems (AHS) · doi:10.1109/ahs.2015.7231154 · Cited by 26
Omidshafiei, S., Agha–mohammadi, A., Chen, Y. F., Üre, N. K., Liu, S.-Y., Lopez, B. T., Surati, R., How, J. P., & Vian, J. L. (2016) · IEEE Control Systems · doi:10.1109/mcs.2016.2602090 · Cited by 21
The implementation of augmented reality in a robotic teleoperation system
Lin, Y., Song, S., & Meng, M. Q. ‐. (2016) · IEEE International Conference on Real-time Computing and Robotics (RCAR) · doi:10.1109/rcar.2016.7784014 · Cited by 17
Safety Aura Visualization for Variable Impedance Actuated Robots
Makhataeva, Z., Zhakatayev, A., & Varol, H. A. (2019) · IEEE/SICE International Symposium on System Integration (SII) · doi:10.1109/sii.2019.8700332 · Cited by 17
Self-citationRobotic arm control using hybrid brain-machine interface and augmented reality feedback
Wang, Y., Zeng, H., Song, A., Xu, B., Li, H., Zhu, L., Wen, P., & Liu, J. (2017) · International IEEE/EMBS Conference on Neural Engineering · doi:10.1109/ner.2017.8008377 · Cited by 16
Lee, A., Lee, J.-H., & Kim, J. (2015) · ETRI Journal · doi:10.4218/etrij.16.0115.0610 · Cited by 15
Pathomon: A Social Augmented Reality Serious Game
Rapp, D., Müller, J., Bucher, K., & von Mammen, S. (2018) · International Conference on Virtual Worlds and Games for Serious Applications (VS-Games) · doi:10.1109/vs-games.2018.8493437 · Cited by 15
Robot gardens: an augmented reality prototype for plant-robot biohybrid systems
von Mammen, S., Hamann, H., & Heider, M. (2016) · ACM Symposium on Virtual Reality Software and Technology (VRST) · doi:10.1145/2993369.2993400 · Cited by 12
Chan, W. P., Quintero, C. P., Pan, M. K. X. J., Sakr, M., der Loos, H. F. M. V., & Croft, E. A. (2022) · River Publishers eBooks · doi:10.1201/9781003340003-8 · Cited by 11
Finding Objects Faster in Dense Environments Using a Projection Augmented Robotic Arm
Gacem, H., Bailly, G., Eagan, J., & Lecolinet, É. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-22698-9_15 · Cited by 8
Augmented reality interface for taping robot
Dinh, H., Yuan, Q., Vietcheslav, I., & Seet, G. (2017) · International Conference on Advanced Robotics (ICAR) · doi:10.1109/icar.2017.8023530 · Cited by 8
Robust Multithreaded Object Tracker through Occlusions for Spatial Augmented Reality
Lee, A., & Jang, I. (2018) · ETRI Journal · doi:10.4218/etrij.2017-0047 · Cited by 7
Simulation system for myoelectric hand prosthesis using augmented reality
He, Y., Fukuda, O., Ide, S., Okumura, H., Yamaguchi, N., & Bu, N. (2017) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio.2017.8324617 · Cited by 6
Cited by
A Review of Extended Reality (XR) Technologies for Manufacturing Training
Doolani, S., Wessels, C., Kanal, V., Sevastopoulos, C., Jaiswal, A., Nambiappan, H., & Makedon, F. (2020) · Technologies · doi:10.3390/technologies8040077 · Cited by 333
Eswaran, M., & Bahubalendruni, M. V. A. R. (2022) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2022.09.016 · Cited by 308
Augmented Reality: A Systematic Review of Its Benefits and Challenges in E-learning Contexts
Alzahrani, N. M. (2020) · Applied Sciences · doi:10.3390/app10165660 · Cited by 268
Tugtekin, U., & Odabasi, H. F. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-10912-0 · Cited by 55
Metadata
- Authors
- Makhataeva, Z., & Varol, H.
- Venue
- Robotics
- Publisher
- MDPI AG
- Published
- 2020-04-02
- Volume
- 9
- Issue
- 2
- Pages
- 21
- ISSN
- 2218-6581 (ISSN-L)
- OpenAlex
- W3014471496
- Cited by
- 266
- References
- 76
- FWCI
- 14.70
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.