CoChat
Sign Up
Cited by 107Social Robot Interaction and HRIRobotic Path Planning AlgorithmsEvacuation and Crowd Dynamics

Principles and Guidelines for Evaluating Social Robot Navigation Algorithms

Open access
Francis, Anthony, Pérez-D’Arpino, Claudia, Li, Chengshu, Xia, Fei, Alahi, Alexandre, Alami, Rachid, Bera, Aniket, Biswas, Abhijat +23 more·2025·ACM Transactions on Human-Robot Interaction
DOI 10.1145/3700599
Open this paper in CoChat — free account

Abstract

A major challenge to deploying robots widely is navigation in human-populated environments, commonly referred to as social robot navigation . While the field of social navigation has advanced tremendously in recent years, the fair evaluation of algorithms that tackle social navigation remains hard because it involves not just robotic agents moving in static environments but also dynamic human agents and their perceptions of the appropriateness of robot behavior. In contrast, clear, repeatable, and accessible benchmarks have accelerated progress in fields like computer vision, natural language processing and traditional robot navigation by enabling researchers to fairly compare algorithms, revealing limitations of existing solutions and illuminating promising new directions. We believe the same approach can benefit social navigation. In this article, we pave the road toward common, widely accessible, and repeatable benchmarking criteria to evaluate social robot navigation. Our contributions include (a) a definition of a socially navigating robot as one that respects the principles of safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and responsiveness to context, (b) guidelines for the use of metrics, development of scenarios, benchmarks, datasets, and simulators to evaluate social navigation, and (c) a design of a social navigation metrics framework to make it easier to compare results from different simulators, robots, and datasets.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3700599

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 127 · Cited by 107

Research works printed but not linked: 153

  1. [1][1] T. Kruse, A. K. Pandey, R. Alami, and A. Kirsch, 'Human-aware robot navigation: A survey,' Robotics and Autonomous Systems , vol. 61, no. 12, pp. 17261743, 2013. -
  2. [2][2] J. Rios-Martinez, A. Spalanzani, and C. Laugier, 'From proxemics theory to socially-aware navigation: A survey,' Intl. Journal of Social Robotics , vol. 7, no. 2, pp. 137-153, 2015. -
  3. [3][3] S. Chik, C. Yeong, E. Su, T. Lim, Y. Subramaniam, and P. Chin, 'A review of social-aware navigation frameworks for service robot in dynamic human environments,' Journal of Telecommunication, Electronic and Computer Engineering (JTEC) , vol. 8, no. 11, pp. 41-50, 2016. -
  4. [4][4] K. Charalampous, I. Kostavelis, and A. Gasteratos, 'Recent trends in social aware robot navigation: A survey,' Robotics and Autonomous Systems , vol. 93, pp. 85-104, 2017. -
  5. [5][5] J. Cheng, H. Cheng, M. Q.-H. Meng, and H. Zhang, 'Autonomous navigation by mobile robots in human environments: A survey,' in 2018 IEEE Intl. Conf. on robotics and biomimetics (ROBIO) . IEEE, 2018, pp. 1981-1986. -
  6. [7][7] C. Mavrogiannis, F. Baldini, A. Wang, D. Zhao, P. Trautman, A. Steinfeld, and J. Oh, 'Core challenges of social robot navigation: A survey,' arXiv preprint arXiv:2103.05668 , 2021. - · arXiv
  7. [8][8] R. Mirsky, X. Xiao, J. Hart, and P. Stone, 'Prevention and resolution of conflicts in social navigation-a survey,' arXiv preprint arXiv:2106.12113 , 2021. - · arXiv
  8. [9][9] R. M¨ oller, A. Furnari, S. Battiato, A. H¨ arm¨ a, and G. M. Farinella, 'A survey on human-aware robot navigation,' Robotics and Autonomous Systems , vol. 145, p. 103837, 2021. -
  9. [10][10] J. Wang, W. P. Chan, P. Carreno-Medrano, A. Cosgun, and E. Croft, 'Metrics for evaluating social conformity of crowd navigation algorithms,' in 2022 IEEE Intl. Conf. on Advanced Robotics and Its Social Impacts (ARSO) . IEEE, 2022, pp. 1-6. -
  10. [11][11] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al. , 'Imagenet large scale visual recognition challenge,' Intl. journal of computer vision , vol. 115, pp. 211-252, 2015. -
  11. [12][12] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, 'A large annotated corpus for learning natural language inference,' in Proc. of the 2015 Conf. on Empirical Methods in Natural Language Processing . Lisbon, Portugal: Association for Computational Linguistics, Sep. 2015, pp. 632-642. [Online]. Available: https: //aclanthology.org/D15-1075 -
  12. [13][13] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, 'Squad: 100,000+ questions for machine comprehension of text,' arXiv preprint arXiv:1606.05250 , 2016. - · arXiv
  13. [14][14] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, 'Superglue: A stickier benchmark for general-purpose language understanding systems,' Advances in neural information processing systems , vol. 32, 2019. -
  14. [15][15] P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva et al. , 'On evaluation of embodied navigation agents,' arXiv preprint arXiv:1807.06757 , 2018. - · arXiv
  15. [16][16] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik et al. , 'Habitat: A platform for embodied ai research,' in Proc. of the IEEE/CVF Intl. Conf. on computer vision , 2019, pp. 9339-9347. -
  16. [18][18] M. Deitke, W. Han, A. Herrasti, A. Kembhavi, E. Kolve, R. Mottaghi, J. Salvador, D. Schwenk, E. VanderBilt, M. Wallingford et al. , 'Robothor: An open simulationto-real embodied ai platform,' in Proc. of the IEEE/CVF Conf. on computer vision and pattern recognition , 2020, pp. 3164-3174. -
  17. [19][19] X. Xiao, B. Liu, G. Warnell, and P. Stone, 'Motion planning and control for mobile robot navigation using machine learning: a survey,' Autonomous Robots , vol. 46, no. 5, pp. 569-597, 2022. -
  18. [20][20] M. Deitke, D. Batra, Y. Bisk, T. Campari, A. X. Chang, D. S. Chaplot, C. Chen, C. P. D'Arpino, K. Ehsani, A. Farhadi et al. , 'Retrospectives on the embodied ai workshop,' arXiv preprint arXiv:2210.06849 , 2022. - · arXiv
  19. [21][21] L. K¨ astner, T. Bhuiyan, T. A. Le, E. Treis, J. Cox, B. Meinardus, J. Kmiecik, R. Carstens, D. Pichel, B. Fatloun et al. , 'Arena-bench: A benchmarking suite for obstacle avoidance approaches in highly dynamic environments,' IEEE RAL , vol. 7, no. 4, pp. 9477-9484, 2022. -
  20. [23][23] A. Nair, F. Jiang, K. Hou, Z. Xu, S. Li, X. Xiao, and P. Stone, 'Dynabarn: Benchmarking metric ground navigation in dynamic environments,' in 2022 IEEE Intl. Symposium on Safety, Security, and Rescue Robotics (SSRR) . IEEE, 2022, pp. 347-352. -
  21. [24][24] M. Everett, Y. F. Chen, and J. P. How, 'Motion planning among dynamic, decision-making agents with deep reinforcement learning,' in IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) , Madrid, Spain, Sep. 2018. [Online]. Available: https: //arxiv.org/pdf/1805.01956.pdf -
  22. [25][25] N. P´ erez-Higueras, R. Otero, F. Caballero, and L. Merino, 'Hunavsim: A ros 2 human navigation simulator for benchmarking human-aware robot navigation,' arXiv preprint arXiv:2305.01303 , 2023. - · arXiv
  23. [26][26] C. Li, F. Xia, R. Mart´ ın-Mart´ ın, M. Lingelbach, S. Srivastava, B. Shen, K. Vainio, C. Gokmen, G. Dharan, T. Jain et al. , 'igibson 2.0: Object-centric simulation for robot learning of everyday household tasks,' arXiv preprint arXiv:2108.03272 , 2021. - · arXiv
  24. [27][27] B. Shen, F. Xia, C. Li, R. Mart´ ın-Mart´ ın, L. Fan, G. Wang, C. P´ erez-D'Arpino, S. Buch, S. Srivastava, L. Tchapmi et al. , 'igibson 1.0: A simulation environment for interactive tasks in large realistic scenes,' in 2021 IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) . IEEE, 2021, pp. 7520-7527. -
  25. [28][28] A. Biswas, A. Wang, G. Silvera, A. Steinfeld, and H. Admoni, 'Socnavbench: A grounded simulation testing framework for evaluating social navigation,' ACM Trans. on Human-Robot Interaction (THRI) , vol. 11, no. 3, pp. 1-24, 2022. -
  26. [29][29] A. Biswas, N. Tsoi, A. Wang, P. Yu, X. He, L. Fu, G. Silvera, M. Vazquez, and A. Steinfeld, 'SEANavBench @ ICRA 2023 workshop website,' https://seannavbench2022.netlify.app/benchmark/ overview, 2022, [Online; accessed 1-Jan-2023]. -
  27. [30][30] S. Pirk, E. Lee, X. Xiao, L. Takayama, A. Francis, and A. Toshev, 'A protocol for validating social navigation policies,' arXiv preprint arXiv:2204.05443 , 2022. - · arXiv
  28. [31][31] X. Xiao, T. Zhang, K. M. Choromanski, T.-W. E. Lee, A. Francis, J. Varley, S. Tu, S. Singh, P. Xu, F. Xia, S. M. Persson, L. Takayama, R. Frostig, J. Tan, C. Parada, and V. Sindhwani, 'Learning model predictive controllers with real-time attention for real-world navigation,' in Conf. on Robot Learning . PMLR, 2022. -
  29. [32][32] C. Cuan, E. Lee, E. Fisher, A. Francis, L. Takayama, T. Zhang, A. Toshev, and S. Pirk, 'Gesture2path: Imitation learning for gesture-aware navigation,' arXiv preprint arXiv:2209.09375 , 2022. - · arXiv
  30. [34][34] T. Kanda and H. Ishiguro, Human-robot interaction in social robotics . CRC Press, 2017. -
  31. [35][35] C. Recchiuto and A. Sgorbissa, 'Diversity-aware social robots meet people: beyond context-aware embodied ai,' arXiv preprint arXiv:2207.05372 , 2022. - · arXiv
  32. [36][36] P. A. Lasota, T. Fong, J. A. Shah et al. , 'A survey of methods for safe human-robot interaction,' Foundations and Trends® in Robotics , vol. 5, no. 4, pp. 261-349, 2017. -
  33. [37][37] P. Boddington, 'Epsrc principles of robotics: commentary on safety, robots as products, and responsibility,' Connection Science , vol. 29, no. 2, pp. 170-176, 2017. -
  34. [38][38] A. Bera, T. Randhavane, R. Prinja, and D. Manocha, 'Sociosense: Robot navigation amongst pedestrians with social and psychological constraints,' in 2017 IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) . IEEE, 2017, pp. 7018-7025. -
  35. [39][39] V. Narayanan, B. M. Manoghar, V. S. Dorbala, D. Manocha, and A. Bera, 'Proxemo: Gait-based emotion learning and multi-view proxemic fusion for socially-aware robot navigation,' in 2020 IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) . IEEE, 2020, pp. 8200-8207. -
  36. [40][40] E. T. Hall, The hidden dimension . Garden City, NY: Doubleday, 1966, vol. 609. -
  37. [41][41] A. D. Dragan, K. C. Lee, and S. S. Srinivasa, 'Legibility and predictability of robot motion,' in HRI . IEEE, 2013. -
  38. [42][42] O. Inbar and J. Meyer, 'Politeness counts: Perceptions of peacekeeping robots,' IEEE Trans. on HumanMachine Systems , vol. 49, no. 3, pp. 232-240, 2019. -
  39. [43][43] T. Randhavane, A. Bera, E. Kubin, A. Wang, K. Gray, and D. Manocha, 'Pedestrian dominance modeling for socially-aware robot navigation,' in 2019 Intl. Conf. on Robotics and Automation (ICRA) . IEEE, 2019, pp. 5621-5628. -
  40. [44][44] E. Cheung, A. Bera, E. Kubin, K. Gray, and D. Manocha, 'Identifying driver behaviors using trajectory features for vehicle navigation,' in 2018 IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) . IEEE, 2018, pp. 3445-3452. -
  41. [45][45] Y. F. Chen, M. Everett, M. Liu, and J. P. How, 'Socially aware motion planning with deep reinforcement learning,' in 2017 IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) . IEEE, 2017, pp. 1343-1350. -
  42. [46][46] H. Kose-Bagci, K. Dautenhahn, and C. L. Nehaniv, 'Emergent dynamics of turn-taking interaction in drumming games with a humanoid robot,' in RO-MAN . IEEE, 2008. -
  43. [47][47] B. Petrak, G. Sopper, K. Weitz, and E. Andr´ e, 'Do you mind if I pass through? Studying the appropriate robot behavior when traversing two conversing people in a hallway setting,' in 2021 30th IEEE Intl. Conf. on Robot & Human Interactive Communication (ROMAN) . IEEE, 2021, pp. 369-375. -
  44. [48][48] P. Trautman and A. Krause, 'Unfreezing the robot: Navigation in dense, interacting crowds,' in 2010 IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems . IEEE, 2010, pp. 797-803. -
  45. [49][49] A. Bera, T. Randhavane, E. Kubin, A. Wang, K. Gray, and D. Manocha, 'The socially invisible robot navigation in the social world using robot entitativity,' in 2018 ieee/rsj Intl. Conf. on intelligent robots and systems (iros) . IEEE, 2018, pp. 4468-4475. -
  46. [50][50] R. Mead and M. J. Matari´ c, 'Autonomous human-robot proxemics: socially aware navigation based on interaction potential,' Autonomous Robots , vol. 41, no. 5, pp. 1189-1201, 2017. -
  47. [51][51] R. Chandra, R. Maligi, A. Anantula, and J. Biswas, 'Socialmapf: Optimal and efficient multi-agent path finding with strategic agents for social navigation,' IEEE RAL , 2023. -
  48. [52][52] N. Suriyarachchi, R. Chandra, J. S. Baras, and D. Manocha, 'Gameopt: Optimal real-time multi-agent planning and control for dynamic intersections,' in 2022 IEEE 25th Intl. Conf. on Intelligent Transportation Systems (ITSC) . IEEE, 2022, pp. 2599-2606. -
  49. [53][53] R. Chandra and D. Manocha, 'Gameplan: Gametheoretic multi-agent planning with human drivers at intersections, roundabouts, and merging,' IEEE RAL , vol. 7, no. 2, pp. 2676-2683, 2022. -
  50. [54][54] R. Chandra, A. Bera, and D. Manocha, 'Using graphtheoretic machine learning to predict human driver behavior,' IEEE Trans. on Intelligent Transportation Systems , vol. 23, no. 3, pp. 2572-2585, 2021. -
  51. [55][55] R. Chandra, M. Wang, M. Schwager, and D. Manocha, 'Game-theoretic planning for autonomous driving among risk-aware human drivers,' in 2022 Intl. Conf. on Robotics and Automation (ICRA) . IEEE, 2022, pp. 2876-2883. -
  52. [56][56] A. Bera, S. Kim, T. Randhavane, S. Pratapa, and D. Manocha, 'Glmp-realtime pedestrian path prediction using global and local movement patterns,' in 2016 IEEE Intl. Conf. on Robotics and Automation (ICRA) . IEEE, 2016, pp. 5528-5535. -
  53. [57][57] A. Nigam and L. D. Riek, 'Social context perception for mobile robots,' in 2015 IEEE/RSJ Intl. Conf. on intelligent robots and systems (IROS) . IEEE, 2015, pp. 3621-3627. -
  54. [58][58] R. Baghel, A. Kapoor, P. Bachiller, R. R. Jorvekar, D. Rodriguez-Criado, and L. J. Manso, 'A toolkit to generate social navigation datasets,' in Advances in Physical Agents II: Proc. of the 21st Intl. Workshop of Physical Agents (WAF 2020) . Springer, 2021, pp. 180-193. -
  55. [59][59] R. Meyes, M. Lu, C. W. de Puiseau, and T. Meisen, 'Ablation studies in artificial neural networks,' arXiv preprint arXiv:1901.08644 , 2019. - · arXiv
  56. [60][60] G. Hoffman and X. Zhao, 'A primer for conducting experiments in human-robot interaction,' ACM Trans. on Human-Robot Interaction (THRI) , vol. 10, no. 1, pp. 1-31, 2020. -
  57. [61][61] T. Kanda, H. Ishiguro, and T. Ishida, 'Psychological analysis on human-robot interaction,' in Proc. 2001 ICRA. IEEE Intl. Conf. on Robotics and Automation (Cat. No. 01CH37164) , vol. 4. IEEE, 2001, pp. 41664173. -
  58. [62][62] A. Kim, J. Han, Y. Jung, and K. Lee, 'The effects of familiarity and robot gesture on user acceptance of information,' in 2013 8th ACM/IEEE Intl. Conf. on Human-Robot Interaction (HRI) . IEEE, 2013, pp. 159160. -
  59. [63][63] M. Paetzel, G. Perugia, and G. Castellano, 'The persistence of first impressions: The effect of repeated interactions on the perception of a social robot,' in Proc. of the 2020 ACM/IEEE Intl. Conf. on humanrobot interaction , 2020, pp. 73-82. -
  60. [64][64] S. Saunderson and G. Nejat, 'Robots asking for favors: The effects of directness and familiarity on persuasive hri,' IEEE RAL , vol. 6, no. 2, pp. 1793-1800, 2021. -
  61. [65][65] J. Biswas and M. Veloso, 'The 1,000-km challenge: Insights and quantitative and qualitative results,' IEEE Intelligent Systems , vol. 31, no. 3, pp. 86-96, 2016. -
  62. [66][66] M. T. Orne, 'On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications.' American psychologist , vol. 17, no. 11, p. 776, 1962. -
  63. [67][67] M. A. Schmuckler, 'What is ecological validity? a dimensional analysis,' Infancy , vol. 2, no. 4, pp. 419436, 2001. -
  64. [68][68] J. F. Kihlstrom, 'Ecological validity and 'ecological validity',' Perspectives on Psychological Science , vol. 16, no. 2, pp. 466-471, 2021. -
  65. [70][70] M. S. Litwin and A. Fink, How to measure survey reliability and validity . Sage, 1995, vol. 7. -
  66. [72][72] H. Kamide, Y. Mae, K. Kawabe, S. Shigemi, M. Hirose, and T. Arai, 'New measurement of psychological safety for humanoid,' in 2012 7th ACM/IEEE Intl. Conf. on Human-Robot Interaction (HRI) . IEEE, 2012, pp. 4956. -
  67. [73][73] N. P´ erez-Higueras, R. Ram´ on-Vigo, F. Caballero, and L. Merino, 'Robot local navigation with learned social cost functions,' in 2014 11th Intl. Conf. on Informatics in Control, Automation and Robotics (ICINCO) , vol. 2. IEEE, 2014, pp. 618-625. -
  68. [74][74] L. J. Manso, P. Nu˜ nez, L. V. Calderita, D. R. Faria, and P. Bachiller, 'Socnav1: A dataset to benchmark and learn social navigation conventions,' Data , vol. 5, no. 1, p. 7, 2020. -
  69. [75][75] P. Bachiller, D. Rodriguez-Criado, R. R. Jorvekar, P. Bustos, D. R. Faria, and L. J. Manso, 'A graph neural network to model disruption in human-aware robot navigation,' Multimedia Tools and Applications , vol. 81, no. 3, pp. 3277-3295, 2022. -
  70. [76][76] H.-T. L. Chiang, A. Faust, M. Fiser, and A. Francis, 'Learning Navigation Behaviors End-to-end with AutoRL,' IEEE RAL , vol. 4, no. 2, pp. 2007-2014, 2019. -
  71. [77][77] A. V. Taylor, E. Mamantov, and H. Admoni, 'Observeraware legibility for social navigation,' in 2022 31st IEEE Intl. Conf. on Robot and Human Interactive Communication (RO-MAN) . IEEE, 2022, pp. 11151122. -
  72. [78][78] E. Marder-Eppstein, E. Berger, T. Foote, B. Gerkey, and K. Konolige, 'The office marathon: Robust navigation in an indoor office environment,' in 2010 IEEE Intl. Conf. on robotics and automation . IEEE, 2010, pp. 300-307. -
  73. [79][79] A. Rosenthal-von der P¨ utten, D. Sirkin, A. Abrams, and L. Platte, 'The forgotten in hri: Incidental encounters with robots in public spaces,' in Companion of the 2020 ACM/IEEE Intl. Conf. on Human-Robot Interaction , 2020, pp. 656-657. -
  74. [80][80] R. Fernandez, N. John, S. Kirmani, J. Hart, J. Sinapov, and P. Stone, 'Passive demonstrations of light-based robot signals for improved human interpretability,' in 2018 27th IEEE Intl. Symposium on Robot and Human Interactive Communication (RO-MAN) . IEEE, 2018, pp. 234-239. -
  75. [81][81] R. Gockley, A. Bruce, J. Forlizzi, M. Michalowski, A. Mundell, S. Rosenthal, B. Sellner, R. Simmons, K. Snipes, A. C. Schultz et al. , 'Designing robots for long-term social interaction,' in 2005 IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems . IEEE, 2005, pp. 1338-1343. -
  76. [82][82] J. Hart, E. Hauser, S. Baker, J. Biswas, J. Jiao, and L. Sentis, 'Longitudinal social impacts of hri over longterm deployments,' in 2022 17th ACM/IEEE Intl. Conf. on Human-Robot Interaction (HRI) . IEEE, 2022, pp. 1258-1260. -
  77. [84][84] R. G. Smart, 'Subject selection bias in psychological research.' Canadian Psychologist/Psychologie canadienne , vol. 7, no. 2, p. 115, 1966. -
  78. [85][85] D. P. Schultz, 'The human subject in psychological research.' Psychological bulletin , vol. 72, no. 3, p. 214, 1969. -
  79. [86][86] R. Feingold-Polak, O. Barzel, and S. Levy-Tzedek, 'A robot goes to rehab: a novel gamified system for long-term stroke rehabilitation using a socially assistive robot-methodology and usability testing,' Journal of NeuroEngineering and Rehabilitation , vol. 18, no. 1, pp. 1-18, 2021. -
  80. [87][87] J. Hart, R. Mirsky, X. Xiao, S. Tejeda, B. Mahajan, J. Goo, K. Baldauf, S. Owen, and P. Stone, 'Using human-inspired signals to disambiguate navigational intentions,' in Social Robotics: 12th Intl. Conf., ICSR 2020, Golden, CO, USA, November 14-18, 2020, Proc. Springer, 2020, pp. 320-331. -
  81. [88][88] E. Cancelli, T. Campari, L. Serafini, A. X. Chang, and L. Ballan, 'Exploiting socially-aware tasks for embodied social navigation,' arXiv preprint arXiv:2212.00767 , 2022. - · arXiv
  82. [90][90] A. Francis, A. Faust, H.-T. L. Chiang, J. Hsu, J. C. Kew, M. Fiser, and T.-W. E. Lee, 'Long-range indoor navigation with prm-rl,' IEEE Trans. on Robotics , vol. 36, no. 4, pp. 1115-1134, 2020. -
  83. [92][92] N. Spatola, B. K¨ uhnlenz, and G. Cheng, 'Perception and evaluation in human-robot interaction: The humanrobot interaction evaluation scale (hries)-a multicomponent approach of anthropomorphism,' Intl. Journal of Social Robotics , vol. 13, no. 7, pp. 1517-1539, 2021. -
  84. [93][93] K. Backor, S. Golde, and N. Nie, 'Estimating survey fatigue in time use study,' in Intl. association for time use research Conf.. Washington, DC . Citeseer, 2007. -
  85. [94][94] M. Galesic and M. Bosnjak, 'Effects of questionnaire length on participation and indicators of response quality in a web survey,' Public opinion quarterly , vol. 73, no. 2, pp. 349-360, 2009. -
  86. [95][95] S. R. Porter, M. E. Whitcomb, and W. H. Weitzer, 'Multiple surveys of students and survey fatigue,' New directions for institutional research , vol. 2004, no. 121, pp. 63-73, 2004. -
  87. [97][97] P. R. Cohen, Empirical methods for artificial intelligence . MIT press Cambridge, MA, 1995, vol. 139. -
  88. [98][98] D. T. Lykken, 'Statistical significance in psychological research.' Psychological bulletin , vol. 70, no. 3p1, p. 151, 1968. -
  89. [99][99] B. G. Tabachnick and L. S. Fidell, Experimental designs using ANOVA . Thomson/Brooks/Cole Belmont, CA, 2007, vol. 724. -
  90. [100][100] M. Tavakol and R. Dennick, 'Making sense of cronbach's alpha,' Intl. journal of medical education , vol. 2, p. 53, 2011. -
  91. [101][101] X.-T. Truong and T. D. Ngo, 'Toward socially aware robot navigation in dynamic and crowded environments: A proactive social motion model,' IEEE Trans. on Automation Science and Engineering , vol. 14, no. 4, pp. 1743-1760, 2017. -
  92. [102][102] X.-T. Truong and T.-D. Ngo, ''to approach humans?': A unified framework for approaching pose prediction and socially aware robot navigation,' IEEE Trans. on Cognitive and Developmental Systems , vol. 10, no. 3, pp. 557-572, 2017. -
  93. [103][103] P. T. Singamaneni, 'Combining proactive planning and situation analysis for human-aware robot navigation,' Theses, Universit´ e Paul Sabatier -Toulouse III, Dec. 2022. [Online]. Available: https://theses.hal.science/tel04006782 -
  94. [104][104] P.-T. Singamaneni, A. Favier, and R. Alami, 'Towards benchmarking human-aware robot navigation: A new perspective and metrics,' in 2023 32nd IEEE Intl. Conf. on Robot and Human Interactive Communication (ROMAN) . IEEE, 2023. -
  95. [105][105] H. Khambhaita and R. Alami, 'Viewing robot navigation in human environment as a cooperative activity,' in Robotics Research , 2020, pp. 285-300. -
  96. [106][106] X. Li and F.-Y. Wang, 'Scenarios engineering: Enabling trustworthy and effective ai for autonomous vehicles,' IEEE Transactions on Intelligent Vehicles , vol. 8, no. 5, pp. 3205-3210, 2023. -
  97. [107][107] X. Li, P. Ye, J. Li, Z. Liu, L. Cao, and F.-Y. Wang, 'From features engineering to scenarios engineering for trustworthy ai: I&i, c&c, and v&v,' IEEE Intelligent Systems , vol. 37, no. 4, pp. 18-26, 2022. -
  98. [108][108] X. Li, Y. Tian, P. Ye, H. Duan, and F.-Y. Wang, 'A novel scenarios engineering methodology for foundation models in metaverse,' IEEE Transactions on Systems, Man, and Cybernetics: Systems , vol. 53, no. 4, pp. 21482159, 2022. -
  99. [109][109] Y. F. Chen, M. Liu, M. Everett, and J. P. How, 'Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning,' in 2017 IEEE Intl. Conf. on robotics and automation (ICRA) . IEEE, 2017, pp. 285-292. -
  100. [111][111] B. Okal and K. O. Arras, 'Learning socially normative robot navigation behaviors with bayesian inverse reinforcement learning,' in Robotics and Automation (ICRA), 2016 IEEE Intl. Conf. on . IEEE, 2016, pp. 2889-2895. -
  101. [112][112] N. J. Wahl, 'An overview of regression testing,' ACM SIGSOFT Software Engineering Notes , vol. 24, no. 1, pp. 69-73, 1999. -
  102. [113][113] N. Koenig and A. Howard, 'Design and use paradigms for gazebo, an open-source multi-robot simulator,' in 2004 IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566) , vol. 3. IEEE, 2004, pp. 2149-2154. -
  103. [114][114] C. Gloor, 'Pedsim: Pedestrian crowd simulation,' URL http://pedsim. silmaril. org , vol. 5, no. 1, 2016. -
  104. [115][115] D. Helbing and P. Molnar, 'Social force model for pedestrian dynamics,' Physical review E , vol. 51, no. 5, p. 4282, 1995. -
  105. [116][116] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, A. Y. Ng et al. , 'Ros: an open- source robot operating system,' in ICRA workshop on open source software , vol. 3, no. 3.2. Kobe, Japan, 2009, p. 5. -
  106. [117][117] C. Rosmann, A. Makarow, F. Hoffmann, and T. Bertram, 'Time-optimal nonlinear model predictive control with minimal control interventions,' in 2017 IEEE Conf. on Control Technology and Applications (CCTA) . IEEE, 2017, pp. 19-24. -
  107. [118][118] D. Fox, W. Burgard, and S. Thrun, 'The dynamic window approach to collision avoidance,' IEEE Robotics & Automation Magazine , vol. 4, no. 1, pp. 23-33, 1997. -
  108. [119][119] C. R¨ osmann, W. Feiten, T. W¨ osch, F. Hoffmann, and T. Bertram, 'Trajectory modification considering dynamic constraints of autonomous robots,' in ROBOTIK 2012; 7th German Conf. on Robotics . VDE, 2012, pp. 1-6. -
  109. [120][120] D. Dugas, J. Nieto, R. Siegwart, and J. J. Chung, 'Navrep: Unsupervised representations for reinforcement learning of robot navigation in dynamic human environments,' in 2021 IEEE Intl. Conf. on Robotics and Automation (ICRA) . IEEE, 2021, pp. 7829-7835. -
  110. [121][121] R. Guldenring, M. G¨ orner, N. Hendrich, N. J. Jacobsen, and J. Zhang, 'Learning local planners for human-aware navigation in indoor environments,' in 2020 IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) . IEEE, 2020, pp. 6053-6060. -
  111. [122][122] D. Perille, A. Truong, X. Xiao, and P. Stone, 'Benchmarking metric ground navigation,' in 2020 IEEE Intl. Symposium on Safety, Security, and Rescue Robotics (SSRR) . IEEE, 2020, pp. 116-121. -
  112. [123][123] D. A. Pomerleau, 'Alvinn: An autonomous land vehicle in a neural network,' in Advances in neural information processing systems , 1989, pp. 305-313. -
  113. [124][124] S. Fujimoto, H. Hoof, and D. Meger, 'Addressing function approximation error in actor-critic methods,' in Intl. Conf. on machine learning . PMLR, 2018, pp. 1587-1596. -
  114. [125][125] M. Everett, Y. F. Chen, and J. P. How, 'Collision avoidance in pedestrian-rich environments with deep reinforcement learning,' IEEE Access , vol. 9, pp. 10 35710 377, 2021. -
  115. [126][126] J. Van Den Berg, S. J. Guy, M. Lin, and D. Manocha, 'Reciprocal n-body collision avoidance,' in Robotics research . Springer, 2011, pp. 3-19. -
  116. [127][127] P. Long, T. Fan, X. Liao, W. Liu, H. Zhang, and J. Pan, 'Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning,' in 2018 IEEE Intl. Conf. on robotics and automation (ICRA) . IEEE, 2018, pp. 6252-6259. -
  117. [128][128] M. Colledanchise and P. ¨ Ogren, Behavior trees in robotics and AI: An introduction . CRC Press, 2018. -
  118. [129][129] A. Lerner, Y. Chrysanthou, and D. Lischinski, 'Crowds by example,' in Computer graphics forum , vol. 26, no. 3. Wiley Online Library, 2007, pp. 655-664. -
  119. [130][130] S. Pellegrini, A. Ess, K. Schindler, and L. Van Gool, 'You'll never walk alone: Modeling social behavior for multi-target tracking,' in 2009 IEEE 12th Intl. Conf. on computer vision . IEEE, 2009, pp. 261-268. -
  120. [132][132] C. P´ erez-D'Arpino, C. Liu, P. Goebel, R. Mart´ ınMart´ ın, and S. Savarese, 'Robot navigation in constrained pedestrian environments using reinforcement learning,' IEEE ICRA , 2021. -
  121. [133][133] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra, 'Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames,' arXiv preprint arXiv:1911.00357 , 2019. - · arXiv
  122. [135][135] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, 'Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,' in ICML . PMLR, 2018. -
  123. [136][136] N. Tsoi, M. Hussein, O. Fugikawa, J. Zhao, and M. V´ azquez, 'An approach to deploy interactive robotic simulators on the web for hri experiments: Results in social robot navigation,' in 2021 IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) . IEEE, 2021, pp. 7528-7535. -
  124. [137][137] E. Heiden, L. Palmieri, L. Bruns, K. O. Arras, G. S. Sukhatme, and S. Koenig, 'Bench-mr: A motion planning benchmark for wheeled mobile robots,' IEEE RAL , vol. 6, no. 3, pp. 4536-4543, 2021. -
  125. [138][138] R. Mart´ ın-Mart´ ın, H. Rezatofighi, A. Shenoi, M. Patel, J. Gwak, N. Dass, A. Federman, P. Goebel, and S. Savarese, 'Jrdb: A dataset and benchmark for visual perception for navigation in human environments,' arXiv preprint arXiv:1910.11792 , 2019. - · arXiv
  126. [139][139] A. Rudenko, T. P. Kucner, C. S. Swaminathan, R. T. Chadalavada, K. O. Arras, and A. J. Lilienthal, 'Th¨ or: Human-robot navigation data collection and accurate motion trajectories dataset,' IEEE RAL , vol. 5, no. 2, pp. 676-682, 2020. -
  127. [140][140] P. Kothari, S. Kreiss, and A. Alahi, 'Human trajectory forecasting: A deep learning perspective,' IEEE Trans. on Intelligent Transportation Systems , 2021. -
  128. [141][141] B. Majecka, 'Statistical models of pedestrian behaviour in the forum,' Master's thesis, School of Informatics, University of Edinburgh , 2009. -
  129. [142][142] A. Robicquet, A. Sadeghian, A. Alahi, and S. Savarese, 'Learning social etiquette: Human trajectory understanding in crowded scenes,' in European Conf. on computer vision . Springer, 2016, pp. 549-565. -
  130. [143][143] H. S. Park, J.-J. Hwang, Y. Niu, and J. Shi, 'Egocentric future localization,' in Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition , 2016, pp. 4697-4705. -
  131. [144][144] T. Chavdarova, P. Baqu´ e, S. Bouquet, A. Maksai, C. Jose, T. Bagautdinov, L. Lettry, P. Fua, L. Van Gool, and F. Fleuret, 'Wildtrack: A multi-camera hd dataset for dense unscripted pedestrian detection,' in Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition , 2018, pp. 5030-5039. -
  132. [145][145] H. Karnan, A. Nair, X. Xiao, G. Warnell, S. Pirk, A. Toshev, J. Hart, J. Biswas, and P. Stone, 'Socially Compliant Navigation Dataset (SCAND),' 2022. [Online]. Available: https://doi.org/10.18738/T8/ 0PRYRH - · DOI
  133. [146][146] --, 'Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,' IEEE RAL , 2022. -
  134. [147][147] D. M. Nguyen, M. Nazeri, A. Payandeh, A. Datar, and X. Xiao, 'Toward human-like social robot navigation: A large-scale, multi-modal, social human navigation dataset,' arXiv preprint arXiv:2303.14880 , 2023. - · arXiv
  135. [148][148] D. Paez-Granados, Y. He, D. Gonon, L. Huber, and A. Billard, '3d point cloud and rgbd of pedestrians in robot crowd navigation: Detection and tracking,' IEEE DataPort , vol. 12, 2021. -
  136. [150][150] R. Bhaskara, M. Chiu, and A. Bera, 'Sg-lstm: Social group lstm for robot navigation through dense crowds,' in 2017 IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) . IEEE, 2023. -
  137. [152][152] Z. Yan, T. Duckett, and N. Bellotto, 'Online learning for human classification in 3d lidar-based tracking,' in 2017 IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) . IEEE, 2017, pp. 864-871. -
  138. [153][153] A. Alahi, V. Ramanathan, and L. Fei-Fei, 'Sociallyaware large-scale crowd forecasting,' in Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition , 2014, pp. 2203-2210. -
  139. [154][154] D. F. Paez-Granados, H. Kadone, M. Hassan, Y. Chen, and K. Suzuki, 'Personal mobility with synchronous trunk-knee passive exoskeleton: Optimizing human-robot energy transfer,' IEEE/ASME Trans. on Mechatronics , vol. 27, no. 5, pp. 3613-3623, 2022. -
  140. [155][155] D. F. P. Granados, H. Kadone, and K. Suzuki, 'Unpowered lower-body exoskeleton with torso lifting mechanism for supporting sit-to-stand transitions,' in 2018 IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) , 2018, pp. 2755-2761. -
  141. [156][156] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, 'Openai gym,' arXiv preprint arXiv:1606.01540 , 2016. - · arXiv
  142. [157][157] C. Chen, Y. Liu, S. Kreiss, and A. Alahi, 'Crowdrobot interaction: Crowd-aware robot navigation with attention-based deep reinforcement learning,' in 2019 Intl. Conf. on Robotics and Automation (ICRA) . IEEE, 2019, pp. 6015-6022. -
  143. [158][158] Y. Liu, S. Li, C. Li, C. Perez-D'Arpino, and S. Savarese, 'Interactive pedestrian simulation in igibson,' in RSS Workshop on Social Robot Navigation , 2021. -
  144. [159][159] A. Favier, P. T. Singamaneni, and R. Alami, 'An intelligent human avatar to debug and challenge human-aware robot navigation systems,' in 2022 17th ACM/IEEE Intl. Conf. on Human-Robot Interaction (HRI) . IEEE, 2022, pp. 760-764. -
  145. [160][160] O. Hauterville, C. Fern´ andez, P. T. Singamaneni, A. Favier, V. Matell´ an, and R. Alami, 'Interactive social agents simulation tool for designing choreographies for human-robot-interaction research,' in ROBOT2022: Fifth Iberian Robotics Conf.: Advances in Robotics, Volume 2 . Springer, 2022, pp. 514-527. -
  146. [161][161] A. Aroor, S. L. Esptein, and R. Korpan, 'Mengeros: A crowd simulation tool for autonomous robot navigation,' in 2017 AAAI Fall Symposium Series , 2017. -
  147. [162][162] N. Tsoi, M. Hussein, J. Espinoza, X. Ruiz, and M. V´ azquez, 'Sean: Social environment for autonomous navigation,' in Proc. of the 8th Intl. Conf. on HumanAgent Interaction , 2020, pp. 281-283. -
  148. [163][163] J. Holtz and J. Biswas, ' SOCIALGYM: A Framework for Benchmarking Social Robot Navigation ,' in Intelligent Robots and Systems (IROS), IEEE/RSJ Intl. Conf. on . IEEE, 2022, pp. 11 246-11 252. -
  149. [164][164] Z. Sprague, R. Chandra, J. Holtz, and J. Biswas, 'Socialgym 2.0: Simulator for multi-agent social robot navigation in shared human spaces,' arXiv preprint arXiv:2303.05584 , 2023. - · arXiv
  150. [165][165] J. Terry, B. Black, N. Grammel, M. Jayakumar, A. Hari, R. Sullivan, L. S. Santos, C. Dieffendahl, C. Horsch, R. Perez-Vicente et al. , 'Pettingzoo: Gym for multiagent reinforcement learning,' Advances in Neural Information Processing Systems , vol. 34, pp. 15 03215 043, 2021. -
  151. [166][166] C. Mavrogiannis, A. M. Hutchinson, J. Macdonald, P. Alves-Oliveira, and R. A. Knepper, 'Effects of distinct robot navigation strategies on human behavior in a crowded environment,' in 2019 14th ACM/IEEE Intl. Conf. on Human-Robot Interaction (HRI) . IEEE, 2019, pp. 421-430. -
  152. [167][167] N. Tsoi, M. Hussein, O. Fugikawa, J. D. Zhao, and M. V´ azquez, 'An approach to deploy interactive robotic simulators on the web for hri experiments: Results in social robot navigation,' in IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) . IEEE, 2021. -
  153. [168][168] A. Francis, C. P´ erez-D'Arpino, C. Li, F. Xia, A. Alahi, A. Bera, A. Biswas, J. Biswas, H.-T. L. Chiang, M. Everett, S. Ha, J. Hart, H. Karnan, T.-W. E. Lee, L. J. Manso, R. Mirsky, S. Pirk, P. T. Singamaneni, P. Stone, A. Taylor, P. Trautman, N. Tsoi, M. V´ azquez, X. Xiao, P. Xu, N. Yokoyama, R. Mart´ ın-Mart´ ın, and A. Toshev, 'Benchmarking social robot navigation across academia and industry,' in Proc. of the AAAI 2023 Spring Symposium on HRI in Academia and Industry: Bridging the Gap. AAAI, 2023.

Authors
Francis, A., Pérez-D’Arpino, C., Li, C., Xia, F., Alahi, A., Alami, R., Bera, A., Biswas, A., Biswas, J., Chandra, R., Chiang, H.-T. L., Everett, M., Ha, S., Hart, J., How, J. P., Karnan, H., Lee, T.-W. E., Manso, L. J., Mirsky, R., Pirk, S., Singamaneni, P. T., Stone, P., Taylor, A. V., Trautman, P., Tsoi, N., Vázquez, M., Xiao, X., Xu, P., Yokoyama, N., Toshev, A., & Martín-Martín, R.
Affiliations
  • Logical Robotics, Simpsonville, South Carolina, USA
  • Nvidia (United States) (US)
  • Stanford University (US)
  • Google (United States) (US)
  • École Polytechnique Fédérale de Lausanne (CH)
  • Centre National de la Recherche Scientifique (FR)
  • Purdue University West Lafayette (US)
  • Carnegie Mellon University (US)
  • The University of Texas at Austin (US)
  • University of Virginia (US)
  • Northeastern University (US)
  • Georgia Institute of Technology (US)
  • Massachusetts Institute of Technology (US)
  • Aston University (GB)
  • Tufts University (US)
  • Christian-Albrechts-Universität zu Kiel (DE)
  • Honda Research Institute, USA (US)
  • Yale University (US)
  • George Mason University (US)
  • Apple (United States) (US)
ORCID
Venue
ACM Transactions on Human-Robot Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2025-02-20
Volume
14
Issue
2
Pages
1-65
ISSN
2573-9522 (ISSN-L)
DOI
10.1145/3700599
OpenAlex
W4384625456
Cited by
107
References
127
Citations by year
2026: 40 · 2025: 44 · 2024: 19 · 2023: 4
FWCI
48.87
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Transactions on Human-Robot Interaction (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Aston Publications Explorer (Aston University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotic Path Planning Algorithms
  • Physical Sciences › Engineering › Ocean Engineering › Evacuation and Crowd Dynamics
Keywords
social robot navigation, benchmarking, evaluation metrics, human-populated environments, human-robot interaction, robot safety, human comfort, legibility, politeness, social competence, proactive behavior, simulators, datasets

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review