CoChat
Sign Up
PreprintCited by 97Human-Automation Interaction and SafetyTeam Dynamics and PerformanceErgonomics and Human Factors

The DOI matches its DataCite record.

Common Metrics to Benchmark Human-Machine Teams (HMT): A Review

Open access
Damacharla, Praveen, Javaid, Ahmad Y., Gallimore, Jennie J., Devabhaktuni, Vijay K.·2020·arXiv
DOI 10.48550/arxiv.2008.04855arXiv 2008.04855
Open this paper in CoChat — free account

Abstract

A significant amount of work is invested in human-machine teaming (HMT) across multiple fields. Accurately and effectively measuring system performance of an HMT is crucial for moving the design of these systems forward. Metrics are the enabling tools to devise a benchmark in any system and serve as an evaluation platform for assessing the performance, along with the verification and validation, of a system. Currently, there is no agreed-upon set of benchmark metrics for developing HMT systems. Therefore, identification and classification of common metrics are imperative to create a benchmark in the HMT field. The key focus of this review is to conduct a detailed survey aimed at identification of metrics employed in different segments of HMT and to determine the common metrics that can be used in the future to benchmark HMTs. We have organized this review as follows: identification of metrics used in HMTs until now, and classification based on functionality and measuring techniques. Additionally, we have also attempted to analyze all the identified metrics in detail while classifying them as theoretical, applied, real-time, non-real-time, measurable, and observable metrics. We conclude this review with a detailed analysis of the identified common metrics along with their usage to benchmark HMTs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2008.04855

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 144

  1. [3][3] P. Griffiths and R. B. Gillespie, "Shared control between human and machine: Haptic display of automation during manual control of vehicle heading," in Proceedings of 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS'04), , 2004, pp. 358-366: IEEE. -
  2. [4][4] M. Johnson, J. M. Bradshaw, P. J. Feltovich, C. M. Jonker, M. B. Van Riemsdijk, and M. Sierhuis, "Coactive design: Designing support for interdependence in joint activity," Journal of Human-Robot Interaction, vol. 3, no. 1, pp. 43-69, 2014. -
  3. [5][5] R. R. Hoffman, J. K. Hawley, and J. M. Bradshaw, "Myths of automation, part 2: Some very human consequences," IEEE Intelligent Systems, no. 2, pp. 82-85, 2014. 6.
  4. [8][8] J. Ball et al. , "The synthetic teammate project," Computational and Mathematical Organization Theory, vol. 16, no. 3, pp. 271299, 2010. 4.
  5. [9][9] - P. C. Cacciabue, "Elements of human-machine systems," in Guide to Applying Human Factors Methods : Springer, 2004, pp. 9-47.
  6. [23][23] - J. L. Burke, R. R. Murphy, E. Rogers, V. J. Lumelsky, and J. Scholtz, "Final report for the DARPA/NSF interdisciplinary study on human-robot interaction," Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, vol. 34, no. 2, pp. 103-112, 2004. - T. J. Wiltshire and S. M. Fiore, "Social Cognitive and Affective Neuroscience in Human-Machine Systems: A Roadmap for Improving Training, Human-Robot Interaction, and Team Performance," Human-Machine Systems, IEEE Transactions on, vol. 44, no. 6, pp. 779-787, 2014. - S. Nikolaidis, R. Ramakrishnan, K. Gu, and J. Shah, "Efficient model learning from joint-action demonstrations for humanrobot collaborative tasks," in Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction , 2015, pp. 189-196: ACM. - M. C. Gombolay, R. A. Gutierrez, S. G. Clarke, G. F. Sturla, and J. A. Shah, "Decision-making authority, team efficiency and human worker satisfaction in mixed human-robot teams," Autonomous Robots, vol. 39, no. 3, pp. 293-312, 2015. - M. C. Gombolay, R. Wilcox, and J. A. Shah, "Fast Scheduling of Robot Teams Performing Tasks With Temporospatial Constraints," IEEE Transactions on Robotics, vol. 34, no. 1, pp. 220-239, Feb 2018. - J. Kim and J. A. Shah, "Improving Team's Consistency of Understanding in Meetings," IEEE Transactions on HumanMachine Systems, vol. 46, no. 5, pp. 625-637, 2016. - S. Nikolaidis, P. Lasota, R. Ramakrishnan, and J. Shah, "Improved human-robot team performance through crosstraining, an approach inspired by human team training practices," The International Journal of Robotics Research, vol. 34, no. 14, pp. 1711-1730, 2015. - D. J. Bruemmer and M. C. Walton, "Collaborative tools for mixed teams of humans and robots," DTIC Document2003. - C. E. Harriott and J. A. Adams, "Modeling Human Performance for Human-Robot Systems," Reviews of Human Factors and Ergonomics, vol. 9, no. 1, pp. 94-130, 2013. - M. D. Manning, C. E. Harriott, S. T. Hayes, J. A. Adams, and A. E. Seiffert, "Heuristic Evaluation of Swarm Metrics' Effectiveness," in Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts , Portland, Oregon, USA, 2015, pp. 17-18: ACM. - C. E. Harriott, A. E. Seiffert, S. T. Hayes, and J. A. Adams, "Biologically-inspired human-swarm interaction metrics," in Proceedings of the Human Factors and Ergonomics Society Annual Meeting , 2014, vol. 58, no. 1, pp. 1471-1475: SAGE Publications. - S. D. Sen and J. A. Adams, "Real-Time Optimal Selection of Multirobot Coalition Formation Algorithms Using Conceptual Clustering," in Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence , 2015. - C. Lizza and C. Friedlander, "The pilot's associate: a forum for the integration of knowledge-based systems and avionics," in Aerospace and Electronics Conference, 1988. NAECON 1988., Proceedings of the IEEE 1988 National , 1988, pp. 1252-1258: IEEE. - G. Champigneux, "In flight mission planning in the Copilote Electronique," AGARD, New Advances in Mission Planning and Rehearsal Systems 12 p(SEE N 94-25008 06-66), 1993. -
  7. [24][24] C. A. Miller and M. D. Hannen, "The Rotorcraft Pilot's Associate: design and evaluation of an intelligent user interface for cockpit information management," Knowledge-Based Systems, vol. 12, no. 8, pp. 443-456, 1999. -
  8. [25][25] P. Urlings and L. C. Jain, "Teaming Human and Machine: a Conceptual Framework," in Hybrid Information Systems : Springer, 2002, pp. 711-721. -
  9. [26][26] H. M. Bloom and N. Christopher, "A framework for distributed and virtual discrete part manufacturing," Proceedings of the CALS EXPO, vol. 96, 1996. -
  10. [27][27] L. Vig and J. A. Adams, "A Framework for Multi-Robot Coalition Formation," in Indian International Conference on Artificial Intelligence (IICAI) , 2005, pp. 347-363: Citeseer. -
  11. [28][28] J.-M. Hoc, "From human-machine interaction to humanmachine cooperation," Ergonomics, vol. 43, no. 7, pp. 833-843, 2000. -
  12. [29][29] E. Salas, C. Prince, D. P. Baker, and L. Shrestha, "Situation awareness in team performance: Implications for measurement and training," Human factors, vol. 37, no. 1, pp. 123-136, 1995. -
  13. [30][30] A. K. Das, R. Fierro, V. Kumar, J. P. Ostrowski, J. Spletzer, and C. J. Taylor, "A vision-based formation control framework," IEEE transactions on robotics and automation, vol. 18, no. 5, pp. 813-825, 2002. -
  14. [32][32] M. Johnson, J. M. Bradshaw, P. J. Feltovich, C. Jonker, M. Sierhuis, and B. van Riemsdijk, "Toward coactivity," in Proceedings of the 5th ACM/IEEE international conference on Human-robot interaction , 2010, pp. 101-102: IEEE Press. -
  15. [33][33] M. Johnson et al. , "Beyond cooperative robotics: The central role of interdependence in coactive design," Intelligent Systems, IEEE, vol. 26, no. 3, pp. 81-88, 2011. -
  16. [35][35] J. M. Bradshaw, P. J. Feltovich, H. Jung, S. Kulkarni, W. Taysom, and A. Uszok, "Dimensions of adjustable autonomy and mixed-initiative interaction," in International Workshop on Computational Autonomy , 2003, pp. 17-39: Springer. -
  17. [36][36] T. T. Hewett et al. , ACM SIGCHI curricula for human-computer interaction . ACM, 1992. -
  18. [38][38] B. D. Argall and A. G. Billard, "A survey of tactile human-robot interactions," Robotics and Autonomous Systems, vol. 58, no. 10, pp. 1159-1176, 2010. -
  19. [39][39] J. Bray-Miners, C. Ste-Croix, and A. Morton, "Human-Robot Interaction Literature Review," DTIC Document2012. -
  20. [41][41] J. C. Licklider, "Man-computer symbiosis," Human Factors in Electronics, IRE Transactions on, no. 1, pp. 4-11, 1960. -
  21. [43][43] M. A. Valentine and A. C. Edmondson, "Team scaffolds: How mesolevel structures enable role-based coordination in temporary groups," Organization Science, vol. 26, no. 2, pp. 405-422, 2014. -
  22. [44][44] K. J. Klein, J. C. Ziegert, A. P. Knight, and Y. Xiao, "Dynamic delegation: Shared, hierarchical, and deindividualized leadership in extreme action teams," Administrative Science Quarterly, vol. 51, no. 4, pp. 590-621, 2006. -
  23. [45][45] G. A. Bigley and K. H. Roberts, "The incident command system: High-reliability organizing for complex and volatile task environments," Academy of Management Journal, vol. 44, no. 6, pp. 1281-1299, 2001. -
  24. [46][46] K. Mackey, "Stages of team development," Software, IEEE, vol. 16, no. 4, pp. 90-91, 1999. -
  25. [49][49] <!-- image --> - education," Mind, brain, and education, vol. 1, no. 1, pp. 3-10, 2007. - M. Farrell, M. Schmitt, and G. Heinemann, "Informal roles and the stages of interdisciplinary team development," Journal of interprofessional care, vol. 15, no. 3, pp. 281-295, 2001. - D. L. Miller, "The stages of group development: A retrospective study of dynamic team processes," Canadian Journal of Administrative Sciences/Revue Canadienne des Sciences de l'Administration, vol. 20, no. 2, pp. 121-134, 2003. -
  26. [50][50] B. W. Tuckman and M. A. C. Jensen, "Stages of small-group development Revisited," Group Facilitation, vol. 2, no. 4, pp. 419-427, 1977. -
  27. [51][51] M. A. Marks, J. E. Mathieu, and S. J. Zaccaro, "A temporally based framework and taxonomy of team processes," Academy of management review, vol. 26, no. 3, pp. 356-376, 2001. -
  28. [52][52] N. Mohr and A. Dichter, "Stages of team development," Team & Organization Development, McGraw-Hill, New York, NY, 1996.
  29. [53][53] M. R. Endsley, "Design and evaluation for situation awareness enhancement," in Proceedings of ergonomics society annual meeting 101: SAGE Publications. -
  30. [54][54] N. Callow, M. J. Smith, L. Hardy, C. A. Arthur, and J. Hardy, its Journal 2. "Measurement of transformational leadership and relationship with team cohesion and performance level," of Applied Sport Psychology, vol. 21, no. 4, pp. 395-412, 2009. -
  31. [56][56] J.-P. Heuzé, N. Raimbault, and P. Fontayne, "Relationships between cohesion, collective efficacy and performance in professional basketball teams: An examination of mediating effects," Journal of sports sciences, vol. 24, no. 1, pp. 59-68, 2006.
  32. [57][57] - D. S. Wilson, E. Ostrom, and M. E. Cox, "Generalizing the core design principles for the efficacy of groups," Journal of Economic Behavior & Organization, vol. 90, pp. S21-S32, 2013.
  33. [60][60] - M. Johnson, C. Jonker, B. Van Riemsdijk, P. J. Feltovich, and J. M. Bradshaw, "Joint activity testbed: Blocks world for teams (BW4T)," in International Workshop on Engineering Societies in the Agents World , 2009, pp. 254-256: Springer. - D. Calisi, L. Iocchi, and D. Nardi, "A unified benchmark framework for autonomous Mobile robots and Vehicles Motion Algorithms (MoVeMA benchmarks)," in Workshop on experimental methodology and benchmarking in robotics research (RSS 2008) , 2008. - R. R. Murphy and D. Schreckenghost, "Survey of metrics for human-robot interaction," in 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , 2013, pp. 197198: IEEE. -
  34. [62][62] A. Steinfeld et al. , "Common metrics for human-robot interaction," in Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction , 2006, pp. 33-40: ACM.
  35. [63][63] - M. R. Elara, C. A. A. Calderon, C. Zhou, and W. S. Wijesoma, "False alarm metrics: Evaluating safety in human robot interactions," in 2010 IEEE Conference on Robotics Automation and Mechatronics (RAM) , 2010, pp. 230-236: IEEE.
  36. [64][64] - R. E. Mohan, W. S. Wijesoma, C. A. Acosta Calderon, and C. Zhou, "False alarm metrics for human-robot interactions in service robots," Advanced Robotics, vol. 24, no. 13, pp. 18411859, 2010. -
  37. [65][65] D. F. Glas, T. Kanda, H. Ishiguro, and N. Hagita, "Teleoperation of Multiple Social Robots," IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, vol. 42, no. 3, pp. 530-544, 2012.
  38. [66][66] - T. A. O'Connell and Y.-Y. Choong, "Metrics for measuring human interaction with interactive visualizations for information analysis," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 2008, pp. 1493-1496: ACM. the human factors and , 1988, vol. 32, no. 2, pp. 97- -
  39. [67][67] G. Sammer et al. , "Relationship between regional hemodynamic activity and simultaneously recorded EEG-theta associated with mental arithmetic-induced workload," Human brain mapping, vol. 28, no. 8, pp. 793-803, 2007. -
  40. [68][68] J. W. Crandall and M. L. Cummings, "Developing performance metrics for the supervisory control of multiple robots," in Proceedings of the ACM/IEEE international conference on Human-robot interaction , 2007, pp. 33-40: ACM. -
  41. [69][69] J. Y. Chen and M. J. Barnes, "Human-agent teaming for multirobot control: a review of human factors issues," HumanMachine Systems, IEEE Transactions on, vol. 44, no. 1, pp. 1329, 2014. -
  42. [70][70] L. Rapolienė, A. Razbadauskas, J. Sąlyga, and A. Martinkėnas, "Stress and Fatigue Management Using Balneotherapy in a Short-Time Randomized Controlled Trial," Evidence-Based Complementary and Alternative Medicine, vol. 2016, 2016. -
  43. [71][71] T. Yazawa et al. , "Measurement of Stress Using DFA Heartbeat Analysis," in Proceedings of the World Congress on Engineering and Computer Science , 2013, vol. 2. -
  44. [72][72] X. Corbillon, G. Simon, A. Devlic, and J. Chakareski, "Viewport-adaptive navigable 360-degree video delivery," in IEEE International Conference on Communications (ICC) , 2017, pp. 1-7: IEEE. -
  45. [73][73] N. Forouzandehmehr, S. M. Perlaza, Z. Han, and H. V. Poor, "A satisfaction game for heating, ventilation and air conditioning control of smart buildings," in 2013 IEEE Global Communications Conference (GLOBECOM) , 2013, pp. 31643169: IEEE. -
  46. [74][74] W. Saad, A. L. Glass, N. B. Mandayam, and H. V. Poor, "Toward a consumer-centric grid: A behavioral perspective," Proceedings of the IEEE, vol. 104, no. 4, pp. 865-882, 2016. -
  47. [78][78] G. Matthews, J. S. Warm, T. H. Shaw, and V. S. Finomore, "Predicting battlefield vigilance: a multivariate approach to assessment of attentional resources," Ergonomics, vol. 57, no. 6, pp. 856-875, 2014. -
  48. [79][79] A. Gorbenko, V. Popov, and A. Sheka, "Robot self-awareness: Exploration of internal states," Applied Mathematical Sciences, vol. 6, no. 14, pp. 675-688, 2012. -
  49. [80][80] D. Schreckenghost, T. Milam, and T. Fong, "Measuring performance in real time during remote human-robot operations with adjustable autonomy," IEEE Intelligent Systems, vol. 25, no. 5, pp. 36-45, 2010. -
  50. [81][81] J. A. Saleh and F. Karray, "Towards generalized performance metrics for human-robot interaction," in 2010 International Conference on Autonomous and Intelligent Systems (AIS) , 2010, pp. 1-6: IEEE. -
  51. [82][82] J. W. Crandall, M. A. Goodrich, D. R. Olsen Jr, and C. W. Nielsen, "Validating human-robot interaction schemes in multitasking environments," Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on, vol. 35, no. 4, pp. 438-449, 2005. -
  52. [83][83] M. Chen, M. Mozaffari, W. Saad, C. Yin, M. Debbah, and C. S. Hong, "Caching in the sky: Proactive deployment of cacheenabled unmanned aerial vehicles for optimized quality-ofexperience," IEEE Journal on Selected Areas in Communications, vol. 35, no. 5, pp. 1046-1061, 2017. -
  53. [85][85] <!-- image --> - Performance and tradeoffs," IEEE Transactions on Wireless Communications, vol. 15, no. 6, pp. 3949-3963, 2016. - E. Zeydan et al. , "Big data caching for networking: Moving from cloud to edge," IEEE Communications Magazine, vol. 54, no. 9, pp. 36-42, 2016. -
  54. [86][86] J. Tadrous, A. Eryilmaz, and H. El Gamal, "Proactive resource allocation: Harnessing the diversity and multicast gains," IEEE Transactions on Information Theory, vol. 59, no. 8, pp. 48334854, 2013.
  55. [87][87] - A. Khreishah, J. Chakareski, and A. Gharaibeh, "Joint caching, routing, and channel assignment for collaborative small-cell cellular networks," IEEE Journal on Selected Areas in Communications, vol. 34, no. 8, pp. 2275-2284, 2016.
  56. [88][88] - S. J. Zaccaro, A. L. Rittman, and M. A. Marks, "Team leadership," The leadership quarterly, vol. 12, no. 4, pp. 451483, 2001. -
  57. [89][89] S.-J. Chen and L. Lin, "Modeling team member characteristics for the formation of a multifunctional team in concurrent engineering," IEEE Transactions on Engineering Management, vol. 51, no. 2, pp. 111-124, 2004. -
  58. [90][90] F. L. Greitzer, "Toward the development of cognitive task difficulty metrics to support intelligence analysis research," in Fourth IEEE Conference on Cognitive Informatics, (ICCI). 2005, pp. 315-320: IEEE. -
  59. [91][91] J. Meyer, "Effects of warning validity and proximity on responses to warnings," Human factors, vol. 43, no. 4, pp. 563572, 2001. -
  60. [92][92] G. Doisy, J. Meyer, and Y. Edan, "The Impact of Human-Robot Interface Design on the Use of a Learning Robot System," IEEE Transactions on Human-Machine Systems, vol. 44, no. 6, pp. 788-795, 2014.
  61. [93][93] - J. A. Shah, J. H. Saleh, and J. A. Hoffman, "Review and synthesis of considerations in architecting heterogeneous teams of humans and robots for optimal space exploration," Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, vol. 37, no. 5, pp. 779-793, 2007.
  62. [98][98] - P. Walker, S. Nunnally, M. Lewis, A. Kolling, N. Chakraborty, and K. Sycara, "Neglect benevolence in human control of swarms in the presence of latency," in 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC) , 2012, pp. 3009-3014: IEEE. - C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, "Measurement," in Experimentation in Software Engineering : Springer Science & Business Media, 2012, pp. 3743. - P. Pina, M. Cummings, J. Crandall, and M. Della Penna, "Identifying generalizable metric classes to evaluate humanrobot teams," in Proc. 3rd Ann. Conf. Human-Robot Interaction , 2008, pp. 13-20. - M. A. Goodrich, S. Kerman, and S.-Y. Jun, "On Leadership and Influence in Human-Swarm Interaction," in AAAI Fall Symposium: Human Control of Bioinspired Swarms , 2012. - M. A. Goodrich, B. Pendleton, P. Sujit, J. Pinto, and J. W. Crandall, "Toward human interaction with bio-inspired teams," in The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 3 , 2011, pp. 1265-1266: International Foundation for Autonomous Agents and Multiagent Systems. -
  63. [99][99] J. D. Lee, "Emerging challenges in cognitive ergonomics: Managing swarms of self-organizing agent-based automation," Theoretical Issues in Ergonomics Science, vol. 2, no. 3, pp. 238250, 2001. -
  64. [100][100] E. Ferrante, A. E. Turgut, C. Huepe, A. Stranieri, C. Pinciroli, and M. Dorigo, "Self-organized flocking with a mobile robot swarm: a novel motion control method," Adaptive Behavior, vol. 20, no. 6, pp. 460-477, 2012.
  65. [101][101] - W. M. Spears, D. F. Spears, J. C. Hamann, and R. Heil, "Distributed, physics-based control of swarms of vehicles," Autonomous Robots, vol. 17, no. 2, pp. 137-162, 2004. -
  66. [102][102] M. D. Mumford, W. A. Baughman, K. V. Threlfall, C. E. Uhlman, and D. P. Costanza, "Personality, adaptability, and - performance: Performance on well-defined problem solving tasks," Human Performance, vol. 6, no. 3, pp. 241-285, 1993. -
  67. [103][103] S. A. Rathus, "A 30-item schedule for assessing assertive behavior," Behavior therapy, vol. 4, no. 3, pp. 398-406, 1973. -
  68. [104][104] M. J. Pearsall and A. P. J. Ellis, "The Effects of Critical Team Member Assertiveness on Team Performance and Satisfaction," Journal of Management, vol. 32, no. 4, pp. 575-594, 2006/08/01 2006. -
  69. [106][106] G. Windle, K. M. Bennett, and J. Noyes, "A methodological review of resilience measurement scales," Health and Quality of Life Outcomes, journal article vol. 9, no. 1, p. 8, 2011. -
  70. [107][107] J. A. Green, D. B. O'Connor, N. Gartland, and B. W. Roberts, "The Chernyshenko Conscientiousness Scales: A new facet measure of conscientiousness," Assessment, vol. 23, no. 3, pp. 374-385, 2016. -
  71. [108][108] V. Dulewicz and M. Higgs, "Can emotional intelligence be measured and developed?," Leadership & Organization Development Journal, vol. 20, no. 5, pp. 242-253, 1999. -
  72. [110][110] T. Heffernan and J. Ling, "The impact of Eysenck's extraversionintroversion personality dimension on prospective memory," Scandinavian Journal of Psychology, vol. 42, no. 4, pp. 321-325, 2001. -
  73. [111][111] I. B. Mauss and M. D. Robinson, "Measures of emotion: A review," Cognition and emotion, vol. 23, no. 2, pp. 209-237, 2009. -
  74. [112][112] R. L. Tomko et al. , "Measuring impulsivity in daily life: the momentary impulsivity scale," Psychological assessment, vol. 26, no. 2, p. 339, 2014. -
  75. [113][113] M. R. Endsley, "Measurement of situation awareness in dynamic systems," Human factors, vol. 37, no. 1, pp. 65-84, 1995. -
  76. [114][114] R. Christensen and G. Knezek, "Comparative measures of grit, tenacity and perseverance," International Journal of Learning, Teaching and Educational Research, vol. 8, no. 1, 2014. -
  77. [115][115] K. Parsons, A. McCormac, M. Butavicius, M. Pattinson, and C. Jerram, "Determining employee awareness using the human aspects of information security questionnaire (HAIS-Q)," Computers & security, vol. 42, pp. 165-176, 2014. -
  78. [116][116] S. G. Hart, "NASA-task load index (NASA-TLX); 20 years later," in Proceedings of the human factors and ergonomics society annual meeting , 2006, vol. 50, no. 9, pp. 904-908: Sage Publications Sage CA: Los Angeles, CA. -
  79. [117][117] P. T. Costa and R. R. McCrae, "Neuroticism, somatic complaints, and disease: is the bark worse than the bite?," Journal of personality, vol. 55, no. 2, pp. 299-316, 1987. -
  80. [118][118] X. Fan et al. , "An Exploratory Study about Inaccuracy and Invalidity in Adolescent Self-Report Surveys," Field Methods, vol. 18, no. 3, pp. 223-244, 2006. -
  81. [119][119] R. R. Wilcox, Introduction to robust estimation and hypothesis testing . Academic press, 2011. -
  82. [120][120] K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack, "Study of subjective and objective quality assessment of video," IEEE transactions on Image Processing, vol. 19, no. 6, pp. 1427-1441, 2010. -
  83. [121][121] J. E. Ware Jr, "Scales for measuring general health perceptions," Health services research, vol. 11, no. 4, p. 396, 1976. -
  84. [122][122] E. Åhsberg, "Dimensions of fatigue in different working populations," Scandinavian journal of psychology, vol. 41, no. 3, pp. 231-241, 2000. -
  85. [124][124] M. J. Barnes and M. F. Matz, "Crew Simulations for Unmanned Aerial Vehicle (UAV) Applications: Sustained Effects, Shift Factors, Interface Issues, and Crew Size," Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 42, no. 1, pp. 143-147, 1998/10/01 1998. -
  86. [125][125] S. Folkard and P. Tucker, "Shift work, safety and productivity," Occupational medicine, vol. 53, no. 2, pp. 95-101, 2003. -
  87. [127][127] <!-- image --> - Proceedings of the Human Factors and Ergonomics Society Annual Meeting , 1999, vol. 43, no. 3, pp. 251-255: SAGE Publications Sage CA: Los Angeles, CA. E. W. Russell, "A multiple scoring method for the assessment of complex memory functions," Psychology, Journal of Consulting and Clinical vol. 43, no. 6, p. 800, 1975. -
  88. [128][128] S. E. Devena and M. W. Watkins, "Diagnostic utility of WISCIV general abilities index and cognitive proficiency index difference scores among children with ADHD," Journal of Applied School Psychology, vol. 28, no. 2, pp. 133-154, 2012. -
  89. [129][129] N. R. Council, Complex operational decision making in networked systems of humans and machines: a multidisciplinary approach . National Academies Press, 2014. -
  90. [130][130] K. G. Shin and P. Ramanathan, "Real-time computing: A new discipline of computer science and engineering," Proceedings of the IEEE, vol. 82, no. 1, pp. 6-24, 1994. -
  91. [131][131] V. Groom and C. Nass, "Can robots be teammates?: Benchmarks in human-robot teams," Interaction Studies, vol. 8, no. 3, pp. 483-500, 2007. -
  92. [132][132] D. R. Olsen and M. A. Goodrich, "Metrics for evaluating humanrobot interactions," in Proceedings of PERMIS , 2003, vol. 2003, p. 4. -
  93. [133][133] M. Smaili, J. Breeman, T. Lombaerts, and O. Stroosma, "A simulation benchmark for aircraft survivability assessment," in Proceedings of 26th International Congress of Aeronautical Sciences , Anchorage, AK, 2008, vol. 9, no. 2, pp. 1-12. -
  94. [134][134] art?," - J. Annett, "Subjective rating scales: science or Ergonomics, vol. 45, no. 14, pp. 966-987, 2002.
  95. [135][135] - L. M. Naismith, J. J. Cheung, C. Ringsted, and R. B. Cavalcanti, "Limitations of subjective cognitive load measures in simulation-based procedural training," Medical education, vol. 49, no. 8, pp. 805-814, 2015.
  96. [136][136] - J. C. Bol and S. D. Smith, "Spillover effects in subjective performance evaluation: Bias and the asymmetric influence of controllability," The Accounting Review, vol. 86, no. 4, pp. 12131230, 2011. -
  97. [137][137] J. W. Crandall and M. L. Cummings, "Identifying predictive metrics for supervisory control of multiple robots," IEEE Transactions on Robotics, vol. 23, no. 5, pp. 942-951, 2007. -
  98. [138][138] J. A. Weekley and R. E. Ployhart, Situational judgment tests: Theory, measurement, and application . Psychology Press, 2013. -
  99. [139][139] D. Dowding and C. Thompson, "Measuring the quality of judgement and decision-making in nursing," Journal of advanced nursing, vol. 44, no. 1, pp. 49-57, 2003.
  100. [141][141] - F. Patterson, V. Ashworth, L. Zibarras, P. Coan, M. Kerrin, and P. O'Neill, "Evaluations of situational judgement tests to assess non-academic attributes in selection," Medical education, vol. 46, no. 9, pp. 850-868, 2012. -
  101. [142][142] J. Clevenger, G. M. Pereira, D. Wiechmann, N. Schmitt, and V. S. Harvey, "Incremental validity of situational judgment tests," Journal of Applied Psychology, vol. 86, no. 3, p. 410, 2001. -
  102. [143][143] K. M. Kowalski-Trakofler, C. Vaught, and T. Scharf, "Judgment and decision making under stress: an overview for emergency managers," International Journal of Emergency Management, vol. 1, no. 3, pp. 278-289, 2003. -
  103. [144][144] H. Chau, C. Wong, F. Chow, and C.-H. F. Fung, "Social judgment theory based model on opinion formation, polarization and evolution," Physica A: Statistical Mechanics and its Applications, vol. 415, pp. 133-140, 2014.
  104. [145][145] - L. Rabin et al. , "Judgment in older adults: Development and psychometric evaluation of the Test of Practical Judgment (TOPJ)," Journal of clinical and experimental neuropsychology, vol. 29, no. 7, pp. 752-767, 2007. 2.
  105. [146][146] - A. S. Clare, M. L. Cummings, and N. P. Repenning, "Influencing Trust for Human-Automation Collaborative Scheduling of Multiple Unmanned Vehicles," Human factors, vol. 57, no. 7, pp. 1208-1218, 2015. -
  106. [147][147] S. Bruni, J. J. Marquez, A. Brzezinski, and M. L. Cummings, "Visualizing operators' cognitive strategies in multivariate optimization," in Proceedings of the Human Factors and Ergonomics Society Annual Meeting , 2006, vol. 50, no. 11, pp. 1180-1184: Sage Publications Sage CA: Los Angeles, CA. -
  107. [148][148] S. Heuer and B. Hallowell, "A novel eye-tracking method to assess attention allocation in individuals with and without aphasia using a dual-task paradigm," Journal of communication disorders, vol. 55, pp. 15-30, 2015. -
  108. [149][149] T. Roderer and C. M. Roebers, "Can you see me thinking (about my answers)? Using eye-tracking to illuminate developmental differences in monitoring and control skills and their relation to performance," Metacognition and learning, vol. 9, no. 1, pp. 123, 2014. -
  109. [150][150] N. Moacdieh and N. Sarter, "Clutter in electronic medical records: examining its performance and attentional costs using eye tracking," Human factors, vol. 57, no. 4, pp. 591-606, 2015. -
  110. [151][151] P. Damacharla, A. Javaid, and V. Devabhaktuni, "Human error prediction using eye tracking to improvise team cohesion in human-machine teams," in 9th International Conference on Applied Human Factors and Ergonomics (AHFE 2018) , Orlando, Florida, USA, 2018: Springer International Publishing. -
  111. [153][153] F. Paas, J. E. Tuovinen, H. Tabbers, and P. W. Van Gerven, "Cognitive load measurement as a means to advance cognitive load theory," Educational psychologist, vol. 38, no. 1, pp. 63-71, 2003. -
  112. [154][154] F. A. Haji, D. Rojas, R. Childs, S. Ribaupierre, and A. Dubrowski, "Measuring cognitive load: Performance, mental effort and simulation task complexity," Medical education, vol. 49, no. 8, pp. 815-827, 2015. -
  113. [155][155] T. A. Nguyen and Y. Zeng, "A physiological study of relationship between designer's mental effort and mental stress during conceptual design," Computer-Aided Design, vol. 54, pp. 3-18, 2014. -
  114. [156][156] G. Matthews, L. E. Reinerman-Jones, D. J. Barber, and J. Abich IV, "The psychometrics of mental workload: Multiple measures are sensitive but divergent," Human factors, vol. 57, no. 1, pp. 125-143, 2015. -
  115. [157][157] T. Fritz, A. Begel, S. C. Müller, S. Yigit-Elliott, and M. Züger, "Using psycho-physiological measures to assess task difficulty in software development," in Proceedings of the 36th International Conference on Software Engineering , 2014, pp. 402-413: ACM. -
  116. [158][158] T. Glenn and S. Monteith, "New measures of mental state and behavior based on data collected from sensors, smartphones, and the Internet," Current psychiatry reports, vol. 16, no. 12, p. 523, 2014. -
  117. [159][159] M. S. Young, K. A. Brookhuis, C. D. Wickens, and P. A. Hancock, "State of science: mental workload in ergonomics," Ergonomics, vol. 58, no. 1, pp. 1-17, 2015. -
  118. [160][160] A. Andre and A. Degani, "Do you know what mode you're in? An analysis of mode error in everyday things," Humanautomation interaction: Research & Practice, Mahwah, NJ: Lawrence Erlbaum, pp. 19-28, 1997. -
  119. [161][161] N. B. Sarter and D. D. Woods, "Mode error in supervisory control of automated systems," in Proceedings of the Human Factors and Ergonomics Society Annual Meeting , 1992, vol. 36, no. 1, pp. 26-29: SAGE Publications Sage CA: Los Angeles, CA. -
  120. [162][162] M. R. Elara, C. A. A. Calderón, C. Zhou, and W. S. Wijesoma, "Experimenting extended neglect tolerance model for human robot interactions in service missions," in 2010 11th International Conference on Control Automation Robotics & Vision (ICARCV) , 2010, pp. 2024-2029: IEEE. -
  121. [163][163] J. A. Saleh, F. Karray, and M. Morckos, "Modelling of robot attention demand in human-robot interaction using finite fuzzy state automata," in Fuzzy Systems (FUZZ-IEEE), 2012 IEEE International Conference on , 2012, pp. 1-8: IEEE. -
  122. [164][164] J. W. Crandall, C. W. Nielsen, and M. A. Goodrich, "Towards predicting robot team performance," in IEEE International Conference on Systems, Man and Cybernetics , 2003, vol. 1, pp. 906-911: IEEE. -
  123. [165][165] T. W. McLain, P. R. Chandler, S. Rasmussen, and M. Pachter, "Cooperative control of UAV rendezvous," in Proceedings of the 2001 American Control Conference,. 2001, vol. 3, pp. 23092314: IEEE. -
  124. [166][166] L. Berardinelli, V. Cortellessa, and A. Di Marco, "Performance modeling and analysis of context-aware mobile software systems," Fundamental Approaches to Software Engineering, pp. 353-367, 2010. -
  125. [167][167] D. Harel, "Statecharts: A visual formalism for complex systems," Science of computer programming, vol. 8, no. 3, pp. 231-274, 1987. <!-- image -->
  126. [168][168] - P. Damacharla, R. R. Junuthula, A. Javaid, and V. Devabhaktuni, "Autonomous ground vehicle error prediction modeling to facilitate human-machine cooperation," in 9th International Conference on Applied Human Factors and Ergonomics (AHFE 2018) Orlando, Florida, USA, 2018: Springer International Publishing. -
  127. [169][169] T. Schlogl, C. Beleznai, M. Winter, and H. Bischof, "Performance evaluation metrics for motion detection and tracking," in Proceedings of the 17th International Conference on Pattern Recognition (ICPR 2004),. 2004, vol. 4, pp. 519-522: IEEE. -
  128. [170][170] F. Serban, J. Vanschoren, J.-U. Kietz, and A. Bernstein, "A ACM 4. survey of intelligent assistants for data analysis," Computing Surveys (CSUR), vol. 45, no. 3, p. 31, 2013.
  129. [171][171] - J. S. Ash, M. Berg, and E. Coiera, "Some unintended consequences of information technology in health care: the nature of patient care information system-related errors," Journal of the American Medical Informatics Association, vol. 11, no. 2, pp. 104-112, 2004. -
  130. [172][172] T. Peynot, J. Underwood, and S. Scheding, "Towards reliable perception for unmanned ground vehicles in challenging conditions," in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , St. Louis, MO, USA, 2009, pp. 1170-1176: IEEE. -
  131. [173][173] S. C. Peters and K. Iagnemma, "An analysis of rollover stability measurement for high-speed mobile robots," in Proceedings 2006 IEEE International Conference on Robotics and Automation (ICRA) , Orlando, FL, USA, 2006, pp. 3711-3716: IEEE. -
  132. [174][174] W. Burgard et al. , "Experiences with an interactive museum tour-guide robot," Artificial intelligence, vol. 114, no. 1-2, pp. 355, 1999.
  133. [175][175] - D. Schreckenghost, T. Fong, H. Utz, and T. Milam, "Measuring robot performance in real-time for nasa robotic reconnaissance operations," in Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems , 2009, pp. 194-202: ACM. -
  134. [176][176] A. V. Carron and L. R. Brawley, "Cohesion: Conceptual and measurement issues," Small Group Research, vol. 31, no. 1, pp. 89-106, 2000. -
  135. [177][177] A. V. Carron and J. R. Ball, "An analysis of the cause-effect characteristics of cohesiveness and participation motivation in intercollegiate hockey," International Review of Sport Sociology, vol. 12, no. 2, pp. 49-60, 1977. -
  136. [178][178] E. Salas, R. Grossman, A. M. Hughes, and C. W. Coultas, "Measuring Team Cohesion," Human Factors, vol. 57, no. 3, pp. 365-374, 2015. -
  137. [179][179] J. Burke and R. Murphy, "RSVP: An investigation of remote shared visual presence as common ground for human-robot teams," in 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI) , 2007, pp. 161-168. -
  138. [180][180] S. J. Yi et al. , "Team thor's entry in the darpa robotics challenge trials 2013," Journal of Field Robotics, vol. 32, no. 3, pp. 315335, 2015.
  139. [182][182] Y. Nakamura, H. Hanafusa, and T. Yoshikawa, "Task-priority based redundancy control of robot manipulators," The International Journal of Robotics Research, vol. 6, no. 2, pp. 315, 1987. -
  140. [183][183] V. Klamroth-Marganska et al. , "Three-dimensional, taskspecific robot therapy of the arm after stroke: a multicentre, parallel-group randomised trial," The Lancet Neurology, vol. 13, no. 2, pp. 159-166, 2014. -
  141. [184][184] P. Damacharla et al. , "Effects of Voice-Based Synthetic Assistant on Performance of Emergency Care Provider in Training," International Journal of Artificial Intelligence in Education, journal article March 19 2018. -
  142. [185][185] S. Prabhala, J. J. Gallimore, and J. R. Lucas, "Evaluating Human Interaction with Automation in a Complex UCAV Control Station Simulation Using Multiple Performance Metrics," in Human-in-the-Loop Simulations : Springer, 2011, pp. 239-258. -
  143. [186][186] J. Gerhardt-Powals, "Cognitive engineering principles for enhancing human-computer performance," International Journal of Human-Computer Interaction, vol. 8, no. 2, pp. 189211, 1996. -
  144. [187][187] H. Cai and Y. Lin, "Modeling of operators' emotion and task performance in a virtual driving environment," International Journal of Human-Computer Studies, vol. 69, no. 9, pp. 571-586, 2011. -

Cited by

Sorted by citations

See all 97

Metadata

Authors
Damacharla, P., Javaid, A. Y., Gallimore, J. J., & Devabhaktuni, V. K.
Affiliations
  • University of Toledo (US)
  • Bowling Green State University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
6
Pages
38637-38655
ISSN
2169-3536 (ISSN-L)
DOI
10.48550/arxiv.2008.04855
arXiv
2008.04855
OpenAlex
W2856991417
Cited by
97
References
0
Citations by year
2026: 3 · 2025: 14 · 2024: 21 · 2023: 24 · 2022: 15 · 2021: 6 · 2020: 10
FWCI
6.77
Type
journal article
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • IEEE Access (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
University of Toledo
Topics
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Social Sciences › Psychology › Social Psychology › Team Dynamics and Performance
  • Social Sciences › Human Factors and Ergonomics › Ergonomics and Human Factors
Keywords
human-machine teams, benchmarking metrics, performance evaluation, human-machine teaming, metric classification, real-time metrics, verification and validation, human-automation interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Common Metrics to Benchmark Human-Machine Teams (HMT): A Review. https://app.cochat.ai/papers/10.48550/arxiv.2008.04855
The paper
Damacharla, P., Javaid, A. Y., Gallimore, J. J., & Devabhaktuni, V. K. (2020). Common Metrics to Benchmark Human-Machine Teams (HMT): A Review. arXiv. https://doi.org/10.48550/arxiv.2008.04855