CoChat

Common Metrics to Benchmark Human-Machine Teams (HMT): A Review

Open access
Damacharla, Praveen, Javaid, Ahmad Y., Gallimore, Jennie J., Devabhaktuni, Vijay K. · 2018 · IEEE Access
Cited by 97

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

Abstract

A significant amount of work is invested in human-machine teaming (HMT) across multiple fields. Accurately and effectively measuring system performance of an HMT is crucial for moving the design of these systems forward. Metrics are the enabling tools to devise a benchmark in any system and serve as an evaluation platform for assessing the performance, along with the verification and validation, of a system. Currently, there is no agreed-upon set of benchmark metrics for developing HMT systems. Therefore, identification and classification of common metrics are imperative to create a benchmark in the HMT field. The key focus of this review is to conduct a detailed survey aimed at identification of metrics employed in different segments of HMT and to determine the common metrics that can be used in the future to benchmark HMTs. We have organized this review as follows: identification of metrics used in HMTs until now, and classification based on functionality and measuring techniques. Additionally, we have also attempted to analyze all the identified metrics in detail while classifying them as theoretical, applied, real-time, non-real-time, measurable, and observable metrics. We conclude this review with a detailed analysis of the identified common metrics along with their usage to benchmark HMTs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/access.2018.2853560

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 18 references

Research works printed but not linked: 167

  1. 10.1145/2480741.2480748 · DOI
  2. 10.1109/IROS.2009.5354484 · DOI
  3. 10.1016/S0004-3702(99)00070-3 · DOI · Paper page
  4. 10.1109/ROBOT.2006.1642269 · DOI
  5. 10.1177/104649640003100105 · DOI · Paper page
  6. 10.1145/1865909.1865950 · DOI
  7. 10.1177/101269027701200203 · DOI · Paper page
  8. damacharla, 2018, Autonomous ground vehicle error prediction modeling to facilitate human-machine cooperation, Proc of the 5th Int Conf on Applied Human Factors and Ergonomics (AHFE), 36
  9. 10.1109/ICPR.2004.1333825 · DOI
  10. bray-miners, 2012, Human-Robot Interaction Literature Review
  11. 10.1016/j.robot.2010.07.002 · DOI · Paper page
  12. 10.1109/MIS.2011.47 · DOI
  13. 10.1109/HRI.2010.5453244 · DOI
  14. 10.1109/TRA.2002.803463 · DOI · Paper page
  15. bainbridge, 2004, Berkshire Encyclopedia of Human-Computer Interaction
  16. 10.1145/2594128 · DOI · Paper page
  17. bradshaw, 2003, Dimensions of adjustable autonomy and mixed-initiative interaction, Proc Int Workshop Comput Auto, 17
  18. dekoven, 2006, A Framework for Supporting Team-work Between Humans and Autonomous Systems
  19. 10.1002/rob.21555 · DOI
  20. 10.1007/978-0-85729-883-6_12 · DOI
  21. 10.1007/s40593-018-0166-3 · DOI
  22. 10.1016/S1474-4422(13)70305-3 · DOI · Paper page
  23. 10.1177/027836498700600201 · DOI · Paper page
  24. 10.1109/ROBOT.2004.1307477 · DOI
  25. 10.1016/j.ijhcs.2011.05.003 · DOI
  26. 10.1080/10447319609526147 · DOI · Paper page
  27. 10.1080/001401300409044 · DOI · Paper page
  28. vig, 2005, A framework for multi-robot coalition formation, Proc Indian Int Conf Artif Intell (IICAI), 347
  29. 10.1145/1228716.1228738 · DOI
  30. 10.1518/001872095779049525 · DOI · Paper page
  31. 10.1177/1541931214581307 · DOI
  32. lizza, 1988, The Pilot’s Associate: A forum for the integration of knowledge-based systems and avionics, Proc IEEE Aerospace Electronics Conf (NAECON), 1252
  33. sen, 2015, Real-time optimal selection of multirobot coalition formation algorithms using conceptual clustering, Proc 29th AAAI Conf Artif Intell, 1
  34. 10.1016/S0950-7051(99)00035-0 · DOI
  35. champigneux, 1993, In flight mission planning in the ‘copilote electronique, New advances in mission planning and rehearsal systems
  36. bloom, 1996, A framework for distributed and virtual discrete part manufacturing, Proc CALS Expo, 1
  37. 10.1023/B:AURO.0000033970.96785.f2 · DOI · Paper page
  38. 10.1007/978-3-7908-1782-9_51 · DOI
  39. 10.1177/1059712312462248 · DOI · Paper page
  40. 10.1111/medu.12773 · DOI · Paper page
  41. 10.1177/0018720814539505 · DOI · Paper page
  42. 10.1016/j.cad.2013.10.002 · DOI
  43. 10.1177/0018720814564594 · DOI
  44. 10.2172/1261061 · DOI
  45. damacharla, 2018, Human error prediction using eye tracking to improvise team cohesion in human-machine teams, Proc of the 5th Int Conf on Applied Human Factors and Ergonomics (AHFE), 47
  46. 10.1177/0018720815587803 · DOI
  47. 10.1177/154193120605001114 · DOI
  48. 10.1016/j.jcomdis.2015.01.005 · DOI
  49. 10.1007/s11409-013-9109-4 · DOI
  50. calisi, 2008, A unified benchmark framework for autonomous Mobile robots and Vehicles Motion Algorithms (MoVeMA benchmarks), Proc Workshop Exp Methodol Benchmarking Robot Res (RSS), 1
  51. 10.1007/978-3-642-10203-5_26 · DOI
  52. 10.1016/j.jebo.2012.12.010 · DOI · Paper page
  53. 10.1080/02640410500127736 · DOI · Paper page
  54. 10.2514/6.2005-6912 · DOI
  55. 10.1080/10413200903204754 · DOI · Paper page
  56. 10.1177/154193128803200221 · DOI · Paper page
  57. mohr, 1996, Stages of Team Development
  58. cannan, 2011, Human-Machine Interaction (HMI) A Survey
  59. 10.1016/0167-6423(87)90035-9 · DOI · Paper page
  60. 10.1007/978-3-642-12029-9_25 · DOI
  61. 10.1109/ACC.2001.946096 · DOI
  62. 10.1109/ICSMC.2003.1243930 · DOI
  63. 10.1109/FUZZ-IEEE.2012.6250792 · DOI
  64. 10.1109/ICARCV.2010.5707803 · DOI
  65. 10.1177/154193129203600108 · DOI
  66. andre, 1997, Do you know what mode you’re in? An analysis of mode error in everyday things, Human-Automation Interaction Research and Practice, 19
  67. 10.5898/JHRI.3.1.Johnson · DOI · Paper page
  68. 10.1109/HAPTIC.2004.1287222 · DOI
  69. sullivan, 2015, F-35 Joint Strike Fighter Assessment Needed to Address Affordability Challenges
  70. 10.1109/MIS.2014.25 · DOI
  71. 10.1007/s10588-010-9065-3 · DOI
  72. 10.1080/00140139.2014.956151 · DOI · Paper page
  73. hicks, 2014, An Investigation of Multiple Unmanned Aircraft Systems Control From the Cockpit of an AH-64 Apache Heli-copter
  74. 10.1111/j.1936-4490.2003.tb00698.x · DOI
  75. 10.1007/978-1-4471-3812-9_2 · DOI
  76. 10.1007/s11920-014-0523-3 · DOI
  77. 10.1109/52.776955 · DOI
  78. 10.2307/3069401 · DOI · Paper page
  79. 10.1080/13561820120068980 · DOI · Paper page
  80. packer, 2002, Multimedia From Wagner to Virtual Reality
  81. 10.2189/asqu.51.4.590 · DOI · Paper page
  82. 10.1287/orsc.2014.0947 · DOI · Paper page
  83. 10.1037/0022-006X.43.6.800 · doi:10.1037/0022-006x.43.6.800 · Paper page
  84. meyer, 1999, Scheduling of actions and reliance onwarnings in a simulated control task, Proc Human Factors Ergonom Soc Annu Meeting, 43, 251, 10.1177/154193129904300326 · DOI
  85. 10.1093/occmed/kqg047 · DOI · Paper page
  86. 10.1177/154193129804200132 · DOI
  87. 10.1109/GLOCOM.2013.6831558 · DOI
  88. 2014, Complex Operational Decision Making in Networked Systems of Humans and Machines A Multidisciplinary Approach
  89. yazawa, 2013, Measurement of stress using DFA heartbeat analysis, Proc World Congr Eng Comput Sci, 2, 596
  90. 10.1155/2016/9631684 · DOI
  91. 10.21236/ADA545552 · DOI
  92. 10.1109/5.259423 · DOI · Paper page
  93. 10.21236/ADA509824 · DOI
  94. 10.1109/JPROC.2016.2520760 · DOI
  95. 10.21236/ADA525910 · DOI
  96. smaili, 2008, A simulation benchmark for aircraft survivability assessment, 26th Int Congr Aeronaut Sci, 9, 1
  97. 10.1080/00140130210166951 · DOI · Paper page
  98. 10.1075/is.8.3.10gro · DOI · Paper page
  99. 10.1080/00140139.2014.899630 · DOI
  100. olsen, 2003, Metrics for evaluating human-robot interactions, Proc PERMIS, 4
  101. gorbenko, 2012, Robot self-awareness: Exploration of internal states, Appl Math Sci, 6, 675
  102. 10.2308/accr-10038 · DOI · Paper page
  103. 10.1111/medu.12732 · DOI · Paper page
  104. weekley, 2013, Situational judgment tests Theory measurement and application, 10.4324/9780203774878 · DOI
  105. 10.1109/TRO.2007.907480 · DOI
  106. 10.1109/HRI.2013.6483569 · DOI
  107. 10.1046/j.1365-2648.2003.02770.x · DOI
  108. 10.1145/1121241.1121249 · DOI · Paper page
  109. elara, 2010, False alarm metrics: Evaluating safety in human robot interactions, Proc IEEE Conf Robot Autom Mechatron (RAM), 230
  110. 10.1163/016918610X527185 · DOI
  111. mcdaniel, 2001, Use of Situational Judgment Tests to Predict Job Performance A Clarification of the Literature
  112. 10.1109/TSMCA.2011.2164243 · DOI
  113. 10.1111/j.1365-2923.2012.04336.x · DOI · Paper page
  114. 10.1145/1357054.1357287 · DOI
  115. 10.1037/0021-9010.86.3.410 · doi:10.1037/0021-9010.86.3.410 · Paper page
  116. 10.1002/hbm.20309 · DOI · Paper page
  117. 10.1504/IJEM.2003.003297 · DOI · Paper page
  118. 10.1145/1228716.1228722 · DOI
  119. 10.1016/j.physa.2014.07.082 · DOI
  120. 10.1109/THMS.2013.2293535 · DOI · Paper page
  121. schechter, 2014, AUVSI Reporter AFRL Targets Seamless Human-Machine Interaction
  122. 10.1080/13825580601025908 · DOI
  123. 10.1007/978-3-642-29044-2_3 · DOI
  124. wilt, 2009, Handbook of Individual Differences in Social Behavior, 27
  125. 10.1109/ICSMC.2012.6378253 · DOI
  126. 10.1109/TSMCC.2007.900644 · DOI
  127. 10.1177/1073191115580639 · DOI
  128. 10.1109/THMS.2014.2331618 · DOI
  129. 10.1518/001872001775870395 · DOI · Paper page
  130. martin, 2003, Academic resilience and the four Cs: Confidence, control, composure, and commitment, Proc NZARE Austral Assoc Res Educ (AARE), 12
  131. 10.1109/COGINF.2005.1532647 · DOI
  132. 10.1177/0149206306289099 · DOI · Paper page
  133. 10.1016/S0005-7894(73)80120-0 · DOI · Paper page
  134. 10.1207/s15327043hup0603_4 · DOI · Paper page
  135. 10.1080/02699930802204677 · DOI · Paper page
  136. 10.1037/a0035083 · DOI · Paper page
  137. 10.1111/1467-9450.00243 · DOI
  138. 10.1109/ICSMC.2011.6084115 · DOI
  139. 10.1080/14639220110104925 · DOI
  140. pina, 2008, Identifying generalizable metric classes to evaluate human-robot teams, Proc 3rd Ann Conf Human-Robot Interaction, 13
  141. goodrich, 2012, On leadership and influence in human-swarm interaction, AAAI Fall Symposium Human Control of Bioinspired Swarms, 1
  142. 10.1109/TSMCC.2004.826287 · DOI
  143. 10.1109/THMS.2014.2343996 · DOI
  144. 10.1145/2696454.2696455 · DOI · Paper page
  145. 10.1007/s10514-015-9457-9 · DOI · Paper page
  146. 10.1109/TRO.2018.2795034 · DOI
  147. 10.1109/THMS.2016.2547186 · DOI
  148. 10.1177/0278364915609673 · DOI
  149. 10.1109/TSMCA.2005.850587 · DOI · Paper page
  150. 10.1177/152822X06289161 · DOI · Paper page
  151. bruemmer, 2003, Collaborative Tools for Mixed Teams of Humans and Robots
  152. 10.1109/AIS.2010.5547049 · DOI
  153. 10.1177/1557234X13501471 · DOI
  154. 10.1109/TWC.2016.2531652 · DOI · Paper page
  155. 10.1145/2701973.2702046 · DOI
  156. wilcox, 2011, Introduction to Robust Estimation and Hypothesis Testing
  157. christensen, 2014, Comparative measures of grit, tenacity and perseverance, International Journal of Learning Teaching and Educational Research, 8, 16
  158. 10.1109/MIS.2010.126 · DOI
  159. 10.1016/j.cose.2013.12.003 · DOI · Paper page
  160. 10.1109/TEM.2004.826011 · DOI
  161. ware, 1976, Scales for measuring general health perceptions, Health Services Res, 11, 396
  162. 10.1111/1467-9450.00192 · DOI · Paper page
  163. lupien, 2013, How to measure stress in humans, Centre for Studies in Human Stress
  164. 10.1109/MCOM.2016.7565185 · DOI · Paper page
  165. 10.1109/TIT.2013.2257911 · DOI
  166. 10.1109/JSAC.2016.2577199 · DOI
  167. 10.1016/S1048-9843(01)00093-5 · DOI · Paper page

Cited by

Metadata

Authors
Damacharla, P., Javaid, A. Y., Gallimore, J. J., & Devabhaktuni, V. K.
Venue
IEEE Access
Published
2018
Volume
6
Pages
38637-38655
ISSN
2169-3536 (ISSN-L)
OpenAlex
W2856991417
Cited by
97
References
18
FWCI
6.44
Type
journal article
Language
en
Open access
gold
Funders
  • University of Toledo
  • Round 1 Award—Improving Healthcare Training and Decision Making Through LVC from the Ohio Federal Research Jobs Commission through Ohio Federal Research Network

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.