CoChat

Common Metrics to Benchmark Human-Machine Teams (HMT): A Review

Open access
Damacharla, Praveen, Javaid, Ahmad Y., Gallimore, Jennie J., Devabhaktuni, Vijay K. · 2018 · IEEE Access
Cited by 97

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Abstract

A significant amount of work is invested in human-machine teaming (HMT) across multiple fields. Accurately and effectively measuring system performance of an HMT is crucial for moving the design of these systems forward. Metrics are the enabling tools to devise a benchmark in any system and serve as an evaluation platform for assessing the performance, along with the verification and validation, of a system. Currently, there is no agreed-upon set of benchmark metrics for developing HMT systems. Therefore, identification and classification of common metrics are imperative to create a benchmark in the HMT field. The key focus of this review is to conduct a detailed survey aimed at identification of metrics employed in different segments of HMT and to determine the common metrics that can be used in the future to benchmark HMTs. We have organized this review as follows: identification of metrics used in HMTs until now, and classification based on functionality and measuring techniques. Additionally, we have also attempted to analyze all the identified metrics in detail while classifying them as theoretical, applied, real-time, non-real-time, measurable, and observable metrics. We conclude this review with a detailed analysis of the identified common metrics along with their usage to benchmark HMTs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/access.2018.2853560

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 17 references

Research works printed but not linked: 168

  1. 10.1145/2480741.2480748 · DOI
  2. 10.1109/IROS.2009.5354484 · DOI
  3. 10.1016/S0004-3702(99)00070-3 · DOI · Paper page
  4. 10.1109/ROBOT.2006.1642269 · DOI
  5. 10.1177/104649640003100105 · DOI · Paper page
  6. 10.1145/1865909.1865950 · DOI
  7. 10.1177/101269027701200203 · DOI · Paper page
  8. damacharla, 2018, Autonomous ground vehicle error prediction modeling to facilitate human-machine cooperation, Proc of the 5th Int Conf on Applied Human Factors and Ergonomics (AHFE), 36
  9. 10.1109/ICPR.2004.1333825 · DOI
  10. bray-miners, 2012, Human-Robot Interaction Literature Review
  11. 10.1016/j.robot.2010.07.002 · DOI · Paper page
  12. 10.1109/MIS.2011.47 · DOI
  13. 10.1109/HRI.2010.5453244 · DOI
  14. 10.1109/TRA.2002.803463 · DOI · Paper page
  15. bainbridge, 2004, Berkshire Encyclopedia of Human-Computer Interaction
  16. 10.1145/2594128 · DOI · Paper page
  17. bradshaw, 2003, Dimensions of adjustable autonomy and mixed-initiative interaction, Proc Int Workshop Comput Auto, 17
  18. dekoven, 2006, A Framework for Supporting Team-work Between Humans and Autonomous Systems
  19. 10.1002/rob.21555 · DOI
  20. 10.1007/978-0-85729-883-6_12 · DOI
  21. 10.1007/s40593-018-0166-3 · DOI
  22. 10.1016/S1474-4422(13)70305-3 · DOI · Paper page
  23. 10.1177/027836498700600201 · DOI · Paper page
  24. 10.1109/ROBOT.2004.1307477 · DOI
  25. 10.1016/j.ijhcs.2011.05.003 · DOI
  26. 10.1080/10447319609526147 · DOI · Paper page
  27. 10.1080/001401300409044 · DOI · Paper page
  28. vig, 2005, A framework for multi-robot coalition formation, Proc Indian Int Conf Artif Intell (IICAI), 347
  29. 10.1145/1228716.1228738 · DOI
  30. 10.1518/001872095779049525 · DOI · Paper page
  31. 10.1177/1541931214581307 · DOI
  32. lizza, 1988, The Pilot’s Associate: A forum for the integration of knowledge-based systems and avionics, Proc IEEE Aerospace Electronics Conf (NAECON), 1252
  33. sen, 2015, Real-time optimal selection of multirobot coalition formation algorithms using conceptual clustering, Proc 29th AAAI Conf Artif Intell, 1
  34. 10.1016/S0950-7051(99)00035-0 · DOI
  35. champigneux, 1993, In flight mission planning in the ‘copilote electronique, New advances in mission planning and rehearsal systems
  36. bloom, 1996, A framework for distributed and virtual discrete part manufacturing, Proc CALS Expo, 1
  37. 10.1023/B:AURO.0000033970.96785.f2 · DOI · Paper page
  38. 10.1007/978-3-7908-1782-9_51 · DOI
  39. 10.1177/1059712312462248 · DOI · Paper page
  40. 10.1111/medu.12773 · DOI · Paper page
  41. 10.1207/S15326985EP3801_8 · DOI · Paper page
  42. 10.1177/0018720814539505 · DOI · Paper page
  43. 10.1016/j.cad.2013.10.002 · DOI
  44. 10.1177/0018720814564594 · DOI
  45. 10.2172/1261061 · DOI
  46. damacharla, 2018, Human error prediction using eye tracking to improvise team cohesion in human-machine teams, Proc of the 5th Int Conf on Applied Human Factors and Ergonomics (AHFE), 47
  47. 10.1177/0018720815587803 · DOI
  48. 10.1177/154193120605001114 · DOI
  49. 10.1016/j.jcomdis.2015.01.005 · DOI
  50. 10.1007/s11409-013-9109-4 · DOI
  51. calisi, 2008, A unified benchmark framework for autonomous Mobile robots and Vehicles Motion Algorithms (MoVeMA benchmarks), Proc Workshop Exp Methodol Benchmarking Robot Res (RSS), 1
  52. 10.1007/978-3-642-10203-5_26 · DOI
  53. 10.1016/j.jebo.2012.12.010 · DOI · Paper page
  54. 10.1080/02640410500127736 · DOI · Paper page
  55. 10.2514/6.2005-6912 · DOI
  56. 10.1080/10413200903204754 · DOI · Paper page
  57. 10.1177/154193128803200221 · DOI · Paper page
  58. mohr, 1996, Stages of Team Development
  59. cannan, 2011, Human-Machine Interaction (HMI) A Survey
  60. 10.1016/0167-6423(87)90035-9 · DOI · Paper page
  61. 10.1007/978-3-642-12029-9_25 · DOI
  62. 10.1109/ACC.2001.946096 · DOI
  63. 10.1109/ICSMC.2003.1243930 · DOI
  64. 10.1109/FUZZ-IEEE.2012.6250792 · DOI
  65. 10.1109/ICARCV.2010.5707803 · DOI
  66. 10.1177/154193129203600108 · DOI
  67. andre, 1997, Do you know what mode you’re in? An analysis of mode error in everyday things, Human-Automation Interaction Research and Practice, 19
  68. 10.5898/JHRI.3.1.Johnson · DOI · Paper page
  69. 10.1109/HAPTIC.2004.1287222 · DOI
  70. sullivan, 2015, F-35 Joint Strike Fighter Assessment Needed to Address Affordability Challenges
  71. 10.1109/MIS.2014.25 · DOI
  72. 10.1007/s10588-010-9065-3 · DOI
  73. 10.1080/00140139.2014.956151 · DOI · Paper page
  74. hicks, 2014, An Investigation of Multiple Unmanned Aircraft Systems Control From the Cockpit of an AH-64 Apache Heli-copter
  75. 10.1111/j.1936-4490.2003.tb00698.x · DOI
  76. 10.1007/978-1-4471-3812-9_2 · DOI
  77. 10.1007/s11920-014-0523-3 · DOI
  78. 10.1109/52.776955 · DOI
  79. 10.2307/3069401 · DOI · Paper page
  80. 10.1080/13561820120068980 · DOI · Paper page
  81. packer, 2002, Multimedia From Wagner to Virtual Reality
  82. 10.2189/asqu.51.4.590 · DOI · Paper page
  83. 10.1287/orsc.2014.0947 · DOI · Paper page
  84. 10.1037/0022-006X.43.6.800 · doi:10.1037/0022-006x.43.6.800 · Paper page
  85. meyer, 1999, Scheduling of actions and reliance onwarnings in a simulated control task, Proc Human Factors Ergonom Soc Annu Meeting, 43, 251, 10.1177/154193129904300326 · DOI
  86. 10.1093/occmed/kqg047 · DOI · Paper page
  87. 10.1177/154193129804200132 · DOI
  88. 10.1109/GLOCOM.2013.6831558 · DOI
  89. 2014, Complex Operational Decision Making in Networked Systems of Humans and Machines A Multidisciplinary Approach
  90. yazawa, 2013, Measurement of stress using DFA heartbeat analysis, Proc World Congr Eng Comput Sci, 2, 596
  91. 10.1155/2016/9631684 · DOI
  92. 10.21236/ADA545552 · DOI
  93. 10.1109/5.259423 · DOI · Paper page
  94. 10.21236/ADA509824 · DOI
  95. 10.1109/JPROC.2016.2520760 · DOI
  96. 10.21236/ADA525910 · DOI
  97. smaili, 2008, A simulation benchmark for aircraft survivability assessment, 26th Int Congr Aeronaut Sci, 9, 1
  98. 10.1080/00140130210166951 · DOI · Paper page
  99. 10.1075/is.8.3.10gro · DOI · Paper page
  100. 10.1080/00140139.2014.899630 · DOI
  101. olsen, 2003, Metrics for evaluating human-robot interactions, Proc PERMIS, 4
  102. gorbenko, 2012, Robot self-awareness: Exploration of internal states, Appl Math Sci, 6, 675
  103. 10.2308/accr-10038 · DOI · Paper page
  104. 10.1111/medu.12732 · DOI · Paper page
  105. weekley, 2013, Situational judgment tests Theory measurement and application, 10.4324/9780203774878 · DOI
  106. 10.1109/TRO.2007.907480 · DOI
  107. 10.1109/HRI.2013.6483569 · DOI
  108. 10.1046/j.1365-2648.2003.02770.x · DOI
  109. 10.1145/1121241.1121249 · DOI · Paper page
  110. elara, 2010, False alarm metrics: Evaluating safety in human robot interactions, Proc IEEE Conf Robot Autom Mechatron (RAM), 230
  111. 10.1163/016918610X527185 · DOI
  112. mcdaniel, 2001, Use of Situational Judgment Tests to Predict Job Performance A Clarification of the Literature
  113. 10.1109/TSMCA.2011.2164243 · DOI
  114. 10.1111/j.1365-2923.2012.04336.x · DOI · Paper page
  115. 10.1145/1357054.1357287 · DOI
  116. 10.1037/0021-9010.86.3.410 · doi:10.1037/0021-9010.86.3.410 · Paper page
  117. 10.1002/hbm.20309 · DOI · Paper page
  118. 10.1504/IJEM.2003.003297 · DOI · Paper page
  119. 10.1145/1228716.1228722 · DOI
  120. 10.1016/j.physa.2014.07.082 · DOI
  121. 10.1109/THMS.2013.2293535 · DOI · Paper page
  122. schechter, 2014, AUVSI Reporter AFRL Targets Seamless Human-Machine Interaction
  123. 10.1080/13825580601025908 · DOI
  124. 10.1007/978-3-642-29044-2_3 · DOI
  125. wilt, 2009, Handbook of Individual Differences in Social Behavior, 27
  126. 10.1109/ICSMC.2012.6378253 · DOI
  127. 10.1109/TSMCC.2007.900644 · DOI
  128. 10.1177/1073191115580639 · DOI
  129. 10.1109/THMS.2014.2331618 · DOI
  130. 10.1518/001872001775870395 · DOI · Paper page
  131. martin, 2003, Academic resilience and the four Cs: Confidence, control, composure, and commitment, Proc NZARE Austral Assoc Res Educ (AARE), 12
  132. 10.1109/COGINF.2005.1532647 · DOI
  133. 10.1177/0149206306289099 · DOI · Paper page
  134. 10.1016/S0005-7894(73)80120-0 · DOI · Paper page
  135. 10.1207/s15327043hup0603_4 · DOI · Paper page
  136. 10.1080/02699930802204677 · DOI · Paper page
  137. 10.1037/a0035083 · DOI · Paper page
  138. 10.1111/1467-9450.00243 · DOI
  139. 10.1109/ICSMC.2011.6084115 · DOI
  140. 10.1080/14639220110104925 · DOI
  141. pina, 2008, Identifying generalizable metric classes to evaluate human-robot teams, Proc 3rd Ann Conf Human-Robot Interaction, 13
  142. goodrich, 2012, On leadership and influence in human-swarm interaction, AAAI Fall Symposium Human Control of Bioinspired Swarms, 1
  143. 10.1109/TSMCC.2004.826287 · DOI
  144. 10.1109/THMS.2014.2343996 · DOI
  145. 10.1145/2696454.2696455 · DOI · Paper page
  146. 10.1007/s10514-015-9457-9 · DOI · Paper page
  147. 10.1109/TRO.2018.2795034 · DOI
  148. 10.1109/THMS.2016.2547186 · DOI
  149. 10.1177/0278364915609673 · DOI
  150. 10.1109/TSMCA.2005.850587 · DOI · Paper page
  151. 10.1177/152822X06289161 · DOI · Paper page
  152. bruemmer, 2003, Collaborative Tools for Mixed Teams of Humans and Robots
  153. 10.1109/AIS.2010.5547049 · DOI
  154. 10.1177/1557234X13501471 · DOI
  155. 10.1109/TWC.2016.2531652 · DOI · Paper page
  156. 10.1145/2701973.2702046 · DOI
  157. wilcox, 2011, Introduction to Robust Estimation and Hypothesis Testing
  158. christensen, 2014, Comparative measures of grit, tenacity and perseverance, International Journal of Learning Teaching and Educational Research, 8, 16
  159. 10.1109/MIS.2010.126 · DOI
  160. 10.1016/j.cose.2013.12.003 · DOI · Paper page
  161. 10.1109/TEM.2004.826011 · DOI
  162. ware, 1976, Scales for measuring general health perceptions, Health Services Res, 11, 396
  163. 10.1111/1467-9450.00192 · DOI · Paper page
  164. lupien, 2013, How to measure stress in humans, Centre for Studies in Human Stress
  165. 10.1109/MCOM.2016.7565185 · DOI · Paper page
  166. 10.1109/TIT.2013.2257911 · DOI
  167. 10.1109/JSAC.2016.2577199 · DOI
  168. 10.1016/S1048-9843(01)00093-5 · DOI · Paper page

Cited by

See all 97

Metadata

Authors
Damacharla, P., Javaid, A. Y., Gallimore, J. J., & Devabhaktuni, V. K.
Venue
IEEE Access
Published
2018
Volume
6
Pages
38637-38655
ISSN
2169-3536 (ISSN-L)
OpenAlex
W2856991417
Cited by
97
References
17
FWCI
6.44
Type
journal article
Language
en
Open access
gold
Funders
  • University of Toledo
  • Round 1 Award—Improving Healthcare Training and Decision Making Through LVC from the Ohio Federal Research Jobs Commission through Ohio Federal Research Network

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Common Metrics to Benchmark Human-Machine Teams (HMT): A Review. https://app.cochat.ai/papers/10.1109/access.2018.2853560
The paper
Damacharla, P., Javaid, A. Y., Gallimore, J. J., & Devabhaktuni, V. K. (2018). Common Metrics to Benchmark Human-Machine Teams (HMT): A Review. IEEE Access. https://doi.org/10.1109/access.2018.2853560

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.