Common Metrics to Benchmark Human-Machine Teams (HMT): A Review
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
A significant amount of work is invested in human-machine teaming (HMT) across multiple fields. Accurately and effectively measuring system performance of an HMT is crucial for moving the design of these systems forward. Metrics are the enabling tools to devise a benchmark in any system and serve as an evaluation platform for assessing the performance, along with the verification and validation, of a system. Currently, there is no agreed-upon set of benchmark metrics for developing HMT systems. Therefore, identification and classification of common metrics are imperative to create a benchmark in the HMT field. The key focus of this review is to conduct a detailed survey aimed at identification of metrics employed in different segments of HMT and to determine the common metrics that can be used in the future to benchmark HMTs. We have organized this review as follows: identification of metrics used in HMTs until now, and classification based on functionality and measuring techniques. Additionally, we have also attempted to analyze all the identified metrics in detail while classifying them as theoretical, applied, real-time, non-real-time, measurable, and observable metrics. We conclude this review with a detailed analysis of the identified common metrics along with their usage to benchmark HMTs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
NASA-Task Load Index (NASA-TLX); 20 Years Later
Hart, S. G. (2006) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120605000909 · Cited by 4,297
A Temporally Based Framework and Taxonomy of Team Processes
Marks, M. A., Mathieu, J. E., & Zaccaro, S. J. (2001) · The Academy of Management Review · doi:10.2307/259182 · Cited by 3,659
Stages of Small-Group Development Revisited
Tuckman, B. W., & Jensen, M. A. C. (1977) · Group & Organization Studies · doi:10.1177/105960117700200404 · Cited by 2,679
A methodological review of resilience measurement scales
Windle, G., Bennett, K. M., & Noyes, J. (2011) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-9-8 · Cited by 2,079
Measurement of Situation Awareness in Dynamic Systems
Endsley, M. R. (1995) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872095779049499 · Cited by 2,039
Licklider, J. C. R. (1960) · IRE Transactions on Human Factors in Electronics · doi:10.1109/thfe2.1960.4503259 · Cited by 1,938
Ash, J. S. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1471 · Cited by 1,789
We Feel, Therefore We Learn: The Relevance of Affective and Social Neuroscience to Education
Immordino‐Yang, M. H., & Damasio, A. (2007) · Mind, Brain, and Education · doi:10.1111/j.1751-228x.2007.00004.x · Cited by 1,463
Study of Subjective and Objective Quality Assessment of Video
Seshadrinathan, K., Soundararajan, R., Bovik, A. C., & Cormack, L. K. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2042111 · Cited by 1,160
Neuroticism, Somatic Complaints, and Disease: Is the Bark Worse than the Bite?
Costa, P. T., & McCrae, R. R. (1987) · Journal of Personality · doi:10.1111/j.1467-6494.1987.tb00438.x · Cited by 1,010
Nasa-task load index (NASA-TLX); 20 years later: (577632012-009)
Hart, S. G. (2006) · PsycEXTRA Dataset · doi:10.1037/e577632012-009 · Cited by 838
Chen, M., Mozaffari, M., Saad, W., Yin, C., Debbah, M., & Hong, C. S. (2017) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2017.2680898 · Cited by 742
Viewport-adaptive navigable 360-degree video delivery
Corbillon, X., Simon, G., Devlic, A., & Chakareski, J. (2017) · 2017 IEEE International Conference on Communications (ICC) · doi:10.1109/icc.2017.7996611 · Cited by 341
Using psycho-physiological measures to assess task difficulty in software development
Fritz, T., Begel, A., Müller, S. C., Yigit-Elliott, S., & Züger, M. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568266 · Cited by 262
Measuring Team Cohesion: Observations from the Science
Salas, E., Grossman, R., Hughes, A. M., & Coultas, C. W. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720815578267 · Cited by 236
Can emotional intelligence be measured and developed?
Dulewicz, V., & Higgs, M. (1999) · Leadership & Organization Development Journal · doi:10.1108/01437739910287117 · Cited by 221
Devena, S. E., & Watkins, M. W. (2012) · Journal of Applied School Psychology · doi:10.1080/15377903.2012.669743 · Cited by 47
Research works printed but not linked: 168
- 10.1145/2480741.2480748 · DOI
- 10.1109/IROS.2009.5354484 · DOI
- 10.1016/S0004-3702(99)00070-3 · DOI · Paper page
- 10.1109/ROBOT.2006.1642269 · DOI
- 10.1177/104649640003100105 · DOI · Paper page
- 10.1145/1865909.1865950 · DOI
- 10.1177/101269027701200203 · DOI · Paper page
- damacharla, 2018, Autonomous ground vehicle error prediction modeling to facilitate human-machine cooperation, Proc of the 5th Int Conf on Applied Human Factors and Ergonomics (AHFE), 36
- 10.1109/ICPR.2004.1333825 · DOI
- bray-miners, 2012, Human-Robot Interaction Literature Review
- 10.1016/j.robot.2010.07.002 · DOI · Paper page
- 10.1109/MIS.2011.47 · DOI
- 10.1109/HRI.2010.5453244 · DOI
- 10.1109/TRA.2002.803463 · DOI · Paper page
- bainbridge, 2004, Berkshire Encyclopedia of Human-Computer Interaction
- 10.1145/2594128 · DOI · Paper page
- bradshaw, 2003, Dimensions of adjustable autonomy and mixed-initiative interaction, Proc Int Workshop Comput Auto, 17
- dekoven, 2006, A Framework for Supporting Team-work Between Humans and Autonomous Systems
- 10.1002/rob.21555 · DOI
- 10.1007/978-0-85729-883-6_12 · DOI
- 10.1007/s40593-018-0166-3 · DOI
- 10.1016/S1474-4422(13)70305-3 · DOI · Paper page
- 10.1177/027836498700600201 · DOI · Paper page
- 10.1109/ROBOT.2004.1307477 · DOI
- 10.1016/j.ijhcs.2011.05.003 · DOI
- 10.1080/10447319609526147 · DOI · Paper page
- 10.1080/001401300409044 · DOI · Paper page
- vig, 2005, A framework for multi-robot coalition formation, Proc Indian Int Conf Artif Intell (IICAI), 347
- 10.1145/1228716.1228738 · DOI
- 10.1518/001872095779049525 · DOI · Paper page
- 10.1177/1541931214581307 · DOI
- lizza, 1988, The Pilot’s Associate: A forum for the integration of knowledge-based systems and avionics, Proc IEEE Aerospace Electronics Conf (NAECON), 1252
- sen, 2015, Real-time optimal selection of multirobot coalition formation algorithms using conceptual clustering, Proc 29th AAAI Conf Artif Intell, 1
- 10.1016/S0950-7051(99)00035-0 · DOI
- champigneux, 1993, In flight mission planning in the ‘copilote electronique, New advances in mission planning and rehearsal systems
- bloom, 1996, A framework for distributed and virtual discrete part manufacturing, Proc CALS Expo, 1
- 10.1023/B:AURO.0000033970.96785.f2 · DOI · Paper page
- 10.1007/978-3-7908-1782-9_51 · DOI
- 10.1177/1059712312462248 · DOI · Paper page
- 10.1111/medu.12773 · DOI · Paper page
- 10.1207/S15326985EP3801_8 · DOI · Paper page
- 10.1177/0018720814539505 · DOI · Paper page
- 10.1016/j.cad.2013.10.002 · DOI
- 10.1177/0018720814564594 · DOI
- 10.2172/1261061 · DOI
- damacharla, 2018, Human error prediction using eye tracking to improvise team cohesion in human-machine teams, Proc of the 5th Int Conf on Applied Human Factors and Ergonomics (AHFE), 47
- 10.1177/0018720815587803 · DOI
- 10.1177/154193120605001114 · DOI
- 10.1016/j.jcomdis.2015.01.005 · DOI
- 10.1007/s11409-013-9109-4 · DOI
- calisi, 2008, A unified benchmark framework for autonomous Mobile robots and Vehicles Motion Algorithms (MoVeMA benchmarks), Proc Workshop Exp Methodol Benchmarking Robot Res (RSS), 1
- 10.1007/978-3-642-10203-5_26 · DOI
- 10.1016/j.jebo.2012.12.010 · DOI · Paper page
- 10.1080/02640410500127736 · DOI · Paper page
- 10.2514/6.2005-6912 · DOI
- 10.1080/10413200903204754 · DOI · Paper page
- 10.1177/154193128803200221 · DOI · Paper page
- mohr, 1996, Stages of Team Development
- cannan, 2011, Human-Machine Interaction (HMI) A Survey
- 10.1016/0167-6423(87)90035-9 · DOI · Paper page
- 10.1007/978-3-642-12029-9_25 · DOI
- 10.1109/ACC.2001.946096 · DOI
- 10.1109/ICSMC.2003.1243930 · DOI
- 10.1109/FUZZ-IEEE.2012.6250792 · DOI
- 10.1109/ICARCV.2010.5707803 · DOI
- 10.1177/154193129203600108 · DOI
- andre, 1997, Do you know what mode you’re in? An analysis of mode error in everyday things, Human-Automation Interaction Research and Practice, 19
- 10.5898/JHRI.3.1.Johnson · DOI · Paper page
- 10.1109/HAPTIC.2004.1287222 · DOI
- sullivan, 2015, F-35 Joint Strike Fighter Assessment Needed to Address Affordability Challenges
- 10.1109/MIS.2014.25 · DOI
- 10.1007/s10588-010-9065-3 · DOI
- 10.1080/00140139.2014.956151 · DOI · Paper page
- hicks, 2014, An Investigation of Multiple Unmanned Aircraft Systems Control From the Cockpit of an AH-64 Apache Heli-copter
- 10.1111/j.1936-4490.2003.tb00698.x · DOI
- 10.1007/978-1-4471-3812-9_2 · DOI
- 10.1007/s11920-014-0523-3 · DOI
- 10.1109/52.776955 · DOI
- 10.2307/3069401 · DOI · Paper page
- 10.1080/13561820120068980 · DOI · Paper page
- packer, 2002, Multimedia From Wagner to Virtual Reality
- 10.2189/asqu.51.4.590 · DOI · Paper page
- 10.1287/orsc.2014.0947 · DOI · Paper page
- 10.1037/0022-006X.43.6.800 · doi:10.1037/0022-006x.43.6.800 · Paper page
- meyer, 1999, Scheduling of actions and reliance onwarnings in a simulated control task, Proc Human Factors Ergonom Soc Annu Meeting, 43, 251, 10.1177/154193129904300326 · DOI
- 10.1093/occmed/kqg047 · DOI · Paper page
- 10.1177/154193129804200132 · DOI
- 10.1109/GLOCOM.2013.6831558 · DOI
- 2014, Complex Operational Decision Making in Networked Systems of Humans and Machines A Multidisciplinary Approach
- yazawa, 2013, Measurement of stress using DFA heartbeat analysis, Proc World Congr Eng Comput Sci, 2, 596
- 10.1155/2016/9631684 · DOI
- 10.21236/ADA545552 · DOI
- 10.1109/5.259423 · DOI · Paper page
- 10.21236/ADA509824 · DOI
- 10.1109/JPROC.2016.2520760 · DOI
- 10.21236/ADA525910 · DOI
- smaili, 2008, A simulation benchmark for aircraft survivability assessment, 26th Int Congr Aeronaut Sci, 9, 1
- 10.1080/00140130210166951 · DOI · Paper page
- 10.1075/is.8.3.10gro · DOI · Paper page
- 10.1080/00140139.2014.899630 · DOI
- olsen, 2003, Metrics for evaluating human-robot interactions, Proc PERMIS, 4
- gorbenko, 2012, Robot self-awareness: Exploration of internal states, Appl Math Sci, 6, 675
- 10.2308/accr-10038 · DOI · Paper page
- 10.1111/medu.12732 · DOI · Paper page
- weekley, 2013, Situational judgment tests Theory measurement and application, 10.4324/9780203774878 · DOI
- 10.1109/TRO.2007.907480 · DOI
- 10.1109/HRI.2013.6483569 · DOI
- 10.1046/j.1365-2648.2003.02770.x · DOI
- 10.1145/1121241.1121249 · DOI · Paper page
- elara, 2010, False alarm metrics: Evaluating safety in human robot interactions, Proc IEEE Conf Robot Autom Mechatron (RAM), 230
- 10.1163/016918610X527185 · DOI
- mcdaniel, 2001, Use of Situational Judgment Tests to Predict Job Performance A Clarification of the Literature
- 10.1109/TSMCA.2011.2164243 · DOI
- 10.1111/j.1365-2923.2012.04336.x · DOI · Paper page
- 10.1145/1357054.1357287 · DOI
- 10.1037/0021-9010.86.3.410 · doi:10.1037/0021-9010.86.3.410 · Paper page
- 10.1002/hbm.20309 · DOI · Paper page
- 10.1504/IJEM.2003.003297 · DOI · Paper page
- 10.1145/1228716.1228722 · DOI
- 10.1016/j.physa.2014.07.082 · DOI
- 10.1109/THMS.2013.2293535 · DOI · Paper page
- schechter, 2014, AUVSI Reporter AFRL Targets Seamless Human-Machine Interaction
- 10.1080/13825580601025908 · DOI
- 10.1007/978-3-642-29044-2_3 · DOI
- wilt, 2009, Handbook of Individual Differences in Social Behavior, 27
- 10.1109/ICSMC.2012.6378253 · DOI
- 10.1109/TSMCC.2007.900644 · DOI
- 10.1177/1073191115580639 · DOI
- 10.1109/THMS.2014.2331618 · DOI
- 10.1518/001872001775870395 · DOI · Paper page
- martin, 2003, Academic resilience and the four Cs: Confidence, control, composure, and commitment, Proc NZARE Austral Assoc Res Educ (AARE), 12
- 10.1109/COGINF.2005.1532647 · DOI
- 10.1177/0149206306289099 · DOI · Paper page
- 10.1016/S0005-7894(73)80120-0 · DOI · Paper page
- 10.1207/s15327043hup0603_4 · DOI · Paper page
- 10.1080/02699930802204677 · DOI · Paper page
- 10.1037/a0035083 · DOI · Paper page
- 10.1111/1467-9450.00243 · DOI
- 10.1109/ICSMC.2011.6084115 · DOI
- 10.1080/14639220110104925 · DOI
- pina, 2008, Identifying generalizable metric classes to evaluate human-robot teams, Proc 3rd Ann Conf Human-Robot Interaction, 13
- goodrich, 2012, On leadership and influence in human-swarm interaction, AAAI Fall Symposium Human Control of Bioinspired Swarms, 1
- 10.1109/TSMCC.2004.826287 · DOI
- 10.1109/THMS.2014.2343996 · DOI
- 10.1145/2696454.2696455 · DOI · Paper page
- 10.1007/s10514-015-9457-9 · DOI · Paper page
- 10.1109/TRO.2018.2795034 · DOI
- 10.1109/THMS.2016.2547186 · DOI
- 10.1177/0278364915609673 · DOI
- 10.1109/TSMCA.2005.850587 · DOI · Paper page
- 10.1177/152822X06289161 · DOI · Paper page
- bruemmer, 2003, Collaborative Tools for Mixed Teams of Humans and Robots
- 10.1109/AIS.2010.5547049 · DOI
- 10.1177/1557234X13501471 · DOI
- 10.1109/TWC.2016.2531652 · DOI · Paper page
- 10.1145/2701973.2702046 · DOI
- wilcox, 2011, Introduction to Robust Estimation and Hypothesis Testing
- christensen, 2014, Comparative measures of grit, tenacity and perseverance, International Journal of Learning Teaching and Educational Research, 8, 16
- 10.1109/MIS.2010.126 · DOI
- 10.1016/j.cose.2013.12.003 · DOI · Paper page
- 10.1109/TEM.2004.826011 · DOI
- ware, 1976, Scales for measuring general health perceptions, Health Services Res, 11, 396
- 10.1111/1467-9450.00192 · DOI · Paper page
- lupien, 2013, How to measure stress in humans, Centre for Studies in Human Stress
- 10.1109/MCOM.2016.7565185 · DOI · Paper page
- 10.1109/TIT.2013.2257911 · DOI
- 10.1109/JSAC.2016.2577199 · DOI
- 10.1016/S1048-9843(01)00093-5 · DOI · Paper page
Cited by
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
Xu, W., Dainoff, M. J., Ge, L., & Gao, Z. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2041900 · Cited by 363
Smart Energy Efficient Home Automation System Using IoT
Vishwakarma, S. K., Upadhyaya, P., Kumari, B., & Mishra, A. K. (2019) · 2019 4th International Conference on Internet of Things: Smart Innovation and Usages (IoT-SIU) · doi:10.1109/iot-siu.2019.8777607 · Cited by 315
Coronado, E., Kiyokawa, T., Ricardez, G. A. G., Ramirez-Alpizar, I. G., Venture, G., & Yamanobe, N. (2022) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2022.04.007 · Cited by 276
Challenges of human—machine collaboration in risky decision-making
Xiong, W., Fan, H., Ma, L., & Wang, C. (2022) · Frontiers of Engineering Management · doi:10.1007/s42524-021-0182-0 · Cited by 95
Metadata
- Authors
- Damacharla, P., Javaid, A. Y., Gallimore, J. J., & Devabhaktuni, V. K.
- Venue
- IEEE Access
- Published
- 2018
- Volume
- 6
- Pages
- 38637-38655
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2856991417
- Cited by
- 97
- References
- 17
- FWCI
- 6.44
- Type
- journal article
- Language
- en
- Open access
- gold
- Funders
- University of Toledo
- Round 1 Award—Improving Healthcare Training and Decision Making Through LVC from the Ohio Federal Research Jobs Commission through Ohio Federal Research Network
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.