The benefits of imperfect diagnostic automation: a synthesis of the literature

“Numerous contextual factors such as human operator knowledge and payoffs may well moderate this value considerably (Lee and See 2004).”

“( 2000 ), diagnostic aiding refers both to stage 1 automation (e.”

“In both of these cases, the automation benefits are likely to increase, to the extent that the single task to be automated is more difficult (Maltz and Shinar 2003, Dixon and Wickens 2006), and/or that concurrent task load is imposed (Parasuraman et al. 1993).”

“If the automation is highly reliable (accurate) and the operator trusts it, compliance rates tend to go up, such that operators respond affirmatively to an automated system’s recommendation (Du et al., 2020; Huegli et al., 2023; Meyer, 2004; Wickens & Dixon, 2007).”

“…k -fold cross validation with six folds to obtain model statistics [24], the resulting RF and LR models classified ERT and non-ERT patients with an accuracy of 76% and 70%, respectively, which was at or above the theoretical threshold for automated decision aids to be more helpful than harmful [25].”

“The reliability of the robot, and level of automation [10, 11, 14, 15], and environmental r ∈ [0, 1], is defined as the probability that the robot can factors such as multi-tasking requirement [18].”

“Such a wide range of sensors provides a large quantity of heterogeneous information which has to be handled properly, in terms of pre-processing, integration and reasoning, in order to effectively support PSIM operators; otherwise, there is the serious risk of overwhelming operators with unnecessary information, warni…”

“Wickens and Dixon (2007) published a literature review investigating the effects of signaling system reliability on human performance, specifically reaction time and accuracy.”

“Taken together, these fi ndings are consistent with the notion that providing recommendations can enhance human decision making (Bartlett & McCarley, 2017, 2021; Wickens & Dixon, 2007).”

“Prior work indicates that people might rely more on automation in a high workload condition [81].”

“Wickens and Dixon (2007) characterized automation performance using a single metric: reliability, or the total proportion of correct responses: (1) where H = total number of hits, CR = correct rejections, M = misses, and FA = false alerts.”

“Other benefits of automation include improving task performance and reducing operator workload (Wickens & Dixon, 2007).”

“…al., 1997), as previous research has stated that false alarms could be more harmful to a user’s performance than miss rate (Bliss, 2003; Dixon et al., 2006; Wickens & Dixon, 2007), while on the other hand a certain level of miss rate is tolerable, especially in demanding multi-tasking situations.”

“According to Muir and colleagues, user acceptance of automation is highly dependent on many factors, one of which is trust that the technology will function as expected (Lee and See 2004; Muir 1987, 1994; Wickens and dixon 2007).”

“When an automated system is not completely reliable the automation risks being useless, harming performance (Wickens and Dixon 2007), decreasing user trust in the system and even causing users to ignore suggestions completely or to disable the suggestions offered by the system.”

“…ratio was motivated by other studies using autonomous decision aid reliability in the 60% – 90% range (Akash et al., 2020; Nunnally, 1978; Petersen et al., 2019; Wickens & Dixon, 2007; Wickens et al., 2020), which has been shown to produce realistic trust dynamics in a short experiment time frame.”

“Probabilities may have little value, though, if a system has such great predictive value that humans just defer to it or such little predictive value that humans stop using it (Meyer, 2004; Parasuraman & Riley, 1997; Wickens & Dixon, 2007).”

“Automation functioned reliably for 24 out of 30 con fl icts, with participants required to manually intervene to the six automation failures (80% automation reliability to ensure that reliability was high enough to encourage individuals to rely on automated advice; Wickens & Dixon, 2007).”

“We chose 60% reliability because systems performing below 70% reliability are considered unreliable (Wickens and Dixon 2007; Kantowitz, Hanowski, and Kantowitz 1997).”

“Their fi ndings regarding performance were aligned with earlier studies in which more reliable automation systems improve performance, but low reliability systems may not be bene fi cial to performance when compared to no-automation control conditions (Wickens & Dixon, 2007).”

“Both imperfect warning systems improved the motorcyclist performance compared to the non-warning system and it seems to facilitate the sensory processing of the target and results are in line with those from [37] and [53].”

“While studies have examined the use of automation reliability as an aid to the user (Dixon and Wickens 2006; Wickens and Dixon 2007; Parasuraman and Manzey 2010) there is a lack of depth in the literature that addresses when an automated agent, who acts like a teammate rather than an aid, suffers…”

“We chose 70% reliability level based on the Wickens and Dixon (2007) study which reported that for imperfect automation systems to be effective; the systems should at least be 70% reliable.”

“In reference to the case of joint automation-human performance, a research revealed that alarms systems should have a reliability factor of 70% [ 15 ].”

“be more harmful to the user’s performance than MISS [1, 4, 10].”

“…et al. (2015, 2016a) nevertheless found that such an imperfect (80% reliable) CRA still supported ATCO conflict avoidance performance, well above the level of unaided manual task performance, and similar findings have been observed with other imperfect automation aids (Wickens and Dixon, 2007).”

“Well-designed automation generally reduces the effort that a person needs to expend and improves task performance (e.g., Wickens & Dixon, 2007).”

“…an important consideration is operator span-of control, i.e., the number of assets that a human is able to operate at any given point in time (e.g. Crandall et al., 2005; Cummings & Guerlain, 2004; Endsley & Rodgers, 1997; Dixon et al., 2005; Wickens & Dixon, 2007; Wickens et. al, 2013).”

“These demands can result in decreased efficiency and safety (Maltz & Shinar, 2003; Sheridan, 1997; Wickens & Dixon, 2007).”

“Wickens and Dixon (2007) used data from 20 studies to identify the value at which automation (relative to performance when no automation is present) fails to benefit the operator or becomes a drawback because of its level of reliability and whether such a value is influenced by task workload.”

“Generally, higher reliability leads to higher signal dependence rates (Manzey, Gerard, & Wiczorek, 2014), quicker signal response times (Chancey, Bliss, Proaps, & Madhavan, 2015), and greater user sensitivity (Rice, 2009) (for review, see Wickens & Dixon, 2007).”

“In an attempt to elicit measurable differences in response behaviors, we elected reliability levels above and below a 71% cutoff (see Wickens & Dixon, 2007, for meta-analysis).”

“That is, when automation is much less than 70% reliable, the operator begins to behave as if there was no automation present (Wickens & Dixon, 2007).”

“This is more likely with automation that is more reliable which acts to increase trust (Wickens & Dixon, 2007).”

“| 411 literature, with an accuracy below 70% automated aids tend to lose their benefits, and aided performance drops below the level of manual performance (Wickens & Dixon, 2007).”

“…of the AI system (e.g., by making it non-functional or non-responsive), we trivialize the proposed research questions, as prior work has widely addressed the relationship between faulty systems and subjective attitudes (Parasuraman, 1997; Parasuraman et al., 1993; Wickens & Dixon, 2007).”

“The FAprone or miss-prone system operated at 70% reliability, at which point operators may still depend upon the automation even though their own unaided performance would likely be superior (Wickens & Dixon, 2007).”

“This stands in contrast with the results of previous studies in performance, where 67% reliability was shown to neither hinder nor help (Wickens & Dixon, 2007), although this study was measuring attention allocation and not task performance.”

“With highly automated tasks, the operator is less vigilant, and, as a result, will have a harder time regaining awareness should something go wrong [3-4].”

“Unfortunately, empirical performance of automation-aided humans is rarely if ever optimal (Parasuraman & Riley, 1997) and often fails to even match the performance of the automation by itself (see Wickens & Dixon, 2007, for review).”

“Although imperfect automation might still contribute to overall performance, [17] concluded that when the reliability level is less than 70%, the imperfect automation would be even worse than no automated aids.”

“We hypothesized that the more reliable the CID aid is the more accurate the Soldier would be at making friend-foeidentifications via target shot choice (Wang et al., 2009, Wickens & Dixon, 2007).”

“…the latter two stages (e.g. Sarter and Schroeder 2001, Rovira et al. 2007), most studies have focused on the early two stages (e.g. Horrey and Wickens 2001, Yeh and Wickens 2001), with particular focus on diagnostic automation (e.g. Dixon et al. 2007, Wickens and Dixon 2007, Rice and Keller 2009).”

“Finally, RADAR can be incrementally deployed with increasing rates of automation over time in order to maximize the benefits of automation and minimize the detriments [39].”

“The effect of automation reliability on task performance has been examined in many studies (Onnasch, 2015; Rovira et al., 2002, 2007; Wickens & Dixon, 2007), which can be classi fi ed as either studies on failure performance or studies on joint human-automation performance (i.e., the combination of…”

“Prior work suggests that the reliability below 70% is considered worse than no automation (Wickens & Dixon, 2007).”

“Prior efforts have suggested that automation reliability below 70% is considered worse than no automation at all [52].”

“Prior work has suggested that automation with reliability below 70% is considered worse than no automation [53].”

“In previous studies, reliability levels between 60% and 80% represented a threshold of performance decline (Rovira et al. 2007; Wickens and Dixon, 2005).”

“Despite the costs of automation unreliability, studies have also shown that imperfect automation can still yield system performance benefits above baseline performance (Wickens & Dixon, 2005).”

“A well-known crossover point beyond which automation is deemed unreliable is 70% (Wickens & Dixon, 2007).”

“Research indicates that the benefits of automated processes decrease when reliability is below approximately 70% [72].”

“However, DS is not guaranteed to be helpful (Parasuraman & Manzey, 2010) and can even compromise performance (Wickens & Dixon, 2007).”

“This reliability level was chosen to acknowledge previous research findings indicating that people’s performance with automation that is less than 70% reliable could be worse than with no automation at all (Wickens & Dixon, 2007).”

“Note that our use of 80% is not meant to suggest clinical or managerial relevance to applying a threshold probability >80%, because we doubt its relevance for that purpose [9,10].”

“…interact can produce outsized positive (Corcoran et al., 1972; Parasuraman, 1987; Thackray & Touchstone, 1989; Woods & Hollnagel, 2006) or negative (Robinson & Sorkin, 1985; Sorkin & Woods, 1985; Dalal & Kasper, 1994; Smith et al., 1997; Wickens & Dixon, 2007) effects on the system as a whole.”

“Researchers have speculated various factors that influence the human ’ s use of the automation including the display design (e.g., Yamani & McCarley, 2018) and aid ’ s reliability (e.g., Wickens & Dixon, 2007).”

“Thus, it functioned as diagnostic aid (Wickens & Dixon, 2007).”

“Automation typically lowers operators’ overall workload [4], and in mundane conditions, automation can maintain more consistent and accurate performance than that obtained by human operators.”

“Wickens and Dixon (2007) reviewed the findings of 20 studies that used automation’s reliability as an independent variable to quantify its effects on task performance.”

“Previous research also showed that for the automated alert that directs attention allocation, reliability higher than 67% would benefit operators than the cost of errors (Wickens & Dixon, 2007).”

“At 67% reliability, where Wickens and Dixon (2007) suggest automation neither helps nor hinders performance, we expected a similar level of allocation behaviors to those without any automation support.”

“These errors decrease task performance and user satisfaction, and causes decision errors and complacency effects (Gajos, Everitt, Tan, Czerwinski, & Weld, 2008; Rovira, McGarry, & Parasuraman, 2007; Wickens et al., 2007).”

“The researcher selected these reliability levels because 60% reliability is the lowest level of reliability at which automation is still useful (Wickens & Dixon, 2007), and 97.25% reliability represents a high reliability level while accounting for the fact that automation is imperfect.”

“Indeed, when reliability falls below a critical threshold (in the range of 70–75%), automation may lead to increased errors and inefficiencies, perhaps yielding inferior performance compared to the absence of automation (Wickens & Dixon, 2007).”

“…filter acted with high reliability (87.5% accuracy), and half were told it acted with low reliability (56.25% accuracy) - past research has demonstrated that 70% reliability is the minimum threshold for automated aids (e.g., Parasuraman & Manzey, 2010; Rovira et al., 2007; Wickens & Dixon, 2007).”

“Theoretically, two metaanalyses by Wickens and Dixon [15] and Rein et al. [16] have shown that automation with reliability below 70% can still yield performance gains, suggesting the need for investigation into trust in this lower reliability range.”

“The reliability followed the suggestion of synthesizing literature that if automation performance falls below 70% reliability, the costs will outweigh the benefits (Wickens & Dixon, 2007).”

“In situations where automation has not reached an adequate level, its control or operation may not yield better results than manual operation, particularly in circumstances that require logical judgment before deciding how to control or operate (Balfe et al., 2015; Wickens & Dixon, 2007).”

“This paper outlines one human-centered machine learning (HCML) approach which deliberately invested in the potential for human-machine teaming (HMT) in all phases.”

“This shift not only allocates valuable resources for strategic investment but also equips SMEs to maintain competitiveness in the dynamic market landscape, swiftly adapting to evolving technologies and processes [3], [4].”

“This ratio of autonomous system failure (4 of 15) aimed to replicate realistic trust dynamics in a short experiment, with a system that was useful despite its errors [11], [28], [52], [53].”

“Moreover, our finding is consistent with past research which suggests an automated system reliability of 70% as the threshold above which users perceive an automated system to be reliable [28].”