The benefits of imperfect diagnostic automation: a synthesis of the literature

“Numerous contextual factors such as human operator knowledge and payoffs may well moderate this value considerably (Lee and See 2004).”

“( 2000 ), diagnostic aiding refers both to stage 1 automation (e.”

“In both of these cases, the automation benefits are likely to increase, to the extent that the single task to be automated is more difficult (Maltz and Shinar 2003, Dixon and Wickens 2006), and/or that concurrent task load is imposed (Parasuraman et al. 1993).”

“[36] C. D. Wickens and S. R. Dixon, “The benefits of imperfect diagnostic automation: A synthesis of the literature,” Theor.”

“With sufficiently valid cues, decisions improve over unaided performance (e.g., Bustamante, Anderson, & Bliss, 2004; Bustamante, Bliss, & Anderson, 2007; Dixon & Wickens, 2004; Maltz & Meyer, 2001; Meyer, 2001; Wickens & Dixon, 2007).”

“Discussions of trust in automation inevitably involve its performance in the environment (Lee & See, 2004; Parasuraman & Riley, 1997; Wickens & Dixon, 2007; Yeh & Wickens, 2001).”

“INTRODUCTION Automation is used to augment human performance by assisting the operator and relieving their workload (Wickens & Hollands, 2000), which can improve task performance (Wickens & Dixon, 2007).”

“Finally, Wickens and Dixon (2005) demonstrated that when the reliability level is below about 70%, operators will often ignore the alerts.”

“…tends to be sub-optimal (e.g. Dixon & Wickens, 2006; Dixon, Wickens & McCarley, 2007; Keller & Rice, in press; Rice, 2009; Rice & Keller, in press; Rice, Trafimow, Clayton & Hunt, in press; Wickens & Dixon, 2007), in that human-automation performance is often inferior to what we would like to see.”

“The automation reliability was set at 80% (Wickens and Dixon 2007), therefore, in each block, 10 trials provided incorrect assistance.”

“In a contrary case, there is a serious risk of overwhelming the operators with unnecessary warnings or alarms: they would not be able anymore to perform their task and they could underestimate critical situations [4, 5].”

“Therefore, alarms at 70% reliability threshold were used to ensure veridicality of the unreliable system (see Wickens & Dixon, 2007).”

“This high compliance rate is actually a rational strategy in order to optimize the amount of correct decisions in interaction with alarm systems and is very surprising, as such high response rates are usually observed in stages having a PPV above .7 (Wickens & Dixon, 2007).”

“A reliability of 80% was chosen as a more realistic accuracy, as systems are recommended to be more than 70% accurate (Wickens & Dixon, 2007) to be helpful.”

“Several other studies have benchmarked the desired reliability of AI to be 60-90% below which human trust is eroded [22].”

“That is, by an improvement in some combination of the miss rate and false alarm rate of the automation diagnosis, as these measures are frequently employed to evaluate warning systems (Meyer, 2004; Meyer & Lee, 2013; Wickens & Dixon, 2007).”

“…range from general, such as synthesizing the literature to estimate when automation performs well enough for the benefits to outweigh the costs (Wickens & Dixon, 2007), to the specific, by conducting a laboratory experiment to examine how automation error types affect performance (Dixon,…”

“This is consistent with previous literature, showing that human operators were adjusting their behavior based on how well the automation was performing [2].”

“…large number of informational cues may lend itself to a more algorithmic approach at least to information acquisition and analysis, two areas of automation in which users are generally more willing to accept occasionally faulty automated outcomes (Parasuraman et al., 2000; Wickens & Dixon, 2007).”

“Human performance in aided decision-making tasks can deteriorate when reliability of automated decision aid is low (Trapsilawati et al., 2015; Wickens & Dixon, 2007).”

“Notably, the 83.3 AP is above the 70% reliability threshold often used to support appropriate use (e.g. trust) in autonomy 27, 28 .”

“Considering the joint automation-human performance and in particular how the level of automation unreliability affects human performance – a research revealed that alarm systems should have a reliability factor of 70% [82].”

“In fact, Wickens and Dixon (2005) found that when the reliability of the automated system was about 70% or below, the performance with the automated system was no better than performance without the automated system.”

“The main objective of this effort was to determine if using a dual-task methodology was an effective way to evaluate an ATC automation prototype.”

“Reliability of this automation can improve task performance (Wickens & Dixon, 2007).”

“…was informed by prior human factors research suggesting that when automation reliability falls below approximately 70%, its costs may outweigh its benefits, highlighting the importance of studying user responses across a range of imperfect, yet still functional, automation performance levels [56].”

“Consistent with research on automation reliability research that showed that reliability of 70% is accepted as applicable by users [56], Bahnsen et al. [5] showed that an AR-cue reliability of 64% (i.e., the cue was always correct but displayed in only 64% of interruptions and otherwise no cue was…”

“This reliability was selected based on a quantitative literature review which suggested 70% accuracy is a “ tipping point ” beyond which unreliable automation becomes detrimental and performance outcomes are typically poorer than unaided (i.e., manual) performance (Wickens & Dixon, 2007).”

“This reliability rate was chosen to induce unreliable behavior while avoiding total disuse of the robot [23, 86].”

“…to improve reliance on them in static decision contexts (e.g., de Visser et al., 2017; Du et al., 2020; Lee et al., 1999; Rieger & Manzey, 2022; Wickens & Dixon, 2007), but it seems a safe assumption that most AI systems will not be implemented in real-world scenarios unless they already offer…”

“An automation bias could result in operators favouring suggestions or information from autonomous machine agents while discounting contradictory information from other sources, including their own judgement (Goddard et al., 2012; Parasuraman & Manzey, 2010; Wickens & Dixon, 2007).”

“Previous researches have shown that errors can influence operators ’ trust, compliance, and reliance on the system (Chancey et al., 2017b; Hutchinson et al., 2023; Liang et al., 2022; Wickens & Dixon, 2007).”

“If the aid’s reliability would be above the reliability threshold of 0.7 (Wickens and Dixon 2007), there would not be enough data in some of the conditions to be meaningfully analysed, which would compromise the usefulness of our design.”

“Though human performance generally is improved with automation, one concern for researchers and practitioners is that automated decision aids can often fail in operational settings (e.g., Wickens & Dixon, 2007).”

“60% was chosen as it is below the standard 70% threshold at which human participants begin to be impacted by poorly performing automation [40].”

“The reliability of both agents reflects a 70% minimal reliability of automated aids that are expected to improve performance compared to manual tasks (Wickens & Dixon, 2007), while maximizing trials in which errors could be observed to differentiate the FAR groups.”

“These selected percentages are taken from previous work detailing critical threshold points of reliability-induced automation complacency (Wickens & Dixon, 2007).”

“Even though perfectly reliable automation rarely exists, the meta-analysis by Wickens and Dixon (2007) indicated that even imperfect automation is beneficial if its reliability reaches 70% .”

“Research using signal detection paradigms has found that performance typically improves when an operator is assisted by an aid of moderate to high reliability (Bartlett & McCarley, 2017; Wickens & Dixon, 2007; Wiczorek & Manzey, 2014).”

“This accuracy level is based on both performance and trust issues that may arise when the system accuracy level is below ~70% (Lee & See, 2004; Wickens & Dixon, 2007).”

“In present day operational environment, automation often acts as a diagnostic aid (Wickens & Dixon, 2007) by gathering information and filtering it for relevance or correlating multiple sources of information.”

“The overall automation reliability will be set at 80%, which is above the threshold for imperfect reliability acceptance (Wickens & Dixon, 2007).”

“Subsequently, Wickens and Dixon (2007), after analyzing a total of 22 studies, concluded that 70% should be the threshold for considering a system as reliable.”

“Wickens and Dixon (2007), used reliability as the predictor variable in their analysis of how automation performance relates to overall system performance.”

“These accuracy levels were chosen partially based on prior research (Wickens & Dixon, 2007), but also to ensure operators would encounter a suitable amount of failures of both systems during the task.”

“This effect partially supports findings by Wickens and Dixon (2007) who propose a critical reliability cut-off around 70% below which automation support cannot be considered as helpful anymore.”

“Furthermore, Wickens and Dixon (2005) demonstrated that when the reliability level is below approximately 70%, operators often ignore the alerts.”

“The more automation is making decisions or taking action, the more accurate it needs to be in order for it to actually provide improvement to the system as a whole [27], while automating information acquisition and assessment or analysis stages should be at least 70% accurate [28].”

“In G-conditions, ingredient references were swapped with 𝑝 err = .40 probability (e.g., “ Teammate is chopping tomato ” became “ Teammate is chopping lettuce ”), yielding 60% reliability, below the 0.70 threshold at which automation becomes counterproductive [64].”

“The underlying AI classification accuracy was equivalent across the SoA conditions, ranging from 70 to 80%, exceeding the commonly cited 70% perceived usefulness threshold (Wickens & Dixon, 2007).”

“The system was configured to achieve approximately 88% reliability in order to be perceived as highly reliable while still producing occasional errors that allow the study of overreliance, consistent with prior human factors research on automation reliability [7, 96, 147].”

“However, our finding in Experiment 2 that participants ignored cues in both high and low reliability conditions runs in contrary to the results of prior studies that found a correlation between performance and reliability (Bliss & Acton, 2003; Wickens & Dixon, 2007).”

“…high consequence stimuli was assessed based on data from 70 participants in previous experiments (J. Kim et al., 2023, 2024 reliability between 70% and 80%, determined based on prior findings that costs tend to outweigh the benefits of automation with reliability below 70% (Wickens & Dixon, 2007).”

“Of the 15 trials each participant completed, the simulated autonomous system offered incorrect solutions in four randomly selected trials to replicate a real-world imperfect autonomous system and elicit realistic trust dynamics (Lee & See, 2004; Wickens & Dixon, 2007).”

“Past work has found that the value of a decision aid tends to decrease with the aid’s reliability. a meta-analysis Wickens and Dixon (2007), for instance, found that aids of reliability below 70% were of little benefit at all to human performance (see also Bartlett et al. in press; Bartlett and…”

“Each broker instance was configured to handle different topics and Quality of Service (QoS) levels, tailored to the specific data flow and throughput requirements of the industrial components [8].”

“Research has shown that higher functional reliability leads to higher automation use and, importantly, that there exists a "cross over point" of 70% reliability – below which the human may be better off without the automation (Wickens & Dixon, 2007).”

“While there is active research on the design of such systems to enhance human-automation performance (Hoff and Bashir, 2015; Wickens and Dixon, 2007), there is comparatively less work examining the role of individual differences in explaining human performance variations with automation (however,…”

“Designing for properly calibrated trust and reliance [4] can be achieved by giving information about how the automation works and what its limitations are [2], [5]–[7].”

“60% reliability was chosen for the Living Room location to ensure the user perceives that location as performing unreliably (Wickens & Dixon, 2007).”

“Further meta-analysis research [23] found an implication that degree of systems reliability should be maintained to be around 72% to benefit performance.”

“With a reliability rate of only 50%, the unreliable version of SearchBot fell well below the 70% threshold identified by Wickens and Dixon (2007).”

“How and to what extent operators adjust their own monitoring behaviour in case of the availability of (imperfect) alarm systems or other decision support has been addressed in several studies (e.g. Parasuraman et al., 1993; Wickens and Dixon, 2007).”

“Prior research suggests that humans tend to misuse MP systems, while they tend to disuse FP systems (Parasuraman and Riley 1997, Wickens and Dixon 2007).”