The benefits of imperfect diagnostic automation: a synthesis of the literature

“Numerous contextual factors such as human operator knowledge and payoffs may well moderate this value considerably (Lee and See 2004).”

“( 2000 ), diagnostic aiding refers both to stage 1 automation (e.”

“In both of these cases, the automation benefits are likely to increase, to the extent that the single task to be automated is more difficult (Maltz and Shinar 2003, Dixon and Wickens 2006), and/or that concurrent task load is imposed (Parasuraman et al. 1993).”

“Wickens and Dixon (2007) proposed that this cutoff value is approximately 70% (with a standard error of ± 14%), on the basis of studies examining the beneficial effects of automation support.”

“The presence of competing tasks has also been found to increase trust in automation in a metaanalysis of studies (Wickens & Dixon, 2007), thus shifting operator attention away from the automation and decreasing SA of the functions being automated (Carsten, Lai, Barnard, Jamson, & Merat, 2012; Kaber…”

“It is also in line with other authors who also departed from the classic approach for similar reasons (e.g., Hutchins, Wickens, Carolan, & Cumming, 2013; Wickens & Dixon, 2007; Wickens et al., 2013; Wickens, Hooey, Gore, Sebok, & Koe-nicke, 2009).”

“These findings have been replicated ( Scerbo, 2007) and modeled computationally (Farrell & Lewandowsky, 2000; Wickens & Dixon, 2007).”

“Such resource scarcity may result either when the task itself is difficult (Maltz & Shinar, 2003) or when the automated task is carried out in a multitask context (C. D. Wickens & Dixon, 2005).”

“In a recent review, Wickens and Dixon (2005) suggested that human performance is sensitive to the level of automation imperfection.”

“This is particularly true of diagnostic automation, which must make diagnoses and long-range predictions in a world of imperfect probabilistic information (Wickens & Dixon, 2007).”

“A similar approach was applied earlier by Wickens and Dixon (2007) who examined stage 2 automation reliability.”

“target cueing is ineffective below a certain level of reliability (approximately 60%–70%) [32], [33].”

“Despite the costs of automation unreliability, studies have also shown that imperfect automation can still yield system performance benefits above baseline performance (Wickens & Dixon, 2005).”

“Note that 75% reliability is within the confidence interval of the automation effectiveness threshold identified by Wickens and Dixon (2007). They also reported that dependence on automation is stronger at higher levels of task demand, even in situations in which the “raw data” are available to the operator.”

“Our results seem to align well with the results of Wickens and Dixon (2007) and suggest that CID performance is indeed similar to the results typically obtained in other automation domains.”

“An additional challenge relevant to automated driving is the presence of concurrent tasks, which further promote the attention allocation away from the automated task (Wickens and Dixon 2007).”

“Using a relatively simple regression model, with weights based on a meta-analysis, Wickens and Dixon (2007) found that such calibration was reasonably good, so long as automation reliability remained above about 80%; but was found wanting when automation erred more frequently.”

“Thus, system designers should use highly reliable automation when possible because less reliable automation tends to harm human–automation performance to such a degree that one might argue that no automation at all is more beneficial than unreliable automation (Wickens & Dixon, 2007).”

“The approach based on d′ asks how low the reliability of an alerting or diagnostic system can get while preserving performance that is superior to that of the unaided human (Wickens & Dixon, 2007; Xu Rantanen & Wickens, 2007).”

“The 3:1 (system maneuver : take-over) ratio was chosen in order to get 75% system reliability for interaction scenarios as proposed by [21].”

“For example, Wickens and Dixon (2007) found that in some cases operators relied on automated aids even when it resulted in worse diagnostic monitoring than when the operator had not used automation at all.”

“In their study of sequential effects on 8 perceptions of job candidates, Wexley et al. (1972) found that up to 12% of variance in ratings of candidates could be attributed to a contrast effect from the two candidates preceding the target candidate.”

“Finally, Wickens and Dixon (2007) demonstrated that when the reliability level is below approximately 70%, operators will often ignore the alerts.”

“We chose 50% reliability as a control condition to match the lottery condition typically used in oxytocin research with economic games and because automation below 70% reliability is generally considered unreliable (Wickens & Dixon, 2007).”

“In many complex systems, the number or complexity of the tasks is too much for human operators to handle safely or efficiently (Sheridan, 1987; Wickens & Dixon, 2007).”

“Although some studies have not shown such negative effects (e.g. Parasuraman et al., 1993a), they are outnumbered by studies which do (for an overview, see Wickens and Dixon, 2007).”

“The system had an overall reliability of at least 85% in all driving scenarios and was greater than the estimated 70% “crossover point” below which unreliable automation appears to be worse than no automation at all (Wickens & Dixon, 2007).”

“Furthermore, Wickens and Dixon (2005) demonstrated that when the reliability level is below approximately 70%, operators often ignore the alerts.”

“Quantitative syntheses of the literature suggest that the ‘crossover point’ from benefits to detriments, at least in high work-load conditions, is around 70% accuracy (Parasuraman & Manzey, 2010; Wickens & Dixon, 2007).”

“If increased transparency can increase the ease of cross checking the validity of recommendations by highlighting key information, then transparency should mitigate the impact of two key factors known to reduce the extent to which operators seek to validate recommendations: 1) high concurrent task demands and 2) high…”

“, 2005) and deteriorate the human operator’s global performance (Dixon et al., 2007; Wickens and Dixon, 2007).”

“Wickens and Dixon (2007) focused on modelling the role of reliability in human interaction with automated diagnostic aids.”

“During disruption, ARS cannot be relied upon to make a correct decision, due to limitations in the algorithms that cannot account for all circumstances or for changes in the timetable since the algorithmswere designed, and therefore this system represents that of an ‘imperfect’ automation system (Wickens and Dixon, 20…”

“The focus of research on the management of automation failure has been on improving system design (e.g. reliability levels, Wickens and Dixon 2007; message style of warnings, Lee, Gore, and Campbell 1999), whereas the issue of training has been given less attention, even though training is critical…”

“Wickens and Dixon (2007) performed a meta-analysis of studies examining automation reliability and human performance and found that human performance improved mono-tonically with system reliability and also that automation with a reliability level less than about 70% was no better than manual…”

“…become less dependent, but even as reliability drops below a threshold at around r = .75 (d′ = 1.35), humans may continue to depend on the imperfect diagnostic automation, even if their performance would be better if this automated advice were ignored (Maltz & Shinar, 2003; Wickens & Dixon, 2007).”

“Automation must therefore be extremely reliable if high levels of trust and usage are to be achieved; Wickens and Dixon (2007), in a meta-analysis, found that less than 70% reliability of automation is worse than useless.”

“For example, diagnostic automated aids associated with a task typically assist the operator by reducing cognitive load (Wickens & Dixon, 2007).”

“Even if the automation errs in occasional judgments, the human operator may be able to achieve a higher sensitivity with the aid’s assistance than without it (Wickens & Dixon, 2007).”

“…as levels of trust dynamically change throughout the course of observations about the automation’s behaviour, it is important that interaction decisions and consequent behaviours should reflect the extant level of trust under uncertainty (Wickens and Dixon 2007; Wang, Jamieson, and Hollands 2009).”

“…is important enough that as long as errors are predictable and the automation error rate is at a consistent rate of approximately 30% or less, most human users will decide to continue to use and benefit from the automation (Parasuraman et al., 2000; Wickens and Dixon, 2007; Wang et al., 2009).”

“57 Consequently, we also elected not to implement this PACU census management system.”

“Because of the effects of false alarms and misses on human behavior, system reliability is often discussed in the automation literature (Wickens & Dixon, 2007).”

“In their review of the literature, Wickens and Dixon (2007) proposed that an automation reliability of about 70% represented a critical inflection point; less than about 70% reliable was not relied upon while reliabilities higher than 70% led to complacency.”

“The reason could be simply the low power associated with analyzing data from single trials (Wickens & Dixon, 2005).”

“Nevertheless, even imperfect acquisition and analysis automation at a reliability as low as 70% , can improve performance as compared to unaided human performance [20].”

“Research on automation reliability on trust and human-automation team has indicated that a reliability of 80% ([25]) or above 70% ([26]) to be a suitable setting for similar studies.”

“Differences across conditions would have suggested variable levels of trust (consistent with Wickens and Dixon, 2007).”

“…reliability level was chosen because previous research has found this to be a critical “crossover” point for performance with imperfect automation (Wickens & Dixon, 2007) and because older adults appear to be sensitive to changes from 80% to 60% decision aid reliability (Fisk, Rogers, &…”

“…level in order to create a clear variation but still have a level that seems to be realistic for false-prone systems and that is still higher than the 70% threshold assumed to be the minimum reliability where automated sys - tems are still considered to be useful (Wickens & Dixon, 2007).”

“Specifically, Wickens and Dixon (2007) found that automation with reliabilities below 70% was, on average, worse than no automation at all.”

“Unreliable diagnostic automation has been shown to negatively affect attentional resources (Wickens, Dixon, Goh, & Hammer, 2005) and to degrade global human operator performance (Dixon, Wickens, & McCarley, 2007; Wickens & Dixon, 2007).”

“Additionally, a reliability percentage of approximately 85% has been commonly used in past studies (e.g., Avril et al., 2022; Wickens & Dixon, 2007).”

“It is intuitive that the number or complexity of the tasks may be too much for operators to handle safely ( Sheridan, 1987 ; Wickens & Dixon, 2007 ); therefore, automation may be used to augment or even to replace these tasks.”

“Workload refers to the load or demand imposed on the human operator and may be either physical or cognitive in nature (Wickens and Dixon 2007).”

“We manipulated task load because evidence from a review of 20 automation reliability studies suggested that dependence on imperfect automation would be stronger under higher task load (because the operator ’ s limited resources are expended; Wickens and Dixon 2007).”

“Single-operator systems still gain performance benefits from automated decision aids even when the information presented by the aid is imperfect or less than fully informative (de Visser & Para-suraman, 2011; Wickens & Dixon, 2007).”

“This is consistent with previous work showing that the performance of a human-automation team will be more efficient than a human completing the same task without automation as long as the reliability of the automation is greater than 70% [75].”

“Following this rationale, assistances with less than approximately 70% of reliability were found to be worse than no assistance (Wickens and Dixon, 2007).”

“Barriers to incorporating ITS (and digital tools, in general) are further compounded for classrooms in developing countries with respect to availability of hardware, infrastruc‐ ture, and appropriate instructional content [11].”

“It has been observed that automation must maintain at least a 70% success level to maintain user trust [43], and that imperfect automation can still be beneficial [44].”

“Interestingly, these findings have been modeled computationally (Farrell & Lewandowsky, 2000; Wickens & Dixon, 2007).”

“A relevant line of prior work has studied how different degrees of reliability of automation support affect human decision-making (Dunning et al., 2024; Rovira et al., 2007; Wickens & Dixon, 2007; de Visser & Parasuraman, 2011).”

“Studies on automation in decision-support have explored the consequences of providing imperfect information [11, 22].”

“…automation during performance tasks (e.g., Bainbridge, 1983; Dixon & Wickens, 2006; Dixon, Wickens, & Chang, 2005; Dixon, Wickens, & McCarley, 2007; Lee & Moray, 1994; Parasuraman & Riley, 1997; Parasuraman, Sheridan, & Wickens, 2000; Rice, in press; Rice et al., in press; Wickens & Dixon, 2007).”

“This is similar to other UAV studies (e.g., 86%, Wohleber et al., 2019; 90%, Chen and Barnes, 2012; 95%, Ruff et al., 2002) and was chosen as automation will always be, to some extent, imperfect but needs to be higher than 70% to still be utilized by operators (Wickens and Dixon, 2007).”

“These levels were chosen to be at least 70% (Wickens & Dixon, 2007) to ensure that automation is beneficial to the performance (Guideline 1).”

“Wickens and Dixon (2005) found that automation with less than approximately 70% reliability is considered worse than not using automation.”

“…error rates have found evidence of misuse rather than disuse, and that disuse is more likely to happen when working with systems with reliability lower than 70 % (Wickens & Dixon, 2007), it implies that the error rate might not have been a direct cause of disuse, or at least not the only one.”

“Dynamically adjusting the system mode, given an understanding of the changes in user’s beliefs over time, might also be crucial given their trust in the automation (Abe & Richardson, 2006; Bliss & Acton, 2003; Riener, Boll, & Kun, 2016; Wickens & Dixon, 2007).”

“90% and 60% were chosen because they are commonly used in Human Factors/HCI studies as the upper and lower bounds of imperfect autonomy (Wickens and Dixon, 2007).”

“Research has shown that autonomous agents with a reliability level below 70% produce such negative results that not implementing any autonomous agent is better [42].”

“%) is well below the 70% threshold traditionally assumed necessary to achieve human-automation team trust (Lee & See, 2004; Wickens & Dixon, 2007).”

“In general, systems above a threshold of .70 reliability tend to improve user performance (Wickens & Dixon, 2007).”