The Adoption of Artificial Intelligence in Health Care and Social Services in Australia: Findings From a Methodologically Innovative National Survey of Values and Attitudes (the AVA-AI Study)

The quote only partly states this claim.

Reducing costs was consistently the least important dimension of AI.

“Although all ethical and social dimensions of AI presented were rated as important, accuracy was consistently the most important and reducing costs the least important.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Both weighted samples were sociodemographically well spread.

“Both weighted samples were sociodemographically well spread.”

May combine two claims.

The AVA-AI study comprises 2 sample components: one from the LIA survey with a responding sample size of 2448 and a web-based panel sample with a responding sample size of 2000.

“The AVA-AI study comprises 2 sample components: one obtained from the Life in Australia (LIA) survey [40] with a responding sample size of 2448 and a web-based panel sample with a responding sample size of 2000.”

• cites [40] Kaczmirek, L, 2019, CSRM Methods SeriesNot checked yet

  Not checked in the free run.

The quote only partly states this claim.

Speed was consistently lower in importance than other dimensions of AI.

“Speed was also consistently lower in importance.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The weighting led to a weighted sample of 1950 for the web-based panel and 2498 for the LIA sample.

“The weighting led to a weighted sample of 1950 for the web-based panel and 2498 for the LIA sample.”

The quote only partly states this claim.

Accuracy was consistently the most important dimension of AI.

“Although all ethical and social dimensions of AI presented were rated as important, accuracy was consistently the most important and reducing costs the least important.”

May combine two claims.

The design effect because of weighting was 1.83 for the combined sample, 1.99 for the LIA, and 1.61 for the web-based panel.

“For the combined sample, the design effect because of weighting was 1.83; for the LIA, it was 1.99; and, for the web-based panel, it was 1.61.”

May combine two claims.

Among the health care scenarios, the only statistically significant differences were in relation to speed (more important in triage) and reducing costs (more important in decision support).

“Taken together, these tables show that, among the health care scenarios, the only statistically significant differences were in relation to speed (more important in triage) and reducing costs (more important in decision support).”

May combine two claims.

A similar pattern was seen in the welfare scenarios-a significant proportion of respondents changed their response among targeted compliance checking, automated parent support without contestability, and nonexplainable job services, in all cases to a more negative response.

“A similar pattern was seen in the welfare scenarios—a significant proportion of respondents changed their response among targeted compliance checking, automated parent support without contestability, and nonexplainable job services, in all cases to a more negative response.”

May combine two claims.

Of those who changed, between 70.8% and 83% changed to a more negative response, and all of these changes were statistically significant.

“Of those who changed, between 70.8% and 83% changed to a more negative response, and all of these changes were statistically significant.”

May combine two claims.

All means for importance of attributes were <3, the midpoint of the scale; t tests against a mean of 3 were statistically significant with P<.001, indicating that more of the distribution of responses was in the extremely or very important categories.

“All means were <3, the midpoint of the scale; t tests against a mean of 3 were statistically significant with P<.001, indicating that more of the distribution of responses was in the extremely or very important categories.”

May combine two claims.

The least support overall was expressed for AI systems that led to physician deskilling (27% support and 48.3% opposition) and those that made diagnostic and treatment recommendations but were not explainable to physicians (29.1% support and 41.6% opposition).

“The least support overall was expressed for AI systems that led to physician deskilling (27% support and 48.3% opposition) and those that made diagnostic and treatment recommendations but were not explainable to physicians (29.1% support and 41.6% opposition).”

May combine two claims.

Once the middle category was excluded, for nonneutral respondents, there were majorities supporting data sharing and targeted compliance checking; a balance on automated parent support without contestability; and a majority opposed to nonexplainable hospital algorithms, nonexplainable job services, and especially deskilling physicians.

“Once the middle category was excluded, Table 6 shows that, for the nonneutral respondents, there were majorities supporting data sharing and targeted compliance checking; a balance on automated parent support without contestability; and a majority opposed to nonexplainable hospital algorithms, nonexplainable job services, and especially deskilling physicians.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Comparisons of the general support and support in specific scenarios and between the scenarios were also analyzed using differences in the means, with similar conclusions.

“Comparisons of the general support and support in specific scenarios and between the scenarios were also analyzed using differences in the means, with similar conclusions.”

May combine two claims.

In the social service scenarios, more statistically significant differences were found, with explanation and cost reduction being more important in automating unemployment benefits and human contact, speed, and personal tailoring being more important in receiving automated carer support advice.

“In the social service scenarios, more statistically significant differences were found, with explanation and cost reduction being more important in automating unemployment benefits and human contact, speed, and personal tailoring being more important in receiving automated carer support advice.”

May combine two claims.

Answers on nonexplainability and deskilling were significantly different, and most were more negative than those on data sharing; answers on deskilling were significantly different, and most were more negative than those on nonexplainability.

“Answers on nonexplainability and deskilling were significantly different, and most were more negative than those on data sharing; answers on deskilling were significantly different, and most were more negative than those on nonexplainability.”

May combine two claims.

A minority of AVA-AI study respondents began the survey with negative general views on AI, and more than 60% expressed support.

“As a minority of AVA-AI study respondents began the survey with negative general views on AI and >60% expressed support”

May combine two claims.

For both explainability scenarios and clinical deskilling, respondents expressed on-balance opposition at a statistically significant level.

“For both explainability scenarios and clinical deskilling, respondents expressed on-balance opposition at a statistically significant level.”

May combine two claims.

The next highest support was for automated systems to identify parents who required assistance to return to work with limited contestability (34.9%) and employment support recommendation systems that were nonexplainable to employment service workers (31.2%).

“The next highest support was for automated systems to identify parents who required assistance to return to work with limited contestability (34.9%) and employment support recommendation systems that were nonexplainable to employment service workers (31.2%).”

May combine two claims.

Despite 45.7% of people changing their responses between the explainability in health care and explainability in welfare questions, people changed their minds in both directions in approximately equal proportions.

“Despite 45.7% of people changing their responses between these 2 questions, people changed their minds in both directions in approximately equal proportions.”

May combine two claims.

Among those taking a positive or negative position, majority support for the development of AI in general was 63.1% when don't know responses were excluded and 81.8% when neutral and don't know responses were excluded, both statistically significantly different from 50%.

“Table 4 clearly demonstrates majority support among those taking a positive or negative position—63.1% when don’t know responses were excluded and 81.8% when neutral and don’t know responses were excluded, with P values indicating that both estimates were statistically significantly different from 50%.”

May combine two claims.

The on-balance support mean score for the development of AI in general was 2.35, statistically significant when tested against the midpoint of 3.00 (P<.001).

“The on-balance support mean score of 2.35 was statistically significant when tested against the midpoint of 3.00 (P<.001).”

May combine two claims.

For targeted compliance checking and noncontestable automated parent support, views were balanced.

“For targeted compliance checking and noncontestable automated parent support, views were balanced.”

May combine two claims.

The estimate for the opposed categories was 13.4% and for the neutral or don't know responses was 26.3%.

“Although the estimate for the support categories was 60.3%, it was only 13.4% for the opposed categories and 26.3% for the neutral or don’t know responses.”

May combine two claims.

When asked about specific scenarios for AI use, respondents were consistently more negative; the reduction in support between the general question and all 6 specific scenarios was statistically significant, and support in the specific scenarios dropped between 17 and 33 percentage points.

“When asked about specific scenarios for AI use, respondents were consistently more negative—the reduction in support between the general question and all 6 specific scenarios was statistically significant, and support expressed in the specific scenarios dropped between 17 and 33 percentage points.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A design effect of 1.83 implies a design factor of 1.35.

“A design effect of 1.83 implies a design factor of 1.35.”

May combine two claims.

Both the AVA-AI study and the Monash survey suggest more positive general views on AI in Australia than in the United States.

“Both the AVA-AI study and the Monash survey suggest more positive general views in Australia than in the United States, although the results of the AVA-AI study are less positive than those of the Monash survey.”

• cites [26] sourceNot checked yet

  Not checked in the free run.

• cites [27] SelwynNCordobaBAndrejevicMCampbellLAI for social good? Australian public attitudes toward AI and societyMonash University202082022-07-28https://www.monash.edu/data-futures-institute/news/ai-for-social-good-australian-public-attitudes-toward-ai-and-societyNot checked yet

  Not checked in the free run.

May combine two claims.

For all questions, estimates of percentages had margins of error (twice the SE) of <3 percentage points, reflecting the relatively large sample size and the reliability of all estimates.

“However, for all questions, estimates of percentages had margins of error (ie, twice the SE) of <3 percentage points, reflecting the relatively large sample size and the reliability of all estimates.”

May combine two claims.

In all 4 scenarios, accuracy was rated as most important on average (1.49-1.61), and the ability of an AI system to reduce system costs was rated as least important (2.30-2.60), especially in health care.

“In all 4 scenarios, accuracy was rated as most important on average (1.49-1.61), and the ability of an AI system to reduce system costs was rated as least important (2.30-2.60), especially in health care.”

May combine two claims.

After accuracy, fairness was the second most important attribute in both social service scenarios (1.80 and 1.81) but, in the health care scenarios, it placed lower relative to other attributes (1.87 and 1.94).

“After accuracy, fairness was the second most important attribute in both social service scenarios (1.80 and 1.81) but, in the health care scenarios, it placed lower relative to other attributes (1.87 and 1.94).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The CIs for questions B01 and C01 tended to be narrower as they were based on the combined sample.

“The CIs for questions B01 and C01 tended to be narrower as they were based on the combined sample.”

May combine two claims.

Respondents made different judgments in specific scenarios-there were statistically significant changes within all pairs except between the questions regarding explainability in health care and in welfare.

“As shown in Table 9, respondents did make different judgments in specific scenarios—there were statistically significant changes within all pairs except between the questions regarding explainability in health care and in welfare.”

May combine two claims.

The effect of weighting on the web-based panel sample was similar to that in the overall sample, although there was very little effect for age and capital cities.

“The effect of weighting was similar to that in the overall sample, although there was very little effect for age and capital cities.”

May combine two claims.

With the neutral score included, the level of support for specific scenarios never reached a majority and ranged from 27% (deskilling physicians) to 42.3% (data sharing for quality care).

“With the neutral score included, the level of support never reached a majority and ranged from 27% (deskilling physicians) to 42.3% (data sharing for quality care).”

May combine two claims.

Respondents had slightly stronger, more diverse, and more negative views on using AI in health care as opposed to in social services.

“Respondents had slightly stronger, more diverse, and more negative views on using AI in health care as opposed to in social services.”

May combine two claims.

Human contact, personal tailoring, and speed were more important for the chatbot than for the benefits system.

“Human contact, personal tailoring, and speed were more important for the chatbot than for the benefits system”

May combine two claims.

Accuracy was always the most highly valued attribute, and reducing costs was always the least highly valued, across health care and social services.

“Two things were consistent: accuracy was always the most highly valued, and reducing costs was always the least highly valued across health care and social services.”

• Also stated in: § Conclusions · plain

  “Respondents consistently rated the accuracy of performance as the most important attribute in an AI system”

May combine two claims.

When asked to make an on-balance judgment about the bundle of attributes most commonly associated with machines versus with humans, respondents strongly preferred human attributes; the estimated proportion preferring machine attributes was 20.3% and human attributes 52%.

“When asked to make an on-balance judgment about the bundle of attributes most commonly associated with machines versus with humans, respondents strongly preferred human attributes.”

• Also stated in: § Final Bundled Attribute Trade-off of AI and Human Attributes · hedged

  “The estimated proportion of those who preferred the machine attributes (categories 1 or 2) was 20.3%, whereas, for human attributes (categories 4 or 5), it was 52%; 27.7% selected a middle position.”

• Also stated in: § Conclusions · plain

  “but only 1 in 5 valued the speed, accuracy, and convenience of AI systems more than continued human contact and discretion in service provision”

May combine two claims.

Respondents in the AVA-AI study were less strongly supportive than respondents in the Monash survey, expressing 27% to 43% support for health care scenarios and 31% to 39% support for social service scenarios.

“Again, respondents in the AVA-AI study were less strongly supportive than respondents in the Monash survey, expressing 27% to 43% support for health care scenarios and 31% to 39% support for social service scenarios.”

• cites [27] SelwynNCordobaBAndrejevicMCampbellLAI for social good? Australian public attitudes toward AI and societyMonash University202082022-07-28https://www.monash.edu/data-futures-institute/news/ai-for-social-good-australian-public-attitudes-toward-ai-and-societyNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Weights for the LIA sample were calculated using generalized regression estimation to adjust for differences in selection probabilities and nonresponse and calibrate to population benchmarks.

“Weights for the LIA sample were calculated using standard methods for a probability sample using generalized regression estimation [41] to adjust for differences in selection probabilities and nonresponse and calibrate to population benchmarks obtained from the population census, current demographic statistics, and the 2017 to 2018 National Health Survey obtained from the Australian Bureau of Stat”

• cites [41] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

The abstract principles and frameworks that have proliferated in AI ethics cannot be sufficient to address ethical issues in practice.

“The abstract principles and frameworks that have proliferated in AI ethics offer accessible ways in to ethical debates, but they cannot be sufficient to address ethical issues in practice [14].”

• cites [14] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

A web-based panel allowed us to generate a relatively large sample, enabling a good level of disaggregation into subpopulations, comparisons between groups, and analysis of associations.

“A web-based panel allowed us to generate a relatively large sample, enabling a good level of disaggregation into subpopulations, comparisons between groups, and analysis of associations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Although the use of weights can reduce bias, there is an associated increase in variances and SEs of the estimates.

“Although the use of weights can reduce bias, there is an associated increase in variances and SEs of the estimates.”

Classified as probably not a substantial claim.

The web-based panel sample was calibrated to the omnibus survey sample using behavioral, lifestyle, and sociodemographic variables.

“The web-based panel sample was calibrated to the omnibus survey sample using behavioral, lifestyle, and sociodemographic variables.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This approach was similar to that used in the study by Zhang and Dafoe, although our approach went further by adjusting for behavioral and lifestyle variables and 2 substantive variables.

“This approach was similar to that used in the study by Zhang and Dafoe [26], although our approach for the AVA-AI study went further by adjusting for behavioral and lifestyle variables and 2 substantive variables.”

• cites [26] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

There are now calls to complement ethical frameworks with other forms of knowledge, including analysis of detailed use cases and investigation of what members of the public think and value regarding the use of AI.

“There are now calls to complement ethical frameworks with other forms of knowledge, including analysis of detailed use cases and investigation of what members of the public think and value regarding the use of AI [15].”

• cites [15] Whittlestone, J, 2019, Ethical and Societal Implications of Algorithms, Data, and Artificial Intelligence: A Roadmap for ResearchNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The instrument underwent multiple rounds of input from investigators and expert colleagues, as well as cognitive testing before the study commenced.

“Before the study commenced, the instrument underwent multiple rounds of input from investigators and expert colleagues, as well as cognitive testing.”

May combine two claims.

Although expressing broad support for AI, people made finely tuned judgments about the acceptability of particular AI applications with different potential benefits and downsides.

“Although expressing broad support for AI, people made finely tuned judgments about the acceptability of particular AI applications with different potential benefits and downsides.”

Classified as probably not a substantial claim.

To enhance adjustment of the web-based panel data, we included 2 substantive questions, a set of behavioral and lifestyle questions, and major sociodemographic variables in both surveys.

“To enhance our adjustment of the web-based panel data in the AVA-AI study, we included 2 substantive questions, a set of behavioral and lifestyle questions, and major sociodemographic variables in both the web-based panel survey and the probability sample–based LIA survey, as indicated in Table 1.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All estimates and analyses were based on a weighted analysis using the largest sample possible.

“All estimates and analyses were based on a weighted analysis using the largest sample possible.”

Classified as probably not a substantial claim.

The web-based panel sample was a nonprobability sample.

“The LIA sample was selected using scientific probability sampling methods, whereas the web-based panel sample was a nonprobability sample.”

Classified as probably not a substantial claim.

Weights must be accounted for in the calculation of estimates and in the statistical inference.

“Weights must be accounted for in the calculation of estimates and in the statistical inference, such as estimates of SEs and the associated CIs obtained from them and P values for any statistical tests used.”

Classified as probably not a substantial claim. May combine two claims.

The methods used accounted for the use of weights in calculating estimates and associated 95% CIs and allowed for testing statistical significance.

“The methods used accounted for the use of weights in calculating estimates and associated 95% CIs and allowed for the testing of statistical significance, assessed when the P value of the relevant statistical test was <.05.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Theoretical and conceptual research in AI ethics needs to be complemented with empirical research to understand the values and judgments of members of the public.

“Theoretical and conceptual research in AI ethics needs to be complemented with empirical research to understand the values and judgments of members of the public, who will be the ultimate recipients of AI-enabled services.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In a final calibration step, the weights were further adjusted to agree with the population benchmarks for these variables.

“In a final calibration step, the weights were further adjusted to agree with the population benchmarks for these variables.”

May combine two claims.

Web-based panels may enable the examination of associations and, with adjustments to reduce biases, improve the estimation of population characteristics.

“but may enable the examination of associations and, with adjustments to reduce biases, improve the estimation of population characteristics [43].”

• cites [43] DiSograCCobbCChanEDennisJCalibrating non-probability internet samples with probability samples using early adopter characteristicsProceedings of the American Statistical Association, Section on Survey Research Joint Statistical Meetings (JSM)2011American Statistical Association, Section on Survey Research Joint Statistical Meetings (JSM)Jul 30 - Aug 4, 2011Miami Beach, FloridaNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

This study was approved by the University of Wollongong Social Sciences Human Research Ethics Committee (protocol number 2019/458).

“This study was approved by the University of Wollongong Social Sciences Human Research Ethics Committee (protocol number 2019/458).”

Classified as probably not a substantial claim.

The AVA-AI study has created one of the first large, robust data sets reflecting public views on the potential use of AI in health care and social services, with particular attention to the ELSI of those technologies.

“The AVA-AI study has created one of the first large, robust data sets reflecting public views on the potential use of AI in health care and social services, with particular attention to the ELSI of those technologies.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The design effects are consistent with the design effect that was due to a weighting of 1.83.

“The design effects are consistent with the design effect that was due to a weighting of 1.83.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For each question in the remaining analyses, the very small proportion of refused and don't know responses were not included and were no more than 8 cases for any of these questions.

“For each question in the remaining analyses, the very small proportion of refused and don’t know responses were not included and were no more than 8 cases for any of these questions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The relatively low level of knowledge of AI in the community and the speculative nature of the questions mean that people's responses to a direct experience of AI may differ from their responses in this survey.

“Although the results suggest that respondents were able to engage with the details of the questions, the relatively low level of knowledge of AI in the community and the speculative nature of the questions mean that people’s responses to a direct experience of AI may differ from their responses in this survey.”

• AssumptionRests on an assumption: The Australian community has a relatively low level of knowledge about AI.

Classified as probably not a substantial claim.

The findings suggest the importance of pursuing an augmentation rather than a replacement role for AI in both health care and social services.

“This suggests the importance of pursuing an augmentation rather than a replacement role for AI in both health care and social services.”

May combine two claims.

The differences in responses between the 2 surveys may show that the ethical and social risks of AI matter to people and will make a difference in their evaluations.

“The differences in responses between the 2 surveys may show that the ethical and social risks of AI matter to people and will make a difference in their evaluations.”

Classified as probably not a substantial claim.

The methodological approach allowed for the collection of detailed information on attitudes for a substantial sample using a relatively low-cost web-based panel while compensating for the potential biases in the creation of such a panel.

“The methodological approach taken allowed for the collection of detailed information on attitudes for a substantial sample using a relatively low-cost web-based panel while compensating for the potential biases in the creation of such a panel.”

May combine two claims.

Respondents highly valued human support and connection and were not prepared to give them up in exchange for accuracy.

“Although they considered attributes such as accuracy to be important if an AI system was to be implemented, they still highly valued human support and connection and were not prepared to give them up in exchange for accuracy (despite the accuracy of AI being highly valued in itself).”

May combine two claims.

Australians want AI systems to augment rather than replace humans in the provision of both health care and social services and to reflect human values.

“Overall, this study suggests that the ethical and social dimensions of AI systems matter to Australians and that Australians want AI systems to augment rather than replace humans in the provision of both health care and social services and to reflect human values.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The lower importance given to cost reduction may reflect a general rejection of instrumental decision-making in policy and of cost-based arguments in public services.

“The lower importance given to cost reduction may reflect a general rejection of instrumental decision-making in policy and of cost-based arguments in public services.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A strength of the design was the use of questions deliberately structured to present both the potential benefits and the potential burdens or harms of AI while attempting to maintain neutral sentiment and avoid normative valence.

“A strength of our design was the use of questions that were deliberately structured to present both the potential benefits and the potential burdens or harms of AI while attempting to maintain neutral sentiment and avoid normative valence in the language used.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This study is one of the largest and most robust surveys of public attitudes toward health care and welfare AI to date.

“To the best of the authors’ knowledge, this study is one of the largest and most robust surveys of public attitudes toward health care and welfare AI to date.”

May combine two claims.

Australians support the idea of AI in a general sense, but their support diminishes when considering the details of particular scenarios and the potential harms or burdens that may accompany any promised benefits.

“Australians support the idea of AI in a general sense, but their support diminishes when considering the details of particular scenarios and the potential harms or burdens that may accompany any promised benefits.”