The Fallacy of AI Functionality

Classified as probably not a substantial claim.

The taxonomy identifies many points of failure, illustrating the scope of the problem.

“Firstly, the sheer number of points of failure we were able to identify illustrates the scope of the problem.”

• AssumptionRests on an assumption: The iterative tagging procedure reliably and validly categorizes each case into the failure modes of the taxonomy.

May combine two claims.

Audits of AI hiring tools focus primarily on ensuring an 80% selection rate for protected classes is satisfied, and rarely mention product validation processes, demonstrating an assumed validity of the prediction task.

“For example, audits of AI hiring tools focus primarily on ensuring an 80% selection rate for protected classes (the so-called 4/5ths rule) is satisfied, and rarely mention product validation processes, demonstrating an assumed validity of the prediction task [52,148,199].”

• cites [52]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [148]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [199]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Functionality is often presented as a consideration secondary to other ethical challenges, but it is a primary concern that often precedes such problems.

“As an overlooked aspect of AI policy, functionality is often presented as a consideration secondary to other ethical challenges.In this paper, we argue that it is a primary concern that often precedes such problems.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Scholars, the press, and policymakers pay too little attention to functionality.

“However, despite this reality, scholars, the press, and policymakers pay too little attention to functionality.”

Probably not a substantial claim: may not pass S4 (based).

A functionality assumption is prevalent in much of the discourse on AI risks.

“We start by calling out what we perceive to be a functionality assumption, prevalent in much of the discourse on AI risks.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Promoters of AI do not tend to question its functionality.

“It is unsurprising that promoters of AI do not tend to question its functionality.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Typically explanations are desired to check the basis for decisions, while assuming the systems work as intended.

“Similarly, to the extent regulators are looking into functionality, explanations aimed at regulators can assist oversight, but typically explanations are desired to check the basis for decisions, while assuming the systems work as intended.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Designers can preempt misuse by implementing instructions, warnings, or error conditions, and failure to do so creates a system that does not function properly.

“It is possible for designers to preempt misuse by implementing instructions, warnings, or error conditions, and failure to do so creates a system that does not function properly.”

May combine two claims.

The accuracy of ETS's method was disputed between experts, with resulting estimates of error rates ranging from 1% to 30%.

“The accuracy of ETS's method was disputed between experts sought by the National Union of Students and the Home Office; the resulting estimates of error rates ranged from 1% to 30%.”

May combine two claims.

ScaleFactor claimed to have AI that could replace accountants for small businesses, but dozens of accountants did most of the work manually.

“Under the hood, however, Instead of software producing financial statements, dozens of accountants did most of it manually from ScaleFactor's Austin headquarters or from an outsourcing office in the Philippines, according to former employees.”

• cites [100]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Due to upstream policy, data or societal choices, AI tasks can be practically impossible for one set of developers and not for another, or for different reasons in different contexts.

“Due to upstream policy, data or societal choices, AI tasks can be practically impossible for one set of developers and not for another, or for different reasons in different contexts.”

Probably not a substantial claim: may not pass S4 (based).

Existing accountability tools to address functionality issues are often overlooked in AI policy literature and practice, due in large part to the assumption of functionality.

“We then discuss the existing accountability tools to address functionality issues, that are often overlooked in AI policy literature and in practice, due in large part to this assumption of functionality.”

May combine two claims.

Expectations based on unreasonable claims can create unearned trust and potential overreliance that hurts parties who purchase the product.

“While this in and of itself may not create unsafe products, expectations based on unreasonable claims can create unearned trust, and a potential overreliance that hurts parties who purchase the product.”

Classified as probably not a substantial claim.

This fairness research question presumes that an unconstrained solution without fairness restrictions is the optimal solution to the problem.

“This research question presumes that an unconstrained solution without fairness restrictions is the optimal solution to the problem.”

Probably not a substantial claim: may not pass S2 (checkable).

Functionality can be difficult to define precisely.

“Functionality can be difficult to define precisely.”

Probably not a substantial claim: may not pass S3 (load-bearing).

AI makes claims to generality while modeling behaviour that is determined by highly constrained and context-specific data.

“Similarly, Raji et al. [149] argue that AI makes claims to generality while modeling behaviour that is determined by highly constrained and context-specific data.”

• cites [149]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Sometimes model failures are anticipated yet difficult to prevent; engineers can sometimes take steps to ensure these points of failure will not cause harm.

“Sometimes model failures are anticipated yet difficult to prevent; in this case, engineers can sometimes take steps to ensure these points of failure will not cause harm.”

Classified as probably not a substantial claim.

The AI label lends itself to inflated claims of functionality that the systems cannot meet.

“Other papers discuss how the AI label lends itself to inflated claims of functionality that the systems cannot meet.”

• cites [102]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [28]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Nagendran et al. found that of 81 non-randomized studies comparing diagnostic deep learning computer vision algorithms against expert clinicians, the median number of clinical experts was 4, full access to datasets and code were unavailable in over 90% of studies, overall risk of bias was high, and adherence to reporting standards was suboptimal.

“Of the 81 non-randomized studies, they found the median number of clinical experts compared to the AI was 4, full access to datasets and code were unavailable in over 90% of studies, the overall risk of bias was high, and adherence to reporting standards were suboptimal, and therefore poorly substantiate their claims.”

• cites [129]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The Epic sepsis prediction model had poor calibration to other hospital settings and discriminated against under-represented demographics when externally validated by Wong et al.

“Similarly, the Epic sepsis prediction model, a product actually implemented at hundreds of hospitals, was recently externally validated by Wong et al. [203], who found that the model had poor calibration to other hospital settings and discriminated against under-represented demographics.”

• cites [203]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

AI products are challenging to engineer correctly in practice.

“There is also some work demonstrating that AI products are challenging to engineer correctly in practice.”

• cites [194]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Out of 12,500 people who appealed their immigration decisions, only 3,600 won their cases, and only a fraction of these were won through actually disproving the allegations of cheating.

“Yet out of 12,500 people who appealed their immigration decisions, only 3,600 won their cases-and only a fraction of these were won through actually disproving the allegations of cheating.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The lack of accountability for false claims makes communication failures particularly pernicious and likely to occur as AI hype continues absent effective regulation.

“Communication failures often co-occur with other functional safety problems, and the lack of accountability for false claims -intentional or otherwise -makes these particularly pernicious and likely to occur as AI hype continues absent effective regulation.”

• AssumptionRests on an assumption: AI hype will continue absent effective regulation.

May combine two claims.

Feldman et al. created a method to remove disparate impact from a model while preserving rank, which only makes sense if the unconstrained system output is correct in the first place.

“Similarly, as an early example of technical fairness solutions, Feldman et al. [60] created a method to remove disparate impact from a model while preserving rank, which only makes sense if the unconstrained system output is correct in the first place.”

• cites [60]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Some ScaleFactor customers received books filled with errors and were forced to rehire accountants or clean up the mess themselves.

“Some customers say they received books filled with errors, and were forced to re-hire accountants, or clean up the mess themselves.[100]”

• cites [100]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Disinformation researchers embrace corporate talking points of a recommendation model that can meaningfully influence consumer behavior to the point of controlling purchases or voting activity, when in actuality these algorithms have little ability to do either.

“As an example, he points to disinformation researchers, who embrace corporate talking points of a recommendation model that can meaningfully influence consumer behavior to the point of controlling their purchases or voting activity-when in actuality, these algorithms have little ability to do either [22,75,88,95,162].”

• cites [22]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [75]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [88]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [95]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [162]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

These proposals often fall short of acknowledging functionality issues in AI deployments and the role of the lack of functional safety in contributing to harm.

“These proposals thus often fall short of acknowledging the functionality issues in AI deployments and the role of the lack of functional safety in contributing to the harm perpetuated by these systems.”

May combine two claims.

From the engineering side, interpretability is usually desired for debugging purposes, so it is focused on functionality.

“From the engineering side, interpretability is usually desired for debugging purposes [23], so it is focused on functionality.”

• cites [23]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The AI research field suffers from scientific validity and evaluation problems.

“Recent work details that the AI research field suffers from scientific validity and evaluation problems [48,79].”

• cites [48]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [79]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Roles in AI development and deployment are becoming more specialized.

“As with other areas of software development, roles in AI development and deployment are becoming more specialized.”

• cites [44]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Functionality is not always ignored outright in policy, but in many cases consideration lacks the emphasis of other concerns.

“These mentions suggest that functionality is not always ignored outright.Sometimes, it is considered in policy, but in many cases, that consideration lacks the emphasis of the other concerns presented.”

Classified as probably not a substantial claim.

Many of the cases in the taxonomy are direct refutations of specific instances of the functionality assumptions in Section 3.

“Many of these cases are direct refutations of the specific instances of the functionality assumptions in Section 3.”

• AssumptionRests on an assumption: The iterative tagging procedure reliably and validly categorizes each case into the failure modes of the taxonomy.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Functionality could also be seen as a statement that a product lives up to the vendor's performance claims, but this is incomplete because specifications chosen by the vendor could be insufficient to solve the problem at hand.

“Functionality could also be seen as a statement that a product lives up to the vendor's performance claims, but this, too, is incomplete; specifications chosen by the vendor could be insufficient to solve the problem at hand.”

Classified as probably not a substantial claim. May combine two claims.

In a classification model, choices such as which input and target variables to use, whether to prioritize accepting true positives or rejecting false negatives, and how to process the training data all factor into determining model outcomes.

“For instance, in a classification model, choices such as which input and target variables to use, whether to prioritize accepting true positives or rejecting false negatives, and how to process the training data all factor into determining model outcomes.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Communication failures often co-occur with other functional safety problems.

“Communication failures often co-occur with other functional safety problems, and the lack of accountability for false claims -intentional or otherwise -makes these particularly pernicious and likely to occur as AI hype continues absent effective regulation.”

May combine two claims.

Concerns around large language models tend to dismiss the practical limitations of what these models can achieve, neglecting more mundane hazards tied to premature deployment of a system that does not work.

“While these are credible threats, concerns around large language models tend to dismiss the practical limitations of what these models can achieve [18], neglecting to address more mundane hazards tied to the premature deployment of a system that does not work [55,189].”

• cites [18]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [55]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [189]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Data validation is a critical step in the construction of a ML system, and the team that built the Idaho benefit system chose to use a highly problematic dataset to train their model.

“Data validation is a critical step in the construction of a ML system, and the team that built the benefit system chose to use a highly problematic dataset to train their model.”

May combine two claims.

The fear of misspecified objectives, runaway feedback loops, and AI alignment presumes the existence of an industry that can get AI systems to execute on any clearly declared objectives, and that the main challenge is to choose and design an appropriate goal.

“While it is not controversial to claim that such concerns are far from being realized [13,42,146], this fear of misspecified objectives, runaway feedback loops, and AI alignment presumes the existence of an industry that can get AI systems to execute on any clearly declared objectives, and that the main challenge is to choose and design an appropriate goal.”

• cites [13]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [42]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [146]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The Nest Labs smoke and carbon monoxide detector was recalled because a wave gesture could unintentionally deactivate it, and the lack of a failsafe made the product dangerous.

“However, the lack of a failsafe to make sure the carbon monoxide detector could not be turned off accidentally made the product dangerous.”

• cites [200]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Functionality is a meaningful AI policy challenge and a necessary first step towards protecting affected communities from algorithmic harm.

“We argue that functionality is a meaningful AI policy challenge, operating as a necessary first step towards protecting affected communities from algorithmic harm.”

May combine two claims.

Large language models used to process queries of AI-powered web search are fragile to misspellings, trivial changes to format, and context changes, which can lead to unexpected results.

“For example, when large language models are used to process the queries of AI-powered web search [131], the models' fragility to misspellings [125,147], or trivial changes to format [19] and context [18] can lead to unexpected results.”

• cites [131]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [125]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [147]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [19]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The Idaho algorithm for Medicaid assistance limits had critical flaws, including corrupt historical data and throwing out two-thirds of records due to data entry errors.

“the data they used to come up with their formula for setting people's assistance limits was corrupt.”

• Also stated in: § Model Implementation Failures. · plain

  “they had to throw out two-thirds of the records they had before they came up with the formula because of data entry errors and data that didn't make sense.”

• cites [177]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some AI development roles focus on managing data, others on modeling, and others on engineering models for speed and scale.

“Some roles focus on managing the data that feeds into models, others specialize in modeling, and others optimally engineer models for speed and scale [44].”

• cites [44]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The draft AI Act in the EU includes requirements for pre-marketing controls to establish products' safety and performance, as well as quality management for high risk systems.

“The draft AI Act in the EU includes requirements for pre-marketing controls to establish products' safety and performance, as well as quality management for high risk systems [190].”

• cites [190]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Policymakers often define AI with respect to how human-like a system is, which could lead to deprioritizing issues more grounded in reality.

“Krafft et al. [110] found that policymakers often define AI with respect to how human-like a system is, and concluded that this could lead to deprioritizing issues more grounded in reality.”

• cites [110]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

US EHR data is often erroneous, miscoded, fragmented, and incomplete, creating a mismatch between available data and intended use.

“US EHR data is often erroneous, miscoded, fragmented, and incomplete [91,92], creating a mismatch between available data and intended use [74].”

• cites [91]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [92]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [74]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Concerns around large language models center on hyped narratives of the models' ability to generate hyper-realistic online content, which could theoretically be used by malicious actors to facilitate harmful misinformation campaigns.

“For example, expressed concerns around large language models centers on hyped narratives of the models' ability to generate hyper-realistic online content, which could theoretically be used by malicious actors to facilitate harmful misinformation campaigns [176,195].”

• cites [176]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [195]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

IBM's Watson in healthcare was sold as a panacea to health care's ills, though it may have been fully capable of performing specific helpful tasks.

“The hype surrounding IBM's Watson in healthcare represents another example where a product that may have been fully capable of performing specific helpful tasks was sold as a panacea to health care's ills.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Sales of goods typically come with a set of implied warranties.

“Sales of goods typically come with a set of implied warranties.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

As AI operates as a general purpose technology prevalent in many industries, there already exists a plethora of governance infrastructure to address functionality issues in related fields of application.

“Thankfully, as AI operates as a general purpose technology prevalent in many of these industries, there already exists a plethora of governance infrastructure to address this issue in related fields of application.”

• AssumptionRests on an assumption: AI operates as a general purpose technology prevalent in many industries.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

There are many organizational interventions that can be deployed to address functionality issues.

“In addition to legal levers, there are many organizational interventions that can be deployed to address the range of functionality issues discussed.”

Probably not a substantial claim: may not pass S3 (load-bearing).

If a person is injured by a defective product, they can sue the producer or seller in products liability.

“In general, if a person is injured by a defective product, they can sue the producer or seller in products liability.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Several other organizational factors can determine and assess the functional safety of a system.

“There are several other organizational factors that can determine and assess the functional safety of a system.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Existing functional safety standards from the automobile, aerospace, defense, and electronics industries can provide a template for approaching functionality challenges within the AI industry.

“Existing functional safety standards from the automobile (ISO 26262), aerospace (US RTCA DO-178C), defense (MIL-STD-882E) and electronics (IEEE IEC 61508 / IEC 61511) industries, amongst others, can provide a template on how to approach this challenge within the AI industry.”

Classified as probably not a substantial claim.

The AI field is now facing the same challenge of dealing with an influx of fraudulent or dysfunctional products.

“The AI field seems to now be facing this same challenge.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Products liability could especially easily apply to engineering failures where the error was foreseeable and an alternative, working version of the product should have been built.

“Products liability could especially easily apply to engineering failures, where the error was foreseeable and an alternative, working version of the product should have been built.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Fraud invariably involves a knowing or intentional misrepresentation that the victim relied on in good faith.

“Fraud has many specific meanings in law, but invariably it involves a knowing or intentional misrepresentation that the victim relied on in good faith.”

May combine two claims.

Defunct AI will breach both implied warranties of merchantability and fitness for particular purpose.

“Defunct AI will breach both these warranties.”

Probably not a substantial claim: may not pass S3 (load-bearing).

CEN and CENELEC were heavily involved in the creation of the EU's draft AI Act.

“CEN (European Committee for Standardisation) and CENELEC (European Committee for Electrotechnical Standardisation), two of three European Standardisation Organisations (ESOs) were heavily involved in the creation of the EU's draft AI Act [190].”

• cites [190]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable).

We cannot take for granted that AI products work.

“We cannot take for granted that AI products work.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The AI field could adopt similar internal audit techniques, especially given the nascent state of reflection and standardization in ML evaluation processes.

“The AI field could certainly adopt similar techniques for increasing the scrutiny of their systems, especially given the nascent state of reflection and standardization common in ML evaluation processes [118].”

• cites [118]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

A $1.4 billion contract with a coalition of high-tech companies led an Indiana governor to continue a welfare automation algorithm that resulted in a 54% increase in denials of welfare applications.

“Similarly, Eubanks [56] demonstrates how a $1.4 billion contract with a coalition of hightech companies led an Indiana governor to stubbornly continue a welfare automation algorithm that resulted in a 54% increase in the denials of welfare applications.”

• cites [56]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Choi analogizes software complexity to car crashes and crashworthiness, arguing that software can cause the type of injury products liability aims to protect from and doctrine should reflect that.

“Choi analogizes this issue to car crashes: car crashes are inevitable, but courts developed the idea of crashworthiness to ask about the car's contribution to the total harm, even if the initial injury was attributable to a product defect [36].”

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

A functionality failure in an AI system could be deemed a product defect.

“It would stand to reason that a functionality failure in an AI system could be deemed a product defect.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In extreme cases, the sale of defunct AI may constitute fraud.

“In extreme cases, the sale of defunct AI may constitute fraud.”

Probably not a substantial claim: may not pass S2 (checkable).

Where dysfunctional AI is being sold and its failures cause substantial harm to consumers, the FTC could step in.

“Thus, where dysfunctional AI is being sold and its failures causes substantial harm to consumers, the FTC could step in.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The National Highway Safety Administration offers similar oversight for automobiles specifically.

“The National Highway Safety Administration offers similar oversight for automobiles specifically.”

Probably not a substantial claim: may not pass S2 (checkable).

If a system is not functional in the sense described, a court should have no problem finding that it is unreasonably defective.

“If a system is not functional, in the sense we describe, a court should have no problem finding that it is unreasonably defective.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Faulty AI products are on the market today, making the problem particularly urgent.

“The fact that faulty AI products are on the market today makes this problem particularly urgent.”

May combine two claims.

All states have consumer protection statutes that bar deceptive practices, and many bar unfair practices as well, like the FTC Act.

“In addition to various federal agencies, all states have consumer protection statutes that bar deceptive practices and many bar unfair practices as well, like the FTC Act [33].”

• cites [33]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Warranty law protects purchasers of defunct AI and certain third parties who stand to benefit from the sale.

“Another area of law implicated by product failure is warranty law, which protects the purchasers of defunct AI and certain third parties who stand to benefit from the sale.”

Probably not a substantial claim: may not pass S3 (load-bearing).

State attorneys general often take active roles as enforcers of consumer protection laws.

“State attorneys general often take active roles as enforcers of those laws [38].”

• cites [38]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

For machine learning products in general, advertised performance on a simple and static data benchmark is much lower than performance on the often more complex and diverse data encountered in practice.

“This follows a pattern seen with machine learning products in general, where the advertised performance on a simple and static data benchmark, is much lower than the performance on the often more complex and diverse data encountered in practice.”

May combine two claims.

Software within a physical system would likely not face the product-definition hurdle, but non-embodied AI might.

“This would likely not be a problem for software that resides within a physical system, but for non-embodied AI, it might pose a hurdle.”

May combine two claims.

Cultural expectations for safety and engineering responsibility impact the quality of output from product development, and fostering a healthy safety culture can increase cooperation on other industry-wide and organizational measures.

“Similarly, cultural expectations for safety and engineering responsibility impact the quality of the output from the product development process -setting these expectations internally and fostering a healthy safety culture can increase cooperation on other industrywide and organizational measures [163].”

• cites [163]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Injuries not related to breach of contract are meant to be worked out in tort law.

“Injuries not related to the breach of contract are meant to be worked out in tort law, as described above.”

May combine two claims.

AI systems developed by public agencies can be adopted as symbols for progress or smokescreens for undesirable policy outcomes, and thus liable to inflated narratives of performance.

“This is particularly a problem in the context of algorithms developed by public agencies -where the AI systems can be adopted as symbols for progress, or smokescreens for undesirable policy outcomes, and thus liable to inflated narratives of performance.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The FTC does not have general rulemaking authority, so most of its actions to date have taken the form of public education and enforcement.

“The Commission does not have general rulemaking authority, so most of its actions to date have taken the form of public education and enforcement.”

May combine two claims.

Internal audits, ranging from documentation exercises to challenge datasets as benchmarks, raised the bar for deployment criteria and matured the product development pipeline.

“Taking the form of anything from documentation exercises to challenge datasets as benchmarks, these processes raised the bar for deployment criteria and matured the product development pipeline in the process [152].”

• cites [152]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The ACLU ran Rekognition with a database of mugshots against members of U.S. Congress using the default setting and found 28 members falsely matched, with people of color shown as a disproportionate share of these errors.

“Building on previous work [31], the ACLU ran Rekognition with a database of mugshots against members of U.S. Congress using the default setting and found 28 members falsely matched within the database, with people of color shown as a disproportionate share of these errors [175].”

• cites [31]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [175]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Product certification and standards development for AI products is a trickier intervention.

“A trickier intervention is the avenue of product certification and standards development for AI products.”

May combine two claims.

A product could be placed on the market claiming an ability it cannot achieve in theory or practice, or fail to be robust to unanticipated but foreseeable uses by consumers.

“As discussed above, a product could be placed on the market claiming the ability to do something it cannot achieve in theory or in practice, or it can fail to be robust to unanticipated but foreseeable uses by consumers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The FTC has authority to make rules regarding unfair or deceptive practices under the Magnuson-Moss Warranty Act.

“The Commission does, however, have authority to make rules regarding unfair or deceptive practices under the Magnuson-Moss Warranty Act.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In many cases, active advocacy of concerned citizens led to policy interventions that effectively changed the tide of these industries.

“In many cases, it required the active advocacy of concerned citizens to lead to the policy interventions that would effectively change the tide of these industries.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

False advertising laws are related and common.

“False advertising laws are related and also common.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Poorly vetted products permeate our lives, and while many accept potential harms as a tradeoff, the claims of the products' benefits go unchallenged.

“Poorly vetted products permeate our lives, and while many readily accept the potential for harms as a tradeoff, the claims of the products' benefits go unchallenged.”

• AssumptionRests on an assumption: Poorly vetted AI products permeate our lives.
• AssumptionRests on an assumption: The claims of AI products' benefits go unchallenged.

Probably not a substantial claim: may not pass S4 (based).

Although functionality issues are not completely ignored in AI policy, the lack of awareness of the range in which these issues arise leads to the problems being inadequately emphasized and poorly addressed by the full scope of accountability tools available.

“Although functionality issues are not completely ignored in AI policy, the lack of awareness of the range in which these issues arise leads to the problems being inadequately emphasized and poorly addressed by the full scope of accountability tools available.”

May combine two claims.

Products liability may provide an avenue for legal accountability for AI functionality failures, though there are potential hurdles.

“Another avenue for legal accountability may come from the tort of products liability, though there are some potential hurdles.”

May combine two claims.

The FTC created no new rules since 1980, but in July 2021 it voted to change internal agency policies to make rulemaking easier.

“Though it has created no new rules since 1980, in July 2021, the FTC voted to change internal agency policies to make it easier to do so [41].”

• cites [41]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Fraud is difficult to prove, and the paper's examples do not clearly indicate fraud, but it is theoretically possible if someone is selling snake oil.

“Fraud is difficult to prove, and our examples do not clearly indicate fraud, but it is theoretically possible if someone is selling snake oil.”

May combine two claims.

Assessment outcomes are mainly for internal use, often to set internal quality assurance standards for deployment and prompt further engineering reflection.

“As a result, assessment outcomes are mainly for internal use only, often just to set internal quality assurance standards for deployment and prompt further engineering reflection during the evaluation process.”

May combine two claims.

Due process requires decisions to not be arbitrary, and AI that doesn't work loses its claim to making decisions on a sound basis.

“Due process requires decisions to not be arbitrary, and AI that doesn't work loses its claim to making decisions on a sound basis [37].”

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Fraud can lead to criminal liability as well.

“Fraud can lead to criminal liability as well.”

May combine two claims.

MD Anderson cancelled IBM Watson contracts after $62 million and many years of effort with no results to show for it.

“At Memorial-Sloan Kettering, after $62 million dollars spent and may years of effort, MD Anderson famously cancelled IBM Watson contracts with no results to show for it [89].”

• cites [89]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Microsoft claimed its machine translation had equal accuracy to humans in Chinese-to-English translations, but outside researchers found that at the document level, with context from nearby sentences and/or compared to human experts, the model did not achieve equal accuracy to human translators.

“However, the study used to make this claim (still prominently displayed in press release materials) was quickly debunked by a series of outside researchers who found that at the document-level, when provided with context from nearby sentences, and/or compared to human experts, the machine translation model did not indeed achieve equal accuracy to human translators [114,185].”

• cites [186]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [114]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [185]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Functionality is a safety risk aligned with profit-oriented goals, and many model logging and evaluation operations tools are available for organizations to leverage in internal inspection, including tools for continuous monitoring of deployed systems.

“Also, as functionality is a safety risk aligned with profit-oriented goals, many model logging and evaluation operations tools are available for organizations to leverage in the internal inspection of their systems -including tools for more continuous monitoring of deployed systems [154,169].”

• cites [154]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [169]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: Functionality is a safety risk aligned with profit-oriented goals.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In tort law, a claim of fraud can lead to compensation necessary to rectify harms from the fraud, as well as punitive damages in egregious cases.

“In tort law, a claim of fraud can lead to compensation necessary to rectify any harms that come from the fraud, as well as punitive damages in egregious cases.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The implied warranty of fitness for particular purpose applies when a seller knows the buyer has a specific purpose in mind and the buyer relies on the seller's skill or judgment about the good's fitness, requiring the good to be fit for that purpose.

“The implied warranty of fitness for particular purpose applies when a seller knows that the buyer has a specific purpose in mind and the buyer is relying on the seller's skill or judgment about the good's fitness, stating that the good is fit for that purpose [188].”

• cites [188]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

These efforts resemble the Partnership on AI's attempt to establish norms on model documentation processes.

“These efforts resemble the Partnership on AI's attempt to establish norms on model documentation processes [153].”

• cites [153]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Collective industry-wide decision-making on critical issues can raise the bar for the entire industry and raise awareness of the importance of handling functionality challenges.

“Collective industry-wide decision-making on critical issues can raise the bar for the entire industry and raise awareness within the industry of the importance of handling functionality challenges.”

May combine two claims.

Based on the FTC's approach to data security, even post-deployment failures, if foreseeable and harmful, can be included among unfair practices, though they are partially attributable to external actors.

“Based on the FTC's approach to data security, in which the Commission has sued companies for failing to adequately secure consumer data in their possession against unknown thirdparty attackers [121], even post-deployment failures-if foreseeable and harmful-can be included among unfair practices, though they partially attributable to external actors.”

• cites [121]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The plaintiff need not have purchased or used the product; it is enough that they were injured by it and the product has a defect that rendered it unsafe.

“The plaintiff need not have purchased or used the product; it is enough that they were injured by it, and the product has a defect that rendered it unsafe.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Due to clear conflicts of interest, self-regulatory approaches are far from adequate oversight, and the presence of regulation incentivizes organizations to take these actions.

“Due to clear conflicts of interest, the self-regulatory approaches described are far from adequate oversight for these challenges, and the presence of regulation does a lot to incentivise organizations to take these actions in the first place.”

May combine two claims.

The transportation industry has a rich history of thorough accident reports and investigations through organizations such as the NTSB, which has assessed damage from known self-driving car crashes from Uber and Tesla.

“Similarly, the transportation industry has a rich history of thorough accident reports and investigations, through organizations such as the National Transportation and Safety Board (NTSB), who have already been responsible for assessing the damage from the few known cases of self-driving car crashes from Uber and Tesla [81].”

• cites [81]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Deceptive claims can be expressed explicitly or implied, such as an aspect of design suggesting a functionality the product lacks.

“Deceptive claims can be expressed explicitly-for example, representation in the sales materials that is inaccurate-or implied, such as an aspect of the design that suggests a functionality the product lacks [82,93].”

• cites [82]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [93]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Organizational interventions provide an immediate path forward in addressing functionality issues.

“However, they do provide an immediate path forward in addressing these issues.”

Classified as probably not a substantial claim. May combine two claims.

The concept of functional safety in engineering design literature encapsulates that a system can be deployed without working very well, and that such performance issues can cause harm worth preventing.

“In broader terms, the concept of functional safety in engineering design literature [163,174] well encapsulates the concerns articulated in this paper-namely that a system can be deployed without working very well, and that such performance issues can cause harm worth preventing.”

• cites [163]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [174]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

For software of nontrivial complexity, it is provably impossible to guarantee bug-free code.

“For software of nontrivial complexity, it is provably impossible to guarantee bug-free code.”

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Choi identifies the complexity of software as a more fundamental hurdle for products liability.

“Writing about software and tort law, Choi [36] identifies the complexity of software as a more fundamental type of hurdle.”

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

California's Unfair Competition Law allows individuals to sue for injunctive relief to halt conduct that violates other laws, even if individuals could not otherwise sue under that law.

“One particular state law worth noting is California's Unfair Competition Law, which allows individuals to sue for injunctive relief to halt conduct that violates other laws, even if individuals could not otherwise sue under that law [2].”

• cites [2]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The Consumer Finance Protection Bureau can regulate harms from products dealing with loans, banking, or other consumer finance issues.

“The Consumer Finance Protection Bureau can regulate harms from products dealing with loans, banking, or other consumer finance issues [4].”

• cites [4]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Formally, the FTC does not have the power to issue fines under its Section 5 authority, but it frequently enters into long-term consent decrees permitting continuing jurisdiction, monitoring, and fines for future violations.

“Formally, the FTC does not have the power to issue fines under its Section 5 authority, but the Commission frequently enters into long-term consent decrees with companies that it sues, permitting continuing jurisdiction, monitoring, and fines for future violations [3,40].”

• cites [3]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

IEEE and ISO regularly shape AI policy conversations, with IEEE attempting to develop a certification program.

“On the U.S. front, industry groups IEEE and ISO regularly shape conversations, with IEEE going so far as to attempt the development of a certification program [72,84].”

• cites [72]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [84]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The Federal Trade Commission is the federal consumer protection agency within the United States with the broadest subject matter jurisdiction.

“The Federal Trade Commission is the federal consumer protection agency within the United States with the broadest subject matter jurisdiction.”

May combine two claims.

Audit results are rarely communicated externally, and disclosure is not mandatory or incentivized.

“The challenge with such audits, however, is that the results are rarely communicated externally and disclosure is not mandatory, nor is it incentivized.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A lack of construct validity would easily subject a model's user to liability.

“A lack of construct validity would easily subject a model's user to liability.”

Classified as probably not a substantial claim. May combine two claims.

No amount of testing can guarantee bug-free software, making it difficult to determine how much testing is enough to be considered reasonable or non-negligent.

“But as no amount of testing can guarantee bug-free software, it will difficult to determine how much testing is enough to be considered reasonable or non-negligent [36,94].”

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [94]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.