Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance

In the paper

“Even when connected to representative organizations, such individuals or communities often lack the vocabulary and ability to flag the issues they observe [34].”

What the sources say

• Disputed

  Section 6.5 (page not provided) says that a specialist machine-learning background was not necessary for meaningful feedback and gives examples of community panelists identifying concerns about predictive policing. The paper acknowledges that technical concepts are difficult to explain, but its finding runs contrary to the attributed claim that these communities lack the ability to flag issues they observe. (LLM, conf=0.86)

  Checked by a reasoning model — may be wrong.

  cites [34] sourceExperimental — accuracy not yet measured

In the paper

“Even in cases when a participatory approach is considered, the responsibility of consulting with external perspectives when considering impacts to articulate in an AIA is still framed as the responsibility of internal stakeholders to coordinate and interpret these interactions [53].”

What the sources say

• Partly supported

  Section 2.3 (page unavailable) says participatory design brings relevant stakeholders into development and that gathering their views and resolving conflicting specifications create work and costs. It does not discuss AI impact assessments (AIAs) or say that internal stakeholders are responsible for coordinating and interpreting external consultations in an AIA. (LLM, conf=0.82)

  cites [53] sourceExperimental — accuracy not yet measured

In the paper

“Together, these watchdogs continue to participate in consequential studies to analyze the biased outcomes of models used in domains as varied as content curation [80], online advertising [40], tenant screening [39,70], child welfare [12], criminal justice [5,10,20,26,78], hiring [63], immigration [24], education [25] or public health [15,56].”

What the sources say

• ?Could not verify

  The Conclusion (page unavailable) describes a drug-safety study that developed and evaluated models for predicting adverse drug interactions. The Discussion mentions possible confounding and indication biases, but neither passage describes a third-party watchdog study of biased model outcomes. I searched for watchdogs, biased outcomes, and model bias in public health. (LLM, conf=0.73)

  cites [80] sourceExperimental — accuracy not yet measured
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [40]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [39]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [70]

In the paper

“The evidence on mandatory rotations is more mixed.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [27]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [22]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [32]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [38]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [43]
• ?Could not verify

  Section 5, “Limitations of Internal Audits” (page unavailable), discusses auditor independence and the limits of internal audits, but not mandatory rotations or evidence about their effects. Searches for “mandatory rotations” and “auditor rotation mixed evidence” found no passage addressing the claim. (LLM, conf=0.73)

  cites [65] sourceExperimental — accuracy not yet measured

In the paper

“A two-year randomized controlled trial showed that environmental third party audits were more accurate when auditors were not paid by the firm being audited, but by a common pool of government distributed funding [18].”

What the sources say

• ?Could not verify

  The 'Some key problems' section discusses conflicts of interest in corporate-funded AI ethics research and independent AI audits, but not environmental audits. Searches for the two-year trial and the comparison between firm-paid and government-funded auditors found no passage addressing the claimed accuracy result. (LLM, conf=0.75)

  cites [18] sourceExperimental — accuracy not yet measured

In the paper

“The longer an auditor gets to know a firm, the more firm-specific expertise the auditor acquires.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [22]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [32]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [38]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [43]
• ?Could not verify

  Sections 2.4 (The Internal Audit) and 3.3.1 (Audit Infrastructure) discuss auditors’ access to organizational information, but neither addresses whether longer tenure increases firm-specific expertise. Searches for auditor tenure and for auditors gaining knowledge of a firm over time found no passage making that point. Page numbers were not available in our copy. (LLM, conf=0.76)

  cites [65] sourceExperimental — accuracy not yet measured

In the paper

“The tenure of auditors is another key determinant of independence [15].”

What the sources say

• ?Could not verify

  Section 4 describes an audit of demographic disparities in mobility-data coverage, and §4.5 applies it to voter turnout. Searches for auditor tenure, appointment length, and independence found no passage addressing whether tenure determines auditor independence. The source therefore does not verify the cited claim. (LLM, conf=0.78)

  cites [15] sourceExperimental — accuracy not yet measured

In the paper

“There is strong social science demonstrating that selection and compensation can undermine independence and, in turn, audit quality.”

What the sources say

• Partly supported

  In “Independence and direction of research” (page unavailable), the paper says funding requirements can affect research independence. “Some key problems” (page unavailable) describes conflicts of interest in corporate-funded AI ethics work and calls for independent audits. It does not present social-science evidence that auditor selection or compensation arrangements undermine audit independence and, in turn, audit quality. (LLM, conf=0.78)

  cites [18] sourceExperimental — accuracy not yet measured

In the paper

“However, even in instances when the algorithmic systems are themselves disclosed, it remains difficult to anticipate which of these deployments are linked to known incidents of harms, and thus challenging to prioritize cases or pursue targets proactively [64].”

What the sources say

• ?Could not verify

  Section 3.1.1 discusses choosing the scope of an audit, and Section 3.2.3 discusses disclosure of audit targets and its effects on auditability. Neither addresses whether disclosed system deployments can be linked to known incidents of harm or whether that difficulty prevents proactive prioritization. Searches for those connections found no supporting passage in our copy. (LLM, conf=0.70)

  cites [64] sourceExperimental — accuracy not yet measured

In the paper

“Similarly, when car owners are given the option to choose their own car inspectors, they select for more lenient car inspectors, resulting in a race to the bottom [10,34].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [10]
• ?Could not verify

  The Background (section 3) discusses oversight of surveillance technologies, and section 6.3 discusses algorithmic accountability. Neither addresses car owners choosing inspectors or a resulting race to the bottom. Searches for car-inspector choice, inspector leniency, and “race to the bottom” found no relevant passage in our copy. (LLM, conf=0.76)

  cites [34] sourceExperimental — accuracy not yet measured

In the paper

“Investigative journalism tools such as Algorithm Tips [83], a database of algorithmic tools being used by public agencies, can also make information on algorithmic deployment more publicly accessible.”

What the sources say

• Partly supported

  The ‘Accountability to decision subjects’ section (p. 6) describes agencies sharing information about their algorithms with journalists and publishing model details, which supports the broader point about public access to information. Our copy does not identify Algorithm Tips or describe a database of public-agency algorithms, so it does not support that specific example. (LLM, conf=0.78)

  cites [83] sourceExperimental — accuracy not yet measured

In the paper

“For instance, it was recurring audits that revealed that Facebook was continuing to discriminate against minority users in job and housing ads,despite earlier communications that it would no longer do so [4,38,41,52,72].”

What the sources say

• Partly supported

  The Concluding Discussion reports that the authors' employment and housing ad tests found racially skewed delivery, and Anti-discrimination rules describes Facebook's earlier pledge to restrict certain targeting categories. The paper does not establish that recurring audits found continued discrimination against minority users generally, or that Facebook had promised to end all such discrimination; it limits its findings to the ads tested. (LLM, conf=0.85)

  cites [4] sourceExperimental — accuracy not yet measured
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [38]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [41]
• ?Could not verify

  Section 2.3 discusses third-party audits, while Section 3 discusses Facebook’s human-rights impact assessment concerning Myanmar. Neither addresses recurring audits of Facebook’s job or housing ads, discrimination against minority users in those ads, or earlier assurances that it would stop. Searches for those terms found no passage supporting the attributed claim. (LLM, conf=0.76)

  cites [52] sourceExperimental — accuracy not yet measured
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [72]

In the paper

“Second, we survey audit systems across domains -e.g., financial, environmental, and health regulation -and show that the institutional design of such audits are far from monolithic.”

In the paper

“Third, all audit systems have some mechanism of access to otherwise confidential information: e.g., access to enter a facility to inspect the premises and records or the reporting or access to financial statements.”

In the paper

“Table 1 shows that a fair number of audit schemes are universal (i.e., they are mandatory for all entities subject to the jurisdiction of the regulator).”

In the paper

“While helpful in aligning internal stakeholders, such an approach is not likely to identify or address the main concerns of external stakeholders [54,66], particularly if that affected population or its representatives are systematically neglected or under-considered as affected non-users.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [54]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [66]

In the paper

“These instructions around audit execution are ultimately attempts at control on audit quality and meaningfulness for accountability, but also reveal the assumption underpinning many of these proposals that internal parties are expected to be the ones conducting compliance audits.”

In the paper

“Developments such as the General Data Protection Regulation (GDPR) [86], the proposed Algorithmic Accountability Act [48], and several of the state-wide and municipal proposals throughout the U.S. [50] tend to focus instead on internal audits commissioned, executed, paid for and controlled by the companies being targeted.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [86]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [48]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [50]

In the paper

“Worse yet -as was observed in the AI hiring tool industry [76,77] -even well-executed algorithmic audit results may get co-opted by audit targets and used to disguise deeper failures.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [76]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [77]

In the paper

“But existing algorithmic accountability policy approaches have neglected the lessons from non-algorithmic domains: notably, the importance of interventions that allow for the effective participation of third parties.”

In the paper

“While the sharp rise in ESG investment strategies is indeed impressive, ESG in many ways challenges, not supports, the turn toward the audits of algorithmic and artificial intelligence (AI) branded products.”

In the paper

“Similarly, the Information Commissioner's Office released a report of”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [32]

In the paper

“Second, the emphasis on internal audits can omit concerns by communities impacted by algorithmic deployments.”

In the paper

“Thirdly, there is acute potential for conflicts of interest with first or second party audits.”

In the paper

“First, many AI policy developments tend to focus on articulating some guidance on what an internal audit should include and potentially enforcing these as expectations for a bare minimum”

In the paper

“Specifically, the assumption behind calls for Algorithmic Impact Assessments (AIAs) remains that audit activities are executed by internal stakeholders.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [67]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [73]

In the paper

“For instance, the New York City Council mandate for hiring tools specifies the details of a”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [50]

In the paper

“Most mainstream interpretations of the AIA continue to center its execution as the responsibility of the assessed organization [67,73].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [67]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [73]

In the paper

“Internal accountability approaches are developed in reference to a limited set of compliance requirements and self-determined AI principles [66].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [66]

In the paper

“For instance, companies developing algorithmic policing tools naturally prioritize the needs of the police users of their products, even though the community whose safety is impacted by the use of such tools may have different expectations and preferences [51].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [51]

In the paper

“Residents in low-income and minority communities, for instance, are less likely to complain about street conditions or”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [48]

In the paper

“Fourth, even when external auditing is contemplated, the focus is on a narrow set of potential auditors.”

In the paper

“Similarly, the warnings from Facebook researchers on addressing mounting bias issues and misinformation campaigns was reported to have been internally suppressed [57].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [57]

In the paper

“Even the work of hired consultants can be manipulated when the publication process is mediated by the company itself, as demonstrated by the conflicted reporting on the Hirevue audit by consulting company ORCAA [21].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [21]

In the paper

“Complaint-initiated audits, however, also raise challenges.”

In the paper

“In some cases, users were purposefully inundated with notices designed to induce them to consent by default [55,57,61,70].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [55]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [57]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [61]

In the paper

“These types of informational disclosures can also distract regulators from more pressing problems.”

In the paper

“Last, the limited audit capacity has led to many instances of audit failures.”

In the paper

“These third-party reporting systems can help focus auditing resources on issues that matter most [81].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [81]

In the paper

“Another research study found that the Vaccine Adverse Event Reporting System (VAERS) database is predictive of true safety concerns with newly released vaccines [26].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [26]

In the paper

“Third-party reporting is most successful when third parties possess high quality information.”

In the paper

“If auditors are tasked with vague inspection mandates, their results may be difficult to understand or translate into enforcement actions or changed practices.”

In the paper

“When an individual's income is reported to the IRS by third parties like banks and employers, for example, that individual is far more likely to be tax compliant than if they self-reported their income [41,47].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [41]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [47]

In the paper

“As a result of this conflict of interest, the ability of internal stakeholders or hired parties to properly scrutinize, honestly describe and effectively alter the algorithmic development process or its outcomes is limited when the incentives to do so directly contradict corporate preferences.”

In the paper

“These notices, however, can easily become overwhelming.”

In the paper

“One analysis of investigated nursing home complaints, for example, found that the complaints predicted performance at subsequent inspections [71].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [71]

In the paper

“Similar to the”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [72]

In the paper

“Similarly, the U.S. Food and Drug Administration has faced challenges in assessing AI-enabled medical devices, largely deferring to company benchmarks on limited, retrospective data [89].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [89]

In the paper

“Eccles et al. (2012) found that, despite the SEC's interpretive guidance, there is significant variation between companies and within industries in climate change reporting, with some companies disclosing no information about climate change, others using boilerplate language in their disclosures, and still others including robust quantitative metrics like GHG emissions and energy use [19].”

What the sources say

• ?Not checked — no full text of the sourcecites [19] source

In the paper

“Still, consumer complaints have proven to be a valuable signal when harms can be difficult to observe.”

In the paper

“One study found that vehicle owners were significantly less likely to keep using car emissions testing stations that fail to report favorably on their vehicles [10,39,63].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [10]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [39]
• ?Not checked — no full text of the sourcecites [63] source

In the paper

“Third parties have already taken advantage of a range of interventions to identify and prioritize potential harms.”

In the paper

“Similar dynamics have been found in credit rating agencies [20,40], supply chain audits [69], and accounting audits [58].”

What the sources say

• ?Not checked — no full text of the sourcecites [20] source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [40]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [69]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [58]

In the paper

“Another study found that credit ratings teams were less accurate in their evaluations when they interacted directly with clients [29].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [29]

In the paper

“Research suggests that auditors who may try to sell additional non-auditing services to their clients are more likely to produce client-favorable audits.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [13]

In the paper

“Current strategies for harm discovery in AI involve ad hoc processes of notice and reporting, limited to situations of harm raised and escalated in public forums such as social media or the press.”

In the paper

“At times, public procurement process disclosure requirements such as the Public Oversight of Surveillance Technology (POST) Act [69] will require that certain (often public) institutions such as the New York Police Department (NYPD) publicly release information on which AI technologies are being used.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [69]

In the paper

“Causholli et al. (2014) found that financial auditors whose clients express willingness to purchase their non-audit services in the future tend to produce lower-quality audits [13].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [13]

In the paper

“As a result, it can be difficult for regulators and the public to identify incidents of algorithmic harm and the public disclosure of algorithmic products in use is necessary to reveal such issues.”

In the paper

“One analysis of nearly 17,000 supplier audits found that supply chain monitors are more lenient when auditing entities they have inspected before [69].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [69]

In the paper

“Much like how FDA's system has enabled a wide range of techniques to surface issues, the incident reporting system can focus auditing resources on issues that matter most [81].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [81]

In the paper

“When communities identify issues with algorithmic deployments, it can be difficult for them to articulate or find concrete evidence for such harms [35].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [35]

In the paper

“There is strong evidence that auditorclient familiarity can induce auditors to make judgments that are favorable to their client.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [69]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [29]

In the paper

“One study of 205 NHS hospitals in England found that hospital cleanliness (as perceived by patients) increased during periods when inspections occurred [75].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [75]

In the paper

“In general, making audit outcomes accessible to the public through open release or a vetted request process improves auditor accountability.”

In the paper

“To date, it still often requires extraordinary effort for investigators to gain sufficient access for a thorough analysis.”

In the paper

“Voluntary audits like these can also be compromised if AI vendors require pre-publication review of audit reports, or if (like HireVue) they mandate that individuals sign an NDA in order to view the audit results.”

In the paper

“Rather than allowing companies to pick their own auditors or dictate the focus of their own audits, the literature suggests that there are significant benefits to a regulator (or third party accreditation body) choosing the auditor or using a central fund, instead of relying on direct payment from audit targets to pay auditors.”

In the paper

“Recent reports have found serious limitations with the voluntary audits undertaken by AI vendors in the hiring space, such as Pymetrics and HireVue.”

In the paper

“Second, a fully public inspection system seems unlikely in the near term, as the scale, expertise, and agility needed in the AI governance space far outstrips the current capacity of government agencies [22].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [22]

In the paper

“Such mediated access, subject to auditor vetting (perhaps by an audit oversight board) and consistent with audit scope, will be critical to enabling third party auditing of AI systems.”

In the paper

“In AI, auditors retained by technology companies are potentially compromised by nondisclosure agreements, financial conflicts, and competition for services.”

In the paper

“Other research shows that clear and rigorous standard-setting can improve the quality of third-party audits.”

In the paper

“For example, some financial auditors must make their communications and records publicly accessible to shareholders in certain U.S. states-this has been proven to improve auditor conduct and lead to higher quality communications [77].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [77]

In the paper

“Even third parties who are contracted to conduct an audit can run into challenges, as access is provided only on the audited target's terms.”

In the paper

“The capacity gap also strongly calls into question the focus of current policy proposals that focus on academic researchers [60].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [60]

In the paper

“Often, such collaborative arrangements also include some control or restriction through contracts of how much of the auditor results can be shared more broadly (as was the case for the Pymetrics audit [88]).”

What the sources say

• ?Not checked — no full text of the sourcecites [88] source

In the paper

“A randomized controlled trial showed that the accuracy and consistency of these inspections improved with training and peer review.”

In the paper

“One study of supply chain monitors found that audit teams who had more training found significantly more violations than audit teams that were less well-trained [69].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [69]

In the paper

“Third, accreditation is not exclusive of some regulatory oversight.”

In the paper

“Another study found that, observing the same conditions, food safety inspectors disagreed on how to implement the health code 60 percent of the time [37].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [37]

In the paper

“Similarly, attempts by social scientists to work directly with Facebook to gain access to data were thwarted by delayed communications, incomplete data releases and the lack of a clear data dictionary, making the released information indecipherable [60].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [60]

In the paper

“Limiting audit access to academic researchers may narrowly focus audit resources and attention on questions that are important for academic novelty, but misaligned with the needs of other communities.”

In the paper

“The National Institute of Standards and Technology, for instance, protects models by having companies run models via a custom Application Program Interface (API) for the Face Recognition Vendor Test (FRVT).”

In the paper

“Additional research has found that audit quality improves with the amount of client-specific expertise held by audit team members [5].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [5]

In the paper

“Such reports would also help cure one of the major emerging informational asymmetries in technology regulation, by opening the black box of algorithmic design.”

In the paper

“Audit results can also inform more general standard-setting around AI performance and engineering practice.”

In the paper

“Separately, the public release of third-party audits can prompt more rigorous auditing practices by allowing for third-party verification.”

In the paper

“One study found that auditors who have been publicly recognized for their accuracy tend to produce more accurate audits than their non-recognized counterparts, particularly when there are conflicts of interest [23].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [23]

In the paper

“Mandatory disclosure requirements, however, are not costless.”

In the paper

“Repeated assessments, including after product release, can be effective at ensuring compliance.”

In the paper

“A restricted transparency regime for registered audit reports could have significant potential benefits.”

In the paper

“Many AI ethics and accountability discussions have been stymied by the lack of concretely stated expectations of performance and evidence [54].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [54]