The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 19 of the 161 sources its claims cite were checked for support.
Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance
Open accessAbstract
Much attention has focused on algorithmic audits and impact assessments to hold developers and users of algorithmic systems accountable. But existing algorithmic accountability policy approaches have neglected the lessons from non-algorithmic domains: notably, the importance of third parties. Our paper synthesizes lessons from other fields on how to craft effective systems of external oversight for algorithmic deployments. First, we discuss the challenges of third party oversight in the current AI landscape. Second, we survey audit systems across domains - e.g., financial, environmental, and health regulation - and show that the institutional design of such audits are far from monolithic. Finally, we survey the evidence base around these design components and spell out the implications for algorithmic auditing. We conclude that the turn toward audits alone is unlikely to achieve actual algorithmic accountability, and sustained focus on institutional design will be required for meaningful third party involvement.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
72 Sources not found · 59 Own interpretation · 30 No basis given · 15 Not checked yet · 11 Couldn’t confirm · 7 No full text of the sources · 6 Partly supported · 3 Own result · 2 Own method · 1 Not supported · accuracy not yet measured
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Even when connected to representative organizations, such individuals or communities often lack the vocabulary and ability to flag the issues they observe [34].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
What the sources say
- Disputedcites [34] sourceExperimental — accuracy not yet measured
Section 6.5 (page not provided) says that a specialist machine-learning background was not necessary for meaningful feedback and gives examples of community panelists identifying concerns about predictive policing. The paper acknowledges that technical concepts are difficult to explain, but its finding runs contrary to the attributed claim that these communities lack the ability to flag issues they observe. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Even in cases when a participatory approach is considered, the responsibility of consulting with external perspectives when considering impacts to articulate in an AIA is still framed as the responsibility of internal stakeholders to coordinate and interpret these interactions [53].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Partly supportedcites [53] sourceExperimental — accuracy not yet measured
Section 2.3 (page unavailable) says participatory design brings relevant stakeholders into development and that gathering their views and resolving conflicting specifications create work and costs. It does not discuss AI impact assessments (AIAs) or say that internal stakeholders are responsible for coordinating and interpreting external consultations in an AIA. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Together, these watchdogs continue to participate in consequential studies to analyze the biased outcomes of models used in domains as varied as content curation [80], online advertising [40], tenant screening [39,70], child welfare [12], criminal justice [5,10,20,26,78], hiring [63], immigration [24], education [25] or public health [15,56].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
What the sources say
- Could not verifycites [80] sourceExperimental — accuracy not yet measured
The Conclusion (page unavailable) describes a drug-safety study that developed and evaluated models for predicting adverse drug interactions. The Discussion mentions possible confounding and indication biases, but neither passage describes a third-party watchdog study of biased model outcomes. I searched for watchdogs, biased outcomes, and model bias in public health. (LLM, conf=0.73)
- Not matchedcites [40]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [39]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [70]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“The evidence on mandatory rotations is more mixed.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [22]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [32]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [38]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [43]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [65] sourceExperimental — accuracy not yet measured
Section 5, “Limitations of Internal Audits” (page unavailable), discusses auditor independence and the limits of internal audits, but not mandatory rotations or evidence about their effects. Searches for “mandatory rotations” and “auditor rotation mixed evidence” found no passage addressing the claim. (LLM, conf=0.73)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“A two-year randomized controlled trial showed that environmental third party audits were more accurate when auditors were not paid by the firm being audited, but by a common pool of government distributed funding [18].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Could not verifycites [18] sourceExperimental — accuracy not yet measured
The 'Some key problems' section discusses conflicts of interest in corporate-funded AI ethics research and independent AI audits, but not environmental audits. Searches for the two-year trial and the comparison between firm-paid and government-funded auditors found no passage addressing the claimed accuracy result. (LLM, conf=0.75)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“The longer an auditor gets to know a firm, the more firm-specific expertise the auditor acquires.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Not matchedcites [22]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [32]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [38]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [43]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [65] sourceExperimental — accuracy not yet measured
Sections 2.4 (The Internal Audit) and 3.3.1 (Audit Infrastructure) discuss auditors’ access to organizational information, but neither addresses whether longer tenure increases firm-specific expertise. Searches for auditor tenure and for auditors gaining knowledge of a firm over time found no passage making that point. Page numbers were not available in our copy. (LLM, conf=0.76)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“The tenure of auditors is another key determinant of independence [15].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Could not verifycites [15] sourceExperimental — accuracy not yet measured
Section 4 describes an audit of demographic disparities in mobility-data coverage, and §4.5 applies it to voter turnout. Searches for auditor tenure, appointment length, and independence found no passage addressing whether tenure determines auditor independence. The source therefore does not verify the cited claim. (LLM, conf=0.78)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“There is strong social science demonstrating that selection and compensation can undermine independence and, in turn, audit quality.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Partly supportedcites [18] sourceExperimental — accuracy not yet measured
In “Independence and direction of research” (page unavailable), the paper says funding requirements can affect research independence. “Some key problems” (page unavailable) describes conflicts of interest in corporate-funded AI ethics work and calls for independent audits. It does not present social-science evidence that auditor selection or compensation arrangements undermine audit independence and, in turn, audit quality. (LLM, conf=0.78)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, even in instances when the algorithmic systems are themselves disclosed, it remains difficult to anticipate which of these deployments are linked to known incidents of harms, and thus challenging to prioritize cases or pursue targets proactively [64].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
What the sources say
- Could not verifycites [64] sourceExperimental — accuracy not yet measured
Section 3.1.1 discusses choosing the scope of an audit, and Section 3.2.3 discusses disclosure of audit targets and its effects on auditability. Neither addresses whether disclosed system deployments can be linked to known incidents of harm or whether that difficulty prevents proactive prioritization. Searches for those connections found no supporting passage in our copy. (LLM, conf=0.70)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Similarly, when car owners are given the option to choose their own car inspectors, they select for more lenient car inspectors, resulting in a race to the bottom [10,34].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [34] sourceExperimental — accuracy not yet measured
The Background (section 3) discusses oversight of surveillance technologies, and section 6.3 discusses algorithmic accountability. Neither addresses car owners choosing inspectors or a resulting race to the bottom. Searches for car-inspector choice, inspector leniency, and “race to the bottom” found no relevant passage in our copy. (LLM, conf=0.76)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
- It may state two separate claims as one. 56
- Probably not a substantial claim. 77
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For instance, Google researchers on the internal AI team were dismissed and blocked from publishing critiques on the large-scale language models [18,27], which would later be revealed to be critical to the company's future product roadmap [23,85,87].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Partly supportedcites [18] sourceExperimental — accuracy not yet measured
The Introduction reports that Google fired Timnit Gebru and Margaret Mitchell and that they were told not to cast Google’s technologies in a negative light in research focused on large language models. It does not establish that these researchers were blocked from publishing the critiques. The later product-roadmap point has separate citations in the citing sentence and is not attributed to this source. (LLM, conf=0.84)
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [85]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [87] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For an audit to successfully measure performance, its scope should be sufficiently delineated.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Audit Precision.
What the sources say
- Partly supportedcites [42] sourceExperimental — accuracy not yet measured
The sections on benchmarking and defining a peer group say that how a comparison group and performance range are defined is crucial to assessing a company's ESG performance. That supports the general need to define the basis of an assessment, but the paper does not address audits or state a requirement about an audit's scope (searched: audit, scope, delineated). No page number was available in our copy. (LLM, conf=0.76)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Voluntary reporting can be unreliable and create reporting bias [4].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
What the sources say
- Could not verifycites [4] sourceExperimental — accuracy not yet measured
The paper’s “Measuring and comparing audiences” section uses self-reported gender and race data and notes that voter-file information may be outdated. Neither that section nor the other passages found when searching for voluntary reporting and reporting bias says that voluntary reporting is unreliable or creates reporting bias. (LLM, conf=0.74)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Similar biases have been found in recent consumer complaints to the Consumer Financial Protection Bureau (CFPB) [65]-although early CFPB complaints, if anything, appeared to be higher from minority neighborhoods [6].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
What the sources say
- Could not verifycites [65] sourceExperimental — accuracy not yet measured
Section 2.1 discusses bias in automated facial analysis, not consumer complaints. Searches for CFPB, consumer complaints, and minority neighborhoods found no passage in our copy addressing the finding attributed to this source. The separate claim about early complaints is attached to citation [6], not this source. (LLM, conf=0.75)
- Not matchedcites [6]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Similar resource constraints prompted the development of”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
What the sources say
- Not matchedcites [35]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [66]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [53] sourceExperimental — accuracy not yet measured
The passages on medical oversight (sections 2.3–2.4; page numbers unavailable) discuss professional review and accountability, not the development of surveillance or complaint databases. Searches for syndromic surveillance, pharmaceutical adverse-event reporting, financial fraud complaints, resource constraints, and real-time audit signals found no passage addressing the claim. (LLM, conf=0.75)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“One study of occupational licensing laws for teachers found that stricter occupational licensing regimes is associated with a wider distribution of student test scores [52], though others concluded that state occupational licensing requirements for teachers had no impact on students' standardized test scores [28].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Could not verifycites [52] sourceExperimental — accuracy not yet measured
The citation to this source is attached to the first clause, about stricter teacher licensing and the distribution of student test scores. Our copy discusses algorithmic impact assessments and accountability, including testing in a different context (section 4.1), but the passages found for searches on teacher occupational licensing and student test scores do not address that association. The second clause is attributed to a different citation. (LLM, conf=0.76)
- Not matchedcites [28]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Could not verify
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Rotation requirements, which require auditors to rotate across audit targets to ostensibly foster independence, are employed in a minority of cases.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“First, we note that nearly all audit programs have specific standards and rules that help to scope the audit.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Second, we note that significant variation exists within the third party audit category, which may be conducted by public agencies (e.g., nursing home inspections) or private parties (e.g., children's toys), which typically also determines whether auditors are compensated by the audited party.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“What this survey highlights most of all is that there are many varieties of audits.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Due to limited regulatory capacity, public inspections appear more likely to be initiated by some complaint or risk-based heuristic, rather than being universally mandated.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Last, in many cases, the audit scheme contemplates post-audit actions, such as recommendations for remediation and disclosure.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Fourth, accreditation schemes exist for all instances where the auditee selects and compensates the third party auditor.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As seen in the instances where auditees choose auditors in Table 1, accreditation is widespread.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As Table 1 shows, third party audits invariably involve privileged access to information.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.1 Evidence Base.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, audited companies could outright ignore or overlook audit results.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Like in ESG, conflicts of interest can compromise the quality of third-party audits of algorithmic systems.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Existing evidence from other regulatory fields suggests that effective target identification mechanisms, independence in selection and compensation, access to data and AI systems, clear and well-scoped audit standards, and post-audit report transparency will be critical to ensure that audits promote, rather than degrade, overall accountability.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Thus, AIAs are often practically executed by similar stakeholders as internal audits, with broader regulatory implications and a purpose framed towards prompting and gauging institutional reflexivity and due diligence, in addition to the audit goals of performance assessment with respect to articulated expectations [47].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [47]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Audits can be qualitative or quantitative, as system expectations can be articulated and assessed in either mode.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.1 Terminology
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Auditing the automated decision systems of a large technology platform can rival the complexity of auditing for human rights violations in a large supply chain.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The scope of an AIA is thus potentially broader than a specific product feature, and the assessment goes beyond just evaluation measurements, to include more reflective articulations of the trade-offs and decision-making inherent in the release and development of a particular algorithmic system.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mediation of auditor access is difficult, and the identification of audit targets and scope uncertain.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“Audits evaluate performance relative to expected behavior, as part of a broader accountability process [62].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.1 Terminology
What the sources say
- Not checked — no full text of the sourcecites [62] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“The impact of this work is undeniable -leading to voluntary moratoriums [30], and several successful lawsuits [75,79]; nation-wide waves of regulatory action and public outcry [74]; congressional hearings [3]; and the recalls or radical redesigns of the deployed algorithms [12,64,65,88].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
What the sources say
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [75]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [79] source
- Not matchedcites [74]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The standards and scope of an algorithmic audit may be unclear -leading to assessments that are unfocused and expensive.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“And due to the ambiguity of what is being audited, ESG certifications risk becoming”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [19] source
- Not matchedcites [37]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“In the same vein, regulatory and institutional interpretations of hate speech often differs dramatically from the understanding of those experiencing hate online [43], leading to deployed moderation tools that further marginalize and censor impacted users rather than protect them [16].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [43]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“By convention, audits are classified into three types [14].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.1 Terminology
What the sources say
- Not matchedcites [14]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“The controversial facial recognition company Clearview AI, for example, claimed to be”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
What the sources say
- Not matchedcites [29]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“There is a clear precedent for situations of auditors compromised by selection, payment, or employment by auditees; auditors struggling to secure access; auditors clumsily navigating diffuse performance standards; and auditees that may elect to share or conceal findings otherwise hidden behind a nondisclosure agreement.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [19] source
- Not matchedcites [37]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This tends to prioritize at most the concerns of regulators or customers, primarily to reduce liability risk, which may not align with core concerns of impacted populations.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [54]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [66]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We note that while third party oversight by non-profit organizations, journalists, and academics have, as noted above, played critical roles in exposing weaknesses to AI systems, such forms of oversight are not conventionally considered audits per se.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.1 Terminology
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Unlike most audits, AIAs are not as strictly defined and tend to include more open-ended inquiry into the impacts of a particular algorithmic deployment [55].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [55]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All of these format and content oriented developments work to restrict and define what is to be included in the audit, focusing on concerns that are preferable for internal auditors to operationalize, and that requires a certain level of access or internal co-operation to achieve.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We conclude that the turn toward audits alone is unlikely to achieve actual algorithmic accountability, and sustained focus on institutional design will be required for meaningful third party involvement.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Identifying audit targets and scope is critical to ensuring that audits are applied to relevant parties, and conducted for the right issues.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1 Target Identification & Audit Scope
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In those instances, risk-based or complaint-based selection of audit targets can help to allocate attention where needed the most.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Companies can themselves adopt incident reporting systems to raise user and public awareness of a system's use in consequential scenarios.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A third party audit where (a) the auditor is selected and compensated by the auditee (b) to conduct an audit of a scope and access determined by the auditee, and (c) with limited post-audit actions may be functionally indistinguishable from firstparty audit, and questionably effective in garnering the expected downstream accountability outcomes.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In sum, while policy attention has been turning toward audits, serious limitations characterize the current AI audit landscape.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Other proposals put the onus on regulatory agencies to serve the primary role of external auditor [36], but regulatory capacity to conduct such audits is currently lacking [22].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [22]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Some have noted, for example, that the privacy disclosure pop-ups implemented in response to the EU's ePrivacy Directive have increased”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
What the sources say
- Not matchedcites [67]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [76]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Third-party reporting has a similar effect on small businesses [2].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
What the sources say
- Not matchedcites [2]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“One study found, for instance, that restaurant compliance notices were not representative of actual food safety status, and pushed inspectors to spend outsize resources on adjudicating the accuracy of these notices rather than addressing safety hazards directly [36].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
What the sources say
- Not matchedcites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Before 2019 and the intervention of the Algorithmic Justice League's Gender Shades audit [10], the U.S. National Institute of Technology (NIST) failed to measure demographic differences in their regular Facial Recognition Vendor's Test (FRVT), thus overlooking for many years that”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [26]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“Similarly, the Algorithmic Accountability Act in 2022, and the Algorithmic Justice and Online Platform Transparency Act of 2021 [49] contemplate obligations on regulated parties with enforcement by the Federal Trade Commission (FTC).”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [49]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When audit resources are scarce, however, it may not be possible to select all parties for an audit.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A future where AI audits are conducted by company employees and consulting companies alone fails to learn the lessons of how third parties have generated robust oversight of AI systems.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This stands in sharp contrast to third party oversight of AI products, where access has proven challenging.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such systems are diverse, impacting a range of industries and stakeholders, and it will require more third party participants, representing a variety of communities and making use of a wide set of tools, to concretely assess a wider range of the harms involved for the implicated stakeholders.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The key to third-party audits is independence.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2 Independence
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although definitions are provided for these terms, there is little guidance for how regulators, auditors, and companies are meant to translate such descriptions into specific performance expectations.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Audit Precision.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The literature suggests that there may be benefits to establishing a national incident reporting system to enable third parties to file complaints about algorithmic systems.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Instead of asking,”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Audit Precision.
What the sources say
- Not matchedcites [82]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“System of Records Notices (SORN), which are intended to notify the public whenever a federal agency creates, modifies, or eliminates records involving personally identifiable information, may alert parties to potential uses of algorithms.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The status quo is that the use of algorithms is rarely disclosed in a transparent manner, making it hard to identify audit targets in the first place [84].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
What the sources say
- Not matchedcites [84]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“Murphy & Sandino (2009) found that compensation consultants recommended higher CEO salaries when those consultants offered additional services to those executives [59].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2.1 Evidence Base.
What the sources say
- Not matchedcites [59]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“Some evidence suggests that auditing can help to standardize information for more consistent reporting [30].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Audit Precision.
What the sources say
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“Another study, however, concluded that while cross-selling opportunities could lead to lower quality statutory audits, they did not have an impact on the quality of independent expert reports provided to takeover targets-though the incentive to cross-sell could have an impact on lowering fees [9].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2.1 Evidence Base.
What the sources say
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“No auditor can audit an entire VLOP single-handedly, and the definition of ADS is so context-dependent and purposefully broad [68] that it does little to help scope an audit.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Audit Precision.
What the sources say
- Not matchedcites [68]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“Another study found that private car emissions testing facilities with more opportunities to cross-sell services to customers were more likely to pass cars that did not meet emissions standards [63].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2.1 Evidence Base.
What the sources say
- Not checked — no full text of the sourcecites [63] source
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“If an auditor violates these independence rules, regulators can impose a range of consequences ranging from monetary sanctions to a bar from the industry [1].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Not matchedcites [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One approach to reduce concerns regarding familiarity is seen in financial auditing, where independence rules prohibit close relationships between auditors and their clients, and prevent the auditor from assisting in the preparation of the financial statements.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Opening audits up for third party participation could also allow for a variety of stakeholders to come in with diverse yet precise inquiries specific to the context, focus and organizational target of which they are most concerned.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Audit Precision.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regulators can use this information to identify specific priority targets and assess fees or penalties based on audit results.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Minimizing conflicts of interest and undue influence helps to ensure the reliability of audit results.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2 Independence
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They could then incentivize third party auditors to inspect repeatedly flagged cases in this database through various means, such as RFPs or bug bounties.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nonetheless, considerations for how to adjust for differences in reporting propensities and actively solicit minority perspectives for incident reporting will be a key component of the system's design.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Sarbanes-Oxley, for example, requires mandatory rotation of the lead audit partner.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Currently, third parties are often forced to operate at great legal risk to gain access to the audited AI system, keeping them vulnerable to various forms of corporate retaliation or obstruction.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.2 Implications for AI Auditing.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The literature is much more mixed on the benefits of licensing schemes; accreditation through a professional organization may be preferred for three reasons.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.2 Implications for AI Auditing.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: Demand for AI audit services will grow significantly in the near future.
- Rests on an assumption: Occupational licensing reduces the supply of services.
In the paper
“Training improves audit quality and more experienced team members are associated with more effective audits [5].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“One study, for example, found that a shift in Belgian financial audit standards towards more rule-based procedures decreased financial auditing errors and increased auditor independence [11].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Another analysis of dental licensing requirements found that more rigorous licensing standards did not improve average dental outcomes of patients [45].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [45]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“While much empirical work has focused on the costs and barriers to entry imposed by occupational licensing, less empirical work has probed whether professional licensing has an impact on quality [44].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [44]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“Another found that PCAOB inspections, which assess the compliance of registered public accounting firms with professional audit rules and standards, improved audit quality by prompting lower-quality auditors to exit the market [17].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [17]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Corporations can also use non-legal strategies to block access, including setting up a paywall, adopting prohibitive Terms of Service conditions, or structuring the product to obscure any clear set of test points.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.2 Implications for AI Auditing.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Fears of legal repercussions or corporate retaliation can weaken the audit inquiry, and professional standards can help determine limited conditions for liability.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.2 Implications for AI Auditing.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Several studies have found that increasing or decreasing licensing requirements for CPAs had no meaningful effect on CPA quality [7,12,16].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [7]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [12]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Similarly, Hirevue commissioned and controlled the scope of its audit by ORCAA, including by mediating any public communication of audit results through their own public relations channels.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2.2 Implications for AI Auditing.
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Another study found that unannounced inspections provided better information about the quality of care at nursing homes, and could reduce the regulatory burdens of conducting inspections-though announced inspections could be better suited to identifying organizational issues [46].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.1 Evidence Base.
What the sources say
- Not matchedcites [46]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“Existing research on this question is mixed.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, the PCAOB experience lends support to the creation of an audit oversight board, housed within a public agency or operating as a separate public interest institution, in order to monitor auditors and to determine the expected scope and standards for audit work.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2.2 Implications for AI Auditing.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Similarly, the Department of Justice and the Department of Housing and Urban Development have engaged in audits to analyze disparate outcomes and compliance to civil rights guidelines for decades [78].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.1 Evidence Base.
What the sources say
- Not matchedcites [78]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“For instance, those making use of crowd-sourcing data collection tools can be interrupted abruptly by claims of violations to privacy or intellectual property rights -NYU researchers studying the Facebook ad delivery algorithm were presented with a cease-and-desist order [58]; German auditors at Algorithm Watch were legally threatened by Instagram [9]; and civil society watch dogs were subpoenaed”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.2 Implications for AI Auditing.
What the sources say
- Not matchedcites [58]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [46]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“Another study found that auditors who were subject to PCAOB inspections produced higher quality audits compared to auditors (typically auditors of foreign SEC registrants) that the PCAOB was not allowed to inspect, suggesting that providing access to parties beyond auditors, such as regulators, may provide additional benefits [51].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.1 Evidence Base.
What the sources say
- Not matchedcites [51]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These auditors make use of a range of tools and methodologies to execute their work, and there is no clear sense of expectations as to what a quality audit entails or who is qualified to execute audit work in the algorithmic context.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.2 Implications for AI Auditing.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Non-academic groups can also take advantage of a broader range of strategies to conduct audits and understand harms to distinct communities.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.2 Implications for AI Auditing.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This tradeoff has also been documented in food safety inspections, where frequent interactions between inspectors and facilities may improve inspector expertise, but can increase the likelihood of social coercion and even bribery [49,68].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Not matchedcites [49]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [68]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Protecting proprietary information is not a proper response, as all audit systems provide some form of privileged access to auditors, and disclosure does not need to be direct nor absolute.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.2 Implications for AI Auditing.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: All audit systems provide some form of privileged access to auditors.
- Rests on an assumption: Disclosure of audit information need not be direct or absolute.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Training, standardization, and accreditation of auditors are also key to ensuring that audits are accurately and consistently conducted.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4 Professionalization
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“One systematic analysis of the monitoring methods employed by the company PricewaterhouseCoopers (PwC), for example, found that the resulting audits omitted several major violations and contained serious deficiencies [62].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.1 Evidence Base.
What the sources say
- Not checked — no full text of the sourcecites [62] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For third-party audits to be effective, auditors must be granted access to enough data to conduct a robust review.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3 Auditor Access
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The literature strongly supports training, standardization, and accreditation for third-party AI auditors.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.2 Implications for AI Auditing.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The FDA, for example, may withdraw an auditor's accreditation if that auditor certifies a supplier that is later linked to an outbreak of foodborne illness or if it refuses to give the FDA access to its records [54].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [54]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“The author attributed these shortcomings in part to management pre-screening of information and interviews for auditors.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.1 Evidence Base.
What the sources say
- Not checked — no full text of the sourcecites [62] source
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A major advantage of the audit approach we articulate is its tangibility: audits can help reveal concrete issues and repeated product failures that can later be resolved by general standards.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.2 Implications for AI Auditing.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Another analysis of restaurant grading across ten jurisdictions found that the practice-meant to protect consumers and improve food sanitation via targeted disclosures-suffered from several serious deficiencies, including grade inflation and grading inconsistencies.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AI policy can foster third party audits with more deliberate institutional design, informed by the audit schemes from other industries.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 6 CONCLUSION
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Public visibility of audit results can improve accountability in several ways.It can prevent companies from being able to disguise or hide undesirable audit outcomes [21] and incentivize better behavior.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
What the sources say
- Not matchedcites [21]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“An analysis of PwC's auditing practices also found that the confidential nature of PwC's audits shielded them from verification by other researchers or NGOs and limited their ability to improve the conditions of the regulated entity [62].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
What the sources say
- Not checked — no full text of the sourcecites [62] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As just a first step in the direction of learning from cognate audit schemes, and because the evidence draws from so many different areas, our survey is necessarily selective and incomplete.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 5 LIMITATIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For instance, IEEE P7013 Inclusion and Application Standards for Automated Facial Analysis Technology industry standard is directly informed by the Gender Shades audit.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.2 Implications for AI Auditing.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“First of all, our case studies and evidence offered above are not exhaustive.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 5 LIMITATIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“One study found that publicly traded companies spend an average of $2.2 million to comply with the Sarbanes-Oxley Act's financial reporting requirements [50].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
What the sources say
- Not matchedcites [50]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A registry of audit results avoids the bias from pre-publication review or the secrecy of findings under NDAs and ensures that audits can provide market incentives for companies to correct behavior, inform enforcement actions, and provide an evidence base for standard setting.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.2 Implications for AI Auditing.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“One study found that mandated disclosure of food inspection results increased the compliance of food businesses with food safety regulations [8], though others have found similar food safety disclosure schemes to be ineffective [31,36].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [31]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our point in this specific text is more simple: as the AI community turns toward audits, it has largely ignored what other audit schemes have to teach us on fostering policy environments conducive to third party participation.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 5 LIMITATIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“The implementation of a restaurant grading scheme shifted valuable inspection resources away from more significant safety hazards [36].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
What the sources say
- Not matchedcites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mere insistence for audits is not enough -in particular, specific interventions will be necessary to allow for the effective participation of third parties, who play a critical role yet continue to face serious and often debilitating challenges when engaged in their work.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 6 CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Limited disclosure of audit results would balance the protection of proprietary information against the benefits of external accountability.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.2 Implications for AI Auditing.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim have been checked yet.
In the paper
“Other commentators have also noted the tension that exists, particularly in the context of algorithmic accountability, between the desire for AI transparency and the desire to enforce existing intellectual property rights.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
Lessons from non-algorithmic audit systems can be transferred to the design of AI audit ecosystems.
“Our paper synthesizes lessons from other fields on how to craft effective systems of external oversight for algorithmic deployments.”
“Existing evidence from other regulatory fields suggests that effective target identification mechanisms, independence in selection and compensation, access to data and AI systems, clear and well-scoped audit standards, and post-audit report transparency will be critical to ensure that audits promote, rather than degrade, overall accountability.”
The paper relies on this without support.
Third-party oversight is necessary for meaningful algorithmic accountability.
“We conclude that the turn toward audits alone is unlikely to achieve actual algorithmic accountability, and sustained focus on institutional design will be required for meaningful third party involvement.”
The paper relies on this without support.
Demand for AI audit services will grow significantly in the near future.
“First, the need for AI audit services is likely to grow significantly in the near future, and substantial evidence suggests that occupational licensing reduces the supply of services.”
The paper relies on this without support.
Occupational licensing reduces the supply of services.
“First, the need for AI audit services is likely to grow significantly in the near future, and substantial evidence suggests that occupational licensing reduces the supply of services.”
The paper relies on this without support.
All audit systems provide some form of privileged access to auditors.
“Protecting proprietary information is not a proper response, as all audit systems provide some form of privileged access to auditors, and disclosure does not need to be direct nor absolute.”
The paper relies on this without support.
Disclosure of audit information need not be direct or absolute.
“Protecting proprietary information is not a proper response, as all audit systems provide some form of privileged access to auditors, and disclosure does not need to be direct nor absolute.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Internal auditors have a contractual relationship with the auditee (Raji et al. 2022)”
From Accountability in Government Use of AI : Citizen Concerns and Preferences (2025) · cites this paper as Raji et al. 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, due to procedural non-transparency, audit process effectiveness is not measurable, perpetuating what Power [53] refers to as the 'regress of mistrust, ' where accountability and trust are shifted away from the audited organizations and onto auditors themselves, who are in turn subjected to more audits and evaluations.”
From A Framework for Assurance Audits of Algorithmic Systems (2024) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Likewise, auditors envisioned audit report databases as accountability tools to facilitate the amalgamation and communication of audit findings to key stakeholders, mirroring interventions such as the U.S. Security and Exchange Commission (SEC) EDGAR database for financial accounting audits [88].”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [88]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“External auditors, by contrast, are independent entities with no contractual ties to the auditee (Raji et al. 2022).”
From Accountability in Government Use of AI : Citizen Concerns and Preferences (2025) · cites this paper as Raji et al. 2022
The cited source says what the claim attributes to it.
In the citing paper
“The algorithmic auditing ecosystem is nascent and currently lacks the structure and standards of other audit domains, such as financial services [42].”
From Auditing Work: Exploring the New York City algorithmic bias audit regime (2024) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“A thorough evaluation alone is not sufficient to hold key stakeholders responsible for system-wide behavior [13, 88]-auditors also need tools that support key components of the accountability process, including target identification, standardization of practice, communication, and advocacy [136].”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [88]
The cited source says what the claim attributes to it.
In the citing paper
“Often defined as independent evaluations of the performance, fairness, legality, or safety of deployed AI systems, the maturity of the audit ecosystem in the technology sector lags far behind other industries such as finance and healthcare [38, 88].”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [88]
The cited source says what the claim attributes to it.
In the citing paper
“who face the threat of legal action under existing privacy and anti-hacking laws [88, 122]”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [88]
The cited source says what the claim attributes to it.
In the citing paper
“Power dynamics between auditors and the audited have a critical impact on accountability [14, 88].”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [88]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
“the U.S. National Institute of Technology (NIST) failed to measure demographic differences”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.2 Implications for AI Auditing.
“The National Institute of Standards and Technology, for instance, protects models”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2.1 Evidence Base.
“Murphy & Sandino (2009) found that compensation consultants recommended higher CEO salaries when those consultants offered additional services to those executives [59].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > References
“148. Kevin J Murphy and Tatiana Sandino. 2010. Executive pay and "independent" compensation consultants.”
Citations
Cites
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Self-citationPrinciples alone cannot guarantee ethical AI
Mittelstadt, B. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0114-4 · Cited by 1,886 · cited 3×
“While helpful in aligning internal stakeholders, such an approach is not likely to identify or address the main concerns of external stakeholders [54, 66], particularly if that affected population or its representatives are”
BackgroundInfluentialData Mining of the Public Version of the FDA Adverse Event Reporting System
Sakaeda, T., Tamon, A., Kadoyama, K., & Okuno, Y. (2013) · International Journal of Medical Sciences · doi:10.7150/ijms.6048 · Cited by 1,071
Does Teacher Certification Matter? High School Teacher Certification Status and Student Achievement
Goldhaber, D. D., & Brewer, D. J. (2000) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737022002129 · Cited by 1,037
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005
Mitigating bias in algorithmic hiring: evaluating claims and practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372828 · Cited by 845
Discrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · cited 1×
“Together, these watchdogs continue to participate in consequential studies to analyze the biased outcomes of models used in domains as varied as content curation [80], online advertising [40], tenant screening [39, 70], child welfare [12], criminal justice [5, 10, 20, 26, 78], hiring [63], immigration [24], education…”
BackgroundFour Things No One Will Tell You About ESG Data
Kotsantonis, S., & Serafeim, G. (2019) · Journal of Applied Corporate Finance · doi:10.1111/jacf.12346 · Cited by 651
Raji, I. D., & Buolamwini, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314244 · Cited by 651
Self-citationWu, E., Wu, K., Daneshjou, R., Ouyang, D., Ho, D. E., & Zou, J. (2021) · Nature Medicine · doi:10.1038/s41591-021-01312-x · Cited by 494
Self-citation
Research works printed but not linked: 28
- 2021. Impact. https://themarkup.org/series/impact Last accessed 25 October 2021 . 2021. Impact. https://themarkup.org/series/impact Last accessed 25 October 2021.
- Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . 2016 . Machine bias . ProPublica , May 23, 2016 (2016), 139--159. Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica, May 23, 2016 (2016), 139--159.
- Ann H. Baier and Lisa Ahramjian . 2020 . Organic Certification of Farms and Businesses Producing Agricultural Products ., 8 pages. Ann H. Baier and Lisa Ahramjian. 2020. Organic Certification of Farms and Businesses Producing Agricultural Products., 8 pages.
- Riccardo Boffo and Robert Patalano . 2020. Esg investing: Practices, progress and challenges. Éditions OCDE , Paris ( 2020 ). Riccardo Boffo and Robert Patalano. 2020. Esg investing: Practices, progress and challenges. Éditions OCDE, Paris (2020).
- Joy Buolamwini and Timnit Gebru . 2018 . Gender shades: Intersectional accuracy disparities in commercial gender classification . In Conference on fairness, accountability and transparency. PMLR, 77--91 . Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77--91.
- Alexandra Chouldechova , Diana Benavides-Prado , Oleksandr Fialko , and Rhema Vaithianathan . 2018 . A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions . In Conference on Fairness, Accountability and Transparency. PMLR, 134--148 . Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency. PMLR, 134--148.
- Alex C Engler . 2021. Independent auditors are struggling to hold AI companies accountable . FastCompany . Alex C Engler. 2021. Independent auditors are struggling to hold AI companies accountable. FastCompany.
- Foxglove. 2020. We put a stop to the A Level grading algorithm! https://www.foxglove.org.uk/2020/08/17/we-put-a-stop-to-the-a-level-gradingalgorithm/ Last accessed 25 October 2021 . Foxglove. 2020. We put a stop to the A Level grading algorithm! https://www.foxglove.org.uk/2020/08/17/we-put-a-stop-to-the-a-level-gradingalgorithm/ Last accessed 25 October 2021.
- Karen Hao . 2020. We read the paper that forced Timnit Gebru out of Google. Here's what it says. Retrieved January 21 ( 2020 ), 2021. Karen Hao. 2020. We read the paper that forced Timnit Gebru out of Google. Here's what it says. Retrieved January 21 (2020), 2021.
- Caroline Haskins , Ryan Mac , and Logan McDonald . 2020. The ACLU Slammed A Facial Recognition Company That Scrapes Photos From Instagram And Facebook. Buzzfeed News ( 2020 ). Caroline Haskins, Ryan Mac, and Logan McDonald. 2020. The ACLU Slammed A Facial Recognition Company That Scrapes Photos From Instagram And Facebook. Buzzfeed News (2020).
- Lauren Kirchner and Matthew Goldstein . 2020 . Access Denied: Faulty Automated Background Checks Freeze Out Renters. (2020). Lauren Kirchner and Matthew Goldstein. 2020. Access Denied: Faulty Automated Background Checks Freeze Out Renters. (2020).
- Ava Kofman and Ariana Tobin . 2019. Facebook Ads Can Still Discriminate Against Women and Older Workers , Despite a Civil Rights Settlement. ProPublica. Retrieved December 12 ( 2019 ), 2020. Ava Kofman and Ariana Tobin. 2019. Facebook Ads Can Still Discriminate Against Women and Older Workers, Despite a Civil Rights Settlement. ProPublica. Retrieved December 12 (2019), 2020.
- Ava Kofman and Ariana Tobin . 2021. Facebook Ads Can Still Discriminate Against Women and Older Workers , Despite a Civil Rights Settlement. ( 2021 ). Ava Kofman and Ariana Tobin. 2021. Facebook Ads Can Still Discriminate Against Women and Older Workers, Despite a Civil Rights Settlement. (2021).
- KATHLEEN McGRORY and NEIL Bedi . 2020. Targeted. The Tampa Bay Times 3 ( 2020 ). KATHLEEN McGRORY and NEIL Bedi. 2020. Targeted. The Tampa Bay Times 3 (2020).
- JB Merrill . 2020. Does facebook still sell discriminatory ads. The Markup ( 2020 ). JB Merrill. 2020. Does facebook still sell discriminatory ads. The Markup (2020).
- Emanuel Moss , Elizabeth AnneWatkins , Ranjit Singh , Madeleine Clare Elish, and Jacob Metcalf . 2021 . Assembling Accountability : Algorithmic Impact Assessment for the Public Interest . Available at SSRN 3877437 (2021). Emanuel Moss, Elizabeth AnneWatkins, Ranjit Singh, Madeleine Clare Elish, and Jacob Metcalf. 2021. Assembling Accountability: Algorithmic Impact Assessment for the Public Interest. Available at SSRN 3877437 (2021).
- Ziad Obermeyer , Brian Powers , Christine Vogeli , and Sendhil Mullainathan . 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 ( 2019 ), 447--453. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 (2019), 447--453.
- Michael O'Leary and Warren Valdmanis . 2021. An ESG Reckoning Is Coming. Harvard Business Review ( 2021 ). Michael O'Leary and Warren Valdmanis. 2021. An ESG Reckoning Is Coming. Harvard Business Review (2021).
- Nathaniel Persily . 2021. Facebook Hides Data Showing It Harms Users. Outside Scholars Need Access. (2021). Last accessed 25 October 2021 . Nathaniel Persily. 2021. Facebook Hides Data Showing It Harms Users. Outside Scholars Need Access. (2021). Last accessed 25 October 2021.
- Elizabeth Pollman . 2019. Corporate Social Responsibility , ESG, and Compliance. Forthcoming , Cambridge Handbook of Compliance (D. Daniel Sokol & Benjamin van Rooij eds.), Loyola Law School, Los Angeles Legal Studies Research Paper 2019 --35 (2019). Elizabeth Pollman. 2019. Corporate Social Responsibility, ESG, and Compliance. Forthcoming, Cambridge Handbook of Compliance (D. Daniel Sokol & Benjamin van Rooij eds.), Loyola Law School, Los Angeles Legal Studies Research Paper 2019--35 (2019).
- Michael Power. 1997. The audit society: Rituals of verification. OUP Oxford. Michael Power. 1997. The audit society: Rituals of verification. OUP Oxford.
- Dillon Reisman , Jason Schultz , Kate Crawford , and Meredith Whittaker . 2018. Algorithmic impact assessments: A practical framework for public agency accountability . AI Now Institute ( 2018 ), 1--22. Dillon Reisman, Jason Schultz, Kate Crawford, and Meredith Whittaker. 2018. Algorithmic impact assessments: A practical framework for public agency accountability. AI Now Institute (2018), 1--22.
- Rashida Richardson . 2021. Defining and Demystifying Automated Decision Systems. Maryland Law Review , Forthcoming ( 2021 ). Rashida Richardson. 2021. Defining and Demystifying Automated Decision Systems. Maryland Law Review, Forthcoming (2021).
- Eva Rosen , Philip ME Garboden, and Jennifer E Cossyleon . 2021 . Racial Discrimination in Housing : How Landlords Use Algorithms and Home Visits to Screen Tenants. American Sociological Review ( 2021), 00031224211029618. Eva Rosen, Philip ME Garboden, and Jennifer E Cossyleon. 2021. Racial Discrimination in Housing: How Landlords Use Algorithms and Home Visits to Screen Tenants. American Sociological Review (2021), 00031224211029618.
- Nathan Sheard. 2021. Banning Government Use of Face Recognition Technology: 2020 Year in Review. Nathan Sheard. 2021. Banning Government Use of Face Recognition Technology: 2020 Year in Review.
- Ariana Tobin . 2019 . HUD sues Facebook over housing discrimination and says the company's algorithms have made the problem worse . ProPublica (March 28 , 2019). Available at https://www.propublica.org/article/hud-sues-facebookhousingdiscrimination-advertising-algorithms (last accessed April 29, 2019) (2019). Ariana Tobin. 2019. HUD sues Facebook over housing discrimination and says the company's algorithms have made the problem worse. ProPublica (March 28, 2019). Available at https://www.propublica.org/article/hud-sues-facebookhousingdiscrimination-advertising-algorithms (last accessed April 29, 2019) (2019).
- Daniel Trielli , Jennifer Stark , and Nicholas Diakopoulos . 2017 . Algorithm tips: A resource for algorithmic accountability in Government . In Proc. Computation+ Journalism Symposium. Daniel Trielli, Jennifer Stark, and Nicholas Diakopoulos. 2017. Algorithm tips: A resource for algorithmic accountability in Government. In Proc. Computation+ Journalism Symposium.
- 10.5555/3152676 · DOI
Cited by
Sorted by citations
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
Hallucination‐Free? Assessing the Reliability of Leading AI Legal Research Tools
Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2025) · Journal of Empirical Legal Studies · doi:10.1111/jels.12413 · Cited by 182
Self-citationBlack-Box Access is Insufficient for Rigorous AI Audits
Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659037 · Cited by 86
AI auditing: The Broken Bus on the Road to AI Accountability
Birhane, A., Steed, R., Ojewale, V., Vecchione, B., & Raji, I. D. (2024) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00037 · Cited by 158
Self-citationThe Gradient of Generative AI Release: Methods and Considerations
Solaiman, I. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593981 · Cited by 90
Characterizing Manipulation from AI Systems
Carroll, M., Chan, A., Ashton, H., & Krueger, D. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623226 · Cited by 131
Chan, A., Ezell, C., Kaufmann, M., Wei, K., Hammond, L., Bradley, H., Bluemke, E., Rajkumar, N., Krueger, D., Kolt, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658948 · Cited by 84
Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
Ojewale, V., Steed, R., Vecchione, B., Birhane, A., & Raji, I. D. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713301 · Cited by 81
Self-citationHorneber, D., & Laumer, S. (2023) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00817-8 · Cited by 101
Deng, W. H., Guo, B., Devrio, A., Shen, H., Eslami, M., & Holstein, K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581026 · Cited by 103
Making It Possible for the Auditing of AI: A Systematic Review of AI Audits and AI Auditability
Li, Y., & Goel, S. (2024) · Information Systems Frontiers · doi:10.1007/s10796-024-10508-8 · Cited by 65
Stogiannos, N., Malik, R., Kumar, A., Barnes, A., Pogose, M., Harvey, H., McEntee, M. F., & Malamateniou, C. (2023) · British Journal of Radiology · doi:10.1259/bjr.20221157 · Cited by 66
Auditing Work: Exploring the New York City algorithmic bias audit regime
Groves, L., Metcalf, J., Kennedy, A., Vecchione, B., & Strait, A. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658959 · Cited by 59
“We join calls from across the FAccT community that similar laws should address the capacity building of algorithm audit expertise, standards of practice, and robust oversight mechanisms [48].”
BackgroundResultInfluentialHartmann, D., de Pereira, J. R. L., Streitbörger, C., & Berendt, B. (2025) · AI and Ethics · doi:10.1007/s43681-024-00595-3 · Cited by 58
Deng, C., Duan, Y., Jin, X., Chang, H., Tian, Y., Liu, H., Wang, Y., Gao, K., Zou, H. P., Jin, Y., et al. (2025) · AI and Ethics · doi:10.1007/s43681-025-00797-3 · Cited by 73
A Framework for Assurance Audits of Algorithmic Systems
Lam, K., Lange, B., Blili-Hamelin, B., Davidovic, J., Brown, S., & Hasan, A. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658957 · Cited by 75
“This lack of clear guidance poses several challenges in the current algorithm auditing industry.”
BackgroundInfluentialHow to design an AI ethics board
Schuett, J., Reuel, A.-K., & Carlier, A. (2025) · AI and Ethics · doi:10.1007/s43681-023-00409-y · Cited by 46
The necessity of AI audit standards boards
Manheim, D., Martin, S., Bailey, M., Samin, M., & Greutzmacher, R. (2025) · AI & Society · doi:10.1007/s00146-025-02320-y · Cited by 45
The Foundation Model Transparency Index
Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2310.12941 · Cited by 46
Three lines of defense against risks from AI
Schuett, J. (2025) · AI & SOCIETY · doi:10.1007/s00146-023-01811-0 · Cited by 41
Artificial intelligence bias auditing – current approaches, challenges and lessons from practice
Lacmanović, S., & Škare, M. (2025) · Review of Accounting and Finance · doi:10.1108/raf-01-2025-0006 · Cited by 43
Blösser, M., & Weihrauch, A. (2023) · European Journal of Marketing · doi:10.1108/ejm-01-2023-0009 · Cited by 35
Abrahams, T. O., Farayola, O. A., Kaggwa, S., Uwaoma, P. U., Hassan, A. O., & Dawodu, S. O. (2024) · Finance & Accounting Research Journal · doi:10.51594/farj.v6i1.706 · Cited by 33
Frontier AI developers need an internal audit function
Schuett, J. (2024) · Risk Analysis · doi:10.1111/risa.17665 · Cited by 50
Seger, E., Dreksler, N., Moulange, R., Dardaman, E., Schuett, J., Wei, K., Winter, C., Arnold, M., Ó hÉigeartaigh, S., Korinek, A., et al. (2023) · doi:10.2139/ssrn.4596436 · Cited by 31
Assessing dual use risks in AI research: necessity, challenges and mitigation strategies
Brenneis, A. (2024) · Research Ethics · doi:10.1177/17470161241267782 · Cited by 36
Contextualizing User Perceptions about Biases for Human-Centered Explainable Artificial Intelligence
Yuan, C. W., Bi, N., Lin, Y.-F., & Tseng, Y. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580945 · Cited by 31
Ononiwu, M., Azonuche, T. I., Okoh, O. F., & Enyejo, J. O. (2023) · International Journal of Scientific Research in Science Engineering and Technology · doi:10.32628/ijsrset232531 · Cited by 31
Should Users Trust Advanced AI Assistants? Justified Trust As a Function of Competence and Alignment
Manzini, A., Keeling, G., Marchal, N., McKee, K. R., Rieser, V., & Gabriel, I. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658964 · Cited by 28
Alam, M. F., Lentsch, A., Yu, N., Barmack, S., Kim, S., Acemoğlu, D., Hart, A. J., Johnson, S., & Ahmed, F. (2024) · doi:10.21428/e4baedd9.e39b392d · Cited by 31
Regulation by Design: Features, Practices, Limitations, and Governance Implications
Prifti, K., Morley, J., Novelli, C., & Floridi, L. (2024) · Minds and Machines · doi:10.1007/s11023-024-09675-z · Cited by 28
Algorithmic Decision-Making, Agency Costs, and Institution-Based Trust
Dowding, K. M., & Taylor, B. R. (2024) · Philosophy & Technology · doi:10.1007/s13347-024-00757-5 · Cited by 26
Beyond Ads: Sequential Decision-Making Algorithms in Law and Public Policy
Henderson, P., Chugg, B., Anderson, B., & Ho, D. E. (2022) · ACM Symposium on Computer Science and Law (CSLAW) · doi:10.1145/3511265.3550439 · Cited by 25
Self-citationFrom Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing
Cen, S. H., & Alur, R. (2024) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3689904.3694711 · Cited by 35
Reclaiming the Digital Commons: A Public Data Trust for Training Data
Chan, A., Bradley, H., & Rajkumar, N. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604658 · Cited by 29
Hartmann, D., Oueslati, A., Staufer, D., Pohlmann, L., Munzert, S., & Heuer, H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713998 · Cited by 29
Algorithmic audits of algorithms, and the law
Merrer, E. L., Pons, R., & Trédan, G. (2023) · AI and Ethics · doi:10.1007/s43681-023-00343-z · Cited by 21
Bound by the Bounty: Collaboratively Shaping Evaluation Processes for Queer AI Harms
Dennler, N. S., Ovalle, A., Singh, A., Soldaini, L., Subramonian, A., Tu, H., Agnew, W. S., Ghosh, A., Yee, K., Peradejordi, I. F., et al. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604682 · Cited by 22
Human-Centered Evaluation and Auditing of Language Models
Xiao, Z., Deng, W. H., Lam, M. S., Eslami, M., Kim, J., Lee, M., & Liao, Q. V. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3636302 · Cited by 26
Accountability in Government Use of AI : Citizen Concerns and Preferences
Feng, N., & Chandra, Y. (2025) · Public Administration · doi:10.1111/padm.70030 · Cited by 18
Kim, S., Ho, J., Li, Y., Fan, B., Yang, W. Y., Ramey, J. B., Fox, S., Zhu, H., Zimmerman, J., & Eslami, M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686905 · Cited by 18
Terms-we-serve-with: Five dimensions for anticipating and repairing algorithmic harm
Rakova, B., Shelby, R. M., & Ma, M. (2023) · Big Data & Society · doi:10.1177/20539517231211553 · Cited by 20
A Framework for Assurance Audits of Algorithmic Systems
Lange, B., Lam, K., Blili-Hamelin, B., Davidovic, J., Brown, S., & Hasan, A. (2024) · PhilPapers (PhilPapers Foundation) · Cited by 17
Lassiter, T. B., & Fleischmann, K. R. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686963 · Cited by 17
Xiong, Y., & Kim, J. K. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100120 · Cited by 17
McClellan, M. (2025) · Journal of risk and financial management · doi:10.3390/jrfm18090475 · Cited by 11
Information Security, Ethics, and Integrity in LLM Agent Interaction
Chen, Y.-J., & Madisetti, V. K. (2025) · Journal of Information Security · doi:10.4236/jis.2025.161010 · Cited by 10
Pitfalls of Evidence-Based AI Policy
Casper, S., Krueger, D., & Hadfield-Menell, D. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i2.14611 · Cited by 25
Access Denied: Meaningful Data Access for Quantitative Algorithm Audits
Zaccour, J., Binns, R., & Rocher, L. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713963 · Cited by 10
Consumer vulnerability within digital platforms as service ecosystems
Lee, K., Bart, Y., & Cauffman, C. (2025) · Journal of Services Marketing · doi:10.1108/jsm-03-2025-0179 · Cited by 9
A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms
Harvey, E., Kizilcec, R. F., & Koenecke, A. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732137 · Cited by 11
Human-Centered Evaluation and Auditing of Language Models
Liu, Y. L., Deng, W. H., Lam, M. S., Eslami, M., Kim, J., Liao, Q. V., Xu, W., Novikova, J., & Xiao, Z. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706729 · Cited by 7
Context-specific certification of AI systems: a pilot in the financial industry
Brogle, K., Kallina, E., Sargeant, H., Shankar, V., Weller, A., Casovan, A., & Bhatt, U. (2025) · AI and Ethics · doi:10.1007/s43681-025-00720-w · Cited by 7
Herrador, M., & Rehberger, J. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.107994 · Cited by 13
Accountability in Algorithmic Systems: From Principles to Practice
Wilkinson, D., Crawford, K., Wallach, H., Raji, I. D., Rakova, B., Singh, R., Strohmayer, A., & Zuckerman, E. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3583747 · Cited by 7
Self-citationIntegrating XAI into Safe AI Development and AI Certification Processes: An Expert-Based Analysis
Frész, B., Göbels, V. P., Omri, S., Brajovic, D., Aichele, A., Kutz, J., Neuhüttler, J., & Huber, M. F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30849-8_13 · Cited by 11
Misabstraction in Sociotechnical Systems
de Troya, Í. M. D. R., Kernahan, J., Doorn, N., Dignum, V., & Dobbe, R. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732122 · Cited by 6
From Incidents to Insights: Patterns of Responsibility following AI Harms
Richards, I., Benn, C., & Zilka, M. (2025) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3757887.3763018 · Cited by 6
The Future of AI in the GCC Post‐NPM Landscape: A Comparative Analysis of Kuwait and the UAE
Albous, M., Alboloushi, B., & Lacheret, A. (2025) · Politics & Policy · doi:10.1111/polp.70084 · Cited by 6
Nieuwenhuizen, E., Trehan, V., & Porumbescu, G. A. (2025) · Public Administration · doi:10.1111/padm.70034 · Cited by 6
Yücel, A., & Bozdoğanoğlu, B. (2024) · Denetişim · doi:10.58348/denetisim.1540801 · Cited by 5
User-based algorithmic auditing
Hacohen, U. Y. (2025) · Theoretical Inquiries in Law · doi:10.1515/til-2025-0012 · Cited by 5
Privacy Concerns in ChatGPT Data Collection and Its Impact on Individuals
Alzamil, L. M., Alhasani, A. M., & Alshehri, S. (2025) · Future Internet · doi:10.3390/fi17110511 · Cited by 5
A Blueprint for Auditing Generative AIPreprint
Mökander, J., Curl, J., & Kshirsagar, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4887768 · Cited by 5
Historical Methods for AI Evaluations, Assessments, and Audits
Sandoval, . C. B., & Jing, F. S. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732093 · Cited by 4
Opportunities and challenges of multidisciplinary algorithmic impact assessments
Sandoval, . C. B., Jing, F. S., Garcia, A. A., Berger, S., Candello, H. C. D. S. P., & Lustig, C. (2025) · Journal of Responsible Innovation · doi:10.1080/23299460.2025.2499302 · Cited by 4
AI, Education, and Academic Publishing in between Knowledge Capitalism and Socialism
Ma, Y. (2025) · Beijing international review of education · doi:10.1177/25902547251390638 · Cited by 4
RECONCILING TRADE SECRETS AND AI PUBLIC TRANSPARENCY
Keller, P., & Aplin, T. F. (2026) · The Cambridge Law Journal · doi:10.1017/s0008197325101086 · Cited by 4
Artificial intelligence and person-centred practice: a critical reflection
Fernandes, J. B., & McCormack, B. G. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02776-2 · Cited by 4
Operational responsibility in AI governance: a user-centric liability framework
Chen, Z. (2026) · AI and Ethics · doi:10.1007/s43681-026-01163-7 · Cited by 5
Toward Patient-Centered AI Fact Labels: Leveraging Extrinsic Trust Cues
Yoo, D. W., Stroud, A. M., Zhu, X., Miller, J. E., & Barry, B. A. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735758 · Cited by 7
Laufer, B., Kleinberg, J. M., Levy, K. E. C., & Nissenbaum, H. F. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623237 · Cited by 4
“It’s Just a Wild, Wild West”: Harnessing Public Procurement as an AI Governance Mechanism
Hudig, A. I., Kallina, E., & Singh, J. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791968 · Cited by 3
“Who is responsible for assessing the adherence to AI-specific requirements, also post-contract award [see e.g. 120, 131, 134]?”
Governing Artificial Intelligence Through Auditing
Oluka, A., & Mashau, P. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-3922-1.ch005 · Cited by 3
Collaboratively Designing and Evaluating Responsible AI Interventions
Deng, W. H., Berman, G., Kaur, H., Schwartz, R., Ibrahim, M. A., Eslami, M., Holstein, K., & Madaio, M. (2024) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3678884.3687136 · Cited by 3
Role and Use of Race in Artificial Intelligence and Machine Learning Models Related to Health
Were, M. C., Li, A., Malin, B., Yin, Z., Coco, J., Collins, B. X., Clayton, E. W., Novak, L. L., Hendricks‐Sturrup, R., Oluyomi, A., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/73996 · Cited by 4
Misalignment or misuse? The AGI alignment tradeoff
Hellrigel-Holderbaum, M., & Dung, L. (2025) · Philosophical Studies · doi:10.1007/s11098-025-02403-y · Cited by 6
High-stakes team based public sector decision making and AI oversightPreprint
Morgan, D., Hashem, Y., Straub, V. J., & Bright, J. (2022) · doi:10.31235/osf.io/arq3w · Cited by 3
A Causal Framework for AI Regulation and AuditingPreprint
Sharkey, L., Ghuidhir, C. N., Braun, D., Scheurer, J., Balesni, M., Bushnaq, L., Stix, C., & Hobbhahn, M. (2024) · Preprints.org · doi:10.20944/preprints202401.1424.v1 · Cited by 9
Improving Group Fairness Assessments with Proxies
Harvey, E., Lee, M. S. A., & Singh, J. P. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3677175 · Cited by 2
Research on the Application of AI Technology in Auditing
Zhang, S. (2024) · Economic management & global business studies. · doi:10.69610/j.emgbs.20240831 · Cited by 2
Governing AI in Public HRM: A Critical Analysis of Taiwan’s Draft Artificial Intelligence Basic Law
Yang, Y.-S. (2025) · Journal of Governance and Public Policy · doi:10.18196/jgpp.v12i3.26878 · Cited by 2
Mapping Research on AI Ethics and Human Rights: A Bibliometric Study with a TCCM-Based Agenda
Singh, J., Thapliyal, S., Kumar, S., Das, S. S., Rawat, S. S., Jha, V. K., Chaudhary, R., Bhagat, R. K., Sharma, S. R., Anand, A., et al. (2026) · F1000Research · doi:10.12688/f1000research.173404.1 · Cited by 2
Discrimination DRIFT: A Practical Framework for Responsible AI
Wyer, S. (2026) · AI and Ethics · doi:10.1007/s43681-026-01211-2 · Cited by 2
Abdallah, A., Methen, W. D. S. A., Diab, M. F. S., Mohamed, G. S. K., Alsarhan, M. F., Aleisa, H. E. H., Alshammari, M. D. M., & Alazzam, M. B. (2025) · EDPACS · doi:10.1080/07366981.2025.2606527 · Cited by 4
Global Approaches to Auditing Artificial Intelligence: A Literature Review
(IPIE), I. P. O. T. I. E., Sylvester, P., Chun, W. H. K., Hannák, A., Mendoza, M., Broussard, M., Сергеевич, Е. Р., Nelson, A., Sandvig, C., Sesan, G., et al. (2024) · doi:10.61452/bwym7397 · Cited by 2
Towards A Global AI Auditing Framework: Assessment and Recommendations
(IPIE), I. P. O. T. I. E., Faveri, B., Johnson-León, M., Sylvester, P., Chun, W. H. K., Hannák, A., Mendoza, M., Broussard, M., Сергеевич, Е. Р., Nelson, A., et al. (2025) · doi:10.61452/zwed1485 · Cited by 2
AI Adoption, Audit Outcomes, and Professional Reputation
Tarawiru, Y., Hasdiana, Bachtiar, Y., & Parman (2025) · Journal of Computer Information Systems · doi:10.1080/08874417.2025.2581114 · Cited by 1
Nainggolan, A. N., & Winoto, A. (2025) · International Conference on Computing and Applied Informatics (ICCAI) · doi:10.1109/iccai65301.2025.11278986 · Cited by 1
Regulatory Frameworks and Ethical Governance of Neural Networks in Computer Science Applications
Kumar, G., Kumari, R., Shukla, A., Yadav, A. P., Verma, R., & Gaur, J. (2025) · IEEE International Symposium on Sustainable Energy, Signal Processing and Cyber Security (iSSSC) · doi:10.1109/isssc66652.2025.11388372 · Cited by 1
Comprehensive and comparative analysis of AI in the energy sector
Worthmann, C., & Malik, K. (2025) · AI and Ethics · doi:10.1007/s43681-025-00728-2 · Cited by 2
Auditoria i control dels sistemes d’intel·ligència artificial d’alt risc al sector públic
Pérez, J. C. (2025) · Revista Catalana de Dret Públic · doi:10.58992/rcdp.i71.2025.4506 · Cited by 1
Jin, A., Asemota, A., Krane, D. E., Adams, N., & Abebe, R. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791552 · Cited by 1
Beyond Explanation: Evidentiary Rights for Algorithmic Accountability
Stewart, M. (2026) · Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3805689.3806532 · Cited by 1
Alamdari, P. A., Klassen, T. Q., & McIlraith, S. A. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812339 · Cited by 1
There is no free benchmark: An institutional view of legal AI benchmarking
Guha, N., Zhang, A. K., Tsang, C., Manning, C. D., Nyarko, J., & Ho, D. E. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2509757122 · Cited by 3
Self-citationA comparative analysis of AI regulation of the EU and Canada with emphasis on oversight
Sarabdeen, J. (2026) · International Journal of Law and Management · doi:10.1108/ijlma-08-2025-0377 · Cited by 1
COMPLIANCE THEATRE: RETHINKING EVALUATION AND ENFORCEMENT IN FRONTIER AI REGULATION
Bartlett, M. (2026) · The Cambridge Law Journal · doi:10.1017/s000819732610155x · Cited by 1
Metadata
- Authors
- Raji, I. D., Xu, P., Honigsberg, C., & Ho, D.
- Affiliations
- University of California, Berkeley (US)
- Stanford University (US)
- Venue
- Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2022-07-26
- Pages
- 557-571
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W4288058319
- Semantic Scholar
- c253ab3e988d996b9253f8437cbdc37c415a2f0b
- Cited by
- 209
- References
- 27
- Citations by year
- 2026: 76 · 2025: 65 · 2024: 37 · 2023: 28 · 2022: 2
- FWCI
- 14.10
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Mozilla Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Digitalization, Law, and Regulation
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- third-party oversight, algorithmic auditing, AI governance, algorithmic accountability, institutional design, external supervision, regulatory audits, financial regulation, environmental regulation, health regulation, algorithmic impact assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
