The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 19 of the 161 sources its claims cite were checked for support.
Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance
Open accessAbstract
Much attention has focused on algorithmic audits and impact assessments to hold developers and users of algorithmic systems accountable. But existing algorithmic accountability policy approaches have neglected the lessons from non-algorithmic domains: notably, the importance of third parties. Our paper synthesizes lessons from other fields on how to craft effective systems of external oversight for algorithmic deployments. First, we discuss the challenges of third party oversight in the current AI landscape. Second, we survey audit systems across domains - e.g., financial, environmental, and health regulation - and show that the institutional design of such audits are far from monolithic. Finally, we survey the evidence base around these design components and spell out the implications for algorithmic auditing. We conclude that the turn toward audits alone is unlikely to achieve actual algorithmic accountability, and sustained focus on institutional design will be required for meaningful third party involvement.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
19 of 161 cited sources checked · 0 supported · 1 flagged · accuracy not yet measured
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Even when connected to representative organizations, such individuals or communities often lack the vocabulary and ability to flag the issues they observe [34].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
What the sources say
- Disputedcites [34] sourceExperimental — accuracy not yet measured
Section 6.5 (page not provided) says that a specialist machine-learning background was not necessary for meaningful feedback and gives examples of community panelists identifying concerns about predictive policing. The paper acknowledges that technical concepts are difficult to explain, but its finding runs contrary to the attributed claim that these communities lack the ability to flag issues they observe. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Even in cases when a participatory approach is considered, the responsibility of consulting with external perspectives when considering impacts to articulate in an AIA is still framed as the responsibility of internal stakeholders to coordinate and interpret these interactions [53].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Partly supportedcites [53] sourceExperimental — accuracy not yet measured
Section 2.3 (page unavailable) says participatory design brings relevant stakeholders into development and that gathering their views and resolving conflicting specifications create work and costs. It does not discuss AI impact assessments (AIAs) or say that internal stakeholders are responsible for coordinating and interpreting external consultations in an AIA. (LLM, conf=0.82)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Together, these watchdogs continue to participate in consequential studies to analyze the biased outcomes of models used in domains as varied as content curation [80], online advertising [40], tenant screening [39,70], child welfare [12], criminal justice [5,10,20,26,78], hiring [63], immigration [24], education [25] or public health [15,56].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
What the sources say
- Could not verifycites [80] sourceExperimental — accuracy not yet measured
The Conclusion (page unavailable) describes a drug-safety study that developed and evaluated models for predicting adverse drug interactions. The Discussion mentions possible confounding and indication biases, but neither passage describes a third-party watchdog study of biased model outcomes. I searched for watchdogs, biased outcomes, and model bias in public health. (LLM, conf=0.73)
- Not matchedcites [40]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [39]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [70]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“The evidence on mandatory rotations is more mixed.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [22]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [32]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [38]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [43]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [65] sourceExperimental — accuracy not yet measured
Section 5, “Limitations of Internal Audits” (page unavailable), discusses auditor independence and the limits of internal audits, but not mandatory rotations or evidence about their effects. Searches for “mandatory rotations” and “auditor rotation mixed evidence” found no passage addressing the claim. (LLM, conf=0.73)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes- Not matched
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“A two-year randomized controlled trial showed that environmental third party audits were more accurate when auditors were not paid by the firm being audited, but by a common pool of government distributed funding [18].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Could not verifycites [18] sourceExperimental — accuracy not yet measured
The 'Some key problems' section discusses conflicts of interest in corporate-funded AI ethics research and independent AI audits, but not environmental audits. Searches for the two-year trial and the comparison between firm-paid and government-funded auditors found no passage addressing the claimed accuracy result. (LLM, conf=0.75)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“The longer an auditor gets to know a firm, the more firm-specific expertise the auditor acquires.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Not matchedcites [22]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [32]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [38]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [43]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [65] sourceExperimental — accuracy not yet measured
Sections 2.4 (The Internal Audit) and 3.3.1 (Audit Infrastructure) discuss auditors’ access to organizational information, but neither addresses whether longer tenure increases firm-specific expertise. Searches for auditor tenure and for auditors gaining knowledge of a firm over time found no passage making that point. Page numbers were not available in our copy. (LLM, conf=0.76)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Not matched
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“The tenure of auditors is another key determinant of independence [15].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Could not verifycites [15] sourceExperimental — accuracy not yet measured
Section 4 describes an audit of demographic disparities in mobility-data coverage, and §4.5 applies it to voter turnout. Searches for auditor tenure, appointment length, and independence found no passage addressing whether tenure determines auditor independence. The source therefore does not verify the cited claim. (LLM, conf=0.78)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“There is strong social science demonstrating that selection and compensation can undermine independence and, in turn, audit quality.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Partly supportedcites [18] sourceExperimental — accuracy not yet measured
In “Independence and direction of research” (page unavailable), the paper says funding requirements can affect research independence. “Some key problems” (page unavailable) describes conflicts of interest in corporate-funded AI ethics work and calls for independent audits. It does not present social-science evidence that auditor selection or compensation arrangements undermine audit independence and, in turn, audit quality. (LLM, conf=0.78)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, even in instances when the algorithmic systems are themselves disclosed, it remains difficult to anticipate which of these deployments are linked to known incidents of harms, and thus challenging to prioritize cases or pursue targets proactively [64].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
What the sources say
- Could not verifycites [64] sourceExperimental — accuracy not yet measured
Section 3.1.1 discusses choosing the scope of an audit, and Section 3.2.3 discusses disclosure of audit targets and its effects on auditability. Neither addresses whether disclosed system deployments can be linked to known incidents of harm or whether that difficulty prevents proactive prioritization. Searches for those connections found no supporting passage in our copy. (LLM, conf=0.70)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Similarly, when car owners are given the option to choose their own car inspectors, they select for more lenient car inspectors, resulting in a race to the bottom [10,34].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [34] sourceExperimental — accuracy not yet measured
The Background (section 3) discusses oversight of surveillance technologies, and section 6.3 discusses algorithmic accountability. Neither addresses car owners choosing inspectors or a resulting race to the bottom. Searches for car-inspector choice, inspector leniency, and “race to the bottom” found no relevant passage in our copy. (LLM, conf=0.76)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Not matched
- It may state two separate claims as one. 56
- Probably not a substantial claim. 77
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For instance, Google researchers on the internal AI team were dismissed and blocked from publishing critiques on the large-scale language models [18,27], which would later be revealed to be critical to the company's future product roadmap [23,85,87].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Partly supportedcites [18] sourceExperimental — accuracy not yet measured
The Introduction reports that Google fired Timnit Gebru and Margaret Mitchell and that they were told not to cast Google’s technologies in a negative light in research focused on large language models. It does not establish that these researchers were blocked from publishing the critiques. The later product-roadmap point has separate citations in the citing sentence and is not attributed to this source. (LLM, conf=0.84)
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [85]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [87] source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For an audit to successfully measure performance, its scope should be sufficiently delineated.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Audit Precision.
What the sources say
- Partly supportedcites [42] sourceExperimental — accuracy not yet measured
The sections on benchmarking and defining a peer group say that how a comparison group and performance range are defined is crucial to assessing a company's ESG performance. That supports the general need to define the basis of an assessment, but the paper does not address audits or state a requirement about an audit's scope (searched: audit, scope, delineated). No page number was available in our copy. (LLM, conf=0.76)
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Voluntary reporting can be unreliable and create reporting bias [4].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
What the sources say
- Could not verifycites [4] sourceExperimental — accuracy not yet measured
The paper’s “Measuring and comparing audiences” section uses self-reported gender and race data and notes that voter-file information may be outdated. Neither that section nor the other passages found when searching for voluntary reporting and reporting bias says that voluntary reporting is unreliable or creates reporting bias. (LLM, conf=0.74)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Similar biases have been found in recent consumer complaints to the Consumer Financial Protection Bureau (CFPB) [65]-although early CFPB complaints, if anything, appeared to be higher from minority neighborhoods [6].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
What the sources say
- Could not verifycites [65] sourceExperimental — accuracy not yet measured
Section 2.1 discusses bias in automated facial analysis, not consumer complaints. Searches for CFPB, consumer complaints, and minority neighborhoods found no passage in our copy addressing the finding attributed to this source. The separate claim about early complaints is attached to citation [6], not this source. (LLM, conf=0.75)
- Not matchedcites [6]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:hedged· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Similar resource constraints prompted the development of”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
What the sources say
- Not matchedcites [35]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [66]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [53] sourceExperimental — accuracy not yet measured
The passages on medical oversight (sections 2.3–2.4; page numbers unavailable) discuss professional review and accountability, not the development of surveillance or complaint databases. Searches for syndromic surveillance, pharmaceutical adverse-event reporting, financial fraud complaints, resource constraints, and real-time audit signals found no passage addressing the claim. (LLM, conf=0.75)
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Not matched
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“One study of occupational licensing laws for teachers found that stricter occupational licensing regimes is associated with a wider distribution of student test scores [52], though others concluded that state occupational licensing requirements for teachers had no impact on students' standardized test scores [28].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Could not verifycites [52] sourceExperimental — accuracy not yet measured
The citation to this source is attached to the first clause, about stricter teacher licensing and the distribution of student test scores. Our copy discusses algorithmic impact assessments and accountability, including testing in a different context (section 4.1), but the passages found for searches on teacher occupational licensing and student test scores do not address that association. The second clause is attributed to a different citation. (LLM, conf=0.76)
- Not matchedcites [28]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Could not verify
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Rotation requirements, which require auditors to rotate across audit targets to ostensibly foster independence, are employed in a minority of cases.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“First, we note that nearly all audit programs have specific standards and rules that help to scope the audit.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Second, we note that significant variation exists within the third party audit category, which may be conducted by public agencies (e.g., nursing home inspections) or private parties (e.g., children's toys), which typically also determines whether auditors are compensated by the audited party.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
Strength:hedged· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“What this survey highlights most of all is that there are many varieties of audits.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Due to limited regulatory capacity, public inspections appear more likely to be initiated by some complaint or risk-based heuristic, rather than being universally mandated.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Last, in many cases, the audit scheme contemplates post-audit actions, such as recommendations for remediation and disclosure.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Fourth, accreditation schemes exist for all instances where the auditee selects and compensates the third party auditor.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As seen in the instances where auditees choose auditors in Table 1, accreditation is widespread.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As Table 1 shows, third party audits invariably involve privileged access to information.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.1 Evidence Base.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, audited companies could outright ignore or overlook audit results.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Like in ESG, conflicts of interest can compromise the quality of third-party audits of algorithmic systems.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Existing evidence from other regulatory fields suggests that effective target identification mechanisms, independence in selection and compensation, access to data and AI systems, clear and well-scoped audit standards, and post-audit report transparency will be critical to ensure that audits promote, rather than degrade, overall accountability.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“Thus, AIAs are often practically executed by similar stakeholders as internal audits, with broader regulatory implications and a purpose framed towards prompting and gauging institutional reflexivity and due diligence, in addition to the audit goals of performance assessment with respect to articulated expectations [47].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [47]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Audits can be qualitative or quantitative, as system expectations can be articulated and assessed in either mode.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.1 Terminology
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Auditing the automated decision systems of a large technology platform can rival the complexity of auditing for human rights violations in a large supply chain.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The scope of an AIA is thus potentially broader than a specific product feature, and the assessment goes beyond just evaluation measurements, to include more reflective articulations of the trade-offs and decision-making inherent in the release and development of a particular algorithmic system.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mediation of auditor access is difficult, and the identification of audit targets and scope uncertain.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
In the paper
“Audits evaluate performance relative to expected behavior, as part of a broader accountability process [62].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.1 Terminology
What the sources say
- Not checked — no full text of the sourcecites [62] source
Strength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“The impact of this work is undeniable -leading to voluntary moratoriums [30], and several successful lawsuits [75,79]; nation-wide waves of regulatory action and public outcry [74]; congressional hearings [3]; and the recalls or radical redesigns of the deployed algorithms [12,64,65,88].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
What the sources say
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [75]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [79] source
- Not matchedcites [74]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The standards and scope of an algorithmic audit may be unclear -leading to assessments that are unfocused and expensive.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“And due to the ambiguity of what is being audited, ESG certifications risk becoming”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [19] source
- Not matchedcites [37]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“In the same vein, regulatory and institutional interpretations of hate speech often differs dramatically from the understanding of those experiencing hate online [43], leading to deployed moderation tools that further marginalize and censor impacted users rather than protect them [16].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [43]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
In the paper
“By convention, audits are classified into three types [14].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.1 Terminology
What the sources say
- Not matchedcites [14]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“The controversial facial recognition company Clearview AI, for example, claimed to be”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
What the sources say
- Not matchedcites [29]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“There is a clear precedent for situations of auditors compromised by selection, payment, or employment by auditees; auditors struggling to secure access; auditors clumsily navigating diffuse performance standards; and auditees that may elect to share or conceal findings otherwise hidden behind a nondisclosure agreement.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 1 INTRODUCTION
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [19] source
- Not matchedcites [37]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Load-bearingStrength:boosted· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This tends to prioritize at most the concerns of regulators or customers, primarily to reduce liability risk, which may not align with core concerns of impacted populations.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [54]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [66]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We note that while third party oversight by non-profit organizations, journalists, and academics have, as noted above, played critical roles in exposing weaknesses to AI systems, such forms of oversight are not conventionally considered audits per se.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.1 Terminology
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Unlike most audits, AIAs are not as strictly defined and tend to include more open-ended inquiry into the impacts of a particular algorithmic deployment [55].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [55]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All of these format and content oriented developments work to restrict and define what is to be included in the audit, focusing on concerns that are preferable for internal auditors to operationalize, and that requires a certain level of access or internal co-operation to achieve.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We conclude that the turn toward audits alone is unlikely to achieve actual algorithmic accountability, and sustained focus on institutional design will be required for meaningful third party involvement.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Abstract
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Identifying audit targets and scope is critical to ensuring that audits are applied to relevant parties, and conducted for the right issues.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1 Target Identification & Audit Scope
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In those instances, risk-based or complaint-based selection of audit targets can help to allocate attention where needed the most.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
Strength:plain· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Companies can themselves adopt incident reporting systems to raise user and public awareness of a system's use in consequential scenarios.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A third party audit where (a) the auditor is selected and compensated by the auditee (b) to conduct an audit of a scope and access determined by the auditee, and (c) with limited post-audit actions may be functionally indistinguishable from firstparty audit, and questionably effective in garnering the expected downstream accountability outcomes.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In sum, while policy attention has been turning toward audits, serious limitations characterize the current AI audit landscape.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Other proposals put the onus on regulatory agencies to serve the primary role of external auditor [36], but regulatory capacity to conduct such audits is currently lacking [22].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [22]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Some have noted, for example, that the privacy disclosure pop-ups implemented in response to the EU's ePrivacy Directive have increased”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
What the sources say
- Not matchedcites [67]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [76]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:hedged· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Third-party reporting has a similar effect on small businesses [2].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
What the sources say
- Not matchedcites [2]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“One study found, for instance, that restaurant compliance notices were not representative of actual food safety status, and pushed inspectors to spend outsize resources on adjudicating the accuracy of these notices rather than addressing safety hazards directly [36].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
What the sources say
- Not matchedcites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Before 2019 and the intervention of the Algorithmic Justice League's Gender Shades audit [10], the U.S. National Institute of Technology (NIST) failed to measure demographic differences in their regular Facial Recognition Vendor's Test (FRVT), thus overlooking for many years that”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [26]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
In the paper
“Similarly, the Algorithmic Accountability Act in 2022, and the Algorithmic Justice and Online Platform Transparency Act of 2021 [49] contemplate obligations on regulated parties with enforcement by the Federal Trade Commission (FTC).”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
What the sources say
- Not matchedcites [49]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When audit resources are scarce, however, it may not be possible to select all parties for an audit.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.1 Evidence Base.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A future where AI audits are conducted by company employees and consulting companies alone fails to learn the lessons of how third parties have generated robust oversight of AI systems.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This stands in sharp contrast to third party oversight of AI products, where access has proven challenging.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 3 THE INSTITUTIONAL DESIGN OF AUDIT SYSTEMS
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such systems are diverse, impacting a range of industries and stakeholders, and it will require more third party participants, representing a variety of communities and making use of a wide set of tools, to concretely assess a wider range of the harms involved for the implicated stakeholders.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 2.2 The Outsized Focus on Internal Audits
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The key to third-party audits is independence.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2 Independence
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although definitions are provided for these terms, there is little guidance for how regulators, auditors, and companies are meant to translate such descriptions into specific performance expectations.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Audit Precision.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The literature suggests that there may be benefits to establishing a national incident reporting system to enable third parties to file complaints about algorithmic systems.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Instead of asking,”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Audit Precision.
What the sources say
- Not matchedcites [82]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“System of Records Notices (SORN), which are intended to notify the public whenever a federal agency creates, modifies, or eliminates records involving personally identifiable information, may alert parties to potential uses of algorithms.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The status quo is that the use of algorithms is rarely disclosed in a transparent manner, making it hard to identify audit targets in the first place [84].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
What the sources say
- Not matchedcites [84]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
In the paper
“Murphy & Sandino (2009) found that compensation consultants recommended higher CEO salaries when those consultants offered additional services to those executives [59].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2.1 Evidence Base.
What the sources say
- Not matchedcites [59]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“Some evidence suggests that auditing can help to standardize information for more consistent reporting [30].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Audit Precision.
What the sources say
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“Another study, however, concluded that while cross-selling opportunities could lead to lower quality statutory audits, they did not have an impact on the quality of independent expert reports provided to takeover targets-though the incentive to cross-sell could have an impact on lowering fees [9].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2.1 Evidence Base.
What the sources say
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“No auditor can audit an entire VLOP single-handedly, and the definition of ADS is so context-dependent and purposefully broad [68] that it does little to help scope an audit.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Audit Precision.
What the sources say
- Not matchedcites [68]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
In the paper
“Another study found that private car emissions testing facilities with more opportunities to cross-sell services to customers were more likely to pass cars that did not meet emissions standards [63].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2.1 Evidence Base.
What the sources say
- Not checked — no full text of the sourcecites [63] source
Strength:hedged· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“If an auditor violates these independence rules, regulators can impose a range of consequences ranging from monetary sanctions to a bar from the industry [1].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Not matchedcites [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One approach to reduce concerns regarding familiarity is seen in financial auditing, where independence rules prohibit close relationships between auditors and their clients, and prevent the auditor from assisting in the preparation of the financial statements.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Opening audits up for third party participation could also allow for a variety of stakeholders to come in with diverse yet precise inquiries specific to the context, focus and organizational target of which they are most concerned.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Audit Precision.
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regulators can use this information to identify specific priority targets and assess fees or penalties based on audit results.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Minimizing conflicts of interest and undue influence helps to ensure the reliability of audit results.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2 Independence
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They could then incentivize third party auditors to inspect repeatedly flagged cases in this database through various means, such as RFPs or bug bounties.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nonetheless, considerations for how to adjust for differences in reporting propensities and actively solicit minority perspectives for incident reporting will be a key component of the system's design.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.1.2 Implications for AI Auditing.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Sarbanes-Oxley, for example, requires mandatory rotation of the lead audit partner.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Currently, third parties are often forced to operate at great legal risk to gain access to the audited AI system, keeping them vulnerable to various forms of corporate retaliation or obstruction.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.2 Implications for AI Auditing.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The literature is much more mixed on the benefits of licensing schemes; accreditation through a professional organization may be preferred for three reasons.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.2 Implications for AI Auditing.
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: Demand for AI audit services will grow significantly in the near future.
- Rests on an assumption: Occupational licensing reduces the supply of services.
In the paper
“Training improves audit quality and more experienced team members are associated with more effective audits [5].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Not matched
In the paper
“One study, for example, found that a shift in Belgian financial audit standards towards more rule-based procedures decreased financial auditing errors and increased auditor independence [11].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Another analysis of dental licensing requirements found that more rigorous licensing standards did not improve average dental outcomes of patients [45].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [45]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“While much empirical work has focused on the costs and barriers to entry imposed by occupational licensing, less empirical work has probed whether professional licensing has an impact on quality [44].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [44]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“Another found that PCAOB inspections, which assess the compliance of registered public accounting firms with professional audit rules and standards, improved audit quality by prompting lower-quality auditors to exit the market [17].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [17]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Corporations can also use non-legal strategies to block access, including setting up a paywall, adopting prohibitive Terms of Service conditions, or structuring the product to obscure any clear set of test points.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.2 Implications for AI Auditing.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Fears of legal repercussions or corporate retaliation can weaken the audit inquiry, and professional standards can help determine limited conditions for liability.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.2 Implications for AI Auditing.
Strength:plain· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Several studies have found that increasing or decreasing licensing requirements for CPAs had no meaningful effect on CPA quality [7,12,16].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [7]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [12]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Similarly, Hirevue commissioned and controlled the scope of its audit by ORCAA, including by mediating any public communication of audit results through their own public relations channels.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2.2 Implications for AI Auditing.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“Another study found that unannounced inspections provided better information about the quality of care at nursing homes, and could reduce the regulatory burdens of conducting inspections-though announced inspections could be better suited to identifying organizational issues [46].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.1 Evidence Base.
What the sources say
- Not matchedcites [46]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:hedged· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“Existing research on this question is mixed.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, the PCAOB experience lends support to the creation of an audit oversight board, housed within a public agency or operating as a separate public interest institution, in order to monitor auditors and to determine the expected scope and standards for audit work.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.2.2 Implications for AI Auditing.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
In the paper
“Similarly, the Department of Justice and the Department of Housing and Urban Development have engaged in audits to analyze disparate outcomes and compliance to civil rights guidelines for decades [78].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.1 Evidence Base.
What the sources say
- Not matchedcites [78]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“For instance, those making use of crowd-sourcing data collection tools can be interrupted abruptly by claims of violations to privacy or intellectual property rights -NYU researchers studying the Facebook ad delivery algorithm were presented with a cease-and-desist order [58]; German auditors at Algorithm Watch were legally threatened by Instagram [9]; and civil society watch dogs were subpoenaed”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.2 Implications for AI Auditing.
What the sources say
- Not matchedcites [58]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [46]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
In the paper
“Another study found that auditors who were subject to PCAOB inspections produced higher quality audits compared to auditors (typically auditors of foreign SEC registrants) that the PCAOB was not allowed to inspect, suggesting that providing access to parties beyond auditors, such as regulators, may provide additional benefits [51].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.1 Evidence Base.
What the sources say
- Not matchedcites [51]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These auditors make use of a range of tools and methodologies to execute their work, and there is no clear sense of expectations as to what a quality audit entails or who is qualified to execute audit work in the algorithmic context.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.2 Implications for AI Auditing.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Non-academic groups can also take advantage of a broader range of strategies to conduct audits and understand harms to distinct communities.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.2 Implications for AI Auditing.
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This tradeoff has also been documented in food safety inspections, where frequent interactions between inspectors and facilities may improve inspector expertise, but can increase the likelihood of social coercion and even bribery [49,68].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > Selection and Compensation.
What the sources say
- Not matchedcites [49]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [68]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Protecting proprietary information is not a proper response, as all audit systems provide some form of privileged access to auditors, and disclosure does not need to be direct nor absolute.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.2 Implications for AI Auditing.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: All audit systems provide some form of privileged access to auditors.
- Rests on an assumption: Disclosure of audit information need not be direct or absolute.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Training, standardization, and accreditation of auditors are also key to ensuring that audits are accurately and consistently conducted.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4 Professionalization
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“One systematic analysis of the monitoring methods employed by the company PricewaterhouseCoopers (PwC), for example, found that the resulting audits omitted several major violations and contained serious deficiencies [62].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.1 Evidence Base.
What the sources say
- Not checked — no full text of the sourcecites [62] source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For third-party audits to be effective, auditors must be granted access to enough data to conduct a robust review.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3 Auditor Access
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The literature strongly supports training, standardization, and accreditation for third-party AI auditors.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.2 Implications for AI Auditing.
Load-bearingStrength:boosted· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“The FDA, for example, may withdraw an auditor's accreditation if that auditor certifies a supplier that is later linked to an outbreak of foodborne illness or if it refuses to give the FDA access to its records [54].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.4.1 Evidence Base.
What the sources say
- Not matchedcites [54]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“The author attributed these shortcomings in part to management pre-screening of information and interviews for auditors.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.3.1 Evidence Base.
What the sources say
- Not checked — no full text of the sourcecites [62] source
Strength:plain· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A major advantage of the audit approach we articulate is its tangibility: audits can help reveal concrete issues and repeated product failures that can later be resolved by general standards.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.2 Implications for AI Auditing.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Another analysis of restaurant grading across ten jurisdictions found that the practice-meant to protect consumers and improve food sanitation via targeted disclosures-suffered from several serious deficiencies, including grade inflation and grading inconsistencies.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AI policy can foster third party audits with more deliberate institutional design, informed by the audit schemes from other industries.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 6 CONCLUSION
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Public visibility of audit results can improve accountability in several ways.It can prevent companies from being able to disguise or hide undesirable audit outcomes [21] and incentivize better behavior.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
What the sources say
- Not matchedcites [21]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
In the paper
“An analysis of PwC's auditing practices also found that the confidential nature of PwC's audits shielded them from verification by other researchers or NGOs and limited their ability to improve the conditions of the regulated entity [62].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
What the sources say
- Not checked — no full text of the sourcecites [62] source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As just a first step in the direction of learning from cognate audit schemes, and because the evidence draws from so many different areas, our survey is necessarily selective and incomplete.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 5 LIMITATIONS
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For instance, IEEE P7013 Inclusion and Application Standards for Automated Facial Analysis Technology industry standard is directly informed by the Gender Shades audit.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.2 Implications for AI Auditing.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“First of all, our case studies and evidence offered above are not exhaustive.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 5 LIMITATIONS
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“One study found that publicly traded companies spend an average of $2.2 million to comply with the Sarbanes-Oxley Act's financial reporting requirements [50].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
What the sources say
- Not matchedcites [50]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A registry of audit results avoids the bias from pre-publication review or the secrecy of findings under NDAs and ensures that audits can provide market incentives for companies to correct behavior, inform enforcement actions, and provide an evidence base for standard setting.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.2 Implications for AI Auditing.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“One study found that mandated disclosure of food inspection results increased the compliance of food businesses with food safety regulations [8], though others have found similar food safety disclosure schemes to be ineffective [31,36].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [31]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our point in this specific text is more simple: as the AI community turns toward audits, it has largely ignored what other audit schemes have to teach us on fostering policy environments conducive to third party participation.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 5 LIMITATIONS
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“The implementation of a restaurant grading scheme shifted valuable inspection resources away from more significant safety hazards [36].”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
What the sources say
- Not matchedcites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mere insistence for audits is not enough -in particular, specific interventions will be necessary to allow for the effective participation of third parties, who play a critical role yet continue to face serious and often debilitating challenges when engaged in their work.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 6 CONCLUSION
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Limited disclosure of audit results would balance the protection of proprietary information against the benefits of external accountability.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.2 Implications for AI Auditing.
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim have been checked yet.
In the paper
“Other commentators have also noted the tension that exists, particularly in the context of algorithmic accountability, between the desire for AI transparency and the desire to enforce existing intellectual property rights.”
§ Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance > 4.5.1 Evidence Base.
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
Lessons from non-algorithmic audit systems can be transferred to the design of AI audit ecosystems.
“Our paper synthesizes lessons from other fields on how to craft effective systems of external oversight for algorithmic deployments.”
“Existing evidence from other regulatory fields suggests that effective target identification mechanisms, independence in selection and compensation, access to data and AI systems, clear and well-scoped audit standards, and post-audit report transparency will be critical to ensure that audits promote, rather than degrade, overall accountability.”
The paper relies on this without support.
Third-party oversight is necessary for meaningful algorithmic accountability.
“We conclude that the turn toward audits alone is unlikely to achieve actual algorithmic accountability, and sustained focus on institutional design will be required for meaningful third party involvement.”
The paper relies on this without support.
Demand for AI audit services will grow significantly in the near future.
“First, the need for AI audit services is likely to grow significantly in the near future, and substantial evidence suggests that occupational licensing reduces the supply of services.”
The paper relies on this without support.
Occupational licensing reduces the supply of services.
“First, the need for AI audit services is likely to grow significantly in the near future, and substantial evidence suggests that occupational licensing reduces the supply of services.”
The paper relies on this without support.
All audit systems provide some form of privileged access to auditors.
“Protecting proprietary information is not a proper response, as all audit systems provide some form of privileged access to auditors, and disclosure does not need to be direct nor absolute.”
The paper relies on this without support.
Disclosure of audit information need not be direct or absolute.
“Protecting proprietary information is not a proper response, as all audit systems provide some form of privileged access to auditors, and disclosure does not need to be direct nor absolute.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Internal auditors have a contractual relationship with the auditee (Raji et al. 2022)”
From Accountability in Government Use of AI : Citizen Concerns and Preferences (2025) · cites this paper as Raji et al. 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, due to procedural non-transparency, audit process effectiveness is not measurable, perpetuating what Power [53] refers to as the 'regress of mistrust, ' where accountability and trust are shifted away from the audited organizations and onto auditors themselves, who are in turn subjected to more audits and evaluations.”
From A Framework for Assurance Audits of Algorithmic Systems (2024) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Likewise, auditors envisioned audit report databases as accountability tools to facilitate the amalgamation and communication of audit findings to key stakeholders, mirroring interventions such as the U.S. Security and Exchange Commission (SEC) EDGAR database for financial accounting audits [88].”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [88]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“External auditors, by contrast, are independent entities with no contractual ties to the auditee (Raji et al. 2022).”
From Accountability in Government Use of AI : Citizen Concerns and Preferences (2025) · cites this paper as Raji et al. 2022
The cited source says what the claim attributes to it.
In the citing paper
“The algorithmic auditing ecosystem is nascent and currently lacks the structure and standards of other audit domains, such as financial services [42].”
From Auditing Work: Exploring the New York City algorithmic bias audit regime (2024) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“A thorough evaluation alone is not sufficient to hold key stakeholders responsible for system-wide behavior [13, 88]-auditors also need tools that support key components of the accountability process, including target identification, standardization of practice, communication, and advocacy [136].”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [88]
The cited source says what the claim attributes to it.
In the citing paper
“Often defined as independent evaluations of the performance, fairness, legality, or safety of deployed AI systems, the maturity of the audit ecosystem in the technology sector lags far behind other industries such as finance and healthcare [38, 88].”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [88]
The cited source says what the claim attributes to it.
In the citing paper
“who face the threat of legal action under existing privacy and anti-hacking laws [88, 122]”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [88]
The cited source says what the claim attributes to it.
In the citing paper
“Power dynamics between auditors and the audited have a critical impact on accountability [14, 88].”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [88]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
NIST is given two different names
MinorThe paper expands NIST as the National Institute of Technology in one passage and the National Institute of Standards and Technology in another.
“the U.S. National Institute of Technology (NIST) failed to measure demographic differences”
“The National Institute of Standards and Technology, for instance, protects models”
Murphy and Sandino study has conflicting publication years
MinorThe discussion dates the Murphy and Sandino study to 2009, while the corresponding appendix reference lists it as published in 2010.
“Murphy & Sandino (2009) found that compensation consultants recommended higher CEO salaries when those consultants offered additional services to those executives [59].”
“148. Kevin J Murphy and Tatiana Sandino. 2010. Executive pay and "independent" compensation consultants.”
Citations
Cites
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Self-citationPrinciples alone cannot guarantee ethical AI
Mittelstadt, B. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0114-4 · Cited by 1,886 · cited 3×
“While helpful in aligning internal stakeholders, such an approach is not likely to identify or address the main concerns of external stakeholders [54, 66], particularly if that affected population or its representatives are”
BackgroundInfluentialData Mining of the Public Version of the FDA Adverse Event Reporting System
Sakaeda, T., Tamon, A., Kadoyama, K., & Okuno, Y. (2013) · International Journal of Medical Sciences · doi:10.7150/ijms.6048 · Cited by 1,071
Does Teacher Certification Matter? High School Teacher Certification Status and Student Achievement
Goldhaber, D. D., & Brewer, D. J. (2000) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737022002129 · Cited by 1,037
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005
Mitigating bias in algorithmic hiring: evaluating claims and practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372828 · Cited by 845
Discrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · cited 1×
“Together, these watchdogs continue to participate in consequential studies to analyze the biased outcomes of models used in domains as varied as content curation [80], online advertising [40], tenant screening [39, 70], child welfare [12], criminal justice [5, 10, 20, 26, 78], hiring [63], immigration [24], education…”
BackgroundFour Things No One Will Tell You About ESG Data
Kotsantonis, S., & Serafeim, G. (2019) · Journal of Applied Corporate Finance · doi:10.1111/jacf.12346 · Cited by 651
Raji, I. D., & Buolamwini, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314244 · Cited by 651
Self-citationWu, E., Wu, K., Daneshjou, R., Ouyang, D., Ho, D. E., & Zou, J. (2021) · Nature Medicine · doi:10.1038/s41591-021-01312-x · Cited by 494
Self-citation
Flagged first, then by citations
Research works printed but not linked: 28
- 2021. Impact. https://themarkup.org/series/impact Last accessed 25 October 2021 . 2021. Impact. https://themarkup.org/series/impact Last accessed 25 October 2021.
- Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . 2016 . Machine bias . ProPublica , May 23, 2016 (2016), 139--159. Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica, May 23, 2016 (2016), 139--159.
- Ann H. Baier and Lisa Ahramjian . 2020 . Organic Certification of Farms and Businesses Producing Agricultural Products ., 8 pages. Ann H. Baier and Lisa Ahramjian. 2020. Organic Certification of Farms and Businesses Producing Agricultural Products., 8 pages.
- Riccardo Boffo and Robert Patalano . 2020. Esg investing: Practices, progress and challenges. Éditions OCDE , Paris ( 2020 ). Riccardo Boffo and Robert Patalano. 2020. Esg investing: Practices, progress and challenges. Éditions OCDE, Paris (2020).
- Joy Buolamwini and Timnit Gebru . 2018 . Gender shades: Intersectional accuracy disparities in commercial gender classification . In Conference on fairness, accountability and transparency. PMLR, 77--91 . Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77--91.
- Alexandra Chouldechova , Diana Benavides-Prado , Oleksandr Fialko , and Rhema Vaithianathan . 2018 . A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions . In Conference on Fairness, Accountability and Transparency. PMLR, 134--148 . Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency. PMLR, 134--148.
- Alex C Engler . 2021. Independent auditors are struggling to hold AI companies accountable . FastCompany . Alex C Engler. 2021. Independent auditors are struggling to hold AI companies accountable. FastCompany.
- Foxglove. 2020. We put a stop to the A Level grading algorithm! https://www.foxglove.org.uk/2020/08/17/we-put-a-stop-to-the-a-level-gradingalgorithm/ Last accessed 25 October 2021 . Foxglove. 2020. We put a stop to the A Level grading algorithm! https://www.foxglove.org.uk/2020/08/17/we-put-a-stop-to-the-a-level-gradingalgorithm/ Last accessed 25 October 2021.
- Karen Hao . 2020. We read the paper that forced Timnit Gebru out of Google. Here's what it says. Retrieved January 21 ( 2020 ), 2021. Karen Hao. 2020. We read the paper that forced Timnit Gebru out of Google. Here's what it says. Retrieved January 21 (2020), 2021.
- Caroline Haskins , Ryan Mac , and Logan McDonald . 2020. The ACLU Slammed A Facial Recognition Company That Scrapes Photos From Instagram And Facebook. Buzzfeed News ( 2020 ). Caroline Haskins, Ryan Mac, and Logan McDonald. 2020. The ACLU Slammed A Facial Recognition Company That Scrapes Photos From Instagram And Facebook. Buzzfeed News (2020).
- Lauren Kirchner and Matthew Goldstein . 2020 . Access Denied: Faulty Automated Background Checks Freeze Out Renters. (2020). Lauren Kirchner and Matthew Goldstein. 2020. Access Denied: Faulty Automated Background Checks Freeze Out Renters. (2020).
- Ava Kofman and Ariana Tobin . 2019. Facebook Ads Can Still Discriminate Against Women and Older Workers , Despite a Civil Rights Settlement. ProPublica. Retrieved December 12 ( 2019 ), 2020. Ava Kofman and Ariana Tobin. 2019. Facebook Ads Can Still Discriminate Against Women and Older Workers, Despite a Civil Rights Settlement. ProPublica. Retrieved December 12 (2019), 2020.
- Ava Kofman and Ariana Tobin . 2021. Facebook Ads Can Still Discriminate Against Women and Older Workers , Despite a Civil Rights Settlement. ( 2021 ). Ava Kofman and Ariana Tobin. 2021. Facebook Ads Can Still Discriminate Against Women and Older Workers, Despite a Civil Rights Settlement. (2021).
- KATHLEEN McGRORY and NEIL Bedi . 2020. Targeted. The Tampa Bay Times 3 ( 2020 ). KATHLEEN McGRORY and NEIL Bedi. 2020. Targeted. The Tampa Bay Times 3 (2020).
- JB Merrill . 2020. Does facebook still sell discriminatory ads. The Markup ( 2020 ). JB Merrill. 2020. Does facebook still sell discriminatory ads. The Markup (2020).
- Emanuel Moss , Elizabeth AnneWatkins , Ranjit Singh , Madeleine Clare Elish, and Jacob Metcalf . 2021 . Assembling Accountability : Algorithmic Impact Assessment for the Public Interest . Available at SSRN 3877437 (2021). Emanuel Moss, Elizabeth AnneWatkins, Ranjit Singh, Madeleine Clare Elish, and Jacob Metcalf. 2021. Assembling Accountability: Algorithmic Impact Assessment for the Public Interest. Available at SSRN 3877437 (2021).
- Ziad Obermeyer , Brian Powers , Christine Vogeli , and Sendhil Mullainathan . 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 ( 2019 ), 447--453. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 (2019), 447--453.
- Michael O'Leary and Warren Valdmanis . 2021. An ESG Reckoning Is Coming. Harvard Business Review ( 2021 ). Michael O'Leary and Warren Valdmanis. 2021. An ESG Reckoning Is Coming. Harvard Business Review (2021).
- Nathaniel Persily . 2021. Facebook Hides Data Showing It Harms Users. Outside Scholars Need Access. (2021). Last accessed 25 October 2021 . Nathaniel Persily. 2021. Facebook Hides Data Showing It Harms Users. Outside Scholars Need Access. (2021). Last accessed 25 October 2021.
- Elizabeth Pollman . 2019. Corporate Social Responsibility , ESG, and Compliance. Forthcoming , Cambridge Handbook of Compliance (D. Daniel Sokol & Benjamin van Rooij eds.), Loyola Law School, Los Angeles Legal Studies Research Paper 2019 --35 (2019). Elizabeth Pollman. 2019. Corporate Social Responsibility, ESG, and Compliance. Forthcoming, Cambridge Handbook of Compliance (D. Daniel Sokol & Benjamin van Rooij eds.), Loyola Law School, Los Angeles Legal Studies Research Paper 2019--35 (2019).
- Michael Power. 1997. The audit society: Rituals of verification. OUP Oxford. Michael Power. 1997. The audit society: Rituals of verification. OUP Oxford.
- Dillon Reisman , Jason Schultz , Kate Crawford , and Meredith Whittaker . 2018. Algorithmic impact assessments: A practical framework for public agency accountability . AI Now Institute ( 2018 ), 1--22. Dillon Reisman, Jason Schultz, Kate Crawford, and Meredith Whittaker. 2018. Algorithmic impact assessments: A practical framework for public agency accountability. AI Now Institute (2018), 1--22.
- Rashida Richardson . 2021. Defining and Demystifying Automated Decision Systems. Maryland Law Review , Forthcoming ( 2021 ). Rashida Richardson. 2021. Defining and Demystifying Automated Decision Systems. Maryland Law Review, Forthcoming (2021).
- Eva Rosen , Philip ME Garboden, and Jennifer E Cossyleon . 2021 . Racial Discrimination in Housing : How Landlords Use Algorithms and Home Visits to Screen Tenants. American Sociological Review ( 2021), 00031224211029618. Eva Rosen, Philip ME Garboden, and Jennifer E Cossyleon. 2021. Racial Discrimination in Housing: How Landlords Use Algorithms and Home Visits to Screen Tenants. American Sociological Review (2021), 00031224211029618.
- Nathan Sheard. 2021. Banning Government Use of Face Recognition Technology: 2020 Year in Review. Nathan Sheard. 2021. Banning Government Use of Face Recognition Technology: 2020 Year in Review.
- Ariana Tobin . 2019 . HUD sues Facebook over housing discrimination and says the company's algorithms have made the problem worse . ProPublica (March 28 , 2019). Available at https://www.propublica.org/article/hud-sues-facebookhousingdiscrimination-advertising-algorithms (last accessed April 29, 2019) (2019). Ariana Tobin. 2019. HUD sues Facebook over housing discrimination and says the company's algorithms have made the problem worse. ProPublica (March 28, 2019). Available at https://www.propublica.org/article/hud-sues-facebookhousingdiscrimination-advertising-algorithms (last accessed April 29, 2019) (2019).
- Daniel Trielli , Jennifer Stark , and Nicholas Diakopoulos . 2017 . Algorithm tips: A resource for algorithmic accountability in Government . In Proc. Computation+ Journalism Symposium. Daniel Trielli, Jennifer Stark, and Nicholas Diakopoulos. 2017. Algorithm tips: A resource for algorithmic accountability in Government. In Proc. Computation+ Journalism Symposium.
- 10.5555/3152676 · DOI
Cited by
Sorted by citations
Items 101–165 of 165
Secure Decentralized Learning with Incentives on UTXO-based Blockchains
Lunardon, L. (2023) · IEEE International Conference on Omni-layer Intelligent Systems (COINS) · doi:10.1109/coins57856.2023.10189235 · Cited by 1
Digital Media's Impact on Corporate Innovation in China's Innovation-Driven Strategy
He, M., & Zhao, W. (2024) · International Conference on New Media Development and Modernized Education (NMDME) · doi:10.4108/eai.13-10-2023.2341289 · Cited by 1
Post-Deployment Regulatory Oversight for General-Purpose Large Language ModelsPreprint
Ezell, C., & Loeb, A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4658623 · Cited by 1
Measuring the Effectiveness of Ethics-Based Auditing – A Semi-Systematic Literature ReviewPreprint
Shafik, E., Puncheva, P., Bharathy, G., & Pannai, E. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5498160 · Cited by 1
Blooming in Muddy Waters; DEI at AI Ethics ConferencesPreprint
Tafesse, E., Young, M., Singh, R., & Metcalf, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4570694
Stein, M., Gandhi, M., Kriecherbauer, T., Oueslati, A., & Trager, R. (2024) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v7i1.31733
Position: Contextual Confidence and Generative AI
Jain, S., Hitzig, Z., & Mishkin, P. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00022
Bawah, K. (2025) · World Journal of Advanced Research and Reviews · doi:10.30574/wjarr.2025.27.2.2874
SUNʼIY INTELLEKT SOHASIDA ALGORITMIK JAVOBGARLIK: IJTIMOIY, HUQUQIY VA TASHKILIY YONDASHUVLAR
Imomniyozov, D. (2025) · Ижтимоий-гуманитар фанларнинг долзарб муаммолари / Актуальные проблемы социально-гуманитарных наук / Actual Problems of Humanities and Social Sciences · doi:10.47390/spr1342v5i8y2025n37
Kelincahan Bisnis dan Maqasid al-Syariah : Menyelaraskan Prinsip Etika dengan Dinamika Ekonomi
Maulidizen, A. (2025) · Jurnal Manajemen Bisnis dan Kewirausahaan · doi:10.55606/jumbiku.v5i3.6106 · Cited by 1
Converging Technologies in Neuromarketing: Analysis of Trends and Ethics
Negrete, J. D. C., Mora, H. V. V., & Mazenett, G. D. J. C. (2025) · European Journal of Sustainable Development · doi:10.14207/ejsd.2025.v14n3p210
Judijanto, L. (2025) · European journal of management, economics and business. · doi:10.59324/ejmeb.2025.2(6).11
Responsible AI for Global Health Equity: A Decentralized Framework
Arockiasamy, J. M. I. (2025) · IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas65609.2025.11269620
Wu, K., Yuhuan, C. H. E. N., Xiao, W., & Jiawei, H. U. (2025) · Research Square · doi:10.21203/rs.3.rs-8037100/v1
Yance, T. V., Setiawan, A., Rahmi, & Khairunissa, A. (2025) · Jurnal Ilmu Hukum · doi:10.30652/rj9e0d37
Almeida, R., & Latini, L. (2025) · The Paris Journal on AI & Digital Ethics · doi:10.65701/m4qf7v2k1n
Biological risk and the AI capability shift
Turner, S. D. (2026) · doi:10.59350/zp9qy-3gg73
Fabiano, L., Arenas, M. V. D. S., & de Souza, V. B. P. (2026) · Derecho y cambio social. · doi:10.54899/dcs.v23i87.4375
Okeke, M. U. C., & Adamides, E. D. (2026) · Preprints.org · doi:10.20944/preprints202602.1282.v1
Okeke, M. U. C., & Abel, C. F. (2026) · Preprints.org · doi:10.20944/preprints202602.1282.v2
AI CHAOS! 1st Workshop on the Challenges for Human Oversight of AI Systems
Schrills, T., Kahr, P. K., Langer, M., Kaur, H., & Gadiraju, U. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742414.3794953
Motea, M., & Oba, P. (2026) · Philosophies · doi:10.3390/philosophies11020034
AI CHAOS! 2nd Workshop on the Challenges for Human Oversight of AI Systems
Khadar, M., Cecil, J., Neut, L. V. D., Banović, N., Baum, K. L., Chancellor, S., Costanza, E., Eslami, M., Feit, A. M., Gaube, S., et al. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3778736
Emon, M. M. H. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-3441-7.ch001
Trustworthy Multi-Agent Framework for Video Thumbnail Selection in Data Asset Ecosystems
Zhang, B., Xu, C., & Huang, G. (2025) · International Conference on Big Data Economy and Information Management (BDEIM) · doi:10.1145/3800000.3800224
Crichton, K., Reddy, A., & Ji, J. (2026) · doi:10.51593/20250002
Ahmed, S., Hossain, Z. M. I., Pranta, R. R., & Diba, K. P. (2026) · Journal of Artificial Intelligence and Technological Development · doi:10.59324/jaitd.2026.2(3).03
Policy, Regulation, and the Public Good of Artificial Intelligence
Gnankob, R. I., Mohapatra, J. K., & Dzakpasu, S. E. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7036-1.ch014
The LLM Mirage: Economic Interests and the Subversion of Weaponization Controls
Gupta, R., & Reddie, A. W. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806538
Hudig, A. I., & Singh, J. P. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806741
Navigating AI at Work: An Ethics-in-Practice Approach
Tinsman, C., Krishan, A., & Neff, G. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812382
The Moral Frontier of the AI Age
Joy, J., & George, J. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-8192-3.ch003
Indahsari, L., Wulandari, V., & Hariansyah, S. (2026) · Referendum · doi:10.62383/referendum.v3i2.1746
Safety Theater in Anthropomorphic AI: Why We Need to Shift from Safety to Accountability Narratives
Maeda, T., & Buening, R. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812282 · Cited by 1
Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs
Hartmann, D., Pohlmann, L., Hanslik, L., Gießing, N., Berendt, B., & Delobelle, P. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1681
Common sense testing and sanity checks in machine learning
Naser, M. Z. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01183-w
Naser, M. Z. (2026) · Safety Science · doi:10.1016/j.ssci.2026.107367
A critical analysis of trustworthy AI tools, mark frameworks, and the implementation chasms
Papademas, M., Karpouzis, K. C., Ziouvelou, X., & Karkaletsis, V. (2026) · AI & Society · doi:10.1007/s00146-026-03364-4
Voinea, D. V. (2026) · Social Sciences · doi:10.3390/socsci15080496
Online Harms AI Audit: Policy Memo
steel, B., Owen, T., & Bridgman, A. (2026) · doi:10.66536/online-harms-ai-audit-policy-memo_2026
VerifyWise—A Modular AI Governance Platform for Data-Driven Compliance Management
Khalilzadeh, M., & Razavi, F. (2026) · International Interdisciplinary Conference on Artificial Intelligence: Engineering, Health, Finance and Humanities (IICAI) · doi:10.1109/iicai70155.2026.11620853
AI Monoculture Risk in Foundation Models: From Market Concentration to Systemic LockoutPreprint
Gupta, D. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7198181
Same System, Opposite Verdicts: Metric Discretion in AI Ethics Audits and the Limits of DisclosurePreprint
Tsaban, S. (2026) · Research Square · doi:10.21203/rs.3.rs-10576395/v1
Begum, F., & Valan, J. A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0408-1.ch003
Lin, W., Li, H., Zhuang, J., Li, Y., Yao, X., Chen, M., & Yang, Y. (2026) · International Conference on Innovation Management and Information Systems (ICIMIS) · doi:10.1145/3821966.3821996
Fairness, Bias, and Ethical Governance in Vision Applications
Gupta, B. P. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9285-1.ch008
Transcending natural evolution in the age of artificial intelligence
Guo, P., Jia, X., Tao, L.-Q., Yang, Y., & Ren, T.-L. (2026) · Science China Information Sciences · doi:10.1007/s11432-026-5041-7
Interorganizational AI Governance in Academic and Grey Literature
Khan, F., & Mäntymäki, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38014-2_24
AI GatekeepersPreprint
Langenbucher, K. C. (2026) · doi:10.67929/ecgi-law-932-2026
Rule-Aware Bayesian Reconstruction and Local Auditing of Hidden Crowd PreferencesPreprint
Huang, Y., Cai, Z., Zhang, Z., & Jia, Y. (2026) · Research Square · doi:10.21203/rs.3.rs-10860635/v1
Radhi, M. K., & Hassan, N. F. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-39720-1_6
From Algorithm to Law: How Artificial Intelligence Redefines Work and Labour Rights in Peru
Corzo, M. G. V. (2026) · Laws · doi:10.3390/laws15050119 · Cited by 1
AÇIK DENETİM BORSASI: DENETLENEN DENETİM TEMELLİ KÜRESEL YAPAY ZEKÂ GÜVENLİK VE YÖNETİŞİM MODELİ
Kalmaz, Ş. (2026) · Akademi Sosyal Bilimler Dergisi · doi:10.65083/asbider.1958208
Galindez, W. (2026) · International Journal of Computational Innovations Intelligent Systems and AI · doi:10.64439/cisai.v2i2.46
Ploumpis, I. C., Eriotis, N., Christopoulos, A. G., Kenourgios, D. F., & Kounadeas, T. (2026) · Journal of risk and financial management · doi:10.3390/jrfm19100796
Asuri, R. (2025) · The Eastasouth Journal of Information System and Computer Science · doi:10.58812/esiscs.v3i02.1247
Sanjeewa, W. (2026) · Proceedings of The International Conference on Advanced Research in Management Economics and Accounting · doi:10.33422/armea.v4i1.2204
Gap analysis of AI ethical guidelines in the medical sector: a comprehensive review
Ashtaiwi, A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02071-9
Shihab, A. H., Mussa, A., Ospan, A., & Delavar, A. (2026) · IEEE International Symposium on Intelligent Systems and Informatics (SISY) · doi:10.1109/sisy71351.2026.11738459
A Two-Dimensional Taxonomy of Bias in Biometric Systems: Separating Sources from Manifestations
Grd, P. (2026) · IEEE International Symposium on Intelligent Systems and Informatics (SISY) · doi:10.1109/sisy71351.2026.11738557
Auditing a Dutch Public Sector Risk Profiling Algorithm Using an Unsupervised Bias Detection Tool
Holstege, F., Jorgensen, M., Padh, K., Parie, J., Proroković, K., Persson, J., & Snoek, L. (2025) · arXiv · doi:10.48550/arxiv.2502.01713
“Inspired by established practices in non-algorithmic disciplines, auditing has the potential to mitigate ethical and legal risks in algorithmic systems through both internal and external oversight (Raji et al., 2022).”
Model evaluation for extreme risks
Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Kokotajlo, D., Marchal, N., Anderljung, M., Kolt, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.15324 · Cited by 66
“Ideally there would exist a rich ecosystem of model auditors providing broad coverage across different risk areas.”
BackgroundMethod
Metadata
- Authors
- Raji, I. D., Xu, P., Honigsberg, C., & Ho, D.
- Affiliations
- University of California, Berkeley (US)
- Stanford University (US)
- Venue
- Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2022-07-26
- Pages
- 557-571
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W4288058319
- Semantic Scholar
- c253ab3e988d996b9253f8437cbdc37c415a2f0b
- Cited by
- 209
- References
- 27
- Citations by year
- 2026: 76 · 2025: 65 · 2024: 37 · 2023: 28 · 2022: 2
- FWCI
- 14.10
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Mozilla Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Digitalization, Law, and Regulation
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- third-party oversight, algorithmic auditing, AI governance, algorithmic accountability, institutional design, external supervision, regulatory audits, financial regulation, environmental regulation, health regulation, algorithmic impact assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
