CoChat
Sign Up
Cited by 62Ethics and Social Impacts of AIDigitalization, Law, and RegulationLegal and Labor Studies

Auditing Work: Exploring the New York City algorithmic bias audit regime

Open access
Groves, Lara, Metcalf, Jacob, Kennedy, Alayna, Vecchione, Briana, Strait, Andrew·2024·The 2024 ACM Conference on Fairness, Accountability, and Transparency
DOI 10.1145/3630106.3658959
Open this paper in CoChat — free account

Abstract

In July 2023, New York City (NYC) implemented the first attempt to create an algorithm auditing regime for commercial machine-learning systems. Local Law 144 (LL 144), requires NYC-based employers using automated employment decision-making tools (AEDTs) in hiring to be subject to annual bias audits by an independent auditor. In this paper, we analyse what lessons can be learned from LL 144 for other national attempts to create algorithm auditing regimes. Using qualitative interviews with 17 experts and practitioners working within the regime, we find LL 144 has failed to create an effective auditing regime: the law fails to clearly define key aspects like AEDTs and what constitutes an independent auditor, leaving auditors, vendors who create AEDTs, and companies using AEDTs to define the law’s practical implementation in ways that failed to protect job applicants. Several factors contribute to this: first, the law was premised on a faulty transparency-driven theory of change that fails to stop biased AEDTs from being used by employers. Second, industry lobbying led to the definition of what constitutes an AEDT being narrowed to the point where most companies considered their tools exempt. Third, we find auditors face enormous practical and cultural challenges gaining access to data from employers and vendors building these tools. Fourth, we find wide disagreement over what constitutes a legitimate auditor and identify four different kinds of ‘auditor roles’ that serve different functions and offer different kinds of services. We conclude with four recommendations for policymakers seeking to create similar bias auditing regimes that use clearer definitions and metrics and more accountability. By exploring LL 144 through the lens of auditors, our paper advances the evidence base around audit as an accountability mechanism, and can provide guidance for policymakers seeking to create similar regimes.

AI snapshot

Written from the paper’s full text.

TL;DR

LL 144, New York City’s first algorithmic bias audit regime for hiring AEDTs, is well-intentioned but failed to create an effective auditing regime because vague definitions, industry lobbying, data-access obstacles, and undefined auditor roles left implementation to industry actors. Interviews with 16 experts and practitioners show very low public compliance—only 19 audits found—and recommend clearer definitions and metrics, auditor standards, better data access, and stronger accountability.

Key findings

  • LL 144 failed to clearly define key terms like AEDTs and “independent auditor”; the “substantially assist” criterion let most companies consider their tools exempt, and the authors found only 19 public audits despite thousands of NYC employers.
  • Auditors disagreed over what constitutes legitimate auditing and split into four roles/services: pre-audit/“audit ready” support, audit report production, additional guidance/mitigation, and certification; some early audits were conducted in-house.
  • Data access and metrics were major obstacles: vendors, NDAs, and APIs restricted data; DCWP guidance allowed discretion and potential cherry-picking; imputed/inferred demographic data persisted despite being prohibited; the metric changed from average to median mid-implementation.
  • The law created a secondary market for wider responsible AI services, but its transparency-and-reputational-pressure theory of change did not protect job applicants, especially with no minimum impact ratio or mandated remediation.

Methods

  • Qualitative interviews with 16 experts and practitioners working within the LL 144 regime, recruited via direct and snowball sampling; 42 invitations were sent, interviews ran July–December 2023, were virtual, and were transcribed.
  • Grounded theory analysis using Atlas.ti; four authors coded inductively, generating 102 initial codes reduced to 16, with each interview recoded by two authors.
  • The authors also qualitatively analyzed the third and final round of public comments from January 2023: 46 total comments, including 13 civil society organizations, 4 audit service firms, 6 employment law firms, 7 system developers, and 3 employment platforms; ad hoc searches identified 19 audits claiming LL 144 relevance.

Limitations

  • Stated: narrow geographical focus on US federal employment and civil rights law; not every auditor with attributable published audits participated; management consulting firms offering AI audits for other legislation (e.g., EU Digital Services Act) were not represented; participants were not compensated.
  • Evident from the text: the qualitative sample is small (16 interviewees) and centers auditor/expert perspectives rather than jobseekers or direct outcome data.
  • Evident from the text: public audit availability is incomplete and there is no central repository, so the baseline compliance rate cannot be known precisely; some findings are more applicable to the US context, such as the four-fifths rule.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3630106.3658959

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 20 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Findingown data
    § 3.4 Public availability of audit reports · p. 7

    Despite LL 144's emphasis on transparency, very few audits were publicly available in the first six months of final rule implementation.

    “Despite the law's emphasis on transparency, very few audits are available publicly in the first six months of final rule implementation.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 4.3 Disagreement over what constitutes a legitimate auditor and legitimate auditing practice · p. 10

    Some auditors explicitly rejected the term 'auditor' to describe their own services.

    “This is why some auditors, like P5 and P6, explicitly rejected the term 'auditor' to describe their own services.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § Auditing Work: Exploring the New York City algorithmic bias audit regime · p. 1

    There is wide disagreement over what constitutes a legitimate auditor in the LL 144 regime.

    “Fourth, we find wide disagreement over what constitutes a legitimate auditor and identify four different kinds of 'auditor roles' that serve different functions and offer different kinds of services.”
    • Also stated in: § 4.3 Disagreement over what constitutes a legitimate auditor and legitimate auditing practice · p. 9 · plain
      “Participants expressed different views on what practices or qualities made them legitimate auditors under LL 144.”
    Quote in the paperSubstantial claim
  • Findingnone given
    § 1 INTRODUCTION · p. 2

    The study's core finding is that the work of algorithm auditing in the LL 144 regime is largely about managing relational dynamics between stakeholders established by the law's accountability structures.

    “The core finding of our study is that the work of algorithm auditing in this regime is largely about managing the relational dynamics between stakeholders established by the accountability structures in the law.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 4.3 Disagreement over what constitutes a legitimate auditor and legitimate auditing practice · p. 9

    Participants expressed different views on what practices or qualities made them legitimate auditors under LL 144.

    “Participants expressed different views on what practices or qualities made them legitimate auditors under LL 144.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 51
  • May combine two claims. 58
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 6

    The four-fifths rule is a bureaucratic compromise that is fundamentally arbitrary and not grounded in empirical evidence, and so may not be useful for judging algorithmic bias.

    “As critics point out, the four-fifths rule is a bureaucratic compromise that is fundamentally arbitrary and not grounded in empirical evidence, and so may not be useful for judging algorithmic bias [15, 49].”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [15] [15] Equal Employment Opportunity Commission. 2023. Assessing Adverse Impact in Software, Algorithms, and Artificial Intelligence Used in Employment Selection Procedures Under Title VII of the Civil Rights Act of 1964. (2023). https://www.eeoc.gov/laws/guidance/select-issuesassessing-adverse-impact-software-algorithms-and-artificial -

      Not checked in the free run.

    • cites [49] [49] Elizabeth Anne Watkins, Michael McKenna, and Jiahao Chen. 2022. The four-fifths rule is not disparate impact: a woeful tale of epistemic trespassing in algorithmic fairness. arXiv preprint arXiv:2202.09519 (2022). source

  • May combine two claims.

    Premisecitation
    § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 5

    The prescribed audit under LL 144 is more accurately described as a 'disparate impact' audit, as it measures a specific form of algorithmic bias.

    “The prescribed audit is more accurately described as a 'disparate impact' audit[3, 4, 20], as it measures a specific form of algorithmic bias [31].”
    Quote in the paperNot a substantial claim1 of 4 sources checked
    • cites [3] [3] Ifeoma Ajunwa, Sorelle Friedler, Carlos E Scheidegger, and Suresh Venkatasubramanian. 2016. Hiring by algorithm: predicting and preventing disparate impact. Available at SSRN (2016). -

      Not checked in the free run.

    • cites [4] [4] Solon Barocas and Andrew D Selbst. 2016. Big data's disparate impact. California law review (2016), 671-732. - source
    • cites [20] [20] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining . 259-268. - source
    • cites [31] [31] New York City Department of Consumer and Worker Protections. 2023. Automated Employment Decision Tools: Frequently Asked Questions. (2023). https://www.nyc.gov/assets/dca/downloads/pdf/about/DCWP-AEDT-FAQ.pdf -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.3 Disagreement over what constitutes a legitimate auditor and legitimate auditing practice · p. 9

    The algorithmic auditing ecosystem is nascent and currently lacks the structure and standards of other audit domains, such as financial services.

    “The algorithmic auditing ecosystem is nascent and currently lacks the structure and standards of other audit domains, such as financial services [42].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [42] [42] Inioluwa Deborah Raji, Peggy Xu, Colleen Honigsberg, and Daniel Ho. 2022. Outsider oversight: Designing a third party audit ecosystem for ai governance. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society . 557-571. - source

  • May combine two claims.

    Findingcitation
    § 3.4 Public availability of audit reports · p. 7

    Surveys indicate AEDTs of all varieties are thought to be widespread, but there is no available data about the baseline rate of adoption of AEDTs generally, let alone systems within LL 144's scope.

    “Surveys indicates that AEDTs of all varieties are thought to be widespread, but there is no available data about the baseline rate of adoption of AEDTs generally, let alone adoption of systems that fall within the scope of LL 144 [9, 24, 46].”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [9] [9] PewResearch Center. 2023. AI in Hiring and Evaluation of Workers: What People Think. (2023). https://www.pewresearch.org/internet/2023/04/20/aiin-hiring-and-evaluating-workers-what-americans-think/ -

      Not checked in the free run.

    • cites [24] [24] Indeed. 2023. The Indeed Global AI Survey: Your Guide to the Future of Hiring. (2023). https://www.indeed.com/lead/the-indeed-ai-report?hl=en# form -

      Not checked in the free run.

    • cites [46] [46] Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the conference on fairness, accountability, and transparency . 59-68. - source

      Section 4 (Takeaways; page unavailable) mentions an algorithm designed to predict good hires only as a hypothetical example. Section 3.4 (Ripple Effect Trap; page unavailable) discusses a risk-assessment questionnaire, not a survey of AEDT adoption. Searches for AEDT adoption rates, employer surveys, and LL 144 found no passage addressing the claimed prevalence or lack of baseline data. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § 3.3 Analysis of the public comments · p. 6

    Of the 46 public comments analyzed, there were 13 civil society organisations, four audit service firms, six employment law firms, seven system developers, and three employment platforms.

    “Of the 46 total comments, we identified 13 civil society organisations (including some authors of this paper), four audit service firms, six employment law firms, seven system developers, and three employment platforms 5 .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.1 Opinions of the law overall were that it was well-intentioned but seriously flawed · p. 8

    Participants with knowledge of the law-drafting process pointed to lobbying efforts by both AI vendors and large employers to water down the definition.

    “Participants with knowledge of the law-drafting process pointed to lobbying efforts by both AI vendors and large employers to water down the definition.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 4.1 Opinions of the law overall were that it was well-intentioned but seriously flawed · p. 8

    Some auditors did not like the law's theory of change and expressed the view that it does not do enough to protect job applicants because audits and notices will not enable them to make an informed decision about whether to be subject to an AEDT or human reviewer, or whether to apply.

    “Some auditors did not like the law's theory of change. The expressed a view that it does not do enough to protect job applicants because audits and notices to applicants, while important, will not enable them to make an informed decision about whether to be subject to an AEDT or a human reviewer, or about whether to apply for the job at all [P11, P13].”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.3 Analysis of the public comments · p. 7

    Civil society commenters frequently called for removal of the 'substantially assist' criterion, arguing it would allow employers to declare themselves out of scope, while industry commenters stated it should remain, which it did.

    “Civil society commenters frequently called for the removal of the 'substantially assist' criterion, arguing it would allow employers to declare themselves out of scope-effectively nullifying the law. In contrast, industry commenters stated that it should remain, which it did.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.3 Disagreement over what constitutes a legitimate auditor and legitimate auditing practice · p. 10

    Some interview subjects felt that offering additional advisory services was not in opposition to the law's definition of being an independent auditor.

    “some felt offering additional advisory services was not in opposition to the law's definition of being an independent auditor [P3].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.3 Disagreement over what constitutes a legitimate auditor and legitimate auditing practice · p. 9

    Several participants suggested that more flexibility about independence would increase the effectiveness of the bias audit.

    “Several participants suggested that more flexibility about independence would increase the effectiveness of the bias audit: 'I needed [to work closely with] our clients to have better data governance practices in order for me to be able to do a better audit for them' [P14].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [P14]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data
    § 3.3 Analysis of the public comments · p. 6

    The study's analysis of public comments found disagreements primarily about the scope of the law and the positionality or independence of auditors.

    “Our analysis revealed disagreements primarily about the scope of the law (which systems would be subject to it) and the positionality of auditors (does the legitimacy of audits require the independence of auditors?).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.3 Analysis of the public comments · p. 7

    In the public comments, hiring platforms and AEDT vendors generally expressed concern that the law was too broad and onerous, while civil society argued the reverse, saying industry actors may try to skirt regulation.

    “Generally, both hiring platforms and AEDT vendors expressed concern about the law being too broad and onerous, with civil society stating the reverse, arguing industry actors may try to skirt regulation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.3 Analysis of the public comments · p. 7

    The study found evidence that AEDT vendors lobbied the government for narrower definitions, language allowing exceptions, and higher control over the audit process.

    “We found evidence that AEDT vendors lobbied the government for narrower definitions, language that allowed for exceptions, and higher degrees of control over the audit process.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given
    § 3.4 Public availability of audit reports · p. 7

    Under LL 144, employers have almost-total discretion to judge whether their systems are in scope, and jobseekers have no formal mechanism to challenge that decision.

    “Under LL 144, employers have almost-total discretion to make a judgement on whether their systems are in scope for multiple reasons, and jobseekers have no formal mechanism to challenge that decision.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.3 Analysis of the public comments · p. 6

    Labour rights advocacy organisations, including labour unions, were not well represented in the public comments, and only a few comments specifically detailed consequences for job seekers.

    “Labour rights advocacy organisations, including labour unions, were not well represented, with the bulk of the civil society comments focusing on issues of algorithmic accountability. Only a few comments specifically detailed the consequences for job seekers of the details in the law.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Opinions of the law overall were that it was well-intentioned but seriously flawed · p. 8

    Most auditors interviewed had a positive opinion about the spirit of LL 144 but agreed it was not a particularly effective auditing mechanism.

    “While most auditors we interviewed had a positive opinion about the spirit of the law, they also agreed that it was not a particularly effective auditing mechanism.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Auditing Work: Exploring the New York City algorithmic bias audit regime · p. 1

    LL 144 fails to clearly define key aspects like AEDTs and what constitutes an independent auditor, leaving auditors, vendors, and employers to define practical implementation in ways that failed to protect job applicants.

    “the law fails to clearly define key aspects like AEDTs and what constitutes an independent auditor, leaving auditors, vendors who create AEDTs, and companies using AEDTs to define the law's practical implementation in ways that failed to protect job applicants.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.3 Disagreement over what constitutes a legitimate auditor and legitimate auditing practice · p. 10

    Other interview subjects felt that a core component of being an independent auditor was that they could not provide advice or guidance on how to mitigate issues.

    “But others, like P5 and P6, felt a core component of being an independent auditor was that they could not provide advice or guidance on how to mitigate these issues.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.1 Opinions of the law overall were that it was well-intentioned but seriously flawed · p. 8

    Many auditors noted a key challenge with the legislation is that it is too open for interpretation when it comes to the scope of the systems.

    “Many auditors [P2, P5, P14] noted a key challenge with the legislation is that 'it's too open for interpretation when it comes to the scope of the systems' [P2].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.3 Disagreement over what constitutes a legitimate auditor and legitimate auditing practice · p. 9

    Two participants felt that the financial incentive in providing a bias audit under LL 144 might create challenges for independence.

    “Two participants felt that the financial incentive in providing a bias audit under LL 144 might create challenges for independence [P1] [P12].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [P1]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [P12]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 4.1 Opinions of the law overall were that it was well-intentioned but seriously flawed · p. 8

    Participants gave strong opinions on employers' discretion to decide if systems meet the 'substantially assist' definition, with one saying the AEDT definition 'creates huge loopholes'.

    “Participants gave strong opinions on employers' discretion to decide if systems meet the 'substantially assist' definition (see 2.3): '[the AEDT definition] creates huge loopholes'.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.4 Public availability of audit reports · p. 7

    The authors identified 19 audits claiming to be related to LL 144, including five from system developers or vendors not subject to audit under the law; some of these audits are no longer available online.

    “We searched for available audits using ad hoc search methods prior to conducting interviews, we identified 19 audits claiming to be related to LL 144, including five from system developers/vendors not subject to audit under the law. Some of these audits are no longer available online.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 12

    Participant P14 noted that the test required using the most favoured group as the reference group and any group that scores less favourably as protected groups, which conflicted with historical ways of doing disparate impact analyses where the protected group is pre-selected before viewing the data.

    “P14 noted that this test required using 'the most favoured group as the reference group, and any group that scores less favourably, those are your protected groups,' conflicted with 'historical' ways of doing disparate impact analyses where the protected group is pre-selected, before viewing the data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.4 LL 144 created an opportunity for more companies to adopt wider responsible AI practices · p. 10

    The creation of a document enabled the wider staff to understand that it is appropriate for them to have a say in how AI is developed and applied in their company.

    “The creation of a document enabled the wider staff to understand 'that it's absolutely appropriate for them to have a say in how AI is developed and applied in their company' [P5].”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data
    § 5.2 Auditing laws need clear and meaningful metrics and definitions of key terms · p. 13

    According to participants, a major source of NYC LL 144's failure is the last-minute addition of language around AEDTs that 'substantially assist', which drastically reduced the number of companies in scope.

    “According to our participants, a major source of NYC LL 144's failure is the last-minute addition of language around AEDTs that 'substantially assist', which drastically reduced the number of companies in scope.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.4 LL 144 created an opportunity for more companies to adopt wider responsible AI practices · p. 10

    Participant P16 offers legal advice along with the audit.

    “Others, like P16, noted that they offer legal advice along with the audit.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 11

    Participant P14 described the audit process as similar to the stages of grief, with stages of denial, despair, and finally acceptance from the client.

    “P14 described how the process of conducting the audit was similar to 'the stages of grief.' They reported stages of denial ('oh, no, we must have given you the wrong data or something like that'), despair ('oh, no, like, this is the worst thing in the world that could happen to us'), and finally acceptance from the client.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 4.4 LL 144 created an opportunity for more companies to adopt wider responsible AI practices · p. 10

    According to P14, the audits ensured that companies had a single AI system document and that their role as an auditor was to provide documentation and a 'grounded language' around responsible AI.

    “According to P14, the audits ensured that companies had a single AI system document and that their role as an auditor was to provide documentation and a 'grounded language' around responsible AI.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given
    § 5.4 Auditing laws must enable smoother data collection for performing tests and providing access to data for certified auditors · p. 14

    One prevailing approach relies on auditors fostering positive relationships with a system's developers, emphasising concerns that evaluations often hinge on selective disclosure, potentially favouring auditors who portray a positive image of the system.

    “One prevailing approach relies on auditors fostering positive relationships with a 14 system's developers, emphasising concerns that evaluations often hinge on selective disclosure, potentially favouring auditors who portray a positive image of the system.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.4 LL 144 created an opportunity for more companies to adopt wider responsible AI practices · p. 10

    Participants P1, P5, P6, P7, and P8 are advising clients on how to get compliance-ready for other possibly forthcoming laws.

    “Others, like P1, P5, P6, P7, and P8, noted they are advising clients on how to get compliance-ready for other possibly forthcoming laws.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given
    § 6 CONCLUSION · p. 14

    Auditors are challenged by navigating the accountability relationships created by the law, including negotiating system access from developers and with legal counsel about whether a system is in scope.

    “Auditors are challenged by navigating the accountability relationships created by the law, including negotiating system access from developers and with legal counsel about whether a system is in scope.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 11

    Participant P2 noted that some clients were frugal with their data but would provide full access after initial results indicated an issue.

    “P2 noted that some clients were 'frugal' with their data but would provide full access after initial results indicated an issue.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 5.3 Establishing an auditing ecosystem requires more standards for and oversight of auditors · p. 13

    These include companies that provide the service of making companies 'audit ready' or advising them on steps they can take to improve their data and responsible AI practices.

    “These include companies that provide the service of making companies 'audit ready' or advising them on steps they can take to improve their data and responsible AI practices.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 11

    Participant P8 described clients who didn't collect data on ethnicity or gender because it's sensitive, and then incorrectly claimed that this lack of data collection meant that bias was impossible.

    “This sense of denial matched the perspective of P8, who described clients who didn't collect data on ethnicity or gender 'because it's sensitive,' and then incorrectly claimed that this lack of data collection meant that bias was impossible.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 11

    Auditors often use tools like APIs or databases that allow controlled access to limited, but relevant, data.

    “Often, auditors use tools like APIs or databases that allow controlled access to limited, but relevant, data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.4 LL 144 created an opportunity for more companies to adopt wider responsible AI practices · p. 10

    Participants P5 and P6 noted that this kind of work was more lucrative than just helping companies conduct an audit.

    “P5 and P6 noted this kind of work was more lucrative than just helping companies conduct an audit.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 11

    Participant P14 noted that vendor cooperation was key and one of the biggest obstacles to clients complying with the NYC law.

    “P14 noted that vendor cooperation was 'key' and 'one of the biggest obstacles to clients complying with the NYC law.'”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 11

    The guidance almost does not talk about sampling data, making it easy to cherry-pick or bias audit results.

    “This included guidance on how to select data for the audit; according to P14, '[the guidance] almost doesn't talk about [sampling data].. So I mean, if we wanted to be in bad faith, we could look at our clients dataset, and we could very easily pick rows that would give stellar results.'”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.3 Establishing an auditing ecosystem requires more standards for and oversight of auditors · p. 13

    Auditors are largely left to their own devices in defining 'good' practice in this space, and interviewees suggested that the DCWP may be ill-equipped to enforce the law.

    “Auditors are largely left to their own devices in defining 'good' practice in this space, and interviewees suggested that the DCWP may be ill-equipped to enforce the law.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 11

    Many clients and vendors were initially resistant to acknowledging biases within their systems, making it difficult for auditors to help them understand their potential biases and potential mitigation strategies.

    “Many clients and vendors were initially resistant to acknowledging biases within their systems, making it difficult for auditors to help them understand their potential biases and potential mitigation strategies.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data
    § 4.4 LL 144 created an opportunity for more companies to adopt wider responsible AI practices · p. 10

    Participant P14 noted that the law, while highly imperfect, had the effect of getting their clients to think about responsible AI and embed new language and ways of thinking into their corporate culture.

    “P14 noted that the law, while highly imperfect, had the effect of getting their clients to think about responsible AI and embed new language and ways of thinking into their corporate culture.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 11

    While explicitly prohibited under LL 144 regulations, the practice of imputing or inferring demographic data persists, resulting in auditors often using names to infer demographic information like race, ethnicity, and gender.

    “While explicitly prohibited under LL 144 regulations, auditors noted that this practice persists, resulting in auditors often using names to infer demographic information like race, ethnicity, and gender.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.4 LL 144 created an opportunity for more companies to adopt wider responsible AI practices · p. 10

    Several participants noted that LL 144 created an incentive for companies and vendors to adopt wider responsible AI and data ethics practices, enabling participants to offer additional responsible AI services.

    “Several participants noted that LL 144 created an incentive for companies and vendors to adopt wider responsible AI and data ethics practices, enabling participants to offer additional responsible AI services.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 12

    This conflict caused the statistical validity of the test to go 'out the window' and enabled 'cherry picking' of data to tell a story that the system was not biased.

    “As a result, this caused the statistical validity of the test to go 'out the window' and enabled 'cherry picking' of data to tell a story that the system was not biased.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 11

    The guidance from DCWP has a lot of discretion and flexibility and could be interpreted by different firms and vendors in different ways.

    “As P14 noted, 'the guidance still has a lot of discretion and flexibility, ' and could be interpreted by different firms and vendors in different ways.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 12

    Participant P10 noted the metric offered by DCWP changed part way through the implementation of the law from the average to the median, which caused delays and additional work.

    “P10 noted the metric offered by DCWP changed part way through the implementation of the law from the average to the median, which caused delays and additional work.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 5

    Because 'assisting' is a feature of the human and organisational aspects of the hiring process, not of the technical system, two employers could use the same model from the same vendor but have different interpretations of whether the output meets LL 144's definition of 'substantial.'

    “However, 'assisting' is a feature of the human and organisational aspects of the hiring process, and not of the technical system. Because of this, two employers could use the same model from the same vendor, but have different interpretations of whether the output meets LL 144's definition of 'substantial.'”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning
    § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 6

    Absent a mandate to remediate discriminatory outcomes from LL 144, auditors must find secondary avenues to achieve their goals and provide valuable services to clients.

    “Absent a mandate to remediate discriminatory outcomes from LL 144, auditors must find secondary avenues to achieve their goals and provide valuable services to clients.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § 1 INTRODUCTION · p. 2

    LL 144's theory of change assumes that providing job-seekers insight into algorithmic systems will lead the public to make better informed decisions and push vendors and employers to create fair(er) systems.

    “Additionally, by providing the public-in this case job-seekers-with insight into the algorithmic systems making impactful decisions about their lives, the public will make better informed decisions and push the vendors building these tools and the companies using them to create fair(er) systems.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning
    § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 5

    The scope of LL 144 is de facto determined by the meaning of 'substantially assist' because no system on the market fully replaces human decision-making in hiring.

    “To the authors' best knowledge, there is no system on the market that fully replaces human decision-making in hiring, and therefore the scope of the law is de facto determined by the meaning of 'substantially assist.'”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: There is no system on the market that fully replaces human decision-making in hiring.

  • May combine two claims.

    Premisecitation
    § Auditing Work: Exploring the New York City algorithmic bias audit regime · p. 1

    LL 144 requires NYC-based employers using AEDTs in hiring or promotion to conduct annual bias audits by independent auditors and to post the resulting audit reports publicly.

    “Local Law 144 (LL 144), requires NYC-based employers using automated employment decision-making tools (AEDTs) in hiring to be subject to annual bias audits by an independent auditor.”
    • Also stated in: § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 4 · plain
      “LL 144 requires only employers/hiring agencies that use the system to conduct a bias audit, but not the developer/vendor/platform that builds or sells the system [31] 1 .”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [31] [31] New York City Department of Consumer and Worker Protections. 2023. Automated Employment Decision Tools: Frequently Asked Questions. (2023). https://www.nyc.gov/assets/dca/downloads/pdf/about/DCWP-AEDT-FAQ.pdf -

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 6

    LL 144 sets no minimum permissible impact ratio and imposes no obligation to cease using a system that an audit discovers to cause disparate impact; only public posting of the audit is required.

    “Most crucially for auditors, LL 144 makes use of conventional measures of bias, yet sets no minimum for permissible impact ratio rates. The law imposes no obligations to cease using any system that an audit discovers to cause disparate impact[31]. All that is required is that the employer make the annual audit publicly available.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [31] [31] New York City Department of Consumer and Worker Protections. 2023. Automated Employment Decision Tools: Frequently Asked Questions. (2023). https://www.nyc.gov/assets/dca/downloads/pdf/about/DCWP-AEDT-FAQ.pdf -

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning
    § 4.2 Auditing regimes create a variety of auditing roles and services · p. 9

    The lack of clear roles, responsibilities, and services around the law furthers the need for additional relational work to comply with the law, and interpretation and implementation are left to auditors and clients, resulting in fractured roles and responsibilities.

    “The lack of clear roles, responsibilities, and services around the law furthers the need for additional relational work to comply with the law. Interpretation and implementation are left to auditors and their clients, resulting in fractured roles and responsibilities.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 6

    While an algorithm auditor might conceive of their role as helping clients reduce discriminatory outcomes and protect civil rights, their mandated role under LL 144 is only the production of audit reports.

    “While an algorithm auditor might conceive of their role as helping clients reduce discriminatory outcomes and protect civil rights-via services such as consulting, model design advice, governance practices, legal guidance, etc.-their mandated role is only the production of audit reports.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Study design and conductown data
    § 2.2 Data analysis · p. 3

    The study used grounded theory epistemology and an inductive coding process with four authors, an initial codebook of 102 codes synthesized to 16, and each interview coded by two separate authors.

    “For our data analysis, we adopted a grounded theory epistemology, allowing us to surface latent themes, patterns, and social interactions from our data [10]. Four authors contributed to the coding process using an inductive approach, allowing us to derive codes based on concepts from the data. We used Atlas.ti data analysis software to support the process. We completed a round of asynchronous”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10] [10] Kathy Charmaz. 2023. Constructing grounded theory: A practical guide through qualitative analysis . sage. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 6

    The authors of LL 144 diverged from existing norms in US employment law when determining what action a company must take upon discovering algorithmic bias.

    “The authors of LL 144 diverged from existing norms in US employment law when determining what action a company must take upon discovering algorithmic bias.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 5

    LL 144 defines race and gender using EEOC categories and excludes other protected categories such as disability, age, and religious status from the audit requirement.

    “LL 144 defines race and gender using categories established by the EEOC [14]. Other protected categories central to AI fairness scholarship and employment civil rights law are not included in the audit requirement, such as disability, age and religious status.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [14] [14] Equal Employment Opportunity Commission. 2022. Definitions of Race and Ethnicity Categories. Data Collection (2022). -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Premisecitation
    § 4.3 Disagreement over what constitutes a legitimate auditor and legitimate auditing practice · p. 10

    The US Sarbanes Oxley Act 2002 strictly prohibits independent auditors from providing certain non-audit services to clients.

    “two participants [P10, P1] highlighted that the US Sarbanes Oxley Act 2002 strictly prohibits independent auditors from providing 'certain non-audit services' to clients [17].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] [17] US Congress. 2002. The Sarbanes Oxley Act. (2002). -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § 3.4 Public availability of audit reports · p. 7

    Given the many thousands of employers in NYC and widespread use of AEDTs, one would reasonably anticipate that more than a handful of employers would fall under the scope of LL 144.

    “However, given the many thousands of employers in NYC and widespread use of AEDTs, one would reasonably anticipate that more than a handful of employers would fall under scope of this law.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.4 Public availability of audit reports · p. 7

    The absence of an audit on an employer's website does not imply absence of an AEDT, non-compliance, or lack of audit.

    “The absence of an audit on an employer's website neither implies the absence of an AEDT (as they may use an AEDT self-determined as out of scope), nor non-compliance (they may use a genuinely out of scope AEDT or none at all), nor even a lack of audit (the process may have been conducted, but the report is not published).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data
    § 4.3 Disagreement over what constitutes a legitimate auditor and legitimate auditing practice · p. 10

    What exactly constitutes a legitimate auditor remains vague and undefined within LL 144 itself, other than the minimal requirement of financial independence from the outcome of the audit report.

    “Like other elements of the law, what exactly constitutes a legitimate auditor remains vague and undefined within the law itself, other than the minimal requirement of financial independence from the outcome of the audit report.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 2.3 Limitation of study · p. 3

    The study's limitation is a narrow geographical focus on implementation of LL 144 and its relationship to US federal employment and civil rights law.

    “This paper reports on empirical research on the implementation of LL 144, and its relationship to United States federal employment law and civil rights law. This gives our research a narrow geographical focus.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.4 Public availability of audit reports · p. 7

    Because there is no central repository and no baseline adoption data, it is not possible to know precisely how many audits are expected or published.

    “Therefore, it is not possible to know precisely how many audits we should expect to find, and lacking a central repository, it is also impossible to know precisely how many audits are published.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 1 INTRODUCTION · p. 2

    LL 144's theory of change assumes that incentivising a market for assessment and transparency documentation will lead assessors to propagate best practices and protect civil rights even without strict regulatory limits on technical system design.

    “The theory of change animating LL 144 is an assumption common in policymaking: by incentivising and supporting a market for assessment and transparency documentation, assessors will be able to propagate best practices and protect civil rights even without strict regulatory limits on the design of technical systems.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning
    § 1 INTRODUCTION · p. 1

    LL 144 requires employers to provide job-seekers a transparency notice about their use of AEDTs and the right to opt out of analysis by the algorithmic system in favour of a human decision process.

    “Additionally, the law requires that those employers provide job-seekers with a transparency notice regarding their use of AEDTs and the right to opt-out of analysis by the algorithmic system in favour of a human decision process.”
    • Also stated in: § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 5 · plain
      “Transparency notice about use of AEDT and right to opt-out for candidates.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 5

    LL 144 defines independent auditors as third-party experts who have no financial stake in the success of the product or the financial outcome of the employer.

    “The law defines independent auditors as third-party experts who have no financial stake in the success of the product or the financial outcome of the employer.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 5

    The law defines AEDTs so that many (perhaps most) systems one might identify as employment decision tools available on the market and offered by recruitment platforms are out of scope, and grants employers significant discretion to decide if their systems are in scope.

    “The law defines AEDTs so that many (perhaps most) of systems one might identify as 'employment decision tools' available on the market and offered by recruitment platforms are out of scope; it also grants employers significant discretion to decide if their systems are in scope.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 2.3 Limitation of study · p. 3

    The authors believe there are generalisable lessons for other jurisdictions/domains, but some findings (e.g., improvements in the law) are more applicable to the United States context.

    “We believe that there are generalisable lessons emerging from our findings that will be of utility to other jurisdictions/domains, but we also recognise that some of the findings (e.g. improvements in the law) are more applicable to the United States context (other nations do not, for example, use the four-fifths rule to measure disparate impact).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning
    § 3.2 Key aspects of the law and its context in US anti-discrimination law · p. 5

    LL 144's definition of machine learning/AI narrows the scope to deep-learning methods and excludes even intentionally discriminatory weights that are set manually.

    “Machine learning/artificial intelligence is defined in terms of outputting a simplified score or ranking on the basis of multiple feature inputs, in which the machine in part algorithmically weights those features. This narrows the scope further to deep-learning methods, and excludes even intentionally discriminatory weights that are set manually.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 11

    Instances of imputation/inferral are particularly problematic in diverse urban settings like NYC and raise questions about the validity of audits conducted under LL 144.

    “These instances of imputation/inferral are particularly problematic in diverse urban settings like NYC and raise questions about the validity of audits conducted under LL 144.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 5.3 Establishing an auditing ecosystem requires more standards for and oversight of auditors · p. 13

    Even in good faith, this leaves the door open for inconsistencies in practice and prevents oversight that might enable ecosystem-level learning.

    “Even in good faith, this leaves the door open for inconsistencies in practice and prevents oversight that might enable ecosystem-level learning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 5.1 Auditing laws must have a theory of change that holds developers accountable, makes audits the source of action/enforceability and prioritises jobseekers · p. 12

    At present, the LL 144 auditing regime is insufficient for creating accountability relationships between jobseekers and deployers or vendors because the lack of prescribed mitigation steps in the event of adverse impact means a candidate would need to rely on plaintiff litigation for further action.

    “At present, the LL 144 auditing regime is insufficient for creating accountability relationships between jobseekers and deployers or vendors: the lack of prescribed mitigation steps in the event of adverse impact means a candidate would need to rely on plaintiff litigation for further action.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 6 CONCLUSION · p. 14

    While the law specifies what the audit should include, it fails to specify the role of the auditor, creating confusion about the appropriate credentials and practices of independent auditors.

    “While the law specifies what the audit should include, it fails to specify the role of the auditor, creating confusion about the appropriate credentials and practices of independent auditors.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § 5.2 Auditing laws need clear and meaningful metrics and definitions of key terms · p. 13

    The statistical metric used by NYC LL 144 (disparate impact, based on the four-fifths rule) is heavily rooted in US law and is not applicable to other jurisdictions outside of the USA.

    “Additionally, the statistical metric used by the NYC LL 144 law (disparate impact, based on the four-fifths rule) is heavily rooted in US law and is not applicable to other jurisdictions outside of the USA.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 5.1 Auditing laws must have a theory of change that holds developers accountable, makes audits the source of action/enforceability and prioritises jobseekers · p. 12

    The law forms a relational regime for industry actors to navigate, with employers who use AEDTs navigating between their responsibility to LL 144, the potential censure of the EEOC, and the social cost of publishing a non-favourable bias audit.

    “Consequently, the law forms a relational regime for industry actors to navigate - employers who use AEDTs must navigate between their responsibility to LL 144, the potential censure of the EEOC, and the social cost of publishing a non-favourable bias audit.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4.5 Auditors faced numerous obstacles around data practices, metrics and cultural attitudes toward data with vendors/clients · p. 11

    LL 144 only imposes obligations on the end-user, the employer, creating a relational dynamic between employer and vendor with auditors mediating in the middle.

    “LL 144 only imposes obligations on the end-user, the employer, creating a relational dynamic between employer and vendor (with auditors mediating in the middle).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data
    § 5.3 Establishing an auditing ecosystem requires more standards for and oversight of auditors · p. 13

    Our interviews suggest these roles are essential.

    “Our interviews suggest these roles are essential, but they must be clearly defined and differentiated in an auditing regime.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 5.4 Auditing laws must enable smoother data collection for performing tests and providing access to data for certified auditors · p. 14

    Consequently, auditors frequently find themselves obligated to declare their intent for data usage, introducing uncertainty and potential bias into the auditing process.

    “Consequently, auditors frequently find themselves obligated to declare their intent for data usage, introducing uncertainty and potential bias into the auditing process.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 5.4 Auditing laws must enable smoother data collection for performing tests and providing access to data for certified auditors · p. 14

    The issue of access is further complicated by a lack of trust in companies releasing only favourable data, a challenge exacerbated by the fact that APIs are often developed and managed by large, for-profit entities.

    “The issue of access is further complicated by a lack of trust in companies releasing only favourable data, a challenge exacerbated by the fact that APIs are often developed and managed by large, for-profit entities.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 5.1 Auditing laws must have a theory of change that holds developers accountable, makes audits the source of action/enforceability and prioritises jobseekers · p. 12

    There is no requirement for the vendor to take action; an employer would need to convince vendors to improve the fairness outcomes of a discriminatory system.

    “Additionally, there is no requirement for the vendor to take action; an employer would need to convince vendors to improve the fairness outcomes of a discriminatory system.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4.3 Disagreement over what constitutes a legitimate auditor and legitimate auditing practice · p. 10

    The responsibility to define auditor roles falls to employers and auditors themselves.

    “Therefore, the responsibility to define auditor roles falls to employers and auditors themselves.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 5.1 Auditing laws must have a theory of change that holds developers accountable, makes audits the source of action/enforceability and prioritises jobseekers · p. 12

    The lack of audits and notices, and the poor readability/interpretability of those that do exist, means job candidates may not directly benefit.

    “In practice, however, the lack of audits and notices (and the poor readability/interpretability of those that do exist) means job candidates may not directly benefit.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § 6 CONCLUSION · p. 14

    The law grants the end-user total discretion over whether their system is in scope, and offers multiple loopholes for employers to move out of scope without addressing discriminatory outcomes.

    “In addition, the law grants the end-user total discretion over whether their system is in scope, and offers multiple loopholes for employers to move out of scope without addressing discriminatory outcomes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § 5.4 Auditing laws must enable smoother data collection for performing tests and providing access to data for certified auditors · p. 14

    Platforms such as Reddit have introduced charges for API access, while others have shut down their APIs altogether.

    “Notably, platforms such as Reddit have introduced charges for API access, while others have shut down their APIs altogether.”
    Quote in the paperNot a substantial claim
84 of 84

  • The paper relies on this without support.

    There is no system on the market that fully replaces human decision-making in hiring.

    “To the authors' best knowledge, there is no system on the market that fully replaces human decision-making in hiring, and therefore the scope of the law is de facto determined by the meaning of 'substantially assist.'”

  • The paper relies on this without support.

    To the authors' knowledge, this is the sole study exploring the LL 144 regime through the lens of auditors and one of the first attempts to use a case study of a bias auditing regime to discern lessons for other attempts.

    “To our knowledge, this is the sole study exploring the LL 144 regime through the lens of auditors and one of the first attempts to use a case study of a bias auditing regime to discern lessons for other attempts.”

  • The paper relies on this without support.

    The interview participants' accounts accurately reflect the implementation of LL 144.

    “Our interview findings and analysis of the law lead us to conclude that despite some modest successes in socialising bias audits among private companies, the law has fallen far short of its ambition to generate an effective bias audit regime that protects job applicants from discriminatory algorithmic tools.”

  • The paper relies on this without support.

    The publicly posted audits found by the authors are representative of all LL 144 audits conducted.

    “Of the thousands of employers in NYC, we could find only 19 who have publicly posted audits.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Multiple audit service providers further indicated that they had clients who paid for an audit for an in-scope system but declined to post it publicly.

    “Multiple audit service providers further indicated that they had clients who paid for an audit for an in-scope system but declined to post it publicly.”

    From Null Compliance: NYC Local Law 144 and the challenges of algorithm accountability (2024) · cites this paper as [28]

    Section 3.4 (p. 7) says an audit may have been conducted without its report being published. It does not say that multiple audit providers reported clients paying for audits of in-scope systems and then declining to publish them. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Several tools surfaced in the survey (e.g., from Holistic.ai, Credo.ai) were explicitly marketed for use for NYC Local Law 144 compliance, but studies of audit practice suggest the produced measures may not be reliable or legally compatible.

    “For instance, several tools surfaced in our survey (e.g., from Holistic.ai, Credo.ai) were explicitly marketed for use for NYC Local Law 144 compliance, but studies of audit practice suggest the produced measures may not be reliable or legally compatible [30, 84, 86].”

    From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [30]

    Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Analyses of the early audit regime highlight concerns about definitional scope, methodological variability, and the risk that compliance may prioritize minimum reporting over substantive fairness improvement.

    “analyses of the early audit regime highlight concerns about definitional scope, methodological variability, and the risk that compliance may prioritize minimum reporting over substantive fairness improvement”

    From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Groves et al., 2024

  • Early analyses of audit disclosures suggest variability in rigor, method choice, and interpretability, raising questions about whether compliance-only audits are sufficient to meaningfully reduce disparate outcomes in practice.

    “early analyses of audit disclosures suggest variability in rigor, method choice, and interpretability, raising questions about whether compliance-only audits are sufficient to meaningfully reduce disparate outcomes in practice”

    From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Groves et al., 2024

  • In a companion study, audit industry workers indicated in interviews that, in their experience, many, if not the majority, of AEDTs on the market violate the four-fifths rule.

    “In a companion study to this one [28], audit industry workers indicated in interviews that, in their experience, many, if not the majority, of AEDTs on the market violate the four-fifths rule.”

    From Null Compliance: NYC Local Law 144 and the challenges of algorithm accountability (2024) · cites this paper as [28]

    Section 3.2 (p. 6) explains the four-fifths rule, and section 4.1 (p. 8) reports auditors’ interview opinions about the law. Neither reports interviewees saying that many or most AEDTs on the market violate the rule. Searches for that estimate and for tools failing the four-fifths threshold found no supporting passage in our copy. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Scope alternates between city agencies and commercial systems only

    Minor

    The introduction says the law mandates audits for city agencies, whereas the history section says the effort ended by regulating only commercial algorithmic hiring systems.

  • Vendor audits are counted as employer audits

    Minor

    The paper says it found 19 audits, including five from developers or vendors not subject to the law. It later describes all 19 as audits publicly posted by employers, overstating the employer count that supports its compliance conclusion.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 12

  1. 2013. ISAE 3000 Revised for IAASB. International Auditing and Assurance Standards Board (2013). https://www.iaasb.org/publications/international-standard-assurance-engagements-isae-3000-revised-assurance-engagements-other-audits-or
  2. Solon Barocas and Andrew D Selbst. 2016. Big data’s disparate impact. California law review (2016), 671–732.
  3. Abeba Birhane, Ryan Steed, Victor Ojewale, Briana Vecchione, and Inioluwa Deborah Raji. 2024. AI auditing: The broken bus on the road to AI accountability. arXiv preprint arXiv:2401.14462 (2024). · arXiv
  4. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77–91.
  5. Equal Employment Opportunity Commission. 2022. Definitions of Race and Ethnicity Categories. Data Collection (2022).
  6. US Congress. 2002. The Sarbanes Oxley Act. (2002). https://sarbanes-oxley-act.com/
  7. Ellen P Goodman and Julia Trehu. 2022. Algorithmic Auditing: Chasing AI Accountability. Santa Clara High Tech. LJ 39 (2022), 289.
  8. Elliot Jones, Jenny Brennan, Connor Dunlop, and Andrew Strait. 2023. Keeping an eye on AI. Ada Lovelace Institute (2023). https://www.adalovelaceinstitute.org/wp-content/uploads/2023/09/ALI_Keeping-an-eye-on-AI-2023.pdf
  9. Lauren Kirchner. 2023. New York City Moves to Create Accountability for Algorithms. ProPublica (2023). https://www.propublica.org/article/new-york-city-moves-to-create-accountability-for-algorithms
  10. Colorado Division of Insurance. 2023. Regulation 10-1-1 Governance and Risk Management Framework Requirements for Life Insurers’ Use of External Consumer Data and Information Sources, Algorithms, and Predictive Models. (2023). https://drive.google.com/file/d/1dlPKJCDo76iHfJZDopQEhTDCmKbuYnNI/view?usp=embed_facebook
  11. UK Parliament. 2022. Online Safety Act 2023. (2022). https://bills.parliament.uk/bills/3137
  12. Jules Pattison-Gordon. 2023. Colorado Aims to Prevent AI-Driven Discrimination in Insurance. GovTech (2023). https://www.govtech.com/policy/colorado-aims-to-prevent-ai-driven-discrimination-in-insurance

Cited by

Sorted by citations

Related

Authors
Groves, L., Metcalf, J., Kennedy, A., Vecchione, B., & Strait, A.
Affiliations
  • The Ada Lovelace Institute (GB)
  • Data & Society Research Institute (US)
  • Independent researcher, USA
ORCID
Venue
The 2024 ACM Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2024-06-03
Pages
1107-1120
DOI
10.1145/3630106.3658959
OpenAlex
W4399362708
Cited by
59
References
15
Citations by year
2026: 22 · 2025: 27 · 2024: 10
FWCI
26.92
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
National Science Foundation (1704425)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Law › Digitalization, Law, and Regulation
  • Social Sciences › Business, Management and Accounting › Industrial relations › Legal and Labor Studies
Keywords
New York City, independent auditors, algorithmic transparency, accountability mechanisms, industry lobbying, data access barriers

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-30.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-30.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review