Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing
Open accessThe DOI matches its Crossref record; 96 of the 97 references it cites were found in a registry or an index, and 1 are not checked yet; 11 of the 44 sources its claims cite were checked for support.
Abstract
Rising concern for the societal implications of artificial intelligence systems has inspired a wave of academic and journalistic literature in which deployed systems are audited for harm by investigators from outside the organizations deploying the algorithms. However, it remains challenging for practitioners to identify the harmful repercussions of their own systems prior to deployment, and, once deployed, emergent issues can become difficult or impossible to trace back to their source.
Verified 2026-11-19 · 1 reference not checked yet
Overview
CoChat Summary
TL;DR
The paper proposes SMACTR, an end-to-end internal algorithmic auditing framework intended to help organizations assess AI systems against declared ethical principles throughout development. It argues that internal audits can complement external accountability by providing pre-deployment access, structured documentation, and risk analysis drawn from aerospace, medical device, and finance practices.
Key findings
- Introduces SMACTR—Scoping, Mapping, Artifact Collection, Testing, and Reflection—as five audit stages, each with documentation requirements that together form an overall audit report and assess decisions against organizational values or AI principles.
- Argues external audits are limited by lack of access to internal processes, intermediate models, and training data, while internal audits can enable proactive pre-deployment intervention and generate artifacts for external review.
- Draws lessons from other industries, citing aerospace’s one commercial airline accident per two million flights, Boeing 787 software estimated at 13 million lines of code, and a 1-in-1,000,000,000 per use hour maximum failure probability for critical aerospace systems; also cites medical-device design controls and finance internal audit infrastructure.
- Uses hypothetical cases—a child abuse screening tool and a smile-detection photo booth—to illustrate the framework; for the smile detector’s CelebA datasheet, it reports 58.1% female and 42% male labels, 77.8% aged 0–45 and 22.1% over 46, and 14.2% lighter-skinned versus 85.8% darker-skinned.
Methods
- Conceptual framework paper: develops SMACTR from auditing practices and artifacts in aerospace, medical devices, and finance, plus responsible-innovation dimensions of anticipation, reflexivity, inclusion, and responsiveness.
- Uses a hypothetical Company X Inc. consultancy with two invented client projects—a child abuse screening tool and Happy-Go-Lucky Inc.’s smile-detection photo booth—to demonstrate audit stages and artifacts; a worked example for the smile case is provided as supplementary material.
- Analysis is narrative and design-oriented, referencing existing standards and tools such as FMEA, model cards, datasheets, design controls, and design history files; no empirical sample or quantitative evaluation is reported.
Limitations
- The paper states that internal auditors necessarily share an organizational interest with the audit target, making an independent and objective viewpoint challenging; audits are not an objective “view from nowhere” and risk becoming reputation management.
- The paper states that internal audits are only one aspect of a broader system of required quality checks and balances, and that determining what systems to audit and why is not covered and is a separate research topic.
- Evident from the text: the framework is not empirically validated, the illustrative cases are hypothetical, and no real deployment outcomes or measured audit effectiveness are reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
9 Valid · 6 Not valid · 27 Not checked · 30 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“While not without faults, safety-critical and regulated industries such as aerospace and medicine have long traditions of auditable processes and design controls that have dramatically improved safety [77,81].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 3 LESSONS FROM AUDITING PRACTICES IN OTHER INDUSTRIES
What the sources say
- Not checked — no full text of the sourcecites [77] matched to “Design controls for the medical device industry” (2013) source
- Could not verifycites [81] matched to “The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions” (2019) sourceExperimental — accuracy not yet measured
The Introduction and §3 discuss biomedical ethics and standards or regulation, but do not say that aerospace or medicine has long used auditable processes and design controls, or that these dramatically improved safety. Searches for aerospace, auditable processes, design controls, and safety-critical industries found no passage addressing that claim. Page numbers are unavailable in our copy. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This is a repeatedly observed phenomenon in tax compliance auditing, where several international surveys of tax compliance demonstrate that a fixed and vetted tax audit methodology is one of the most effective strategies to convince companies to respect audit results and pay their full taxes [22,53].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.3 Audit Integrity and Procedural Justice
What the sources say
- Partly supportedcites [22] matched to “Perception on justice, trust and tax compliance behavior in Malaysia” (2017) sourceExperimental — accuracy not yet measured
The Regression Test section reports that Malaysian respondents would be more compliant if tax authorities treated them fairly in tax procedures. The Research Methodology section describes a questionnaire of academics, not several international surveys of companies. The paper does not establish that a fixed, vetted audit methodology is especially effective at securing acceptance of audit results or full tax payment. (LLM, conf=0.86)
- Partly supportedcites [53]Experimental — accuracy not yet measured
The “Justice in the taxation context” section reports that taxpayers are generally more compliant when treated fairly and respectfully, and the General discussion links perceived unfair treatment to resistance to paying back tax (page numbers unavailable). It does not establish that a fixed, vetted audit methodology is among the most effective strategies, or report international surveys demonstrating that effect for companies. (LLM, conf=0.83)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Traditional metrics for artificial intelligence like loss may conceal fairness concerns, social impact risks or abstraction errors [69].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.3.2 Artifact: Ethnographic Field Study.
What the sources say
- Supportedcites [69] matched to “Fairness and Abstraction in Sociotechnical Systems” (2019) source
The Introduction (page not provided) says that narrow assessments of machine-learning performance abstract away social context and can miss information needed for fair outcomes—an abstraction error. The Formalism Trap section adds that mathematical measures cannot capture the full range of fairness concerns, while the Takeaways discuss unintended social consequences. The paper names AUC, precision and recall rather than loss, but the claim uses loss as an illustrative metric. (LLM, conf=0.88)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Internal auditors' direct access to systems can thus help extend traditional external auditing paradigms by incorporating additional information typically unavailable for external evaluations to reveal previously unidentifiable risks.”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.4 The Internal Audit
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The essential inclusion of independent domain experts and marginalized groups in the ethical review process “has the potential to lead to more rigorous critical reflection because their experiences will often be precisely those that are most needed in identifying problematic background assumptions and revealing limitations with research questions, models, or methodologies” [33].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.2.1 Artifact: Ethical Review of System Use Case.
What the sources say
- Not checked — no full text of the sourcecites [33] matched to “25 Years of Feminist Empiricism and Standpoint Theory: Where Are We Now?” (2010) source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
19 more claims
Create a free account to see all 72 claims and every source verdict.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 37
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This disparity poses risks, as practitioners may unintentionally embed their own biases into AI system design and implementation (Martin 2019;Raji et al. 2020).”
From Accountability in Government Use of AI : Citizen Concerns and Preferences (2025) · cites this paper as Raji et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Work on AI safety and reproducibility has long pointed out the crucial importance of training data for understanding model performance, ensuring reproducibility, and assessing legal exposure [2, 25, 29, 46, 58].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While there is a growing interest in auditing algorithmic systems [125,126,128,138,154], this feld generally lacks defned standards and records could be incomplete or not relating to information needed for accountability purposes [15,127].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [128]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Raji et al. (2020) proposed an internal algorithmic auditing framework spanning data collection, model development, testing, deployment, and monitoring, with stage-specific artifacts that enable traceability and accountability.”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Raji et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Practical approaches to implement such ideas typically revolve around the systematic collection and curation of data on various aspects of systems [2, 28, 46].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747 · cited 1×
“A hypothetical datasheet for this system is filled out using studies on the CelebA dataset, with which the smile detector is built [44, 47].”
MethodThe global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Product development: Past research, present findings and future directions
Brown, S. L., & Eisenhardt, K. M. (1995) · The Academy of Management Review · doi:10.2307/258850 · Cited by 3,777 · cited 1×
“There are also intersections between this framework and just efective product development theory [5], as many of the components of audit design refocus the product development process to prioritize the user and their ultimate well-being, resulting in a more efective product performance outcome.”
Background(2018) · Cited by 3,510
Christiane, Druml, Wolzt, & Pleiner (2001) · Acta Scientiarum Naturalium Universitatis Sunyatseni · Cited by 3,419
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196
Developing a framework for responsible innovation
Stilgoe, J., Owen, R., & Macnaghten, P. (2013) · Research Policy · doi:10.1016/j.respol.2013.05.008 · Cited by 3,184 · cited 1×
“To design our audit procedure, we suggest complementing formal risk assessment methodologies with ideas from responsible innovation, which stresses four key dimensions: anticipation, reflexivity, inclusion and responsiveness [73], as well as system-theoretic concepts that help grapple with increasing complexity and co…”
MethodAlgorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Self-citation(2014) · Journal of the Korean Medical Association · doi:10.5124/jkma.2014.57.11.899 · Cited by 2,566
Research works printed but not linked: 7
- Virginia Eubanks. 2018. A child abuse prediction model fails poor families. Wired Magazine (2018).
- Jeremy Goldhaber-Fiebert and Lea Prince. 2019. Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny CountyâĂŹs Child Welfare Office. Pittsburgh: Allegheny County.[Google Scholar] (2019).
- Nitin Kohli, Renata Barreto, and Joshua A Kroll. 2018. Translation Tutorial: A Shared Lexicon for Research and Practice in Human-Centered Software Systems. In 1st Conference on Fairness, Accountability, and Transparancy. New York, NY, USA. 7.
- Joshua A Kroll, Solon Barocas, Edward W Felten, Joel R Reidenberg, David G Robinson, and Harlan Yu. 2016. Accountable algorithms. U. Pa. L. Rev. 165 (2016), 633.
- Institute of Internal Auditors. Research Foundation and Institute of Internal Au- ditors. 2007. The Professional Practices Framework. Inst of Internal Auditors.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22 (2014).
- Tim Vanderveen. 2005. Averting highest-risk errors is first priority. Patient Safety and Quality Healthcare 2 (2005), 16-21.
Cited by
Items 501–600 of 1,412
Historical Methods for AI Evaluations, Assessments, and Audits
Sandoval, . C. B., & Jing, F. S. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732093 · Cited by 12
“…following sections, we introduce historical analysis to the growing critical literature on methodological considerations for AI audits [26, 94, 111, 114, 147, 154] and position it as a form of sociotechnical evaluation that can be integrated within assessment protocols to expand the harms…”
InfluentialArtificial Intelligence and Robotics Addressing COVID-19 Pandemic’s Challenges
David, W., & King‐Okoye, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-70740-8_18 · Cited by 4
Artificial Intelligence and the Internet of Things Forensics in a National Security Context
Montasari, R. (2023) · Advances in information security · doi:10.1007/978-3-031-21920-7_4 · Cited by 5
Lessons from Workers' Inquiry: A Sociotechnical Approach to Audits of Algorithmic Management Systems
Dalal, S. (2024) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/3665598 · Cited by 6
Obioha-Val, O. A. (2024) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2024/v17i11523 · Cited by 10
Rigopoulos, G. (2024) · WSEAS TRANSACTIONS ON BUSINESS AND ECONOMICS · doi:10.37394/23207.2024.21.192 · Cited by 4
Bias in Federated Learning: Factors, Effects, Mitigations, and Open Issues
Benmalek, M., & Seddiki, A. (2024) · Ingénierie des systèmes d information · doi:10.18280/isi.290605 · Cited by 13
The challenges of being an in-house AI ethicist and how to overcome them
Burema, D. (2025) · Journal of Responsible Innovation · doi:10.1080/23299460.2024.2445322 · Cited by 8
Big Data Versus Big GPU: Evolving Requirements and Governance Dynamics of AI Training Data
Cheng, L., Gong, X., & Zhao, Y. (2025) · International Journal of Digital Law and Governance · doi:10.1515/ijdlg-2025-0004 · Cited by 5
Addressing Data Scarcity with Synthetic Data: A Secure and GDPR-Compliant Cloud-Based Platform
Borovits, N., Bardelloni, G., Hashemi, H., Tulsiani, M., Tamburri, D. A., & van den Heuvel, W. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3732937 · Cited by 4
Transparency, Compliance, And Contestability When Code Is(n’t) Law
Hicks, A. (2022) · New Security Paradigms Workshop · doi:10.1145/3584318.3584854 · Cited by 12
“There are suggestions for ways to audit the design [69] of AI systems as well as releasing information about the models themselves [64] and the datasets that they are trained on [34].”
Wu, Y.-H. (2024) · AI and Ethics · doi:10.1007/s43681-024-00535-1 · Cited by 8
“AI models should not be considered immune from the sociopolitical influences of the larger world introduced by the hands of those who build them [39].”
Certification of Model Robustness in Active Class Selection
Bunse, M., & Morik, K. J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86520-7_17 · Cited by 5
Krause, D. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5050631 · Cited by 7
Refusing to Fall Behind: The Ethical Obligation to Embrace AI in Mental Health Social Work
Flaherty, H. B., & Krishnan, P. (2025) · Journal of Evidence-Based Social Work · doi:10.1080/26408066.2025.2553018 · Cited by 8
Algorithmic Justice: Navigating AI's Role in Cybersecurity and Legal Transformation
Ali, S. I. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-3114-0.ch007 · Cited by 5
How to Reduce Bias in the Life Cycle of a Data Science Project
Banks, G. C., Tonidandel, S., Dou, W., Gerson, M. J., Xu, D., & Yavorsky, J. E. (2025) · Journal of Business and Psychology · doi:10.1007/s10869-025-10022-x · Cited by 6
“As advanced technology and society as a whole change, the need for auditing the life cycle of the AI approach has become a necessary step to keep such technology accountable and to achieve its full potential (Raji et al., 2020).”
Panigutti, C., Yela, D. F., Porcaro, L., Bertrand, A., & Garrido, J. S. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732052 · Cited by 14
“For example, many authors distinguish between first, second, and third-party algorithm audits based on the auditor’s independence from the audit target [21], or between internal and external audits based on the level of access to the algorithmic systems being examined [90].”
What is AI safety? What do we want it to be?
Harding, J., & Kirk‐Giannini, C. D. (2025) · Philosophical Studies · doi:10.1007/s11098-025-02367-z · Cited by 25
Building Trust in Artificial Intelligence
Dordunoo, L. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-5686-0.ch010 · Cited by 3
Li, D. (2025) · Chinese Public Administration Review · doi:10.1177/15396754251359961 · Cited by 6
“Organizations such as AI Now Institute, Algorithm Watch, and Access Now developed comprehensive governance frameworks emphasizing rights-based approaches, democratic participation, and strong accountability mechanisms(Raji et al., 2020).”
Kumar, D., Suthar, N., Rodriguez, R. V., & Hemachandran, K. (2025) · Journal of Information Communication and Ethics in Society · doi:10.1108/jices-07-2025-0174 · Cited by 6
Tamer, O. Ü., McDonald, B. D., Hémici, F., & Κοντόγεωργα, Γ. (2025) · Sayıştay Dergisi · doi:10.52836/sayistay.1819606 · Cited by 3
Yang, S. (2026) · Science as Culture · doi:10.1080/09505431.2026.2628729 · Cited by 3
“…as SMACTR (Scoping– Mapping–Artefact collection–Testing–Conclusion–Recommendations) model similarly cast accountability as a sequence of documentation and testing steps whose outputs are reports authored within organizations, judged against their own values and risk tolerances (Raji et al., 2020).”
Artificial intelligence as a moral mediator: emotional reciprocity driving happiness in hospitality
Ramírez, R. R., Leal‐Solís, A., & Roets, A. O. (2026) · AI & Society · doi:10.1007/s00146-026-02953-7 · Cited by 3
“EAIG embodies the extent to which AI responsibly aggregates operational and emotional data without profiling individuals, enforces fair treatment, and embeds ethical oversight throughout design and deployment (Lucas et al. 2025; Raji et al. 2020).”
Evangelista, E. D. L., & Bukhari, S. M. S. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100594 · Cited by 13
Mohamed, R. A. A. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2026.2649549 · Cited by 4
“…requires learner-authored rewriting, and normalizes verification when system output is ambiguous – design imperatives consistent with human–AI interaction guidance and accountable AI documentation principles (Amershi et al., 2019; Ji et al., 2023; Mitchell et al., 2019; Raji et al., 2020).”
InfluentialEthical Precision in Nanoscale Brain Interfacing
Wood, G. M. D. O. (2026) · Advanced Science · doi:10.1002/advs.202524255 · Cited by 3
Security and Privacy of Large Language Models: Threat Taxonomy, Ethical Implications, and Governance
Pribisalić, M., & Martinčić-Ipšić, S. (2026) · AI · doi:10.3390/ai7050152 · Cited by 9
Operational responsibility in AI governance: a user-centric liability framework
Chen, Z. (2026) · AI and Ethics · doi:10.1007/s43681-026-01163-7 · Cited by 8
Saraboji, K., Gopalakrishnan, K., Sood, D., Kaur, A., Shivaram, S., Helgeson, S. A., Arunachalam, S. P., & Mitra, D. (2026) · AI in Medicine · doi:10.3390/aimed1020016 · Cited by 4
Towards adaptive categories: Dimensional governance for agentic AI
Engin, Z., & Hand, D. J. (2026) · Data & Policy · doi:10.1017/dap.2026.10083 · Cited by 25
“Lifecycle governance models promote oversight across system development and deployment stages (Raji et al., 2020; Cobbe et al., 2021), while human-centred (Capel and Brereton, 2023) and trustworthy AI (Li et al., 2023) frameworks foreground the role of transparency, contestability, and calibrated…”
Sokolovsky, A., & Arnaboldi, L. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119836 · Cited by 5
Automated Boundary Identification for Machine Learning Classifiers
Dobslaw, F., & FELDT, R. H. (2024) · IEEE/ACM International Workshop on Search-Based Software Testing (SBST) · doi:10.1145/3643659.3643927 · Cited by 2
The future of medicine or a threat? Artificial intelligence representation in Chicago Med
Nádasi, E., & Héder, M. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03810-y · Cited by 4
“Therefore, there is an issue of assigning responsibility: decisions are made by an agent who cannot bear responsibility for them, creating a “ gap ” (Barnes 2020) in the chain of accountability.”
Psychological Impact of AI Moderation
Subrahmanyam, S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-0335-2.ch008 · Cited by 4
Exploring Culturally Informed AI Assistants: A Comparative Study of ChatBlackGPT and ChatGPT
Egede, L., Haque, E. A., Thompson, G., Boyd, A. E., Smith, A. D. R., & Johnson, B. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720136 · Cited by 7
Mediating The Marginal: A Quantitative Analysis of Curated LGBTQ+ Content on Instagram
Souza, G., Lutz, N., & Turner, K. M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713618 · Cited by 7
RogueGPT: transforming ChatGPT-4 into a rogue AI with dis-ethical tuning
Buscemi, A., & Proverbio, D. (2025) · AI and Ethics · doi:10.1007/s43681-025-00750-4 · Cited by 6
“One prominent approach involves the development of model cards and documentation practices that provide detailed information about the model’s capabilities, limitations, and potential biases [12, 23].”
Xiong, P., & Zhang, Y. (2025) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2025.2561927 · Cited by 5
RetractedErlei, A., Cau, F. M., Georgiev, R., Kumar, S. C., Bizer, K., & Gadiraju, U. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791420 · Cited by 7
“Participants could select an AI system from the pool for delegation by first clicking on it—which makes the selected AI system light up with a green border—and then clicking the ‘ Delegate decision to AI ’ button to complete the current trial.”
Creative Provocations: Speculations on the Future of Creativity, Technology & Learning
Henriksen, D., & Mishra, P. (2022) · Creativity theory and action in education · doi:10.1007/978-3-031-14549-0 · Cited by 4
Centring dignity in algorithm development: testing a Dignity Lens
Ruster, L. P., Oliva‐Altamirano, P., & Daniell, K. A. (2022) · Australian Conference on Human-Computer Interaction (OzCHI) · doi:10.1145/3572921.3572938 · Cited by 5
“To date, much of the work to operationalise AI ethics principles focuses on principles such as fairness [26], accountability [39], and transparency [11, 38, 55].”
BackgroundGanesh, B. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604751 · Cited by 4
ÉTICA Y GÉNERO EN LA IA: IDENTIFICAR SESGOS DE GÉNERO EN IA MEDIANTE PENSAMIENTO COMPLEJO
Pérez, M. N. V. (2024) · Revista Iberoamericana de Complejidad y Ciencias Económicas · doi:10.48168/ricce.v2n2p49 · Cited by 6
A Minimalist Account of the Right To Explanation
Grote, T., & Paulo, N. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00888-3 · Cited by 6
“We outline the basic features of our proposal by discussing literature on model auditing and reliability in machine learning (Raji et al., 2020; Grote et al., 2024 ) and on the standardized documentation of model properties via model cards (Mitchell et al., 2019).”
Enhancing Transparency in Large Language Models to Meet EU AI Act Requirements
Kalodanis, K., Rizomiliotis, P., Feretzakis, G., Kalles, D., Verykios, V. S., & Anagnostopoulos, D. (2024) · Pan-Hellenic Conference on Progress in Computing and Informatics (PCI) · doi:10.1145/3716554.3716597 · Cited by 7
AI Data Development: A Scorecard for the System Card Framework
Bahiru, T. K., Tibebu, H., & Kakadiaris, I. A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-8687-2_42 · Cited by 3
Taking the Confusion Out of Multinomial Confusion Matrices and Imbalanced Classes
Lovell, D. R., McCarron, B., Langfield, B., Tran, K. A., & Bradley, A. P. (2021) · Communications in computer and information science · doi:10.1007/978-981-16-8531-6_2 · Cited by 7
Digital Transformation, Leadership, and Markets
Hydros, A. K. M., & Chaudhry, U. B. (2022) · Advanced sciences and technologies for security applications · doi:10.1007/978-3-030-98225-6_8 · Cited by 5
A Pragmatic Approach to Regulating Artificial Intelligence: A Technology Regulator's PerspectivePreprint
Ellul, J., McCarthy, S., Sammut, T., Brockdorff, J., Scerri, M., & Pace, G. J. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3827303 · Cited by 9
“To minimise costs and burden for the various involved stakeholders algorithmic and computational auditing and compliance have been proposed [36, 49].”
BackgroundArtificial moral cognition: Learning from developmental psychologyPreprint
Weidinger, L., Reinecke, M. G., & Haas, J. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/tnf4e · Cited by 3
Thomas, C., Roberts, H., Mökander, J., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4660737 · Cited by 3
Verifiable AI Governance: The Four Tests Standard (4TS) and Proof-Carrying DecisionsPreprint
Meyman, E. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5688982 · Cited by 3
TGE-360: A Mature, Executable and Evidence-Governed Taqrīb Governance EcosystemPreprint
Moghadasnian, S. (2026) · doi:10.24108/preprints-3115499 · Cited by 3
Practices for Engineering Trustworthy Machine Learning Applications
Serban, A., van der Blom, K., Hoos, H. H., & Visser, J. (2021) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/wain52551.2021.00021 · Cited by 33
Reducing organizational inequalities associated with algorithmic controls
Li, Y., & Xiang, B. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00137-0 · Cited by 11
“Research on trustworthy AI has emerged in the past few years, which generally examines AI accountability, transparency, explainability, interpretability, fairness, safety, privacy, and security [19, 20, 22, 23, 110–113].”
Emerging Paradigms in Non-Profit Governance: A Comprehensive Analysis of Disruptive Innovations
Santos, J. (2024) · Advances in logistics, operations, and management science book series · doi:10.4018/979-8-3693-3723-3.ch001 · Cited by 7
Governing the Digital Society: Platforms, Artificial Intelligence, and Public Values
(2025) · Amsterdam University Press eBooks · doi:10.1515/9789048562725 · Cited by 3
Malin, C. D., Fleiß, J., & Thalmann, S. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1561322 · Cited by 10
Chakraborty, C., & Howard, A. (2025) · IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS) · doi:10.1109/ethics65148.2025.11098392 · Cited by 3
Ethical Leadership Challenges in the Age of Artificial Intelligence: An In-depth Analysis
Bannor, F. O., & Baysah, J. O. (2025) · Pan-African journal of education and social sciences · doi:10.56893/pajes2025v06i02.06 · Cited by 8
Governance of AI-Based Algorithms
Okolo, C. T. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_84 · Cited by 4
Shin, E. Y., & Shin, D. (2025) · Transforming Government People Process and Policy · doi:10.1108/tg-09-2025-0273 · Cited by 8
Systematic Mapping Study on AI Integration in CI/CD Pipelines
Berrouyne, O., Bajta, M. E., & Benelallam, I. (2025) · International Conference on Optimization and Applications (ICOA) · doi:10.1109/icoa66896.2025.11236947 · Cited by 2
A theoretical model of AI bias mitigation: incentives, regulation and equilibrium
Bera, S., & Bera, I. (2025) · Journal of Information Communication and Ethics in Society · doi:10.1108/jices-06-2025-0138 · Cited by 2
Operationalising ethical AI governance: a triadic framework for the European AI act
Baeyaert, J. (2025) · AI and Ethics · doi:10.1007/s43681-025-00882-7 · Cited by 4
Developing an artificial intelligence ethics governance checklist for the legal community
Kelley, S. (2025) · AI and Ethics · doi:10.1007/s43681-025-00841-2 · Cited by 5
“…used to implement principles in prac - tice is an AI ethics governance checklist (also referred to as a standards checklist or standard framework) [27, 34, 35 ]; a checklist that facilitates a risk assessment of AI systems within a firm’s existing AI model development or vendor selection processes.”
Rahman, M. I., Hossain, M. R., Sayem, S. M., & Emdad, F. B. (2025) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2025.100342 · Cited by 8
UK AI Governance at a Crossroads: Charting the Path Ahead
Ünver, M. B., & Odusola-Stevenson, S. (2025) · International journal of mass communication. · doi:10.6000/2818-3401.2025.03.12 · Cited by 4
Desai, D., & Desai, A. A. (2026) · Analytics · doi:10.3390/analytics5010008 · Cited by 3
Coordination transparency: governing distributed agency in AI systems
Bohr, J. (2026) · AI & Society · doi:10.1007/s00146-026-02853-w · Cited by 8
“Audits focused on abstract fairness metrics struggle to capture how platform integration, deployment context, and downstream interaction effects shape impacts in practice (Raji et al. 2020; Selbst et al. 2019).”
Interdisciplinary expertise to advance human-centered explainable AI
Bennett, C. R., Cole-Lewis, H. J., Farquhar, S. A., Hammel, N., Babenko, B., Lang, O., Fleck, M. S., Traynis, I., Lau, C. T., Liu, Y., et al. (2026) · Social Science & Medicine · doi:10.1016/j.socscimed.2026.119036 · Cited by 13
Computational text analysis on unstructured police data: a scoping review
Lukmanjaya, W., Halmich, C., Butler, T. G., Cook, D., & Karystianis, G. (2026) · Crime Science · doi:10.1186/s40163-026-00272-2 · Cited by 8
Borkowska, K., & Jackson, D. (2026) · AI & Society · doi:10.1007/s00146-026-02921-1 · Cited by 6
“A more constructive approach treats risks as identifiable and manageable through concrete controls—documentation, evaluation, redress, and oversight—rather than as abstract threats (Beck 1992; Mitchell et al. 2019; Raji et al. 2020).”
InfluentialGoverning Artificial Intelligence Through Auditing
Oluka, A., & Mashau, P. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-3922-1.ch005 · Cited by 4
Robotic Bias and Its Dimensions
Furlanis, T., Brščić, D., & Kanda, T. (2026) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3757279.3785613 · Cited by 3
Measuring agentic AI adoption and control frameworks in finance
Mustafa, A. U., & Aysan, A. F. (2026) · Modern Finance · doi:10.61351/mf.v4i1.557 · Cited by 2
“…but the ability to produce accountability artifacts such as documentation, monitoring, evidence logging, and auditable controls that allow autonomy to be scaled without violating safety, compliance, or reputational constraints (Mitchell et al., 2019; Raji et al., 2020; Gebru et al., 2021).”
InfluentialAlgorithmic HRM: Opportunities and Ethical Challenges in AI-Driven Talent Acquisition
Rismawati, R., & Salju, S. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-4307-5.ch001 · Cited by 2
Nandal, N. (2026) · EDPACS · doi:10.1080/07366981.2026.2667950 · Cited by 4
Social stereotypes in AI text-to-image generation
Barve, S., Mao, A., Shi, J. M., Juneja, P., & Saha, K. (2026) · AI and Ethics · doi:10.1007/s43681-026-01146-8 · Cited by 19
Fujii, Y. (2026) · AI · doi:10.3390/ai7060218 · Cited by 3
When Saying “No” Is Not Enough: Cognitive-Action Decoupling and the Illusion of Safety in LLM Agents
Yu, S., Carroll, F., & Bentley, B. L. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812378 · Cited by 2
Discrimination DRIFT: A Practical Framework for Responsible AI
Wyer, S. (2026) · AI and Ethics · doi:10.1007/s43681-026-01211-2 · Cited by 3
Surveilled urban futures: How AI shapes power, privacy, and participation in smart cities
Yiğitcanlar, T., Shaamala, A., Marasinghe, R., Liu, K., Hossain, S. T., & Xu, Y. (2026) · Urban Studies · doi:10.1177/00420980261461440 · Cited by 4
The Continuity Trap in Data Science Health Research
Adebamowo, C., Adebamowo, S. N., Akintola, A., Ikhane, P. A., Akintola, S., Ogundiran, T. O., Jegede, A. S., Adeyemo, O., Callier, S., Imam-Tamim, M. K., et al. (2026) · Journal of Medical Internet Research · doi:10.2196/98699 · Cited by 3
Yalçın, O. G. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-87687-6_17 · Cited by 3
Using structured ethical techniques to facilitate reasoning in technology ethics
Murphy, M. (2023) · AI and Ethics · doi:10.1007/s43681-023-00371-9 · Cited by 3
“These practices also try to integrate in the design and deployment of technical systems by translating ethical values into actual design requirements for developers [17].”
Guiding the Next Technological Revolution: Principles for Responsible AI and Nanotech Progress
Gutierrez, R. (2023) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-0368-9.ch011 · Cited by 7
On Audit and Certification of Machine Learning Systems
Namiot, D., & Sneps-Sneppe, M. (2023) · Conference of Open Innovations Association (FRUCT) · doi:10.23919/fruct60429.2023.10328174 · Cited by 2
“Another such project is presented in the work of Google employees [35].”
BackgroundCharting the Ethical Course: Navigating AI Advancements in Communication Education
Le-Nguyen, H.-T., & Tran, T. T. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-0831-8.ch011 · Cited by 4
Thick Data Analytics (TDA): An Iterative and Inductive Framework for Algorithmic Improvement
Nguyen, M., Eulalio, T. Y., Marafino, B. J., Rose, C. C., Chen, J. H., & Baiocchi, M. (2024) · The American Statistician · doi:10.1080/00031305.2024.2327535 · Cited by 2
“To promote the use of feedback mechanisms in addition to retrospective analysis of model predictions, a recent shift in the literature of model assessment investigates unusual behaviors, errors, and failures (Liu et al. 2022; Raji et al. 2020; Oakden-Rayner et al. 2020, 2022).”
Who Pilots the Copilots? Mapping a Generative AI’s Actor-Network to Assess Its Educational Impacts
Balzan, F., Munarini, M., & Angeli, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64299-9_42 · Cited by 8
Sattlegger, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70804-6_9 · Cited by 3
Managing Ethical Challenges: Ensuring Equity and Integrity in AI-Powered Assessment
Dar, A. A., Yadav, S. S., Tripathi, R. K., Albalawi, O., Jain, A., & Gautam, P. L. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-6170-2.ch011 · Cited by 7
A Review of Text Generation Based on Large Language Model
刘, 津. (2025) · Artificial Intelligence and Robotics Research · doi:10.12677/airr.2025.141019 · Cited by 2
Bohlens, C. (2025) · Advances in human and social aspects of technology book series · doi:10.4018/979-8-3693-9035-1.ch011 · Cited by 3
Redefining Work in the Age of AI: Navigating Employment, Automation, and Ethics
Giri, A., & Sharma, A. K. M. M. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-3384-7.ch009 · Cited by 3
The Role of Artificial Intelligence in Achieving Audit Quality in the Kingdom of Bahrain
Astal, A. Y. M. A., Aldoseri, J., & Al-Mesaiadeen, J. M. (2025) · Studies in systems, decision and control · doi:10.1007/978-3-031-84628-1_81 · Cited by 3
Ethical AI Auditor for Bias Detecting in AI Models Using Adversarial Debiasing
A, S. G., Geerthik, S., Ida, J. J., & Jubi, S. A. (2025) · International Conference on Advances in Modern Age Technologies for Health and Engineering Science (AMATHE) · doi:10.1109/amathe65477.2025.11081330 · Cited by 3
Metadata
- Authors
- Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P.
- Affiliations
- Partnership on AI
- Google (United States) (US)
- ORCID
- Venue
- Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2020-01-27
- Pages
- 33-44
- OpenAlex
- W3001807593
- Cited by
- 2,245
- References
- 83
- Citations by year
- 2026: 1097 · 2025: 454 · 2024: 243 · 2023: 177 · 2022: 149 · 2021: 99 · 2020: 24
- FWCI
- 35.91
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- AI accountability, algorithmic auditing, harm detection, algorithmic transparency, societal impact of AI, algorithmic governance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.