Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing
Open accessThe DOI matches its Crossref record; 96 of the 97 references it cites were found in a registry or an index, and 1 are not checked yet; 11 of the 44 sources its claims cite were checked for support.
Abstract
Rising concern for the societal implications of artificial intelligence systems has inspired a wave of academic and journalistic literature in which deployed systems are audited for harm by investigators from outside the organizations deploying the algorithms. However, it remains challenging for practitioners to identify the harmful repercussions of their own systems prior to deployment, and, once deployed, emergent issues can become difficult or impossible to trace back to their source.
Verified 2026-11-19 · 1 reference not checked yet
Overview
CoChat Summary
TL;DR
The paper proposes SMACTR, an end-to-end internal algorithmic auditing framework intended to help organizations assess AI systems against declared ethical principles throughout development. It argues that internal audits can complement external accountability by providing pre-deployment access, structured documentation, and risk analysis drawn from aerospace, medical device, and finance practices.
Key findings
- Introduces SMACTR—Scoping, Mapping, Artifact Collection, Testing, and Reflection—as five audit stages, each with documentation requirements that together form an overall audit report and assess decisions against organizational values or AI principles.
- Argues external audits are limited by lack of access to internal processes, intermediate models, and training data, while internal audits can enable proactive pre-deployment intervention and generate artifacts for external review.
- Draws lessons from other industries, citing aerospace’s one commercial airline accident per two million flights, Boeing 787 software estimated at 13 million lines of code, and a 1-in-1,000,000,000 per use hour maximum failure probability for critical aerospace systems; also cites medical-device design controls and finance internal audit infrastructure.
- Uses hypothetical cases—a child abuse screening tool and a smile-detection photo booth—to illustrate the framework; for the smile detector’s CelebA datasheet, it reports 58.1% female and 42% male labels, 77.8% aged 0–45 and 22.1% over 46, and 14.2% lighter-skinned versus 85.8% darker-skinned.
Methods
- Conceptual framework paper: develops SMACTR from auditing practices and artifacts in aerospace, medical devices, and finance, plus responsible-innovation dimensions of anticipation, reflexivity, inclusion, and responsiveness.
- Uses a hypothetical Company X Inc. consultancy with two invented client projects—a child abuse screening tool and Happy-Go-Lucky Inc.’s smile-detection photo booth—to demonstrate audit stages and artifacts; a worked example for the smile case is provided as supplementary material.
- Analysis is narrative and design-oriented, referencing existing standards and tools such as FMEA, model cards, datasheets, design controls, and design history files; no empirical sample or quantitative evaluation is reported.
Limitations
- The paper states that internal auditors necessarily share an organizational interest with the audit target, making an independent and objective viewpoint challenging; audits are not an objective “view from nowhere” and risk becoming reputation management.
- The paper states that internal audits are only one aspect of a broader system of required quality checks and balances, and that determining what systems to audit and why is not covered and is a separate research topic.
- Evident from the text: the framework is not empirically validated, the illustrative cases are hypothetical, and no real deployment outcomes or measured audit effectiveness are reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
9 Valid · 6 Not valid · 27 Not checked · 30 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“While not without faults, safety-critical and regulated industries such as aerospace and medicine have long traditions of auditable processes and design controls that have dramatically improved safety [77,81].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 3 LESSONS FROM AUDITING PRACTICES IN OTHER INDUSTRIES
What the sources say
- Not checked — no full text of the sourcecites [77] matched to “Design controls for the medical device industry” (2013) source
- Could not verifycites [81] matched to “The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions” (2019) sourceExperimental — accuracy not yet measured
The Introduction and §3 discuss biomedical ethics and standards or regulation, but do not say that aerospace or medicine has long used auditable processes and design controls, or that these dramatically improved safety. Searches for aerospace, auditable processes, design controls, and safety-critical industries found no passage addressing that claim. Page numbers are unavailable in our copy. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This is a repeatedly observed phenomenon in tax compliance auditing, where several international surveys of tax compliance demonstrate that a fixed and vetted tax audit methodology is one of the most effective strategies to convince companies to respect audit results and pay their full taxes [22,53].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.3 Audit Integrity and Procedural Justice
What the sources say
- Partly supportedcites [22] matched to “Perception on justice, trust and tax compliance behavior in Malaysia” (2017) sourceExperimental — accuracy not yet measured
The Regression Test section reports that Malaysian respondents would be more compliant if tax authorities treated them fairly in tax procedures. The Research Methodology section describes a questionnaire of academics, not several international surveys of companies. The paper does not establish that a fixed, vetted audit methodology is especially effective at securing acceptance of audit results or full tax payment. (LLM, conf=0.86)
- Partly supportedcites [53]Experimental — accuracy not yet measured
The “Justice in the taxation context” section reports that taxpayers are generally more compliant when treated fairly and respectfully, and the General discussion links perceived unfair treatment to resistance to paying back tax (page numbers unavailable). It does not establish that a fixed, vetted audit methodology is among the most effective strategies, or report international surveys demonstrating that effect for companies. (LLM, conf=0.83)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Traditional metrics for artificial intelligence like loss may conceal fairness concerns, social impact risks or abstraction errors [69].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.3.2 Artifact: Ethnographic Field Study.
What the sources say
- Supportedcites [69] matched to “Fairness and Abstraction in Sociotechnical Systems” (2019) source
The Introduction (page not provided) says that narrow assessments of machine-learning performance abstract away social context and can miss information needed for fair outcomes—an abstraction error. The Formalism Trap section adds that mathematical measures cannot capture the full range of fairness concerns, while the Takeaways discuss unintended social consequences. The paper names AUC, precision and recall rather than loss, but the claim uses loss as an illustrative metric. (LLM, conf=0.88)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Internal auditors' direct access to systems can thus help extend traditional external auditing paradigms by incorporating additional information typically unavailable for external evaluations to reveal previously unidentifiable risks.”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.4 The Internal Audit
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The essential inclusion of independent domain experts and marginalized groups in the ethical review process “has the potential to lead to more rigorous critical reflection because their experiences will often be precisely those that are most needed in identifying problematic background assumptions and revealing limitations with research questions, models, or methodologies” [33].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.2.1 Artifact: Ethical Review of System Use Case.
What the sources say
- Not checked — no full text of the sourcecites [33] matched to “25 Years of Feminist Empiricism and Standpoint Theory: Where Are We Now?” (2010) source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
19 more claims
Create a free account to see all 72 claims and every source verdict.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 37
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This disparity poses risks, as practitioners may unintentionally embed their own biases into AI system design and implementation (Martin 2019;Raji et al. 2020).”
From Accountability in Government Use of AI : Citizen Concerns and Preferences (2025) · cites this paper as Raji et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Work on AI safety and reproducibility has long pointed out the crucial importance of training data for understanding model performance, ensuring reproducibility, and assessing legal exposure [2, 25, 29, 46, 58].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While there is a growing interest in auditing algorithmic systems [125,126,128,138,154], this feld generally lacks defned standards and records could be incomplete or not relating to information needed for accountability purposes [15,127].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [128]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Raji et al. (2020) proposed an internal algorithmic auditing framework spanning data collection, model development, testing, deployment, and monitoring, with stage-specific artifacts that enable traceability and accountability.”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Raji et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Practical approaches to implement such ideas typically revolve around the systematic collection and curation of data on various aspects of systems [2, 28, 46].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747 · cited 1×
“A hypothetical datasheet for this system is filled out using studies on the CelebA dataset, with which the smile detector is built [44, 47].”
MethodThe global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Product development: Past research, present findings and future directions
Brown, S. L., & Eisenhardt, K. M. (1995) · The Academy of Management Review · doi:10.2307/258850 · Cited by 3,777 · cited 1×
“There are also intersections between this framework and just efective product development theory [5], as many of the components of audit design refocus the product development process to prioritize the user and their ultimate well-being, resulting in a more efective product performance outcome.”
Background(2018) · Cited by 3,510
Christiane, Druml, Wolzt, & Pleiner (2001) · Acta Scientiarum Naturalium Universitatis Sunyatseni · Cited by 3,419
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196
Developing a framework for responsible innovation
Stilgoe, J., Owen, R., & Macnaghten, P. (2013) · Research Policy · doi:10.1016/j.respol.2013.05.008 · Cited by 3,184 · cited 1×
“To design our audit procedure, we suggest complementing formal risk assessment methodologies with ideas from responsible innovation, which stresses four key dimensions: anticipation, reflexivity, inclusion and responsiveness [73], as well as system-theoretic concepts that help grapple with increasing complexity and co…”
MethodAlgorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Self-citation(2014) · Journal of the Korean Medical Association · doi:10.5124/jkma.2014.57.11.899 · Cited by 2,566
Research works printed but not linked: 7
- Virginia Eubanks. 2018. A child abuse prediction model fails poor families. Wired Magazine (2018).
- Jeremy Goldhaber-Fiebert and Lea Prince. 2019. Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny CountyâĂŹs Child Welfare Office. Pittsburgh: Allegheny County.[Google Scholar] (2019).
- Nitin Kohli, Renata Barreto, and Joshua A Kroll. 2018. Translation Tutorial: A Shared Lexicon for Research and Practice in Human-Centered Software Systems. In 1st Conference on Fairness, Accountability, and Transparancy. New York, NY, USA. 7.
- Joshua A Kroll, Solon Barocas, Edward W Felten, Joel R Reidenberg, David G Robinson, and Harlan Yu. 2016. Accountable algorithms. U. Pa. L. Rev. 165 (2016), 633.
- Institute of Internal Auditors. Research Foundation and Institute of Internal Au- ditors. 2007. The Professional Practices Framework. Inst of Internal Auditors.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22 (2014).
- Tim Vanderveen. 2005. Averting highest-risk errors is first priority. Patient Safety and Quality Healthcare 2 (2005), 16-21.
Cited by
Items 601–700 of 1,412
Stevens, N., & Gieseking, J. J. (2026) · Transactions of the Institute of British Geographers · doi:10.1111/tran.70046 · Cited by 2
Oguine, O. C., Garcia, A. A., Muller, M. J., & Badillo-Urquiola, K. A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791346 · Cited by 7
“Recent scholarship highlights the epistemic value of lived experience, particularly among communities that are systematically marginalized or disproportionately impacted by sociotechnical harms [24, 82, 103].”
Duin, A. H., & Pedersen, I. (2021) · Studies in computational intelligence · doi:10.1007/978-3-030-70928-0_4 · Cited by 3
Railway Transport Planning and Management
de Luca, S., Pace, R. D., & Fiori, C. (2021) · IntechOpen eBooks · doi:10.5772/intechopen.95682 · Cited by 2
Beck, C. A. M. R., Boff, M. M., & Piaia, T. C. (2022) · Revista Eletrônica Direito e Política · doi:10.14210/rdp.v17n2.p509-546 · Cited by 11
More Than Just a Policy: Day-to-Day Effects of Data Governance on the Data Scientist
Marivate, V. N. (2023) · Springer eBooks · doi:10.1007/978-3-031-24498-8_7 · Cited by 2
Explainability for All: Care Ethics for Implementing Artificial Intelligence
Tracey, O., & Irish, R. K. (2023) · IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas57930.2023.10306199 · Cited by 2
Recommender Systems and Autonomy: A Role for Regulation of Design, Rights, and Transparency
Djeffal, C., & Hitrova, C. (2020) · Indian Journal of Law and Technology · doi:10.55496/jfsp9105 · Cited by 2
Progressing Towards Responsible AIPreprint
Scantamburlo, T., Cortés, À., & Schacht, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.07326 · Cited by 11
“help companies and their engineering teams audit AI systems before deploying them [56], or the People + AI Guidebook to help user experience (UX) professionals and product managers follow a human-centered approach to AI [27].”
BackgroundThe Care Label Concept: A Certification Suite for Trustworthy and Resource-Aware Machine LearningPreprint
Morik, K. J., Kotthaus, H., Heppe, L., Heinrich, D., Fischer, R., Pauly, A., & Piatkowski, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.00512 · Cited by 8
“Trustworthy AI answers the societal demands [2] and surveys methods for increasing trustworthiness [23] or proposes AI internal audits [41].”
BackgroundHow Good Is NLP? A Sober Look at NLP Tasks through the Lens of Social ImpactPreprint
Jin, Z., Chauhan, G., Tse, B., Sachan, M., & Mihalcea, R. F. (2021) · arXiv (Cornell University) · doi:10.3929/ethz-b-000527311 · Cited by 3
Borsci, S., Lehtola, V., Nex, F., Yang, M. Y., Augustijn, E.-W., Bagheriye, L., Brüne, C., Kounadi, O., Li, J. J., Moreira, J., et al. (2023) · doi:10.32920/23808690 · Cited by 2
WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AIPreprint
Deng, W. H., Wang, C., Han, H. Z., Hong, J., Holstein, K., & Eslami, M. (2025) · Qeios · doi:10.32388/b1lxnv · Cited by 4
Bridging the Divide: A Systems Thinking Approach to Inclusivity in AI Development and EducationPreprint
Sarkar, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5097669 · Cited by 5
Global Approaches to Auditing Artificial Intelligence: A Literature Review
(IPIE), I. P. O. T. I. E., Sylvester, P., Chun, W. H. K., Hannák, A., Mendoza, M., Broussard, M., Сергеевич, Е. Р., Nelson, A., Sandvig, C., Sesan, G., et al. (2024) · doi:10.61452/bwym7397 · Cited by 7
Towards A Global AI Auditing Framework: Assessment and Recommendations
(IPIE), I. P. O. T. I. E., Faveri, B., Johnson-León, M., Sylvester, P., Chun, W. H. K., Hannák, A., Mendoza, M., Broussard, M., Сергеевич, Е. Р., Nelson, A., et al. (2025) · doi:10.61452/zwed1485 · Cited by 6
Strategies for Culturally Responsive AI in Education: Mitigating Bias and Enhancing Student Outcomes
Oyetade, K., Harmse, A., & Zuva, T. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-6932-5_2 · Cited by 3
Designing Data Science Software for Social Care Organisations
Reinmund, T. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3577062 · Cited by 1
Centering the Margins: Outlier-Based Identification of Harmed Populations in Toxicity Detection
Raman, V., Fleisig, E., & Klein, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.579 · Cited by 8
Research on the Application of AI Technology in Auditing
Zhang, S. (2024) · Economic management & global business studies. · doi:10.69610/j.emgbs.20240831 · Cited by 2
Bustelo, J. L. (2024) · Revista Iberoamericana de Complejidad y Ciencias Económicas · doi:10.48168/ricce.v2n4p5 · Cited by 2
Talking About the Assumption in the Room
Mothilal, R. K., Lalani, F. M., Ahmed, S. I., Guha, S., & Sultana, S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713958 · Cited by 6
“…a significant number of these toolkits—some very popular such as model cards [87] or datasheets [37]—that do not include direct interrogative questions to invoke assumptions, but rather frame questions on various phases of an ML workflow intended to unearth hidden assumptions [29, 62, 100, 102].”
Harnessing the Power of AI With Deep Learning: Innovations and Implications
Upender, T., Sathiyamoorthy, C., Udayakumar, K., Kirubakaran, D., Purushothaman, V., & Raja, K. T. (2025) · International Conference on Computational, Communication and Information Technology (ICCCIT) · doi:10.1109/icccit62592.2025.10928038 · Cited by 1
“Data privacy concerns, algorithmic bias, and the need for explainable AI are critical issues that must be addressed to ensure the responsible and equitable deployment of these technologies [13][14].”
Exploring the Multi-Dimensional Risks of Generative AI in Ethics, Technology, and Society
Abuhashish, F., & Ismail, I. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-0832-6.ch003 · Cited by 2
van Schie, G., Candidatu, L., & Huyskes, D. (2025) · Amsterdam University Press eBooks · doi:10.1515/9789048562725-016 · Cited by 1
Ponmalar, S., Kumar, K. N., Balaji, R. V., Rohith, T., & Kaarthick, A. K. N. (2025) · International Conference on Augmented Intelligence and Sustainable Systems (ICAISS) · doi:10.1109/icaiss61471.2025.11042179 · Cited by 3
The Individual and Organizational Foundations of Ethical Decision-Making
Brahmi, H., & Kaneb, Y. E. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1737-3.ch004 · Cited by 1
Machine Learning in Public Governance: A Systematic Review of Applications, Trends and Challenges
Nuruly, Y., Sansyzbayeva, G., Ashirbekova, L., & Tazhiyeva, S. (2025) · The economy: strategy and practice · doi:10.51176/1997-9967-2025-2-85-103 · Cited by 2
Impact of Artificial Intelligence in Audit. Bibliometric Analysis
Florea, E. C. B., Preda, M., & Olteanu, A. L. (2025) · Proceedings of the ... International Conference on Business Excellence · doi:10.2478/picbe-2025-0010 · Cited by 2
Igoche, B. I., Matthew, O., & Olabanji, D. (2025) · Knowledge · doi:10.3390/knowledge5030015 · Cited by 4
“To counter these risks, institutions need to establish ethical governance frameworks that include algorithmic audits, explainability protocols such as Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP) and stakeholder feedback mechanisms [23–26].”
AI for Academic Research: Transforming the Research Landscape
Rischard, M. W., Crevar, A. R., & Dubey, P. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-5097-4.ch006 · Cited by 2
Орлов, О. (2025) · Public administration aspects · doi:10.15421/152520 · Cited by 10
AI for Comprehensive Naval Project Management: From Render to Launch
Cortés, D. M., & León, K. (2025) · Ciencia y tecnología de buques · doi:10.25043/19098642.276 · Cited by 1
“• Audit trails documenting human intervention in the decision process, ensuring traceability and accountability at each stage [114].”
Ethical Entrepreneurship: Navigating the Path to Responsible Business
Kayyali, M. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1107-4.ch007 · Cited by 1
Balancing Automation and Trust: The Future of Digital Insurance in African Markets
Anning‐Dorson, T. (2025) · Springer eBooks · doi:10.1007/978-3-031-97210-2_12 · Cited by 2
AI and HR: Exploring the Ethical Implications of Algorithmic Hiring
Mahapatra, T. (2025) · Training and Development Journal · doi:10.5958/2231-069x.2025.00001.8 · Cited by 1
Blockchain as a Governance Layer for AGI Ethics
Saxena, D. N. (2025) · Scientific Journal of Artificial Intelligence and Blockchain Technologies · doi:10.63345/sjaibt.v2.i1.110 · Cited by 2
Responsible Data Practices and Generative AI: Organizational, Legal, and Technical Perspectives
Chowdhury, R. (2025) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780198940272.013.0003 · Cited by 2
Algocracy and Capillaries of Power
Muhammady, I. G. E., & Rizkika, H. L. (2025) · Jurnal Penelitian Ilmu Sosial dan Eksakta · doi:10.47134/trilogi.v5i1.1699 · Cited by 1
PREDICTIVE MODELING FOR THE DETECTION OF ANOMALIES IN PUBLIC CONTRACTS
de Sousa, S. B. L. (2025) · Revista de Geopolítica · doi:10.56238/revgeov16n5-072 · Cited by 1
“Hence, the use of fairness metrics and algorithmic auditing protocols (Raji et al., 2020) should accompany any deployment to avoid reinforcing inequities in the public procurement system.”
Ethical Considerations in Requirements Engineering for AI Applications
Ramachandran, M. (2025) · Springer eBooks · doi:10.1007/978-981-95-2909-4_7 · Cited by 1
AI Ethics by Design with AI Reference Architecture: Embedding Ethics by Design in AI Architecture
Ramachandran, M. (2025) · Springer eBooks · doi:10.1007/978-981-95-2909-4_9 · Cited by 1
System Card+: Responsible AI Framework for Decision Support Systems
Tibebu, H., & Kakadiaris, I. A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-8687-2_30 · Cited by 1
Understanding Sensitive Attribute Association Bias in Recommendation Embedding Algorithms
Beattie, L., Corpus, I., Lin, L. H., & Ravichandran, P. (2026) · ACM Transactions on Recommender Systems · doi:10.1145/3777548 · Cited by 4
“Broad system-level frameworks such as łSMACTRž or łSIIMž provide essential structure to approaching the ambiguous problem space of auditing for harms and bias in industry systems [ 7, 48].”
The future of productivity: digital surrogacy
Aeon, B. (2025) · AI & Society · doi:10.1007/s00146-025-02791-z · Cited by 1
Auditoria i control dels sistemes d’intel·ligència artificial d’alt risc al sector públic
Pérez, J. C. (2025) · Revista Catalana de Dret Públic · doi:10.58992/rcdp.i71.2025.4506 · Cited by 1
“En la línia de Raji et al. (2020) i Zenda (2022), es pot manllevar dels estudis mediambientals el concepte de governança urgent, que distingeix entre auditar la fiabilitat del sistema i els perjudicis socials (Raji et al., 2020).”
InfluentialA development framework for human work integrated AI systems in manufacturing
Yamamoto, Y., Muñoz, Á. A., & Sandström, K. (2025) · Frontiers in Manufacturing Technology · doi:10.3389/fmtec.2025.1601903 · Cited by 1
A review of ethical AI frameworks in product development: taking stock and moving forward
Wecker, L., Nielsen, J. A., Nasrollahi, K., & Ploug, T. (2025) · AI and Ethics · doi:10.1007/s43681-025-00952-w · Cited by 2
From Innovation to Impact the Future Ethical Landscape and Governance of Large Language Models
Rehman, S. U. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-5017-2.ch014 · Cited by 1
Min, C.-H., Lee, D.-G., & Kwak, J. (2026) · Electronics · doi:10.3390/electronics15020307 · Cited by 1
Engineering Data Architectures for AI/ML Integration in Regulated Manufacturing
Shubina, V., Ranti, T., Juppo, A. M., & Mäkilä, T. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-14518-5_5 · Cited by 2
Unified Federated AI Framework for Credit Scoring
Kaarat, A., Batthula, V. J. R., & Segall, R. S. (2026) · International Journal of Artificial Intelligence · doi:10.4018/ijaibm.398859 · Cited by 2
Resilient Systems: AI-Mediated Communication and Frontline Public Safety
Barrett, J. R., & Korostelina, K. V. (2026) · Sustainability · doi:10.3390/su18021071 · Cited by 1
Chinnaraju, A., & Loganathan, K. A. (2026) · International Journal of Latest Technology in Engineering Management & Applied Science · doi:10.51583/ijltemas.2026.150100018 · Cited by 6
AI Governance for Equitable Urban Resilience
Sharma, V. C., Babber, K., & Tyagi, P. K. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-5571-9.ch011 · Cited by 1
Shababi, H. (2026) · AI and Ethics · doi:10.1007/s43681-026-00986-8 · Cited by 1
Ethics and Compliance: Navigating Data Privacy in AI Applications
Awasthi, A. P., Singh, N. K., & Mondal, B. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-5238-1.ch012 · Cited by 1
A speculative realist typology of AI fairness: an Object-Oriented Onto-Ethics
Dhrubo, A. M., & Brohi, A. A. (2026) · AI and Ethics · doi:10.1007/s43681-026-00982-y · Cited by 1
The ethics of AI: Pursuing accountability for assured data to make right decisions
Sayyadi, M. (2026) · Business Information Review · doi:10.1177/02663821261427241 · Cited by 1
“Boards should mandate testing procedures to minimize harm and identify potential biases or failures before deploying systems (Raji et al., 2020).”
Public acceptance of various ways to use and treat AI: A predictive moral framework
Eriksson, K., Karlsson, S., Vartanova, I., & Strimling, P. (2026) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2026.100257 · Cited by 5
Can generative artificial intelligence be considered a cognitive subject? An analytic analysis
Gouveia, S. S., & Wang, Y. (2026) · AI & Society · doi:10.1007/s00146-026-02924-y · Cited by 4
“For example, Raji et al. (2020) propose internal algorithmic auditing as a way of reducing accountability gaps by embedding evaluation and documentation across the system lifecycle.”
Reconsidering the Rules: Legal and Ethical Frameworks for the Governance of Foundation Models
Ranjan, K. P. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0000-7.ch005 · Cited by 1
Mind Games: Decoding the Subconscious Triggers of Manipulation Using Consumer Neuroscience and AI
Iorga, A. M., Berceanu, D. C.-A., & Lepar, A. M. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1152-4.ch012 · Cited by 1
GBP-Audit: AI Bias Intervention with Built-In Safety-First Guardrails
Doku, R., Osafo, N. Y. A., & Rawat, D. B. (2025) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi67134.2025.00052 · Cited by 1
“GBP-Audit adopts calibration and AUROC guardrails explicitly to implement calibration-preserving fairness edits. e) Governance, auditability, and documentation: Model Cards [39], Datasheets for Datasets [40], and internal audit frameworks [41] emphasize structured transparency.”
Watson, S., & Ennion, M. (2026) · Learning Media and Technology · doi:10.1080/17439884.2026.2642199 · Cited by 6
Harnessing AI for the Future of Talent
Kaur, G., Vats, N. K., Kaur, A., & Suri, R. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-3867-5.ch010 · Cited by 1
Brunyé, T. T., Petrimoulx, M. V., & Cantelon, J. A. (2026) · Sensors · doi:10.3390/s26072034 · Cited by 3
Ray, G., & Molla, N. (2026) · Journal of Digital Security and Forensics · doi:10.29121/digisecforensics.v3.i1.2026.92 · Cited by 1
Dadelo, S. (2026) · Applied Sciences · doi:10.3390/app16073380 · Cited by 2
Bhatia, R., & Mathur, M. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01030-8 · Cited by 2
Operationalizing Responsible AI Policies with LLMs: an End-to-End Monitoring Prototype
Zieliński, P., Kelly, M., Barreto, R. D. C., & Tahaei, M. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3799675 · Cited by 4
“…is intended as decision support rather than an automated compliance oracle: it can reduce missed requirements and help prevent non-compliant deployments that may harm users, but it cannot guarantee coverage of all relevant risks, as it is limited by policy coverage and evaluation availability [28].”
Who legitimises the AI algorithm? Leadership, volatility and the governance of algorithmic authority
Anning‐Dorson, T. (2026) · Strategy and Leadership · doi:10.1108/sl-12-2025-0453 · Cited by 1
Nwokeji, J. C., Nkwo, M. S., Ikwunne, T. A., & Yeerbo, M. (2026) · AI and Ethics · doi:10.1007/s43681-026-01151-x · Cited by 1
Pimenow, S., Pimenowa, O., Prus, P., & Zieliński, M. (2026) · Sustainability · doi:10.3390/su18104892 · Cited by 6
Beyond Bias Detection: Community Auditors and Normative Reasoning in AI Oversight CSCW006
Jackson, C., Ahmad, T., Raj, S. D., & Wu, N. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3788042 · Cited by 1
Nodipha, O. G., & Maphala, B. P. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1833850 · Cited by 4
AI Algorithmic Bias, Neurodiscrimination, and Neurorights: Towards Conceptual Clarity in NeuroAI
Wang, M. Q. (2026) · Neuroethics · doi:10.1007/s12152-026-09649-1 · Cited by 5
Бондаренко, Д. В. (2026) · Regional and Branch Economy · doi:10.47576/2949-1916.2026.4.4.021 · Cited by 1
Antoro, H., Budhijanto, D., & Somawijaya (2026) · Jurnal IUS Kajian Hukum dan Keadilan · doi:10.29303/ius.v14i1.1880 · Cited by 6
Pan, Y., Luo, C., Han, Q., & Wang, Z. (2026) · Journal of Open Innovation Technology Market and Complexity · doi:10.1016/j.joitmc.2026.100806 · Cited by 3
Beyond Explanation: Evidentiary Rights for Algorithmic Accountability
Stewart, M. (2026) · Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3805689.3806532 · Cited by 1
Data protection in the era of large language models: challenges and opportunities
Revesai, Z., & Chipfumbu, C. T. (2026) · AI and Ethics · doi:10.1007/s43681-026-01223-y · Cited by 5
Fehér, K., McKelvey, F. R., Rodgers, L., & Marinov, R. (2026) · Futures · doi:10.1016/j.futures.2026.103879 · Cited by 5
Chambers, L. M., & Davenport, D. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806542 · Cited by 1
“…technical work: approaches that foreground social and community values in design praxes [23, 41, 78, 79, 97]; documentation and reporting artifacts such as model cards [57] and datasheets [39]; and accountability-oriented practices including algorithmic audits and evaluations [51, 69, 88, 92].”
Suherman, A., Maulana, H. F., & Hasrullah, H. (2026) · Journal of Information Communication and Ethics in Society · doi:10.1108/jices-07-2025-0162 · Cited by 1
Bridging Causal Reasoning and Fuzzy Calibration for Trustworthy AI Risk Assessment
McConomy, G., Liu, J., Glass, D., Bond, R., & Toner, H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2497-5_18 · Cited by 1
León-Ariza, S. A., Orobio, M. C., Ibáñez-Gutiérrez, H. M., García-Serna, J., & Vivas, D. A. (2026) · Medical Teacher · doi:10.1080/0142159x.2026.2685794 · Cited by 4
Accessible GenAI UI Generation with Post-render Verification
Jerry, B., Moreno, L. D. L. R., & Martínez, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31285-3_47 · Cited by 1
Kwan Hong, T. A. N. (2026) · International Journal of Law Management & Humanities · doi:10.63108/ijlmh.12537 · Cited by 6
Aârab, H. (2026) · International Conference on Human Centric Artificial Intelligence (ICHCAI) · doi:10.1109/ichcai70183.2026.11607581 · Cited by 2
Latif, W. B., Mussayeva, I., Chowdhury, S., Yakub, S., Amin, M. B., Nusrat, F., & Salehin, R. (2026) · Corporate Law & Governance Review · doi:10.22495/clgrv8i3p16 · Cited by 1
Shi, J., & Sirgy, M. J. (2026) · Applied Research in Quality of Life · doi:10.1007/s11482-026-10649-9 · Cited by 2
Exploring AI governance for sustainable decision-making: evidence from six case studies
Almeida, F. L. (2026) · Journal of Ethics in Entrepreneurship and Technology · doi:10.1108/jeet-04-2026-0023 · Cited by 2
Ethically Designing Customer and Employee Experiences in the Age of AI and Knowledge Automation
Nayak, S., Malik, A., & Budhwar, P. (2026) · Australasian Marketing Journal (AMJ) · doi:10.1177/14413582261454384 · Cited by 1
Kabir-Chisty, M. (2026) · AI and Ethics · doi:10.1007/s43681-026-01296-9 · Cited by 2
VQFlow: A Benchmark Dataset for Encrypted Video Streaming Traffic across QoS Configurations
Zhao, Z., Li, Z., Li, T., & Zhang, F. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770854.3785690 · Cited by 1
“We adhere to strict ethical guidelines throughout the data collection and release process [31, 32].”
Kaya, M. (2026) · Kamu Yönetimi ve Teknoloji Dergisi · doi:10.58307/kaytek.1967200 · Cited by 1
Alibašić, H. (2026) · Systems · doi:10.3390/systems14080984 · Cited by 3
A normatively grounded hybrid model of AI governance: insights from the teaching of Leo XIV
Fel, S., Choroszewicz, M., & Kozak, J. (2026) · AI and Ethics · doi:10.1007/s43681-026-01336-4 · Cited by 1
“They must not preclude auditing or access to an effective remedy [63, 64].”
Liu, B. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1896492 · Cited by 1
Metadata
- Authors
- Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P.
- Affiliations
- Partnership on AI
- Google (United States) (US)
- ORCID
- Venue
- Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2020-01-27
- Pages
- 33-44
- OpenAlex
- W3001807593
- Cited by
- 2,245
- References
- 83
- Citations by year
- 2026: 1097 · 2025: 454 · 2024: 243 · 2023: 177 · 2022: 149 · 2021: 99 · 2020: 24
- FWCI
- 35.91
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- AI accountability, algorithmic auditing, harm detection, algorithmic transparency, societal impact of AI, algorithmic governance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.