Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing
Open accessThe DOI matches its Crossref record; 96 of the 97 references it cites were found in a registry or an index, and 1 are not checked yet; 11 of the 44 sources its claims cite were checked for support.
Abstract
Rising concern for the societal implications of artificial intelligence systems has inspired a wave of academic and journalistic literature in which deployed systems are audited for harm by investigators from outside the organizations deploying the algorithms. However, it remains challenging for practitioners to identify the harmful repercussions of their own systems prior to deployment, and, once deployed, emergent issues can become difficult or impossible to trace back to their source.
Verified 2026-11-19 · 1 reference not checked yet
Overview
CoChat Summary
TL;DR
The paper proposes SMACTR, an end-to-end internal algorithmic auditing framework intended to help organizations assess AI systems against declared ethical principles throughout development. It argues that internal audits can complement external accountability by providing pre-deployment access, structured documentation, and risk analysis drawn from aerospace, medical device, and finance practices.
Key findings
- Introduces SMACTR—Scoping, Mapping, Artifact Collection, Testing, and Reflection—as five audit stages, each with documentation requirements that together form an overall audit report and assess decisions against organizational values or AI principles.
- Argues external audits are limited by lack of access to internal processes, intermediate models, and training data, while internal audits can enable proactive pre-deployment intervention and generate artifacts for external review.
- Draws lessons from other industries, citing aerospace’s one commercial airline accident per two million flights, Boeing 787 software estimated at 13 million lines of code, and a 1-in-1,000,000,000 per use hour maximum failure probability for critical aerospace systems; also cites medical-device design controls and finance internal audit infrastructure.
- Uses hypothetical cases—a child abuse screening tool and a smile-detection photo booth—to illustrate the framework; for the smile detector’s CelebA datasheet, it reports 58.1% female and 42% male labels, 77.8% aged 0–45 and 22.1% over 46, and 14.2% lighter-skinned versus 85.8% darker-skinned.
Methods
- Conceptual framework paper: develops SMACTR from auditing practices and artifacts in aerospace, medical devices, and finance, plus responsible-innovation dimensions of anticipation, reflexivity, inclusion, and responsiveness.
- Uses a hypothetical Company X Inc. consultancy with two invented client projects—a child abuse screening tool and Happy-Go-Lucky Inc.’s smile-detection photo booth—to demonstrate audit stages and artifacts; a worked example for the smile case is provided as supplementary material.
- Analysis is narrative and design-oriented, referencing existing standards and tools such as FMEA, model cards, datasheets, design controls, and design history files; no empirical sample or quantitative evaluation is reported.
Limitations
- The paper states that internal auditors necessarily share an organizational interest with the audit target, making an independent and objective viewpoint challenging; audits are not an objective “view from nowhere” and risk becoming reputation management.
- The paper states that internal audits are only one aspect of a broader system of required quality checks and balances, and that determining what systems to audit and why is not covered and is a separate research topic.
- Evident from the text: the framework is not empirically validated, the illustrative cases are hypothetical, and no real deployment outcomes or measured audit effectiveness are reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
9 Valid · 6 Not valid · 27 Not checked · 30 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“While not without faults, safety-critical and regulated industries such as aerospace and medicine have long traditions of auditable processes and design controls that have dramatically improved safety [77,81].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 3 LESSONS FROM AUDITING PRACTICES IN OTHER INDUSTRIES
What the sources say
- Not checked — no full text of the sourcecites [77] matched to “Design controls for the medical device industry” (2013) source
- Could not verifycites [81] matched to “The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions” (2019) sourceExperimental — accuracy not yet measured
The Introduction and §3 discuss biomedical ethics and standards or regulation, but do not say that aerospace or medicine has long used auditable processes and design controls, or that these dramatically improved safety. Searches for aerospace, auditable processes, design controls, and safety-critical industries found no passage addressing that claim. Page numbers are unavailable in our copy. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This is a repeatedly observed phenomenon in tax compliance auditing, where several international surveys of tax compliance demonstrate that a fixed and vetted tax audit methodology is one of the most effective strategies to convince companies to respect audit results and pay their full taxes [22,53].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.3 Audit Integrity and Procedural Justice
What the sources say
- Partly supportedcites [22] matched to “Perception on justice, trust and tax compliance behavior in Malaysia” (2017) sourceExperimental — accuracy not yet measured
The Regression Test section reports that Malaysian respondents would be more compliant if tax authorities treated them fairly in tax procedures. The Research Methodology section describes a questionnaire of academics, not several international surveys of companies. The paper does not establish that a fixed, vetted audit methodology is especially effective at securing acceptance of audit results or full tax payment. (LLM, conf=0.86)
- Partly supportedcites [53]Experimental — accuracy not yet measured
The “Justice in the taxation context” section reports that taxpayers are generally more compliant when treated fairly and respectfully, and the General discussion links perceived unfair treatment to resistance to paying back tax (page numbers unavailable). It does not establish that a fixed, vetted audit methodology is among the most effective strategies, or report international surveys demonstrating that effect for companies. (LLM, conf=0.83)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Traditional metrics for artificial intelligence like loss may conceal fairness concerns, social impact risks or abstraction errors [69].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.3.2 Artifact: Ethnographic Field Study.
What the sources say
- Supportedcites [69] matched to “Fairness and Abstraction in Sociotechnical Systems” (2019) source
The Introduction (page not provided) says that narrow assessments of machine-learning performance abstract away social context and can miss information needed for fair outcomes—an abstraction error. The Formalism Trap section adds that mathematical measures cannot capture the full range of fairness concerns, while the Takeaways discuss unintended social consequences. The paper names AUC, precision and recall rather than loss, but the claim uses loss as an illustrative metric. (LLM, conf=0.88)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Internal auditors' direct access to systems can thus help extend traditional external auditing paradigms by incorporating additional information typically unavailable for external evaluations to reveal previously unidentifiable risks.”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.4 The Internal Audit
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The essential inclusion of independent domain experts and marginalized groups in the ethical review process “has the potential to lead to more rigorous critical reflection because their experiences will often be precisely those that are most needed in identifying problematic background assumptions and revealing limitations with research questions, models, or methodologies” [33].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.2.1 Artifact: Ethical Review of System Use Case.
What the sources say
- Not checked — no full text of the sourcecites [33] matched to “25 Years of Feminist Empiricism and Standpoint Theory: Where Are We Now?” (2010) source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
19 more claims
Create a free account to see all 72 claims and every source verdict.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 37
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This disparity poses risks, as practitioners may unintentionally embed their own biases into AI system design and implementation (Martin 2019;Raji et al. 2020).”
From Accountability in Government Use of AI : Citizen Concerns and Preferences (2025) · cites this paper as Raji et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Work on AI safety and reproducibility has long pointed out the crucial importance of training data for understanding model performance, ensuring reproducibility, and assessing legal exposure [2, 25, 29, 46, 58].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While there is a growing interest in auditing algorithmic systems [125,126,128,138,154], this feld generally lacks defned standards and records could be incomplete or not relating to information needed for accountability purposes [15,127].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [128]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Raji et al. (2020) proposed an internal algorithmic auditing framework spanning data collection, model development, testing, deployment, and monitoring, with stage-specific artifacts that enable traceability and accountability.”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Raji et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Practical approaches to implement such ideas typically revolve around the systematic collection and curation of data on various aspects of systems [2, 28, 46].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747 · cited 1×
“A hypothetical datasheet for this system is filled out using studies on the CelebA dataset, with which the smile detector is built [44, 47].”
MethodThe global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Product development: Past research, present findings and future directions
Brown, S. L., & Eisenhardt, K. M. (1995) · The Academy of Management Review · doi:10.2307/258850 · Cited by 3,777 · cited 1×
“There are also intersections between this framework and just efective product development theory [5], as many of the components of audit design refocus the product development process to prioritize the user and their ultimate well-being, resulting in a more efective product performance outcome.”
Background(2018) · Cited by 3,510
Christiane, Druml, Wolzt, & Pleiner (2001) · Acta Scientiarum Naturalium Universitatis Sunyatseni · Cited by 3,419
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196
Developing a framework for responsible innovation
Stilgoe, J., Owen, R., & Macnaghten, P. (2013) · Research Policy · doi:10.1016/j.respol.2013.05.008 · Cited by 3,184 · cited 1×
“To design our audit procedure, we suggest complementing formal risk assessment methodologies with ideas from responsible innovation, which stresses four key dimensions: anticipation, reflexivity, inclusion and responsiveness [73], as well as system-theoretic concepts that help grapple with increasing complexity and co…”
MethodAlgorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Self-citation(2014) · Journal of the Korean Medical Association · doi:10.5124/jkma.2014.57.11.899 · Cited by 2,566
Research works printed but not linked: 7
- Virginia Eubanks. 2018. A child abuse prediction model fails poor families. Wired Magazine (2018).
- Jeremy Goldhaber-Fiebert and Lea Prince. 2019. Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny CountyâĂŹs Child Welfare Office. Pittsburgh: Allegheny County.[Google Scholar] (2019).
- Nitin Kohli, Renata Barreto, and Joshua A Kroll. 2018. Translation Tutorial: A Shared Lexicon for Research and Practice in Human-Centered Software Systems. In 1st Conference on Fairness, Accountability, and Transparancy. New York, NY, USA. 7.
- Joshua A Kroll, Solon Barocas, Edward W Felten, Joel R Reidenberg, David G Robinson, and Harlan Yu. 2016. Accountable algorithms. U. Pa. L. Rev. 165 (2016), 633.
- Institute of Internal Auditors. Research Foundation and Institute of Internal Au- ditors. 2007. The Professional Practices Framework. Inst of Internal Auditors.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22 (2014).
- Tim Vanderveen. 2005. Averting highest-risk errors is first priority. Patient Safety and Quality Healthcare 2 (2005), 16-21.
Cited by
Items 401–500 of 1,412
Towards a Non-Ideal Methodological Framework for Responsible ML
Mothilal, R. K., Guha, S., & Ahmed, S. I. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642501 · Cited by 16
“To address these gaps in theorizing and understanding practitioners’ methodological approaches, we ask the following three broad research questions in our study:”
BackgroundMachine Learning Advancing Diversity Equity and Inclusion in Data-Driven HR Practices
Muralidhar, L. B., Sathyanarayana, N., Swapna, H. R., Hukkeri, S. V., & Sultana, P. H. R. (2025) · Advances in human resources management and organizational development book series · doi:10.4018/979-8-3373-0149-5.ch019 · Cited by 8
When AI is wrong: addressing liability challenges in women's healthcare
Marotta, A. (2022) · Journal of Computer Information Systems · doi:10.1080/08874417.2022.2089773 · Cited by 7
Safety Issues in Human-Machine Collaboration and Possible Countermeasures
Ma, L., & Wang, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-05890-5_21 · Cited by 10
The goldilocks framework: towards selecting the optimal approach to conducting AI projects
Джусупова, Р., Bosch, J., & Olsson, H. H. (2022) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3522664.3528595 · Cited by 10
Recommender Systems and Autonomy: A Role for Regulation of Design, Rights, and Transparency
Djeffal, C., Hitrova, C., & Magrani, E. (2021) · Indian Journal of Law and Technology · doi:10.55496/jgdn9214 · Cited by 10
Pierson, C. M., & Hildt, E. (2023) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.792 · Cited by 7
“Similar mechanisms have been proposed, such as embedding ethicists in the design process (van Wynsberghe & Robbins, 2014), ethical foresight analysis (Floridi & Strait, 2020), or algorithmic auditing (Raji et al, 2020).”
FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines
Barker, M., Kallina, E., Ashok, D., Collins, K. M., Casovan, A., Weller, A., Talwalkar, A., Chen, V., & Bhatt, U. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623239 · Cited by 15
Responsible AI for Trusted AI-powered Enterprise Platforms
Hoi, S. C. H. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3575784 · Cited by 10
Beyond Fairness Metrics: Roadblocks and Challenges for Ethical AI in PracticePreprint
Chen, J., Storchan, V., & Kurshan, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.06217 · Cited by 34
Measurement as governance in and for responsible AIPreprint
Jacobs, A. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05658 · Cited by 24
Law and the Emerging Political Economy of Algorithmic AuditsPreprint
Terzis, P., Veale, M., & Gaumann, N. (2024) · doi:10.31219/osf.io/xvqz7 · Cited by 7
Gautam, S., Chandra, M., De, A., Chakravorti, T., Malik, G., & De Choudhury, M. (2025) · arXiv · doi:10.48550/arxiv.2508.06849 · Cited by 7
Toward Data Sovereignty: Justice-oriented and Community-based AI Education
Moudgalya, S. K., & Swaminathan, S. (2024) · Research on Equity and Sustained Participation in Engineering, Computing, and Technology (RESPECT) · doi:10.1145/3653666.3656107 · Cited by 14
“Attempts to address these dangers include algorithmic auditing [17].”
BackgroundAbdu, A. A., Chambers, L. M., Mulligan, D., & Jacobs, A. Z. (2024) · Conference on Fairness, Accountability and Transparency · doi:10.1145/3630106.3658962 · Cited by 12
“Transparency efforts have called for visibility around choices about data, processes, or mathematical properties of algorithms [67], e.g., through documentation of the development process [39, 61, 102] or structured disclosures of data properties [51, 57, 88].”
BackgroundFormalising Anti-Discrimination Law in Automated Decision Systems
Sargeant, H., & Magnusson, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732015 · Cited by 17
“The common conflation of fairness and legal non-discrimination has led to a predominant focus on outcome disparity as the main indicator of unfairness, yet unlawful discrimination is both broader and more detailed.”
Sivakaminathan, S. S., & Musi, E. (2025) · AI & Society · doi:10.1007/s00146-025-02564-8 · Cited by 22
A Review of Ethical Challenges in AI for Emergency Management
Yuan, X., Guo, Q., Dadson, Y. A., Goodarzi, M., Jung, J., Dong, Y., Albert, N., Gayle, D. B., Sharma, P., Ogunbayo, O. T., et al. (2025) · Knowledge · doi:10.3390/knowledge5030021 · Cited by 9
“Many AI systems, specifically deep learning models, are like black boxes, making it hard to understand how decisions are made or to figure out who’s responsible when something goes wrong [105], to explain the underlying decision-making processes and output rationale [76,106].”
AI and Diversity: Ensuring Fairness in Automated Decision-Making
Subrahmanyam, S., & Baz, B. E. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-3104-1.ch002 · Cited by 6
Evolving from Rules to Learning in Urban Modeling and Planning Support Systems
Cai, Z. (2025) · Urban Science · doi:10.3390/urbansci9120508 · Cited by 9
Allen, L. K., DeLiema, D., & Kendeou, P. (2025) · Educational Psychologist · doi:10.1080/00461520.2025.2588213 · Cited by 10
“Whereas software engineers maintain access to these algorithms for internal audits and adjustments (Keyes & Austin, 2022; Raji et al., 2020), ordinary users can only speculate about how they function based on their search outputs or news feeds.”
Rahman, M. I. (2026) · Nurse Education in Practice · doi:10.1016/j.nepr.2026.104756 · Cited by 13
Fairness, explainability, privacy, and robustness for trustworthy algorithmic decision-making
Majumdar, S. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-85713-0.00017-7 · Cited by 10
Montasari, R. (2024) · Advanced sciences and technologies for security applications · doi:10.1007/978-3-031-50454-9_10 · Cited by 10
Gammelgård-Larsen, A., van Berkel, N., Skov, M. B., & Kjeldskov, J. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650886 · Cited by 8
Employing Hybrid AI Systems to Trace and Document Bias in ML Pipelines
Russo, M., Chudasama, Y., Purohit, D., Sawischa, S. F., & Vidal, M.-E. (2024) · IEEE Access · doi:10.1109/access.2024.3427388 · Cited by 11
“…the importance of operating under practices that enable others to easily see the actions performed by the professionals and teams working at all steps of the ML pipeline [12], [13], [14], while also making an urgent call for data stewardship [15] and responsible data management practices [7].”
BackgroundSoftware Systems Compliance with the AI Act: Lessons Learned from an International Challenge
Scantamburlo, T., Falcarin, P., Veneri, A., Fabris, A., Gallese, C., Billa, V., Rotolo, F., & Marcuzzi, F. (2024) · doi:10.1145/3643691.3648589 · Cited by 14
“For example, SMACTR is an internal audit framework used by the industry encompassing distinct stages: Scoping, Mapping, Artifact Collection, Testing, Reflection, and Post-Audit [55].”
MethodThe Digital Siren’s Call: Accepting Unethical AI Advice
Xu, C., Xu, H., Sun, Y., & Xiong, W. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2400396 · Cited by 8
“However, the deployment of these AI models also presents new challenges, including the need for comprehensive data governance to prevent bias and ensure privacy (Binns, 2018; Raji et al., 2020).”
AI in the Assessment Ecosystem: A Human–Centered AI Perspective
von Davier, A. A., & Burstein, J. C. (2024) · Intelligent systems reference library · doi:10.1007/978-3-031-71232-6_6 · Cited by 15
Kumar, A. B., Sanjaya, K., & Saleem, M. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-9395-6.ch009 · Cited by 9
Toward empowering AI governance with redress mechanisms
Pi, Y., & Proctor, M. (2025) · Cambridge Forum on AI Law and Governance · doi:10.1017/cfl.2025.9 · Cited by 20
“Defining accountability – who is responsible or answerable for an AI system, its behavior and its potential impacts – is crucial for those seeking redress (Raji et al., 2020).”
Boykin, C. M., Dasch, S. T., Rice, V., Lakshminarayanan, V. R., Togun, T. A., & Brown, S. M. (2021) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3465416.3483302 · Cited by 11
The Self-Synchronisation of AI Ethical Principles
Light, R., & Panai, E. (2022) · Digital Society · doi:10.1007/s44206-022-00023-1 · Cited by 8
“Ethicists must conceive of audits as tools to develop an ethical environment because “audits are tools for interrogating complex processes, often to determine whether they comply with company policy, industry standards or regulations” (Raji et al., 2020, p. 34).”
BackgroundMethodAssessing the Impact of Cognitive Biases in AI Project Development
Bernault, C., Juan, S., Delmas, A., André, J., Rodier, M., & Kaadoud, I. C. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35891-3_24 · Cited by 11
Human Expectations and Perceptions of Learning in Machine Teaching
Mishra, S., & Rzeszotarski, J. M. (2023) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3565472.3595612 · Cited by 8
Developing responsible AI practices at the Smithsonian Institution
Dikow, R. B., DiPietro, C., Trizna, M. G., BredenbeckCorp, H., Bursell, M. G., Ekwealor, J. T. B., Hodel, R. G. J., Lopez, N., Mattingly, W., Munro, J., et al. (2023) · Research Ideas and Outcomes · doi:10.3897/rio.9.e113334 · Cited by 11
A comprehensive review of safety audits: ensuring workplace safety and compliance in Zimbabwe
Shabani, T., Jerie, S., & Shabani, T. (2024) · Safety and Reliability · doi:10.1080/09617353.2024.2343958 · Cited by 15
“…gaps or weaknesses in their safety management systems and ensure that established processes are being followed consistently (chowdhury & shil, 2019; Raji et al., 2020). this proposes that process audits gives insights on how safety policies are integrated into daily operations within the…”
Data Portability as a Tool for Audit
Zwiebelmann, Z., & Henderson, T. (2021) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3460418.3479343 · Cited by 9
Eitzel, M. V. (2021) · Research Ideas and Outcomes · doi:10.3897/rio.7.e71553 · Cited by 8
Futuristic Technologies for Smart Manufacturing
Tyagi, A. K., Mishra, A. K., Vedavathi, N., Kakulapati, V., & Sajidha, S. A. (2023) · Wiley eBooks · doi:10.1002/9781394213948.ch21 · Cited by 6
Ethical Considerations in AI-Generated Content on Social Media
Medhat, M., Ayoub, L. W., Daher, M., & Mohamed, K. M. (2025) · Studies in Big Data · doi:10.1007/978-3-031-83911-5_52 · Cited by 5
AI-driven data governance for smart cities: balancing privacy, efficiency, and public trust
Mastrogiovanni, S. (2025) · South Florida Journal of Development · doi:10.46932/sfjdv6n9-029 · Cited by 5
Sustainable Outsourcing in an AI-Driven World: Ethical, Economic, and Environmental Perspectives
Kumar, M., Rani, P., Mangla, N., Nguyen, N. T., & Kaur, K. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1270-5.ch001 · Cited by 6
The future of open human feedback
Don-Yehiya, S., Burtenshaw, B., Astudillo, R. F., Osborne, C., Jaiswal, M., Kuo, T.-S., Zhao, W., Shenfeld, I., Peng, A., Yurochkin, M., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01038-2 · Cited by 23
“This includes removing personally identifiable information, such as names, emails and IP addresses 25,119 , which can be especially challenging when interleaved in a chat 120 and because personally identifiable information definitions vary by application 117 .”
El‐Bassel, N., David, J., Mukherjee, T. I., Aggarwal, M., Wu, E., Gilbert, L., Walters, S. T., Chandler, R. K., Hunt, T., Frye, V. A., et al. (2025) · Implementation Science · doi:10.1186/s13012-025-01447-2 · Cited by 11
“To mitigate algorithmic bias, we propose dataset diversification, subgroup performance evaluation, and participatory bias audits in collaboration with community stakeholders [34, 66].”
Alam, N. (2025) · Journal of Evidence-Based Social Work · doi:10.1080/26408066.2025.2547219 · Cited by 10
Building Trustworthy AI: Transparency, Fairness, and Governance in the Digital Age
Liu, E. (2025) · AI & Future Society · doi:10.63802/afs.v1.i1.80 · Cited by 9
Mapping Stakeholder Needs to Multi-Sided Fairness in Candidate Recommendation for Algorithmic Hiring
Kaya, M., & Bogers, T. (2025) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3705328.3748079 · Cited by 10
Huang, C., & Yao, X. (2025) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2025.3593452 · Cited by 7
Is there consistency in ethical sensitivity in artificial intelligence? A review of language models
Tutar, H., Bilan, S., & Şentürk, Ü. (2025) · Human Technology · doi:10.14254/1795-6889.2025.21-2.4 · Cited by 5
“In this context, artificial intelligence systems should not be used in critical decision support systems without testing their ethical sensitivity level (Raji et al., 2020).”
Influentialden Bulck, H. V., Kulig, M., Hyzen, A., Puppis, M., & Paulussen, S. (2025) · European Journal of Communication · doi:10.1177/02673231251385761 · Cited by 10
“Following Raji et al. ’ s (2020) approach, we fi rst clarify the objective behind each standard: promoting the dissemination of and access to epistemically valuable content (reliability), that answers the greatest number of questions (power), to the broadest and most diverse audiences (fecundity),…”
Jarrahi, M. H., Karami, A., Conway, P., Memariani, A., & Lutz, C. (2025) · Technology in Society · doi:10.1016/j.techsoc.2025.103127 · Cited by 10
“These mechanisms should involve technical error detection, feedback from multiple stakeholders, and assessment impacts that reflect how bias evolves over time and downstream in dynamic organizational and social contexts (Raji et al., 2020).”
Brintrup, A. M., Baryannis, G., Tiwari, A., Ratchev, S. M., Martínez-Arellano, G., & Singh, J. (2025) · Data-Centric Engineering · doi:10.1017/dce.2025.10032 · Cited by 47
“Note, however, that there is a clear realisation that issues of trustworthy AI are inherently socio-technical (Kroll et al., 2017; Raji et al., 2020), and require a con-sideration of technical, organisational, and human processes aspects, throughout the technology development, operation and use (Cobbe, Lee, and Singh,…”
Formal and computational foundations for implementing Affective Sovereignty in emotion AI systems
Kim, R. S. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01000-0 · Cited by 8
Abass, S. A. E., Altamimi, A. S. A., Aldakhil, T., & Prajapati, P. (2026) · Critical Reviews in Analytical Chemistry · doi:10.1080/10408347.2026.2694610 · Cited by 6
Harrison, G., Bryson, K., Bamba, A. E. B., Dovichi, L., Binion, A. H., Borem, A., & Ur, B. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642755 · Cited by 11
“While the specific procedures vary [40, 47, 50], these approaches rely on somebody with the power to inspect a model and decide whether it should be deployed.”
BackgroundReinmund, T., Kunze, L., & Jirotka, M. D. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642247 · Cited by 9
“…how accuracy of the ML system will be monitored and maintained, including over time and across sub-groups of the target population, how deviations from expected system behaviour will be identified and addressed, and what steps will be taken to comply with regional data protection regulation [79].”
Authenticity in algorithm-aided decision-making
Karlan, B. (2024) · Synthese · doi:10.1007/s11229-024-04716-7 · Cited by 5
“If Hallie were made aware of algorithmic audits (e.g. Raji et al. 2020), especially the poor audit fairness scores for some resume filtering algorithms, she could avoid or otherwise contextualize the output of the algorithm in her final decision.”
Ethical Concerns With Regards to Artificial Intelligence: A National Public Poll in Taiwan
Chang, W. L.-Y., Liao, Y.-K., Chao, E., Liu, S.-Y., & Lee, T. S. (2024) · IEEE Access · doi:10.1109/access.2024.3458893 · Cited by 12
“Some researchers suggested that accountability deals with the uncertainty around who is responsible for AI-caused outcomes, which needs a holistic framework encompassing many ethics principles to guide system designs and professional training programs [21], [22], [23].”
BackgroundDoing Responsibilities with Automated Grading Systems: An Empirical Multi-Stakeholder Exploration
Figueras, C., Rossitto, C., & Pargman, T. C. (2024) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3679318.3685334 · Cited by 8
TAPS Responsibility matrix: a tool for responsible data science by design
Urovi, V., Çelebi, R., Sun, C., Kumar, P., Rieswijk, L., Erard, M., Yılmaz, A., Moodley, K., & Dumontier, M. (2024) · Journal of Responsible Innovation · doi:10.1080/23299460.2024.2414530 · Cited by 8
Ethics, Algorithms, and the Rules of Evidence: New Era of AI-Driven Forensics
Kumar, A. B., & Sanjaya, K. (2025) · IGI Global eBooks · doi:10.4018/979-8-3693-9405-2.ch006 · Cited by 7
Pinning down an octopus: towards an operational definition of AI systems in the EU AI Act
Tomić, S., & Štimac, V. (2025) · Journal of European Public Policy · doi:10.1080/13501763.2025.2534648 · Cited by 11
Schedl, M., Gómez, E., & Lex, E. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532683 · Cited by 10
Aboukadri, S., Ouaddah, A., & Mezrioui, A. (2022) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-031-20601-6_5 · Cited by 5
Embedding Ethics and Trustworthiness for Sustainable AI in Earth Sciences: Where Do We Begin?
Dias, P. A., & Lunga, D. (2022) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss46834.2022.9883030 · Cited by 5
“[12], proposed to help close the so-called accountability gap in development/deployment of AI systems by ensuring enough information is collected at each stage to support auditing across all steps involved in system development.”
BackgroundFairness as adequacy: a sociotechnical view on model evaluation in machine learning
Grote, T. (2023) · AI and Ethics · doi:10.1007/s43681-023-00280-x · Cited by 9
“Second, while there are different proposals in the ML community by now for holistic model auditings, these largely shy away from prescriptive methodological considerations (see [66].”
Data Recording for Responsible Robotics
Škerlj, J., Braun, M., Witz, S., Breuer, S., Bak, M., Scholz, S., Naceri, A., Müller, R., Haddadin, S., & Eisenberger, I. (2023) · IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO) · doi:10.1109/arso56563.2023.10187414 · Cited by 7
Rethinking Machine Learning Benchmarks in the Context of Professional Codes of Conduct
Henderson, P., Hu, J., Diab, M., & Pineau, J. (2024) · ACM Symposium on Computer Science and Law (CSLAW) · doi:10.1145/3614407.3643708 · Cited by 8
“…et al. 2019], Datasheets [Gebru et al. 2021], Reward Reports [Gilbert et al. 2022], Holistic Assessments [Liang et al. 2022], Internal Audits [Raji et al. 2020], and others, have sought to improve transparency into the limitations of language-based foundation models (including translation…”
BackgroundAI and Equity in Higher Education: Ensuring Inclusivity in the Algorithmic Classroom
Diene, A. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-1565-1.ch001 · Cited by 9
Fairness Certification for Natural Language Processing and Large Language Models
Freiberger, V., & Buchmann, E. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-66329-1_39 · Cited by 13
Estrategias de diversidad e inclusión en procesos de reclutamiento asistidos por IA
Jara-Álvarez, G. X., Salgado-Ortiz, P. J., Santander-Salmon, E. S., & Quiñónez-Cabeza, B. M. (2024) · Revista Científica Ciencia y Método · doi:10.55813/gaea/rcym/v2/n4/53 · Cited by 7
Muslihatun, F. A. N., Hantono, B. S., & Fauziati, S. (2021) · International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE) · doi:10.1109/icitisee53823.2021.9655975 · Cited by 14
AccTEF: A Transparency and Accountability Evaluation Framework for Ontology-Based Systems
Basereh, M., Caputo, A., & Brennan, R. (2022) · International Journal of Semantic Computing · doi:10.1142/s1793351x22400013 · Cited by 7
Building a Culture of Safety for AI: Perspectives and ChallengesPreprint
Manheim, D. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4491421 · Cited by 18
Pushing Boundaries or Crossing Lines? The Complex Ethics of ChatGPT JailbreakingPreprint
Boxleitner, A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4604736 · Cited by 6
A Blueprint for Auditing Generative AIPreprint
Mökander, J., Curl, J., & Kshirsagar, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4887768 · Cited by 7
Seeds of Sovereignty: Designing Regenerative AI for Plural EpistemologiesPreprint
Kadel, A. (2025) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/f9e65_v1 · Cited by 6
Sridharan, V. (2025) · Journal of Computer Science and Technology Studies · doi:10.32996/jcsts.2025.7.5.56 · Cited by 5
Talent Analytics for Workforce Forecasting, Fostering Employee Engagement, and Turnover Prediction
Sanjay, K., & Khalsa, S. K. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-1942-1.ch017 · Cited by 4
Regulating Emotion AI in the United States: Insights from Empirical Inquiry
Ingber, A. S., & Andalibi, N. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732014 · Cited by 6
“However, audits are both challenging [18, 90] and raise ethical questions, particularly when they concern biometric data [89].”
Algorithmic Governance: Gender Bias in AI-Generated Policymaking?
Voutyrakou, D. A., & Skordoulis, C. D. (2025) · Human-Centric Intelligent Systems · doi:10.1007/s44230-025-00109-2 · Cited by 12
Adam, S. H., Eltahir, I. A. E., Alnor, N. H. A., Taha, M. A., & Musa, E. H. E. (2025) · WSEAS TRANSACTIONS ON BUSINESS AND ECONOMICS · doi:10.37394/23207.2025.22.184 · Cited by 7
Velotto, G. (2025) · AI and Ethics · doi:10.1007/s43681-025-00880-9 · Cited by 4
“Audit practice translates these lessons into mechanisms—impact assessments, model cards/datasheets, red-teaming, and recurring post-deploy - ment audits with documented findings and responses [ 81 , 104].”
InfluentialEvaluating the Social Impact of Generative AI Systems
Solaiman, I., Talat, Z., Agnew, W. S., Ahmad, L., Baker, D., Blodgett, S. L., Chen, C., Daumé, H., Dodge, J., Duan, I., et al. (2025) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780198940272.013.0025 · Cited by 10
“, [190]) are powerful tools that necessarily depend on the sector of deployment.”
BackgroundSelf-citationVasiliki, M., Serdaris, P., Antoniadis, I. I., & Spinthiropoulos, K. G. (2025) · Digital · doi:10.3390/digital6010001 · Cited by 7
Yadav, A. (2025) · International Review of Law Computers & Technology · doi:10.1080/13600869.2025.2602108 · Cited by 4
Public Perceptions of Algorithmic Bias and Fairness in Cloud-Based Decision Systems
Alhosban, A., Gaire, R., & Al-Ababneh, H. A. (2025) · Standards · doi:10.3390/standards6010002 · Cited by 4
Emotionally based strategic communications as a new tool in defensive cognitive warfare
Ćosić, K., Popović, S., & Wiederhold, B. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1751406 · Cited by 4
Gupta, A. K., Mishra, S. S., Priyanka, P., & Verma, A. (2025) · Global AI Summit - International Conference on Artificial Intelligence and Emerging Technology (AI Summit) · doi:10.1109/aisummit66170.2025.11410968 · Cited by 8
Frimpong, V., & Botchey, O. K. (2026) · Businesses · doi:10.3390/businesses6020018 · Cited by 6
“Both industry and academia emphasise the importance of algorithmic auditing and responsible AI practices, underscoring the need for governance tools such as auditability, documentation, and accountability frameworks (Raji et al., 2020; Koshiyama et al., 2024).”
Exploring the Intersection of AI, Language, and Law: A Bibliometric Analysis
Lan, Z., & Zhang, X. (2026) · SAGE Open · doi:10.1177/21582440261431000 · Cited by 10
“…model cards with disaggregated performance reporting across jurisdictions and languages, and end-to-end internal algorithmic auditing that links design choices to harm hypotheses and remediation plans (Bender & Friedman, 2018; Gebru et al., 2021; Mitchell et al., 2019; Raji et al., 2020).”
InfluentialAlgorithms, policing, and race
Ugwudike, P. (2023) · Routledge eBooks · doi:10.4324/9781003176619-16 · Cited by 9
Impact Assessment as a Legal Design Pattern—A “Timeless Way” of Managing Future Risks?
Wernick, A. (2024) · Digital Society · doi:10.1007/s44206-024-00111-4 · Cited by 12
“The challenges may concern predicting the impact of AI-based technology throughout its production (Raji et al., 2020 ) and its life cycle, considering the contextual factors (Wieringa, 2020 ).”
InfluentialAhmed, H. O. K. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20307-7 · Cited by 4
“Continuous development and adaptation to changing language and cultural dynamics are made possible by this iterative approach [44].”
Ethical Considerations in the Use of AI and Big Data in Corporate Decision-Making
Sharma, G. K., Tyagi, E., & Chaudhary, A. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-9015-3.ch017 · Cited by 4
Ferguson, B., Baldwin, A., Harvey, C. L., & Henderson, A. J. (2025) · Nursing Inquiry · doi:10.1111/nin.70001 · Cited by 6
“AI systems require careful design to be sensitive to human inputs and incorporate them into the risk assessment process (Leslie 2019; Raji et al. 2020; Shneiderman 2020).”
Leveraging Ethics for Supporting Students' Artificial Intelligence Skills Development in Namibia
Uunona, G. N., & Goosen, L. (2025) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-8965-2.ch011 · Cited by 4
Prospective Algorithmic Accountability and the Role of the News Media
Diakopoulos, N. A. (2025) · Philosophy of engineering and technology · doi:10.1007/978-3-031-87974-6_8 · Cited by 20
Metadata
- Authors
- Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P.
- Affiliations
- Partnership on AI
- Google (United States) (US)
- ORCID
- Venue
- Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2020-01-27
- Pages
- 33-44
- OpenAlex
- W3001807593
- Cited by
- 2,245
- References
- 83
- Citations by year
- 2026: 1097 · 2025: 454 · 2024: 243 · 2023: 177 · 2022: 149 · 2021: 99 · 2020: 24
- FWCI
- 35.91
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- AI accountability, algorithmic auditing, harm detection, algorithmic transparency, societal impact of AI, algorithmic governance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.