Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing
Open accessThe DOI matches its Crossref record; 96 of the 97 references it cites were found in a registry or an index, and 1 are not checked yet; 11 of the 44 sources its claims cite were checked for support.
Abstract
Rising concern for the societal implications of artificial intelligence systems has inspired a wave of academic and journalistic literature in which deployed systems are audited for harm by investigators from outside the organizations deploying the algorithms. However, it remains challenging for practitioners to identify the harmful repercussions of their own systems prior to deployment, and, once deployed, emergent issues can become difficult or impossible to trace back to their source.
Verified 2026-11-19 · 1 reference not checked yet
Overview
CoChat Summary
TL;DR
The paper proposes SMACTR, an end-to-end internal algorithmic auditing framework intended to help organizations assess AI systems against declared ethical principles throughout development. It argues that internal audits can complement external accountability by providing pre-deployment access, structured documentation, and risk analysis drawn from aerospace, medical device, and finance practices.
Key findings
- Introduces SMACTR—Scoping, Mapping, Artifact Collection, Testing, and Reflection—as five audit stages, each with documentation requirements that together form an overall audit report and assess decisions against organizational values or AI principles.
- Argues external audits are limited by lack of access to internal processes, intermediate models, and training data, while internal audits can enable proactive pre-deployment intervention and generate artifacts for external review.
- Draws lessons from other industries, citing aerospace’s one commercial airline accident per two million flights, Boeing 787 software estimated at 13 million lines of code, and a 1-in-1,000,000,000 per use hour maximum failure probability for critical aerospace systems; also cites medical-device design controls and finance internal audit infrastructure.
- Uses hypothetical cases—a child abuse screening tool and a smile-detection photo booth—to illustrate the framework; for the smile detector’s CelebA datasheet, it reports 58.1% female and 42% male labels, 77.8% aged 0–45 and 22.1% over 46, and 14.2% lighter-skinned versus 85.8% darker-skinned.
Methods
- Conceptual framework paper: develops SMACTR from auditing practices and artifacts in aerospace, medical devices, and finance, plus responsible-innovation dimensions of anticipation, reflexivity, inclusion, and responsiveness.
- Uses a hypothetical Company X Inc. consultancy with two invented client projects—a child abuse screening tool and Happy-Go-Lucky Inc.’s smile-detection photo booth—to demonstrate audit stages and artifacts; a worked example for the smile case is provided as supplementary material.
- Analysis is narrative and design-oriented, referencing existing standards and tools such as FMEA, model cards, datasheets, design controls, and design history files; no empirical sample or quantitative evaluation is reported.
Limitations
- The paper states that internal auditors necessarily share an organizational interest with the audit target, making an independent and objective viewpoint challenging; audits are not an objective “view from nowhere” and risk becoming reputation management.
- The paper states that internal audits are only one aspect of a broader system of required quality checks and balances, and that determining what systems to audit and why is not covered and is a separate research topic.
- Evident from the text: the framework is not empirically validated, the illustrative cases are hypothetical, and no real deployment outcomes or measured audit effectiveness are reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
9 Valid · 6 Not valid · 27 Not checked · 30 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“While not without faults, safety-critical and regulated industries such as aerospace and medicine have long traditions of auditable processes and design controls that have dramatically improved safety [77,81].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 3 LESSONS FROM AUDITING PRACTICES IN OTHER INDUSTRIES
What the sources say
- Not checked — no full text of the sourcecites [77] matched to “Design controls for the medical device industry” (2013) source
- Could not verifycites [81] matched to “The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions” (2019) sourceExperimental — accuracy not yet measured
The Introduction and §3 discuss biomedical ethics and standards or regulation, but do not say that aerospace or medicine has long used auditable processes and design controls, or that these dramatically improved safety. Searches for aerospace, auditable processes, design controls, and safety-critical industries found no passage addressing that claim. Page numbers are unavailable in our copy. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This is a repeatedly observed phenomenon in tax compliance auditing, where several international surveys of tax compliance demonstrate that a fixed and vetted tax audit methodology is one of the most effective strategies to convince companies to respect audit results and pay their full taxes [22,53].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.3 Audit Integrity and Procedural Justice
What the sources say
- Partly supportedcites [22] matched to “Perception on justice, trust and tax compliance behavior in Malaysia” (2017) sourceExperimental — accuracy not yet measured
The Regression Test section reports that Malaysian respondents would be more compliant if tax authorities treated them fairly in tax procedures. The Research Methodology section describes a questionnaire of academics, not several international surveys of companies. The paper does not establish that a fixed, vetted audit methodology is especially effective at securing acceptance of audit results or full tax payment. (LLM, conf=0.86)
- Partly supportedcites [53]Experimental — accuracy not yet measured
The “Justice in the taxation context” section reports that taxpayers are generally more compliant when treated fairly and respectfully, and the General discussion links perceived unfair treatment to resistance to paying back tax (page numbers unavailable). It does not establish that a fixed, vetted audit methodology is among the most effective strategies, or report international surveys demonstrating that effect for companies. (LLM, conf=0.83)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Traditional metrics for artificial intelligence like loss may conceal fairness concerns, social impact risks or abstraction errors [69].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.3.2 Artifact: Ethnographic Field Study.
What the sources say
- Supportedcites [69] matched to “Fairness and Abstraction in Sociotechnical Systems” (2019) source
The Introduction (page not provided) says that narrow assessments of machine-learning performance abstract away social context and can miss information needed for fair outcomes—an abstraction error. The Formalism Trap section adds that mathematical measures cannot capture the full range of fairness concerns, while the Takeaways discuss unintended social consequences. The paper names AUC, precision and recall rather than loss, but the claim uses loss as an illustrative metric. (LLM, conf=0.88)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Internal auditors' direct access to systems can thus help extend traditional external auditing paradigms by incorporating additional information typically unavailable for external evaluations to reveal previously unidentifiable risks.”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.4 The Internal Audit
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The essential inclusion of independent domain experts and marginalized groups in the ethical review process “has the potential to lead to more rigorous critical reflection because their experiences will often be precisely those that are most needed in identifying problematic background assumptions and revealing limitations with research questions, models, or methodologies” [33].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.2.1 Artifact: Ethical Review of System Use Case.
What the sources say
- Not checked — no full text of the sourcecites [33] matched to “25 Years of Feminist Empiricism and Standpoint Theory: Where Are We Now?” (2010) source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
19 more claims
Create a free account to see all 72 claims and every source verdict.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 37
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This disparity poses risks, as practitioners may unintentionally embed their own biases into AI system design and implementation (Martin 2019;Raji et al. 2020).”
From Accountability in Government Use of AI : Citizen Concerns and Preferences (2025) · cites this paper as Raji et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Work on AI safety and reproducibility has long pointed out the crucial importance of training data for understanding model performance, ensuring reproducibility, and assessing legal exposure [2, 25, 29, 46, 58].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While there is a growing interest in auditing algorithmic systems [125,126,128,138,154], this feld generally lacks defned standards and records could be incomplete or not relating to information needed for accountability purposes [15,127].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [128]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Raji et al. (2020) proposed an internal algorithmic auditing framework spanning data collection, model development, testing, deployment, and monitoring, with stage-specific artifacts that enable traceability and accountability.”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Raji et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Practical approaches to implement such ideas typically revolve around the systematic collection and curation of data on various aspects of systems [2, 28, 46].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747 · cited 1×
“A hypothetical datasheet for this system is filled out using studies on the CelebA dataset, with which the smile detector is built [44, 47].”
MethodThe global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Product development: Past research, present findings and future directions
Brown, S. L., & Eisenhardt, K. M. (1995) · The Academy of Management Review · doi:10.2307/258850 · Cited by 3,777 · cited 1×
“There are also intersections between this framework and just efective product development theory [5], as many of the components of audit design refocus the product development process to prioritize the user and their ultimate well-being, resulting in a more efective product performance outcome.”
Background(2018) · Cited by 3,510
Christiane, Druml, Wolzt, & Pleiner (2001) · Acta Scientiarum Naturalium Universitatis Sunyatseni · Cited by 3,419
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196
Developing a framework for responsible innovation
Stilgoe, J., Owen, R., & Macnaghten, P. (2013) · Research Policy · doi:10.1016/j.respol.2013.05.008 · Cited by 3,184 · cited 1×
“To design our audit procedure, we suggest complementing formal risk assessment methodologies with ideas from responsible innovation, which stresses four key dimensions: anticipation, reflexivity, inclusion and responsiveness [73], as well as system-theoretic concepts that help grapple with increasing complexity and co…”
MethodAlgorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Self-citation(2014) · Journal of the Korean Medical Association · doi:10.5124/jkma.2014.57.11.899 · Cited by 2,566
Research works printed but not linked: 7
- Virginia Eubanks. 2018. A child abuse prediction model fails poor families. Wired Magazine (2018).
- Jeremy Goldhaber-Fiebert and Lea Prince. 2019. Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny CountyâĂŹs Child Welfare Office. Pittsburgh: Allegheny County.[Google Scholar] (2019).
- Nitin Kohli, Renata Barreto, and Joshua A Kroll. 2018. Translation Tutorial: A Shared Lexicon for Research and Practice in Human-Centered Software Systems. In 1st Conference on Fairness, Accountability, and Transparancy. New York, NY, USA. 7.
- Joshua A Kroll, Solon Barocas, Edward W Felten, Joel R Reidenberg, David G Robinson, and Harlan Yu. 2016. Accountable algorithms. U. Pa. L. Rev. 165 (2016), 633.
- Institute of Internal Auditors. Research Foundation and Institute of Internal Au- ditors. 2007. The Professional Practices Framework. Inst of Internal Auditors.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22 (2014).
- Tim Vanderveen. 2005. Averting highest-risk errors is first priority. Patient Safety and Quality Healthcare 2 (2005), 16-21.
Cited by
Items 301–400 of 1,412
Building Knowledge through Action: Considerations for Machine Learning in the Workplace
Lindley, S. E., & Wilkins, D. J. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3584947 · Cited by 18
“Furthermore, delegating the responsibility of verification to workplace systems raises complex and unanswered societal-level questions about who is accountable for decisions that are made by, or with influential support from, ML [e.g., Doshi-Velez et al., 2017; Raji et al., 2020].”
BackgroundTransforming Leadership: The Role of Artificial Intelligence in Modern Organizational Management
Ali, Q., & Mahmood, M. A. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-8960-7.ch009 · Cited by 14
Time to audit your AI algorithms
Sandu, I., Wiersma, M., & Manichand, D. (2022) · Maandblad voor Accountancy en Bedrijfseconomie · doi:10.5117/mab.96.90108 · Cited by 12
“In one hypothetical example in Raji et al. (2020), a smile detection algorithm used to automatically trigger cameras in physical photo booths can lead to threats to the principle of fairness & non‑discrimination if the algorithm disproportionately impacts people with disabilities or different…”
BackgroundMethodInfluentialWeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI
Deng, W. H., Claire, W., Han, H. Z., Hong, J., Holstein, K., & Eslami, M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757702 · Cited by 27
The Role of Workers in AI Ethics and Governance
Nedzhvetskaya, N., & Tan, J. (2022) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197579329.013.68 · Cited by 28
Using model cards for ethical reflection: a qualitative exploration
Nunes, J. L., Barbosa, G. D. J., de Souza, C. S., Lopes, H., & Barbosa, S. D. J. (2022) · Brazilian Symposium on Human Factors in Computing Systems (IHC) · doi:10.1145/3554364.3559117 · Cited by 27
“As such, we, as researchers, should investigate how developers make use of tools like the MC and others [2, 21, 24, 27] and how they impact their own forms of reasoning.”
BackgroundStakeholder-accountability model for artificial intelligence projects
Maxmetrics, H., & Miller, G. J. (2022) · Journal of Economics and Management · doi:10.22367/jem.2022.44.18 · Cited by 24
“An organization’s ethical and value framework drives the AI system’s business model and constitutes the practical guidelines and policies for the project’s governance (Raji et al., 2020; Shneiderman, 2020).”
BackgroundChallenges of Machine Learning and AI (What Is Next?), Responsible and Ethical AI
Gkontra, P., Quaglio, G., Garmendia, A. T., & Lekadir, K. (2023) · Springer eBooks · doi:10.1007/978-3-031-36678-9_17 · Cited by 22
Hilbert, M., Thakur, A., Flores, P. M., Zhang, X., Bhan, J. Y., Bernhard, P., & Ji, F. (2023) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2023.102743 · Cited by 30
Ethical considerations of AI in financial decision
Wang, Y. (2024) · Computing and artificial intelligence. · doi:10.59400/cai.v2i1.1290 · Cited by 27
Auditing and Generating Synthetic Data with Controllable Trust Trade-offs
Belgodere, B. M., Dognin, P., Ivankay, A., Melnyk, I., Mroueh, Y., Mojsilović, A., Navrátil, J., Nitsure, A., Padhi, I., Rigotti, M., et al. (2024) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2024.3477976 · Cited by 41
“…on specific technical and societal aspects, it’s important to note that both the FAccT and STS literature encompass a broader spectrum of topics, including governance, closing the accountability gap [90] , ethics, and the far-reaching societal implications of emerging technologies [87], [103].”
BackgroundAuditing Flood Vulnerability Geo-Intelligence Workflow for Biases
Masinde, B. K., Gevaert, C. M., Nagenborg, M., den Homberg, M. J. C. V., Margutti, J., Gortzak, I., & Zevenbergen, J. A. (2024) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi13120419 · Cited by 21
Nullius in Explanans: an ethical risk assessment for explainable AI
Nannini, L., Huyskes, D., Panai, E., Pistilli, G., & Tartaro, A. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09800-7 · Cited by 18
“Automated risk assessment tools that adapt to changes in the system or its operating environment are employed, using dynamic risk assessment methods (Raji et al., 2020; Raveendran et al., 2022; Tartaro et al., 2024).”
Schmidt, J.-H., Bartsch, S. C., Adam, M., & Benlian, A. (2025) · Business & Information Systems Engineering · doi:10.1007/s12599-024-00914-2 · Cited by 22
“…current governance tools and methods for promoting accountability in AI systems development, such as AI principles or audits (e.g., Floridi 2019; Raji et al. 2020), often fall short due to their abstract nature and reliance on subjective judgments, which hinder effective communication (Berente…”
Maclure, J., & Morin-Martel, A. (2025) · Digital Society · doi:10.1007/s44206-025-00174-x · Cited by 19
“To help alleviate some of these issues, Raji et al. ( 2020 ) suggest that the audits of AI projects be conducted internally within the firms by auditors having full access to the relevant information but who did not work on the project.”
Navigating AI conformity: A design framework to assess fairness, explainability, and performance
von Zahn, M., Zacharias, J., Lowin, M., Chen, J., & Hinz, O. F. (2025) · Electronic Markets · doi:10.1007/s12525-025-00770-2 · Cited by 22
“…practices (Mökander & Floridi, 2022), declining system performance leading to poorer decision-making (dos Reis et al., 2016), biased outputs that disadvantage certain demographic groups (Barocas & Selbst, 2016), and a lack of accountability for decisions made by the AI (Raji et al., 2020).”
Hopkins, A., & Booth, S. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 12
Silano, J. A. (2024) · Healthcare Management Forum · doi:10.1177/08404704241257144 · Cited by 12
“38 Such tools can enable development teams to assess how an AI model performs under various conditions, including its effectiveness across diverse human health datasets.”
BackgroundShvetsova, O. A., Katalshov, D., & Lee, S.-K. (2025) · Applied Sciences · doi:10.3390/app15137298 · Cited by 26
Munonye, W. C., Ajonye, G. O., Ahonsi, S. O., Munonye, D. I., Akinloye, O. A., & Ikechukwu, O. C. (2025) · Cleaner and Responsible Consumption · doi:10.1016/j.clrc.2025.100324 · Cited by 19
Muppala, M. (2025) · International Conference on Emerging Research in Electronics, Computer Science and Technology (ICERECT) · doi:10.1109/icerect65215.2025.11378387 · Cited by 15
Promoting Ethical Uses in Artificial Intelligence Applied to Education
Matias, A., & Zipitria, I. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-32883-1_53 · Cited by 18
Manifestations of xenophobia in AI systems
Tomašev, N., Maynard, J. L., & Gabriel, I. (2024) · AI & Society · doi:10.1007/s00146-024-01893-4 · Cited by 25
“Recent advances in understanding and mitigating social biases in language models (Zmigrod et al. 2019; Huang et al. 2019b; Li et al. 2020; Liang et al. 2021; Ousidhoum et al. 2021; Nozza et al. 2021), and computer vision systems (Joo et al. 2020; Wang et al. 2020a), coupled with a development of…”
Black, E., Gillis, T. B., & Hall, Z. Y. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658928 · Cited by 22
“…call for fairness testing over time, insinuating some concern for robustness to distribution shift, and also call for adversarial testing [52, 66], suggesting that it is important for a model to perform well in unseen environments, very few papers particularly point to the need for robust…”
BackgroundOpportunities and Challenges of Using Artificial Intelligence in Securing Cyber-Physical Systems
Nweke, L. O., & Yayilgan, S. Y. (2024) · Springer eBooks · doi:10.1007/978-3-031-57452-8_7 · Cited by 17
A Comprehensive Framework for Transparent and Explainable AI Sensors in Healthcare
Bouderhem, R. (2024) · International Electronic Conference on Sensors and Applications (ECSA) · doi:10.3390/ecsa-11-20524 · Cited by 19
Fahad, S. K. A., Zhengkui, D. W., Ng, P. C., Wong, N., Ng, A. B., & See, S. C. W. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-80334-5_19 · Cited by 18
Abrar, M., Rehman, M. U., Khalid, S., & Ullah, R. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-025-11447-9 · Cited by 18
“AI models must be designed to avoid unauthorized access, breaches, or misuse of personal health information (Raji et al. 2020).”
Almahmoud, J., DeLine, R. A., & Drucker, S. M. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3463934 · Cited by 18
“Furthermore, showing and explaining where models fail is an approach that been appropriated from practices in the industry such as failure modes and effects analysis (FMEA) [54].”
BackgroundSubverting Fair Image Search with Generative Adversarial Perturbations
Ghosh, A., Jagielski, M., & Wilson, C. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533128 · Cited by 13
“fenses [32], to models that are hardened against adversarial perturbation techniques [5, 43], to auditing checklists [76] and tools that”
BackgroundFeeling fixes: Mess and emotion in algorithmic audits
Keyes, O., & Austin, J. (2022) · Big Data & Society · doi:10.1177/20539517221113772 · Cited by 25
“Of particular interest for our purposes is the proposals for “ internal audits ”— that is, for audit processes designed to operate within organizations developing algorithmic systems (Raji et al., 2020; Rakova et al., 2021).”
BackgroundResultAlgorithm Assurance: Auditing Applications of Artificial Intelligence
Boer, A., de Beer, L., & van Praat, F. (2022) · Progress in IS · doi:10.1007/978-3-031-11089-4_7 · Cited by 24
Ethics Auditing Framework for Trustworthy AI: Lessons from the IT Audit Literature
Zinda, N. (2022) · Digital ethics lab yearbook · doi:10.1007/978-3-031-09846-8_12 · Cited by 15
MLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualities
Maffey, K. R., Dotterrer, K., Niemann, J., Cruickshank, I. J., Lewis, G. A., & Kästner, C. (2023) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier58687.2023.00012 · Cited by 17
Inteligencia artificial y su impacto en la transformación de la gestión financiera
Herrera-Sánchez, M. J., & Casanova-Villalba, C. I. (2024) · Space Scientific Journal of Multidisciplinary · doi:10.63618/omd/ssjm/v2/n1/43 · Cited by 18
Auditing Digital Platforms for Discrimination in Economic Opportunity AdvertisingPreprint
Kingsley, S., Wang, C., Mikhalenko, A., Sinha, P., & Kulkarni, C. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.09656 · Cited by 24
“Digital platform audits have been generally hard to conduct[23, 34, 35].”
BackgroundDecoloniality impact assessment for AI
Eke, D., Chavarriaga, R., & Stahl, B. (2025) · AI & Society · doi:10.1007/s00146-025-02649-4 · Cited by 21
“Additionally, several frameworks address bias (UnBias 2018; Raji et al. 2020; ISO 2025; DSIT, UK 2025), fairness (UnBias 2018), and equality (EqualAI 2025), particularly in high-stakes domains, such as hiring (AI Now Institute 2018), policing, and recidivism prediction (Pennsylvania Board of…”
The Centers and Margins of Modeling Humans in Well-being Technologies
Zhu, J., Sanches, P., Tsaknaki, V., van der Maden, W., & Kaklopoulou, I. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713940 · Cited by 15
“Our proposed approach can complement existing quantitative auditing methods towards more situated qualitative approaches in auditing [22, 105].”
Heyn, H.-M., Habibullah, K. M., Knauss, E., Horkoff, J., Borg, M., Knauss, A., & Li, P. J. (2023) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain58948.2023.00011 · Cited by 23
AI and Auditing: Enhancing Audit Efficiency and Effectiveness with Artificial Intelligence
Lidiana, L. (2024) · Accounting Studies and Tax Journal (COUNT) · doi:10.62207/g0wpn394 · Cited by 25
“Internal audits, executed by dedicated teams within organizations, can inform decisions on AI technology development, especially when risks outweigh benefits (Raji, 2020).”
BackgroundSoni, V. (2024) · International Journal of Innovative Science and Research Technology (IJISRT) · doi:10.38124/ijisrt/ijisrt24may2203 · Cited by 16
Conceptualizing Automated Decision-Making in Organizational Contexts
Boos, A. K. (2024) · Philosophy & Technology · doi:10.1007/s13347-024-00773-5 · Cited by 20
“…Wachter, 2020; Véliz, 2021), algorithmic discrimination (Barocas & Selbst, 2016; Chouldechova, 2017; Binns, 2022; Fazelpour et al., 2022; Lazar & Stone, 2023), and the lack of transparency and accountability (Lepri et al., 2018; Zerilli et al., 2019; Raji et al., 2020; Busuioc, 2020; Kroll, 2020).”
Beyond Traditional Language Learning: EFL Student Views on ChatGPT in Saudi Arabia
Alharthi, S. M. (2024) · Arab World English Journal · doi:10.24093/awej/call10.2 · Cited by 28
“Acceptance, and not resistance, occurs when AI systems reflect on cultural values such as community well-being and are presented through familiar frameworks (Raji et al., 2020).”
BackgroundA comprehensive analysis of gender, racial, and prompt-induced biases in large language models
Torres, N., Ulloa, C., Araya, I., Ayala, M., & Jara, S. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-024-00696-6 · Cited by 20
The Value of Disagreement in AI Design, Evaluation, and Alignment
Fazelpour, S., & Fleisher, W. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732146 · Cited by 10
“These efforts have significantly advanced our understanding of how central concerns such as reliability and validity [23, 59], bias and fairness [6, 33], transparency and explainability [40, 119], privacy [55, 85], and accountability [74, 95] interact with sociotechnical AI systems.”
Interpretable trading pattern designed for machine learning applications
Sokolovsky, A., Arnaboldi, L., Bacardit, J., & Groß, T. (2023) · Machine Learning with Applications · doi:10.1016/j.mlwa.2023.100448 · Cited by 17
Güvenilir Yapay Zeka ve İç Denetim
Ağdeniz, Ş. (2024) · Denetişim · doi:10.58348/denetisim.1384391 · Cited by 15
Oladele, R. I., Orelaja, A., & Akinwande, O. T. (2024) · International Journal of Latest Technology in Engineering Management & Applied Science · doi:10.51583/ijltemas.2024.130207 · Cited by 24
Auditing the Audits: Lessons for Algorithmic Accountability from Local Law 144's Bias Audits
Gerchick, M. K., Encarnación, R., Tanigawa-Lau, C., Armstrong, L., Gutiérrez, A., & Metaxa, D. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732004 · Cited by 10
“Evaluating internal governance is a crucial part of algorithm audits [81, 102].”
Henderson, M. D. (2025) · Leadership & Organization Development Journal · doi:10.1108/lodj-05-2025-0319 · Cited by 15
Hauer, M. P., Krafft, T. D., & Zweig, K. A. (2023) · Data & Policy · doi:10.1017/dap.2023.30 · Cited by 16
“Raji et al. (2020) present internal algorithm audits, performed by external experts, as an approach to check whether ethical expectations are met.”
BackgroundInfluentialA democratic curriculum for the challenges of post-truth
Nally, D. (2024) · Curriculum Perspectives · doi:10.1007/s41297-024-00228-z · Cited by 15
“This will be particularly concerning if AI assistants are used to delegate decision-making, even if it is enacted by students, educators and policymakers with the intention of mistake-proofing any judgement-based work they complete (Schiff, 2022, following Raji et al., 2020).”
Tran, M. H., & Fiesler, C. L. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671109 · Cited by 16
“…the course content, instructors also might be able to draw from a growing body of research on concrete tools such as checklists and audits for considering ethics and fairness for topics like machine learning [16, 32, 39, 44], in addition to teaching strategies for tackling more ambiguous problems.”
Ethics and Transparency in Secure Web Model Generation
Sindiramutty, S. R., Prabagaran, K. R. V., Jhanjhi, N. Z., Murugesan, R. K., Malik, N. A., & Hussain, M. (2024) · Advances in information security, privacy, and ethics book series · doi:10.4018/979-8-3693-5415-5.ch012 · Cited by 16
Ethics and Artificial Intelligence in the Interdisciplinary Collaborations of Smart Care
Hine, C., & Barnaghi, P. M. (2024) · Science Technology & Human Values · doi:10.1177/01622439241302519 · Cited by 11
Warranty Provisions: Machine-Learning Versus Human Estimates
Becker, M., & Schölzel, S. (2025) · European Accounting Review · doi:10.1080/09638180.2024.2444521 · Cited by 14
“…efficiency, and standardization, as well as explainabil-ity to the auditor as prerequisites for widespread adoption, and previous artificial intelligence research enumerates various challenges in operationalizing and auditing machine learning systems (Raji et al., 2020; Shankar et al., 2022).”
The role of bank governance in managing the risks associated with banking institutions
Alnor, N. H. A., Mohammed, O. A. A., Al‐Matari, E. M., Ahmed, A., Benlaria, H., Elhefni, A. H. M., Kouki, F., & Elshaabany, M. M. (2024) · International Journal of ADVANCED AND APPLIED SCIENCES · doi:10.21833/ijaas.2024.04.021 · Cited by 11
Chang, H.-C. H., Bui, M., & McIlwain, C. D. (2022) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2022.348 · Cited by 13
“Google, in collaboration with Inioluwa Raji, has also envisioned means of an end-to-end framework for internal audits, primarily focused on the various stages of an algorithms production and deployment [19].”
Making the Unaccountable Internet: The Changing Meaning of Accounting in the Early ARPANET
Cooper, A. F., & Vidan, G. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533137 · Cited by 14
“…of “public accountability” from Bovens [13] into research concerning algorithmic accountability [64, 127], while others have attempted to institute cultures of accountability through ex ante engineering standards of care [45, 89] or post hoc audits and impact assessments [3, 87, 90, 108, 120].”
BackgroundFair Machine Learning Under Partial Compliance
Dai, J. C., Fazelpour, S., & Lipton, Z. C. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462521 · Cited by 19
“Existing approaches to auditing have focused on examining the performance of a single algorithmic decision maker [26, 51, 54].”
BackgroundA Matrix for Selecting Responsible AI Frameworks
Narayanan, M., & Schoeberl, C. (2023) · doi:10.51593/20220029 · Cited by 18
Li, Y., & Goel, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4787236 · Cited by 9
Metaverse Tourism: Opportunities, AI-Driven Marketing, and Ethical Challenges in Virtual Travel
Skandali, D. (2025) · Encyclopedia · doi:10.3390/encyclopedia5030135 · Cited by 12
“Real-time, granular consent dashboards; • Algorithmic transparency and third-party audits to detect and mitigate bias [31,32]; • Decentralized or self-sovereign identity (SSI) architectures to give users control over disclosure and reuse [33]; and • Alignment with robust regulatory instruments such…”
An Explainable AI Tool for Operational Risks Evaluation of AI Systems for SMEs
Han, T. A., Pandit, D., Joneidy, S., Hasan, M. M., Hossain, J., Tania, M. H., Hossain, M. A., & Nourmohammadi, N. (2023) · International Conference on Software, Knowledge, Information Management and Applications (SKIMA) · doi:10.1109/skima59232.2023.10387301 · Cited by 11
“Independent auditing and testing regimes tailored to AI systems can help build trust [29].”
BackgroundDai, Z. (2024) · International Conference on Frontiers of Artificial Intelligence and Machine Learning (FAIML) · doi:10.1145/3653644.3653662 · Cited by 17
AI4Bangladesh: AI Ethics for Bangladesh - Challenges, Risks, Principles, and Suggestions
Mayeesha, T. T., Islam, F., & Ahmed, N. (2024) · International Conference on Information and Communication Technologies and Development (ICTD) · doi:10.1145/3700794.3700820 · Cited by 13
Obura, E. A., & Emoit, P. I. (2024) · Africa Education Review · doi:10.1080/18146627.2024.2440351 · Cited by 24
“This will ensure accountability and appropriate use of the AI tool in research to protect the authors from plagiarism, liability, and punishment for not using AI responsibly (Kohli 2018; Raji et al. 2020).”
A Survey on the Recent Advancements in Human-Centered Dialog Systems
Oruche, R., Goruganthu, S. K., Akula, R., Cheng, X., Goni, M. A., Shibo, B. W., Kee, K. F., Zampieri, M., & Calyam, P. P. (2025) · ACM Computing Surveys · doi:10.1145/3729220 · Cited by 17
Calado, S., & Veloso, C. M. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-1742-7.ch004 · Cited by 10
Ezema, K., Chandler, C., Southwell, R. V., Cholendiran, N., & D’Mello, S. K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714059 · Cited by 19
A systematic review of algorithm auditing processes to assess bias and risks in AI systems
Funda, V. (2025) · Journal of Infrastructure Policy and Development · doi:10.24294/jipd11489 · Cited by 17
“The studies by Raji et al. (2020) and Shneiderman (2020) highlight the focus on internal auditing practices in commercial settings and the development of human-centered AI governance strategies.”
Deng, W. H., Barocas, S., & Vaughan, J. W. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711076 · Cited by 23
“…these contributions with literature from CSCW, design, and the broader fields of HCI and social science, we illuminate paths for future researchers, industry practitioners and organizations, and policy makers to better support responsible research practices in industry settings and beyond.”
A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms
Harvey, E., Kizilcec, R. F., & Koenecke, A. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732137 · Cited by 22
Pepple, D., & Muthuthantrige, N. (2025) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2025.100878 · Cited by 22
Interpretable Machine Learning: A Comprehensive Review of Foundations, Methods, and the Path Forward
Fridkin, S., & Bendersky, M. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70075 · Cited by 38
ACROCPoLis: A Descriptive Framework for Making Sense of Fairness
Aler Tubella, A., Coelho Mollo, D., Dahlgren Lindström, A., Devinney, H., Dignum, V., Ericson, P., Jonsson, A., Kampik, T., Lenaerts, T., Mendez, J. A., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594059 · Cited by 8
“Considering a broad category of Actors allows for including companies and organizations, which often hold accountability for the Outcomes of the process [49], as well as communities and individuals affected by a given process, whose perspectives are key to fairness assessments [7].”
BackgroundGenerative AI in Terms of Its Ethical Problems for Both Teachers and Learners: Striking a Balance
Le-Nguyen, H.-T., & Tran, T. T. (2023) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-0074-9.ch006 · Cited by 23
Awareness of Unethical Artificial Intelligence and its Mitigation Measures
HÖLLER, S., Dilger, T., Spieß, T., Ploder, C., & Bernsteiner, R. C. (2023) · European Journal of Interdisciplinary Studies · doi:10.24818/ejis.2023.17 · Cited by 10
AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks
Schnitzer, R., Hapfelmeier, A., Gaube, S., & Zillner, S. (2024) · Frontiers of artificial intelligence, ethics and multidisciplinary applications. · doi:10.1007/978-981-99-9836-4_27 · Cited by 20
“…19 [16,36,46] AIH 2 [25,42] AIH 8 [24,37] AIH 14 [2,30] AIH 20 [16,25,42,46] AIH 3 [42,48] AIH 9 [34,42] AIH 15 [46,48] AIH 21 [16,48] AIH 4 [13,32,40] AIH 10 [34,42] AIH 16 [16,48] AIH 22 [44] AIH 5 [3,42] AIH 11 [12,46] AIH 17 [6,48] AIH 23 [16,25,42,46] AIH 6 [25,42] AIH 12 [42,43] AIH 18…”
Navigating the decision‐making landscape of AI in risk finance: Techno‐accountability unveiled
Issa, H., Dakroub, R., Lakkis, H., & Jaber, J. (2024) · Risk Analysis · doi:10.1111/risa.14336 · Cited by 14
“For the third phase (risk analysis), because of the likelihood of high variability, the testing of any technological system should be based on risk prioritization to reduce the accountability gap (Raji et al., 2020).”
How do AI systems fail socially?: an engineering risk analysis approach
Rismani, S., & Moon, A. (2021) · IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS) · doi:10.1109/ethics53270.2021.9632769 · Cited by 8
Navigating the Audit Landscape: A Framework for Developing Transparent and Auditable XR
Norval, C., Cloete, R., & Singh, J. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594090 · Cited by 13
“This brings about transparency, thus helping support accountability processes concerning the broader socio-technical system (see e.g. [14, 26, 62, 92, 115, 116]).”
BackgroundInfluentialOlson, M., Liu, S., Anirudh, R., Thiagarajan, J. J., Bremer, P., & Wong, W.-K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00771 · Cited by 8
Beyond Bias: Artificial Intelligence and Social JusticePreprint
Sloan, R. H., & Warner, R. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3530090 · Cited by 9
Fairness On The Ground: Applying Algorithmic Fairness Approaches to Production SystemsPreprint
Bakalar, C., Barreto, R. D. C., Bergman, S., Bogen, M., Chern, B., Corbett‐Davies, S., Hall, M. A., Kloumann, I. M., Lam, M. S., Candela, J. Q., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06172 · Cited by 49
“Recognizing that documentation and metrics on their own will not necessarily lead to meaningful fairness improvements, others have proposed a structured frameworks for internal algorithmic auditing informed by organizational values [53].”
Aligning AI Regulation to Sociotechnical ChangePreprint
Maas, M. M. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3871635 · Cited by 9
SAF: Stakeholders’ Agreement on Fairness in the Practice of Machine Learning Development
Curto, G., & Comim, F. (2023) · Science and Engineering Ethics · doi:10.1007/s11948-023-00448-y · Cited by 11
“…design processes (Lehr & Ohm, 2017 ), the quality management of such systems (Horch, 1996 ) and the auditing practices at each stage of ML end-to-end processes development (Raji et al., 2020 ), to date there is not a process describing how to manage fairness decision making throughout ML design.”
BackgroundBertrand, A., Eagan, J., Maxwell, W., & Brand, J. L. M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642326 · Cited by 10
The democratization dilemma: When everyone is an expert, who do we trust?
Mahajan, S. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04734-x · Cited by 11
“As highlighted in algorithmic accountability scholarship, external audits can uncover hidden biases, promote regulatory compliance, and strengthen public trust in AI systems (Ananny and Crawford 2018; Raji et al. 2020).”
Human-Centered Evaluation and Auditing of Language Models
Liu, Y. L., Deng, W. H., Lam, M. S., Eslami, M., Kim, J., Liao, Q. V., Xu, W., Novikova, J., & Xiao, Z. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706729 · Cited by 7
“It is also known that practitioners engage in extensive internal auditing (e.g., red teaming [16]) to mitigate the safety concerns and other risks of LLMs, although there is a lack of trans-parency and shared practices in how such audits are performed [39, 43, 44, 46].”
Managing Transparency Fairness Accountability in AI for Sustainable Human Resource Management
Setiawati, R., Kusmara, A., & Kuusk, T. (2025) · International Conference on Creative Communication and Innovative Technology (ICCIT) · doi:10.1109/iccit65724.2025.11166968 · Cited by 19
“• IBM uses Watson to personalize employee training, significantly improving efficiency and satisfaction [17].”
Why AI governance should be a focal issue for gerontology
Berridge, C. W., & Ho, A. (2025) · The Gerontologist · doi:10.1093/geront/gnaf234 · Cited by 11
Isaifan, R. J., Tawalbeh, A., & Hasna, M. O. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1640780 · Cited by 17
Culturally responsive AI chatbots: From framework to field evidence
Naidoo, V., & CHADHA, K. K. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100224 · Cited by 15
Evaluating the quality of GenAI applications in software engineering: a multi-case study
Yu, L., Alégroth, E., Chatzipetrou, P., & Gorschek, T. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10759-2 · Cited by 11
“They often function as ‘black boxes’, making conventional evaluation techniques inadequate (Raji et al. 2020).”
Digital platform sovereignty through Ubuntu: Re-imagining AI-enabled work future in Africa
Anning‐Dorson, T. (2025) · Electronic Markets · doi:10.1007/s12525-025-00864-x · Cited by 13
“…AI further intensifies governance gaps around disclosure, watermarking, provenance and auditability; without clear duties to explain model behaviour and training data lineage, opacity compounds informational disadvantage for workers and regulators (Kellogg et al., 2020; Raji et al., 2020).”
InfluentialDynamic calibration of trust and trustworthiness in AI-enabled systems
Liebherr, M., Enkel, E., Law, E. L., Mousavi, M. R., Sammartino, M., & Sieberg, P. M. (2026) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-026-00840-6 · Cited by 17
Translating the Public Service Media Remit into Metrics and Algorithms
Grün, A., & Neufeld, X. (2022) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3523227.3547380 · Cited by 12
Beyond the ML Model: Applying Safety Engineering Frameworks to Text-to-Image Development
Rismani, S., Shelby, R. M., Smart, A., Santos, R. D., Moon, A., & Rostamzadeh, N. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604685 · Cited by 27
“These methods differ from other assessment processes for ML systems, such as algorithmic impact assessments and auditing practices, in their focus on identifying, evaluating, and connecting anticipated harms to a design decision and mitigation in the development process [59, 67].”
BackgroundMethodInfluentialSelf-citationMeasuring an artificial intelligence language model’s trust in humans using machine incentives
Johnson, T., & Obradovich, N. (2024) · Journal of Physics Complexity · doi:10.1088/2632-072x/ad1c69 · Cited by 17
“…and require behavioral analysis [11, 29], the analysis of machine behavior has illuminated algorithmic biases [12, 13], the nuances of AI errors [14], practical methods for auditing AI behavior throughout the development process [30, 31], and AI models’ skill in judgment and decision making [15].”
Method
Metadata
- Authors
- Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P.
- Affiliations
- Partnership on AI
- Google (United States) (US)
- ORCID
- Venue
- Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2020-01-27
- Pages
- 33-44
- OpenAlex
- W3001807593
- Cited by
- 2,245
- References
- 83
- Citations by year
- 2026: 1097 · 2025: 454 · 2024: 243 · 2023: 177 · 2022: 149 · 2021: 99 · 2020: 24
- FWCI
- 35.91
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- AI accountability, algorithmic auditing, harm detection, algorithmic transparency, societal impact of AI, algorithmic governance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.