Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing
Open accessThe DOI matches its Crossref record; 96 of the 97 references it cites were found in a registry or an index, and 1 are not checked yet; 11 of the 44 sources its claims cite were checked for support.
Abstract
Rising concern for the societal implications of artificial intelligence systems has inspired a wave of academic and journalistic literature in which deployed systems are audited for harm by investigators from outside the organizations deploying the algorithms. However, it remains challenging for practitioners to identify the harmful repercussions of their own systems prior to deployment, and, once deployed, emergent issues can become difficult or impossible to trace back to their source.
Verified 2026-11-19 · 1 reference not checked yet
Overview
CoChat Summary
TL;DR
The paper proposes SMACTR, an end-to-end internal algorithmic auditing framework intended to help organizations assess AI systems against declared ethical principles throughout development. It argues that internal audits can complement external accountability by providing pre-deployment access, structured documentation, and risk analysis drawn from aerospace, medical device, and finance practices.
Key findings
- Introduces SMACTR—Scoping, Mapping, Artifact Collection, Testing, and Reflection—as five audit stages, each with documentation requirements that together form an overall audit report and assess decisions against organizational values or AI principles.
- Argues external audits are limited by lack of access to internal processes, intermediate models, and training data, while internal audits can enable proactive pre-deployment intervention and generate artifacts for external review.
- Draws lessons from other industries, citing aerospace’s one commercial airline accident per two million flights, Boeing 787 software estimated at 13 million lines of code, and a 1-in-1,000,000,000 per use hour maximum failure probability for critical aerospace systems; also cites medical-device design controls and finance internal audit infrastructure.
- Uses hypothetical cases—a child abuse screening tool and a smile-detection photo booth—to illustrate the framework; for the smile detector’s CelebA datasheet, it reports 58.1% female and 42% male labels, 77.8% aged 0–45 and 22.1% over 46, and 14.2% lighter-skinned versus 85.8% darker-skinned.
Methods
- Conceptual framework paper: develops SMACTR from auditing practices and artifacts in aerospace, medical devices, and finance, plus responsible-innovation dimensions of anticipation, reflexivity, inclusion, and responsiveness.
- Uses a hypothetical Company X Inc. consultancy with two invented client projects—a child abuse screening tool and Happy-Go-Lucky Inc.’s smile-detection photo booth—to demonstrate audit stages and artifacts; a worked example for the smile case is provided as supplementary material.
- Analysis is narrative and design-oriented, referencing existing standards and tools such as FMEA, model cards, datasheets, design controls, and design history files; no empirical sample or quantitative evaluation is reported.
Limitations
- The paper states that internal auditors necessarily share an organizational interest with the audit target, making an independent and objective viewpoint challenging; audits are not an objective “view from nowhere” and risk becoming reputation management.
- The paper states that internal audits are only one aspect of a broader system of required quality checks and balances, and that determining what systems to audit and why is not covered and is a separate research topic.
- Evident from the text: the framework is not empirically validated, the illustrative cases are hypothetical, and no real deployment outcomes or measured audit effectiveness are reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
9 Valid · 6 Not valid · 27 Not checked · 30 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“While not without faults, safety-critical and regulated industries such as aerospace and medicine have long traditions of auditable processes and design controls that have dramatically improved safety [77,81].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 3 LESSONS FROM AUDITING PRACTICES IN OTHER INDUSTRIES
What the sources say
- Not checked — no full text of the sourcecites [77] matched to “Design controls for the medical device industry” (2013) source
- Could not verifycites [81] matched to “The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions” (2019) sourceExperimental — accuracy not yet measured
The Introduction and §3 discuss biomedical ethics and standards or regulation, but do not say that aerospace or medicine has long used auditable processes and design controls, or that these dramatically improved safety. Searches for aerospace, auditable processes, design controls, and safety-critical industries found no passage addressing that claim. Page numbers are unavailable in our copy. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This is a repeatedly observed phenomenon in tax compliance auditing, where several international surveys of tax compliance demonstrate that a fixed and vetted tax audit methodology is one of the most effective strategies to convince companies to respect audit results and pay their full taxes [22,53].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.3 Audit Integrity and Procedural Justice
What the sources say
- Partly supportedcites [22] matched to “Perception on justice, trust and tax compliance behavior in Malaysia” (2017) sourceExperimental — accuracy not yet measured
The Regression Test section reports that Malaysian respondents would be more compliant if tax authorities treated them fairly in tax procedures. The Research Methodology section describes a questionnaire of academics, not several international surveys of companies. The paper does not establish that a fixed, vetted audit methodology is especially effective at securing acceptance of audit results or full tax payment. (LLM, conf=0.86)
- Partly supportedcites [53]Experimental — accuracy not yet measured
The “Justice in the taxation context” section reports that taxpayers are generally more compliant when treated fairly and respectfully, and the General discussion links perceived unfair treatment to resistance to paying back tax (page numbers unavailable). It does not establish that a fixed, vetted audit methodology is among the most effective strategies, or report international surveys demonstrating that effect for companies. (LLM, conf=0.83)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Traditional metrics for artificial intelligence like loss may conceal fairness concerns, social impact risks or abstraction errors [69].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.3.2 Artifact: Ethnographic Field Study.
What the sources say
- Supportedcites [69] matched to “Fairness and Abstraction in Sociotechnical Systems” (2019) source
The Introduction (page not provided) says that narrow assessments of machine-learning performance abstract away social context and can miss information needed for fair outcomes—an abstraction error. The Formalism Trap section adds that mathematical measures cannot capture the full range of fairness concerns, while the Takeaways discuss unintended social consequences. The paper names AUC, precision and recall rather than loss, but the claim uses loss as an illustrative metric. (LLM, conf=0.88)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Internal auditors' direct access to systems can thus help extend traditional external auditing paradigms by incorporating additional information typically unavailable for external evaluations to reveal previously unidentifiable risks.”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.4 The Internal Audit
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The essential inclusion of independent domain experts and marginalized groups in the ethical review process “has the potential to lead to more rigorous critical reflection because their experiences will often be precisely those that are most needed in identifying problematic background assumptions and revealing limitations with research questions, models, or methodologies” [33].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.2.1 Artifact: Ethical Review of System Use Case.
What the sources say
- Not checked — no full text of the sourcecites [33] matched to “25 Years of Feminist Empiricism and Standpoint Theory: Where Are We Now?” (2010) source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
19 more claims
Create a free account to see all 72 claims and every source verdict.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 37
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This disparity poses risks, as practitioners may unintentionally embed their own biases into AI system design and implementation (Martin 2019;Raji et al. 2020).”
From Accountability in Government Use of AI : Citizen Concerns and Preferences (2025) · cites this paper as Raji et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Work on AI safety and reproducibility has long pointed out the crucial importance of training data for understanding model performance, ensuring reproducibility, and assessing legal exposure [2, 25, 29, 46, 58].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While there is a growing interest in auditing algorithmic systems [125,126,128,138,154], this feld generally lacks defned standards and records could be incomplete or not relating to information needed for accountability purposes [15,127].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [128]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Raji et al. (2020) proposed an internal algorithmic auditing framework spanning data collection, model development, testing, deployment, and monitoring, with stage-specific artifacts that enable traceability and accountability.”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Raji et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Practical approaches to implement such ideas typically revolve around the systematic collection and curation of data on various aspects of systems [2, 28, 46].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747 · cited 1×
“A hypothetical datasheet for this system is filled out using studies on the CelebA dataset, with which the smile detector is built [44, 47].”
MethodThe global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Product development: Past research, present findings and future directions
Brown, S. L., & Eisenhardt, K. M. (1995) · The Academy of Management Review · doi:10.2307/258850 · Cited by 3,777 · cited 1×
“There are also intersections between this framework and just efective product development theory [5], as many of the components of audit design refocus the product development process to prioritize the user and their ultimate well-being, resulting in a more efective product performance outcome.”
Background(2018) · Cited by 3,510
Christiane, Druml, Wolzt, & Pleiner (2001) · Acta Scientiarum Naturalium Universitatis Sunyatseni · Cited by 3,419
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196
Developing a framework for responsible innovation
Stilgoe, J., Owen, R., & Macnaghten, P. (2013) · Research Policy · doi:10.1016/j.respol.2013.05.008 · Cited by 3,184 · cited 1×
“To design our audit procedure, we suggest complementing formal risk assessment methodologies with ideas from responsible innovation, which stresses four key dimensions: anticipation, reflexivity, inclusion and responsiveness [73], as well as system-theoretic concepts that help grapple with increasing complexity and co…”
MethodAlgorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Self-citation(2014) · Journal of the Korean Medical Association · doi:10.5124/jkma.2014.57.11.899 · Cited by 2,566
Research works printed but not linked: 7
- Virginia Eubanks. 2018. A child abuse prediction model fails poor families. Wired Magazine (2018).
- Jeremy Goldhaber-Fiebert and Lea Prince. 2019. Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny CountyâĂŹs Child Welfare Office. Pittsburgh: Allegheny County.[Google Scholar] (2019).
- Nitin Kohli, Renata Barreto, and Joshua A Kroll. 2018. Translation Tutorial: A Shared Lexicon for Research and Practice in Human-Centered Software Systems. In 1st Conference on Fairness, Accountability, and Transparancy. New York, NY, USA. 7.
- Joshua A Kroll, Solon Barocas, Edward W Felten, Joel R Reidenberg, David G Robinson, and Harlan Yu. 2016. Accountable algorithms. U. Pa. L. Rev. 165 (2016), 633.
- Institute of Internal Auditors. Research Foundation and Institute of Internal Au- ditors. 2007. The Professional Practices Framework. Inst of Internal Auditors.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22 (2014).
- Tim Vanderveen. 2005. Averting highest-risk errors is first priority. Patient Safety and Quality Healthcare 2 (2005), 16-21.
Cited by
Items 201–300 of 1,411
A Systematic Review of Responsible Artificial Intelligence Principles and Practice
Gunasekara, L., El-Haber, N., Nagpal, S., Moraliyage, H., Issadeen, Z., Manic, M., & Silva, D. D. (2025) · Applied System Innovation · doi:10.3390/asi8040097 · Cited by 70
Regulating artificial intelligence: a technology regulator's perspective
Ellul, J., Pace, G. J., McCarthy, S., Sammut, T., Brockdorff, J., & Scerri, M. (2021) · International Conference on Artificial Intelligence and Law · doi:10.1145/3462757.3466093 · Cited by 43
“Quite a number of ethical frameworks have been proposed— it suffices to note that “at least 63 public-private initiatives have produced statements describing high-level principles, values and other tenets to guide the ethical development, deployment and governance of AI” [14].”
BackgroundTowards eXplainable Artificial Intelligence (XAI) in Tax Law: The Need for a Minimum Legal Standard
Kuźniacki, B., Almada, M., Tyliński, K., Górski, L., Winogradska, B., & Zeldenrust, R. (2022) · World Tax Journal · doi:10.59403/2yhh9pa · Cited by 31
Predictive Algorithms in Justice Systems and the Limits of Tech-Reformism
Ugwudike, P. (2022) · International Journal for Crime, Justice and Social Democracy · doi:10.5204/ijcjsd.2189 · Cited by 29
“But internal audits should not override the need for independent third-party ethical audits for external accountability (Brown, Davidovic and Hasan 2021; Raji et al. 2020).”
BackgroundModeling rabbit-holes on YouTube
Merrer, E. L., Trédan, G., & Yesilkanat, A. (2023) · Social Network Analysis and Mining · doi:10.1007/s13278-023-01105-9 · Cited by 40
“This paper thus seeks to articulate a precise definition of the RH effect so that reproducible and automated means of measure are made possible (rather than relying on manual tagging [3, 4]).”
The Digital Transformation in Health: How AI Can Improve the Performance of Health Systems
Periáñez, Á., del Río, A. F., Nazarov, I., Jané, E., Hassan, M., Rastogi, A., & Tang, D. (2024) · Health Systems & Reform · doi:10.1080/23288604.2024.2387138 · Cited by 59
Not All Clinical AI Monitoring Systems Are Created Equal: Review and Recommendations
Feng, J., Xia, F., Singh, K., & Pirracchio, R. (2025) · NEJM AI · doi:10.1056/aira2400657 · Cited by 28
Constantinides, M., Bogucka, E. P., Quercia, D., Kallio, S., & Tahaei, M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686927 · Cited by 59
“The automatically generated report plays a crucial role in this process by providing a summary of the guidelines that were effectively implemented, and of those that should be considered for future development.”
Actionable Approaches to Promote Ethical AI in Libraries
Bubinger, H., & Dinneen, J. D. (2021) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.528 · Cited by 45
“Notably, the SMACTR framework (Raji et al., 2020) provides a promising guide to particular auditing steps (Scoping, Mapping, Artefact Collection, Testing, Reflection, and post-audit) and the necessary documents (e.”
MethodRegulating Government AI and the Challenge of Sociotechnical Design
Engstrom, D. F., & Haim, A. (2023) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-120522-091626 · Cited by 48
Should Users Trust Advanced AI Assistants? Justified Trust As a Function of Competence and Alignment
Manzini, A., Keeling, G., Marchal, N., McKee, K. R., Rieser, V., & Gabriel, I. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658964 · Cited by 28
“• The creation of internal teams and practices, which operate independently of those building AI assistants, that are responsible for conducting rigorous internal testing and evaluation of models underpinning assistants (e.g. red teamers and dogfooding [102]).”
BackgroundEthics dumping in artificial intelligence
Bélisle-Pipon, J.-C., & Victor, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1426761 · Cited by 26
The Escalating AI’s Energy Demands and the Imperative Need for Sustainable Solutions
León, M. (2024) · WSEAS TRANSACTIONS on SYSTEMS archive · doi:10.37394/23202.2024.23.46 · Cited by 35
Algorithmic accountabilities and health systems: A review and sociomaterial approach
Donia, J. (2025) · Big Data & Society · doi:10.1177/20539517251334099 · Cited by 33
“These code-related goals, while themselves (Coglianese and Lehr, 2019; Mayernik, 2017), algorithmic impact assessments (Ada Lovelace Institute, 2021), audits (Raji et al., 2020), and public algo-rithm registries (City of Amsterdam, 2020).”
Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)
Neumann, A., Kirsten, E., Zafar, M. B., & Singh, J. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732038 · Cited by 77
A Sociotechnical Audit: Assessing Police Use of Facial Recognition
Radiya-Dixit, E., & Neff, G. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594084 · Cited by 45
“An audit is a tool for analysis or inspection, often to evaluate the compliance of a system with respect to predefined standards [1, 65, 79].”
MethodKing, T. C., Roberts, H., Mökander, J., Tsamados, A., Taddeo, M., & Floridi, L. (2024) · AI & Society · doi:10.1007/s00146-024-01950-y · Cited by 52
“RAIS is an open-source framework, derived from the internal algorithm audit framework developed by Raji et al. (2020), designed to help VCs assess early-stage AI companies and technologies across areas of social and ethical impact, regulatory compliance, and technical risk.”
Sloane, M., & Wüllhorst, E. (2025) · Data & Policy · doi:10.1017/dap.2024.54 · Cited by 33
“…to better harmonize with the opaque ways algorithms sort people into groups (Wachter, 2022), or the evaluation of existing governance measures, such as impact assessments and audits that are common in other domains, for AI systems (Stahl et al., 2023; Nahmias & Perel, 2020; D et al., 2020).”
Reliability Testing for Natural Language Processing Systems
Tan, S., Joty, S., Baxter, K., Taeihagh, A., Bennett, G. A., & Kan, M. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.321 · Cited by 49
“…mindful of the biases in the NLP pipeline (Bender and Friedman, 2018; Mitchell et al., 2019; Waseem et al., 2021) and holding creators accountable via audits (Raji et al., 2020; Brundage et al., 2020), we argue for the need to evaluate an NLP system’s reliability in diverse operating conditions.”
BackgroundMethodInfluentialPiloting a Survey-Based Assessment of Transparency and Trustworthiness with Three Medical AI Tools
Fehr, J., Jaramillo-Gutierrez, G., Oala, L., Gröschel, M. I., Bierwirth, M., Balachandran, P., Leite, A. W., & Lippert, C. (2022) · Healthcare · doi:10.3390/healthcare10101923 · Cited by 37
“Two previous works provided internal assessment frameworks for organizations who want to assess if their AI tool meets the ethical expectations for trustworthy AI [28,29].”
BackgroundToward Sociotechnical AI: Mapping Vulnerabilities for Machine Learning in Context
Dobbe, R., & Wolters, A. (2024) · Minds and Machines · doi:10.1007/s11023-024-09668-y · Cited by 25
Beyond model interpretability: socio-structural explanations in machine learning
Smart, A., & Kasirzadeh, A. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02056-1 · Cited by 25
Self-citationData Quality in the Age of AI: A Review of Governance, Ethics, and the FAIR Principles
Guillen-Aguinaga, M., Aguinaga-Ontoso, E., Guillen-Aguinaga, L., Guillen-Grima, F., & Aguinaga-Ontoso, I. (2025) · Data · doi:10.3390/data10120201 · Cited by 37
Krook, J., Winter, P., Downer, J., & Blockx, J. (2025) · AI and Ethics · doi:10.1007/s43681-025-00674-z · Cited by 48
“For example, Raji et al. (2020) describes how ‘AIAs tools are designed to be publicly accessible documents that detail the potential impacts and ethical considerations of deploying AI systems.’”
The necessity of AI audit standards boards
Manheim, D., Martin, S., Bailey, M., Samin, M., & Greutzmacher, R. (2025) · AI & Society · doi:10.1007/s00146-025-02320-y · Cited by 67
“As Raji et al. (2020) argued, auditing needs to be present throughout the development of a model.”
InfluentialA framework for assessing AI ethics with applications to cybersecurity
Bruschi, D., & Diomede, N. (2022) · AI and Ethics · doi:10.1007/s43681-022-00162-8 · Cited by 37
Towards Implementing Responsible AI
Sanderson, C., Lu, Q., Douglas, D., Xu, X., Zhu, L., & Whittle, J. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10021121 · Cited by 28
The five tests: designing and evaluating AI according to indigenous Māori principles
Munn, L. (2024) · AI & SOCIETY · doi:10.1007/s00146-023-01636-x · Cited by 24
“In response, some researchers have developed tools to technically audit AI, testing the system and producing reports to reveal risks and threats (Raji et al. 2020).”
What does it mean to be a responsible AI practitioner: An ontology of roles and skills
Rismani, S., & Moon, A. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604702 · Cited by 43
“AI ethics principles [31], in-depth auditing toolkits [11, 39, 46], checklists [5, 35], codebases [4, 8], standards and regulations [1, 6] have been proposed by many different international actors.”
BackgroundReward Reports for Reinforcement Learning
Gilbert, T. K., Lambert, N., Dean, S., Zick, T., Snoswell, A., & Mehta, S. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604698 · Cited by 63
“Meanwhile, many audit mechanisms attempt to confirm either internally or externally whether a given system conforms to a legacy standard or regulation [46].”
BackgroundHohma, E., & Lütge, C. (2023) · AI · doi:10.3390/ai4040046 · Cited by 39
Responsible AI starts with the artifact: Challenging the concept of responsible AI in IS research
Mikalef, P., Benlian, A., Conboy, K., & Tarafdar, M. (2025) · European Journal of Information Systems · doi:10.1080/0960085x.2025.2506875 · Cited by 45
“Aspects such as accuracy and reliability assume that through specified actions, the decisions and outcomes of AI systems can be precisely foreseen, or at least that the decisional space is predictable (Raji et al., 2020; Ryan, 2020).”
Wang, S., Cooper, N., & Eby, M. R. (2024) · Big Data & Society · doi:10.1177/20539517241290220 · Cited by 43
“Recent toolkits (Krafft et al., 2021), frameworks (Raji et al., 2020), and studies (Costanza-Chock et al., 2022) of algorithmic auditing advocate for the involvement of affected people and communities early and throughout auditing processes.”
BackgroundHuman-Artificial Intelligence in Management Functions: A Synergistic Symbiosis Relationship
Islami, X., & Mulolli, E. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2439615 · Cited by 51
“When humans rely on AI to lead, the potential for an accountability gap often arises, as it can be unclear who bears responsibility for failures (Raji et al. 2020).”
Torkamaan, H., Steinert, S., Pera, M. S., Kudina, O., Freire, S. K., Verma, H., Kelly, S., Sekwenz, M.-T., Yang, J., van Nunen, K., et al. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2431068 · Cited by 43
The Present and Future of Accountability for AI Systems: A Bibliometric Analysis
Bartsch, S. C., Nguyen, L. H., Schmidt, J.-H., Du, G., Adam, M., Benlian, A., & Sunyaev, A. (2025) · Information Systems Frontiers · doi:10.1007/s10796-025-10636-9 · Cited by 41
“1, researchers might explore how technical applications and tools, such as explanation interfaces (e.g., Ribeiro et al., 2016) or AI audits (e.g., Raji et al., 2020 ) are interpreted differently by generative AI systems’ stakeholders, such as healthcare professionals or regulators.”
Fairness in Agreement With European Values: An Interdisciplinary Perspective on AI Regulation
Colmenarejo, A. B., Nannini, L., Rieger, A., Scott, K. M., Zhao, X., Patro, G. K., Kasneci, G., & Kinder-Kurlanda, K. E. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534158 · Cited by 42
“Other research examples, once informed on practitioners’ needs, focused on designing different AI fairness solutions: checklists to be aligned with teams’ workflows and organizational ad-hoc processes, fairness frameworks or internal algorithmic auditing protocols designed for industrial applications [61, 91].”
BackgroundThe Concept of Accountability in AI Ethics and Governance
Lechterman, T. M. (2022) · The Oxford Handbook of AI Governance · doi:10.1093/oxfordhb/9780197579329.013.10 · Cited by 23
International Governance of Civilian AI: A Jurisdictional Certification Approach
Trager, R., Harack, B., Reuel, A., Carnegie, A., Heim, L., Ho, L., Kreps, S., Lall, R., Larter, O., hÉigeartaigh, S. Ó., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.15514 · Cited by 23
Torkamaan, H., Tahaei, M., Buijsman, S., Xiao, Z., Wilkinson, D., & Knijnenburg, B. P. (2024) · Human-computer interaction series · doi:10.1007/978-3-031-55109-3_2 · Cited by 40
Towards a Better Understanding of Evaluating Trustworthiness in AI Systems
Kemmerzell, N., Schreiner, A., Khalid, H., Schalk, M., & Bordoli, L. (2025) · ACM Computing Surveys · doi:10.1145/3721976 · Cited by 45
“…openness, this strategy centers around examining the completeness and transparency of documentation regarding the AI system’s development process [52, 242], including data sources, algorithms, decision-making processes, and compliance with ethical guidelines [163, 205] and legal requirements [133].”
Attard-Frost, B., & Widder, D. G. (2025) · Big Data & Society · doi:10.1177/20539517251340603 · Cited by 59
“Documentation, disclosure, and explanation of the data and computational resources, processes, and outcomes involved in ML development activities and automated decision-making activities is another transparency concern (Miceli et al., 2022b; Mitchell et al., 2019; Raji et al., 2020).”
Opportunities and Challenges of Foundation Models in Industrial Manufacturing
Ayyat, M., Osman, M., & Nadeem, T. (2025) · IEEE Access · doi:10.1109/access.2025.3589777 · Cited by 31
“Governance, Regulatory Compliance, and ROI Justification: Successful FM deployment requires organizational governance , clear validation pipelines, regulatory alignment, and quantifiable ROI [204], [235].”
How Good Is NLP? A Sober Look at NLP Tasks through the Lens of Social Impact
Jin, Z., Chauhan, G., Tse, B., Sachan, M., & Mihalcea, R. F. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.273 · Cited by 45
Spotting When Algorithms Are Wrong
Buijsman, S., & Veluwenkamp, H. (2022) · Minds and Machines · doi:10.1007/s11023-022-09591-0 · Cited by 29
“It has, moreover, also been suggested as a basis for AI ethics audits (Raji et al., 2020).”
Explainable Artificial Intelligence for Smart Grid Intrusion Detection Systems
Yayla, A., Haghnegahdar, L., & Dincelli, E. (2022) · IT Professional · doi:10.1109/mitp.2022.3163731 · Cited by 30
Boch, A., Ryan, S., Kriebitz, A., Amugongo, L. M., & Lütge, C. (2023) · Robotics · doi:10.3390/robotics12040110 · Cited by 32
Yu, J. (2025) · Transport Policy · doi:10.1016/j.tranpol.2025.05.030 · Cited by 22
Nezami, N., Haghighat, P., Gándara, D., & Anahideh, H. (2024) · Education Sciences · doi:10.3390/educsci14020136 · Cited by 32
Frontier AI developers need an internal audit function
Schuett, J. (2024) · Risk Analysis · doi:10.1111/risa.17665 · Cited by 55
“Let us first define some of the key terms used in this article, namely, “internal audit,” “AI,” and “frontier AI,” as well as related terms like “general-purpose AI (GPAI),” “foundation model,” and “artificial general intelligence (AGI).””
The Doctor and the Algorithm: Promise, Peril, and the Future of Health AI
Graham, S. S. (2022) · doi:10.1093/oso/9780197644461.001.0001 · Cited by 32
Taking Algorithms to Courts: A Relational Approach to Algorithmic Accountability
Metcalf, J., Singh, R., Moss, E., Tafesse, E., & Watkins, E. A. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594092 · Cited by 21
Towards a Responsible AI Metrics Catalogue: A Collection of Metrics for AI Accountability
Xia, B., Lu, Q., Zhu, L., Lee, S. U., Liu, Y., & Xing, Z. (2024) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3644815.3644959 · Cited by 74
Fairness Without Demographic Data: A Survey of Approaches
Ashurst, C., & Weller, A. (2023) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3617694.3623234 · Cited by 21
“Algorithmic impact assessments [112], audits [28, 108] and avenues for complaints can surface fairness concerns.”
BackgroundMihaljević, H., Müller, I., Dill, K., Yollu-Tok, A., & von Grafenstein, M. (2023) · AI & Society · doi:10.1007/s00146-023-01726-w · Cited by 31
“…no matter who provides them, cannot capture all relevant impacts resulting from the usage of the respective products, as they cannot shed light on process-and organization-related aspects (Green and Chen 2021; Raji et al. 2020) as well as the comprehensive social processes (Engler 2021a; Kim 2017).”
InfluentialA Comparative Assessment and Synthesis of Twenty Ethics Codes on AI and Big Data
Loi, M., Heitz, C., & Christen, M. (2020) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds49233.2020.00015 · Cited by 28
The epistemic revolution of AI: reconfiguring the foundations of scientific knowledge
Cruz‐Aguilar, M. A. (2025) · AI & Society · doi:10.1007/s00146-025-02658-3 · Cited by 34
From Demo to Design in Teaching Machine Learning
Bilstrup, K.-E. K., Kaspersen, M. H., Assent, I., Enni, S. A., & Petersen, M. G. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3534634 · Cited by 30
“As a response, a number of recent papers in the ML field suggest new ways of reflecting on and documenting the ML process [45, 52, 55, 59, 61].”
BackgroundIntegrating Fairness in the Software Design Process: An Interview Study With HCI and ML Experts
Ryan, S., Nadal, C., & Doherty, G. J. (2023) · IEEE Access · doi:10.1109/access.2023.3260639 · Cited by 33
“This has included co-design checklists [13], auditing and accountability frameworks [32], and other toolkits [33] focused on the assessment and potential mitigation of perceived (un)fairness.”
BackgroundReclaiming the Digital Commons: A Public Data Trust for Training Data
Chan, A., Bradley, H., & Rajkumar, N. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604658 · Cited by 43
Lu, Q., Luo, Y., Zhu, L., Tang, M., Xu, X., & Whittle, J. (2023) · IEEE Intelligent Systems · doi:10.1109/mis.2023.3320437 · Cited by 33
“To try to bridge the principle-algorithmic gap, further guidance, such as guidebooks a,b question banks, 2 check-lists, 3 and documentation templates, 4,5 have begun to emerge.”
BackgroundHuman-Centered Evaluation and Auditing of Language Models
Xiao, Z., Deng, W. H., Lam, M. S., Eslami, M., Kim, J., Lee, M., & Liao, Q. V. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3636302 · Cited by 40
Law and the Emerging Political Economy of Algorithmic Audits
Terzis, P., Veale, M., & Gaumann, N. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658970 · Cited by 39
“The idea of auditing online platforms and algorithms as a form of governance is not new nor is its critique [24, 31, 69, 109, 106].”
BackgroundEthical Guidelines and Principles in the Context of Artificial Intelligence
de Cerqueira, J. A. S., Tives, H. A., & Canedo, E. D. (2021) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3466933.3466969 · Cited by 29
“Alguns exemplos de frameworks para implementar ética em IA incluem: [1] e [36] do setor privado, [27] da academia, [35] parceria”
BackgroundBubinger, H., & Dinneen, J. D. (2024) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2024.102943 · Cited by 28
Confounding Complexity of Machine Action: A Hobbesian Account of Machine Responsibility
Sætra, H. S. (2020) · International Journal of Technoethics · doi:10.4018/ijt.20210101.oa1 · Cited by 18
“Furthermore, much work in responsible AI and robotics involve attempts to increase accountability, transparency, and the traceability of error in AI and robotic systems (Raji et al., 2020; Winfield & Jirotka, 2017).”
BackgroundAuditing the Information Quality of News-Related Queries on the Alexa Voice Assistant
Dambanemuya, H. K., & Diakopoulos, N. A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449157 · Cited by 28
“Periodic auditing of information quality from voice assistants may be necessary, or could be imagined as a more fully integrated aspect of the engineering development process [54].”
BackgroundThe Switch, the Ladder, and the Matrix: Models for Classifying AI Systems
Mökander, J., Sheth, M., Watson, D., & Floridi, L. (2023) · Minds and Machines · doi:10.1007/s11023-022-09620-y · Cited by 28
“Moreover, the industry lacks both incentives and useful tools to translate abstract principles into verifiable criteria (Morley et al., 2020; Raji et al., 2020).”
BackgroundAI Accountability: Approaches, Affecting Factors, and Challenges
Raja, A. K., & Zhou, J. J. (2023) · Computer · doi:10.1109/mc.2023.3238390 · Cited by 35
Nishal, S., Sinchai, J., & Diakopoulos, N. A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637419 · Cited by 33
“Implementing internal audits of these systems could also provide a systematic understanding of what the computed subsidies could be missing, and bring to light broader bias patterns that arise if they are used uncritically [101], leading to better design and well-informed usage.”
BackgroundPeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software
Jiang, W., Yasmin, J., Jones, J., Synovic, N. M., Kuo, J., Bielanski, N., Tian, Y., Thiruvathukal, G. K., & Davis, J. C. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644907 · Cited by 49
A Capability Approach to AI Ethics
Ratti, E., & Graves, M. (2025) · American Philosophical Quarterly · doi:10.5406/21521123.62.1.01 · Cited by 23
Accountability in Government Use of AI : Citizen Concerns and Preferences
Feng, N., & Chandra, Y. (2025) · Public Administration · doi:10.1111/padm.70030 · Cited by 18
Towards Model-Agnostic Ensemble Explanations
Bobek, S., Bałaga, P., & Nalepa, G. J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77970-2_4 · Cited by 24
“Another example of auditing framework was presented in [12].”
BackgroundRhea, A. K., Markey, K., D’Arinzo, L., Schellmann, H., Sloane, M., Squires, P., & Stoyanovich, J. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534189 · Cited by 26
“Several audits and framework documents emphasize the importance of algorithmic impact assessment and stakeholder evaluations [7, 17, 41, 43, 44, 59].”
BackgroundQuaye, J. A., Parrish, A., Inel, O., Rastogi, C., Kirk, H. R., Kahng, M., van Liemt, E., Bartolo, M., Tsang, J., White, J., et al. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658913 · Cited by 52
“Initial works have red-teamed risks in domains like human-AI dialogue [18, 19, 35] and T2I generation [26, 37, 43, 58].”
BackgroundBalayn, A., Yurrita, M., Rancourt, F., Casati, F., & Gadiraju, U. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713787 · Cited by 50
“As for circumventing the risks that reliance under miscalibra-tion and distrust bring, organizations could focus on (i) promoting transparency and enhanced accountability for contentious decisions [29, 106], and (ii) ensuring the upholding of clear distributed responsibilities [99, 139].”
Naja, I., Marković, M., Edwards, P. J., Pang, W., Cottrill, C., & Williams, R. A. (2022) · IEEE Access · doi:10.1109/access.2022.3188967 · Cited by 33
“…the approach outlined in this paper on the assumption that AI systems, algorithms, or machines should not be treated as legally responsible or liable agents [29], [30], and that as a result, human agents or organizations that created or influenced the behaviors of AI systems should be accountable.”
BackgroundA Framework for Assurance Audits of Algorithmic Systems
Lange, B., Lam, K., Blili-Hamelin, B., Davidovic, J., Brown, S., & Hasan, A. (2024) · PhilPapers (PhilPapers Foundation) · Cited by 17
Kuehnert, B., Kim, R., Forlizzi, J., & Heidari, H. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732191 · Cited by 47
Chow, J., Lee, R., & Wu, H. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01493-8 · Cited by 27
Making Sense of AI Systems Development
Dolata, M., & Crowston, K. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3338857 · Cited by 22
“Past research assumes that sense-making is an ongoing process which mostly happens implicitly, but suggests that paying attention to sensemaking processes is beneficial as it raises sensitivity to the possibility of problems and the need to respond [25].”
AI research ethics is in its infancy: the EU’s AI Act can make it a grown-up
Rességuier, A., & Ufert, F. (2023) · Research Ethics · doi:10.1177/17470161231220946 · Cited by 29
“The third area of adaptation concerns the need to ensure transparency from the stage of research through proper reporting mechanisms , a key component of AI governance that is essential to promote accountability, conduct audits (Raji et al., 2020), and check for potential biases in the system.”
BackgroundReconfiguring Participatory Design to Resist AI Realism
Gautam, A. (2024) · Participatory Design Conference · doi:10.1145/3661455.3669867 · Cited by 21
“Among the unique ones, AI systems often lack transparency in how it was created and evaluated, in data collection, model development, fine-tuning, and impact evaluation [23, 39, 40, 42].”
BackgroundDecoloniality, Digital-coloniality and Computer Programming Education
Geyser, H. (2024) · ACM Transactions on Computing Education · doi:10.1145/3702332 · Cited by 21
“When image recognition software misrecognises race and gender, causing lived harm [54, 55, 159–161], it may not be intentional.”
Havrda, M., & Rakova, B. (2020) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc42975.2020.9283137 · Cited by 27
“Various scholars have also proposed checklist approaches [18] as well as end-to-end algorithmic auditing frameworks [19].”
BackgroundKasinidou, M., Kleanthous, S., Orphanou, K., & Otterbacher, J. (2021) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3430665.3456311 · Cited by 28
Business Data Ethics: Emerging Trends in the Governance of Advanced Analytics and AIPreprint
Hirsch, D. D., Bartley, T., Chandrasekaran, A., Norris, D., Parthasarathy, S., & Turner, P. N. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3828239 · Cited by 19
From Explainability to Ineffability? ML Tarot and the Possibility of Inspiriting Design
Lustig, C., & Rosner, D. K. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533552 · Cited by 16
Peace and Prosperity for the Digital Age? The Colonial Political Economy of European AI Governance
Carmel, E., & Paul, R. (2022) · IEEE Technology and Society Magazine · doi:10.1109/mts.2022.3173340 · Cited by 25
Beattie, L., Taber, D., & Cramer, H. S. M. (2022) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3523227.3547403 · Cited by 21
“began to address this need by introducing SMACTR to provide a framework for a wider auditing context [12].”
BackgroundHuman achievement and artificial intelligence
Karlan, B. (2023) · Ethics and Information Technology · doi:10.1007/s10676-023-09713-x · Cited by 20
Perceptive Biases in Construction Mediation: Evidence and Application of Artificial Intelligence
Cao, N., Cheung, S. O., & Li, K. (2023) · Buildings · doi:10.3390/buildings13102460 · Cited by 23
Supporting User Engagement in Testing, Auditing, and Contesting AI
Deng, W. H., Lam, M. S., Cabrera, Á. A., Metaxa, D., Eslami, M., & Holstein, K. (2023) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3584931.3611279 · Cited by 23
Why Predictive Algorithms are So Risky for Public Sector BodiesPreprint
Waller, M., & Waller, P. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3716166 · Cited by 20
ESR: Ethics and Society Review of Artificial Intelligence ResearchPreprint
Bernstein, M. S., Levi, M., Magnus, D. C., Rajala, B., Satz, D. M., & Waeiss, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11521 · Cited by 30
“Further, it remains unclear whose responsibility ethics is within an organization [54], despite a need to audit AI systems throughout the development process [63].”
BackgroundFrom Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing
Cen, S. H., & Alur, R. (2024) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3689904.3694711 · Cited by 55
“Raji et al. [107] propose an end-to-end framework for internal audits, e.g., those which are voluntarily conducted by a company or model provider, which nevertheless provides guidance on audits more broadly.”
BackgroundSkeja, A., & Sadiku-Dushi, N. (2025) · Leadership & Organization Development Journal · doi:10.1108/lodj-06-2025-0530 · Cited by 22
An HCAI Methodological Framework: Putting it Into Action to Enable Human-Centered AI
Xu, W., Gao, Z., & Dainoff, M. J. (2026) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2025.3631590 · Cited by 97
Building Knowledge through Action: Considerations for Machine Learning in the Workplace
Lindley, S. E., & Wilkins, D. J. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3584947 · Cited by 18
“Furthermore, delegating the responsibility of verification to workplace systems raises complex and unanswered societal-level questions about who is accountable for decisions that are made by, or with influential support from, ML [e.g., Doshi-Velez et al., 2017; Raji et al., 2020].”
Background
Metadata
- Authors
- Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P.
- Affiliations
- Partnership on AI
- Google (United States) (US)
- ORCID
- Venue
- Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2020-01-27
- Pages
- 33-44
- OpenAlex
- W3001807593
- Cited by
- 2,245
- References
- 83
- Citations by year
- 2026: 1097 · 2025: 454 · 2024: 243 · 2023: 177 · 2022: 149 · 2021: 99 · 2020: 24
- FWCI
- 35.91
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- AI accountability, algorithmic auditing, harm detection, algorithmic transparency, societal impact of AI, algorithmic governance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.