Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing
Open accessThe DOI matches its Crossref record; 96 of the 97 references it cites were found in a registry or an index, and 1 are not checked yet; 11 of the 44 sources its claims cite were checked for support.
Abstract
Rising concern for the societal implications of artificial intelligence systems has inspired a wave of academic and journalistic literature in which deployed systems are audited for harm by investigators from outside the organizations deploying the algorithms. However, it remains challenging for practitioners to identify the harmful repercussions of their own systems prior to deployment, and, once deployed, emergent issues can become difficult or impossible to trace back to their source.
Verified 2026-11-19 · 1 reference not checked yet
Overview
CoChat Summary
TL;DR
The paper proposes SMACTR, an end-to-end internal algorithmic auditing framework intended to help organizations assess AI systems against declared ethical principles throughout development. It argues that internal audits can complement external accountability by providing pre-deployment access, structured documentation, and risk analysis drawn from aerospace, medical device, and finance practices.
Key findings
- Introduces SMACTR—Scoping, Mapping, Artifact Collection, Testing, and Reflection—as five audit stages, each with documentation requirements that together form an overall audit report and assess decisions against organizational values or AI principles.
- Argues external audits are limited by lack of access to internal processes, intermediate models, and training data, while internal audits can enable proactive pre-deployment intervention and generate artifacts for external review.
- Draws lessons from other industries, citing aerospace’s one commercial airline accident per two million flights, Boeing 787 software estimated at 13 million lines of code, and a 1-in-1,000,000,000 per use hour maximum failure probability for critical aerospace systems; also cites medical-device design controls and finance internal audit infrastructure.
- Uses hypothetical cases—a child abuse screening tool and a smile-detection photo booth—to illustrate the framework; for the smile detector’s CelebA datasheet, it reports 58.1% female and 42% male labels, 77.8% aged 0–45 and 22.1% over 46, and 14.2% lighter-skinned versus 85.8% darker-skinned.
Methods
- Conceptual framework paper: develops SMACTR from auditing practices and artifacts in aerospace, medical devices, and finance, plus responsible-innovation dimensions of anticipation, reflexivity, inclusion, and responsiveness.
- Uses a hypothetical Company X Inc. consultancy with two invented client projects—a child abuse screening tool and Happy-Go-Lucky Inc.’s smile-detection photo booth—to demonstrate audit stages and artifacts; a worked example for the smile case is provided as supplementary material.
- Analysis is narrative and design-oriented, referencing existing standards and tools such as FMEA, model cards, datasheets, design controls, and design history files; no empirical sample or quantitative evaluation is reported.
Limitations
- The paper states that internal auditors necessarily share an organizational interest with the audit target, making an independent and objective viewpoint challenging; audits are not an objective “view from nowhere” and risk becoming reputation management.
- The paper states that internal audits are only one aspect of a broader system of required quality checks and balances, and that determining what systems to audit and why is not covered and is a separate research topic.
- Evident from the text: the framework is not empirically validated, the illustrative cases are hypothetical, and no real deployment outcomes or measured audit effectiveness are reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
9 Valid · 6 Not valid · 27 Not checked · 30 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“While not without faults, safety-critical and regulated industries such as aerospace and medicine have long traditions of auditable processes and design controls that have dramatically improved safety [77,81].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 3 LESSONS FROM AUDITING PRACTICES IN OTHER INDUSTRIES
What the sources say
- Not checked — no full text of the sourcecites [77] matched to “Design controls for the medical device industry” (2013) source
- Could not verifycites [81] matched to “The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions” (2019) sourceExperimental — accuracy not yet measured
The Introduction and §3 discuss biomedical ethics and standards or regulation, but do not say that aerospace or medicine has long used auditable processes and design controls, or that these dramatically improved safety. Searches for aerospace, auditable processes, design controls, and safety-critical industries found no passage addressing that claim. Page numbers are unavailable in our copy. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This is a repeatedly observed phenomenon in tax compliance auditing, where several international surveys of tax compliance demonstrate that a fixed and vetted tax audit methodology is one of the most effective strategies to convince companies to respect audit results and pay their full taxes [22,53].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.3 Audit Integrity and Procedural Justice
What the sources say
- Partly supportedcites [22] matched to “Perception on justice, trust and tax compliance behavior in Malaysia” (2017) sourceExperimental — accuracy not yet measured
The Regression Test section reports that Malaysian respondents would be more compliant if tax authorities treated them fairly in tax procedures. The Research Methodology section describes a questionnaire of academics, not several international surveys of companies. The paper does not establish that a fixed, vetted audit methodology is especially effective at securing acceptance of audit results or full tax payment. (LLM, conf=0.86)
- Partly supportedcites [53]Experimental — accuracy not yet measured
The “Justice in the taxation context” section reports that taxpayers are generally more compliant when treated fairly and respectfully, and the General discussion links perceived unfair treatment to resistance to paying back tax (page numbers unavailable). It does not establish that a fixed, vetted audit methodology is among the most effective strategies, or report international surveys demonstrating that effect for companies. (LLM, conf=0.83)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Traditional metrics for artificial intelligence like loss may conceal fairness concerns, social impact risks or abstraction errors [69].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.3.2 Artifact: Ethnographic Field Study.
What the sources say
- Supportedcites [69] matched to “Fairness and Abstraction in Sociotechnical Systems” (2019) source
The Introduction (page not provided) says that narrow assessments of machine-learning performance abstract away social context and can miss information needed for fair outcomes—an abstraction error. The Formalism Trap section adds that mathematical measures cannot capture the full range of fairness concerns, while the Takeaways discuss unintended social consequences. The paper names AUC, precision and recall rather than loss, but the claim uses loss as an illustrative metric. (LLM, conf=0.88)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Internal auditors' direct access to systems can thus help extend traditional external auditing paradigms by incorporating additional information typically unavailable for external evaluations to reveal previously unidentifiable risks.”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.4 The Internal Audit
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The essential inclusion of independent domain experts and marginalized groups in the ethical review process “has the potential to lead to more rigorous critical reflection because their experiences will often be precisely those that are most needed in identifying problematic background assumptions and revealing limitations with research questions, models, or methodologies” [33].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.2.1 Artifact: Ethical Review of System Use Case.
What the sources say
- Not checked — no full text of the sourcecites [33] matched to “25 Years of Feminist Empiricism and Standpoint Theory: Where Are We Now?” (2010) source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
19 more claims
Create a free account to see all 72 claims and every source verdict.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 37
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This disparity poses risks, as practitioners may unintentionally embed their own biases into AI system design and implementation (Martin 2019;Raji et al. 2020).”
From Accountability in Government Use of AI : Citizen Concerns and Preferences (2025) · cites this paper as Raji et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Work on AI safety and reproducibility has long pointed out the crucial importance of training data for understanding model performance, ensuring reproducibility, and assessing legal exposure [2, 25, 29, 46, 58].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While there is a growing interest in auditing algorithmic systems [125,126,128,138,154], this feld generally lacks defned standards and records could be incomplete or not relating to information needed for accountability purposes [15,127].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [128]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Raji et al. (2020) proposed an internal algorithmic auditing framework spanning data collection, model development, testing, deployment, and monitoring, with stage-specific artifacts that enable traceability and accountability.”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Raji et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Practical approaches to implement such ideas typically revolve around the systematic collection and curation of data on various aspects of systems [2, 28, 46].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747 · cited 1×
“A hypothetical datasheet for this system is filled out using studies on the CelebA dataset, with which the smile detector is built [44, 47].”
MethodThe global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Product development: Past research, present findings and future directions
Brown, S. L., & Eisenhardt, K. M. (1995) · The Academy of Management Review · doi:10.2307/258850 · Cited by 3,777 · cited 1×
“There are also intersections between this framework and just efective product development theory [5], as many of the components of audit design refocus the product development process to prioritize the user and their ultimate well-being, resulting in a more efective product performance outcome.”
Background(2018) · Cited by 3,510
Christiane, Druml, Wolzt, & Pleiner (2001) · Acta Scientiarum Naturalium Universitatis Sunyatseni · Cited by 3,419
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196
Developing a framework for responsible innovation
Stilgoe, J., Owen, R., & Macnaghten, P. (2013) · Research Policy · doi:10.1016/j.respol.2013.05.008 · Cited by 3,184 · cited 1×
“To design our audit procedure, we suggest complementing formal risk assessment methodologies with ideas from responsible innovation, which stresses four key dimensions: anticipation, reflexivity, inclusion and responsiveness [73], as well as system-theoretic concepts that help grapple with increasing complexity and co…”
MethodAlgorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Self-citation(2014) · Journal of the Korean Medical Association · doi:10.5124/jkma.2014.57.11.899 · Cited by 2,566
Research works printed but not linked: 7
- Virginia Eubanks. 2018. A child abuse prediction model fails poor families. Wired Magazine (2018).
- Jeremy Goldhaber-Fiebert and Lea Prince. 2019. Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny CountyâĂŹs Child Welfare Office. Pittsburgh: Allegheny County.[Google Scholar] (2019).
- Nitin Kohli, Renata Barreto, and Joshua A Kroll. 2018. Translation Tutorial: A Shared Lexicon for Research and Practice in Human-Centered Software Systems. In 1st Conference on Fairness, Accountability, and Transparancy. New York, NY, USA. 7.
- Joshua A Kroll, Solon Barocas, Edward W Felten, Joel R Reidenberg, David G Robinson, and Harlan Yu. 2016. Accountable algorithms. U. Pa. L. Rev. 165 (2016), 633.
- Institute of Internal Auditors. Research Foundation and Institute of Internal Au- ditors. 2007. The Professional Practices Framework. Inst of Internal Auditors.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22 (2014).
- Tim Vanderveen. 2005. Averting highest-risk errors is first priority. Patient Safety and Quality Healthcare 2 (2005), 16-21.
Cited by
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724
Shneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768
“However, they can produce important internal efforts such as Google’s internal review and algorithmic auditing framework [92] (see Section 3.”
BackgroundTrustworthy Artificial Intelligence: A Review
Kaur, D., Uslu, S., Rittichier, K. J., & Durresi, A. (2023) · ACM Computing Surveys · doi:10.1145/3491209 · Cited by 709
“In-the-loop 2 [55, 55, 65, 132, 152, 188] 2, 3 [159, 174]”
BackgroundLanger, M., Oster, D., Speith, T., Hermanns, H., Kästner, L., Schmidt, E., Sesing, A., & Baum, K. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103473 · Cited by 652
“With increasing use of artificial systems, accountability gaps might emerge [134, 135].”
BackgroundTrustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“7, documentation is an effective and important approach to enhance the system’s transparency and accountability by tracking, guiding, and auditing its entire lifecycle [272] and serves as a cornerstone of building a trustworthy AI system.”
BackgroundMethodInfluentialResponsible Artificial Intelligence Governance: A Review and Research Framework
Papagiannidis, E., Mikalef, P., & Conboy, K. (2025) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2024.101885 · Cited by 565
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548
“Model or algorithmic audits [264] can be used to assess and document such crucial accountability considerations.”
BackgroundSchiff, D. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00270-2 · Cited by 489
“This could include, for example, requiring educational institutions that use algorithms to provide publicly accessible explanations for any decisions made, document these decisions, and establish venues to contest potentially problematic results or processes (Raji et al. 2020).”
BackgroundMethodMunn, L. (2023) · AI and Ethics · doi:10.1007/s43681-022-00209-w · Cited by 355
“The article thus concludes by highlighting alternatives: the first thinks more broadly about AI justice, considering sociopolitical dynamics and systems of oppression [14, 46]; the second thinks more narrowly, focusing on concrete issues like accuracy, auditing, and governance [25, 67].”
BackgroundLanger, M., & Landers, R. N. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106878 · Cited by 338
Carrington, A. M., Manuel, D. G., Fieguth, P. W., Ramsay, T., Osmani, V., Wernly, B., Bennett, C., Hawken, S., Magwood, O., Sheikh, Y., et al. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3145392 · Cited by 339
Software Engineering for AI-Based Systems: A Survey
Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 335
“Furthermore, five articles propose ML quality certification and assessment to mitigate quality risks of deployed AI systems [12, 94, 145, 163, 224].”
BackgroundInfluentialRakova, B., Yang, J., Cramer, H., & Chowdhury, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449081 · Cited by 329
“Internal efforts in turn have access to more data, ensure that auditing can be completed before public releases, develop processes for companies, and allow companies to take responsibility for their impact [29].”
BackgroundInfluentialPredictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., DasSarma, N., Henighan, T., Jones, A., Joseph, N., Kernion, J., Mann, B., Askell, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2202.07785 · Cited by 324
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306
“[36] for a discussion of some lessons for AI from auditing in other industries.”
BackgroundAuditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Should ChatGPT be biased? Challenges and risks of bias in large language models
Ferrara, E. (2023) · First Monday · doi:10.5210/fm.v28i11.13346 · Cited by 282
“• Accountability: It is essential for developers and stakeholders to establish a clear framework for accountability, which may include monitoring the performance of AI models, addressing biases and errors, and responding to the concerns of users and affected communities [94, 95, 96].”
BackgroundShould ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models
Ferrara, E. (2023) · arXiv · doi:10.48550/arxiv.2304.03738 · Cited by 282
Ethics as a Service: A Pragmatic Operationalisation of AI Ethics
Morley, J., Elhalal, A., Garcia, F., Kinsey, L., Mökander, J., & Floridi, L. (2021) · Minds and Machines · doi:10.1007/s11023-021-09563-w · Cited by 266
Ethics-Based Auditing to Develop Trustworthy AI
Mökander, J., & Floridi, L. (2021) · Minds and Machines · doi:10.1007/s11023-021-09557-8 · Cited by 268
Trustworthy ai: A computational perspective
Liu, H., Wang, Y., Fan, W., Liu, X., Li, Y., Jain, S., Liu, Y., Jain, A., & Tang, J. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3546872 · Cited by 261
“SMACTR [265] is a recent internal auditing framework proposed by researchers from Google and Partnership on AI.”
MethodHutchinson, B., Smart, A., Hanna, A., Denton, R., Greer, C., Kjartansson, O., Barnes, P., & Mitchell, M. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445918 · Cited by 279
“or opportunistically. From an observer’s perspective, these practices replace abductive and a posteriori reasoning about dataset provenance with careful documentation, facilitating reviews and audits [114]. They help us to avoid the trap of being blindly “data-driven” by poorly-understood datasets— mistaking taking a…”
BackgroundInfluentialSelf-citationThe Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251
Self-citationLiu, X., Glocker, B., McCradden, M. M., Ghassemi, M., Denniston, A. K., & Oakden-Rayner, L. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00003-6 · Cited by 266
Lo Piano, S. (2020) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-0501-9 · Cited by 229
“Raji et al. (2020) suggest that a process of algorithmic auditing within the software-development company could help in tackling some of the ethical issues raised.”
BackgroundThe ethical debate about the gig economy: A review and critical analysis
Tan, Z. M., Aggarwal, N., Cowls, J., Morley, J., Taddeo, M., & Floridi, L. (2021) · Technology in Society · doi:10.1016/j.techsoc.2021.101594 · Cited by 225
National strategic artificial intelligence plans: A multi-dimensional analysis
Fatima, S., Desouza, K. C., & Dawson, G. S. (2020) · Economic Analysis and Policy · doi:10.1016/j.eap.2020.07.008 · Cited by 216
A systematic review of artificial intelligence impact assessments
Stahl, B. C., Antoniou, J., Bhalla, N., Brooks, L., Jansen, P., Lindqvist, B., Kirichenko, A., Marchal, S., Rodrigues, R., Santiago, N., et al. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10420-8 · Cited by 212
Technology readiness levels for machine learning systems
Lavin, A., Gilligan-Lee, C. M., Visnjic, A., Ganju, S., Newman, D., Ganguly, S., Lange, D., Baydin, A. G., Sharma, A., Gibson, A., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33128-9 · Cited by 209
Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations
Mökander, J., Morley, J., Taddeo, M., & Floridi, L. (2021) · Science and Engineering Ethics · doi:10.1007/s11948-021-00319-4 · Cited by 207
“Such checklists are simple tools that help designers get a more informed view of edge cases and system failures (Raji et al. 2020).”
BackgroundInfluentialPredictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533229 · Cited by 207
“Along with innovations in governance of AI organizations, and work on best practices, we also believe governments should invest in better methods to assure the benefits of systems being deployed - specifically, governments should support efforts to measure and monitor the capabilities (both harmful and beneficial) of…”
MethodThe algorithm audit: Scoring the algorithms that score us
Brown, S., Davidovic, J., & Hasan, A. (2021) · Big Data & Society · doi:10.1177/2053951720983865 · Cited by 204
“…are either too high level to be put into practice without further guidance (Barocas et al., 2013; Floridi et al., 2018; Mittelstadt et al., 2016; Raji et al., 2020; Sandvig et al., 2014), or they focus on narrow and often technical notions of fairness, bias, or transparency (Mitchell et al.,…”
BackgroundAlgorithmic Impact Assessments and Accountability: The Co-construction of Impacts
Metcalf, J., Moss, E., Watkins, E. A., Singh, R., & Elish, M. C. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445935 · Cited by 195
“As momentum builds for algorithmic impact assessments (AIAs) such as those laid out in several promising frameworks [58, 96, 97, 102], to serve as a mechanism for addressing the unintended consequences of algorithmic systems, the question of what constitutes an “impact” in the context of an AIA has never been more imp…”
A Review of Practical AI for Remote Sensing in Earth Sciences
Janga, B., Asamani, G., Sun, Z., & Cristea, N. (2023) · Remote Sensing · doi:10.3390/rs15164112 · Cited by 194
“As discussed above, maintaining integrity in remote sensing data involves multiple aspects, including data quality, data integrity, and the prevention of tampering or manipulation [179].”
BackgroundProblematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Bandy, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449148 · Cited by 193
Friha, O., Amine Ferrag, M., Kantarci, B., Cakmak, B., Ozgun, A., & Ghoualmi-Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 189
Recommendations for ethical and responsible use of artificial intelligence in digital agriculture
Dara, R., Hazrati Fard, S. M., & Kaur, J. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.884192 · Cited by 186
The TESCREAL bundle: Eugenics and the promise of utopia through artificial general intelligence
Gebru, T., & Torres, É. P. (2024) · First Monday · doi:10.5210/fm.v29i4.13636 · Cited by 181
Self-citationEthical and legal challenges of AI in marketing: an exploration of solutions
Kumar, D., & Suthar, N. (2024) · Journal of Information, Communication and Ethics in Society · doi:10.1108/jices-05-2023-0068 · Cited by 180
Singhal, A., Neveditsin, N., Tanveer, H., & Mago, V. (2024) · JMIR Medical Informatics · doi:10.2196/50048 · Cited by 172
Shen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171
“In response to these concerns, scholars have proposed a number of approaches to audit algorithmic systems for biased, discriminatory, or otherwise harmful behaviors 1 — both internally, within the organizations responsible for developing and maintaining these systems, and externally [73, 76].”
BackgroundEthics by design for artificial intelligence
Brey, P., & Dainow, B. (2024) · AI and Ethics · doi:10.1007/s43681-023-00330-4 · Cited by 169
Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers
Solanki, P., Grundy, J., & Hussain, W. (2023) · AI and Ethics · doi:10.1007/s43681-022-00195-z · Cited by 157
A manifesto on explainability for artificial intelligence in medicine
Combi, C., Amico, B., Bellazzi, R., Holzinger, A., Moore, J. H., Zitnik, M., & Holmes, J. H. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102423 · Cited by 163
AI auditing: The Broken Bus on the Road to AI Accountability
Birhane, A., Steed, R., Ojewale, V., Vecchione, B., & Raji, I. D. (2024) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00037 · Cited by 158
“Online platform audits, for example, cite social science and critical race studies as inspiration [13, 14], as do several audits of automated decision”
BackgroundMethodInfluentialSelf-citationDesigning for Responsible Trust in AI Systems: A Communication Perspective
Liao, Q. V., & Sundar, S. S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533182 · Cited by 165
Floridi, L., Holweg, M., Taddeo, M., Amaya, J., Mökander, J., & Wen, Y. (2022) · doi:10.2139/ssrn.4064091 · Cited by 157
Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support
Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., & Wallach, H. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512899 · Cited by 156
Mökander, J., Axente, M., Casolari, F., & Floridi, L. (2022) · Minds and Machines · doi:10.1007/s11023-021-09577-4 · Cited by 153
“However, external audits are fundamentally limited by a lack of access to internal processes at the audited organisation (Raji et al., 2020).”
BackgroundInfluentialGhobakhloo, M., Fathi, M., Iranmanesh, M., Vilkas, M., Grybauskas, A., & Amran, A. (2024) · Journal of Manufacturing Technology Management · doi:10.1108/jmtm-12-2023-0530 · Cited by 203
Auditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152
“However, previous research suggests that technology providers too have strong incentives to subject the AI systems they design and deploy to independent audits (Falco et al., 2021; Raji et al., 2020).”
BackgroundBogina, V., Hartman, A., Kuflik, T., & Shulner-Tal, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00248-0 · Cited by 142
“Raji et al. (2020) suggested a framework for managing the auditing process in an industrial setting to ensure responsible development in AI.”
BackgroundReviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems
Cobbe, J., Lee, M. S. A., & Singh, J. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445921 · Cited by 150
“Even where assessments are not legally required, they are good practice, and robust internal (in-organisation) assessment processes have value [60].”
BackgroundHarms from Increasingly Agentic Algorithmic Systems
Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594033 · Cited by 149
Building and Auditing Fair Algorithms: A Case Study in Candidate Screening
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445928 · Cited by 133
“…algorithm auditing to reverse engineering by outsiders, in that the goal is to make “black-box” systems more transparent regardless of the system creator’s intent [13], this conceptualization has since been expanded to include audits carried out by ethically and morally-conscious insiders [51].”
BackgroundInfluentialCooper, A. F., Moss, E., Laufer, B., & Nissenbaum, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533150 · Cited by 129
Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
Gajos, K. Z., & Mamykina, L. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511138 · Cited by 127
Corporate Governance of Artificial Intelligence in the Public Interest
Cihon, P., Schuett, J., & Baum, S. D. (2021) · Information · doi:10.3390/info12070275 · Cited by 123
“Additionally, management can encourage or require the use of documentation methods that generate and distribute information needed to ensure compliance with AI principles [88].”
BackgroundFrom human resources to human rights: Impact assessments for hiring algorithms
Yam, J., & Skorburg, J. A. (2021) · Ethics and Information Technology · doi:10.1007/s10676-021-09599-7 · Cited by 170
“First, it is widely recognized that “the AI industry lacks proven methods to translate principles into practice and AI principles have been criticized for being vague” (Raji et al., 2020, p. 34).”
BackgroundRoberts, H., Zhang, J., Bariach, B., Cowls, J., Gilburt, B., Juneja, P., Tsamados, A., Ziosi, M., Taddeo, M., & Floridi, L. (2024) · AI & SOCIETY · doi:10.1007/s00146-022-01596-8 · Cited by 123
“These audits could be undertaken internally, based on regulator, academic, or industry guidance (Raji et al. 2020; Mökander et al. 2021; Mökander and Floridi 2021), 15 or by an external organisation offering AI auditing services.”
Actionable Principles for Artificial Intelligence Policy: Three Pathways
Stix, C. (2021) · Science and Engineering Ethics · doi:10.1007/s11948-020-00277-3 · Cited by 121
“to enable the auditing of AI systems by third parties) (Raji et al. 2020).”
BackgroundBuilding Human Values into Recommender Systems: An Interdisciplinary Synthesis
Stray, J., Halevy, A., Assar, P., Hadfield-Menell, D., Boutilier, C., Ashar, A., Bakalar, C., Beattie, L., Ekstrand, M., Leibowicz, C., et al. (2024) · ACM Transactions on Recommender Systems · doi:10.1145/3632297 · Cited by 122
“While companies can and do perform internal algorithm audits of various types (Chowdhury, 2021; Raji et al., 2020) regulation could require external audits of recommender systems to mitigate these concerns (Singh and Doty, 2021a).”
BackgroundOutsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance
Raji, I. D., Xu, P., Honigsberg, C., & Ho, D. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534181 · Cited by 209
Self-citationOperationalising AI governance through ethics-based auditing: an industry case study
Mökander, J., & Floridi, L. (2023) · AI and Ethics · doi:10.1007/s43681-022-00171-7 · Cited by 129
“Rather, organisations should undertake a gap finding and filling exercise, adding ethics-based evaluation criteria to existing procurement processes.”
BackgroundEmployee Perceptions of the Effective Adoption of AI Principles
Kelley, S. (2022) · Journal of Business Ethics · doi:10.1007/s10551-022-05051-y · Cited by 118
“Raji et al. (2020) propose an algorithmic auditing framework aimed to help organizations assess the fit of their AI decisions given their AIPs.”
BackgroundInfluentialBuçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
“Therefore, we conducted a limited internal audit [54] of our work to check for inequalities that might emerge and to understand how different groups of people are affected by the introduction of cognitive forcing functions into the AI-assisted decision-making processes.”
MethodResponsible AI Pattern Catalogue: A Collection of Best Practices for AI Governance and Engineering
Lu, Q., Zhu, L., Xu, X., Whittle, J., Zowghi, D., & Jacquet, A. (2024) · ACM Computing Surveys · doi:10.1145/3626234 · Cited by 121
“Continuous documentation using templates helps track the evolution of artifacts and clarify the context in which AI systems are trustworthy [7, 8, 9, 10, 11, 12].”
BackgroundSuresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445088 · Cited by 117
““anticipate ethics-related failures before launch" [97], bias or mistake detection [89, 101, 103, 125], understand skewness and biases in input data [35], find unknown vulnerabilities and flaws [5, 38] (8/58) T3: Understand the extent of the information the model is using data entanglement [97], be…”
BackgroundInfluentialMökander, J., Axente, M., Casolari, F., & Floridi, L. (2022) · PhilPapers (PhilPapers Foundation) · Cited by 115
Sætra, H. S. (2022) · Sustainable Development · doi:10.1002/sd.2438 · Cited by 161
How Different Groups Prioritize Ethical Values for Responsible AI
Jakesch, M., Buçinca, Z., Amershi, S., & Olteanu, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533097 · Cited by 116
A survey on artificial intelligence assurance
Batarseh, F. A., Freeman, L., & Huang, C. (2021) · Journal Of Big Data · doi:10.1186/s40537-021-00445-7 · Cited by 156
“A recent publication by Raji et al. (2020) shows a study from the Google team claiming that they are “aiming to close the accountability gap of AI” using an internal audit system (at Google).”
BackgroundA Survey of Privacy Risks and Mitigation Strategies in the Artificial Intelligence Life Cycle
Shahriar, S., Allana, S., Hazratifard, S. M., & Dara, R. (2023) · IEEE Access · doi:10.1109/access.2023.3287195 · Cited by 114
A toolbox for surfacing health equity harms and biases in large language models
Pfohl, S. R., Cole-Lewis, H., Sayres, R., Neal, D., Asiedu, M., Dieng, A., Tomasev, N., Rashid, Q. M., Azizi, S., Rostamzadeh, N., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03258-2 · Cited by 113
Self-citationWhat about investors? ESG analyses as tools for ethics-based AI auditing
Minkkinen, M., Niukkanen, A., & Mäntymäki, M. (2024) · AI & SOCIETY · doi:10.1007/s00146-022-01415-0 · Cited by 109
“Issues may become visible and material to stakeholders only through controversies, and by then, significant harm may have occurred to individuals and groups (cf. Raji et al. 2020).”
Policy advice and best practices on bias and fairness in AI
Álvarez, J. M., Colmenarejo, A. B., Elobaid, A., Fabbrizzi, S., Fahimi, M., Ferrara, A., Ghodsi, S., Mougan, C., Papageorgiou, I. T., Lobo, P. R., et al. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09746-w · Cited by 160
“…the development of documentation practices and accompanying artefacts that enhance AI audit pipelines (Gebru et al., 2021; Raji & Yang, 2019; Stoyanovich et al., 2022; Raji et al., 2020), thus enabling stakeholders to easily inspect all the actions performed across the many steps of the pipeline.”
Designing AI Using a Human-Centered Approach: Explainability and Accuracy Toward Trustworthiness
Schoenherr, J. R., Abbas, R., Michael, K., Rivas, P., & Anderson, T. D. (2023) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2023.3257627 · Cited by 105
Contestable AI by Design: Towards a Framework
Alfrink, K., Keller, I., Kortuem, G., & Doorn, N. (2022) · Minds and Machines · doi:10.1007/s11023-022-09611-z · Cited by 169
“Future efforts may seek to compare our proposed framework to more generic ethical, responsible and accountable AI frameworks (e.g. Cobbe et al., 2021; Hutchinson et al., 2021; Mohseni 2019; Raji et al., 2020).”
Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 103
Deng, W. H., Guo, B., Devrio, A., Shen, H., Eslami, M., & Holstein, K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581026 · Cited by 103
“Yet achieving this goal requires that companies are actually held accountable for addressing the issues that user auditors uncover [16, 39, 68, 78].”
BackgroundGonzález-Gonzalo, C., Thee, E. F., Klaver, C. C. W., Lee, A. Y., Schlingemann, R. O., Tufail, A., Verbraak, F., & Sánchez, C. I. (2022) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2021.101034 · Cited by 103
“Raji et al. (2020) have recently C. Gonz ´ alez-Gonzalo et al. proposed a framework for end-to-end internal algorithmic auditing.”
AI Ethics Principles in Practice: Perspectives of Designers and Developers
Sanderson, C., Douglas, D., Lu, Q., Schleiger, E., Whittle, J., Lacey, J., Newnham, G., Hajkowicz, S., Robinson, C., & Hansen, D. (2023) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2023.3257303 · Cited by 99
“As such, including recommendations and suggestions from relevant literature [41], [53], [60], [62], [63] may yield a richer picture.”
BackgroundUnderstanding accountability in algorithmic supply chains
Cobbe, J., Veale, M., & Singh, J. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594073 · Cited by 115
“Addressing the ‘many hands’ problem is increasingly a concern in the algorithmic accountability literature (implicitly or explicitly), with proposals in recent years for accountability for algorithmic systems to operate at an organisational level [25, 40, 60, 62, 67, 75, 89, 98, 110].”
BackgroundAI ethics education: A systematic literature review
Wiese, L. J., Patil, I., Schiff, D. S., & Magana, A. J. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100405 · Cited by 251
”Because AI is 100% right and safe”: User Attitudes and Sources of AI Authority in India
Kapania, S., Siy, O., Clapper, G., SP, A. M., & Sambasivan, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517533 · Cited by 90
A survey on interpretable reinforcement learning
Glanois, C., Weng, P., Zimmer, M., Li, D., Yang, T., Hao, J., & Liu, W. (2024) · Machine Learning · doi:10.1007/s10994-024-06543-w · Cited by 243
“We encourage the curious reader to look further at the gener-ous work of other researchers investigating these tangent questions (such as Crawford et al. (2016), Raji et al. (2020), Daly et al. (2019), Yu et al. (2018), Commission (2019) to mention only a few).”
Krafft, T. D., Zweig, K. A., & König, P. D. (2022) · Regulation & Governance · doi:10.1111/rego.12369 · Cited by 95
Burr, C., & Leslie, D. (2021) · AI and Ethics · doi:10.1007/s43681-022-00178-0 · Cited by 147
Minkkinen, M., Laine, J., & Mäntymäki, M. (2022) · Digital Society · doi:10.1007/s44206-022-00022-2 · Cited by 88
“Because CAAI requires continuous access to AI systems, it appears most relevant to internal audit functions within organizations (cf. Raji et al., 2020b; Tronto et al., 2021) as opposed to external auditing conducted by independent auditors.”
BackgroundFairLens: Auditing black-box clinical decision support systems
Panigutti, C., Perotti, A., Panisson, A., Bajardi, P., & Pedreschi, D. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102657 · Cited by 86
Edge Intelligence—Research Opportunities for Distributed Computing Continuum Systems
Pujol, V. C., Donta, P. K., Morichetta, A., Murturi, I., & Dustdar, S. (2023) · IEEE Internet Computing · doi:10.1109/mic.2023.3284693 · Cited by 110
An Action-Oriented AI Policy Toolkit for Technology Audits by Community Advocates and Activists
Krafft, P. M., Young, M., Katell, M., Lee, J. E., Narayan, S., Epstein, M., Dailey, D., Herman, B., Tam, A., Guetler, V., et al. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445938 · Cited by 83
“Raji and colleagues explicate a far-ranging organizational process for achieving algorithmic accountability through internal auditing [45].”
BackgroundLiao, Q. V., Subramonyam, H., Wang, J., & Wortman Vaughan, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580652 · Cited by 86
“Recent years have seen a growing interest in studying practitioners’ practices, challenges, and gaps in dealing with RAI issues, such as fairness [35, 55, 74, 75, 82], transparency [18, 57, 66], and accountability [21, 81].”
BackgroundDocumenting Computer Vision Datasets: An Invitation to Reflexive Data Practices
Miceli, M., Yang, T., Naudts, L., Schuessler, M., Serbanescu, D., & Hanna, A. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445880 · Cited by 82
“ment for potential fairness concerns prior to the release of a model trained on such data [42, 43].”
BackgroundResponsible AI in Africa: Challenges and Opportunities
(2023) · Social and Cultural Studies of Robots and AI · doi:10.1007/978-3-031-08215-3 · Cited by 82
Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
Ojewale, V., Steed, R., Vecchione, B., Birhane, A., & Raji, I. D. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713301 · Cited by 81
Self-citationHudley, A. H. C., Mallinson, C., & Bucholtz, M. (2024) · doi:10.1093/oso/9780197755303.001.0001 · Cited by 91
Metadata
- Authors
- Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P.
- Affiliations
- Partnership on AI
- Google (United States) (US)
- ORCID
- Venue
- Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2020-01-27
- Pages
- 33-44
- OpenAlex
- W3001807593
- Cited by
- 2,245
- References
- 83
- Citations by year
- 2026: 1097 · 2025: 454 · 2024: 243 · 2023: 177 · 2022: 149 · 2021: 99 · 2020: 24
- FWCI
- 35.91
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- AI accountability, algorithmic auditing, harm detection, algorithmic transparency, societal impact of AI, algorithmic governance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.