Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing
Open accessThe DOI matches its Crossref record; 96 of the 97 references it cites were found in a registry or an index, and 1 are not checked yet; 11 of the 44 sources its claims cite were checked for support.
Abstract
Rising concern for the societal implications of artificial intelligence systems has inspired a wave of academic and journalistic literature in which deployed systems are audited for harm by investigators from outside the organizations deploying the algorithms. However, it remains challenging for practitioners to identify the harmful repercussions of their own systems prior to deployment, and, once deployed, emergent issues can become difficult or impossible to trace back to their source.
Verified 2026-11-19 · 1 reference not checked yet
Overview
CoChat Summary
TL;DR
The paper proposes SMACTR, an end-to-end internal algorithmic auditing framework intended to help organizations assess AI systems against declared ethical principles throughout development. It argues that internal audits can complement external accountability by providing pre-deployment access, structured documentation, and risk analysis drawn from aerospace, medical device, and finance practices.
Key findings
- Introduces SMACTR—Scoping, Mapping, Artifact Collection, Testing, and Reflection—as five audit stages, each with documentation requirements that together form an overall audit report and assess decisions against organizational values or AI principles.
- Argues external audits are limited by lack of access to internal processes, intermediate models, and training data, while internal audits can enable proactive pre-deployment intervention and generate artifacts for external review.
- Draws lessons from other industries, citing aerospace’s one commercial airline accident per two million flights, Boeing 787 software estimated at 13 million lines of code, and a 1-in-1,000,000,000 per use hour maximum failure probability for critical aerospace systems; also cites medical-device design controls and finance internal audit infrastructure.
- Uses hypothetical cases—a child abuse screening tool and a smile-detection photo booth—to illustrate the framework; for the smile detector’s CelebA datasheet, it reports 58.1% female and 42% male labels, 77.8% aged 0–45 and 22.1% over 46, and 14.2% lighter-skinned versus 85.8% darker-skinned.
Methods
- Conceptual framework paper: develops SMACTR from auditing practices and artifacts in aerospace, medical devices, and finance, plus responsible-innovation dimensions of anticipation, reflexivity, inclusion, and responsiveness.
- Uses a hypothetical Company X Inc. consultancy with two invented client projects—a child abuse screening tool and Happy-Go-Lucky Inc.’s smile-detection photo booth—to demonstrate audit stages and artifacts; a worked example for the smile case is provided as supplementary material.
- Analysis is narrative and design-oriented, referencing existing standards and tools such as FMEA, model cards, datasheets, design controls, and design history files; no empirical sample or quantitative evaluation is reported.
Limitations
- The paper states that internal auditors necessarily share an organizational interest with the audit target, making an independent and objective viewpoint challenging; audits are not an objective “view from nowhere” and risk becoming reputation management.
- The paper states that internal audits are only one aspect of a broader system of required quality checks and balances, and that determining what systems to audit and why is not covered and is a separate research topic.
- Evident from the text: the framework is not empirically validated, the illustrative cases are hypothetical, and no real deployment outcomes or measured audit effectiveness are reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
9 Valid · 6 Not valid · 27 Not checked · 30 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“While not without faults, safety-critical and regulated industries such as aerospace and medicine have long traditions of auditable processes and design controls that have dramatically improved safety [77,81].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 3 LESSONS FROM AUDITING PRACTICES IN OTHER INDUSTRIES
What the sources say
- Not checked — no full text of the sourcecites [77] matched to “Design controls for the medical device industry” (2013) source
- Could not verifycites [81] matched to “The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions” (2019) sourceExperimental — accuracy not yet measured
The Introduction and §3 discuss biomedical ethics and standards or regulation, but do not say that aerospace or medicine has long used auditable processes and design controls, or that these dramatically improved safety. Searches for aerospace, auditable processes, design controls, and safety-critical industries found no passage addressing that claim. Page numbers are unavailable in our copy. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This is a repeatedly observed phenomenon in tax compliance auditing, where several international surveys of tax compliance demonstrate that a fixed and vetted tax audit methodology is one of the most effective strategies to convince companies to respect audit results and pay their full taxes [22,53].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.3 Audit Integrity and Procedural Justice
What the sources say
- Partly supportedcites [22] matched to “Perception on justice, trust and tax compliance behavior in Malaysia” (2017) sourceExperimental — accuracy not yet measured
The Regression Test section reports that Malaysian respondents would be more compliant if tax authorities treated them fairly in tax procedures. The Research Methodology section describes a questionnaire of academics, not several international surveys of companies. The paper does not establish that a fixed, vetted audit methodology is especially effective at securing acceptance of audit results or full tax payment. (LLM, conf=0.86)
- Partly supportedcites [53]Experimental — accuracy not yet measured
The “Justice in the taxation context” section reports that taxpayers are generally more compliant when treated fairly and respectfully, and the General discussion links perceived unfair treatment to resistance to paying back tax (page numbers unavailable). It does not establish that a fixed, vetted audit methodology is among the most effective strategies, or report international surveys demonstrating that effect for companies. (LLM, conf=0.83)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Traditional metrics for artificial intelligence like loss may conceal fairness concerns, social impact risks or abstraction errors [69].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.3.2 Artifact: Ethnographic Field Study.
What the sources say
- Supportedcites [69] matched to “Fairness and Abstraction in Sociotechnical Systems” (2019) source
The Introduction (page not provided) says that narrow assessments of machine-learning performance abstract away social context and can miss information needed for fair outcomes—an abstraction error. The Formalism Trap section adds that mathematical measures cannot capture the full range of fairness concerns, while the Takeaways discuss unintended social consequences. The paper names AUC, precision and recall rather than loss, but the claim uses loss as an illustrative metric. (LLM, conf=0.88)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Internal auditors' direct access to systems can thus help extend traditional external auditing paradigms by incorporating additional information typically unavailable for external evaluations to reveal previously unidentifiable risks.”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 2.4 The Internal Audit
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The essential inclusion of independent domain experts and marginalized groups in the ethical review process “has the potential to lead to more rigorous critical reflection because their experiences will often be precisely those that are most needed in identifying problematic background assumptions and revealing limitations with research questions, models, or methodologies” [33].”
§ Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing > 4.2.1 Artifact: Ethical Review of System Use Case.
What the sources say
- Not checked — no full text of the sourcecites [33] matched to “25 Years of Feminist Empiricism and Standpoint Theory: Where Are We Now?” (2010) source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
19 more claims
Create a free account to see all 72 claims and every source verdict.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 37
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This disparity poses risks, as practitioners may unintentionally embed their own biases into AI system design and implementation (Martin 2019;Raji et al. 2020).”
From Accountability in Government Use of AI : Citizen Concerns and Preferences (2025) · cites this paper as Raji et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Work on AI safety and reproducibility has long pointed out the crucial importance of training data for understanding model performance, ensuring reproducibility, and assessing legal exposure [2, 25, 29, 46, 58].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While there is a growing interest in auditing algorithmic systems [125,126,128,138,154], this feld generally lacks defned standards and records could be incomplete or not relating to information needed for accountability purposes [15,127].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [128]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Raji et al. (2020) proposed an internal algorithmic auditing framework spanning data collection, model development, testing, deployment, and monitoring, with stage-specific artifacts that enable traceability and accountability.”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Raji et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Practical approaches to implement such ideas typically revolve around the systematic collection and curation of data on various aspects of systems [2, 28, 46].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [46]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747 · cited 1×
“A hypothetical datasheet for this system is filled out using studies on the CelebA dataset, with which the smile detector is built [44, 47].”
MethodThe global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Product development: Past research, present findings and future directions
Brown, S. L., & Eisenhardt, K. M. (1995) · The Academy of Management Review · doi:10.2307/258850 · Cited by 3,777 · cited 1×
“There are also intersections between this framework and just efective product development theory [5], as many of the components of audit design refocus the product development process to prioritize the user and their ultimate well-being, resulting in a more efective product performance outcome.”
Background(2018) · Cited by 3,510
Christiane, Druml, Wolzt, & Pleiner (2001) · Acta Scientiarum Naturalium Universitatis Sunyatseni · Cited by 3,419
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196
Developing a framework for responsible innovation
Stilgoe, J., Owen, R., & Macnaghten, P. (2013) · Research Policy · doi:10.1016/j.respol.2013.05.008 · Cited by 3,184 · cited 1×
“To design our audit procedure, we suggest complementing formal risk assessment methodologies with ideas from responsible innovation, which stresses four key dimensions: anticipation, reflexivity, inclusion and responsiveness [73], as well as system-theoretic concepts that help grapple with increasing complexity and co…”
MethodAlgorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Self-citation(2014) · Journal of the Korean Medical Association · doi:10.5124/jkma.2014.57.11.899 · Cited by 2,566
Research works printed but not linked: 7
- Virginia Eubanks. 2018. A child abuse prediction model fails poor families. Wired Magazine (2018).
- Jeremy Goldhaber-Fiebert and Lea Prince. 2019. Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny CountyâĂŹs Child Welfare Office. Pittsburgh: Allegheny County.[Google Scholar] (2019).
- Nitin Kohli, Renata Barreto, and Joshua A Kroll. 2018. Translation Tutorial: A Shared Lexicon for Research and Practice in Human-Centered Software Systems. In 1st Conference on Fairness, Accountability, and Transparancy. New York, NY, USA. 7.
- Joshua A Kroll, Solon Barocas, Edward W Felten, Joel R Reidenberg, David G Robinson, and Harlan Yu. 2016. Accountable algorithms. U. Pa. L. Rev. 165 (2016), 633.
- Institute of Internal Auditors. Research Foundation and Institute of Internal Au- ditors. 2007. The Professional Practices Framework. Inst of Internal Auditors.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22 (2014).
- Tim Vanderveen. 2005. Averting highest-risk errors is first priority. Patient Safety and Quality Healthcare 2 (2005), 16-21.
Cited by
Items 101–200 of 1,412
Hudley, A. H. C., Mallinson, C., & Bucholtz, M. (2024) · doi:10.1093/oso/9780197755303.001.0001 · Cited by 91
Responsible AI in Africa—Challenges and Opportunities
Okolo, C. T., Aruleba, K., & Obaido, G. (2023) · Social and Cultural Studies of Robots and AI · doi:10.1007/978-3-031-08215-3_3 · Cited by 79
Supporting Human-AI Collaboration in Auditing LLMs with LLMs
Rastogi, C., Ribeiro, M. T., King, N., Nori, H., & Amershi, S. (2023) · arXiv · doi:10.48550/arxiv.2304.09991 · Cited by 79
The Gradient of Generative AI Release: Methods and Considerations
Solaiman, I. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593981 · Cited by 90
“Auditing must be considered both pre- and post-deployment as impacts from a system may not be detectable pre-deployment and when deployed, impacts may be difficult to trace back to a specific system [74].”
BackgroundLam, M. S., Gordon, M. L., Metaxa, D., Hancock, J. T., Landay, J. A., & Bernstein, M. S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555625 · Cited by 77
“If system developers host their own end-user audits, users could directly share their reports with the developers in a closed setting; this would bear similarities to a first- or second-party audit [59, 72].”
BackgroundSupporting Human-AI Collaboration in Auditing LLMs with LLMs
Rastogi, C., Tulio Ribeiro, M., King, N., Nori, H., & Amershi, S. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604712 · Cited by 78
“Raji et al. (2020) introduce a framework for algorithm auditing to be applied throughout the algorithm’s internal development lifecycle.”
MethodBlack-Box Access is Insufficient for Rigorous AI Audits
Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659037 · Cited by 86
“Without careful institutional design, the incentives of developers and auditors may result in audits that do not consistently align with public interest [11, 25, 67, 218, 251, 252].”
BackgroundThylstrup, N. B. (2022) · Media Culture & Society · doi:10.1177/01634437211060226 · Cited by 98
“The notion of traceability often used in AI ethics discourses derives from aerospace and safety-critical software engineering, where it signifies the ability to trace each component back along a supply chain (Raji et al., 2020) and development life cycle.”
BackgroundYurrita, M., Draws, T., Balayn, A., Murray-Rust, D., Tintarev, N., & Bozzon, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581161 · Cited by 108
“However, moving towards algorithmic decision-making processes that enhance decision subjects’ feelings of justice requires that empirical studies are part of broader eforts to create methodological tools that consider diferent stakeholders’ (including decision subjects) viewpoints in the design and evaluation processe…”
BackgroundDesigning Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M. R., Vaughan, J. W., Wadsworth, W. D., & Wallach, H. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462610 · Cited by 72
“[69], which proposes the “Scoping, Mapping, Artifact Collection, Testing, and Reflection” (SMACTR) framework for conducting comprehensive internal audits of AI systems via a rigorous, multi-stage process.”
BackgroundZhang, A., Walker, O., Nguyen, K., Dai, J., Chen, A., & Lee, M. K. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579601 · Cited by 72
Dixon, R. B. L. (2022) · AI and Ethics · doi:10.1007/s43681-022-00205-0 · Cited by 106
“[9] proposed situating AI principles as the standard for evaluating the development of the AI life cycle and in internal audits when formalized guidelines are not available.”
BackgroundTowards artificial intelligence at scale in the chemical industry
Chiang, L. H., Braun, B., Wang, Z., & Castillo, I. (2022) · AIChE Journal · doi:10.1002/aic.17644 · Cited by 91
Rethinking open source generative AI: open washing and the EU AI Act
Liesenfeld, A., & Dingemanse, M. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659005 · Cited by 123
AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries
Cihon, P., Kleinaltenkamp, M. J., Schuett, J., & Baum, S. D. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3077595 · Cited by 66
“These ethics principles are used as a “North Star” guide in today’s certification programs ([36] p.35) and will likely remain relevant as the technology matures.”
Improving accountability in recommender systems research through reproducibility
Bellogín, A., & Said, A. (2021) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-021-09302-x · Cited by 84
“However, there is no consensus on whether these agents should be held accountable in a human-like manner or whether humans should always be the only actors who are ultimately responsible for technological artifacts (Jobin et al. 2019; Raji et al. 2020).”
BackgroundLet Me Take Over: Variable Autonomy for Meaningful Human Control
Methnani, L., Aler Tubella, A., Dignum, V., & Theodorou, A. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.737072 · Cited by 66
“, current moral and legal frameworks fail to explicitly answer who should be held responsible for the actions taken by an autonomous system and how (Raji et al., 2020; Santoni de Sio and Mecacci, 2021).”
BackgroundMethodInfluentialBuilding Trustworthy AI Solutions: A Case for Practical Solutions for Small Businesses
Crockett, K. A., Colyer, E., Gerber, L., & Latham, A. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3137091 · Cited by 96
Healthsheet: Development of a Transparency Artifact for Health Datasets
Rostamzadeh, N., Mincu, D., Roy, S., Smart, A., Wilcox, L., Pushkarna, M., Schrouff, J., Amironesei, R., Moorosi, N., & Heller, K. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533239 · Cited by 72
“As a step towards more accountable ML practices[28], several guidelines and frameworks have been proposed around data transparency, collection and use in ML [9, 10, 22, 23, 49, 56].”
BackgroundSelf-citationConfronting Power and Corporate Capture at the FAccT Conference
Young, M., Katell, M., & Krafft, P. M. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533194 · Cited by 63
Ethical Challenges in the Development of Virtual Assistants Powered by Large Language Models
Piñeiro-Martín, A., García-Mateo, C., Docío-Fernández, L., & López-Pérez, M. D. C. (2023) · Electronics · doi:10.3390/electronics12143170 · Cited by 64
“…(since there is no collective awareness of the real possibilities or the risks that its use could entail) and, as mentioned in [32], “the industry lacks useful tools and incentives to translate high-level ethics principles to verifiable and actionable criteria for designing and deploying AI” [33].”
BackgroundFederatedTrust: A solution for trustworthy federated learning
Sánchez, P. M. S., Celdrán, A. H., Xie, N., Bovet, G., Pérez, G. M., & Stiller, B. (2023) · Future Generation Computer Systems · doi:10.1016/j.future.2023.10.013 · Cited by 103
Holding AI to Account: Challenges for the Delivery of Trustworthy AI in Healthcare
Procter, R., Tolmie, P., & Rouncefield, M. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3577009 · Cited by 92
“2022) and frameworks for auditing compliance (Raji et al. 2020) and we aim to investigate how these are defined and are then embedded into organisational practice.”
Mökander, J., & Axente, M. (2023) · AI & SOCIETY · doi:10.1007/s00146-021-01286-x · Cited by 62
Kim, J. Y., Hasan, A., Kellogg, K. C., Ratliff, W., Murray, S. G., Suresh, H., Valladares, A., Shaw, K., Tobey, D., Vidal, D. E., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000390 · Cited by 61
Self-citationEvaluation Gaps in Machine Learning Practice
Hutchinson, B., Rostamzadeh, N., Greer, C., Heller, K., & Prabhakaran, V. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533233 · Cited by 62
“…a top-down examination of the context and goal of the evaluation before the evaluation design even begins, and their call for careful documentation of the evaluation “remit”—i.e., official responsibilities—is in line with more recent work calling for stakeholder transparency for ML [82, 136].”
BackgroundResultInfluentialSelf-citationGurjar, K., Jangra, A., Baber, H., Islam, M., & Sheikh, S. A. (2024) · IEEE Engineering Management Review · doi:10.1109/emr.2024.3355973 · Cited by 103
“To address these concerns, businesses should prioritize transparency,fairness,and accountabilityintheirAIsystems, andworktodevelopAIsystems thatarealignedwithethicaland socialvalues[136],[137].”
BackgroundSystematizing Audit in Algorithmic Recruitment
Kazim, E., Koshiyama, A. S., Hilliard, A., & Polle, R. (2021) · Journal of Intelligence · doi:10.3390/jintelligence9030046 · Cited by 60
Kawakami, A., Chowdhary, S., Iqbal, S. T., Liao, Q. V., Olteanu, A., Suh, J., & Saha, K. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610207 · Cited by 100
“These include, taxonomy of AI failures [147], guidelines for designing human-AI interaction technologies [6], taxonomy of ethical tensions between AI and practitioners [35], taxonomies of roles of AI consumers [8, 175], question-bank of explainable AI [108], and guidelines for minimizing risks with automated emotion r…”
BackgroundHartmann, D., de Pereira, J. R. L., Streitbörger, C., & Berendt, B. (2025) · AI and Ethics · doi:10.1007/s43681-024-00595-3 · Cited by 58
Community-in-the-loop: towards pluralistic value creation in AI, or—why AI needs business ethics
Häußermann, J. J., & Lütge, C. (2022) · AI and Ethics · doi:10.1007/s43681-021-00047-2 · Cited by 57
Addressing bias in artificial intelligence for public health surveillance
Flores, L., Kim, S., & Young, S. D. (2023) · Journal of Medical Ethics · doi:10.1136/jme-2022-108875 · Cited by 75
“53 Not only should developers of algorithms test and audit each component within the AI lifecycle, but a collective set of standards and fairness checks to reduce bias are also needed among corporations, institutions and government agencies.”
BackgroundFarsight: Fostering Responsible AI Awareness During AI Application Prototyping
Wang, Z. J., Kulkarni, C., Wilcox, L., Terry, M., & Madaio, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642335 · Cited by 55
Requirements for Trustworthy Artificial Intelligence – A Review
Kaur, D., Uslu, S., & Durresi, A. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-57811-4_11 · Cited by 85
“[23] proposed an internal auditing framework for algorithmic auditing to keep check on the development life cycle of AI systems.”
BackgroundAI Documentation: A path to accountability
Königstorfer, F., & Thalmann, S. (2022) · Journal of Responsible Technology · doi:10.1016/j.jrt.2022.100043 · Cited by 55
Maiti, M., Kayal, P., & Vujko, A. (2025) · Future Business Journal · doi:10.1186/s43093-025-00462-5 · Cited by 128
Making It Possible for the Auditing of AI: A Systematic Review of AI Audits and AI Auditability
Li, Y., & Goel, S. (2024) · Information Systems Frontiers · doi:10.1007/s10796-024-10508-8 · Cited by 105
“Given the criticality of identifying risks for harmful consequences prior to deployment (Raji et al., 2020), there is a need for collaboration between internal and external auditors.”
InfluentialProbing Pre-Trained Language Models for Cross-Cultural Differences in Values
Arora, A., Kaffee, L.-A., & Augenstein, I. (2023) · Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP) · doi:10.18653/v1/2023.c3nlp-1.12 · Cited by 57
“Thus, when working with cultural values, an auditing system (Raji et al., 2020) with these value systems in mind and one that takes into account the downstream use case should be employed.”
Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studies
Vecchione, B., Levy, K., & Barocas, S. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483294 · Cited by 51
Kim, J., Lee, J., Jang, K. M., & Lourentzou, I. (2023) · Telematics and Informatics · doi:10.1016/j.tele.2023.102085 · Cited by 63
Corvite, S., Roemmich, K., Rosenberg, T. I., & Andalibi, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579600 · Cited by 97
“Ethical and responsible EAI must square with these harmful and unjust implications [136, 144], such as by exploring how data subjects’ privacy may be protected and preserved with any EAI technologies in the workplace.”
BackgroundReed, R. (2021) · Religions · doi:10.3390/rel12060401 · Cited by 80
Certification Labels for Trustworthy AI: Insights From an Empirical Mixed-Method Study
Scharowski, N., Benk, M., Kühne, S. J., Wettstein, L., & Brühlmann, F. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593994 · Cited by 75
“However, whether auditing should be conducted by independent third-parties or internally within organizations is a topic of ongoing academic discussion [17, 38, 54], with both approaches having their advantages and drawbacks.”
BackgroundSociotechnical Safety Evaluation of Generative AI Systems
Weidinger, L., Rauh, M., Marchal, N., Manzini, A., Hendricks, L. A., Mateos-Garcia, J., Bergman, S., Kay, J., Griffin, C., Bariach, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.11986 · Cited by 49
A Framework for Assurance Audits of Algorithmic Systems
Lam, K., Lange, B., Blili-Hamelin, B., Davidovic, J., Brown, S., & Hasan, A. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658957 · Cited by 75
“This process can act as a complement to the work of an internal audit team that evaluates the algorithmic system along its development lifecycle, 13 and produces critical documents and artifacts—such as model or ethical risk analysis results, system cards, and technical testing reports [56].”
BackgroundMachine Learning for Health: Algorithm Auditing & Quality Control
Oala, L., Murchison, A. G., Balachandran, P., Choudhary, S., Fehr, J., Leite, A. W., Goldschmidt, P. G., Johner, C., Schörverth, E. D. M., Nakasi, R., et al. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01783-y · Cited by 48
They shall be fair, transparent, and robust: auditing learning analytics systems
Simbeck, K. (2023) · AI and Ethics · doi:10.1007/s43681-023-00292-7 · Cited by 83
“Often, edge cases or corner cases are used that describe scenarios with low probabilities but possibly severe consequences [14, 40, 83].”
InfluentialHorneber, D., & Laumer, S. (2023) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00817-8 · Cited by 127
“Algorithm audits can be either focused on technical aspects such as reliability and robustness or on problematic behaviors and the societal consequences of ML systems (Raji et al. 2020).”
BackgroundHagendorff, T. (2021) · Minds and Machines · doi:10.1007/s11023-021-09573-8 · Cited by 47
Ramesh, D., Kameswaran, V., Wang, D., & Sambasivan, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533237 · Cited by 47
“, to help the users make informed choices when interacting with ADS) [10, 21, 42, 83, 100, 119]; or be embedded in other mechanisms such as algorithmic audits [26, 46, 112] and impact assessments [101, 102].”
BackgroundThe emergence of artificial intelligence ethics auditing
Schiff, D. S., Kelley, S., & Ibáñez, J. C. (2024) · Big Data & Society · doi:10.1177/20539517241299732 · Cited by 130
“…planning (scoping and mapping), performing (artifact collection and testing), and reporting (re fl ection and post-audit stage) (Porter et al., 2014: 8; Raji et al., 2020), an association we visually map in Figure 3 with individual activities listed as part of an updated account and framework of…”
InfluentialToward accountable human-centered AI: rationale and promising directions
Qadir, J., Islam, M. Q., & Al-Fuqaha, A. (2022) · Journal of Information, Communication and Ethics in Society · doi:10.1108/jices-06-2021-0059 · Cited by 46
Algorithmic bias, fairness, and inclusivity: a multilevel framework for justice-oriented AI
Panarese, P., Grasso, M. M., & Solinas, C. (2025) · AI & Society · doi:10.1007/s00146-025-02451-2 · Cited by 107
AI-deploying organizations are key to addressing ‘perfect storm’ of AI risks
Curtis, C., Gillespie, N., & Lockey, S. (2023) · AI and Ethics · doi:10.1007/s43681-022-00163-7 · Cited by 43
“Deeper understanding of how these organizations can translate the principles of trustworthy AI into practice is needed, including methods for the early detection of errors and a focus on contestability and accountability [54].”
Raji, I. D., & Buolamwini, J. (2023) · Communications of the ACM · doi:10.1145/3571151 · Cited by 45
Self-citationAI in the Public Eye: Investigating Public AI Literacy Through AI Art
Hemment, D., Currie, M., Bennett, S., Elwes, J., Ridler, A., Sinders, C., Vidmar, M., Hill, R., & Warner, H. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594052 · Cited by 44
“Racial justice and design justice approaches addressing the area of FAccT (fairness, accountability and transparency) in AI call for involving the expertise of affected and minority communities in efforts around algorithmic transparency and governance [43][51].”
BackgroundHutchinson, B., Smart, A., Hanna, A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13561 · Cited by 44
“or opportunistically. From an observer’s perspective, these practices replace abductive and a posteriori reasoning about dataset provenance with careful documentation, facilitating reviews and audits [114]. They help us to avoid the trap of being blindly “data-driven” by poorly-understood datasets— mistaking taking a…”
BackgroundInfluentialSelf-citationThe Model Card Authoring Toolkit: Toward Community-centered, Deliberation-driven AI Design
Shen, H., Wang, L., Deng, W. H., Brusse, C., Velgersdijk, R., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533110 · Cited by 43
“They have developed methods and tools for informing and engaging users in model design, development and deployment [47, 51] as well as for them to analyze and contest the outcomes of ML systems [42].”
MethodA principles-based ethics assurance argument pattern for AI and autonomous systems
Porter, Z., Habli, I., McDermid, J. A., & Kaas, M. H. L. (2023) · AI and Ethics · doi:10.1007/s43681-023-00297-2 · Cited by 72
Hauptman, A. I., Schelble, B. G., Duan, W., Flathmann, C., & McNeese, N. J. (2024) · Cognition Technology & Work · doi:10.1007/s10111-024-00765-7 · Cited by 67
“These explanations are key to providing a degree of accountability over the AI (Raji et al. 2020).”
Seamful XAI: Operationalizing Seamful Design in Explainable AI
Ehsan, U., Liao, Q. V., Passi, S., Riedl, M. O., & Daumé, H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637396 · Cited by 105
“However, recent work studying how practitioners deal with RAI issues on the ground, including AI fairness [15, 35, 54, 65], transparency [31, 37, 48], accountability [70], and overall harms mitigation [4, 71], report that practitioners grapple with tremendous challenges.”
BackgroundBoretti, A. A. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00171-y · Cited by 63
AI Regulation Is (not) All You Need
Lucaj, L., van der Smagt, P., & Benbouzid, D. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594079 · Cited by 40
“This process allows the developers to detect the potential unintended consequences before the deployment [63].”
InfluentialEhsan, U., Singh, R., Metcalf, J., & Riedl, M. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533186 · Cited by 40
Considerations in the reliability and fairness audits of predictive models for advance care planning
Lu, J., Sattler, A., Wang, S., Khaki, A. R., Callahan, A., Fleming, S., Fong, R., Ehlert, B., Li, R. C., Shieh, L., et al. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.943768 · Cited by 40
Do, K., Pang, R. Y., Jiang, J., & Reinecke, K. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581347 · Cited by 62
Berman, G., Goyal, N., & Madaio, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642398 · Cited by 40
Towards a multi-stakeholder value-based assessment framework for algorithmic systems
Yurrita, M., Murray-Rust, D., Balayn, A., & Bozzon, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533118 · Cited by 39
“Since there are value trade-offs, like the one outlined in our example use case, and certain value-specific criteria are mutually exclusive, we follow the claim made by Raji and Smart [147], and advocate for standpoint diversity.”
InfluentialHopkins, A., & Booth, S. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462527 · Cited by 40
Three lines of defense against risks from AI
Schuett, J. (2025) · AI & SOCIETY · doi:10.1007/s00146-023-01811-0 · Cited by 41
“It does not merely mean an audit that is done internally (Raji et al 2020).”
Sociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Advertising
Lam, M. S., Pandit, A., Kalicki, C. H., Gupta, R., Sahoo, P., & Metaxa, D. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610209 · Cited by 39
“1Some recent work has discussed methods to facilitate effective internal audits [58], but this strategy is not always possible.”
BackgroundEthical Considerations in AI Development
Singh, B. P., & Joshi, A. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-2964-1.ch010 · Cited by 47
Ugwudike, P. (2022) · AI and Ethics · doi:10.1007/s43681-021-00117-5 · Cited by 39
“The aim is to anticipate potentially harmful outcomes including adverse social impact [16] such as negative discrimination [17].”
BackgroundYampolskiy, R. V. (2024) · AI and Ethics · doi:10.1007/s43681-024-00420-x · Cited by 80
“Inspection and audit mechanisms: Establishing robust inspection and audit mechanisms [40-42], both on-site and remote, is essential for verifying AI treaty compliance.”
Beyond External Constraints: The Missing Dimension of AI GovernancePreprint
CHEN, L. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6449738 · Cited by 121
Putting AI Ethics into Practice: The Hourglass Model of Organizational AI Governance
Mäntymäki, M., Minkkinen, M., Birkstedt, T., & Viljanen, M. (2022) · arXiv · doi:10.48550/arxiv.2206.00335 · Cited by 38
“The life cycle perspective is essential because AI systems are IT systems that need to be governed during their entire life cycles, from conception to operation and monitoring (Ibáñez & Olmeda, 2021; Laato, Birkstedt, et al., 2022; Laato, Mäntymäki, Minkkinen, et al., 2022; Raji et al., 2020).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38
Freiesleben, T., König, G., Molnar, C., & Tejero-Cantero, Á. (2024) · Minds and Machines · doi:10.1007/s11023-024-09691-z · Cited by 82
“The auditor examines model properties e.g. for debugging, to check if the model satisfies legal or ethical norms, or to improve understanding of the model by intervening on it (Raji et al. 2020).”
From human-centered to symbiotic artificial intelligence: a focus on medical applications
Desolda, G., Esposito, A., Lanzilotti, R., Piccinno, A., & Costabile, M. F. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20414-5 · Cited by 69
Many hands make many fingers to point: challenges in creating accountable AI
Slota, S. C., Fleischmann, K. R., Greenberg, S. R., Verma, N., Cummings, B., Li, L., & Shenefiel, C. (2021) · AI & Society · doi:10.1007/s00146-021-01302-0 · Cited by 65
“AI systems are often characterized by an inequitable risk distribution, where those who are already vulnerable disproportionately experience their harms (Raji et al. 2020).”
BackgroundThe AI ethics maturity model: a holistic approach to advancing ethical data science in organizations
Krijger, J., Thuis, T., de Ruiter, M., Ligthart, E., & Broekman, I. (2022) · AI and Ethics · doi:10.1007/s43681-022-00228-7 · Cited by 61
(2022) · doi:10.1145/3491102.3501868 · Cited by 36
“[39, 79]), and we extend this focus to the development of ML datasets.”
BackgroundBorsci, S., Lehtola, V., Nex, F., Yang, M. Y., Augustijn, E.-W., Bagheriye, L., Brüne, C., Kounadi, O., Li, J. J., Moreira, J., et al. (2022) · AI & Society · doi:10.1007/s00146-021-01383-x · Cited by 50
“Concurrently, the wide usage of AI-powered by large-scale data and DL algorithms in areas of high societal impact raises a lot of concerns regarding fairness, accountability, and transparency of decision-making (Raji et al. 2020).”
BackgroundSloane, M., & Zakrzewski, J. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533156 · Cited by 59
Towards intellectual freedom in an AI Ethics Global Community
Ebell, C., Baeza-Yates, R., Benjamins, R., Cai, H., Coeckelbergh, M., Duarte, T., Hickok, M., Jacquet, A., Kim, A., Krijger, J., et al. (2021) · AI and Ethics · doi:10.1007/s43681-021-00052-5 · Cited by 34
Milošević, T., Royen, K. V., & Davis, B. (2022) · International Journal of Bullying Prevention · doi:10.1007/s42380-022-00117-x · Cited by 57
“…measures; how it can inform social scientific work but also to emphasise its limitations (sometimes unintended and unexamined), flaws in design and how its implementation can further existing social inequalities, which is a widely researched and acknowledged issue (Gebru, 2019; Raji et al., 2020).”
BackgroundArtificial Intelligence and Bias Towards Marginalised Groups: Theoretical Roots and Challenges
Mergen, A., Çetin-Kiliç, N., & Özbilgin, M. F. (2025) · International perspectives on equality, diversity and inclusion · doi:10.1108/s2051-233320250000012004 · Cited by 58
Cajueiro, D. O., & Celestino, V. R. R. (2025) · Journal of Economy and Technology · doi:10.1016/j.ject.2025.07.001 · Cited by 69
A Moral Political Economy: Present, Past, and Future
Carugati, F., & Levi, M. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108872942 · Cited by 43
Tech Worker Organizing for Power and Accountability
Boag, W., Suresh, H., Lepe, B., & D'Ignazio, C. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533111 · Cited by 34
“[56], co-designing checklists for AI fairness with practitioners [43], and a series of questions for self-assessment of whether AI products are respecting human rights [65].”
BackgroundRismani, S., Shelby, R., Smart, A., Jatho, E., Kroll, J., Moon, A., & Rostamzadeh, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581407 · Cited by 33
Self-citationAdam, H., Yang, M. Y., Cato, K., Baldini, I., Senteio, C., Celi, L. A., Zeng, J., Singh, M., & Ghassemi, M. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534203 · Cited by 33
“thoroughly and continuously audited for racial bias both before and after deployment [21, 47, 59].”
BackgroundAn external stability audit framework to test the validity of personality prediction in AI hiring
Rhea, A. K., Markey, K., D’Arinzo, L., Schellmann, H., Sloane, M., Squires, P., Khan, F. A., & Stoyanovich, J. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00861-0 · Cited by 57
“As argued by several authors, stakeholder interests should be central to the task of scoping [11,23,42,50,51,52,53,66,73].”
BackgroundInfluentialA Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms
Coston, A., Kawakami, A., Zhu, H., Holstein, K., & Heidari, H. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00050 · Cited by 79
Online Certification of Preference-Based Fairness for Personalized Recommender Systems
Do, V., Corbett‐Davies, S., Atif, J., & Usunier, N. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i6.20606 · Cited by 66
“This type of internal audit (Raji et al., 2020) requires more access than usual external audits that focus on recommendation parity, but this is necessary to explore user preferences.”
BackgroundHenriksen, A., Enni, S. A., & Bechmann, A. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462564 · Cited by 54
“In contrast, researchers at Google have presented an end-to-end framework for conducting internal audits during the AI development process [78].”
ResultLewicki, K., Lee, M. S. A., Cobbe, J., & Singh, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581463 · Cited by 30
“Hey SyRI, tell me about algorithmic accountability”: Lessons from a landmark case
Wieringa, M. (2023) · Data & Policy · doi:10.1017/dap.2022.39 · Cited by 53
Aspirations and Practice of ML Model Documentation: Moving the Needle with Nudging and Traceability
Bhat, A., Coursey, A., Hu, G., Li, S., Nahar, N., Zhou, S., Kästner, C., & Guo, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581518 · Cited by 68
Agarwal, A., & Agarwal, H. (2023) · AI and Ethics · doi:10.1007/s43681-023-00266-9 · Cited by 49
Metadata
- Authors
- Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P.
- Affiliations
- Partnership on AI
- Google (United States) (US)
- ORCID
- Venue
- Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2020-01-27
- Pages
- 33-44
- OpenAlex
- W3001807593
- Cited by
- 2,245
- References
- 83
- Citations by year
- 2026: 1097 · 2025: 454 · 2024: 243 · 2023: 177 · 2022: 149 · 2021: 99 · 2020: 24
- FWCI
- 35.91
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- AI accountability, algorithmic auditing, harm detection, algorithmic transparency, societal impact of AI, algorithmic governance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.