Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index; 33 of the 53 sources its claims cite were checked for support.
Abstract
This paper introduces reviewability as a framework for improving the accountability of automated and algorithmic decisionmaking (ADM) involving machine learning. We draw on an understanding of ADM as a socio-technical process involving both human and technical elements, beginning before a decision is made and extending beyond the decision itself. While explanations and other model-centric mechanisms may assist some accountability concerns, they often provide insufficient information of these broader ADM processes for regulatory oversight and assessments of legal compliance. Reviewability involves breaking down the ADM process into technical and organisational elements to provide a systematic framework for determining the contextually appropriate record-keeping mechanisms to facilitate meaningful review - both of individual decisions and of the process as a whole. We argue that a reviewability framework, drawing on administrative law's approach to reviewing human decision-making, offers a practical way forward towards more a more holistic and legally-relevant form of accountability for ADM.
Overview
CoChat Summary
TL;DR
This paper introduces “reviewability” as a framework for improving accountability of automated and algorithmic decision-making (ADM) involving machine learning by breaking ADM into technical and organisational stages and specifying contextually appropriate record-keeping and logging. It argues that model-centric explanations are insufficient for regulatory oversight and legal compliance, and that a holistic, process-based approach drawing on English administrative law can better support meaningful accountability.
Key findings
- Model-focused transparency and explanation mechanisms are insufficient because ADM is a socio-technical process with human and organisational elements; problems often originate outside the model, in purposes, assumptions, design, and use.
- Reviewability requires contextually appropriate information that is (i) relevant to the accountability relationships involved, (ii) accurate, complete, and representative, (iii) proportionate to the required transparency, and (iv) comprehensible by the relevant forums.
- The paper proposes a framework with four stages: Commissioning (procurement, problem definition, impact assessment), Model building (data collection, pre-processing, model training, model testing), Decision-making (deployment, use, consequences), and Investigation (audit, disclosure).
- It draws on accountability as involving an actor, a forum, an account, and consequences, and cites five kinds of accountability relationship: political, legal, administrative, professional, and social. It also identifies three generic actor roles: managers, developers, and users.
Methods
- Conceptual and legal-theoretical framework development, drawing on English administrative law, accountability literature, and prior work applying administrative law to ADM.
- No empirical sample or quantitative analysis is reported; the paper synthesises related research and proposes a systematic framework for reviewable ADM.
- Main analysis breaks the ADM process into stages and steps, then identifies record-keeping and logging mechanisms that could provide contextually appropriate information for review of individual decisions and the process as a whole.
Limitations
- The paper states that reviewability cannot on its own address all potential problems with ADM; wider examination of political economy and socio-economic context, as well as legal protections for individuals and groups, is also needed.
- It states that transparency has limits: reviewable systems may not capture broader business processes, wider context, interactions with other socio-technical processes, business models, or questions of power; transparency also has temporal limitations and is open to subjective interpretation and contestation.
- It states that implementation research is needed on what record-keeping and logging are appropriate at each step, what information to retain, and how best to present it to forums; recording everything can be impractical and can create a “transparency paradox.”
- Evident from the text: the framework is conceptual and normative, with no empirical evaluation or deployment study reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
23 Valid · 10 Not valid · 12 Not checked · 78 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Administrative law as a framework and judicial review as an oversight mechanism understand decision-making as a process beginning before decisions are made and extending to their consequences and effects.”
§ Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems > 1 INTRODUCTION
What the sources say
- Supportedcites [13] 13. Jennifer Cobbe. 2019. Administrative Law and the Machines of Government: Judicial Review of Automated Public-Sector Decision-Making. Legal Studies (2019). source
The Review of ADM (p. 9) says judicial review of an individual decision may need to examine earlier choices about training data, system design and testing. The discussion of reasons (p. 22) says administrative-law requirements depend on how serious and consequential a decision is. Together, these support the claim that oversight considers both what precedes a decision and its effects. (LLM, conf=0.86)
- Not checked — no full text of the sourcecites [14] 14. Jennifer Cobbe, Michelle Seng Ah Lee, Heleen Janssen, and Jatinder Singh. 2020. Centring the Rule of Law in the Digital State. IEEE Computer (2020). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Models are only one aspect of ADM and cannot by themselves provide accounts of the process as a whole [66], whether to technical, legal, or other forums.”
§ Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems > 2.2.1 Meaningful accountability needs more than models.
What the sources say
- Supportedcites [66] 66. Jatinder Singh, Jennifer Cobbe, and Chris Norval. 2020. Decision Provenance: Harnessing Data Flow for Accountable Systems. IEEE Access 7 (2020). source
In “V. Case Study: Machine Learning Pipelines” (page unavailable), the paper describes models as components of interconnected systems and says accountability requires information about data flows, model use, and the downstream effects of decisions. “B. Technical Assistance” (page unavailable) connects system-wide transparency to both technical oversight and legal accountability. This supports the claim that a model alone cannot account for the whole decision-making process. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Rather, people are concerned with broader aspects of ADM: the purposes, roles, and outcomes of decision-making processes as a whole, as it is the whole process that affects them rather than simply certain technical aspects of it [26].”
§ Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems > 2.2.1 Meaningful accountability needs more than models.
What the sources say
- Partly supportedcites [26] 26. Lilian Edwards and Michael Veale. 2017. Slave to the Algorithm? Why a 'Right to an Explanation' is Probably Not the Remedy You Are Looking For. Duke Law & Technology Review 17 (2017). sourceExperimental — accuracy not yet measured
The Introduction (p. 2) says the public is troubled by systems that govern their lives and produce damaging or unfair results. The Conclusion (p. 64) favors broader algorithmic governance over explanations of a black box’s interior. These passages support the contrast with purely technical explanations, but they do not establish that people are specifically concerned with the purposes and roles of decision-making processes as a whole. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, understanding ADM as a sociotechnical process, we argue that there is another form of opacityunwitting opacity, where those responsible for designing, developing, deploying, and using systems simply don't think to record relevant organisational aspects of ADM processes (perhaps unaware of their relevance for meaningful accountability [12]).”
§ Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems > 2.1 Accountability in ADM
What the sources say
- Partly supportedcites [12] 12. Richard Cloete, Chris Norval, and Jatinder Singh. 2020. A Call for Auditable Virtual, Augmented and Mixed Reality. sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Though explanations can assist some specific concerns, like model engineering [7], explanations or other approaches focused on how the model itself works or has reached a particular outcome may miss much of what is important [75].”
§ Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems > 2.2 Limitations of model-focused mechanisms
What the sources say
- Supportedcites [7] 7. Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yun- han Jia, Joydeep Ghosh, Ruchir Puri, José M. F. Moura, and Peter Eckersley. 2020. Explainable Machine Learning in Deployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* '20). Association for Computing Machinery, New York, NY, USA, 648?657. https://doi.org/10.1145/3351095.3375624 sourceExperimental — accuracy not yet measured
The paper’s §3.4 Key Takeaways says that a description of how a model responds to an input change may not explain anything useful to stakeholders or reveal the underlying causal mechanism. Section 5.1 Establish Clear Desiderata likewise emphasizes identifying stakeholders’ needs and the purpose of an explanation. Together, these support the hedged claim that focusing on model behavior or a particular prediction can miss important concerns. (LLM, conf=0.86)
- Supportedcites [75] 75. Maranke Wieringa. 2020. What to account for when accounting for algorithms: a systematic literature review on algorithmic accountability. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 1-18. sourceExperimental — accuracy not yet measured
Section 3.2 says that understanding a system’s technical workings can be useful but is insufficient for accountability because it does not explain why the system was adopted or who set its specifications. Section 3.4 similarly argues that explanations focused on technical aspects can miss the system’s social context, development, and effects. (LLM, conf=0.94)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
24 more claims · 8 with their cited sources checked
Create a free account to see all 123 claims and every source verdict.
- Probably not a substantial claim. 87
- It may state two separate claims as one. 24
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Accountability has five types: legal responsibility to judicial bodies; political accountability to elected representatives and similar entities; administrative accountability to auditors and regulatory agencies; professional responsibility to internal and external colleagues; and social responsibility to community members and the public (Cobbe et al., 2021).”
From Reconciling Administrative Efficiency with Procedural Safeguards in the ADM System (2025) · cites this paper as Cobbe et al., 2021
In the citing paper
“Transparency which supports broader accountability processes on training sets, models and internal processes behind these services, as well as in the widersense of acknowledgement of known-limitations, past failures, and lessons learnt [27,131] can help move AIaaS from the opaque, unknown and potentially harmful 'black-boxes' of today, to a more understandable, accepting of their own limitations,”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [27]
Citations
Cites
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · cited 2×
“…of the research landscape show a vast number of mechanisms for developing ‘interpretable’ or ‘explainable’ ML systems have been proposed, seeking to provide explanations and other interpretable accounts of model behaviour to a range of forums, many of which have come in the last few years [4, 32].”
BackgroundThe Black Box Society: The Secret Algorithms That Control Money and Information
Pasquale, F. (2015) · doi:10.4159/harvard.9780674736061 · Cited by 3,796
A few useful things to know about machine learning
Domingos, P. (2012) · Communications of the ACM · doi:10.1145/2347736.2347755 · Cited by 3,326
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 5×
“Burrell identifies three forms of algorithmic opacity [10]; while Burrell refers primarily to model opacity, these also relate to broader ADM processes.”
BackgroundMethodInfluentialModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245 · cited 1×
“Even where assessments are not legally required, they are good practice, and robust internal (in-organisation) assessment processes have value [60].”
BackgroundEuropean Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · cited 1×
“Some proposals have seemingly been prompted by academic debate about the existence, nature, extent, and utility of a so-called ‘right to an explanation’ [26, 31, 44, 64, 74] in GDPR, which was passed in 2016.”
BackgroundAnanny, M., & Crawford, K. (2018) · New Media & Society · doi:10.1177/1461444816676645 · Cited by 1,836 · cited 8×
“There are limits to transparency itself as a mechanism for supporting accountability [2, 26, 68].”
BackgroundInfluential
Research works printed but not linked: 14
- Philip Alston. 2019. Report of the Special Rapporteur on extreme poverty and human rights. United Nations General Assembly A/74/493 (2019). https://undocs.org/A/74/493
- Mark Bovens. 2006. Analysing and Assessing Public Accountability. A Conceptual Framework. EUROGOV, European Governance Papers No. C-06-01 (2006).
- Cary Coglianese and Daniel Lehr. 2017. Regulating by Robot: Administrative Decision Making in the Machine-Learning Era. Georgetown Law Journal 105 (2017).
- European Commission. 2020. White Paper On Artificial Intelligence - A European Approach to Excellence and Trust. COM(2020) 65 (2020).
- 10.1145/2347736 · doi:10.1145/2347736
- Avi Feller, Emma Pierson, Sam Corbett-Davies, and Sharad Goel. 2016. A computer program used for bail and sentencing decisions was labeled biased against blacks. It's actually not that clear. The Washington Post (2016).
- Agata Foryciarz, Daniel Leufer, and Katarzyna Szymielewicz. 2020. Black-Boxed Politics: Opacity is a Choice in AI Systems. Panoptykon Foundation (17 January 2020). https://en.panoptykon.org/articles/black-boxed-politics-opacity-choice-ai-systems
- 10.1093/idpl · doi:10.1093/idpl
- David Lehr and Paul Ohm. 2017. Playing with the Data: What Legal Scholars Should Learn About Machine Learning. UC Davis Law Review 51 (2017).
- House of Common Science and Technology Committee. 2018. Algorithms in Decision-Making. Fourth Report of Session 2017-19 HC 351 (2018).
- Douglas Pyper. 2020. Research Briefing: The Public Sector Equality Duty and Equality Impact Assessments. House of Commons Library (2020). https://commonslibrary.parliament.uk/research-briefings/sn06591/
- Nick Seaver. 2013. Knowing Algorithms. paper presented at Media in Transition 8 (2013).
- Cynthia Stohl, Michael Stohl, and Paul M Leonardi. 2016. Managing Opacity: Information Visibility and the Paradox of Transparency in the Digital Age. International Journal of Communication 10 (2016).
- European Union. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). Official Journal of the European Union L119 (4 May 2016), 1--88.
Cited by
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
“Three ideas underpin the promise of auditing as an AI governance mechanism: that procedural regularity and transparency contribute to good governance [8], [9]; that proactivity in the design of AI systems helps identify risks and prevent harm before it occurs [10], [11]; and, that the independence between the auditor…”
BackgroundA literature review on applications of explainable artificial intelligence (XAI)
Kalasampath, K., Spoorthi, K. N., Sajeev, S., Kuppa, S. S., Ajay, K., & Maruthamuthu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3546681 · Cited by 180
“By comprehending the capabilities and restrictions of each kind of explanation, developers, and stakeholders can develop AI systems that are more efficient and reliable, thus advancing ethical and responsible AI practices [114].”
Auditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152
“Reflecting that convergence, previous research has developed procedures to audit AI systems for transparency and explainability (Cobbe et al., 2021; Mittelstadt, 2016), bias and fairness (Bartley et al., 2021; Morina et al., 2019) and accountability (Busuioc, 2021; Metcalf et al., 2021).”
MethodAI and Global Governance: Modalities, Rationales, Tensions
Veale, M., Matus, K., & Gorwa, R. (2023) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-020223-040749 · Cited by 134
“Issues of accountability can be separated from neither the much longer sociotechnical process (Cobbe et al. 2021) nor the need for an active, critical audience (Kemper & Kolkman 2018).”
BackgroundMethodOperationalising AI governance through ethics-based auditing: an industry case study
Mökander, J., & Floridi, L. (2023) · AI and Ethics · doi:10.1007/s43681-022-00171-7 · Cited by 129
“National regulators like the UK Information Commissioner’s Office have provided guidance on how to audit AI systems [18], and professional services firms like PwC, EY, Deloitte and KPMG have all developed auditing (or ‘assurance’) procedures to help clients ensure that the AI systems they design and deploy are legal,…”
BackgroundPolicy advice and best practices on bias and fairness in AI
Álvarez, J. M., Colmenarejo, A. B., Elobaid, A., Fabbrizzi, S., Fahimi, M., Ferrara, A., Ghodsi, S., Mougan, C., Papageorgiou, I. T., Lobo, P. R., et al. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09746-w · Cited by 160
“…algorithmic recruitment (Kazim et al., 2021), online housing markets (Asplund et al., 2020), resource allocation systems (Coston et al., 2021); and more general processes related to the design (Katell et al., 2019) and vision of these systems as socio-technical processes (Cobbe et al., 2021).”
Contestable AI by Design: Towards a Framework
Alfrink, K., Keller, I., Kortuem, G., & Doorn, N. (2022) · Minds and Machines · doi:10.1007/s11023-022-09611-z · Cited by 169
“Future efforts may seek to compare our proposed framework to more generic ethical, responsible and accountable AI frameworks (e.g. Cobbe et al., 2021; Hutchinson et al., 2021; Mohseni 2019; Raji et al., 2020).”
Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99
“Meaningful assessments are inherently contextual [28].”
BackgroundSelf-citationUnderstanding accountability in algorithmic supply chains
Cobbe, J., Veale, M., & Singh, J. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594073 · Cited by 115
“A particular difficulty, however, is that accountability is contextual [5, 12, 25, 80, 109, 110].”
BackgroundInfluentialSelf-citationBurr, C., & Leslie, D. (2021) · AI and Ethics · doi:10.1007/s43681-022-00178-0 · Cited by 147
Stakeholder roles in artificial intelligence projects
Miller, G. J. (2022) · Project Leadership and Society · doi:10.1016/j.plas.2022.100068 · Cited by 142
Minkkinen, M., Laine, J., & Mäntymäki, M. (2022) · Digital Society · doi:10.1007/s44206-022-00022-2 · Cited by 88
Lee, M. S. A., Floridi, L., & Singh, J. (2021) · AI and Ethics · doi:10.1007/s43681-021-00067-y · Cited by 71
Self-citationThe Conflict Between Explainable and Accountable Decision-Making Algorithms
Lima, G., Grgić-Hlača, N., Jeong, J. K., & Cha, M. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3534628 · Cited by 98
Making It Possible for the Auditing of AI: A Systematic Review of AI Audits and AI Auditability
Li, Y., & Goel, S. (2024) · Information Systems Frontiers · doi:10.1007/s10796-024-10508-8 · Cited by 105
“…leads to any subsequent actions or requires remedial response, such as system debugging and penalty actions, details of such post-audit actions and remedies should be recorded (Cobbe et al., 2021 A few studies propose the potential contribution of IT audit experiences and methods in AI audits.”
InfluentialAccountability in AI: From principles to industry-specific accreditation
Percy, C., Dragičević, S., Sarkar, S., & Garcez, A. D. (2022) · AI Communications · doi:10.3233/aic-210080 · Cited by 71
Norval, C., Cornelius, K., Cobbe, J., & Singh, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533133 · Cited by 35
“Eliciting these types of responses is indicative of an effective disclosure – facilitating the scrutiny of organisational practices, and better informing users as to how their personal data is being used, enabling them to appropriately respond [13].”
BackgroundInfluentialSelf-citationPossible Bias in Supervised Deep Learning Algorithms for CT Lung Nodule Detection and Classification
Sourlos, N., Wang, J., Nagaraj, Y., van Ooijen, P. M. A., & Vliegenthart, R. (2022) · Cancers · doi:10.3390/cancers14163867 · Cited by 47
“Therefore, a better approach is to seek reviewability in which we do not necessarily have explanations, but we expose the decision-making process including human processes, structures, and systems around a model [51].”
BackgroundLewicki, K., Lee, M. S. A., Cobbe, J., & Singh, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581463 · Cited by 30
Self-citation“Hey SyRI, tell me about algorithmic accountability”: Lessons from a landmark case
Wieringa, M. (2023) · Data & Policy · doi:10.1017/dap.2022.39 · Cited by 53
“One of the difficulties that academics face is how to operationalize and make accountability in practice (e.g., Cobbe et al., 2021; Kroll, 2021).”
BackgroundRethinking responsible AI from ethical pillars to sociotechnical practice
Ibitoye, A., Nkwo, M. S., & Orji, R. (2025) · AI and Ethics · doi:10.1007/s43681-025-00809-2 · Cited by 56
“…and value alignment, is analysed in turn, drawing on established frameworks such as Fairness in Machine Learning [ 20 ], Explainable AI [ 21], algorithmic accountability and Accountability for Reasonableness [ 22, 23], human rights-based approaches to AI [ 24], and value-sensitive design [ 25].”
Lessons Learned from Assessing Trustworthy AI in Practice
Vetter, D., Amann, J., Bruneault, F., Coffee, M. P., Düdder, B., Gallucci, A., Gilbert, T. K., Hagendorff, T., van Halem, I., Hickman, E., et al. (2023) · Digital Society · doi:10.1007/s44206-023-00063-1 · Cited by 43
“…ForHumanity, 2021), Brown et al.’s framework for ethical algorithm audits (Brown et al., 2021), the Reviewability framework presented by Cobbe et al. (Cobbe et al., 2021), Felländer et al.’s DRESS-eAI (Felländer et al., 2022), or the capAI procedure developed by Floridi et al. (Floridi et al.,…”
MethodArtificial Intelligence Project Success Factors—Beyond the Ethical Principles
Miller, G. J. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-030-98997-2_4 · Cited by 39
Towards a Better Understanding of Evaluating Trustworthiness in AI Systems
Kemmerzell, N., Schreiner, A., Khalid, H., Schalk, M., & Bordoli, L. (2025) · ACM Computing Surveys · doi:10.1145/3721976 · Cited by 45
“…openness, this strategy centers around examining the completeness and transparency of documentation regarding the AI system’s development process [52, 242], including data sources, algorithms, decision-making processes, and compliance with ethical guidelines [163, 205] and legal requirements…”
Software documentation is not enough! Requirements for the documentation of AI
Königstorfer, F., & Thalmann, S. (2021) · Digital Policy Regulation and Governance · doi:10.1108/dprg-03-2021-0047 · Cited by 35
Lee, M. S. A., & Singh, J. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462572 · Cited by 47
“This involves breaking down the technical and organisational elements of ML-driven decision-making [6].”
Self-citationA Room With an Overview: Toward Meaningful Transparency for the Consumer Internet of Things
Norval, C., & Singh, J. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3318369 · Cited by 32
“It follows that there is a need for meaningful trans-parency [17]: the provision of information in a manner that caters to the needs and expectations of the user to facilitate their effective oversight, scrutiny, and review over these technologies [18].”
BackgroundSelf-citationDrake, A., Keller, P., Pietropaoli, I., Puri, A., Maniatis, S. M., Tomlinson, J., Maxwell, J., Fussey, P., Pagliari, C., Smethurst, H., et al. (2021) · International Review of Law Computers & Technology · doi:10.1080/13600869.2021.1999075 · Cited by 33
“The discussions implied that it is useful to start from relatively simple norms, like active communications describing an algorithm, its purpose and where to seek further information (BritainThinks 2021), or systematic record-keeping (Cobbe, Lee, and Singh 2021; Win fi eld and Jirotka 2017).”
BackgroundMind The Gap: Designers and Standards on Algorithmic System Transparency for Users
Schor, B. G. S., Norval, C., Charlesworth, E., & Singh, J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642531 · Cited by 17
Self-citationKallina, E., Bohné, T., & Singh, J. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732069 · Cited by 36
Self-citationIntimate Data Sharing: Enhancing Transparency and Control in Fertility Tracking
Hudig, A. I., & Singh, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714089 · Cited by 25
Self-citationArtificial Intelligence Project Success Factors: Moral Decision-Making with Algorithms
Miller, G. J. (2021) · Annals of Computer Science and Information Systems · doi:10.15439/2021f26 · Cited by 24
“The reviewability framework [41] recommends maintaining data collection”
MethodBurr, C., & Leslie, D. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3937983 · Cited by 13
Wang, C., Yin, Y., & Hu, H. (2026) · Technology in Society · doi:10.1016/j.techsoc.2026.103264 · Cited by 43
Stakeholder-accountability model for artificial intelligence projects
Maxmetrics, H., & Miller, G. J. (2022) · Journal of Economics and Management · doi:10.22367/jem.2022.44.18 · Cited by 24
“The study framework “offers a legally-grounded, holistic, systematic, and practical framework for making algorithmic systems mean-ingfully accountable” (Cobbe et al., 2021, p. 607).”
BackgroundA Review of User Profiling Based on Social Networks
Wu, W., Ghazali, M., & Huspi, S. H. (2024) · IEEE Access · doi:10.1109/access.2024.3430987 · Cited by 13
“…and algorithmic decision-making (ADM) is an evaluability framework that decomposes administrative management processes into technical and organizational elements and is used to determine contextual record-keeping mechanisms, thereby providing a purposeful review of decisions and processes [155].”
BackgroundNemat, A. T., Becker, S. J., Lucas, S., Thomas, S., Gadea, I., & Charton, J. E. (2023) · Minds and Machines · doi:10.1007/s11023-023-09654-w · Cited by 14
“…organisations that employ them meet ethical standards (Epstein et al., 2018; Saleiro et al., 2018; AI Ethics Impact Group, 2020; Chmielinski, 2020 ; Cobbe et al., 2021 ; ForHumanity, 2021; Poretschkin, 2021; Grellette, 2022; Mökander et al., 2021, 2022a; Zicari et al., 2021; Floridi et al., 2022;…”
A Matrix for Selecting Responsible AI Frameworks
Narayanan, M., & Schoeberl, C. (2023) · doi:10.51593/20220029 · Cited by 18
Meaningful Transparency for Clinicians: Operationalising HCXAI Research with Gynaecologists
Schor, B. G. S., Kallina, E., Singh, J., & Blackwell, A. F. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658971 · Cited by 20
“To become meaningful, it must be contextually appropriate , i.e. the information transferred to clinicians must be relevant, accurate, proportionate, and comprehensible to them [26].”
BackgroundInfluentialSelf-citationHow do AI systems fail socially?: an engineering risk analysis approach
Rismani, S., & Moon, A. (2021) · IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS) · doi:10.1109/ethics53270.2021.9632769 · Cited by 8
Navigating the Audit Landscape: A Framework for Developing Transparent and Auditable XR
Norval, C., Cloete, R., & Singh, J. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594090 · Cited by 13
“For one, a key consideration for the auditability of XR relates to the overall intent of the XR system, i.e. understanding the system’s aims, how it will likely be operated, and the potential risks that might arise (e.g. information about ‘commissioning’, per [26]).”
BackgroundInfluentialSelf-citationNigatu, H. H., Pickoff-White, L., Canny, J., & Chasins, S. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594093 · Cited by 13
Valdivia, A., Hyde-Vaamonde, C., & Marcos, J. G. (2024) · AI & Society · doi:10.1007/s00146-024-02016-9 · Cited by 25
“However, there is no data available to assess whether this limitation on the EPV could have entailed potential risks to partners.”
Risk Identification Questionnaire for Unintended Bias in Machine Learning Development LifecyclePreprint
Lee, M. S. A., & Singh, J. P. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3777093 · Cited by 6
Self-citationRights Out of Sight: Data Practices and Transparency Gaps in Smart Consumer IoT Ecosystems
Hudig, A. I., Mandalari, A. M., Norval, C., Haddadi, H., Binns, R., & Singh, J. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732154 · Cited by 10
Self-citationOn the Complexities of Testing for Compliance with Human Oversight Requirements in AI Regulation
Langer, M., Lazar, V., & Baum, K. L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-07132-3_11 · Cited by 27
“Moreover, without concrete standards, virtually any implementation could be considered compliant [16].”
Accountable AI for Healthcare IoT Systems
Bagave, P., Westberg, M., Dobbe, R., Janssen, M., & Ding, A. Y. (2022) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa56441.2022.00013 · Cited by 16
Lee, M. S. A., Floridi, L., & Singh, J. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3679975 · Cited by 10
Self-citationBouwmeester, M. (2025) · European Journal of Social Security · doi:10.1177/13882627251330438 · Cited by 9
“In response to the different drivers of opacity (black box problem, intentional opacity and technical illiteracy, see more extensively Cobbe et al. 2021), various mechanisms are proposed to enhance ‘ algorithmic accountability ’ (Busuioc, 2021), such as specialised parliamentary committees (Moulds,…”
Brintrup, A. M., Baryannis, G., Tiwari, A., Ratchev, S. M., Martínez-Arellano, G., & Singh, J. (2025) · Data-Centric Engineering · doi:10.1017/dce.2025.10032 · Cited by 47
“In practice, transparency generally will not solve issues with the technology (Ananny and Crawford, 2018), but can provide a basis for supporting recourse, repair, and accountability more generally (Cobbe et al., 2021; Williams et al., 2022).”
InfluentialSelf-citationThe tensions of data sharing for human rights: A modern slavery case study
Hancock, J., Mahesh, S., Cobbe, J., Singh, J., & Mazumder, A. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658949 · Cited by 5
“Accountability mechanisms may be based on formal legal frameworks—such as provisions giving data subjects certain rights over their personal data—as well as technical systems [17, 18].”
BackgroundSelf-citationPowar, J., Janssen, H., Cloete, R., & Singh, J. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732164 · Cited by 11
“These approaches aim to rebalance the interests and power dynamics of data ecosystems in favour of those to whom data relates or ‘belongs’, who previously had limited involvement in the use of that data, a lack of control that is reinforced by systemic power and information asymmetries [22, 23, 68].”
Self-citationAuditable Augmented/Mixed/Virtual Reality: The Practicalities of Mobile System Transparency
Cloete, R., Norval, C., & Singh, J. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3495001 · Cited by 7
Self-citationGive and Take: Perceptions of a Conversational Coach Agent in Fitness Trackers MHCI014
Loerakker, M. B., Stefanidi, E., Niess, J., Mildner, T., & Woźniak, P. W. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3743718 · Cited by 13
“Transparency becomes especially meaningful given it is ‘contextually appropriate’ [26, 28].”
Operationalising digital ethics: establishment of an ethics evaluation tool for data analytics
Charton, J. E., Lucas, S., Vogd, W., Halter, W., & Becker, S. J. (2025) · AI & Society · doi:10.1007/s00146-025-02621-2 · Cited by 7
“…certification processes or ethics-based audits (Epstein et al. 2018; Saleiro et al. 2018; AI Ethics Impact Group 2020; Chmielinski et al. 2020; Cobbe et al. 2021; ForHumanity 2021; Poretschkin et al. 2021; Grellette 2022; Mökander et al. 2021, 2022; Zicari et al. 2021; Floridi et al. 2022;…”
Velotto, G. (2025) · AI and Ethics · doi:10.1007/s43681-025-00880-9 · Cited by 4
“Contestability-by-design pairs explanations with those procedures so disagreement becomes governable rather than rhetorical [25 , 26 , 104].”
The Prediction of Leadership Degree Based on Machine Learning
Skrynnyk, O., & Vasylieva, T. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-14841-5_6 · Cited by 4
Equitable AI Audits: evaluating the evaluators in today's world
Munarini, M., Brusseau, J., & Angeli, L. (2024) · International Conference on Theory and Practice of Electronic Governance · doi:10.1145/3680127.3680140 · Cited by 6
Beyond aversion – principles of appropriate algorithmic decision-making in human resource management
Strohmeier, S., Becker, M., & Scheer-Weller, E. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128954 · Cited by 9
Rethinking the judicial duty to state reasons in the age of automation?
Hendrickx, V. (2025) · Cambridge Forum on AI Law and Governance · doi:10.1017/cfl.2025.11 · Cited by 8
Towards adaptive categories: Dimensional governance for agentic AI
Engin, Z., & Hand, D. J. (2026) · Data & Policy · doi:10.1017/dap.2026.10083 · Cited by 25
“Lifecycle governance models promote oversight across system development and deployment stages (Raji et al., 2020; Cobbe et al., 2021), while human-centred (Capel and Brereton, 2023) and trustworthy AI (Li et al., 2023) frameworks foreground the role of transparency, contestability, and calibrated…”
Los seres humanos en la toma de decisiones automatizada en el marco del RGPD y la Ley de IA
Levitina, A. (2024) · Revista CIDOB d Afers Internacionals · doi:10.24241/rcai.2024.138.3.121 · Cited by 8
“It’s Just a Wild, Wild West”: Harnessing Public Procurement as an AI Governance Mechanism
Hudig, A. I., Kallina, E., & Singh, J. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791968 · Cited by 3
Self-citationTrouble at Sea: Data and digital technology challenges for maritime human rights concerns
Hancock, J., Hui, R., Singh, J., & Mazumder, A. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658950 · Cited by 8
“…(1) trans-parency within technical systems and their surroundings, as has been studied extensively by technology and privacy researchers (e.g. [19, 28, 51, 65]); (2) transparency within markets and supply chains for data and related tech (e.g. regarding suppliers, product specifications,…”
BackgroundSelf-citationMaking Algorithmic Systems More Meaningfully Transparent for Users Through Visualisation
Schor, B. G. S., Singh, J., & Blackwell, A. F. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532107.3532875 · Cited by 6
“As a growing number of systems have been demonstrated to impact entire communities negatively [20], transparency —which can contribute to redressing information and power asymmetries [26]— is largely recognised as helpful and important, eg. to support accountability [14, 27].”
BackgroundMethodSelf-citationReducing organizational inequalities associated with algorithmic controls
Li, Y., & Xiang, B. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00137-0 · Cited by 11
“[19, 22, 113].”
InfluentialHuman–AI hybrid finance: from AI tools to decision systems
Kou, G., Li, Y., Wang, H.-M., & Wang, X. (2026) · Financial Innovation · doi:10.1186/s40854-026-00941-w · Cited by 7
“Fourth, it should be auditable : the explanation and the human response to it should be recorded in a way that supports later review (Cobbe et al. 2021).”
IoTracker: A probabilistic event tracking approach for data-intensive IoT Smart Applications
Biondi, G., Prati, R. C., Borelli, F., Silva, D., de Oliveira, N. G., & Kamienski, C. (2022) · Internet of Things · doi:10.1016/j.iot.2022.100556 · Cited by 2
Touameur, O., & Harrag, F. (2023) · International Engineering Conference on Electrical, Energy, and Artificial Intelligence (EICEEAI) · doi:10.1109/eiceeai60672.2023.10590431 · Cited by 2
“However, this factor would be in the cost of security because making ADMs transparent will expose them to vulnerable attacks [17].”
BackgroundKallina, E., Leeb, C. M., & Singh, J. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790959 · Cited by 1
Self-citationLee, M. S. A., Floridi, L., & Singh, J. (2022) · Digital ethics lab yearbook · doi:10.1007/978-3-031-09846-8_11 · Cited by 3
Self-citationTrucks Don’t Mean Trump: Diagnosing Human Error in Image Analysis
Zamfirescu-Pereira, J. D., Chen, J., Wen, E., Koenecke, A., Garg, N., & Pierson, E. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533145 · Cited by 6
“Second, algorithmic tools are often designed to be used by humans, so understanding the imperfections of the human in the loop is necessary to ensure the system as a whole is fair [19, 45, 64].”
BackgroundParadata as a Tool for Legal Analysis: Utilising Data-on-Data Related Processes
Enqvist, L. (2024) · Knowledge management and organizational learning · doi:10.1007/978-3-031-53946-6_13 · Cited by 2
Causality-based accountability mechanisms for socio-technical systems
Ibrahim, A., Kyriakopoulos, S., & Pretschner, A. (2021) · Journal of Responsible Technology · doi:10.1016/j.jrt.2021.100016 · Cited by 6
Improving Group Fairness Assessments with Proxies
Harvey, E., Lee, M. S. A., & Singh, J. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3677175 · Cited by 3
Self-citationMohanta, A., Garikapati, K., Sarangi, N. C., Samanta, S., & Mohanty, S. (2025) · Advances in human and social aspects of technology book series · doi:10.4018/979-8-3693-9035-1.ch012 · Cited by 3
Alanazi, M. (2026) · Nursing in Critical Care · doi:10.1111/nicc.70537 · Cited by 2
Asif, M., Ersoy, A., Shahzad, M. F., Ahmad, M. B., & Ashfaq, M. (2026) · Tourism and Hospitality Research · doi:10.1177/14673584261474503 · Cited by 1
Algorytm jako informacja publiczna w prawie europejskim
Mazur, J. (2021) · Wydawnictwa Uniwersytetu Warszawskiego eBooks · doi:10.31338/uw.9788323551355 · Cited by 4
Exploring Generative AI as Personally Effective Decision-Making Tools: A Thought Experiment
Hai-Jew, S. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-6230-3.ch014 · Cited by 1
The Role of the EU Right to Good Administration in Shaping Algorithmic Decision Systems
Jan, B. (2024) · European yearbook of constitutional law · doi:10.1007/978-94-6265-647-5_8 · Cited by 1
Framework for Managing Artificial Intelligence (AI) Projects: Avoiding Harms, Losses, and Damages
Miller, G. J. (2025) · Information systems engineering and management · doi:10.1007/978-3-031-80275-1_7 · Cited by 4
Harkens, A. (2025) · Data science, machine intelligence, and law · doi:10.1007/978-3-031-86813-9_9 · Cited by 2
The Algorithmic Construction of Epistemic Injustice
McInerney, T. (2025) · Palgrave socio-legal studies · doi:10.1007/978-3-032-07581-9_5 · Cited by 9
Measuring the Effectiveness of Ethics-Based Auditing – A Semi-Systematic Literature ReviewPreprint
Shafik, E., Puncheva, P., Bharathy, G., & Pannai, E. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.5498160 · Cited by 1
A Typology of Explanations for Explainability-by-DesignPreprint
Tsakalakis, N., Stalla-Bourdillon, S., Huynh, T. D., & Moreau, L. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5020690
Penovski, R., Srčič, S., & Scherer, R. (2025) · doi:10.2196/preprints.75491
Automation and Human Synergy: Redefining Work in the Digital Enterprise
Katiyar, J. (2025) · European journal of computer science and information technology · doi:10.37745/ejcsit.2013/vol13n48164176
Responsible Artificial Intelligence for Healthcare: Trends, Challenges, and Opportunities
Baker, S., & Ahn, E. (2025) · Springer eBooks · doi:10.1007/978-3-031-88024-7_1
Truby, J. (2025) · Oxford University Press eBooks · doi:10.1093/9780198945215.003.0155
Counterfactual Explanations via Locally-guided Sequential Algorithmic Recourse
Small, E. A., Clark, J. N., McWilliams, C., Ambler, M. T., Sokol, K., Chan, J., Salim, F. D., & Santos‐Rodríguez, R. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3770065 · Cited by 4
Reconciling Administrative Efficiency with Procedural Safeguards in the ADM System
Sharaf Addin, E. (2025) · Bratislava Law Review · doi:10.46282/blr.2025.9.2.1116
“…bodies; political accountability to elected representatives and similar entities; administrative accountability to auditors and regulatory agencies; professional responsibility to internal and external colleagues; and social responsibility to community members and the public (Cobbe et al., 2021).”
Socio-Institutional Issues and Policy Directions for Advanced Sports Policy in the AX Era
Kim, M., & Kwak, M. (2025) · 한국여성체육학회지 · doi:10.16915/jkapesgw.2025.12.39.4.1
Civil Rights in the Age of AI: Challenges and Strategies
Abdurakhmanov, A. Y., Boltayev, A., Urazmetov, M., & Ahmad, M. (2026) · IGI Global eBooks · doi:10.4018/979-8-3693-3920-6.ch001
Gropper, J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6156746
Devin, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6135630 · Cited by 3
Epistemic Debt: The Economics Of Unwarranted Institutional ReliancePreprint
Devin, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6135728 · Cited by 1
Tasking AI Fairly. How to Empower AI Practitioners With sXAI?
Alpsancar, S., & Stamboliev, E. (2026) · Springer eBooks · doi:10.1007/978-981-96-5290-7_29
Baryannis, G., & Papadakis, E. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6501399
Liu, Q. (2026) · International Journal of Digital Law and Governance · doi:10.1515/ijdlg-2025-0021
“…logging systems that record AI inputs, outputs, decision rationale, and human interventions in a traceable and tamper-proof manner for audit purposes (Cobbe et al. 2021); second, de fi ne tiered review thresholds calibrated to case severity, such that the more serious the case, the higher the…”
Metadata
- Authors
- Cobbe, J., Lee, M. S. A., & Singh, J.
- Affiliations
- University of Cambridge (GB)
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 598-609
- OpenAlex
- W3134395196
- Semantic Scholar
- 1b94ddf67a6e0bef90ef83fad8301a3d71c41096
- Cited by
- 150
- References
- 52
- Citations by year
- 2026: 63 · 2025: 29 · 2024: 15 · 2023: 13 · 2022: 18 · 2021: 11
- FWCI
- 12.86
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- Funders
- Engineering and Physical Sciences Research Council (EP/P024394/1, EP/R033501/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Digitalization, Law, and Regulation
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- reviewability, automated decision-making, algorithmic decision-making, accountability, machine learning, administrative law, regulatory oversight, legal compliance, record keeping, socio-technical processes, administrative review
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.