Interpretable algorithmic forensics
Open accessThe DOI matches its Crossref record; 35 of the 37 references it cites were found in a registry or an index; 10 of the 48 sources its claims cite were checked for support.
Abstract
One of the most troubling trends in criminal investigations is the growing use of “black box” technology, in which law enforcement rely on artificial intelligence (AI) models or algorithms that are either too complex for people to understand or they simply conceal how it functions. In criminal cases, black box systems have proliferated in forensic areas such as DNA mixture interpretation, facial recognition, and recidivism risk assessments. The champions and critics of AI argue, mistakenly, that we face a catch 22: While black box AI is not understandable by people, they assume that it produces more accurate forensic evidence. In this Article, we question this assertion, which has so powerfully affected judges, policymakers, and academics. We describe a mature body of computer science research showing how “glass box” AI—designed to be interpretable—can be more accurate than black box alternatives. Indeed, black box AI performs predictably worse in settings like the criminal system. Debunking the black box performance myth has implications for forensic evidence, constitutional criminal procedure rights, and legislative policy. Absent some compelling—or even credible—government interest in keeping AI as a black box, and given the constitutional rights and public safety interests at stake, we argue that a substantial burden rests on the government to justify black box AI in criminal cases. We conclude by calling for judicial rulings and legislation to safeguard a right to interpretable forensic AI.
Overview
CoChat Summary
TL;DR
This article argues that the “black box performance myth”—the assumption that noninterpretable AI is more accurate than interpretable AI—is unsupported, especially in criminal forensic settings. It calls for a strong legal, evidentiary, and constitutional presumption of glass box, interpretable AI in criminal cases.
Key findings
- For tabular and image data, interpretable models are generally as accurate as the best black box models on benchmark datasets; in criminal recidivism prediction, a simple model using age, gender, and prior criminal record was “just as predictable” as COMPAS, which could rely on up to 137 inputs.
- In DNA mixture interpretation, a New York judge noted that “estimates as to the likelihood of an incorrect conclusion where there actually are four or more contributors [to the DNA sample] run to over 50%”; after source code disclosure, concerns about the Forensic Statistical Tool led to its discontinuation.
- Criminal justice data are often noisy and incomplete: in 2021, during the transition from UCR to NIBRS, about 40% of law enforcement agencies did not report data to the FBI.
- Courts have often deferred to black box forensic AI, but the authors propose a glass box presumption requiring the government to show a substantial or compelling interest to justify noninterpretable AI; they note 10 states have passed restrictions on certain law enforcement uses of facial recognition, but none require glass box AI.
Methods
- Design: conceptual and legal-policy article synthesizing computer science research, constitutional criminal procedure doctrine, evidence law, and case law; it is not an empirical study.
- Data or sample: cited machine learning benchmark studies, criminal justice risk assessment examples such as COMPAS and the Forensic Statistical Tool, court rulings including *State v. Loomis*, *People v. Collins*, *People v. Thompson*, *Commonwealth v. Foley*, and *People v. Wakefield*, plus statutes and regulatory developments.
- Main analysis: distinguishes interpretability, transparency, and explainability; reviews evidence on interpretable versus black box accuracy; and applies due process, confrontation, equal protection, discovery, and expert-evidence principles to argue for a right to interpretable forensic AI.
Limitations
- Stated: the authors acknowledge that future noninterpretable AI systems may show substantial, demonstrated performance advantages, and that black box use might be justified by a compelling interest such as national security, though it should still be independently vetted.
- Stated: constitutional rights may be insufficient because they are unevenly enforced in criminal cases, indigent defendants face barriers to obtaining experts and discovery, and no U.S. legislative enactments or proposals have required open or glass box forensic evidence.
- Evident from the text: the article presents no new empirical experiment, dataset, or quantitative meta-analysis; its claims about accuracy rely on cited studies and selected legal examples.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 18 Not valid · 33 Not checked · 51 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“For tabular data, very small interpretable models can often perform as well as black box models on benchmark datasets if they are optimized carefully.”
§ Interpretable algorithmic forensics > Interpretable Versus Explainable AI > The Black Box Performance Myth.
What the sources say
- Supportedcites [22] C. Rudin, Machine learning: Fundamental principles and 10 grand challenges. Stat. Surv. 16, 1– 85 (2022). source
The “Static datasets” section says interpretable models generally do not lose accuracy to black-box models on benchmarks and that simple models perform surprisingly well on tabular data. The “Sparse logical models” section adds that fully optimized single trees can match ensembles or neural networks on many problems, illustrating this with a nine-leaf tree for the Monk 2 dataset. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Thus, for either tabular data or images, interpretable models are generally as accurate as the best of the black box models when applied to benchmark datasets (22).”
§ Interpretable algorithmic forensics > Interpretable Versus Explainable AI > The Black Box Performance Myth.
What the sources say
- Supportedcites [22] C. Rudin, Machine learning: Fundamental principles and 10 grand challenges. Stat. Surv. 16, 1– 85 (2022). source
The “Static datasets” section says interpretable models do not generally lose accuracy relative to black-box models on benchmark datasets, including challenging computer-vision benchmarks. It also says prediction accuracy is similar across most algorithms for tabular data. Page numbers are not provided in the supplied copy. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Researchers in the risk prediction field have long found that a small number of simple factors are modestly predictive: largely age, gender, and prior criminal activity (33).”
§ Interpretable algorithmic forensics > Interpretable Versus Explainable AI > The Superiority of Interpretable AI in Forensics.
What the sources say
- Partly supportedcites [33] J. Monahan, J. Skeem, Risk assessment in criminal sentencing. Ann. Rev. Clinical Psych. 12, 489 (2016). 26666966 sourceExperimental — accuracy not yet measured
The Pennsylvania Commission on Sentencing section describes a predictive scale that includes age, gender, and several measures of prior arrests. Selecting an Instrument also says a few items can provide substantial predictive utility and reports moderate accuracy for validated tools. But the paper does not establish that prediction rests largely on those three factors; it identifies other predictive dimensions and says broader estimates outperform criminal history alone. Page numbers were not supplied for these passages. (LLM, conf=0.84)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Basic typographical errors in inputs to black box recidivism prediction models have led to catastrophic errors deeply affecting people’s lives (29).”
§ Interpretable algorithmic forensics > Interpretable Versus Explainable AI > Special Cause For Accuracy Concerns in Forensics.
What the sources say
- Partly supportedcites [29] C. Rudin, C. Wang, B. Coker, The age of secrecy and unfairness in recidivism prediction. Harv. Data Sci. Rev. 2, 1 (2020). sourceExperimental — accuracy not yet measured
Section 3 (pp. 20–21) describes a data-input error involving the proprietary COMPAS model and a prisoner whose parole was denied. But the paper’s example of a mistyped input preceding a murder involved a simple, transparent scoring system, not a black-box model (Introduction, p. 2; Section 3, p. 19). The claim conflates these cases when it attributes that catastrophic outcome to black-box models. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In fact, quite a lot of effort has been made by scientists to understand how COMPAS depends on important variables like race and age, without a lot of success (59, 60).”
§ Interpretable algorithmic forensics > Interpretable Versus Explainable AI > Forensic AI in the Courts.
What the sources say
- Partly supportedcites [59] J. Dressel, H. Farid, The accuracy, fairness, and limits of predicting recidivism. Sci. Adv. 4, eaao5580 (2018). 29376122 PMC5777393 sourceExperimental — accuracy not yet measured
The Results (Algorithmic assessment; page unavailable) support the claim that researchers investigated COMPAS despite its undisclosed inner workings. But the paper reports a substantial result: a simple linear classifier using age and prior convictions matched COMPAS’s predictive accuracy. It therefore does not support the broad characterization that this effort had little success; it also says race is not an explicit input, though other inputs may correlate with it. (LLM, conf=0.85)
- Partly supportedcites [60] E. Angelino, N. Larus-Stone, D. Alabi, M. Seltzer, C. Rudin, " Learning certifiably optimal rule lists" in 2017 Proceedings of the 23rd ACM SIGKDD Int’l Conf. on Knowledge Discovery and Data Mining (ACM, 2017), p. 35. sourceExperimental — accuracy not yet measured
The Introduction (p. 2) supports the 'without a lot of success' half: because COMPAS is a black box, proprietary model, 'no one (outside of the creators of COMPAS) can determine the reason or extent of the bias ... nor can anyone determine the reason for any particular prediction.' But our copy nowhere records 'quite a lot of effort ... by scientists' — the investigations it credits are ProPublica's journalism (Larson et al., 2016) and, in §6.5 (p. 47), work subsequent to this paper (Dressel and Farid, 2018). Moreover the paper's own analysis in 'Comparison of CORELS and COMPAS by race' (pp. 41-42) claims some success rather than none: transparent rule lists built only from prior-crime counts, age and gender predict similarly to COMPAS for blacks and whites and 'indicate possible explanations' for the racial disparity, and the authors state their analysis 'does not indicate [bias] is necessarily true as long as criminal history and age are allowed to be considered as features.' (LLM, conf=0.68)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
33 more claims · 2 with their cited sources checked
Create a free account to see all 111 claims and to verify the 4 references still open.
- Probably not a substantial claim. 54
- It may state two separate claims as one. 41
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Interpretable algorithmic forensics > Interpretable Versus Explainable AI > A Glass Box Presumption.
“So far 10 states have passed restrictions on certain law enforcement uses of FRT, but these restrictions are not all likely to address the problem we discuss here (62).”
§ Interpretable algorithmic forensics > References
“62. Regulation of the European Parliament and of the Council, Laying Down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (April 2021)”
Citations
Cites
Can we open the black box of AI?
Castelvecchi, D. (2016) · Nature · doi:10.1038/538020a · Cited by 1,773 · cited 1×
“One called it as difficult to understand black box AI as it is to “understand the networks inside” the human brain (17).”
BackgroundExplainable AI: From black box to glass box
Rai, A. (2020) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-019-00710-5 · Cited by 1,317 · cited 1×
“Some of the most trenchant critics of black box AI similarly emphasize how AI derives its efficiency and effectiveness from its “inherently uninterpretable” associations and processes (16).”
BackgroundInterpretable Machine Learning: Fundamental Principles and 10 Grand ChallengesPreprint
Rudin, C., Chen, C.-F., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2021) · Statistics Survey · doi:10.48550/arxiv.2103.11251 · Cited by 1,254
Self-citationThe accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071 · cited 1×
“In fact, quite a lot of effort has been made by scientists to understand how COMPAS depends on important variables like race and age, without a lot of success (59, 60).”
BackgroundInterpretable machine learning: Fundamental principles and 10 grand challenges
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958 · cited 1×
“Thus, for either tabular data or images, interpretable models are generally as accurate as the best of the black box models when applied to benchmark datasets (22).”
BackgroundSelf-citationThis Looks Like That: Deep Learning for Interpretable Image Recognition
Chen, C., Li, O., Tao, C., Barnett, A. J., Su, J., & Rudin, C. (2018) · arXiv · doi:10.48550/arxiv.1806.10574 · Cited by 702 · cited 2×
“The system not only explained how it reached its results, but provided visual justifications for it, by showing what features of a bird, for example, led it to conclude that it was a red- bellied woodpecker (30).”
BackgroundSelf-citationThe Growth of Incarceration in the United States: Exploring Causes and Consequences
(2014) · National Academies Press eBooks · doi:10.17226/18613 · Cited by 599
Rudin, C., & Radin, J. (2019) · Harvard Data Science Review · doi:10.1162/99608f92.5a8a3a3d · Cited by 432 · cited 2×
“As one of us has put it simply: “Why Are We Using Black Box Models in AI When We Don’t Need To?””
BackgroundSelf-citationThe Judicial Demand for Explainable Artificial Intelligence
Deeks, A. S. (2019) · Columbia Law Review · Cited by 303
(2018) · Harvard journal of law & technology · Cited by 295
Research works printed but not linked: 4
- H. B. Dixon, Artificial intelligence: Benefits and unknown risks. ABA J. 60, 1 (2021).
- B. L. Garrett, C. Rudin, The right to a glass box: Rethinking the use of AI in criminal cases. Cornell Law Rev. 109 (forthcoming).
- T. Z. Zarsky, Transparent predictions. Univ. Ill. Law Rev. 2013, 1503, 1520 (2013).
- A. Roth, Trial by machine. Georgetown Law J. 104, 1245, 1300 (2016).
Cited by
Artificial Intelligence and the Future of Humans
Ovidius University of Constanta, D. O. T., & Ciocan, T.-C. (2023) · DIALOGO · doi:10.51917/dialogo.2023.10.1.15 · Cited by 243
Beyond principlism: practical strategies for ethical AI use in research practices
Lin, Z. (2025) · AI and Ethics · doi:10.1007/s43681-024-00585-5 · Cited by 40
“In contrast, making AI processes transparent and comprehensible, as exempli - fied by interpretable and explainable AI [40 , 41 ], while not How to measure biases in LLMs?”
de Souza, F. G., Bhansali, S., Pal, K., Maranhão, F. D. S., Oliveira, M. S., Valladão, V. S., Silva, D. S. B. E., & Silva, G. B. (2024) · Materials · doi:10.3390/ma17051088 · Cited by 42
Krook, J., Winter, P., Downer, J., & Blockx, J. (2025) · AI and Ethics · doi:10.1007/s43681-025-00674-z · Cited by 48
“Similarly, Garrett et al. (2023) [46] found numerous black box AI algorithms in the criminal justice system (e.g., analysing complex DNA mixtures, conducting risk assessments in pretrial decision-making and sentencing, and as part of facial recognition systems used by law enforcement to identify…”
Artificial intelligence in forensic mental health: A review of applications and implications
Olawade, D. B., Ayoola, F. I., Ebo, T. O., Asaolu, A. J., Egbon, E., & David-Olawade, A. C. (2025) · Journal of Forensic and Legal Medicine · doi:10.1016/j.jflm.2025.102895 · Cited by 33
Metadata
- Authors
- Garrett, B. L., & Rudin, C.
- Affiliations
- Duke University (US)
- Corresponding author
- Garrett, Brandon L.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2023-10-02
- Volume
- 120
- Issue
- 41
- Pages
- e2301842120-e2301842120
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC10576126
- OpenAlex
- W4387246019
- Semantic Scholar
- 7af943edd1f51a581cb029b3db7cdc90411a7099
- Cited by
- 68
- References
- 37
- Citations by year
- 2026: 30 · 2025: 17 · 2024: 18 · 2023: 3
- FWCI
- 4.84
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Iowa State University Digital Repository (Iowa State University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Artificial Intelligence in Law
- Keywords
- explainable AI, black-box algorithms, forensic evidence, DNA mixture interpretation, face recognition, recidivism risk assessment, AI legislation
- MeSH terms
- Algorithms, Artificial Intelligence, Forensic Medicine, Humans, Law Enforcement, Criminals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Interpretable algorithmic forensics. https://app.cochat.ai/papers/10.1073/pnas.2301842120
- The paper
- Garrett, B. L., & Rudin, C. (2023). Interpretable algorithmic forensics. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2301842120
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.