CoChat

Interpretable algorithmic forensics

Open access
Garrett, Brandon L., Rudin, Cynthia · 2023 · Proceedings of the National Academy of Sciences

The DOI matches its Crossref record; 35 of the 37 references it cites were found in a registry or an index; 10 of the 48 sources its claims cite were checked for support.

Abstract

One of the most troubling trends in criminal investigations is the growing use of “black box” technology, in which law enforcement rely on artificial intelligence (AI) models or algorithms that are either too complex for people to understand or they simply conceal how it functions. In criminal cases, black box systems have proliferated in forensic areas such as DNA mixture interpretation, facial recognition, and recidivism risk assessments. The champions and critics of AI argue, mistakenly, that we face a catch 22: While black box AI is not understandable by people, they assume that it produces more accurate forensic evidence. In this Article, we question this assertion, which has so powerfully affected judges, policymakers, and academics. We describe a mature body of computer science research showing how “glass box” AI—designed to be interpretable—can be more accurate than black box alternatives. Indeed, black box AI performs predictably worse in settings like the criminal system. Debunking the black box performance myth has implications for forensic evidence, constitutional criminal procedure rights, and legislative policy. Absent some compelling—or even credible—government interest in keeping AI as a black box, and given the constitutional rights and public safety interests at stake, we argue that a substantial burden rests on the government to justify black box AI in criminal cases. We conclude by calling for judicial rulings and legislation to safeguard a right to interpretable forensic AI.

Overview

CoChat Summary

TL;DR

This article argues that the “black box performance myth”—the assumption that noninterpretable AI is more accurate than interpretable AI—is unsupported, especially in criminal forensic settings. It calls for a strong legal, evidentiary, and constitutional presumption of glass box, interpretable AI in criminal cases.

Key findings

  • For tabular and image data, interpretable models are generally as accurate as the best black box models on benchmark datasets; in criminal recidivism prediction, a simple model using age, gender, and prior criminal record was “just as predictable” as COMPAS, which could rely on up to 137 inputs.
  • In DNA mixture interpretation, a New York judge noted that “estimates as to the likelihood of an incorrect conclusion where there actually are four or more contributors [to the DNA sample] run to over 50%”; after source code disclosure, concerns about the Forensic Statistical Tool led to its discontinuation.
  • Criminal justice data are often noisy and incomplete: in 2021, during the transition from UCR to NIBRS, about 40% of law enforcement agencies did not report data to the FBI.
  • Courts have often deferred to black box forensic AI, but the authors propose a glass box presumption requiring the government to show a substantial or compelling interest to justify noninterpretable AI; they note 10 states have passed restrictions on certain law enforcement uses of facial recognition, but none require glass box AI.

Methods

  • Design: conceptual and legal-policy article synthesizing computer science research, constitutional criminal procedure doctrine, evidence law, and case law; it is not an empirical study.
  • Data or sample: cited machine learning benchmark studies, criminal justice risk assessment examples such as COMPAS and the Forensic Statistical Tool, court rulings including *State v. Loomis*, *People v. Collins*, *People v. Thompson*, *Commonwealth v. Foley*, and *People v. Wakefield*, plus statutes and regulatory developments.
  • Main analysis: distinguishes interpretability, transparency, and explainability; reviews evidence on interpretable versus black box accuracy; and applies due process, confrontation, equal protection, discovery, and expert-evidence principles to argue for a right to interpretable forensic AI.

Limitations

  • Stated: the authors acknowledge that future noninterpretable AI systems may show substantial, demonstrated performance advantages, and that black box use might be justified by a compelling interest such as national security, though it should still be independently vetted.
  • Stated: constitutional rights may be insufficient because they are unevenly enforced in criminal cases, indigent defendants face barriers to obtaining experts and discovery, and no U.S. legislative enactments or proposals have required open or glass box forensic evidence.
  • Evident from the text: the article presents no new empirical experiment, dataset, or quantitative meta-analysis; its claims about accuracy rely on cited studies and selected legal examples.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1073/pnas.2301842120

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

9 Valid · 18 Not valid · 33 Not checked · 51 Own results and interpretations

33 more claims · 2 with their cited sources checked

Create a free account to see all 111 claims and to verify the 4 references still open.

See all claims — free account
  • Probably not a substantial claim. 54
  • It may state two separate claims as one. 41

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 35 references

Research works printed but not linked: 4

  1. H. B. Dixon, Artificial intelligence: Benefits and unknown risks. ABA J. 60, 1 (2021).
  2. B. L. Garrett, C. Rudin, The right to a glass box: Rethinking the use of AI in criminal cases. Cornell Law Rev. 109 (forthcoming).
  3. T. Z. Zarsky, Transparent predictions. Univ. Ill. Law Rev. 2013, 1503, 1520 (2013).
  4. A. Roth, Trial by machine. Georgetown Law J. 104, 1245, 1300 (2016).

Cited by

See all 68

Metadata

Authors
Garrett, B. L., & Rudin, C.
Affiliations
Duke University (US)
Corresponding author
Garrett, Brandon L.
Venue
Proceedings of the National Academy of Sciences
Published
2023-10-02
Volume
120
Issue
41
Pages
e2301842120-e2301842120
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
OpenAlex
W4387246019
Cited by
68
References
37
Citations by year
2026: 30 · 2025: 17 · 2024: 18 · 2023: 3
FWCI
4.84
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Iowa State University Digital Repository (Iowa State University) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Law › Artificial Intelligence in Law
Keywords
explainable AI, black-box algorithms, forensic evidence, DNA mixture interpretation, face recognition, recidivism risk assessment, AI legislation
MeSH terms
Algorithms, Artificial Intelligence, Forensic Medicine, Humans, Law Enforcement, Criminals

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Interpretable algorithmic forensics. https://app.cochat.ai/papers/10.1073/pnas.2301842120
The paper
Garrett, B. L., & Rudin, C. (2023). Interpretable algorithmic forensics. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2301842120

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.