Transparency, Governance and Regulation of Algorithmic Tools Deployed in the Criminal Justice System: a UK Case Study
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
We present a survey of tools used in the criminal justice system in the UK in three categories: data infrastructure, data analysis, and risk prediction. Many tools are currently in deployment, offering potential benefits, including improved efficiency and consistency. However, there are also important concerns. Transparent information about these tools, their purpose, how they are used, and by whom is difficult to obtain. Even when information is available, it is often insufficient to enable a satisfactory evaluation. More work is needed to establish governance mechanisms to ensure that tools are deployed in a transparent, safe and ethical way. We call for more engagement with stakeholders and greater documentation of the intended goal of a tool, how it will achieve this goal compared to other options, and how it will be monitored in deployment. We highlight additional points to consider when evaluating the trustworthiness of deployed tools and make concrete proposals for policy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“However, algorithms can behave in fragile, hard to predict ways, such that small changes in input features can lead to unexpectedly large changes in model outputs [72].”
BackgroundLum, K., & Isaac, W. (2016) · Significance · doi:10.1111/j.1740-9713.2016.00960.x · Cited by 611 · cited 1×
“We note the use of similar tools in the US, particularly Perdpol , which was also trialled in the UK by Kent Police and the Metropolitan Police and was criticised for potential racial bias [27, 56].”
MethodThe Australian Human Rights Commission
Santow, E. (2019) · The Legal Protection of Rights in Australia · doi:10.5040/9781509919857.ch-007 · Cited by 592
Oswald, M., Grace, J., Urwin, S., & Barnes, G. C. (2018) · Information & Communications Technology Law · doi:10.1080/13600834.2018.1458455 · Cited by 247 · cited 3×
“Qualifying individuals are offered a four month intervention program tailored to their needs and are not charged if they successfully complete it [39, 67].”
BackgroundMethodEthics of Data and Analytics: Concepts and Cases
Martin, K. (2022) · Auerbach Publications eBooks · doi:10.1201/9781003278290 · Cited by 246
Flores, A. W., Bechtel, K., & Lowenkamp, C. T. (2016) · Federal probation · Cited by 240
Transparency: Motivations and Challenges
Weller, A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_2 · Cited by 198 · cited 1×
“Ensuring this requirement is met can prove more 13 For further discussion of the challenges surrounding algorithmic transparency, see [7, 10, 82] difficult when the tool is proprietary and is supplied by an external company, which is often the case.”
BackgroundSelf-citationDisparate Impact in Big Data Policing
Selbst, A. D. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2819182 · Cited by 192 · cited 1×
“Fairness: There is extensive literature reporting bias against minority groups within algorithms deployed in the criminal justice system [9, 58].”
BackgroundStatistical Procedures for Forecasting Criminal Behavior: A Comparative Assessment
Berk, R. A., & Bleich, J. (2013) · Criminology & Public Policy · doi:10.1111/1745-9133.12047 · Cited by 180
The four dimensions of contestable AI diagnostics - A patient-centric approach to explainable AI
Ploug, T., & Holm, S. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101901 · Cited by 148 · cited 1×
“Contestability: The ability to contest decisions made or assisted by an automated system is an important safeguard in the trustworthy deployment of algorithmic tools [28, 31, 41].”
Background
Research works printed but not linked: 20
- Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . 2016. Machine bias . In Ethics of Data and Analytics . Auerbach Publications , 254--264. Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. In Ethics of Data and Analytics. Auerbach Publications, 254--264.
- Tao Cheng . 2021. Crime, Policing and Citizenship (CPC) - Space-Time Interactions of Dynamic Networks . https://gtr.ukri.org/projects?ref=EP%2FJ004197%2F1 Accessed : 2021 . Tao Cheng. 2021. Crime, Policing and Citizenship (CPC) - Space-Time Interactions of Dynamic Networks. https://gtr.ukri.org/projects?ref=EP%2FJ004197%2F1 Accessed: 2021.
- Alexandra Chouldechova . 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5, 2 ( 2017 ), 153--163. Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5, 2 (2017), 153--163.
- Diane Coyle and Adrian Weller . 2020. "Explaining" machine learning reveals policy challenges. Science 368, 6498 ( 2020 ), 1433--1434. Diane Coyle and Adrian Weller. 2020. "Explaining" machine learning reveals policy challenges. Science 368, 6498 (2020), 1433--1434.
- Danielle Ensign , Sorelle A Friedler , Scott Neville , Carlos Scheidegger , and Suresh Venkatasubramanian . 2018 . Runaway feedback loops in predictive policing . In Conference on Fairness, Accountability and Transparency. PMLR, 160--171 . Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. 2018. Runaway feedback loops in predictive policing. In Conference on Fairness, Accountability and Transparency. PMLR, 160--171.
- Seena Fazel , Matthias Burghart , Thomas Fanshawe , Sharon Danielle Gil , John Monahan, and Rongqin Yu. 2021 . The predictive performance of criminal risk assessment tools used at sentencing. Seena Fazel, Matthias Burghart, Thomas Fanshawe, Sharon Danielle Gil, John Monahan, and Rongqin Yu. 2021. The predictive performance of criminal risk assessment tools used at sentencing.
- Seena Fazel and Achim Wolf . 2018. Selecting a risk assessment tool to use in practice: a 10-point guide. Evidence-based mental health 21, 2 ( 2018 ), 41--43. Seena Fazel and Achim Wolf. 2018. Selecting a risk assessment tool to use in practice: a 10-point guide. Evidence-based mental health 21, 2 (2018), 41--43.
- Centre for Data Ethics and Innovation. 2021. Review into bias in algorithmic decision-making. Centre for Data Ethics and Innovation. 2021. Review into bias in algorithmic decision-making.
- Fieke Jansen . 2018. Data Driven Policing in the Context of Europe. Data Justice Lab ( 2018 ). Fieke Jansen. 2018. Data Driven Policing in the Context of Europe. Data Justice Lab (2018).
- Deirdre K Mulligan , Daniel Kluttz , and Nitin Kohli . 2019. Shaping our tools: Contestability as a means to promote responsible algorithmic decision making in the professions. Available at SSRN 3311894 ( 2019 ). Deirdre K Mulligan, Daniel Kluttz, and Nitin Kohli. 2019. Shaping our tools: Contestability as a means to promote responsible algorithmic decision making in the professions. Available at SSRN 3311894 (2019).
- The National Police Chiefs Council (NPCC) , Association of Police, and Crime Commissioners . 2020 . National Policing Digital Strategy 2020--2030. https://www.apccs.police.uk/media/4886/national-policing-digital-strategy-2020--2030.pdf The National Police Chiefs Council (NPCC), Association of Police, and Crime Commissioners. 2020. National Policing Digital Strategy 2020--2030. https://www.apccs.police.uk/media/4886/national-policing-digital-strategy-2020--2030.pdf
- ACRO Criminal Records Office . [n. d.]. Police National Computer Services. https://www.acro.police.uk/PNC-services Accessed : 2022 . ACRO Criminal Records Office. [n. d.]. Police National Computer Services. https://www.acro.police.uk/PNC-services Accessed: 2022.
- Frank Pasquale . 2019. The Black Box Society . Harvard University Press , Cambridge, MA . Frank Pasquale. 2019. The Black Box Society. Harvard University Press, Cambridge, MA.
- The Metropolitan Police . [n. d.]. Gangs violence matrix. https://www.met.police.uk/police-forces/metropolitan-police/areas/about-us/about-the-met/gangs-violence-matrix/ Accessed : 2021 . The Metropolitan Police. [n. d.]. Gangs violence matrix. https://www.met.police.uk/police-forces/metropolitan-police/areas/about-us/about-the-met/gangs-violence-matrix/ Accessed: 2021.
- West Yorkshire Police . [n. d.]. National Data Analytics Solution (NDAS) Privacy Notice. https://www.westyorkshire.police.uk/advice/modern-slavery/national-data-analytics-solution-ndas-privacy-notice Accessed : 2022 . West Yorkshire Police. [n. d.]. National Data Analytics Solution (NDAS) Privacy Notice. https://www.westyorkshire.police.uk/advice/modern-slavery/national-data-analytics-solution-ndas-privacy-notice Accessed: 2022.
- Predpol. [n. d.]. Kent Police Use PredPol To Prevent Violent Crime. https://www.predpol.com/kent-police-use-predpol-to-prevent-violent-crime/ Accessed : 2022 . Predpol. [n. d.]. Kent Police Use PredPol To Prevent Violent Crime. https://www.predpol.com/kent-police-use-predpol-to-prevent-violent-crime/ Accessed: 2022.
- Edward Santow . 2021 . Intelligence: Coding calues into the future . In The Public Square Project, Peter Lewis and Jordan Guiao (Eds.). Melbourne University Press , Melbourne . Edward Santow. 2021. Intelligence: Coding calues into the future. In The Public Square Project, Peter Lewis and Jordan Guiao (Eds.). Melbourne University Press, Melbourne.
- Andrew Selbst . 2018 . Disparate Impact in Big Data Policing . Georgia Law Review 52 , 1 (2018), 3373 . Andrew Selbst. 2018. Disparate Impact in Big Data Policing. Georgia Law Review 52, 1 (2018), 3373.
- Daniel Sprick . 2019. Predictive Policing in China: An Authoritarian Dream of Public Security. Naveiñ Reet: Nordic Journal of Law and Social Research (NNJLSR) No 9 ( 2019 ). Daniel Sprick. 2019. Predictive Policing in China: An Authoritarian Dream of Public Security. Naveiñ Reet: Nordic Journal of Law and Social Research (NNJLSR) No 9 (2019).
- Niche Technology . [n. d.]. Niche RMS Customer Profiles. https://nicherms.com/region/uk/ Accessed : 2021 . Niche Technology. [n. d.]. Niche RMS Customer Profiles. https://nicherms.com/region/uk/ Accessed: 2021.
Cited by
OECD (2025) · OECD eBooks · doi:10.1787/795de142-en · Cited by 324
Artificial intelligence in government: Concepts, standards, and a unified framework
Straub, V. J., Morgan, D., Bright, J., & Margetts, H. Z. (2023) · Government Information Quarterly · doi:10.1016/j.giq.2023.101881 · Cited by 175
Harms from Increasingly Agentic Algorithmic Systems
Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594033 · Cited by 149
“extensively studied the ways in which automated decision-making can disempower individuals, by impairing human decision-making [Green and Chen 2019; Green 2021] or subjecting individuals to oppressive institutions [Barabas et al. 2020, 2018; Green 2020; Zilka et al. 2022].”
BackgroundInfluentialSelf-citationChan, A., Ezell, C., Kaufmann, M., Wei, K., Hammond, L., Bradley, H., Bluemke, E., Rajkumar, N., Krueger, D., Kolt, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658948 · Cited by 84
de Jong, S., Paananen, V., Tag, B., & van Berkel, N. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710946 · Cited by 32
Metadata
- Authors
- Zilka, M., Sargeant, H., & Weller, A.
- Affiliations
- University of Cambridge (GB)
- Venue
- Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2022-07-26
- Pages
- 880-889
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W4281799859
- Cited by
- 28
- References
- 15
- Citations by year
- 2026: 3 · 2025: 11 · 2024: 8 · 2023: 6
- FWCI
- 6.21
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Apollo (cc-by)
- Funders
- General Sir John Monash Foundation
- EPSRC (EP/V025279/1)
- Leverhulme Trust (ECF-2021-429)
- The Alan Turing Institute
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Artificial Intelligence in Law
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- algorithmic tools, criminal justice system, algorithmic transparency, governance, regulation, risk prediction, trustworthiness, stakeholder engagement, algorithmic accountability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Transparency, Governance and Regulation of Algorithmic Tools Deployed in the Criminal Justice System: a UK Case Study. https://app.cochat.ai/papers/10.1145/3514094.3534200
- The paper
- Zilka, M., Sargeant, H., & Weller, A. (2022). Transparency, Governance and Regulation of Algorithmic Tools Deployed in the Criminal Justice System: a UK Case Study. Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society. https://doi.org/10.1145/3514094.3534200
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
