CoChat

Is that your final decision? Multi-stage profiling, selective effects, and Article 22 of the GDPR

Open access
Binns, Reuben, Veale, Michael · 2021 · International Data Privacy Law

Published version of a preprint

The DOI matches its Crossref record; none of the 39 sources its claims cite is checked for support yet.

Abstract

Provisions in many data protection laws require a legal basis, or at the very least safeguards, for significant, solely automated decisions; Article 22 of the GDPR is the most notable. Little attention has been paid to Article 22 in light of decision-making processes with multiple stages, potentially both manual and automated, and which together might impact upon decision subjects in different ways. Using stylized examples grounded in real-world systems, we raise five distinct complications relating to interpreting Article 22 in the context of such multi-stage profiling systems. These are: the potential for selective automation on subsets of data subjects despite generally adequate human input; the ambiguity around where to locate the decision itself; whether ‘significance’ should be interpreted in terms of any ‘potential’ effects or only selectively in terms of ‘realised’ effects; the potential for upstream automation processes to foreclose downstream outcomes despite human input; and that a focus on the final step may distract from the status and importance of upstream processes. We argue that the nature of these challenges will make it difficult for courts or regulators to distil a set of clear, fair, and consistent interpretations for many realistic contexts. Data protection law provides a set of rights and obligations in relation to the processing of personal data. Amongst its substantive principles, such as lawfulness, fairness, transparency, and various procedural elements and risk-based measures that apply to personal data processing, it also addresses the use of automated decision-making systems. Provisions around automated decisions are not new, having been part of the data protection toolbox for several decades.1 The Data Protection Directive 1995 regulated automated decision-making;2 however in practice this was largely theoretical and the rights and obligations therein were rarely invoked. However, this previously dormant corner of data protection has begun to awaken in recent years, partly as a result of the rise of automation in the private and public sector, and partly due to new interest, novel concepts, and safeguards in the General Data Protection Regulation (GDPR).3 Article 22 of the GDPR contains provisions which restrict the use of automated decision making systems where they are ‘solely automated’ and have ‘legal or similarly significant effects’. The exact remedies the provision provides4 as well as their utility5 are subject to a debate that this paper does not seek to engage in. Instead, we focus solely on the scope of Article 22. A similarly worded provision exists in the Law Enforcement Directive, the GDPR’s sister instrument for police and criminal justice.6 Furthermore, many other laws internationally share a similar or near identical construction to Article 22.7 Whilst our analysis focuses on the GDPR to study this provision, the issues we raise are relevant more broadly. The relevant section of Article 22, Article 22(1) reads ‘The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.’ Recital 71, which comments on Article 22, reads (in its relevant section) ‘The data subject should have the right not to be subject to a decision, which may include a measure, evaluating personal aspects relating to him or her which is based solely on automated processing and which produces legal effects concerning him or her or similarly significantly affects him or her, such as automatic refusal of an online credit application or e-recruiting practices without any human intervention. Such processing includes “profiling” that consists of any form of automated processing of personal data evaluating the personal aspects relating to a natural person, in particular to analyse or predict aspects concerning the data subject's performance at work, economic situation, health, personal preferences or or or where it produces legal effects concerning him or her or similarly significantly affects him or her.’ in these provisions have been in and The of of the has not been to any concerning Article 22 or its in the Data Protection The ambiguity of to in this is the nature of it is to a decision on automated other of human or is to an automated decision not solely The most significant that exists on this from the Data Protection to several including that the human should be in a to the and the of the and not they should have the to the automated and they should and are generally in this with the of of private in the of the and where the for decisions to be on automated processing in for safeguards to apply is The ambiguity of decisions have a on the data the examples of credit and in Recital of the has with several other examples of decisions effects as significant, including which from or is on a set of relating to human and These are which

Overview

CoChat Summary

TL;DR

This paper examines how Article 22 of the GDPR applies to multi-stage profiling systems where automation and human involvement, and the significance of effects, vary across stages or outcomes. It identifies five complications that make it difficult for courts or regulators to produce clear, fair, and consistent interpretations of Article 22 in realistic contexts.

Key findings

  • The paper raises five distinct complications: selective automation on subsets of data subjects despite generally adequate human input; ambiguity around where to locate the decision itself; whether “significance” should be interpreted in terms of any “potential” effects or only “realised” effects; the potential for upstream automation to foreclose downstream outcomes despite human input; and the risk that focusing on the final step misleads as to the status of upstream processes.
  • It distinguishes three roles automation can play in multi-stage profiling: “Supporting” (providing information to a decision-maker), “Triaging” (determining which cases reach a human or another automated process), and “Summarizing” (consolidating human decisions).
  • The paper argues that none of the complications have easy answers without drifting radically from the set-up of Article 22, and that courts or regulators will find it difficult to distil clear, fair, and consistent interpretations within multi-level profiling systems.
  • It suggests the “potential” approach to significance—whether a decision mechanism is reasonably foreseeable as significant for some individuals—makes more sense than a purely “realised” approach, while noting this can cast the net too wide and create practical problems.

Methods

  • Conceptual and legal analysis of Article 22 of the GDPR and similarly worded provisions, using stylized examples grounded in real-world systems such as COMPAS, the TSA Secondary Security Screening Selection list, the Allegheny Family Screening Tool, UK Visas and Immigration screening, anti-money-laundering detection, loan assessment, and automated summarization of human scores.
  • No empirical dataset or statistical analysis is used; the paper draws on legal texts (GDPR Article 22 and Recital 71, Law Enforcement Directive Article 11), EDPB/Article 29 Working Party guidance, CJEU cases, and prior literature.
  • The analysis organizes multi-stage profiling into supporting, triaging, and summarising structures, then examines five challenges and possible ways forward.

Limitations

  • The paper states it does not engage in the debate over the exact remedies Article 22 provides or their utility, focusing solely on the scope of Article 22.
  • The paper states it does not consider interactions between decisions over time, such as through retraining or updating a model, which may create important feedback effects.
  • Evident from the text: the analysis is conceptual and illustrative rather than empirical, and the examples are stylized; the paper also notes the CJEU has not yet been asked to interpret Article 22 or its predecessor.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/idpl/ipab020

Cites

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 0 Not valid · 32 Not checked · 57 Own results and interpretations

25 more claims

Create a free account to see all 89 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 51
  • It may state two separate claims as one. 22

Create a free account to see these claims.

Citations

Cited by

Metadata

Authors
Binns, R., & Veale, M.
Affiliations
  • University of Oxford (GB)
  • University College London (GB)
Corresponding author
Veale, Michael
Venue
International Data Privacy Law
Published
2021-10-20
Volume
11
Issue
4
Pages
319-332
ISSN
  • 2044-3994 (ISSN-L)
  • 2044-4001
OpenAlex
W3206170428
Cited by
65
References
0
Citations by year
2026: 16 · 2025: 17 · 2024: 16 · 2023: 6 · 2022: 9 · 2021: 1
FWCI
52.16
Type
journal article
Language
en
Open access
Locations
  • International Data Privacy Law (publishedVersion, cc-by)
  • UCL Discovery (University College London) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • SocArXiv (OSF Preprints) (submittedVersion, cc-by-nd)
  • SocArXiv (OSF Preprints) (submittedVersion)
Funders
  • Engineering and Physical Sciences Research Council (EP/S035362/1)
  • Digital Charter Fellowship from the Alan Turing Institute and Department for Digital
Topics
  • Social Sciences › Law › Digitalization, Law, and Regulation
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Business, Management and Accounting › Industrial relations › Legal and Labor Studies
Keywords
GDPR Article 22, automated decision-making, human oversight, personal data protection, Law Enforcement Directive, algorithmic decision-making, human-in-the-loop systems, data protection impact assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.