CoChat
Sign Up
Cited by 22Ethics and Social Impacts of AIResearch Data Management PracticesData Analysis and Archiving

A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication

Open access
Luccioni, Alexandra Sasha, Corry, Frances, Sridharan, Hamsini, Ananny, Mike, Schultz, Jason, Crawford, Kate·2022·2022 ACM Conference on Fairness Accountability and Transparency
DOI 10.1145/3531146.3533086
Open this paper in CoChat — free accountOpen PDF

Abstract

Datasets are central to training machine learning (ML) models. The ML community has recently made significant improvements to data stewardship and documentation practices across the model development life cycle. However, the act of deprecating, or deleting, datasets has been largely overlooked, and there are currently no standardized approaches for structuring this stage of the dataset life cycle. In this paper, we study the practice of dataset deprecation in ML, identify several cases of datasets that continued to circulate despite having been deprecated, and describe the different technical, legal, ethical, and organizational issues raised by such continuations. We then propose a Dataset Deprecation Framework that includes considerations of risk, mitigation of impact, appeal mechanisms, timeline, post-deprecation protocols, and publication checks that can be adapted and implemented by the ML community. Finally, we propose creating a centralized, sustainable repository system for archiving datasets, tracking dataset modifications or deprecations, and facilitating practices of care and stewardship that can be integrated into research and publication processes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3531146.3533086

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Data provenance frameworks developed in AI/ML have incorporated perspectives from libraries and archives, feminist ethics, and the food industry.

    “Data provenance frameworks developed in the context of AI and ML have incorporated perspectives from libraries and archives, feminist ethics and even the food industry (Holland et al. 2020;Jo and Gebru 2020;Luccioni et al. 2022).”

    From Can AI provenance inform archival provenance? (2026) · cites this paper as Luccioni et al. 2022

    Section 2 (p. 3) connects dataset stewardship with feminist philosophies of care, and Section 5.3 (p. 14) proposes collaboration with university libraries on dataset documentation. The paper does not say its framework draws on the food industry, nor does it establish that provenance frameworks incorporated perspectives from libraries and archives as a field. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Documenting where data come from can serve technical goals, for instance, so that researchers are able to monitor drift and contamination in datasets.

    “Documenting where data come from can also serve technical goals, for instance, so that researchers are able to monitor drift and contamination in datasets (Luccioni et al. 2022).”

    From Can AI provenance inform archival provenance? (2026) · cites this paper as Luccioni et al. 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 42 references couldn’t be matched to a known paper.

Research works printed but not linked: 8

  1. [6][6] Jack Bandy and Nicholas Vincent. 2021. Addressing" Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus. arXiv preprint arXiv:2105.05241 (2021). - · arXiv
  2. [8][8] Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In 9th Annual Conference of the Special Interest Group for Computing, Information and Society . -
  3. [10][10] Thomas Baumhauer, Pascal Schöttle, and Matthias Zeppelzauer. 2020. Machine unlearning: Linear filtration for logit-based classifiers. arXiv preprint arXiv:2002.02730 (2020). - · arXiv
  4. [11][11] Misha Benjamin, Paul Gagnon, Negar Rostamzadeh, Chris Pal, Yoshua Bengio, and Alex Shee. 2019. Towards standardization of data licenses: The montreal data license. arXiv preprint arXiv:1903.12262 (2019). - · arXiv
  5. [15][15] Geoffrey C Bowker and Susan Leigh Star. 2000. Sorting things out: Classification and its consequences . MIT press. -
  6. [19][19] Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency . PMLR, 77-91. -
  7. [20][20] Dan L Burk. 2019. Algorithmic fair use. The University of Chicago Law Review 86 (2019), 283. -
  8. [22][22] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2020. Extracting training data from large language models. arXiv preprint arXiv:2012.07805 (2020). - · arXiv

Cited by

Sorted by citations

Related

Authors
Luccioni, A. S., Corry, F., Sridharan, H., Ananny, M., Schultz, J., & Crawford, K.
Affiliations
  • Hugging Face (US)
  • University of Southern California (US)
  • New York University (US)
ORCID
Venue
2022 ACM Conference on Fairness Accountability and Transparency
Publisher
ACM
Published
2022-06-20
Pages
199-212
DOI
10.1145/3531146.3533086
OpenAlex
W4283156082
Cited by
22
References
42
Citations by year
2026: 5 · 2025: 4 · 2024: 7 · 2023: 5 · 2022: 1
FWCI
4.22
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Alfred. P Sloan Foundation
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
  • Social Sciences › Sociology and Political Science › Data Analysis and Archiving
Keywords
data stewardship, dataset documentation, data archiving, data repositories, machine learning data, data ethics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review