CoChat
Sign Up
PreprintCited by 175Ethics and Social Impacts of AIData Quality and ManagementResearch Data Management Practices

The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards

Open access
Holland, Sarah, Hosny, Ahmed, Newman, Sarah, Joseph, Joshua, Chmielinski, Kasia·2018·arXiv
DOI 10.48550/arxiv.1805.03677arXiv 1805.03677
Open this paper in CoChat — free accountOpen PDF

Abstract

Artificial intelligence (AI) systems built on incomplete or biased data will often exhibit problematic outcomes. Current methods of data analysis, particularly before model development, are costly and not standardized. The Dataset Nutrition Label (the Label) is a diagnostic framework that lowers the barrier to standardized data analysis by providing a distilled yet comprehensive overview of dataset "ingredients" before AI model development. Building a Label that can be applied across domains and data types requires that the framework itself be flexible and adaptable; as such, the Label is comprised of diverse qualitative and quantitative modules generated through multiple statistical and probabilistic modelling backends, but displayed in a standardized format. To demonstrate and advance this concept, we generated and published an open source prototype with seven sample modules on the ProPublica Dollars for Docs dataset. The benefits of the Label are manyfold. For data specialists, the Label will drive more robust data analysis practices, provide an efficient way to select the best dataset for their purposes, and increase the overall quality of AI models as a result of more robust training datasets and the ability to check for issues at the time of model development. For those building and publishing datasets, the Label creates an expectation of explanation, which will drive better data collection practices. We also explore the limitations of the Label, including the challenges of generalizing across diverse datasets, and the risk of using "ground truth" data as a comparison dataset. We discuss ways to move forward given the limitations identified. Lastly, we lay out future directions for the Dataset Nutrition Label project, including research and public policy agendas to further advance consideration of the concept.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1805.03677

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recently, researchers have proposed standardized forms of communicating characteristics of datasets used in machine learning.

    “Recently, researchers have proposed standardized forms of communicating characteristics of datasets used in machine learning [3, 21, 28] to help users understand the context in which the datasets should be used.”

    From Model Cards for Model Reporting (2019) · cites this paper as [28]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 23

  1. [1]1. [Davenport TH, Harris JG. Automated decision making comes of age. MIT Sloan Management Review. Massachusetts Institute of Technology, Cambridge, MA; 2005;46: 83.](http://paperpile.com/b/JgfICH/bCfK)
  2. [2]2. [Skeem JL, Lowenkamp C. Risk, Race, & Recidivism: Predictive Bias and Disparate Impact. 2016; doi: 10.2139/ssrn.2687339](http://paperpile.com/b/JgfICH/irmm) · DOI
  3. [3]3. [Corbett-Davies S, Pierson E, Feller A, Goel S, Huq A. Algorithmic Decision Making and the Cost of Fairness. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY, USA: ACM; 2017. pp. 797-806.](http://paperpile.com/b/JgfICH/sVYQ)
  4. [5]5. [Buolamwini J, Gebru T. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In: Friedler SA, Wilson C, editors. Proceedings of the 1st Conference on Fairness, Accountability and Transparency. New York, NY, USA: PMLR; 2018. pp. 77-91.](http://paperpile.com/b/JgfICH/GLEY)
  5. [8]8. [Union E. Directive 2009/28/EC of the European Parliament and of the Council of 23 April 2009 on the promotion of the use of energy from renewable sources and amending and subsequently repealing Directives 2001/77/EC and 2003/30/EC. Official Journal of the European Union. 2009;5: 2009.](http://paperpile.com/b/JgfICH/lWsW)
  6. [11]11. [Balasubramanian SK, Cole C. Consumers' search and use of nutrition information: The challenge and promise of the nutrition labeling and education act. J Mark. American Marketing Association; 2002;66: 112-127.](http://paperpile.com/b/JgfICH/xW42)
  7. [12]12. [Moorman C. A Quasi Experiment to Assess the Consumer and Informational Determinants of Nutrition Information Processing Activities: The Case of the Nutrition Labeling and Education Act. Journal of Public Policy & Marketing. American Marketing Association; 1996;15: 28-44.](http://paperpile.com/b/JgfICH/PsiT)
  8. [13]13. [Guthrie JF, Fox JJ, Cleveland LE, Welsh S. Who uses nutrition labeling, and what effects does label use have on diet quality? J Nutr Educ. 1995;27: 163-172.](http://paperpile.com/b/JgfICH/LKc2)
  9. [15]15. [Future. Journal of Public Policy & Marketing. American Marketing Association; 1996;15: 148-150.](http://paperpile.com/b/JgfICH/2iLY)
  10. [15]15. [Petruccelli PJ. Consumer and Marketing Implications of Information Provision: The Case of the Nutrition Labeling and Education Act of 1990. Journal of Public Policy & Marketing. American Marketing Association; 1996;15: 150-153.](http://paperpile.com/b/JgfICH/eV6p)
  11. [16]16. [Teisl MF, Levy AS, Others. Does nutrition labeling lead to healthier eating? Journal of Food Distribution Research. FDRS FOOD DISTRIBUTION RESEARCH SOCIETY; 1997;28: 18-27.](http://paperpile.com/b/JgfICH/6yT5)
  12. [17]17. [Drichoutis AC, Lazaridis P, Nayga RM Jr. Consumers' use of nutritional labels: a review of research studies and issues. Academy of marketing science review. Academy of Marketing Science Review; 2006;2006: 1.](http://paperpile.com/b/JgfICH/wtbj)
  13. [19]19. [Borra S. Consumer perspectives on food labels--. Am J Clin Nutr. Oxford University Press; 2006;83: 1235S-1235S.](http://paperpile.com/b/JgfICH/WjMV)
  14. [20]20. [Burton S, Biswas A, Netemeyer R. Effects of Alternative Nutrition Label Formats and Nutrition Reference Information on Consumer Perceptions, Comprehension, and Product Evaluations. Journal of Public Policy & Marketing. American Marketing Association; 1994;13: 36-47.](http://paperpile.com/b/JgfICH/GueP)
  15. [21]21. [Ciocchetti CA. The future of privacy policies: A privacy nutrition label filled with fair information practices. John Marshall J Comput Inf Law. HeinOnline; 2008;26: 1.](http://paperpile.com/b/JgfICH/dNAS)
  16. [22]22. [Kelley PG, Bresee J, Cranor LF, Reeder RW. A nutrition label for privacy. Proceedings of the 5th Symposium on Usable Privacy and Security. ACM; 2009. p. 4.](http://paperpile.com/b/JgfICH/efEo)
  17. [23]23. [Kelley PG, Cesca L, Bresee J, Cranor LF. Standardizing Privacy Notices: An Online Study of the Nutrition Label Approach. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, NY, USA: ACM; 2010. pp. 1573-1582.](http://paperpile.com/b/JgfICH/jgrF)
  18. [25]25. [Gebru T, Morgenstern J, Vecchione B, Vaughan JW, Wallach H, Iii HD, et al. Datasheets for Datasets [Internet]. arXiv [cs.DB]. 2018. Available: http://arxiv.org/abs/1803.09010](http://paperpile.com/b/JgfICH/EWZx) · arXiv
  19. [27]27. [Hunt A, Thomas D. The pragmatic programmer: from journeyman to master. Addison-Wesley Professional; 2000.](http://paperpile.com/b/JgfICH/D3jT)
  20. [32]32. [Mansinghka V, Tibbetts R, Baxter J, Shafto P, Eaves B. BayesDB: A probabilistic programming system for querying the probable implications of data [Internet]. arXiv [cs.AI]. 2015. Available: http://arxiv.org/abs/1512.05006](http://paperpile.com/b/JgfICH/RDMs) · arXiv
  21. [33]33. [Berger JO. Statistical Decision Theory and Bayesian Analysis. Springer Science & Business Media; 2013.](http://paperpile.com/b/JgfICH/4WdQ)
  22. [34]34. [Surendra H MHS. A Review Of Synthetic Data Generation Methods For Privacy Preserving Data Publishing. INTERNATIONAL JOURNAL OF SCIENTIFIC & TECHNOLOGY RESEARCH. 2017;6. Available:](http://paperpile.com/b/JgfICH/hoyj)
  23. [5]5. [http://www.ijstr.org/final-print/mar2017/A-Review-Of-Synthetic-Data-Generation-Methods-For-Priv acy-Preserving-Data-Publishing.pdf](http://www.ijstr.org/final-print/mar2017/A-Review-Of-Synthetic-Data-Generation-Methods-For-Privacy-Preserving-Data-Publishing.pdf)

Cited by

Sorted by citations

Authors
Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K.
Affiliations
Harvard University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1805.03677
arXiv
1805.03677
OpenAlex
W2802642435
Cited by
175
References
2
Citations by year
2026: 1 · 2025: 8 · 2024: 13 · 2023: 26 · 2022: 21 · 2021: 48 · 2020: 35 · 2019: 21 · 2018: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Data Quality and Management
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
Keywords
dataset quality, data quality standards, data bias, data completeness, data collection practices

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review