Datasheets for datasets
Open accessThe DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index.
Abstract
Documentation to facilitate communication between dataset creators and consumers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Paradata needs may include the preservation of novel forms of information, such as datasheets explaining training datasets or model cards documenting an AI tool, and of these documents' relation to archival collections processed using them [ 26 ].”
From Positioning Paradata: A Conceptual Frame for AI Processual Documentation in Archives and Recordkeeping Contexts (2023) · cites this paper as [26]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Significant critiques of LLMs have been raised, including concerns about bias and fairness, misinformation, opaqueness due to a lack of interpretability and explainability, and environmental impact (Bender et al., 2021; Gebru et al., 2021; Strubell et al., 2019).”
From Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments (2023) · cites this paper as Gebru et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“If ML driven algorithms are considered as epistemic resources that help us to improve understanding and/or make better predictions, then the statistical nature of ML de facto prevents groups underrepresented in the data from contributing to this epistemic resource on an equal ground to dominant groups (Gebru et al., 2021)”
From Algorithmic profiling as a source of hermeneutical injustice (2025) · cites this paper as Gebru et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The authors found that providing examples of what the requirements can look like helps to guide practitioners in clarifying their own rules.”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [24]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“I show how archival theories have been animated in documenting data in the context of AI and ML, including through documentation frameworks like Datasheets for Datasets”
From Can AI provenance inform archival provenance? (2026) · cites this paper as Gebru et al. 2021
12 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments
Huang, G. B., Mattar, M., Berg, T. L., & Miller, E. G. (2008) · HAL · Cited by 4,822
Minimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109
(2018) · Cited by 3,510
Self-citationA sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts
Pang, B., & Lee, L. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1218990 · Cited by 3,301
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“Researchers at Google published follow-up work on model cards that document machine learning models [20] and released a data card (a lightweight version of a datasheet) along with the Open Images dataset [17].”
BackgroundSelf-citationMan is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · cited 2×
“Of particular concern are recent examples showing that machine learning models can reproduce or amplify unwanted societal biases reflected in training data [4, 5, 9].”
BackgroundThe Effects of Automobile Safety Regulation
Peltzman, S. (1975) · Journal of Political Economy · doi:10.1086/260352 · Cited by 1,531
The Recent Past and Near Future of Risk and/or Need Assessment
Andrews, D. A., Bonta, J., & Wormith, J. S. (2006) · Crime & Delinquency · doi:10.1177/0011128705281756 · Cited by 1,507 · cited 1×
“Domains such as criminal justice (Garvie et al., 2016; Systems, 2017; Andrews et al., 2006), hiring and employment (Mann & O’Neil, 2016), critical infrastructure (O’Connor, 2017; Chui, 2017), and finance (Lin, 2012) all increasingly depend on machine learning methods.”
Background
Research works printed but not linked: 7
- Chui, G. Project will use AI to prevent or minimize electric grid failures, 2017.
- Garvie, C., Bedoya, A., and Frankle, J. The Perpetual Line-Up: Unregulated Police Face Recognition in America. Georgetown Law, Center on Privacy & Technology, Washington, D.C., 2016.
- Holstein, K., Vaughan, J., Daumé, H, Dudík, M., and Wallach, H. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of 2019 ACM CHI Conf. on Human Factors in Computing Systems.
- Krasin, I. et al. OpenImages: A public dataset for large-scale multi-label and multi-class image classification, 2017.
- O'Connor, M. How AI Could Smarten Up Our Water System, 2017.
- Doha Supply Systems. Facial Recognition, 2017.
- Yagcioglu, S., Erdem, A., Erdem, E., and Ikizler-Cinbis, N. RecipeQA: A challenge dataset for multimodal comprehension of cooking recipes. In Proceedings of the 2018 Conf. on Empirical Methods in Natural Language Processing.
Cited by
Items 101–200 of 1,039
Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases
Steed, R., & Caliskan, A. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445932 · Cited by 271
“Our method can be used to assess task-agnostic biases contained in a dataset to enhance transparency [28, 51], but bias mitigation for unsupervised transfer learning is a challenging open problem.”
MethodUse of artificial intelligence in critical care: opportunities and obstacles
Pinsky, M. R., Bedoya, A., Bihorac, A., Celi, L., Churpek, M., Economou-Zavlanos, N. J., Elbers, P., Saria, S., Liu, V., Lyons, P. G., et al. (2024) · Critical Care · doi:10.1186/s13054-024-04860-z · Cited by 160
“Clinicians must understand the indications and contraindications of relevant clinical AI models, including the ability to interpret and appraise published models and training datasheets associated with a given AI tool across various demographic populations [69, 70].”
BackgroundVision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155
Explainable Artificial Intelligence for Neuroscience: Behavioral Neurostimulation
Fellous, J.-M., Sapiro, G., Rossi, A., Mayberg, H., & Ferrante, M. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.01346 · Cited by 155
“Characterization of the input data can also be done by annotating and standardizing them, using for example documentation approaches such as datasheets (Gebru et al., 2018).”
Legalbench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Guha, N., Nyarko, J., Ho, D. E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D., et al. (2023) · doi:10.2139/ssrn.4583531 · Cited by 154
Fields, J., Chovanec, K., & Madiraju, P. (2024) · IEEE Access · doi:10.1109/access.2024.3349952 · Cited by 246
“Further, [118] have argued that training data must be made open to assess sources of bias, thus dovetailing the concerns over fairness and equity with concerns over intellectual property.”
BackgroundREFORMS: Consensus-based Recommendations for Machine-learning-based Science
Kapoor, S., Cantrell, E. M., Peng, K., Pham, T. H., Bail, C. A., Gundersen, O. E., Hofman, J. M., Hullman, J., Lones, M. A., Malik, M. M., et al. (2024) · Science Advances · doi:10.1126/sciadv.adk3452 · Cited by 129
“This follows recent calls for more granular details about data, such as by Gebru et al. ( 98 ).”
Utility is in the Eye of the User: A Critique of NLP Leaderboards
Ethayarajh, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.393 · Cited by 147
Auditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152
“…2016; Leveson, 2011), law (Laux et al., 2021; Selbst, 2021), media and communication studies (Bandy & Diakopoulos, 2019; Sandvig et al., 2014), social science (Metaxa et al., 2021; Vecchione et al., 2021), philosophy (Boddington, 2017; Dafoe, 2017) and organisational studies (Guszcza et al., 2018).”
BackgroundUnderstanding artificial intelligence ethics and safety
Leslie, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05684 · Cited by 134
Drukker, K., Chen, W., Gichoya, J. W., Gruszauskas, N. P., Kalpathy-Cramer, J., Koyejo, S., Myers, K. J., Sá, R. C., Sahiner, B., Whitney, H. M., et al. (2023) · Journal of Medical Imaging · doi:10.1117/1.jmi.10.6.061104 · Cited by 149
Carobene, A., Padoan, A., Cabitza, F., Banfi, G., & Plebani, M. (2023) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2023-1136 · Cited by 219
“AI models can exhibit and perpetuate biases present in their training data, raising signi fi cant ethical concerns [36].”
The Contestation of Tech Ethics: A Sociotechnical Approach to Technology Ethics in Practice
Green, B. (2021) · Journal of Social Computing · doi:10.23919/jsc.2021.0018 · Cited by 140
“As one senior researcher in a tech company notes, “the ethics system that you create has to be something that people feel adds value and is not a massive roadblock that adds no value, because if it is a roadblock that has no value, people literally will not do it, because they do not have to”[79].”
BackgroundInfluentialMeasuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142
“We follow the recommendations of Gebru et al. (2018) and provide a datasheet for the ETHICS dataset in this section.”
MethodA survey on bias in visual datasets
Fabbrizzi, S., Papadopoulos, S., Ntoutsi, E., & Kompatsiaris, I. Y. (2022) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2022.103552 · Cited by 222
Harms from Increasingly Agentic Algorithmic Systems
Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594033 · Cited by 149
Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics
Wong, R. Y., Madaio, M. A., & Merrill, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579621 · Cited by 135
Z-Inspection ® : A Process to Assess Trustworthy AI
Zicari, R. V., Brodersen, J., Brusseau, J., Dudder, B., Eichhorn, T., Ivanov, T., Kararigas, G., Kringen, P., McCullough, M., Moslein, F., et al. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3066209 · Cited by 134
“data sets [48], [64], [68]–[73], models [74], and services [68].”
BackgroundCooper, A. F., Moss, E., Laufer, B., & Nissenbaum, H. F. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533150 · Cited by 243
“For data-driven algorithmic systems in particular, they have taken the form of annotations [13], audits [5], and frameworks concerning the appropriate use of data and other artifacts, which are often developed and used in the production of AI/ML systems [22, 53, 61, 85, 89, 114].”
BackgroundOperationalising AI governance through ethics-based auditing: an industry case study
Mökander, J., & Floridi, L. (2023) · AI and Ethics · doi:10.1007/s43681-022-00171-7 · Cited by 129
Administration by Algorithm? Public Management Meets Public Sector Machine Learning
Veale, M., & Brass, I. (2019) · Algorithmic Regulation · doi:10.1093/oso/9780198838494.003.0006 · Cited by 105
Representation Bias in Data: A Survey on Identification and Resolution Techniques
Shahbazi, N., Lin, Y., Asudeh, A., & Jagadish, H. V. (2023) · ACM Computing Surveys · doi:10.1145/3588433 · Cited by 132
XAIR: A Systematic Metareview of Explainable AI (XAI) Aligned to the Software Development Process
Clement, T., Kemmerzell, N., Abdelaal, M., & Amberg, M. (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5010006 · Cited by 131
Responsible AI Pattern Catalogue: A Collection of Best Practices for AI Governance and Engineering
Lu, Q., Zhu, L., Xu, X., Whittle, J., Zowghi, D., & Jacquet, A. (2023) · ACM Computing Surveys · doi:10.1145/3626234 · Cited by 271
“In addition to supply chain details of the components, context documents (like model cards [81] for reporting AI models, and datasheets for the datasets [41] used to train AI models) can be integrated to the bill of materials registry.”
MethodRed Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.07858 · Cited by 130
Rethinking Fairness: An Interdisciplinary Survey of Critiques of Hegemonic ML Fairness Approaches
Weinberg, L. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13196 · Cited by 127
“…these “datasheets for datasets” have the potential to “mitigate unwanted biases in machine learning systems, facilitate greater reproducibility of machine learning results, and help researchers and practitioners select more appropriate datasets for their chosen tasks” (Gebru et al., 2021, p. 2).”
BackgroundInfluentialStructural racism in precision medicine: leaving no one behind
Geneviève, L. D., Martani, A., Shaw, D., Elger, B. S., & Wangmo, T. (2020) · BMC Medical Ethics · doi:10.1186/s12910-020-0457-8 · Cited by 147
“Another solution put forward by Gebru and colleagues [54] is the concept of datasheets for datasets, which would help tackle the issues surrounding biases in training datasets for machine learning communities.”
BackgroundInfluentialCorporate Governance of Artificial Intelligence in the Public Interest
Cihon, P., Schuett, J., & Baum, S. D. (2021) · Information · doi:10.3390/info12070275 · Cited by 195
Societal Biases in Language Generation: Progress and Challenges
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.330 · Cited by 131
“Paullada et al. (2020), Bender and Fried-man (2018), and Gebru et al. (2018) provide more comprehensive surveys and frameworks that focus on aspects of data creation and management that could lead to biases, and we refer readers to their works for more discussion.”
Background(2019) · doi:10.1145/3351095.3372862 · Cited by 167
“A number of proposed standards have been created around datasets for ML [2–4, 14, 22, 23, 39], which are often framed as potential ways to mitigate bias and improve transparency and accountability.”
Typology of Risks of Generative Text-to-Image Models
Bird, C., Ungless, E., & Kasirzadeh, A. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604722 · Cited by 121
Understanding accountability in algorithmic supply chains
Cobbe, J., Veale, M., & Singh, J. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594073 · Cited by 255
“Yet organisation-focused tools to provide information on points in the AI lifecycle (such as [39, 75]) are of limited help where information about interconnections be-tween actors is needed.”
BackgroundInfluentialHuman-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2021) · arXiv · doi:10.48550/arxiv.2110.10790 · Cited by 120
“See examples in FactSheets [11] and Datasheets [39]”
MethodWidder, D. G., & Nafus, D. (2023) · Big Data & Society · doi:10.1177/20539517231177620 · Cited by 119
“2019) and datasheets for datasets (Gebru et al. 2021) hold promise for communication between people across modules and organizations.”
BackgroundThe Prompt Report: A Systematic Survey of Prompt Engineering Techniques
Schulhoff, S., Ilie, M., Balepur, N., Kahadze, K., Liu, A., Si, C., Li, Y., Gupta, A., Han, H., Schulhoff, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.06608 · Cited by 118
Shah, C. S., & Bender, E. M. (2022) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3498366.3505816 · Cited by 216
“ponents and training sets (following proposals such as [6, 30, 50] but instantiated for specific systems), work on documenting and understanding bias perpetuation such as [54], work on designing interfaces that make the system properties transparent to all types of”
BackgroundConcealed Data Poisoning Attacks on NLP Models
Wallace, E., Zhao, T., Feng, S., & Singh, S. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.13 · Cited by 115
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective
Kiritchenko, S., Nejadgholi, I., & Fraser, K. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12590 · Cited by 172
Morris, M. R. (2020) · Communications of the ACM · doi:10.1145/3356727 · Cited by 114
Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
Cabitza, F., Campagner, A., & Basile, V. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25840 · Cited by 114
“2019); the Data Statements (Bender and Friedman 2018); or the Datasheets for Datasets (Gebru et al. 2021) initiative, followed by IBM with their AI FactSheets (Richards et al.”
BackgroundMethodInfluentialA toolbox for surfacing health equity harms and biases in large language models
Pfohl, S. R., Cole-Lewis, H., Sayres, R., Neal, D., Asiedu, M., Dieng, A., Tomasev, N., Rashid, Q. M., Azizi, S., Rostamzadeh, N., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03258-2 · Cited by 113
15 challenges for AI: Or what AI (currently) can’t do
Hagendorff, T., & Wezel, K. (2020) · AI & SOCIETY · doi:10.1007/s00146-019-00886-y · Cited by 113
“The latter are guided by subjective value assumptions, for example when selecting certain training data (Gebru et al. 2018), pre-processing data sets, defining thresholds, choosing certain models, interpreting outcomes, etc.”
BackgroundUche-Anya, E. N., Anyane‐Yeboa, A., Berzin, T. M., Ghassemi, M., & May, F. P. (2022) · Gut · doi:10.1136/gutjnl-2021-326271 · Cited by 140
Xie, J., Sun, L., & Zhao, Y. F. (2025) · Engineering · doi:10.1016/j.eng.2024.04.024 · Cited by 112
Decision Provenance: Harnessing Data Flow for Accountable Systems
Singh, J., Cobbe, J., & Norval, C. (2019) · IEEE Access · doi:10.1109/access.2018.2887201 · Cited by 110
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3267446 · Cited by 394
Establishing Data Provenance for Responsible Artificial Intelligence Systems
Werder, K., Ramesh, B., & Zhang, R. (2022) · ACM Transactions on Management Information Systems · doi:10.1145/3503488 · Cited by 188
“For example, data provenance is needed to develop a data information sheet [31] that provides details on the most important variables influencing an AI-based system’s recommendations.”
BackgroundPolicy advice and best practices on bias and fairness in AI
Álvarez, J. M., Colmenarejo, A. B., Elobaid, A., Fabbrizzi, S., Fahimi, M., Ferrara, A., Ghodsi, S., Mougan, C., Papageorgiou, I. T., Lobo, P. R., et al. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09746-w · Cited by 160
“Existing frameworks for the elaboration of documentation include: Datasheets for Datasets (Gebru et al., 2021), Dataset Nutrition Labels (Chmielinski et al., 2022), Data Statements (Bender & Friedman, 2018), Data Readiness Report (Afzal et al., 2021), and Model Cards for Models (Mitchell et al., 2019).”
InfluentialJacobs, P. G., Herrero, P., Facchinetti, A., Vehi, J., Kovatchev, B., Breton, M. D., Cinar, A., Nikita, K. S., Doyle, F. J., Bondia, J., et al. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2023.3331297 · Cited by 113
“Best practice 4: Include a data sheet [80] for the data set used in training, validation and testing.”
MethodKLUE: Korean Language Understanding EvaluationPreprint
Park, S., Moon, J., Kim, S., Cho, W. I., Han, J., Park, J., Song, C., Kim, J., Song, Y., Oh, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.09680 · Cited by 304
“We adopt the recently proposed documentation frameworks; datasheets [41] and data statements [6].”
MethodMeasuring algorithmically infused societies
Wagner, C., Strohmaier, M., Olteanu, A., Kıcıman, E., Contractor, N., & Eliassi-Rad, T. (2021) · Nature · doi:10.1038/s41586-021-03666-1 · Cited by 144
“They also fail to define and justify the criteria and metrics for assessing the quality of a measurement model, describe and justify the data selection and preprocessing methods, and acknowledge and mitigate biases in the data 69,73–75 .”
BackgroundRepairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123
Efficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 236
“Qualitative tools include documentation for datasets and model attributes (Gebru et al., 2021).”
MethodContestable AI by Design: Towards a Framework
Alfrink, K., Keller, I., Kortuem, G., & Doorn, N. (2022) · Minds and Machines · doi:10.1007/s11023-022-09611-z · Cited by 169
“Mitchell et al. (2019) and Gebru et al. (2020) offer examples of possible documentation approaches.”
Mylrea, M., & Robinson, N. (2023) · Entropy · doi:10.3390/e25101429 · Cited by 166
“By adhering to ethical principles during data collection, preprocessing, and model training, AI systems can mitigate biases, promote fairness, and ensure equitable benefits for all individuals and communities [54,68].”
BackgroundLessons from Archives: Strategies for Collecting Sociocultural Data in Machine LearningPreprint
Jo, E. S., & Gebru, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.10389 · Cited by 102
Self-citationInteractive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101
“We also look to work on datasheets for datasets to include more detailed information of the model’s data [21].”
BackgroundThe Principles of Data-Centric AI
Jarrahi, M. H., Memariani, A., & Guha, S. (2023) · Communications of the ACM · doi:10.1145/3571724 · Cited by 204
Open-source tools for behavioral video analysis: Setup, methods, and best practices
Luxem, K., Sun, J. J., Bradley, S. P., Krishnan, K., Yttri, E., Zimmermann, J., Pereira, T. D., & Laubach, M. (2023) · eLife · doi:10.7554/elife.79305 · Cited by 98
“Guidance for how to implement data sharing can be found in several existing efforts of the machine learning community (Gebru et al., 2021; Hutchinson et al., 2021; Stoyanovich and Howe, 2019).”
BackgroundTest of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
Fatemi, B., Kazemi, M., Tsitsulin, A., Malkan, K., Yim, J., Palowitch, J., Seo, S., Halcrow, J., & Perozzi, B. (2024) · arXiv · doi:10.48550/arxiv.2406.09170 · Cited by 101
What Will it Take to Fix Benchmarking in Natural Language Understanding?
Bowman, S. R., & Dahl, G. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.385 · Cited by 95
AI Ethics Principles in Practice: Perspectives of Designers and Developers
Sanderson, C., Douglas, D., Lu, Q., Schleiger, E., Whittle, J., Lacey, J., Newnham, G. J., Hajkowicz, S. A., Robinson, C., & Hansen, D. T. (2023) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2023.3257303 · Cited by 167
“A record should be kept of the methods used for pre-processing (or cleaning) data that is then used as inputs the ML model [28].”
BackgroundMethodExploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99
“In addition, Boyd et al. found that context documents like Datasheets [41] could better scaffold an ML practitioner’s process of issue discovery, understanding, and ethical decision-making around ML training datasets[19].”
BackgroundInfluentialAnik, A. I., & Bunt, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445736 · Cited by 103
“proposed the concept of providing a datasheet for each dataset to document, for example, its motivation, creation, composition, intended uses, distribution, and maintenance [44].”
BackgroundInfluentialGeiger, R. S., Cope, D., Ip, J., Lotosh, M., Shah, A., Weng, J., & Tang, R. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00144 · Cited by 141
Widder, D. G., Zhen, D., Dabbish, L., & Herbsleb, J. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594012 · Cited by 94
Retiring Adult: New Datasets for Fair Machine LearningPreprint
Ding, F., Hardt, M., Miller, J. A., & Schmidt, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04884 · Cited by 599
“Influential works relating to the ethics and fairness of machine learning recognize the centrality of datasets, pointing to significant harms associated with data, as well as better data practices [11, 17, 21, 25, 27].”
BackgroundMethodDesigning Responsible AI: Adaptations of UX Practice to Meet Responsible AI Challenges
Wang, Q., Madaio, M., Kane, S., Kapania, S., Terry, M., & Wilcox, L. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581278 · Cited by 90
Labelling instructions matter in biomedical image analysis
Rädsch, T., Reinke, A., Weru, V., Tizabi, M. D., Schreck, N., Kavur, A. E., Pekdemir, B., Roß, T., Kopp‐Schneider, A., & Maier‐Hein, L. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00625-5 · Cited by 121
A large-scale audit of dataset licensing and attribution in AI
Longpre, S., Mahari, R., Chen, A., Obeng-Marnu, N., Sileo, D., Brannon, W., Muennighoff, N., Khazam, N., Kabbara, J., Perisetla, K., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00878-8 · Cited by 307
“Altogether, we hope this tooling enables and promotes the thorough documentation proposed in previous work 24,40,67,68”
Synthetic data in biomedicine via generative artificial intelligence
van Breugel, B., Liu, T., Oglić, D., & van der Schaar, M. (2024) · Nature Reviews Bioengineering · doi:10.1038/s44222-024-00245-7 · Cited by 149
“Largescale generative models, such as the large language models DALLE 88 and ChatGPT, can be applied for text and image generation, a process we call queryspecific generation.”
Overview of leakage scenarios in supervised machine learning
Sasse, L., Nicolaisen‐Sobesky, E., Dukart, J., Eickhoff, S. B., GOETZ, M. A., Hamdan, S., Komeyer, V., Kulkarni, A. J., Lahnakoski, J. M., Love, B. C., et al. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01193-8 · Cited by 145
“Data processing and quality reporting have also been discussed (e.g. Datasheets for Datasets [20] and Data Nutrition Labels [30]).”
MethodFair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor
Nissim, M., van Noord, R., & van der Goot, R. (2020) · Computational Linguistics · doi:10.1162/coli_a_00379 · Cited by 88
“(and emphasise) the presence of human bias. Using such sensational “party tricks" (Gonen and Goldberg 2019) is harmful, as they get easily propagated both in science itself (Jha and Mamidi 2017; Gebru et al. 2018; Mohammad et al. 2018), even outside NLP and AI (McQuillan2018), as well as in popularised articles o…”
BackgroundGarcia Santa Cruz, B., Bossa, M. N., Sölter, J., & Husch, A. D. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102225 · Cited by 89
Liao, Q. V., Subramonyam, H., Wang, J., & Wortman Vaughan, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580652 · Cited by 86
“While there has been extensive work on AI documentation [12, 45, 50, 51, 78, 80], who the consumers are and how they are consuming it has not been well studied empirically.”
BackgroundMethodInfluentialSelf-citationA Total Error Framework for Digital Traces of Human Behavior on Online Platforms
Sen, I., Flöck, F., Weller, K., Weiß, B., & Wagner, C. (2021) · Public Opinion Quarterly · doi:10.1093/poq/nfab018 · Cited by 173
“Of course, demographic attribute inference itself is a task that may be affected by various errors (Karimi et al., 2016; McCormick et al., 2017) and can be especially problematic if there are different error rates for different groups of people (Buolamwini and Gebru, 2018).”
BackgroundShow Your Work: Improved Reporting of Experimental Results
Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1224 · Cited by 233
“Similar to other calls for improved reporting in machine learning (Mitchell et al., 2019; Gebru et al., 2018), we recommend pairing experimental results with the information from this checklist in a structured format (see examples provided in Appendix A).”
MethodSign Language Recognition, Generation, and Translation: An Interdisciplinary PerspectivePreprint
Bragg, D., Koller, O., Bellard, M., Berke, L., Boudrealt, P., Braffort, A., Caselli, N. K., Huenerfauth, M., Kacorri, H., Verhoef, T., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.08597 · Cited by 87
“In particular, demographics may be important for understanding biases and generalizability of systems trained on the data [44].”
BackgroundChalkidis, I., Fergadiotis, M., & Androutsopoulos, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.559 · Cited by 184
“We also provide a detailed Dataset Card (Gebru et al., 2018) for the MULTI-EURLEX dataset in Appendix D.”
MethodA Checklist to Combat Cognitive Biases in Crowdsourcing
Draws, T., Rieger, A., Inel, O., Gadiraju, U., & Tintarev, N. (2021) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v9i1.18939 · Cited by 86
Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI
Ehsan, U., Saha, K., De Choudhury, M., & Riedl, M. O. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579467 · Cited by 85
Addressing bias in generative AI: Challenges and research opportunities in information management
Wei, X., Kumar, N., & Zhang, H. (2025) · Information & Management · doi:10.1016/j.im.2025.104103 · Cited by 174
Why ‘open’ AI systems are actually closed, and why this matters
Widder, D. G., Whittaker, M., & West, S. M. (2024) · Nature · doi:10.1038/s41586-024-08141-1 · Cited by 86
“…a closed element of many AI offerings advertising themselves as open: many large-scale AI models described as open neglect to provide even basic information about the underlying data used to train the system 77 , let alone offering the underlying training data openly or documenting its provenance.”
Chan, A., Ezell, C., Kaufmann, M., Wei, K., Hammond, L., Bradley, H., Bluemke, E., Rajkumar, N., Krueger, D., Kolt, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658948 · Cited by 84
“We refer to the set of such additional information as an agent card , drawing inspiration from previous work on documenting AI systems [24, 68, 69, 112].”
BackgroundMethodA Survey of Data Quality Requirements That Matter in ML Development Pipelines
Priestley, M., O’donnell, F., & Simperl, E. (2023) · Journal of Data and Information Quality · doi:10.1145/3592616 · Cited by 86
“This includes datasheets [23, 35], data statements [9] and checklists [58].”
BackgroundMethodResultInfluentialAn Action-Oriented AI Policy Toolkit for Technology Audits by Community Advocates and Activists
Krafft, P. M., Young, M., Katell, M. A., Lee, J. E., Narayan, S., Epstein, M., Dailey, D., Herman, B., Tam, A., Guetler, V., et al. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445938 · Cited by 120
Ibáñez, J. C., & Olmeda, M. V. (2021) · AI & Society · doi:10.1007/s00146-021-01267-0 · Cited by 125
“They should not consider only their datasets and how they are selected (Gebru et al. 2018) but also how they process, use and trace them (de Bruin and Floridi 2017) and how their algorithms are coded or implemented (Kitchin 2016).”
BackgroundTackling bias in AI health datasets through the STANDING TOGETHER initiative
Ganapathi, S., Palmer, J., Alderman, J. E., Calvert, M., Espinoza, C., Gath, J., Ghassemi, M., Heller, K., Mckay, F., Karthikesalingam, A., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01987-w · Cited by 82
Human-centred artificial intelligence: a contextual morality perspective
van Berkel, N., Tag, B., Gonçalves, J., & Hosio, S. J. (2020) · Behaviour and Information Technology · doi:10.1080/0144929x.2020.1818828 · Cited by 105
Understanding and Evaluating Racial Biases in Image Captioning
Zhao, D., Wang, A., & Russakovsky, O. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01456 · Cited by 169
“Researchers have advocated for increased transparency of datasets [27, 35], including developing tools to steer researcher intervention [8, 70].”
BackgroundLearning from data with structured missingness
Mitra, R., McGough, S. F., Chakraborti, T., Holmes, C. C., Copping, R., Hagenbuch, N., Biedermann, S., Noonan, J., Lehmann, B., Shenvi, A., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-022-00596-z · Cited by 139
“As a start, introducing standardised processes for documenting missingness in datasets would help ensure any conclusions appropriately acknowledge these biases (76; 77).”
SegmentMeIfYouCan: A Benchmark for Anomaly SegmentationPreprint
Chan, R., Lis, K., Uhlemeyer, S., Blum, H., Honari, S., Siegwart, R., Fua, P., Salzmann, M., & Rottmann, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14812 · Cited by 224
“The following section is a complete answer to the datasheet questions from [48].”
BackgroundA survey on ethical principles of AI and implementations
Zhou, J., Chen, F., Berry, A., Reed, M., Zhang, S., & Savage, S. (2020) · 2020 IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci47803.2020.9308437 · Cited by 99
“[38] proposed that every dataset be accompanied with a datasheet that documents its motivation, composition, collection process, preprocessing/cleaning/ labeling, recommended uses, distribution, and maintenance.”
BackgroundInfluentialWang, A. Y., Wang, D., Drozdal, J., Muller, M. J., Park, S., Weisz, J. D., Liu, X., Wu, L., & Dugan, C. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3489465 · Cited by 121
“…cells created by Themisto ); how many times a participant ignored the generated recommendations and manually created the documentation ( markdown cells created by human ); and how many time a markdown cell is co-created by human and Themisto( markdown cells co-created by human and Themisto…”
BackgroundChen, J., Wang, D., Shao, Z., Zhang, X., Ruan, M., Li, H., & Li, J. (2023) · Buildings · doi:10.3390/buildings13092285 · Cited by 145
Stoyanovich, J., Howe, B., & Jagadish, H. V. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3415478.3415570 · Cited by 76
“Several recent projects, including the Dataset Nutrition Label project [24], Datasheets for Datasets [19], and Model Cards [39], are proposing to use metadata to support interpretability.”
MethodMarkl, N. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533117 · Cited by 153
“This includes where, how, when, why, about whom or what, by whom and for whom the dataset was compiled and, where relevant, who annotated it [52].”
BackgroundPublic procurement of artificial intelligence systems: new risks and future proofing
Hickok, M. (2024) · AI & SOCIETY · doi:10.1007/s00146-022-01572-2 · Cited by 81
Improving fairness in machine learning systems: What do industry practitioners need?Preprint
Holstein, K., Vaughan, J. W., Daumé, H., Dudík, M., & Wallach, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.05239 · Cited by 81
“many industry contexts have some control over data collection and curation. Future research should support these practitioners in collecting and curating high quality datasets in the first place (cf. [23, 40, 59]). In contrast to the fair ML literature, HCI research on ML developer tools has often focused on the desig…”
Self-citationFormalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AIPreprint
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07487 · Cited by 81
“rk that proposes standardized documentations to communicate the performance characteristics of trained AI models. The examples of such documentations are: data statements [6], datasheets for datasets [18], model cards [54], reproducibility checklists [56], fairness checklists [48], and factsheets [2]. We illustrate th…”
Background
Metadata
- Authors
- Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K.
- Affiliations
- DAIR Institute (US)
- University of Washington (US)
- Cornell University (US)
- Microsoft Research New York City (United States) (US)
- University of Maryland, College Park, MD
- University of Southern California (US)
- ORCID
- Venue
- Communications of the ACM
- Published
- 2021-11-19
- Volume
- 64
- Issue
- 12
- Pages
- 86-92
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2795038878
- Semantic Scholar
- 0df347f5e3118fac7c351917e3a497899b071d1e
- Cited by
- 2,469
- References
- 37
- Citations by year
- 2026: 788 · 2025: 402 · 2024: 298 · 2023: 379 · 2022: 221 · 2021: 217
- FWCI
- 252.30
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Communications of the ACM (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation
- Alfred P. Sloan Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Artificial Intelligence › Semantic Web and Ontologies
- Keywords
- dataset documentation, data provenance, data sharing, data reuse, metadata
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.