The DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index.
Datasheets for datasets
Open accessAbstract
Documentation to facilitate communication between dataset creators and consumers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Paradata needs may include the preservation of novel forms of information, such as datasheets explaining training datasets or model cards documenting an AI tool, and of these documents' relation to archival collections processed using them [ 26 ].”
From Positioning Paradata: A Conceptual Frame for AI Processual Documentation in Archives and Recordkeeping Contexts (2023) · cites this paper as [26]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Significant critiques of LLMs have been raised, including concerns about bias and fairness, misinformation, opaqueness due to a lack of interpretability and explainability, and environmental impact (Bender et al., 2021; Gebru et al., 2021; Strubell et al., 2019).”
From Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments (2023) · cites this paper as Gebru et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“If ML driven algorithms are considered as epistemic resources that help us to improve understanding and/or make better predictions, then the statistical nature of ML de facto prevents groups underrepresented in the data from contributing to this epistemic resource on an equal ground to dominant groups (Gebru et al., 2021)”
From Algorithmic profiling as a source of hermeneutical injustice (2025) · cites this paper as Gebru et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The authors found that providing examples of what the requirements can look like helps to guide practitioners in clarifying their own rules.”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [24]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“I show how archival theories have been animated in documenting data in the context of AI and ML, including through documentation frameworks like Datasheets for Datasets”
From Can AI provenance inform archival provenance? (2026) · cites this paper as Gebru et al. 2021
12 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments
Huang, G. B., Mattar, M., Berg, T. L., & Miller, E. G. (2008) · HAL · Cited by 4,822
Minimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109
(2018) · Cited by 3,510
Self-citationA sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts
Pang, B., & Lee, L. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1218990 · Cited by 3,301
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“Researchers at Google published follow-up work on model cards that document machine learning models [20] and released a data card (a lightweight version of a datasheet) along with the Open Images dataset [17].”
BackgroundSelf-citationMan is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · cited 2×
“Of particular concern are recent examples showing that machine learning models can reproduce or amplify unwanted societal biases reflected in training data [4, 5, 9].”
BackgroundThe Effects of Automobile Safety Regulation
Peltzman, S. (1975) · Journal of Political Economy · doi:10.1086/260352 · Cited by 1,531
The Recent Past and Near Future of Risk and/or Need Assessment
Andrews, D. A., Bonta, J., & Wormith, J. S. (2006) · Crime & Delinquency · doi:10.1177/0011128705281756 · Cited by 1,507 · cited 1×
“Domains such as criminal justice (Garvie et al., 2016; Systems, 2017; Andrews et al., 2006), hiring and employment (Mann & O’Neil, 2016), critical infrastructure (O’Connor, 2017; Chui, 2017), and finance (Lin, 2012) all increasingly depend on machine learning methods.”
Background
Research works printed but not linked: 7
- Chui, G. Project will use AI to prevent or minimize electric grid failures, 2017.
- Garvie, C., Bedoya, A., and Frankle, J. The Perpetual Line-Up: Unregulated Police Face Recognition in America. Georgetown Law, Center on Privacy & Technology, Washington, D.C., 2016.
- Holstein, K., Vaughan, J., Daumé, H, Dudík, M., and Wallach, H. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of 2019 ACM CHI Conf. on Human Factors in Computing Systems.
- Krasin, I. et al. OpenImages: A public dataset for large-scale multi-label and multi-class image classification, 2017.
- O'Connor, M. How AI Could Smarten Up Our Water System, 2017.
- Doha Supply Systems. Facial Recognition, 2017.
- Yagcioglu, S., Erdem, A., Erdem, E., and Ikizler-Cinbis, N. RecipeQA: A challenge dataset for multimodal comprehension of cooking recipes. In Proceedings of the 2018 Conf. on Empirical Methods in Natural Language Processing.
Cited by
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953
“In §F.1 we provide a Dataset Card for SA-1B, following [39], in a list of questions and answers.”
MethodLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
The Ethics of AI Ethics -- An Evaluation of Guidelines
Hagendorff, T. (2019) · arXiv · doi:10.48550/arxiv.1903.03425 · Cited by 1,907
“aware data mining (Gebru et al. 2018), as well as privacy (Baron and Musolesi 2017).”
BackgroundInfluentialScientific discovery in the age of artificial intelligence.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06221-2 · Cited by 1,823
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Self-citationRahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364
“Further, when systems learn from data, their failures are linked to imperfections in data or how data was collected, leading some to argue for adapted reporting mechanisms for datasets 47 and models 48 .”
BackgroundSoftware Engineering for Machine Learning: A Case Study
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., & Zimmermann, T. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00042 · Cited by 1,135
“propose to use data sheets inspired by electronics to more transparently and reliably track the metadata characteristics of these datasets [34].”
MethodLeakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
The ethics of AI in health care: A mapping review
Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113172 · Cited by 1,105
“Similarly, although some technical solutions have been put forward for mitigating issues with data bias (Gebru et al., 2018; Holland et al., 2018) and data quality (Dai et al., 2018) and ensuring social inclusion in decision-making (Balthazar et al., 2018; Friedman et al., 2017; Rahwan, 2018),…”
BackgroundBender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854
Ferrara, E. (2023) · Sci · doi:10.3390/sci6010003 · Cited by 875
“Documenting such dataset biases and augmentation procedures is of paramount importance [27–29].”
BackgroundBaker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993
“Corporate voices have been prominent in discussions of algorithmic bias in broader computer science discourse (Blodgett et al., 2020; Cramer et al., 2019; Gebru et al., 2018; Mitchell et al., 2021), but have been relatively less prominent in education.”
BackgroundFlamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
“[30] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford.”
BackgroundInfluentialFrom Recognition to Cognition: Visual Commonsense Reasoning
Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00688 · Cited by 769
“A datasheet is a list of questions that accompany datasets that are released, in part so that people think hard about the phenomena in their data [23].”
BackgroundSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763
“Gaps between the purpose and the assumptions under which these datasets and models are built and those under which they are being used can lead to performance disparities, affecting research validity and producing misleading results (e.g., Blodgett et al., 2016; Buolamwini and Gebru, 2018).”
BackgroundAlgorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687
“Documenting all of these choices, from data collection, sampling, and measurement to data processing, enables modeling that appropriately accounts for the specific conditions under which data have been collected (Gebru et al. 2018, Holland et al. 2018).”
MethodMachine learning for medical imaging: methodological failures and recommendations for the future
Varoquaux, G., & Cheplygina, V. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00592-y · Cited by 656
“Possibilities include data sheets [Gebru et al., 2018], which describe the data collection procedure, and model cards [Mitchell”
Trustworthy Artificial Intelligence: A Review
Kaur, D., Uslu, S., Rittichier, K. J., & Durresi, A. (2023) · ACM Computing Surveys · doi:10.1145/3491209 · Cited by 709
“Another similar approach [75] is creating a data sheet consisting of all information related to the data collection process, dataset features, and recommended use.”
BackgroundBias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 1,245
“Curation of additional language resources should value inclusivity over convenience, and documentation should follow practices such as Bender & Friedman (2018) and Gebru et al. (2021). Furthermore, stakeholders must ensure that the process of collecting data itself does not contribute to further harms.”
BackgroundChallenges in deploying machine learning: A survey of case studies
Paleyes, A., Urma, R.-G., & Lawrence, N. D. (2023) · ACM Computing Surveys · doi:10.1145/3533378 · Cited by 638
Lekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957
“…(eg, hyperparameters), training and testing data, evaluation criteria and results, biases and other limitations, and periodic audits and updates 49-51 ; (c) a publication based on existing AI reporting the bmj | BMJ 2025;388:e081554 | doi: 10.1136/bmj-2024-081554 standards 13 15 52 ; and (d) a…”
InfluentialFairness of artificial intelligence in healthcare: review and recommendations
Ueda, D., Kakinuma, T., Fujita, S., Kamagata, K., Fushimi, Y., Ito, R., Matsui, Y., Nozaki, T., Nakaura, T., Fujima, N., et al. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01474-3 · Cited by 607
“Data bias refers to problems arising from the collection and organization of data used in AI training that can poten‑ tially have harmful effects on fairness and accuracy [67].”
BackgroundMethodSurvey of Explainable AI Techniques in Healthcare
Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023) · Sensors · doi:10.3390/s23020634 · Cited by 591
Sex and gender analysis improves science and engineering
Tannenbaum, C., Ellis, R. P., Eyssel, F., Zou, J., & Schiebinger, L. (2019) · Nature · doi:10.1038/s41586-019-1657-6 · Cited by 581
“Beyond automotive safety systems, the importance of anthropometric characteristics, such as the carrying angle of the elbow or the shape and size of the human knee, can be used to guide sex-specific design for artificial joints, limb prostheses and occupational protective gear 50,51 .”
BackgroundTrustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
Advances, challenges and opportunities in creating data for trustworthy AI
Liang, W., Tadesse, G. A., Ho, D., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00516-1 · Cited by 559
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445923 · Cited by 545
“rk that proposes standardized documentations to communicate the performance characteristics of trained AI models. The examples of such documentations are: data statements [6], datasheets for datasets [18], model cards [54], reproducibility checklists [56], fairness checklists [48], and factsheets [2]. We illustrate th…”
BackgroundTowards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548
“The proliferation of tools such as datasheets and model cards are intended to fill that gap [245, 246].”
BackgroundUnderstanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527
“…for future research, but could mean, for example, accompanying a model with a summary of characteristics of the data on which it was trained and evaluated in order to help end users reason about scenarios in which the model is most likely to achieve its stated performance on held-out data [10, 19].”
BackgroundSelf-citationde Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499
“It is recommended that the amount of collected data is suf fi ciently large for the intended purpose 6,8,12,15,20,22,26,46 – 49 , is ideally prespeci fi ed 8 and should be clearly reported 3,14,37,46,50 .”
BackgroundSchiff, D. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00270-2 · Cited by 489
Preiksaitis, C., & Rose, C. (2023) · JMIR Medical Education · doi:10.2196/48785 · Cited by 552
Madaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436
“Moreover, despite recent work on methods for documenting datasets [9, 41, 50] and models [67], similar methods do not yet exist for other components or stages of the lifecycle.”
MethodSelf-citationFactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434
“We expect that AI will soon go through the same evolution that other technologies have gone through (see [8] for an excellent review of the evolution of safety standards in different industries).”
BackgroundMethodInfluentialAlgorithmic bias in data-driven innovation in the age of AI
Akter, S., McCarthy, G., Sajib, S., Michael, K., Dwivedi, Y. K., D’Ambra, J., & Shen, K. N. (2021) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2021.102387 · Cited by 440
Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective
Bragg, D., Koller, O., Bellard, M., Berke, L., Boudreault, P., Braffort, A., Caselli, N., Huenerfauth, M., Kacorri, H., Verhoef, T., et al. (2019) · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3308561.3353774 · Cited by 422
“In particular, demographics may be important for understanding biases and generalizability of systems trained on the data [44].”
BackgroundThieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406
“00388 [66] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumeé III, and Kate Crawford.”
BackgroundThe Ethics of AI Ethics: An Evaluation of Guidelines
Hagendorff, T. (2020) · PhilPapers (PhilPapers Foundation) · Cited by 394
Munn, L. (2023) · AI and Ethics · doi:10.1007/s43681-022-00209-w · Cited by 355
“The article thus concludes by highlighting alternatives: the first thinks more broadly about AI justice, considering sociopo-litical dynamics and systems of oppression [14, 46]; the second thinks more narrowly, focusing on concrete issues like accuracy, auditing, and governance [25, 67].”
BackgroundCausality matters in medical imaging
Castro, D. C., Walker, I., & Glocker, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17478-w · Cited by 364
“Scholars in FAT have proposed checklist-style guidelines for reporting datasets 59 and models 60 , for example, and have been investigating sources of failure for machine-learning models, among which is poor reporting 61 .”
MethodSantos, T. T., de Souza, L. L., dos Santos, A. A., & Avila, S. (2020) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2020.105247 · Cited by 359
“e 1. For a subset of 2,020 clusters, binary masks are provided for instance segmentation. Appendix A presents a detailed description for the dataset, following the guidelines proposed by Gebru et al. [14] for dataset characterization, and including information about cameras, field location, pre-processing and file forma…”
MethodInfluentialThe ethics of algorithms: key problems and solutions
Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01154-8 · Cited by 384
“Even while bias may be anticipated and protected variables excluded from the data, unanticipated proxies for these variables could still be used to reconstruct biases, leading to “bias by proxy” that is difficult to detect and avoid (Fuster et al. 2017; Gillis and Spiess 2019).”
Khan, S. M., Liu, X., Nath, S., Korot, E., Faes, L., Wagner, S. K., Keane, P. A., Sebire, N. J., Burton, M. J., & Denniston, A. K. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30240-5 · Cited by 334
Datasets: A Community Library for Natural Language Processing
Lhoest, Q., Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.emnlp-demo.21 · Cited by 343
“…are being created with a variety of different procedures, from crowd-sourcing to scraping to synthetic generation, which need to be taken into account when evaluating which is most appropriate for a given purpose and ought to be immediately apparent to prospective users (Gebru et al., 2018).”
BackgroundThe Effects of Data Quality on Machine Learning Performance on Tabular Data
Mohammed, S., Budach, L., Feuerpfeil, M., Ihde, N., Nathansen, A., Noack, N., Patzlaff, H., Naumann, F., & Harmouch, H. (2025) · Information Systems · doi:10.1016/j.is.2025.102549 · Cited by 331
“Furthermore, some studies presented a vision of data quality assessment tools [29], an automation of data quality verification [62] or a methodology to summarize the quality of a dataset as datasheets [24], nutritional labels [70], and data cards [71].”
Power to the People? Opportunities and Challenges for Participatory AI
Birhane, A., Isaac, W., Prabhakaran, V., Diaz, M., Elish, M. C., Gabriel, I., & Mohamed, S. (2022) · Equity and Access in Algorithms Mechanisms and Optimization · doi:10.1145/3551624.3555290 · Cited by 330
“…including: auditing methods for machine learning based interventions [48, 75], public consultation methods such as citizen juries [6, 83] or joint problem formulation [61], information disclosures such as model cards [45, 63] or datasheets [37, 44], and artefact co-development [42, 54, 79].”
BackgroundHealth data poverty: an assailable barrier to equitable digital health care
Ibrahim, H., Liu, X., Zariffa, N., Morris, A. D., & Denniston, A. K. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30317-4 · Cited by 326
Predictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., DasSarma, N., Henighan, T., Jones, A., Joseph, N., Kernion, J., Mann, B., Askell, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2202.07785 · Cited by 324
“, through the use of model cards [48] and data sheets [26] to document the capabilities, drawbacks, and other salient details of models).”
MethodAdministration by Algorithm? Public Management meets Public Sector Machine Learning
Veale, M., & Brass, I. (2019) · doi:10.31235/osf.io/mwhnb · Cited by 316
Inherent Tradeoffs in the Fair Determination of Risk Scores
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603197 · Cited by 310
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306
“EN dataset. structured, task-specific NLP datasets, best practices have emerged around documenting the collection process, composition, intended uses, and other characteristics (Bender and Friedman, 2018; Gebru et al., 2018; Hutchinson et al., 2021).”
BackgroundToward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306
“(18)Model cards for model reporting [28] and data sheets for datasets [29] reveal information about AI systems publicly, and future work in third party auditing could build on such tools, as advocated by Raji and Smart et al.”
BackgroundMethodDeduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.577 · Cited by 304
The value of standards for health datasets in artificial intelligence-based applications
Arora, A., Alderman, J. E., Palmer, J., Ganapathi, S., Laws, E., McCradden, M. D., Oakden-Rayner, L., Pfohl, S. R., Ghassemi, M., McKay, F., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02608-w · Cited by 294
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Bhatt, U., Antorán, J., Zhang, Y., Liao, Q. V., Sattigeri, P., Fogliato, R., Melançon, G., Krishnan, R., Stanley, J., Tickoo, O., et al. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462571 · Cited by 300
Responsible AI—two frameworks for ethical design practice
Peters, D., Vold, K., Robinson, D., & Calvo, R. A. (2020) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2020.2974991 · Cited by 293
“Yet, moving effectively from ethical theory and principles into context specific, actionable practice is proving a signifi-cant barrier for the widespread uptake of systematic ethical impact analysis in software engineering [13], [14].”
BackgroundMethodCausality matters in medical imaging
Castro, D. C., Walker, I. D., & Glocker, B. (2020) · RePEc: Research Papers in Economics · Cited by 288
Reproducibility in machine learning for health research: Still a ways to go
McDermott, M. B. A., Wang, S., Marinsek, N., Ranganath, R., Foschini, L., & Ghassemi, M. (2021) · Science Translational Medicine · doi:10.1126/scitranslmed.abb1655 · Cited by 273
Explainability fact sheets: a framework for systematic assessment of explainable approaches
Sokol, K., & Flach, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372870 · Cited by 264
AI Art and its Impact on Artists
Jiang, H. H., Brown, L., Cheng, J., Khan, M., Gupta, A., Workman, D., Hanna, A., Flowers, J., & Gebru, T. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604681 · Cited by 265
Self-citationRoom-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
Ku, A., Anderson, P., Patel, R., Ie, E., & Baldridge, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.356 · Cited by 274
Large image datasets: A pyrrhic win for computer vision?
Birhane, A., & Prabhu, V. U. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00158 · Cited by 257
“Much along the lines of model cards [55] and datasheet for datasets [35], we propose dissemination of dataset audit cards.”
MethodPutting AI ethics to work: are the tools fit for purpose?
Ayling, J., & Chapman, A. (2022) · AI and Ethics · doi:10.1007/s43681-021-00084-x · Cited by 253
“A second phase saw a more technical approach from the computer science community focusing on fairness, accountability and transparency as an engineering ‘ethical-by-design’ problem-solving exercise [20–22].”
BackgroundGoverning AI safety through independent audits
Falco, G., Shneiderman, B., Badger, J., Carrier, R., Dahbura, A., Danks, D., Eling, M., Goodloe, A., Gupta, J., Hart, C., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00370-7 · Cited by 250
Predictive biases in natural language processing models: A conceptual framework and overview
Shah, D. S., Schwartz, H. A., & Hovy, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.468 · Cited by 241
“Similarly, Datasheets Gebru et al. (2018) are suggested to cover the lifecycle of data including “motivation for dataset creation; dataset composition; data collection process; data preprocessing; dataset distribution; dataset maintenance; and legal and ethical considerations”.”
BackgroundLiu, X., Glocker, B., McCradden, M. M., Ghassemi, M., Denniston, A. K., & Oakden-Rayner, L. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00003-6 · Cited by 266
Interpretable Machine Learning
Boehmke, B., & Greenwell, B. (2019) · Hands-On Machine Learning with R · doi:10.1201/9780367816377-16 · Cited by 230
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
“…data or model cards, documentation of the data and who is represented within it, as well as how the model was made, assumptions that were made, how the model behaves in relation to diverse groups of people, and how well the model performs in relation to those groups [45, 55, 68, 105, 191, 222].”
BackgroundRe-imagining Algorithmic Fairness in India and Beyond
Sambasivan, N., Arnesen, E., Hutchinson, B., Doshi, T., & Prabhakaran, V. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445896 · Cited by 228
MERLOT: Multimodal Neural Script Knowledge ModelsPreprint
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02636 · Cited by 493
“In this section, we present a DataSheet [37, 12] for YT-Temporal-180M, synthesizing many of the other analyses we performed in this paper.”
MethodTechnology readiness levels for machine learning systems
Lavin, A., Gilligan-Lee, C. M., Visnjic, A., Ganju, S., Newman, D., Ganguly, S., Lange, D., Baydin, A. G., Sharma, A., Gibson, A., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33128-9 · Cited by 209
Bias in Large Language Models: Origin, Evaluation, and Mitigation
Guo, Y., Guo, M., Su, J., Yang, Z., Zhu, M., Li, H., Qiu, M., & Liu, S. S. (2024) · arXiv · doi:10.48550/arxiv.2411.10915 · Cited by 208
“Additionally, fostering interdisciplinary collaborations to address ethical challenges and develop adaptive regulatory frameworks (Gebru et al., 2021), while optimizing the balance between model performance and fairness (Joniak and Aizawa, 2022), remains crucial.”
Nasarian, E., Alizadehsani, R., Acharya, U. R., & Tsui, K.-L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102412 · Cited by 207
Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models
Schramowski, P., Brack, M., Deiseroth, B., & Kersting, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02157 · Cited by 219
“: “[data that] if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety” [12], which is for example also reflected by the OpenAI content policy7 that applies to the use of DALL-E [29].”
BackgroundMalgaroli, M., Hull, T. D., Zech, J. M., & Althoff, T. (2023) · Translational Psychiatry · doi:10.1038/s41398-023-02592-2 · Cited by 201
“We focused on service provision research as an important area for mapping out advancements directly relevant to clinical care.”
BackgroundChan, B., Schweter, S., & Möller, T. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.598 · Cited by 187
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
“Creating an audit trail of the LLM development process to provide chronological documentary evidence of the development of an LLM’s capabilities, including information about its intended purpose, design specifications and choices, as well as training data and testing procedures through the generation of model cards [8…”
MethodThe diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review
Wallace, W., Chan, C., Chidambaram, S., Hanna, L., Iqbal, F. M., Acharya, A., Normahani, P., Ashrafian, H., Markar, S. R., Sounderajah, V., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00667-w · Cited by 186
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models
Wang, Z. J., Montoya, E., Munechika, D., Yang, H., Hoover, B., & Chau, D. H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.51 · Cited by 171
Alderman, J. E., Palmer, J., Laws, E., McCradden, M. D., Ordish, J., Ghassemi, M., Pfohl, S. R., Rostamzadeh, N., Cole-Lewis, H., Glocker, B., et al. (2025) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00224-3 · Cited by 170
Ethical and legal challenges of AI in marketing: an exploration of solutions
Kumar, D., & Suthar, N. (2024) · Journal of Information, Communication and Ethics in Society · doi:10.1108/jices-05-2023-0068 · Cited by 180
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
Kreutzer, J., Caswell, I., Wang, L., Wahab, A., van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00447 · Cited by 179
“, 2018), as well as generalist work such as (Gebru et al., 2018).”
BackgroundQuality at a Glance: An Audit of Web-Crawled Multilingual Datasets
Kreutzer, J., Caswell, I., Wang, L., Wahab, A., van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.12028 · Cited by 179
“, 2018), as well as generalist work such as (Gebru et al., 2018).”
BackgroundAI-driven healthcare: A review on ensuring fairness and mitigating bias
Chinta, S. V., Wang, Z., Palikhe, A., Zhang, X., Kashif, A., Smith, M. A., Liu, J., & Zhang, W. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000864 · Cited by 171
“This inclusion helps in identifying and addressing potential blind spots in AI training datasets, which, if overlooked, could perpetuate biases and inequalities in healthcare delivery [103].”
Ethics by design for artificial intelligence
Brey, P., & Dainow, B. (2024) · AI and Ethics · doi:10.1007/s43681-023-00330-4 · Cited by 169
The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review
Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01196-4 · Cited by 169
“While some initiatives cover the entire ML system and development pipeline (e.g., MINIMAR 39 , FactSheets 40 ), others are concerned with documentation surrounding the model (e.g., Model Cards 41 ), and still others concentrate on the documentation of datasets (e.g., Datasheets 42 , STANDING Together 43,44 , Dataset N…”
BackgroundInfluentialAligning AI With Shared Human Values
Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2008.02275 · Cited by 517
“We follow the recommendations of Gebru et al. (2018) and provide a datasheet for the ETHICS dataset in this section.”
MethodCBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark
Zhang, N., Chen, M., Bi, Z., Liang, X., Li, L., Shang, X., Yin, K., Tan, C., Xu, J., Huang, F., et al. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.544 · Cited by 272
Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers
Solanki, P., Grundy, J., & Hussain, W. (2023) · AI and Ethics · doi:10.1007/s43681-022-00195-z · Cited by 157
“The full documentation processes are outlined in [45, 164] for ML datasets, and in [110, 111] for NLP data.”
BackgroundMethodQA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2022) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 327
“The field of AI ethics has drawn much attention to undesirable social biases and called for documenting the speaker demographics [24, 102].”
BackgroundAlgorithmic realism: expanding the boundaries of algorithmic thought
Green, B., & Viljoen, S. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372840 · Cited by 161
“Speci c interventions of this sort include the methods of algorithmic fairness and approaches to improve data and model documentation [46, 69, 99].”
MethodMERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound
Zellers, R., Lu, J., Lu, X., Yu, Y., Zhao, Y., Salehi, M., Kusupati, A., Hessel, J., Farhadi, A., & Choi, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01589 · Cited by 160
“For a Datasheet [41], please see the MERLOT paper [128].”
BackgroundMethodImage Representations Learned With Unsupervised Pre-Training Contain Human-like Biases
Steed, R., & Caliskan, A. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445932 · Cited by 270
“Our method can be used to assess task-agnostic biases contained in a dataset to enhance transparency [28, 51], but bias mitigation for unsupervised transfer learning is a challenging open problem.”
Method
Metadata
- Authors
- Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K.
- Affiliations
- DAIR Institute (US)
- University of Washington (US)
- Cornell University (US)
- Microsoft Research New York City (United States) (US)
- University of Maryland, College Park, MD
- University of Southern California (US)
- ORCID
- Venue
- Communications of the ACM
- Published
- 2021-11-19
- Volume
- 64
- Issue
- 12
- Pages
- 86-92
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2795038878
- Semantic Scholar
- 0df347f5e3118fac7c351917e3a497899b071d1e
- Cited by
- 2,469
- References
- 37
- Citations by year
- 2026: 788 · 2025: 402 · 2024: 298 · 2023: 379 · 2022: 221 · 2021: 217
- FWCI
- 252.30
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Communications of the ACM (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation
- Alfred P. Sloan Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Artificial Intelligence › Semantic Web and Ontologies
- Keywords
- dataset documentation, data provenance, data sharing, data reuse, metadata
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.