Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
In this paper, we propose data statements as a design solution and professional practice for natural language processing technologists, in both research and development. Through the adoption and widespread use of data statements, the field can begin to address critical scientific and ethical issues that result from the use of data from certain populations in the development of technology for other populations. We present a form that data statements can take and explore the implications of adopting them as part of regular practice. We argue that data statements will help alleviate issues related to exclusion and bias in language technology, lead to better precision in claims about how natural language processing research can generalize and thus better engineering results, protect companies from public embarrassment, and ultimately lead to language technology that meets its users in their own preferred linguistic style and furthermore does not misrepresent them to others.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“or lead to adverse effects from deploying suboptimal or harmful models (Bender and Friedman, 2018) on tasks for which they are ill-suited (Raji et al., 2022).”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Bender and Friedman, 2018
The cited source says what the claim attributes to it.
In the citing paper
“Numerous authors have observed that ML systems perform better for users whose demographic characteristics match those represented in the training data [9,36].”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Thumbs up?: sentiment classification using machine learning techniques
Pang, B., Lee, L., & Vaithyanathan, S. (2002) · Proceedings of the ACL-02 conference on Empirical methods in natural language processing - EMNLP '02 · doi:10.3115/1118693.1118704 · Cited by 7,024 · cited 1×
“Here, our focus is on linguistic data. the review (e.g., Pang et al., 2002) or the hashtag #sarcasm used to identify sarcastic language (e.g., Kreuz and Caucci, 2007).”
BackgroundThe Study of Second Language Acquisition
Brown, C., & Ellis, R. (1996) · The Modern Language Journal · doi:10.2307/329065 · Cited by 4,199
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
The Social Stratification of English in New York City.
Crockett, H. J., & Labov, W. (1968) · American Sociological Review · doi:10.2307/2092899 · Cited by 3,084
Accurate unlexicalized parsing
Klein, D., & Manning, C. D. (2003) · Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - ACL '03 · doi:10.3115/1075096.1075150 · Cited by 3,008
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter
Waseem, Z., & Hovy, D. (2016) · Proceedings of the NAACL Student Research Workshop · doi:10.18653/v1/n16-2013 · Cited by 1,655 · cited 4×
“Waseem and Hovy (2016) provide the following complete list of terms used in their initial scrape: MKR, asian drive, feminazi, immigrant, nigger, sjw, WomenAgainstFeminism, blameonenotall, islam terrorism, notallmen, victimcard, victim card, arab terror, gamergate, jsil, racecard, race card. dataset…”
BackgroundMethodInfluential
Cited by
Items 201–300 of 865
Chen, C., & Sundar, S. S. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580805 · Cited by 36
“This approach is echoed by the recently proposed documentation paradigm in XAI community, which advocates detailed reporting of training data and performance characteristics of machine learning models at the outset of model development [5, 19, 22, 41, 45].”
MethodLeins, K., Lau, J. H., & Baldwin, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.261 · Cited by 45
Dingemanse, M., & Liesenfeld, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.385 · Cited by 66
Bandy, J., & Vincent, N. (2021) · arXiv · doi:10.48550/arxiv.2105.05241 · Cited by 36
“Table [1] highlights the summary statistics of our corpus.”
BackgroundRaji, I. D., & Yang, J. (2019) · arXiv · doi:10.48550/arxiv.1912.06166 · Cited by 36
“A common focus across templates, even outside of data-specific work, is on clarifying the details of data provenance, for both the training and testing data used in ML development [4, 2, 11].”
BackgroundTeaching Responsible Data Science: Charting New Pedagogical Territory
Lewis, A., & Stoyanovich, J. (2021) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00241-7 · Cited by 62
van der Goot, R., Sharaf, I., Imankulova, A., Üstün, A. B., Stepanović, M., Ramponi, A., Khairunnisa, S. O., Komachi, M., & Plank, B. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.197 · Cited by 80
Hupont, I., Tolan, S., Güneş, H., & Gómez, E. (2022) · Scientific Reports · doi:10.1038/s41598-022-14981-6 · Cited by 56
AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level
Seker, A., Bandel, E., Bareket, D., Brusilovsky, I., Greenfeld, R. S., & Tsarfaty, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.4 · Cited by 73
Global Challenges in the Standardization of Ethics for Trustworthy AI
Lewis, D., Hogan, L., Filip, D., & Wall, P. J. (2020) · Journal of ICT Standardization · doi:10.13052/jicts2245-800x.823 · Cited by 34
“Such standardization effort can build on existing proposals for supporting trustworthiness in the exchange of AI training data sets [22][23].”
BackgroundUsing audio stimuli in acceptability judgment experiments
Sedarous, Y., & Namboodiripad, S. (2020) · Language and Linguistics Compass · doi:10.1111/lnc3.12377 · Cited by 37
The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool
Gerchick, M., Jegede, T., Shah, T., Gutierrez, A., Beiers, S., Shemtov, N., Xu, K., Samant, A., & Horowitz, A. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594081 · Cited by 34
“Viewing design decisions as policy choices, a related body of work focuses on the actions and functions of data scientists, engineers, researchers, and other actors involved in the design of algorithmic systems, including in shaping the datasets used to build machine learning (ML) systems [11, 67, 68, 77], defining th…”
BackgroundAssessing Algorithmic Fairness in Automatic Classifiers of Educational Forum Posts
Sha, L., Rakovic, M., Whitelock-Wainwright, A., Carroll, D., Yew, V. M., Gasevic, D., & Chen, G. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-78292-4_31 · Cited by 34
Hassan, S., Huenerfauth, M., & Alm, C. O. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.267 · Cited by 32
CounterFAccTual: How FAccT Undermines Its Organizing Principles
Gansky, B., & McDonald, S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533241 · Cited by 34
Mitigating Gender Bias in Natural Language Processing: Literature ReviewPreprint
Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.-W., & Wang, W. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.08976 · Cited by 257
“Other important aspects such as model/data transparency (Mitchell et al., 2019; Bender and Friedman, 2018) and privacy preservation (Reddy and Knight, 2016; Elazar and Goldberg, 2018; Li et al.”
BackgroundToward Artificial Empathy for Human-Centered Design
Zhu, Q., & Luo, J. (2023) · Journal of Mechanical Design · doi:10.1115/1.4064161 · Cited by 44
NUBIA: NeUral Based Interchangeability Assessor for Text Generation
Kane, H., Kocyigit, M. Y., Abdalla, A., Ajanoh, P., & Coulibali, M. (2020) · arXiv · doi:10.48550/arxiv.2004.14667 · Cited by 32
“This description should not only include architectures but also training data and fine-tuning data (Mitchell et al., 2019; Gebru et al., 2018; Bender and Friedman, 2018).”
BackgroundNLPositionality: Characterizing Design Biases of Datasets and Models
Santy, S., Liang, J. T., Bras, R. J. L., Reinecke, K., & Sap, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.505 · Cited by 165
Olá, Bonjour, Salve! XFORMAL: A Benchmark for Multilingual Formality Style Transfer
Briakou, E., Lu, D., Zhang, K., & Tetreault, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.256 · Cited by 89
Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Papers
Peng, K., Mathur, A., & Narayanan, A. (2021) · arXiv · doi:10.48550/arxiv.2108.02922 · Cited by 31
“Researchers have worked to make sense of ethical considerations involved in dataset creation [43, 69, 32], have proposed ways to identify and mitigate biases in datasets [11, 82], have developed means to protect the privacy of individuals in datasets [70, 91], and have improved methods to document datasets [35, 47, 12…”
BackgroundResources for Turkish natural language processing: A critical survey
Çöltekin, Ç., Doğruöz, A. S., & Çetinoğlu, Ö. (2022) · Language Resources and Evaluation · doi:10.1007/s10579-022-09605-4 · Cited by 49
“Being aware of the earlier recommendations (e.g., Ide et al., 2017; Bender and Friedman, 2018; Gebru et al., 2020) for resource creation efforts and their descriptions would be useful for any annotation or curation project.”
BackgroundMultilingual Open Information Extraction: Challenges and Opportunities
Claro, D. B., Souza, M., Xavier, C. C., & Oliveira, L. J. G. D. (2019) · Information · doi:10.3390/info10070228 · Cited by 29
“Notice that the focus of the area on the English language (particularly, based on prose using in journalistic or encyclopedic texts) and its particular characteristics, although natural from an engineering point of view due to the availability of resources, may induce important bias in the area [70].”
BackgroundOn the Ethical Limits of Natural Language Processing on Legal Text
Tsarapatsanis, D., & Αλέτρας, Ν. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.314 · Cited by 64
Ruddit: Norms of Offensiveness for English Reddit Comments
Hada, R., Sudhir, S., Mishra, P., Yannakoudakis, H., Mohammad, S. M., & Shutova, E. (2021) · doi:10.18653/v1/2021.acl-long.210 · Cited by 57
“We follow the format provided by Bender and Friedman (2018) to discuss the ethical considerations for our dataset.”
MethodUnderstanding and identifying the use of emotes in toxic chat on Twitch
Kim, J., Wohn, D. Y., & Cha, M. (2021) · Online Social Networks and Media · doi:10.1016/j.osnem.2021.100180 · Cited by 38
Interrogating Data Work as a Community of Practice
Rothschild, A., Meng, A., DiSalvo, C. F., Johnson, B., Shapiro, B. R., & DiSalvo, B. J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555198 · Cited by 39
“There are other contextualization practices proposed for large-scale datasets: Bender & Friedman proposed data statements, essentially a short biography, to accompany NLP datasets [1] and Gebru et al.”
MethodThe Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning
Hessel, J., Hwang, J. D., Park, J. S., Zellers, R., Bhagavatula, C., Rohrbach, A., Saenko, K., & Choi, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_32 · Cited by 76
A domain-specific language for describing machine learning datasets
Giner-Miguelez, J., Gómez, A., & Cabot, J. (2023) · Journal of Computer Languages · doi:10.1016/j.cola.2023.101209 · Cited by 45
“For instance, theData Statementswork [3] emphasizes the need to annotate natural language datasets with additional metadata such as the demographics of data gatherers and data annotators (those labelling the data to prepare it for the training phase), and the specific context of the text in the dataset.”
BackgroundBigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.15877 · Cited by 29
“We follow [11] to create the datacard for BigCodeBench , where we tend to summarize and centralize all information that might be relevant for the benchmark analysis.”
MethodSmallset Timelines: A Visual Representation of Data Preprocessing Decisions
Lucchesi, L. R., Kuhnert, P., Davis, J. L., & Xie, L. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533175 · Cited by 37
“Datasheets for datasets [19], model cards [34], FactSheets [1], data statements [2], and the Dataset Nutrition Label [25] are proposed templates and frameworks for recording information about a dataset, including changes made to it.”
MethodSihab-Us-Sakib, S., Rahman, M. R., Forhad, M. S. A., & Aziz, M. A. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100104 · Cited by 50
Constantinides, M., Bogucka, E. P., Quercia, D., Kallio, S., & Tahaei, M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686927 · Cited by 59
“In total, we formulated a set of 16 techniques based on relevant literature sources [5, 10, 19, 24, 31, 34, 41, 42, 44, 54, 59, 66, 67, 84, 93, 94].”
A Statutory Article Retrieval Dataset in French
Louis, A., & Spanakis, G. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.468 · Cited by 93
Documenting Data Production Processes: A Participatory Approach for Data Work
Miceli, M., Yang, T., Alvarado Garcia, A., Posada, J., Wang, S. M., Pohl, M., & Hanna, A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555623 · Cited by 28
A Methodology for Creating AI FactSheets
Richards, J., Piorkowski, D., Hind, M., Houde, S., & Mojsilović, A. (2020) · arXiv · doi:10.48550/arxiv.2006.13796 · Cited by 28
“Our mechanism for transparent AI documentation, called FactSheets [1], takes a more general approach to AI transparency than previous work [2, 5, 7, 11, 14] in several ways.”
BackgroundMethodPerspectivist approaches to natural language processing: a survey
Frenda, S., Abercrombie, G., Basile, V., Pedrani, A., Panizzon, R., Cignarella, A. T., Marco, C. S., & Bernardi, D. (2024) · Language Resources and Evaluation · doi:10.1007/s10579-024-09766-4 · Cited by 127
“…step towards transparency about the perspectives involved in the datasets is the reflection provided by Prabhakaran et al. (2021), who propose a set of recommendations to increase the transparency of datasets, following up on the existing data statement presented in Bender and Friedman (2018).”
ASHABot: An LLM-Powered Chatbot to Support the Informational Needs of Community Health Workers
Ramjee, P., Chhokar, M., Sachdeva, B., Meena, M., Abdullah, H., Vashistha, A., Nagar, R., & Jain, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713680 · Cited by 69
“While researchers have proposed developer-centric tools like design playbooks [36] and documentation protocols [10, 37] to enhance AI safety, the effectiveness of these tools remains underexplored [11], and AI safety continues to be a complex, evolving challenge, particularly in high-stakes,…”
DaN+: Danish Nested Named Entities and Lexical Normalization
Plank, B., Jensen, K. N., & van der Goot, R. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.583 · Cited by 50
“Following (Bender and Friedman, 2018), the following outlines the data statement for DAN+: A.”
BackgroundHell Hath No Fury? Correcting Bias in the NRC Emotion Lexicon
Zad, S., Jimenez, J., & Finlayson, M. A. (2021) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2021.woah-1.11 · Cited by 40
“…furthermore, might be reflective of relatively obvious biases, which in light of recent work demonstrating the built-in biases of various AI and NLP resources (Bolukbasi et al., 2016; Bender and Friedman, 2018; Mehrabi et al., 2019; Blodgett et al., 2020), it would be good to try to correct for.”
BackgroundAugmented Datasheets for Speech Datasets and Ethical Decision-Making
Papakyriakopoulos, O., Choi, A. S. G., Thong, W. E., Zhao, D., Andrews, J. T. A., Bourke, R., Xiang, A., & Koenecke, A. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594049 · Cited by 58
“posals have since followed, including nutrition labels [99], data statements [17], data cards [210], data briefs [68], model info sheets [117], and causal datasheets [36].”
BackgroundThe “Collections as ML Data” checklist for machine learning and cultural heritage
Lee, B. C. G. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24765 · Cited by 75
“…and best practices that machine learning researchers and practitioners (those who utilize machine learning in research, experiments, and operational contexts) can consult while creating datasets, training models, and operationalizing these systems (Bender & Friedman, 2018; Mitchell et al., 2019).”
InfluentialSaura, J. R., Palacios‐Marqués, D., & Soriano, D. R. (2023) · The Journal of Technology Transfer · doi:10.1007/s10961-023-10023-z · Cited by 43
Handling and Presenting Harmful Text in NLP Research
Kirk, H., Birhane, A., Vidgen, B., & Derczynski, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.35 · Cited by 25
“Each section is designed to be filled in as a statement (such as in a data statement (Bender and Friedman, 2018) or model card (Mitchell et al.”
BackgroundIbrahim, S. T., Li, M., Patel, J., & Katapally, T. R. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.109859 · Cited by 42
Shimorina, A., & Belz, A. (2022) · Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval) · doi:10.18653/v1/2022.humeval-1.6 · Cited by 26
On learning and representing social meaning in NLP: a sociolinguistic perspective
Nguyen, D., Rosseel, L., & Grieve, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.50 · Cited by 56
“One key step is documenting the datasets (Bender and Friedman, 2018; Gebru et al., 2018).”
BackgroundMultiLexNorm: A Shared Task on Multilingual Lexical Normalization
van der Goot, R., Ramponi, A., Zubiaga, A., Plank, B., Müller, B., Vicente, I. S., Ljubešić, N., Çetinoğlu, Ö., Mahendra, R., Çolakoğlu, T., et al. (2021) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2021.wnut-1.55 · Cited by 53
Multisource AI Scorecard Table for System EvaluationPreprint
Blasch, E., Sung, J., & Nguyen, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.03985 · Cited by 39
“These efforts include: Betting understanding the data that underlies a machine learning system through Datasheets [36], data statements [37], and “nutrition labels”[38]; Clarifying the intended use cases of machine learning algorithms and minimizing their usage in contexts for which they are…”
BackgroundCroissant: A Metadata Format for ML-Ready Datasets
Akhtar, M., Benjelloun, O., Conforti, C., Gijsbers, P., Giner-Miguelez, J., Jain, N., Kuchnik, M., Lhoest, Q., Marcenac, P., Maskey, M., et al. (2024) · Workshop on Data Management for End-to-End Machine Learning (DEEM) · doi:10.1145/3650203.3663326 · Cited by 104
“Data statements [21], Datasheets [6], and Data cards [22] were proposed to fill the lack of standardized dataset documentation.”
Kraft, A., & Soulier, E. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658981 · Cited by 25
“Besides aforementioned efforts to facilitate more diverse contribution, we also need transparent documentation practices that allow scrutiny of knowledge bases and their original knowers [7, 27].”
BackgroundSystematic analysis of 32,111 AI model cards characterizes documentation practice in AI
Liang, W., Rajani, N. F., Yang, X., Ozoani, E., Wu, E., Chen, Y. T., Smith, D. S., & Zou, J. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00857-z · Cited by 101
“Data-focused documentation tools 23–25 —including datasheets for datasets 18 and data nutrition labels 17,52 —serve as a vehicle for communication between dataset creators and consumers, addressing the lack of industry standards for documenting AI datasets.”
A Survey on Online Aggression: Content Detection and Behavioral Analysis on Social Media
Mane, S., Kundu, S., & Sharma, R. R. (2025) · ACM Computing Surveys · doi:10.1145/3711125 · Cited by 34
“Recognizing the potential influence of annotators’ backgrounds on the annotation process [23, 25], researchers have implemented strategies to mitigate bias.”
Afzal, S., Rajmohan, C., Kesarwani, M., Mehta, S., & Patel, H. (2021) · IEEE International Conference on Smart Data Services (SMDS) · doi:10.1109/smds53860.2021.00016 · Cited by 50
On the Gap between Adoption and Understanding in NLP
Bianchi, F., & Hovy, D. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.340 · Cited by 35
“A way forward Following Bender and Friedman (2018) on providing a data statement, we believe that a code statement similar to that offered by Mitchell et al. (2019) should be provided along with a paper.”
Sachdeva, P., Barreto, R., von Vacano, C., & Kennedy, C. J. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533216 · Cited by 43
“of their background and perspective on label generation, has long been underappreciated within the machine learning community [7, 49].”
BackgroundD’Arcy, A., & Bender, E. M. (2022) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-015324 · Cited by 43
Self-citationGround Truth Or Dare: Factors Affecting The Creation Of Medical Datasets For Training AI
Zając, H. D., Avlona, N.-R., Kensing, F., Andersen, T. O., & Shklovski, I. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604766 · Cited by 49
“This has implications for scholarly discussions that focus on developing documentation frameworks that support the responsible and informed use of complex datasets [2, 5, 20, 31, 44].”
BackgroundSystemic Biases in Sign Language AI Research: A Deaf-Led Call to Reevaluate Research Agendas
Desai, A., De Meulder, M., Hochgesang, J. A., Kocab, A., & Lu, A. X. (2024) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/4qt6t64rxwd2 · Cited by 25
Singhal, P., Gokhale, S., Shah, A., Jain, D. K., Walambe, R. A., Ekárt, A., & Kotecha, K. V. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-06659-1 · Cited by 29
“Bias mitigating methods in NLP work on (non-exhaustive) multiple strategies: • Debias training corpora [32], • Debias word embeddings [3, 25], • Including data statements [33], • Re-weighting data points based on their distribution using leaned latent distribution [34].”
RuSemShift: a dataset of historical lexical semantic change in Russian
Rodina, J., & Kutuzov, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.90 · Cited by 25
Reliability Testing for Natural Language Processing Systems
Tan, S., Joty, S., Baxter, K., Taeihagh, A., Bennett, G. A., & Kan, M. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.321 · Cited by 49
Implementing Ai Principles: Frameworks, Processes, and ToolsPreprint
Boza, P., & Evgeniou, T. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3783124 · Cited by 36
Ali, H., Pham, D.-T., Alam, S., Schultz, M., Li, M. Z., Wang, Y., Itoh, E., & Dương, V. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103256 · Cited by 32
Assessing risk of bias in toxicological studies in the era of artificial intelligence
Härtung, T., Hoffmann, S., & Whaley, P. (2025) · Archives of Toxicology · doi:10.1007/s00204-025-03978-5 · Cited by 33
Measuring and mitigating language model biases in abusive language detection
Song, R., Giunchiglia, F., Li, Y., Shi, L., & Xu, H. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103277 · Cited by 32
Pistilli, G., Ferrandis, C. M., Jernite, Y., & Mitchell, M. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594002 · Cited by 36
Evaluation of African American Language Bias in Natural Language Generation
Deas, N., Grieser, J. A., Kleiner, S., Patton, D. U., Turcan, E., & McKeown, K. R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.421 · Cited by 92
Zampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., & Çöltekin, Ç. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3950378 · Cited by 27
Data Authenticity, Consent, and Provenance for AI Are All Broken: What Will It Take to Fix Them?
Longpre, S., Mahari, R., Obeng-Marnu, N., Brannon, W., South, T., Kabbara, J., & Pentland, A. (2024) · doi:10.21428/e4baedd9.a650f77d · Cited by 40
Features or Spurious Artifacts? Data-centric Baselines for Fair and Robust Hate Speech Detection
Ramponi, A., & Tonelli, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.221 · Cited by 52
“…future research on the topic, we also release (iv) code to reproduce all experiments, and (v) disaggregated lexical artifact annotations, more broadly (vi) suggesting the inclusion of dataset artifacts in data statements (Bender and Friedman, 2018), which can be easily revealed using our codebase.”
MethodResultThe landscape of data and AI documentation approaches in the European policy context
Micheli, M., Hupont, I., Delipetrev, B., & Soler-Garrido, J. (2023) · Ethics and Information Technology · doi:10.1007/s10676-023-09725-7 · Cited by 42
“The results highlight that many initiatives are also suitable for authorities and auditors (e.g., Baracaldo et al. (2022), Bender and Friedman (2018), Grasso et al. (2020), Stoy-anovich and Howe (2019)), which in practice require specific information elements at a sufficient level of detail to carry…”
BackgroundInfluentialData Statements: From Technical Concept to Community Practice
McMillan-Major, A., Bender, E. M., & Friedman, B. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3594737 · Cited by 21
“…Bender and Friedman as “a characterization of a dataset that provides context to allow developers and users to better understand how experimental results might generalize, how software might be appropriately deployed, and what biases might be reflected in systems built on the software” [2, p. 587].”
BackgroundMethodResultInfluentialSelf-citationvan Miltenburg, E., van der Lee, C., & Krahmer, E. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.51 · Cited by 56
Armengol-Estapé, J., Carrino, C. P., Rodriguez-Penagos, C. G., Bonet, O. D. G., Armentano-Oller, C., González-Agirre, A., Melero, M., & Villegas, M. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.437 · Cited by 24
Utilizing subjectivity level to mitigate identity term bias in toxic comments classification
Zhao, Z., Zhang, Z., & Hopfgartner, F. (2022) · Online Social Networks and Media · doi:10.1016/j.osnem.2022.100205 · Cited by 29
Adaptive Sampling Strategies to Construct Equitable Training Datasets
Cai, W., Encarnación, R., Chern, B., Corbett‐Davies, S., Bogen, M., Bergman, S., & Goel, S. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533203 · Cited by 43
Understanding Implementation Challenges in Machine Learning Documentation
Chang, J., & Custis, C. (2022) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3551624.3555301 · Cited by 43
“Templates such as Datasheets for Datasets [12], Dataset Nutrition Labels [5], and Data statements [3] have been created as structured formats for documenting dataset’s motivation, composition, collection process, recommended uses, characteristics of actors involved, ethical concerns.”
BackgroundMethod(2021) · arXiv (Cornell University) · Cited by 21
Kuehnert, B., Kim, R., Forlizzi, J., & Heidari, H. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732191 · Cited by 47
Barclay, I., Taylor, H., Preece, A. D., Taylor, I., Verma, D. C., & de Mel, G. R. (2020) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.6129 · Cited by 26
Lancaster, C., Schulenberg, K., Flathmann, C., McNeese, N. J., & Freeman, G. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3632121 · Cited by 38
“…to achieve this could be through constructing and sharing in collaborative organizational case studies, utilizing data statements on development projects, and other immersive approaches to understanding the potential ramifications of their work validated in the AI ethics literature [19, 26, 99].”
BackgroundParticipant flow diagrams for health equity in AI
Ellen, J., de Matos, J. C. R. G., Viola, M., Gallifant, J., Quion, J., Celi, L. A., & Hussein, N. S. A. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104631 · Cited by 23
Nee, J., Smith, G. M., Sheares, A. M., & Rustagi, I. (2021) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3465416.3483301 · Cited by 24
“Developers can document this information using tools like data statements for NLP and be transparent about the limitations of the tools built with a given dataset [16].”
MethodThe Sanction of Authority: Promoting Public Trust in AIPreprint
Knowles, B., & Richards, J. T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.04221 · Cited by 20
Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias
González, A. V., Barrett, M., Hvingelby, R., Webster, K., & Søgaard, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.209 · Cited by 31
Understanding who uses Reddit: Profiling individuals with a self-reported bipolar disorder diagnosis
Jagfeld, G., Lobban, F. A., Rayson, P., & Jones, S. H. (2021) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.18653/v1/2021.clpsych-1.1 · Cited by 34
Mohammad, S. M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.573 · Cited by 54
A Large-Scale English Multi-Label Twitter Dataset for Cyberbullying and Online Abuse Detection
Salawu, S., Lumsden, J., & He, Y. (2021) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2021.woah-1.16 · Cited by 19
“As identified by Bender and Friedman (2018), researchers should therefore be mindful of potential annotators’ biases when creating online abuse datasets.”
BackgroundFairEd: A Systematic Fairness Analysis Approach Applied in a Higher Educational Context
Vásquez, J., Gitiaux, X., Ortega, C., & Rangwala, H. (2022) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3506860.3506902 · Cited by 29
“[8] and [30] define frameworks to help stakeholders provide a context around the data.”
BackgroundBLIP: Facilitating the Exploration of Undesirable Consequences of Digital Technologies
Pang, R. Y., Santy, S., Just, R., & Reinecke, K. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642054 · Cited by 30
“Additionally, there are calls for researchers within different computing communities to accurately report the design considerations of their datasets [9, 38], models [67, 89], and tasks [56, 68] as well as evaluate and de-bias their products [4, 8, 104, 111].”
BackgroundTowards Generating Consumer Labels for Machine Learning Models
Seifert, C., Scherzinger, S., & Wiese, L. (2019) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi48466.2019.00033 · Cited by 31
“Similarly, Bender and Friedman proposed Data Statements [7] specifically tailored toward data sets for natural language processing, adding for instance speaker characteristics.”
BackgroundInfluentialUniversal Joy A Data Set and Results for Classifying Emotions Across Languages
Lamprinidis, S., Bianchi, F., Hardt, D., & Hovy, D. (2021) · CBS Research Portal (Copenhagen Business School) · Cited by 20
Guiding Principles for Participatory Design-inspired Natural Language Processing
Caselli, T., Cibin, R., Conforti, C., Encinas, E., & Teli, M. (2021) · Workshop on NLP for Positive Impact (NLP4PI) · doi:10.18653/v1/2021.nlp4posimpact-1.4 · Cited by 51
“It is known that NLP tools struggle with tail phenomena (Ettinger et al., 2017) and are subject to bias (Bender and Friedman, 2018).”
BackgroundFairness and social bias quantification in Large Language Models for sentiment analysis
Radaideh, M. I., Kwon, O. H., & Radaideh, M. I. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113569 · Cited by 37
Benchmarking Machine Reading Comprehension: A Psychological Perspective
Sugawara, S., Stenetorp, P., & Aizawa, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.137 · Cited by 29
“The rubric is in a similar motivation with dataset statements (Bender and Friedman, 2018; Gebru et al., 2018); however, taking the validity aspects into account would improve its substance.”
MethodDescribeML: a tool for describing machine learning datasets
Giner-Miguelez, J., Gómez, A., & Cabot, J. (2022) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C) · doi:10.1145/3550356.3559087 · Cited by 30
“As another example, a natural language dataset gathered from Australian speakers could drop the accuracy of models trained to support users of the United States due to the different language styles [1].”
BackgroundMethodAssessing and Mitigating Bias in Artificial Intelligence: A Review
Sinha, A., Sapra, D., Sinwar, D., Singh, V., & Raghuwanshi, G. (2023) · Recent Advances in Computer Science and Communications · doi:10.2174/2666255816666230523114425 · Cited by 24
The Sentiment Problem: A Critical Survey towards Deconstructing Sentiment Analysis
Venkit, P. N., Srinath, M., Gautam, S., Venkatraman, S., Gupta, V. K., Passonneau, R. J., & Wilson, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.848 · Cited by 41
Metadata
- Authors
- Bender, E. M., & Friedman, B.
- Affiliations
- University of Washington (US)
- Corresponding author
- Bender, Emily M.
- Friedman, Batya
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2018-12
- Volume
- 6
- Pages
- 587-604
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2911227954
- Semantic Scholar
- 97bfa89addc6e5d76361e4c1e296949cad887b86
- Cited by
- 854
- References
- 46
- Citations by year
- 2026: 108 · 2025: 96 · 2024: 102 · 2023: 154 · 2022: 127 · 2021: 177 · 2020: 68 · 2019: 20 · 2018: 1
- FWCI
- 132.62
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Science Foundation (1302709)
- University of Washington
- York University
- Macquarie University
- University of California, San Diego
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- natural language processing, algorithmic bias, underrepresented groups, ethical practice, linguistic diversity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.