Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
In this paper, we propose data statements as a design solution and professional practice for natural language processing technologists, in both research and development. Through the adoption and widespread use of data statements, the field can begin to address critical scientific and ethical issues that result from the use of data from certain populations in the development of technology for other populations. We present a form that data statements can take and explore the implications of adopting them as part of regular practice. We argue that data statements will help alleviate issues related to exclusion and bias in language technology, lead to better precision in claims about how natural language processing research can generalize and thus better engineering results, protect companies from public embarrassment, and ultimately lead to language technology that meets its users in their own preferred linguistic style and furthermore does not misrepresent them to others.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“or lead to adverse effects from deploying suboptimal or harmful models (Bender and Friedman, 2018) on tasks for which they are ill-suited (Raji et al., 2022).”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Bender and Friedman, 2018
The cited source says what the claim attributes to it.
In the citing paper
“Numerous authors have observed that ML systems perform better for users whose demographic characteristics match those represented in the training data [9,36].”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Thumbs up?: sentiment classification using machine learning techniques
Pang, B., Lee, L., & Vaithyanathan, S. (2002) · Proceedings of the ACL-02 conference on Empirical methods in natural language processing - EMNLP '02 · doi:10.3115/1118693.1118704 · Cited by 7,024 · cited 1×
“Here, our focus is on linguistic data. the review (e.g., Pang et al., 2002) or the hashtag #sarcasm used to identify sarcastic language (e.g., Kreuz and Caucci, 2007).”
BackgroundThe Study of Second Language Acquisition
Brown, C., & Ellis, R. (1996) · The Modern Language Journal · doi:10.2307/329065 · Cited by 4,199
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
The Social Stratification of English in New York City.
Crockett, H. J., & Labov, W. (1968) · American Sociological Review · doi:10.2307/2092899 · Cited by 3,084
Accurate unlexicalized parsing
Klein, D., & Manning, C. D. (2003) · Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - ACL '03 · doi:10.3115/1075096.1075150 · Cited by 3,008
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter
Waseem, Z., & Hovy, D. (2016) · Proceedings of the NAACL Student Research Workshop · doi:10.18653/v1/n16-2013 · Cited by 1,655 · cited 4×
“Waseem and Hovy (2016) provide the following complete list of terms used in their initial scrape: MKR, asian drive, feminazi, immigrant, nigger, sjw, WomenAgainstFeminism, blameonenotall, islam terrorism, notallmen, victimcard, victim card, arab terror, gamergate, jsil, racecard, race card. dataset…”
BackgroundMethodInfluential
Cited by
Items 301–400 of 865
What is research data “misuse”? And how can it be prevented or mitigated?
Pasquetto, I. V., Cullen, Z., Thomer, A. K., & Wofford, M. F. (2024) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24944 · Cited by 24
“…in the open science and data curation literature, researchers in the machine learning fairness community also suggest improving data documentation practices as ways to prevent data misuse (Bender & Friedman, 2018; Gebru et al., 2021; Mitchell et al., 2019; Paullada et al., 2021; Peng et al., 2021).”
Detecting Fine-Grained Cross-Lingual Semantic Divergences without Supervision by Learning to Rank
Briakou, E., & Carpuat, M. J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.121 · Cited by 33
Investigating Robustness of Dialog Models to Popular Figurative Language Constructs
Jhamtani, H., Gangal, V., Hovy, E. H., & Berg-Kirkpatrick, T. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.592 · Cited by 33
“On replacing figurative language with its literal counterpart into such contexts, the replaced literal text, which is typically English of a more formal and written variety, ends up being much more out of sync with the context in the lexico-syntactic/stylistic sense than it is for DailyDialog.”
MethodInfluential“It’s Like the Value System in the Loop”: Domain Experts’ Values Expectations for NLP Automation
Showkat, D., & Baumer, E. P. S. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533483 · Cited by 25
“Even if it improves efciency or provides novel insights [16], a system that is trained with one demographic and is tested on a diferent demographic can perpetuate, reproduce, or even enhance biased outcomes [13, 14, 120].”
BackgroundScoping review on natural language processing applications in counselling and psychotherapy
Laricheva, M., Liu, Y., Shi, E., & Wu, A. D. (2024) · British Journal of Psychology · doi:10.1111/bjop.12721 · Cited by 26
“When the sample data for training are not carefully examined before NLP, it may introduce selection bias (Bender & Friedman, 2018) and lead to poor performance for certain demographic groups who are not well represented in the sample.”
SD-QA: Spoken Dialectal Question Answering for the Real World
Faisal, F., Keshava, S., Alam, M. M. I., & Anastasopoulos, A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.281 · Cited by 61
Brintrup, A. M. (2020) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190066727.013.24 · Cited by 22
What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
Luccioni, A. S., & Viviano, J. D. (2021) · arXiv · doi:10.48550/arxiv.2105.02732 · Cited by 17
“…high quality and safe neural language models will likely require the community to adopt more mindful data collection practices (Gehman et al., 2020; Bender and Friedman, 2018; Gebru et al., 2018; Jo and Gebru, 2020; Paullada et al., 2020; Bender et al., 2021), establish standardized filtering…”
BackgroundChatGPT’s performance in university admissions tests in mathematics
Udías, Á., Alonso‐Ayuso, A., Álfaro, C., Algar, M. J., Cuesta, M., Fernández-Isabel, A., Gómez, J., Lancho, C., Cano, E. L., de Diego, I. M., et al. (2024) · International Electronic Journal of Mathematics Education · doi:10.29333/iejme/15517 · Cited by 32
Improving Governance Outcomes Through AI Documentation: Bridging Theory and Practice
Winecoff, A., & Bogen, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713814 · Cited by 47
“For instance, in the initial case studies of an extensive data documentation framework, practitioners reported that completing the documentation took two to three hours, excluding the time needed to gather the required information [9].”
InfluentialLarsen, B. C. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462591 · Cited by 23
Addressing religious hate online: from taxonomy creation to automated detection
Ramponi, A., Testa, B., Tonelli, S., & Ježek, E. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1128 · Cited by 21
Transcending the “Male Code”: Implicit Masculine Biases in NLP Contexts
Seaborn, K., Chandra, S., & Fabre, É. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581017 · Cited by 29
“previous efforts on data statements [12] and model cards [78] by”
BackgroundMethodLi, Y., Du, M., Wang, X., & Wang, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.797 · Cited by 54
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
Gehrmann, S., Bhattacharjee, A., Mahendiran, A., Wang, A., Papangelis, A., Madaan, A., McMillan-Major, A., Shvets, A., Upadhyay, A., Bohnet, B., et al. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.27 · Cited by 27
Das, D., Guha, S., & Semaan, B. (2023) · Workshop on Cross-Cultural Considerations in NLP (C3NLP) · doi:10.18653/v1/2023.c3nlp-1.8 · Cited by 49
“In this work, we followed (Bender and Friedman, 2018)’s guidelines for ethical considerations that recommend reflecting on curation rationales, language variety, demographic, and text characteristics, among other things.”
BackgroundMasader: Metadata Sourcing for Arabic Text and Speech Data Resources
Alyafeai, Z., Masoud, M., Ghaleb, M., & Al-shaibani, M. S. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3rot2sy5uudb · Cited by 24
Style is NOT a single variable: Case Studies for Cross-Stylistic Language Understanding
Kang, D., & Hovy, E. H. (2021) · doi:10.18653/v1/2021.acl-long.185 · Cited by 48
“Following the data statement (Bender and Friedman, 2018), we cite and introduce individual datasets with their data statistics in Table 2.”
MethodBias and comparison framework for abusive language datasets
Wich, M., Eder, T., Kuwatly, H. A., & Groh, G. (2021) · AI and Ethics · doi:10.1007/s43681-021-00081-0 · Cited by 23
“[18] and [5] proposed datasheets for datasets to document their characteristics because the machine learning or NLP communities do not have a standardized approach.”
BackgroundUncertainty visualization: Fundamentals and recent developments
Hägele, D., Schulz, C., Beschle, C. A., Booth, H., Butt, M., Barth, A., Deußen, O., & Weiskopf, D. (2022) · it - Information Technology · doi:10.1515/itit-2022-0033 · Cited by 23
“Making a differentiation and concomitant visualization of ambiguities and the context in which they are (un)resolvable, is an important factor in the on-going discussions on bias in artificial intelligence [11, 30, 6, 39, 18].”
BackgroundThe Ethics of Computational Social Science
Leslie, D. (2023) · Springer eBooks · doi:10.1007/978-3-031-16624-2_4 · Cited by 20
Saliency Cards: A Framework to Characterize and Compare Saliency Methods
Boggust, A., Suresh, H., Strobelt, H., Guttag, J. V., & Satyanarayan, A. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593997 · Cited by 30
“Saliency cards are complementary to documentation standards for machine learning datasets [8, 11, 22, 29, 33, 35, 50, 60] and models [2, 19, 50, 51, 65, 67].”
BackgroundCrossRE: A Cross-Domain Dataset for Relation Extraction
Bassignana, E., & Plank, B. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.263 · Cited by 47
Bound by the Bounty: Collaboratively Shaping Evaluation Processes for Queer AI Harms
Dennler, N. S., Ovalle, A., Singh, A., Soldaini, L., Subramonian, A., Tu, H., Agnew, W. S., Ghosh, A., Yee, K., Peradejordi, I. F., et al. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604682 · Cited by 36
“To assess AI harms, companies have used bias evaluation benchmarks [6, 21, 46], dataset and model documentation [4, 25, 42], and other auditing processes [41, 50, 51].”
MethodRejecting nativeness to produce a more accurate and just Linguistics
Birkeland, A., Block, A., Craft, J. T., Sedarous, Y., Wang, S., Wu, A., & Namboodiripad, S. (2024) · Language · doi:10.1353/lan.2024.a937197 · Cited by 18
“…about cognition, language, and/or society, while claims about minoritized people or lesser-studied languages are treated as being particularized (e.g. Brekhus 1998, Henrich et al. 2010, Anand et al. 2011, Bender & Friedman 2018, Clancy & Davis 2019, Sedarous & Namboodiripad 2020, Remedios 2022).”
Classifying constructive comments
Kolhatkar, V., Thain, N., Sorensen, J. S., Dixon, L., & Taboada, M. (2023) · First Monday · doi:10.5210/fm.v28i4.13163 · Cited by 42
Fifty Shades of Grey: In Praise of a Nuanced Approach Towards Trustworthy Design
Thornton, L., Knowles, B., & Blair, G. S. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445871 · Cited by 26
“Transparency in particular is leaned on heavily in the trusted AI community, increasingly instantiated through forms of documentation [1, 11, 52, 70, 109].”
BackgroundBeyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources
Yu, X., Chatterjee, T., Asai, A., Hu, J., & Choi, E. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.273 · Cited by 59
Confounding Tradeoffs for Neural Network QuantizationPreprint
Garg, S., Jain, A., Lou, J., & Nahmias, M. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06366 · Cited by 25
Unsilencing colonial archives via automated entity recognition
Luthra, M., Todorov, K., Jeurgens, C., & Colavizza, G. (2023) · Journal of Documentation · doi:10.1108/jd-02-2022-0038 · Cited by 46
Will Artificial Intelligence Be “Better” Than Humans in the Management of Syncope?
Dipaola, F., Gebska, M. A., Gatti, M., Levra, A. G., Parker, W. H., Menè, R., Lee, S., Costantino, G., Barsotti, E. J., Shiffer, D., et al. (2024) · JACC Advances · doi:10.1016/j.jacadv.2024.101072 · Cited by 14
“38 To improve integration of AI in medical decision-making scenarios, the development of models that provide transparent and interpretable results 39 is essential (see Table 1 ).”
BackgroundAboelazm, K. S. (2025) · International Journal of Law and Management · doi:10.1108/ijlma-06-2024-0208 · Cited by 24
Easy, Reproducible and Quality-Controlled Data Collection with CROWDAQ
Ning, Q., Wu, H., Dasigi, P., Dua, D., Gardner, M., Logan, R. L., Marasović, A., & Nie, Z. (2020) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2020.emnlp-demos.17 · Cited by 12
The ROOTS Search Tool: Data Transparency for LLMs
Piktus, A., Akiki, C., Villegas, P., Laurençon, H., Dupont, G., Luccioni, A. S., Jernite, Y., & Rogers, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.29 · Cited by 59
Artificial Intelligence Strategies for National Security and Safety StandardsPreprint
Blasch, E., Sung, J., Nguyen, T., Daniel, C. P., & Mason, A. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.05727 · Cited by 25
“These efforts include: Datasheets [42], data statements [43], and “nutrition la-bels” [44] to better understand the data that underlies a machine learning system; Cards to clarify the intended use cases of machine learning algorithms and minimize their usage in contexts for which they are not…”
BackgroundRissanen Data Analysis: Examining Dataset Characteristics via Description LengthPreprint
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03872 · Cited by 16
“Our work opens up ample opportunity for future work: automatically uncovering dataset biases when writing data statements (Gebru et al., 2018; Bender and Friedman, 2018), selecting the datasets to include in future benchmarks, discovering which capabilities are helpful for different tasks, and also…”
BackgroundResultMundt, M., Lang, S., Delfosse, Q., & Kersting, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03331 · Cited by 29
“…search (Lindauer & Hutter, 2020), human-centered machine learning model cards (Mitchell et al., 2019), or general dataset sheet spec-ifications (Bender & Friedman, 2018; Gebru et al., 2018), which all make valuable propositions to overcome existing gaps through the creation of standardized…”
Artificial Intelligence for Earth System Predictability (AI4ESP) (2021 Workshop Report)
Hickmon, N. L., Varadharajan, C., Hoffman, F. M., Wainwright, H. M., & Collis, S. (2022) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/1888810 · Cited by 19
Tan, M., Lee, H., Wang, D., & Subramonyam, H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637358 · Cited by 14
“Research in data-centric AI has primarily addressed the downstream harms of low-quality data through the creation of numerous frameworks for facilitating data accountability and transparency through clear documentation practices [3, 7, 15, 18, 25, 68, 71, 93].”
BackgroundEcosystem Graphs: The Social Footprint of Foundation Models
Bommasani, R., Soylu, D., Liao, T., Creel, K., & Liang, P. (2023) · doi:10.21203/rs.3.rs-2961271/v1 · Cited by 14
Knowledge Workers' Perspectives on AI Training for Responsible AI Use
Zhang, A., & Lee, M. K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714100 · Cited by 25
“Different tools have also been created to facilitate critical reflection about responsible AI amongst the practitioners developing or engineering these systems [8, 25, 45].”
Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline
Kapania, S., Ballard, S. A., Kessler, A., & Vaughan, J. W. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732005 · Cited by 29
“While there are plenty of resources for data and model documentation [9, 14, 25, 66], there is almost no existing guidance on what to document for synthetic data that is created and used in such myriad ways.”
Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution
Emelin, D., & Sennrich, R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.670 · Cited by 40
“Since our work introduces a novel resource, we include a Data Statement (Bender and Friedman, 2018) as a concise overview of its provenance and construction.”
What’s the Meaning of Superhuman Performance in Today’s NLU?
Tedeschi, S., Bos, J., Declerck, T., Hajič, J., Hershcovich, D., Hovy, E. H., Koller, A., Krek, S., Schockaert, S., Sennrich, R., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.697 · Cited by 57
Natural Language Processing and Text Mining (Turning Unstructured Data into Structured)
Bagheri, A., Giachanou, A., Mosteiro, P. J., & Verberne, S. (2023) · Springer eBooks · doi:10.1007/978-3-031-36678-9_5 · Cited by 21
Lu, X., Pan, L., Liu, Q., Nakov, P., & Kan, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.483 · Cited by 93
Kirk, H. R., Bean, A., Vidgen, B., Röttger, P., & Hale, S. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.148 · Cited by 90
An Ontology for Standardising Trustworthy AI
Lewis, D., Filip, D., & J. Pandit, H. (2021) · Factoring Ethics in Technology, Policy Making, Regulation and AI · doi:10.5772/intechopen.97478 · Cited by 13
“Such AI trustworthiness characteristics can be for example associated with: the datasets used to train data-driven machine learning AI [47][48][49]; trained ML models [50][51] or AI-based product or services [52].”
BackgroundMethodIt’s the Meaning That Counts: The State of the Art in NLP and Semantics
Hershcovich, D., & Donatelli, L. E. (2021) · Künstliche Intell. · doi:10.1007/s13218-021-00726-6 · Cited by 17
MS-Mentions: Consistently Annotating Entity Mentions in Materials Science Procedural Text
O’Gorman, T., Jensen, Z., Mysore, S., Huang, K., Mahbub, R., Olivetti, E., & McCallum, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.101 · Cited by 14
“Speaker demographic and Language Variety following (Bender and Friedman, 2018): Demographics of paper authors were not collected.”
BackgroundMethodGender Bias in Machine Translation Systems
Ullmann, S. (2022) · Social and cultural studies of robots and AI · doi:10.1007/978-3-030-88615-8_7 · Cited by 17
Tackling Documentation Debt: A Survey on Algorithmic Fairness Datasets
Fabris, A., Messina, S., Silvello, G., & Susto, G. A. (2022) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3551624.3555286 · Cited by 24
“Another qualitative framework is represented by Data statements [8], which is tailored for NLP, requiring domain-specific information on language variety and speaker demographics.”
BackgroundCOBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statements
Zhou, X., Zhu, H., Yerukola, A., Davidson, T., Hwang, J. D., Swayamdipta, S., & Sap, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.392 · Cited by 60
Non-portability of Algorithmic Fairness in IndiaPreprint
Sambasivan, N., Arnesen, E., Hutchinson, B., & Prabhakaran, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.03659 · Cited by 32
“Transparency on datasets, processes, and models ( e.g., [36, 22, 14]), openly discussing limitations, failure cases, and intended use-cases can help bolster the practical limits to applying computing to human problems.”
BackgroundHi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models
Smith, E. M., & Williams, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03300 · Cited by 13
“Collecting annotators’ perspectives (c.f. Bender and Friedman 2018) can provide additional context for our results.”
Burr, C., & Leslie, D. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3937983 · Cited by 13
Li, Y., & Goel, S. (2025) · Advances in Accounting · doi:10.1016/j.adiac.2025.100842 · Cited by 27
Accountable Data: The Politics and Pragmatics of Disclosure Datasets
Poirier, L. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533201 · Cited by 21
“Recent scholarship in information studies and critical data studies has demonstrated the significance of strengthening existing data documentation standards [12, 33], and bringing such components of data provenance to the fore.”
BackgroundUsing model cards for ethical reflection: a qualitative exploration
Nunes, J. L., Barbosa, G. D. J., de Souza, C. S., Lopes, H., & Barbosa, S. D. J. (2022) · Brazilian Symposium on Human Factors in Computing Systems (IHC) · doi:10.1145/3554364.3559117 · Cited by 27
“They focus on different parts of the ML development process: on datasets [4, 16, 18]; on trained models [21]; or even on the development process itself [19].”
BackgroundImproving Gender-Related Fairness in Sentence Encoders: A Semantics-Based Approach
Dolci, T., Azzalini, F., & Tanelli, M. (2023) · Data Science and Engineering · doi:10.1007/s41019-023-00211-0 · Cited by 23
“The social bias in the geometry of the model is then of course reflected in downstream applications like web search, CV parsing or hate speech detection [3, 7, 43].”
Thorny Roses: Investigating the Dual Use Dilemma in Natural Language Processing
Kaffee, L.-A., Arora, A., Talat, Z., & Augenstein, I. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.932 · Cited by 31
Liu, Y., & Martens, J.-B. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103227 · Cited by 20
Meiser, M., & Zinnikus, I. (2024) · Energies · doi:10.3390/en17091992 · Cited by 21
Machine learning data practices through a data curation lens: An evaluation framework
Bhardwaj, E., Gujral, H., Wu, S., Zogheib, C., Maharaj, T., & Becker, C. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658955 · Cited by 44
“For example, data statements for natural language processing (NLP) datasets contain specifications on demographic information about the dataset annotator, quality of the dataset, provenance, etc. [12].”
BackgroundDesign, Development, and Deployment of Context-Adaptive AI Systems for Enhanced User Adoption
Lee, C. P. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3638195 · Cited by 18
“Efforts are directed towards enhancing transparency in technical models [28, 37], encompassing frameworks for documenting models and datasets [2, 4, 7], methods for elucidating model outputs [17, 31, 34], and strategies for effectively communicating uncertainty [5, 8].”
A comprehensive review of techniques for documenting artificial intelligence
Königstorfer, F. (2024) · Digital Policy Regulation and Governance · doi:10.1108/dprg-01-2024-0008 · Cited by 18
Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology
Hada, R., Husain, S., Gumma, V., Diddee, H., Yadavalli, A., Seth, A., Kulkarni, N., Gadiraju, U., Vashistha, A., Seshadri, V., et al. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659017 · Cited by 31
“We use the framework by Bender and Friedman [7] to discuss the ethical considerations for our work.”
MethodThe Venus score for the assessment of the quality and trustworthiness of biomedical datasets
Chicco, D., Fabris, A., & Jurman, G. (2025) · BioData Mining · doi:10.1186/s13040-024-00412-x · Cited by 18
Joint Rumour Stance and Veracity Prediction
Lillie, A. E., Middelboe, E. R., & Derczynski, L. (2019) · DSpace repository (University of Tartu) · Cited by 11
Analyzing Social Media Research: A Data Quality and Research Reproducibility Perspective
Srivastava, A. K., & Mishra, R. (2021) · IIM Kozhikode Society & Management Review · doi:10.1177/22779752211011810 · Cited by 15
“…social networks, which lead to leaking of personal information, online harassment such as stalking and sexual abuse, misrepresentation, public embarrassment, hate speech, racial bias, etc. (Al-Qurishi et al., 2018; Bender & Friedman, 2018; Choi et al., 2015; Jiang et al., 2013; Sap et al., 2019).”
BackgroundHopkins, A., & Booth, S. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 12
Could a Conversational AI Identify Offensive Language?
da Silva, D. A., Louro, H. D. B., Gonçalves, G. S., Marques, J., Dias, L. A. V., da Cunha, A. M., & Tasinaffo, P. M. (2021) · Information · doi:10.3390/info12100418 · Cited by 15
Jin, Z., Wang, X., Cheng, F., Sun, C., Liu, Q., & Qu, H. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3236380 · Cited by 14
“For example, clear and detailed documentation of benchmark datasets should be carefully considered [22], [23].”
BackgroundTrigger Warning Assignment as a Multi-Label Document Classification Problem
Wiegmann, M., Wolska, M. A., Schröder, C., Borchardt, O., Stein, B., & Potthast, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.676 · Cited by 22
Gender Bias in AI Recruitment Systems: A Sociological-and Data Science-based Case Study
Njoto, S., Cheong, M., Lederman, R., McLoughney, A. J., Ruppanner, L., & Wirth, A. I. (2022) · IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas55053.2022.10227106 · Cited by 30
“One area constantly being investigated in the area of fair machine learning, which has been alluded to in our literature review, is the inherent biases in language models [29], [40], [54], [55].”
BackgroundMisinformation on Twitter During the Danish National Election: A Case Study
Derczynski, L., Albert-Lindqvist, T. O., Bendsen, M. V., Inie, N., Pedersen, V. D., & Pedersen, J. E. (2019) · Conference for Truth and Trust Online (TTO) · doi:10.36370/tto.2019.16 · Cited by 12
“The statement (Bender and Friedman, 2018) for the Party-supporting and Candidate data is:”
BackgroundBreeding Gender-aware Direct Speech Translation Systems
Gaido, M., Savoldi, B., Bentivogli, L., Negri, M., & Turchi, M. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.350 · Cited by 36
Customizing Triggers with Concealed Data Poisoning.Preprint
Wallace, E., Zhao, T. Z., Feng, S., & Singh, S. (2020) · arXiv (Cornell University) · Cited by 12
Societal Biases in Language Generation: Progress and ChallengesPreprint
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04054 · Cited by 12
Casey, A., Davidson, E. M., Grover, C., Tobin, R., Grivas, A., Zhang, H., Schrempf, P., O’Neil, A. Q., Lee, L., Walsh, M., et al. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1184919 · Cited by 12
“Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated English Text
Hada, R., Seth, A., Diddee, H., & Bali, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.115 · Cited by 39
Gómez-Adorno, H., Bel-Enguix, G., Sierra, G., Barajas, J., & Alvarez-Giron, W. (2024) · Informatics · doi:10.3390/informatics11020024 · Cited by 17
“We followed the guidelines specified by [57] to create this data statement.”
MethodAI-Powered Smart Digital Libraries
Meesad, P., & Mingkhwan, A. (2024) · Studies in big data · doi:10.1007/978-3-031-69216-1_10 · Cited by 20
Fine-Grained Fairness Analysis of Abusive Language Detection Systems with CheckList
Manerba, M. M., & Tonelli, S. (2021) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2021.woah-1.9 · Cited by 12
“Data Statement (Bender and Friedman, 2018): templates and related labels were manually defined by the first author, a non-native English speaker.”
BackgroundDataset Geography: Mapping Language Data to Language Users
Faisal, F., Wang, Y., & Anastasopoulos, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.239 · Cited by 23
Trust and Transparency in Contact Tracing ApplicationsPreprint
Hobson, S. F., Hind, M., Mojsilović, A., & Varshney, K. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.11356 · Cited by 15
“Thus, there has been a recent focus on transparent reporting mechanisms for AI systems, focusing on datasets [5], [6], [7], models [8], [9] and services [8].”
BackgroundHavens, L., Terras, M., Bach, B., & Alex, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.05911 · Cited by 28
“Within the NLP research community, Bender and Friedman (2018) recommend improved documentation practices to mitigate emergent, technical, and preexisting biases.”
BackgroundMethodDeduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2021) · arXiv · doi:10.48550/arxiv.2107.06499 · Cited by 11
Najjar, A. A., Ashqar, H. I., Darwish, O., & Hammad, E. M. (2025) · Information · doi:10.3390/info16090767 · Cited by 33
Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond
Dycke, N., Kuznetsov, I., & Gurevych, I. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.23 · Cited by 10
“In line with the recent work in responsible data handling for NLP (Rogers et al., 2021; Bender and Friedman, 2018) and previous discussions on ar X iv :2 20 1.”
Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection
Inel, O., Draws, T., & Aroyo, L. M. (2023) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v11i1.27547 · Cited by 25
Radha, S. K., Kuehlkamp, A., & Nabrzyski, J. (2025) · ACM Journal on Responsible Computing · doi:10.1145/3768151 · Cited by 16
“Existing templates, such as Datasheets for Datasets [38], Dataset Nutrition Labels [80], and Data Statements [17], have been recommended by academic and industry stakeholders to improve documentation standards.”
Hirmer, S. A., Leonard, A., Tumwesige, J., & Conforti, C. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.186 · Cited by 22
Listening to Affected Communities to Define Extreme Speech: Dataset and Experiments
Maronikolakis, A., Wisiorek, A., Nann, L., Jabbar, H., Udupa, S., & Schuetze, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.87 · Cited by 33
Interrater disagreement resolution: A systematic procedure to reach consensus in annotation tasks
Oortwijn, Y., Ossenkoppele, T., & Betti, A. (2021) · UvA-DARE (University of Amsterdam) · Cited by 10
A Matrix for Selecting Responsible AI Frameworks
Narayanan, M., & Schoeberl, C. (2023) · doi:10.51593/20220029 · Cited by 18
Multi-Agent Risks from Advanced AI
Hammond, L., Chan, A., Clifton, J., Hoelscher-Obermaier, J., Khan, A., McLean, E., Smith, C., Barfuss, W., Foerster, J., Gavenčiak, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.14143 · Cited by 305
Manu, M., Makhija, F., Bist, A. S., Kharayat, P. S., Singh, A. K., & Goyal, S. (2024) · International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE) · doi:10.1109/icacite60783.2024.10617437 · Cited by 16
“Also, AI powered the natural language processing which enables machine to understand and interact with human language [32] and due to this specialized coders are not required any more, as we have seen various examples above that how tech companies are laying off the employees every day.”
BackgroundLimitations in speech recognition for young adults with down syndrome
Cibrian, F. L., Chen, Y. ‘., Anderson, K., Abrahamsson, C. M., & Motti, V. G. (2025) · Universal Access in the Information Society · doi:10.1007/s10209-025-01197-4 · Cited by 13
“All these measures have been widely used in studies involving speech recognition [20, 85–89].”
Metadata
- Authors
- Bender, E. M., & Friedman, B.
- Affiliations
- University of Washington (US)
- Corresponding author
- Bender, Emily M.
- Friedman, Batya
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2018-12
- Volume
- 6
- Pages
- 587-604
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2911227954
- Semantic Scholar
- 97bfa89addc6e5d76361e4c1e296949cad887b86
- Cited by
- 854
- References
- 46
- Citations by year
- 2026: 108 · 2025: 96 · 2024: 102 · 2023: 154 · 2022: 127 · 2021: 177 · 2020: 68 · 2019: 20 · 2018: 1
- FWCI
- 132.62
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Science Foundation (1302709)
- University of Washington
- York University
- Macquarie University
- University of California, San Diego
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- natural language processing, algorithmic bias, underrepresented groups, ethical practice, linguistic diversity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.