Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
In this paper, we propose data statements as a design solution and professional practice for natural language processing technologists, in both research and development. Through the adoption and widespread use of data statements, the field can begin to address critical scientific and ethical issues that result from the use of data from certain populations in the development of technology for other populations. We present a form that data statements can take and explore the implications of adopting them as part of regular practice. We argue that data statements will help alleviate issues related to exclusion and bias in language technology, lead to better precision in claims about how natural language processing research can generalize and thus better engineering results, protect companies from public embarrassment, and ultimately lead to language technology that meets its users in their own preferred linguistic style and furthermore does not misrepresent them to others.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“or lead to adverse effects from deploying suboptimal or harmful models (Bender and Friedman, 2018) on tasks for which they are ill-suited (Raji et al., 2022).”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Bender and Friedman, 2018
The cited source says what the claim attributes to it.
In the citing paper
“Numerous authors have observed that ML systems perform better for users whose demographic characteristics match those represented in the training data [9,36].”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Thumbs up?: sentiment classification using machine learning techniques
Pang, B., Lee, L., & Vaithyanathan, S. (2002) · Proceedings of the ACL-02 conference on Empirical methods in natural language processing - EMNLP '02 · doi:10.3115/1118693.1118704 · Cited by 7,024 · cited 1×
“Here, our focus is on linguistic data. the review (e.g., Pang et al., 2002) or the hashtag #sarcasm used to identify sarcastic language (e.g., Kreuz and Caucci, 2007).”
BackgroundThe Study of Second Language Acquisition
Brown, C., & Ellis, R. (1996) · The Modern Language Journal · doi:10.2307/329065 · Cited by 4,199
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
The Social Stratification of English in New York City.
Crockett, H. J., & Labov, W. (1968) · American Sociological Review · doi:10.2307/2092899 · Cited by 3,084
Accurate unlexicalized parsing
Klein, D., & Manning, C. D. (2003) · Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - ACL '03 · doi:10.3115/1075096.1075150 · Cited by 3,008
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter
Waseem, Z., & Hovy, D. (2016) · Proceedings of the NAACL Student Research Workshop · doi:10.18653/v1/n16-2013 · Cited by 1,655 · cited 4×
“Waseem and Hovy (2016) provide the following complete list of terms used in their initial scrape: MKR, asian drive, feminazi, immigrant, nigger, sjw, WomenAgainstFeminism, blameonenotall, islam terrorism, notallmen, victimcard, victim card, arab terror, gamergate, jsil, racecard, race card. dataset…”
BackgroundMethodInfluential
Cited by
Items 801–865 of 865
Self-Supervised Modality Complementation (SSMC)
Lin, Z., & Li, Q. (2026) · Wireless networks · doi:10.1007/978-3-032-20416-5_4
Liar, Liar, LLM on Fire: Investigating Deception in Turkish Text Generation
Bölücü, N., Yücel, Z., & Çetintaş, D. (2026) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3811025
Silenced by design: Marginal voices and algorithmic bias in AI-generated climate discourse
Brancaccio, M. (2026) · Discourse & Society · doi:10.1177/09579265261449258
“This yielded a dataset of 90 responses in total (Table 2), documented in full in Supplemental Appendix 1, keeping with principles of transparent dataset description (Bender and Friedman, 2018).”
Bridging the silos in affective AI: a critical perspective from data to society
Inoshita, K. (2026) · AI & Society · doi:10.1007/s00146-026-03324-y · Cited by 3
Evaluating Data Availability Statements in Academic Literature: A Scalable AI Approach
Wong, J. Y. M., Cheung, A. C. Y., Feng, S., Man, F. L., & Chiu, D. K. W. (2026) · Serials Review · doi:10.1080/00987913.2026.2669760
A Novel Dual-Loop Causality-Traceable Retrieval Framework for Long-Horizon Conversational Agents
Chan, D. Y., Cheng, C.-Y., Wang, J.-F., & Tseng, S.-P. (2026) · Electronics · doi:10.3390/electronics15112373 · Cited by 1
Rodrigo-Ginés, F.-J., Chamorro-Padial, J., & Rodríguez-Díaz, P. (2026) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai68641.2026.11536395 · Cited by 1
“Despite their technical sophistication, these systems encounter profound epistemological limitations when attempting to model a phenomenon that is not purely linguistic but socio-cognitive and interpretive in nature [3], [4].”
Feltrero, R., & López-López, L.-F. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0042-9_5
Safety Assurance Through Red-Teaming, MLOps, and Continuous Monitoring for Medical LLMs
Pandey, A. K., Sharma, S., Kumar, R., & Kumar, G. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7862-6.ch014
Sibaee, S., Ahmed, S., Alharbi, A. I., Nacar, O., Ammar, A., Alhabashi, Y., & Boulila, W. (2026) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3820657
Gök, M., & YILMAZ, T. Y. (2026) · European Journal of Education · doi:10.1111/ejed.70737
Ghost in the cache: How data decay shapes the unseen landscape of AI memory
Gaweł, H. (2026) · Memory Mind & Media · doi:10.1017/mem.2026.10039
“Complementarily, media and communication scholarship draws our attention to the discursive ramifications of such leakage (Bender and Friedman 2018).”
AlDahoul, N., Karim, H. B. A., & Tan, M. J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6900538
Voices in the Loop: Mapping Participatory AI
Mushkani, R. A. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812241
“Where available, project records should link to datasheets, model cards, and data statements [2, 19, 30], while preserving participation metadata that those artifacts usually omit.”
A Human-Centric Framework for Data Attribution in Large Language Models
Wuehrl, A., Ruckdeschel, M., Lo, K., & Rogers, A. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812236 · Cited by 1
“The NLP community has also contributed to relevant discussions of LLM data governance [118], data documentation [45, 94, 113, 156], and data markets [35, 83].”
Havens, L., Terras, M., Bach, B., & Alex, B. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812283
“Still other work proposes new practices to encourage greater reflection during AI system creation, including dataset collection [14, 54], model design [80, 103], and evaluation [87, 114, 155].”
Responsible NLP: Why performance is no longer enough
Lamsiyah, S. (2026) · Natural language processing. · doi:10.1017/nlp.2026.10030 · Cited by 1
“Data statements (Bender and Friedman 2018), datasheets for datasets (Gebru et al. 2021), and Downloaded from https://www.cambridge.org/core.”
Lessons from archival provenance: Reframing data creator relationships in AI
Maemura, E. (2026) · Cambridge Forum on AI Culture and Society · doi:10.1017/cfc.2026.10021
“…approach of the DPI proposal as well as a broader trend toward documentation standards for transparent, responsible and explainable AI (e.g., Bender and Friedman 2018; Gebru et al. 2021; Holland et al. 2018; Krause 2019; McMillan-Major et al. 2023; Pushkarna and Zaldivar 2022; Zelenka and…”
Nagaraj, P., Muneeswaran, V., Amer, A., Hafez, M., Muthiah, R., Islam, M. T., & Ijaz, M. F. (2025) · Asian Conference on Communication and Networks (ASIANComNet) · doi:10.1109/asiancomnet68615.2025.11579569
ZHOU, H., & Chan, H. Y. (2026) · Data in Brief · doi:10.1016/j.dib.2026.113321
Barut, E. (2026) · Uluslararası Dil Edebiyat ve Kültür Araştırmaları Dergisi · doi:10.37999/udekad.1934778
Zheng, H., & Burhanudeen, H. (2026) · Journal of Visualized Experiments · doi:10.3791/71320
WhiteTesseract: Reframing the Interpretation of Cultural Heritage through XR and Conversational AI
Li, J., Liu, Z., Jin, X., Fushimi, T., & Ochiai, Y. (2026) · Journal on Computing and Cultural Heritage · doi:10.1145/3821568 · Cited by 1
“However, concerns persist about factual accuracy, hallucination risks in historical contexts, and ethical alignment—especially when LLMs are used in educational or heritage settings [6, 37].”
Reproducibility in Information Retrieval
Maistro, M., Breuer, T., & Ferro, N. (2026) · Foundations and Trends® in Information Retrieval · doi:10.1108/ftinr-05-2025-0087 · Cited by 1
The Third Option: integrating AI training‑data licensing into scholarly publishing workflows
Ochiai, M. (2026) · Insights the UKSG journal · doi:10.1629/uksg.776
Measuring complex constructs in large-scale text with computational social mixed methods
Herderich, A., Lasser, J., Galešić, M., Aroyehun, S., García, D., & Garland, J. (2026) · Behavior Research Methods · doi:10.3758/s13428-026-03046-5
“There are alternative or complementary approaches like data statements for natural language processing (Bender & Friedman, 2018) or data nutrition labels for automated decision-making systems (Chmielinski et al., 2022).”
Dawodi, M., Pfeffer, J., Sun, Y., & Baktash, J. A. (2026) · AI & Society · doi:10.1007/s00146-026-03201-8 · Cited by 1
“Annotation frameworks should therefore prioritize fine-grained or multi-label taxonomies and transparent documentation of guidelines, annotator training, and disa-greement handling (Bender and Friedman 2018; Blodg-ett et al. 2020).”
InfluentialRemache, A., Refae, G. A. E., Remache, R., Belarbi, A. K., Remache, S. A. Q., & Almheiri, S. (2026) · International Conference on Artificial Intelligence Systems (AIS) · doi:10.1109/ais69919.2026.11621192
“These recommendations are consistent with data statements for NLP [14], datasheets for datasets [15], and model cards for model reporting [16].”
Developing artificial intelligence benchmarks for entrepreneurial tasks
Fox, J. D. (2026) · Small Enterprise Research · doi:10.1080/13215906.2026.2705499
iBRAHIMOĞLU, N., Demirezen, M. U., & Yıldız, F. (2026) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-026-01490-x · Cited by 2
BRIDGE-E: A Complexity-Aware Framework for Policy Coherence
Bouderhem, R. (2026) · SpringerBriefs in energy · doi:10.1007/978-3-032-22978-6_7
MiLTL: A Cross-Modal Contradiction Cascade for On-Device Voice Phishing Detection
Jung, B., & Chang, J. (2026) · Algorithms · doi:10.3390/a19090723
Béchara, H., Manohara, K., Dasandi, N., & Mikhaylov, S. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817597
“We follow emerging best practices in dataset documentation, including data statements [3], datasheets for datasets [7], and model cards [12].”
Ankam, J. (2024) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.5469 · Cited by 14
“Model cards and dataset documentation improve transparency about intended use, evaluation, and provenance [21]–[23].”
HEART-Synth: A Synthetic Resident-Patient Dialogue Dataset for Medical Error Disclosure Assessment
Ghimire, A., Amsaad, F., & Adkins, S. M. (2026) · IEEE National Aerospace and Electronics Conference (NAECON) · doi:10.1109/naecon70028.2026.11674942
“Data statements and datasheets argue that dataset creators should report composition, provenance, intended use, and limitations rather than presenting a dataset as a neutral object [14], [15].”
Editorial: Benchmarking and the Construction of Entities in the Humanities
Kwok, J. C. Y. (2026) · Journal of Open Humanities Data · doi:10.5334/johd.588
Measuring and Mitigating Bias and Discrimination in AI
Ghalme, G. S. (2026) · Springer eBooks · doi:10.1007/978-3-032-22368-5_3
Bharadwaj, S., Gangurde, R. D., & Mishra, A. M. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-9028-4.ch001
Walkiewicz, J. (2026) · Wroclaw University of Economics and Business eBooks · doi:10.15611/2026.53.1.04
Research, G. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7301221
AI-driven Acceleration and the Evaluation Bottleneck in Science: A Conceptual FrameworkPreprint
Rana, P., Varshney, L. R., & Varshney, K. R. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7318643
Is This Funny to You?: A Reader-Aware Humour Classification Dataset
Roland, K., & King, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37249-9_3
Komendantova, N., & Pesov, P. (2026) · Statistical Journal of the IAOS · doi:10.1177/18747655261480969
Close to Some, Distant to OthersEvaluator-Product Distance Dispersion and Polarized ReceptionPreprint
Du, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7389430
Close to Some, Distant to OthersEvaluator-Product Distance Dispersion and Polarized ReceptionPreprint
Du, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7387975
Uluk, M. (2026) · Özgür Yayınları eBooks · doi:10.58830/ozgur.pub1391
Eraydın, Y. Y., & Uzun, C. (2026) · Sustainability · doi:10.3390/su18179172
Creely, E., & Mehta, R. (2026) · Linguistics and Education · doi:10.1016/j.linged.2026.101585 · Cited by 1
Suladay, E. J. C., Ganesh, K., Lagman, M. P., Burgos, M. P. J., Torres, J. C., & Alvaro, M. M. F. (2026) · International Conference on Cloud Computing and Big Data Analytics (ICCCBDA) · doi:10.1109/icccbda69573.2026.11676426
“Calibration studies will refine operating thresholds (60), (61), and transparency will be strengthened through Model Cards and Data Statements (62), (63).”
Maemura, E., & Byrne, H. (2026) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70113
“For instance, Data Statements for NLP originally developed by Bender and Friedman (2018) were refined through a series of community workshops as reported by McMillan-Major et al. (2024).”
Wang, X., Yu, H., & Niu, F. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10090315
UzEnParC 1.0: A Cross-Script, Metadata-Rich Uzbek-English Parallel Corpus for Low-Resource NLP
Boltayevich, E. B., & Amirkulov, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37870-5_16
The same model, different responsibilities: Responsible NLP under the EU AI Act
Lamsiyah, S. (2026) · Natural language processing. · doi:10.1017/nlp.2026.10036
Evidence aware multimodal auditing of museum image metadata consistencyPreprint
yue, P., hou, J., & zhang, X. (2026) · Research Square · doi:10.21203/rs.3.rs-10971995/v1
A dataset of Chinese news from government websites in the Amdo Tibetan region (2013–2024)
Jia, R. E. N., Lei, G. U. O., Xuemei, C. A. O., Haoyu, D. O. N. G., & Yang, M. (2026) · China Scientific Data · doi:10.11922/11-6035.nbsdc.2025.0230.zh
Polleti, G., Gerardi, F. F., & Cozman, F. G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39904-5_23
Spivak, E. (2026) · Research Square · doi:10.21203/rs.3.rs-11311689/v1
FakenewsBR: A Unified Portuguese Misinformation Corpus for Reproducible Data Integration Research
Mariano, K. D. P., Almada, F. L. N., Dutra, M. A., Monteiro, V. E. D. S., Gomes, J. R. S., Filho, A. R. G., & Soares, A. D. S. (2026) · doi:10.5753/dsw.2026.249457
“Data redundancy and near-duplicate narratives often bias evaluations, yet systematic deduplication and standardized documentation [Bender and Friedman 2018] remain scarce in Portuguese resources.”
Teaching AI, teaching bias: addressing gender inequality in public health education
Dietrich, A., Czok, K., & Ilskens, K. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1952627
“Data Statements were developed specifically for language datasets; their purpose should be distinguished from the wider documentation needed for clinical datasets (39).”
Tsoni, R., Μιχιώτης, Α., Κουτούζης, Μ., & Verykios, V. S. (2026) · Pan-Hellenic Conference on Progress in Computing and Informatics (PCI) · doi:10.1145/3829216.3829329
“Complementary reporting practices such as data statements, datasheets, and model cards enhance transparency about intended use, data provenance, and group performance [1, 14, 28].”
Charity Without Competence: Domain Knowledge and the Limits of Interpretation in AI Ethics
Younas, A., & Zeng, Y. (2026) · Philosophies · doi:10.3390/philosophies11050196
GovSynth: A Synthetic Evidence Corpus for Cybersecurity Governance Audit Across Frameworks
Yang, S.-C., Cha, S.-C., Wu, T.-J., & Lin, C.-C. (2026) · International Conference on Computer and Communications Management (ICCCM) · doi:10.1145/3841323.3841356
“It releases under CC BY 4.0 [7] with a datasheet [8] and a data statement [4].”
InfluentialThe Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492
Disembodied Machine Learning: On the Illusion of Objectivity in NLP
Waseem, Z., Lulz, S., Bingel, J., & Augenstein, I. (2021) · arXiv · doi:10.48550/arxiv.2101.11974 · Cited by 34
“Bender and Friedman (2018) propose ‘data statements’, a tool to describe and expose representational biases in the processes of developing datasets from collection to annotation.”
Background
Metadata
- Authors
- Bender, E. M., & Friedman, B.
- Affiliations
- University of Washington (US)
- Corresponding author
- Bender, Emily M.
- Friedman, Batya
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2018-12
- Volume
- 6
- Pages
- 587-604
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2911227954
- Semantic Scholar
- 97bfa89addc6e5d76361e4c1e296949cad887b86
- Cited by
- 854
- References
- 46
- Citations by year
- 2026: 108 · 2025: 96 · 2024: 102 · 2023: 154 · 2022: 127 · 2021: 177 · 2020: 68 · 2019: 20 · 2018: 1
- FWCI
- 132.62
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Science Foundation (1302709)
- University of Washington
- York University
- Macquarie University
- University of California, San Diego
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- natural language processing, algorithmic bias, underrepresented groups, ethical practice, linguistic diversity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.