Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107

Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages

McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754

A large annotated corpus for learning natural language inference

Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342

Know What You Don’t Know: Unanswerable Questions for SQuAD

Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

Mihaylov, T., Clark, P. E., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1260 · Cited by 1,602

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1421 · Cited by 919

On the Opportunities and Risks of Foundation Models

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676

Shortcut learning in deep neural networks

Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224

Artificial Intelligence and Business Value: a Literature Review

Enholm, I. M., Papagiannidis, E., Mikalef, P., & Krogstie, J. (2022) · Information Systems Frontiers · doi:10.1007/s10796-021-10186-w · Cited by 1,046

Beyond accuracy: behavioral testing of NLP models with CheckList

Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890

Adversarial NLI: A New Benchmark for Natural Language Understanding

Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.441 · Cited by 656

An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction

Larson, S., Mahendran, A., Peper, J. J., Clarke, C., Lee, A., Hill, P., Kummerfeld, J. K., Leach, K., Laurenzano, M. A., Tang, L., et al. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1131 · Cited by 535

WinoGrande: an adversarial winograd schema challenge at scale

Sakaguchi, K., Bras, R. L., Bhagavatula, C., & Choi, Y. (2021) · Communications of the ACM · doi:10.1145/3474381 · Cited by 356

WILDS: A Benchmark of in-the-Wild Distribution Shifts

Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518

Evaluating Models’ Local Decision Boundaries via Contrast Sets

Gardner, M., Artzi, Y., Basmov, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.117 · Cited by 323

Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00370 · Cited by 242

Neural Unsupervised Domain Adaptation in NLP—A Survey

Ramponi, A., & Plank, B. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.603 · Cited by 311

CommonGen: A constrained text generation challenge for generative commonsense reasoning

Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., & Ren, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.165 · Cited by 239

Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

Davani, A. M., Díaz, M., & Prabhakaran, V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00449 · Cited by 294

Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Liang, P. P., Zadeh, A., & Morency, L. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 438

Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills

Smith, E. M., Williamson, M., Shuster, K., Weston, J., & Boureau, Y.-L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.183 · Cited by 207

Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection

Vidgen, B., Thrush, T., Waseem, Z., & Kiela, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.132 · Cited by 174

Gender Bias in Machine Translation

Savoldi, B., Gaido, M., Bentivogli, L., Negri, M., & Turchi, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00401 · Cited by 229

Symbolic Knowledge Distillation: from General Language Models to Commonsense Models

West, P., Bhagavatula, C., Hessel, J., Hwang, J. D., Jiang, L., Bras, R. J. L., Lu, X., Welleck, S., & Choi, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.341 · Cited by 209

Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

Sap, M., Swayamdipta, S., Vianna, L., Zhou, X., Choi, Y., & Smith, N. A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.431 · Cited by 224

QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension

Rogers, A., Gardner, M., & Augenstein, I. (2022) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 213

From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models

Feng, S., Park, C. Y., Liu, Y., & Tsvetkov, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.656 · Cited by 149

Rise of Artificial Intelligence in Business and Industry

Bharadiya, J. P., Thomas, R. K., & Ahmed, F. (2023) · Journal of Engineering Research and Reports · doi:10.9734/jerr/2023/v25i3893 · Cited by 152

Towards Faithful Model Explanation in NLP: A Survey

Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 196

Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power?

Miceli, M., Posada, J., & Yang, T. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492853 · Cited by 125

WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation

Liu, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.508 · Cited by 178

Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs

Bugliarello, E., Cotterell, R., Okazaki, N., & Elliott, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00408 · Cited by 133

Offensive, aggressive, and hate speech analysis: From data-centric to human-centered approach

Kocoń, J., Figas, A., Gruza, M., Puchalska, D., Kajdanowicz, T., & Kazienko, P. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102643 · Cited by 103

Reasoning Over Paragraph Effects in Situations

Lin, K., Tafjord, O., Clark, P. E., & Gardner, M. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5808 · Cited by 110

Measure and Improve Robustness in NLP Models: A Survey

Wang, X., Wang, H., & Yang, D. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.339 · Cited by 90

Reframing Human-AI Collaboration for Generating Free-Text Explanations

Wiegreffe, S., Hessel, J., Swayamdipta, S., Riedl, M. O., & Choi, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.47 · Cited by 121

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

Bhargava, P., Валентинович, Д. А., & Rogers, A. (2021) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2021.insights-1.18 · Cited by 96

A Checklist to Combat Cognitive Biases in Crowdsourcing

Draws, T., Rieger, A., Inel, O., Gadiraju, U., & Tintarev, N. (2021) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v9i1.18939 · Cited by 86

TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions

Qiang, N., Wu, H., Han, R., Peng, N., Gardner, M., & Roth, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.88 · Cited by 92

OCNLI: Original Chinese Natural Language Inference

Hu, H., Richardson, K., Xu, L. J., Li, L., Kübler, S., & Moss, L. S. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.314 · Cited by 112

Identifying and Measuring Annotator Bias Based on Annotators’ Demographic Characteristics

Kuwatly, H. A., Wich, M., & Groh, G. (2020) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2020.alw-1.21 · Cited by 102

Stance Detection Benchmark: How Robust is Your Stance Detection?

Schiller, B., Daxenberger, J., & Gurevych, I. (2021) · Künstliche Intell. · doi:10.1007/s13218-021-00714-w · Cited by 92

Competency Problems: On Finding and Removing Artifacts in Language Data

Gardner, M., Merrill, W., Dodge, J., Peters, M., Ross, A., Singh, S., & Smith, N. A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.135 · Cited by 81

MisRoBÆRTa: Transformers versus Misinformation

Truică, C., & Apostol, E. (2022) · Mathematics · doi:10.3390/math10040569 · Cited by 78

Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning StrategiesPreprint

Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.02235 · Cited by 70

Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation

Bartolo, M., Thrush, T., Jia, R., Riedel, S., Stenetorp, P., & Kiela, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.696 · Cited by 94

Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks

Rogers, A., Kovaleva, O., Downey, M., & Rumshisky, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6398 · Cited by 73

Data and its (dis)contents: A survey of dataset development and use in machine learning research

Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548

Semantic Answer Similarity for Evaluating Question Answering Models

Risch, J., Möller, T., Gutsch, J., & Pietsch, M. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.15 · Cited by 66

Gender biases within Artificial Intelligence and ChatGPT: Evidence, Sources of Biases and Solutions

Ho, J. Q. H., Hartanto, A., Koh, A., & Majeed, N. M. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100145 · Cited by 83

More Bang for Your Buck: Natural Perturbation for Robust Question Answering

Khashabi, D., Khot, T., & Sabharwal, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.12 · Cited by 62

Learning from Task Descriptions

Weller, O., Lourie, N., Gardner, M., & Peters, M. E. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.105 · Cited by 67

MultiModalQA: Complex Question Answering over Text, Tables and ImagesPreprint

Talmor, A., Yoran, O., Catav, A., Lahav, D., Wang, Y., Asai, A., Ilharco, G., Hajishirzi, H., & Berant, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06039 · Cited by 149

Teach Me to Explain: A Review of Datasets for Explainable NLP.Preprint

Wiegreffe, S., & Marasović, A. (2021) · arXiv (Cornell University) · Cited by 57

Impact of politically biased data on hate speech classification

Wich, M., Bauer, J., & Groh, G. (2020) · Workshop on Abusive Language Online · doi:10.18653/v1/2020.alw-1.7 · Cited by 54

Prompting Contrastive Explanations for Commonsense Reasoning Tasks

Paranjape, B., Michael, J., Ghazvininejad, M., Hajishirzi, H., & Zettlemoyer, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.366 · Cited by 64

Evaluating NLP Models via Contrast SetsPreprint

Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · Cited by 65

Teach Me to Explain: A Review of Datasets for Explainable Natural Language ProcessingPreprint

Wiegreffe, S., & Marasović, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12060 · Cited by 128

Changing the World by Changing the Data

Rogers, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.170 · Cited by 51

Intrinsic Evaluation of Summarization Datasets

Bommasani, R., & Cardie, C. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.649 · Cited by 60

I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling

Nie, Y., Williamson, M., Bansal, M., Kiela, D., & Weston, J. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.134 · Cited by 66

Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets

Wu, Y., Gardner, M., Stenetorp, P., & Dasigi, P. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.190 · Cited by 69

Analyzing Dataset Annotation Quality Management in the Wild

Klie, J.-C., de Castilho, R. E., & Gurevych, I. (2024) · Computational Linguistics · doi:10.1162/coli_a_00516 · Cited by 66

Hate Speech Classifiers Learn Normative Social Stereotypes

Davani, A. M., Atari, M., Kennedy, B., & Dehghani, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00550 · Cited by 73

Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models

Röttger, P., Seelawi, H., Nozza, D., Talat, Z., & Vidgen, B. (2022) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2022.woah-1.15 · Cited by 61

Explaining Question Answering Models through Text GenerationPreprint

Latcinnik, V., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05569 · Cited by 60

Annotating Online Misogyny

Zeinert, P., Inie, N., & Derczynski, L. (2021) · doi:10.18653/v1/2021.acl-long.247 · Cited by 49

Don’t Blame the Annotator: Bias Already Starts in the Annotation Instructions

Parmar, M., Mishra, S. R., Geva, M., & Baral, C. R. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.130 · Cited by 59

Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition

Ghaddar, A., Langlais, P., Rashid, A., & Rezagholizadeh, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00386 · Cited by 40

Evaluating Models' Local Decision Boundaries via Contrast SetsPreprint

Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02709 · Cited by 132

A taxonomy and review of generalization research in NLPPreprint

Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2022) · arXiv (Cornell University) · doi:10.3929/ethz-b-000638992 · Cited by 41

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey

Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38

Automated quality assessment of cognitive behavioral therapy sessions through highly contextualized language representations

Flemotomos, N., Martínez, V. R., Chen, Z., Creed, T. A., Atkins, D. C., & Narayanan, S. S. (2021) · PLoS ONE · doi:10.1371/journal.pone.0258639 · Cited by 48

Ethical Issues Arising Due to Bias in Training A.I. Algorithms in Healthcare and Data Sharing as a Potential Solution

Goankar, B., Cook, K., & Macyszyn, L. J. (2020) · AI Ethics Journal · doi:10.47289/aiej20200916 · Cited by 43

Measuring a hate speech spectrum with faceted Rasch item response theory and perspective-aware, explainable-by-design deep learningPreprint

Kennedy, C. J., Bacon, G., Sahn, A., & von Vacano, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.10277 · Cited by 140

Personal Bias in Prediction of Emotions Elicited by Textual Opinions

Miłkowski, P., Gruza, M., Kanclerz, K., Kazienko, P., Grimling, D., & Kocoń, J. (2021) · doi:10.18653/v1/2021.acl-srw.26 · Cited by 34

HateCheck: Functional Tests for Hate Speech Detection Models

Röttger, P., Vidgen, B., Nguyen, D., Waseem, Z., Margetts, H. Z., & Pierrehumbert, J. B. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.4 · Cited by 287

“Will You Find These Shortcuts?” A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification

Bastings, J., Ebert, S., Zablotskaia, P., Sandholm, A., & Filippova, K. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.64 · Cited by 65

Multi-hop Question Answering

Mavi, V., Jangra, A., & Jatowt, A. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000102 · Cited by 119

The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions

Zhou, X., Nie, Y., Tan, H., & Bansal, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.659 · Cited by 32

What Question Answering can Learn from Trivia Nerds

Boyd-Graber, J. L., & Börschinger, B. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.662 · Cited by 31

Investigating Annotator Bias with a Graph-Based Approach

Wich, M., Kuwatly, H. A., & Groh, G. (2020) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2020.alw-1.22 · Cited by 32

Case Study: Deontological Ethics in NLP

Prabhumoye, S., Boldt, B., Salakhutdinov, R., & Black, A. W. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.297 · Cited by 35

COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences

Singh, S., Wen, N., Hou, Y., Alipoormolabashi, P., Wu, T.-L., Ma, X., & Peng, N. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.78 · Cited by 48

ANLIzing the Adversarial Natural Language Inference DatasetPreprint

Williams, A., Thrush, T., & Kiela, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12729 · Cited by 40

Shortcutted Commonsense: Data Spuriousness in Deep Learning of Commonsense Reasoning

Branco, R., Branco, A., Rodrigues, J. A., & Silva, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.113 · Cited by 36

Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning

Sainz, O., González-Dios, I., de Lacalle, O. L., Min, B., & Agirre, E. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.187 · Cited by 31

Minimizing bias when using artificial intelligence in critical care medicine

Ranard, B. L., Park, S., Jia, Y., Zhang, Y., Alwan, F. S., Celi, L. A., & Lusczek, E. R. (2024) · Journal of Critical Care · doi:10.1016/j.jcrc.2024.154796 · Cited by 27

CrossAug

Lee, M.-W., Won, S., Kim, J., Lee, H., Park, C., & Jung, K. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482078 · Cited by 27

CREAK: A Dataset for Commonsense Reasoning over Entity KnowledgePreprint

Onoe, Y., Zhang, M. J. Q., Choi, E., & Durrett, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.01653 · Cited by 61

Logiqa 2.0—An improved dataset for logical reasoning in natural language understanding

Liu, H., Liu, J., Cui, L., Teng, Z., Duan, N., Zhou, M., & Zhang, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3293046 · Cited by 26

Benefits of Intermediate Annotations in Reading Comprehension

Dua, D., Singh, S., & Gardner, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.497 · Cited by 26

Linguistically-Informed Transformations (LIT): A Method for Automatically Generating Contrast Sets

Li, C., Shengshuo, L., Liu, Z., Wu, X., Zhou, X., & Steinert‐Threlkeld, S. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.12 · Cited by 30

Assessing Annotator Identity Sensitivity via Item Response Theory: A Case Study in a Hate Speech Corpus

Sachdeva, P., Barreto, R., von Vacano, C., & Kennedy, C. J. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533216 · Cited by 25

Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power?Preprint

Miceli, M., Posada, J., & Yang, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08131 · Cited by 25

Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation

Rebuffel, C., Scialom, T., Soulier, L., Piwowarski, B., Lamprier, S., Staiano, J., Scoutheeten, G., & Gallinari, P. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.633 · Cited by 24

Reliability Testing for Natural Language Processing Systems

Tan, S., Joty, S., Baxter, K., Taeihagh, A., Bennett, G. A., & Kan, M. (2021) · doi:10.18653/v1/2021.acl-long.321 · Cited by 31

What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?

Nangia, N., Sugawara, S., Trivedi, H. K., Warstadt, A., Vania, C., & Bowman, S. R. (2021) · doi:10.18653/v1/2021.acl-long.98 · Cited by 29

Proceedings of the 2nd Workshop on Machine Reading for Question Answering

(2019) · doi:10.18653/v1/d19-58 · Cited by 26

Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA

Bitton, Y., Stanovsky, G., Schwartz, R., & Elhadad, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.9 · Cited by 31

Listening to Mental Health Crisis Needs at Scale: Using Natural Language Processing to Understand and Evaluate a Mental Health Crisis Text Messaging Service

Liu, Z., Peach, R. L., Lawrance, E., Noble, A., Ungless, M. A., & Barahona, M. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.779091 · Cited by 24

Quizbowl: The Case for Incremental Question AnsweringPreprint

Rodríguez, P., Feng, S., Iyyer, M., He, H., & Boyd-Graber, J. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04792 · Cited by 47

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop

(2021) · doi:10.18653/v1/2021.acl-srw · Cited by 22

Promoting Graph Awareness in Linearized Graph-to-Text Generation

Hoyle, A., Marasović, A., & Smith, N. A. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.82 · Cited by 20

Does Putting a Linguist in the Loop Improve NLU Data Collection?

Parrish, A., Huang, W., Agha, O., Lee, S.-H., Nangia, N., Warstadt, A., Aggarwal, K., Allaway, E., Linzen, T., & Bowman, S. R. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.421 · Cited by 23

Digital History and Hermeneutics

University of Luxembourg, de la Recherche, F. N., Fickers, A., & Tatarinov, J. (2022) · De Gruyter eBooks · doi:10.1515/9783110723991 · Cited by 27

Toward Annotation Efficiency in Biased Learning Settings for Natural Language Processing

Effland, T. (2022) · Columbia Academic Commons (Columbia University) · doi:10.7916/znbv-rz34

Large Language Models for Data Annotation and Synthesis: A Survey

Tan, Z., Li, D., Wang, S., Beigi, A., Jiang, B., Bhattacharjee, A., Karami, M., Li, J., Cheng, L., & Liu, H. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.54 · Cited by 178

Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks

Röttger, P., Vidgen, B., Hovy, D., & Pierrehumbert, J. B. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.13 · Cited by 140

AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators

He, X., Lin, Z., Gong, Y., Jin, A.-L., Zhang, H., Chen, L., Jiao, J., Yiu, S., Duan, N., & Chen, W. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-industry.15 · Cited by 272

A Systematic Comparison of Large Language Models for Data Annotation in NER Tasks

Haq, M. U. U., Rigoni, D., & Sperduti, A. (2026) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4qnuuw7rjs24