Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107

Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages

McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754

A large annotated corpus for learning natural language inference

Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342

Know What You Don’t Know: Unanswerable Questions for SQuAD

Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

Mihaylov, T., Clark, P. E., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1260 · Cited by 1,602

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1421 · Cited by 919

Natural Language Processing: Challenges and Future Directions

Kaddari, Z., Mellah, Y., Berrich, J., Belkasmi, M. G., & Bouchentouf, T. (2020) · Lecture notes in networks and systems · doi:10.1007/978-3-030-53970-2_22 · Cited by 22

Multitask Personalized Recognition of Emotions Evoked by Textual Content

Miłkowski, P., Saganowski, S., Gruza, M., Kazienko, P., Piasecki, M., & Kocoń, J. (2022) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops53856.2022.9767502 · Cited by 20

Image Retrieval from Contextual Descriptions

Krojer, B., Adlakha, V., Vineet, V., Goyal, Y., Ponti, E. M., & Reddy, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.241 · Cited by 38

Angler: Helping Machine Translation Practitioners Prioritize Model Improvements

Robertson, S., Wang, Z. J., Moritz, D., Kery, M. B., & Hohman, F. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580790 · Cited by 22

An Empirical Study of Invariant Risk MinimizationPreprint

Choe, Y. J., Ham, J., & Park, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05007 · Cited by 36

A Comprehensive Literature Review on Children’s Databases for Machine Learning Applications

Al-Azani, S., Sait, S. M., & Al-Utaibi, K. A. (2022) · IEEE Access · doi:10.1109/access.2022.3146008 · Cited by 20

Sources of Uncertainty in Supervised Machine Learning—a Statisticians’ View

Gruber, C., Schenk, P. O., Schierholz, M., Kreuter, F., & Kauermann, G. (2026) · Statistical Science · doi:10.1214/25-sts1003 · Cited by 85

HINT: Integration Testing for AI-based features with Humans in the Loop

Chen, Q. Z., Schnabel, T., Nushi, B., & Amershi, S. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511141 · Cited by 24

Hidden Biases in Unreliable News Detection Datasets

Zhou, X., Elfardy, H., Christodoulopoulos, C., Butler, T., & Bansal, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.211 · Cited by 20

Investigating and Mitigating Biases in Crowdsourced Data

Hettiachchi, D., Sanderson, M., Gonçalves, J., Hosio, S. J., Kazai, G., Lease, M., Schaekermann, M., & Yılmaz, E. (2021) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3462204.3481729 · Cited by 17

Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants

Bartolo, M., Thrush, T., Riedel, S., Stenetorp, P., Jia, R., & Kiela, D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.275 · Cited by 28

ASRS-CMFS vs. RoBERTa: Comparing Two Pre-Trained Language Models to Predict Anomalies in Aviation Occurrence Reports with a Low Volume of In-Domain Data Available

Kierszbaum, S., Klein, T., & Lapasset, L. (2022) · Aerospace · doi:10.3390/aerospace9100591 · Cited by 20

Do We Need to Create Big Datasets to Learn a Task?

Mishra, S. R., & Sachdeva, B. (2020) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2020.sustainlp-1.23 · Cited by 19

SD-QA: Spoken Dialectal Question Answering for the Real World

Faisal, F., Keshava, S., Alam, M. M. I., & Anastasopoulos, A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.281 · Cited by 26

Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation BenchmarkingPreprint

Ma, Z., Ethayarajh, K., Thrush, T., Jain, S., Wu, L., Jia, R., Potts, C., Williams, A., & Kiela, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06052 · Cited by 21

Perceptual Score: What Data Modalities Does Your Model Perceive?Preprint

Gat, I., Schwartz, I., & Schwing, A. G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.14375 · Cited by 20

DQI: Measuring Data Quality in NLPPreprint

Mishra, S. R., Arunkumar, A., Sachdeva, B., Bryan, C., & Baral, C. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00816 · Cited by 38

Low-cost tree crown dieback estimation using deep learning-based segmentation

Allen, M. J., Moreno‐Fernández, D., Ruiz‐Benito, P., Grieve, S., & Lines, E. R. (2024) · Environmental Data Science · doi:10.1017/eds.2024.16 · Cited by 21

Uncovering labeler bias in machine learning annotation tasks

Haliburton, L., Leusmann, J., Welsch, R., Ghebremedhin, S., Isaakidis, P., Schmidt, A., & Mayer, S. (2024) · AI and Ethics · doi:10.1007/s43681-024-00572-w · Cited by 15

RUN through the Streets: A New Dataset and Baseline Models for Realistic Urban Navigation

Paz-Argaman, T., & Tsarfaty, R. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1681 · Cited by 15

Neural Conversational QA: Learning to Reason vs Exploiting Patterns

Verma, N., Sharma, A., Madan, D., Contractor, D., Kumar, H., & Joshi, S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.589 · Cited by 14

Is My Model Using the Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoning

Gupta, V., Bhat, R. A., Ghosal, A., Shrivastava, M., Singh, M., & Srikumar, V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00482 · Cited by 16

Benchmarking Multimodal Regex Synthesis with Complex Structures

Ye, X., Chen, Q., Dillig, I., & Durrett, G. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.541 · Cited by 14

What’s the Meaning of Superhuman Performance in Today’s NLU?

Tedeschi, S., Bos, J., Declerck, T., Hajič, J., Hershcovich, D., Hovy, E. H., Koller, A., Krek, S., Schockaert, S., Sennrich, R., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.697 · Cited by 25

The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values

Kirk, H. R., Bean, A., Vidgen, B., Röttger, P., & Hale, S. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.148 · Cited by 48

HardEval: Focusing on Challenging Tokens to Assess Robustness of NER.

Bernier-Colborne, G., & Langlais, P. (2020) · NPARC · Cited by 12

Detecting Community Sensitive Norm Violations in Online Conversations

Park, C. Y., Mendelsohn, J., Radhakrishnan, K., Jain, K., Kanakagiri, T., Jurgens, D., & Tsvetkov, Y. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.288 · Cited by 14

Analyzing Dynamic Adversarial Training Data in the Limit

Wallace, E., Williams, A., Jia, R., & Kiela, D. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.18 · Cited by 25

Incorporating Stock Market Signals for Twitter Stance Detection

Conforti, C., Berndt, J., Pilehvar, M. T., Giannitsarou, C., Toxvaerd, F., & Collier, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.281 · Cited by 13

Evidence-based Verification for Real World Information Needs.Preprint

Thorne, J. P., Glockner, M., Vallejo, G., Vlachos, A., & Gurevych, I. (2021) · arXiv (Cornell University) · Cited by 13

Leveraging Digital Intelligence for Community Well-Being

Narayan, R. (2020) · International Journal of Community Well-Being · doi:10.1007/s42413-020-00085-4 · Cited by 14

ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding

Ghosh, S., & Srivastava, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.276 · Cited by 19

Multi-Step Inference for Reasoning Over Paragraphs

Liu, J., Gardner, M., Cohen, S. B., & Lapata, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.245 · Cited by 18

tWT–WT: A Dataset to Assert the Role of Target Entities for Detecting Stance of Tweets

Kaushal, A., Saha, A., & Ganguly, N. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.303 · Cited by 12

Latent Space Explanation by Intervention

Gat, I., Lorberbom, G., Schwartz, I., & Hazan, T. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19948 · Cited by 14

Project PIAF: Building a Native French Question-Answering DatasetPreprint

Keraron, R., Lancrenon, G., Bras, M., Allary, F., Moyse, G., Scialom, T., Soriano-Morales, E.-P., & Staiano, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00968 · Cited by 15

Mitigating Large Language Model Bias: Automated Dataset Augmentation and Prejudice Quantification

Mondal, D., & Lipizzi, C. (2024) · Computers · doi:10.3390/computers13060141 · Cited by 10

The Authenticity Gap in Human Evaluation

Ethayarajh, K., & Jurafsky, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.406 · Cited by 10

Closing the Loop: Testing ChatGPT to Generate Model Explanations to Improve Human Labelling of Sponsored Content on Social Media

Bertaglia, T., Huber, S., Goanță, C., Spanakis, G., & Iamnitchi, A. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44067-0_11 · Cited by 13

What Sounds “Right” to Me? Experiential Factors in the Perception of Political Ideology

Shen, Q., & Rosé, C. P. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.152 · Cited by 10

Annotation Curricula to Implicitly Train Non-Expert Annotators

Lee, J.-U., Klie, J.-C., & Gurevych, I. (2022) · Computational Linguistics · doi:10.1162/coli_a_00436 · Cited by 13

Biased by Design? Evaluating Bias and Behavioral Diversity in LLM Annotation of Real-World and Synthetic Hotel Reviews

Voutsa, M. C., Tsapatsoulis, N., & Djouvas, C. (2025) · AI · doi:10.3390/ai6080178 · Cited by 10

Overestimation of Syntactic Representation in Neural Language Models

Kodner, J., & Gupta, N. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.160 · Cited by 13

Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation

Shinoda, K., Sugawara, S., & Aizawa, A. (2021) · doi:10.18653/v1/2021.acl-srw.21 · Cited by 10

Toward Annotator Group Bias in Crowdsourcing

Liu, H., Thekinen, J., Mollaoglu, S., Tang, D. Q., Yang, J. S., Cheng, Y., Liu, H., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.126 · Cited by 16

Robust Domain Adaptation: Representations, Weights and Inductive Bias

Bouvier, V., Very, P., Chastagnol, C., Tami, M., & Hudelot, C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67658-2_21 · Cited by 9

I like fish, especially dolphins: Addressing Contradictions in Dialogue ModelingPreprint

Nie, Y., Williamson, M., Bansal, M., Kiela, D., & Weston, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.13391 · Cited by 9

A Survey on Out-of-Distribution Evaluation of Neural NLP Models

Li, X., Liu, M., Gao, S. B., & Buntine, W. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/749 · Cited by 25

Exploring the occupational biases and stereotypes of Chinese large language models

Jiang, L., Zhu, G., Sun, J., Cao, J., & Wu, J. X. (2025) · Scientific Reports · doi:10.1038/s41598-025-03893-w · Cited by 9

A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressions

Udagawa, T., Yamazaki, T., & Aizawa, A. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.67 · Cited by 9

A Case Study of Efficacy and Challenges in Practical Human-in-Loop Evaluation of NLP Systems Using Checklist

Bhatt, S., Jain, R., Dandapat, S., & Sitaram, S. (2021) · Cited by 8

Defining a New NLP Playground

Li, S., Han, C., Yu, P., Edwards, C., Li, M., Wang, X., Fung, Y. R., Yu, C., Tetreault, J., Hovy, E. H., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.799 · Cited by 11

If You Want to Go Far Go Together: Unsupervised Joint Candidate Evidence Retrieval for Multi-hop Question Answering

Yadav, V., Bethard, S. J., & Surdeanu, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.363 · Cited by 9

SyGNS: A Systematic Generalization Testbed Based on Natural Language Semantics

Yanaka, H., Mineshima, K., & Inui, K. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.10 · Cited by 10

Interpretable modular knowledge reasoning for machine reading comprehension

Ren, M., Huang, H., & Gao, Y. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-06975-2 · Cited by 8

Bias Mitigation for Toxicity Detection via Sequential Decisions

Cheng, L., Mosallanezhad, A., Silva, Y. N., Hall, D. L., & Liu, H. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531945 · Cited by 8

Curing the SICK and Other NLI Maladies

Kalouli, A.-L., Hu, H., Webb, A. F., Moss, L. S., & de Paiva, V. (2022) · Computational Linguistics · doi:10.1162/coli_a_00465 · Cited by 8

Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)

(2022) · doi:10.18653/v1/2022.woah-1 · Cited by 8

On Learning Interpreted Languages with Recurrent Models

Paperno, D. (2022) · Computational Linguistics · doi:10.1162/coli_a_00431 · Cited by 7

Bias-aware ranking from pairwise comparisons

Ferrara, A., Bonchi, F., Fabbri, F., Karimi, F., & Wagner, C. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-024-01024-z · Cited by 7

Harmonization Sometimes Harms

Klenner, M., Göhring, A., & Amsler, M. (2020) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-197961 · Cited by 7

Effects of Naturalistic Variation in Goal-Oriented Dialog

Ganhotra, J., Moore, R. J., Joshi, S., & Wadhawan, K. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.358 · Cited by 7

Robustness and Sensitivity of BERT Models Predicting Alzheimer’s Disease from Text

Novikova, J. (2021) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2021.wnut-1.37 · Cited by 11

Investigating Multi-source Active Learning for Natural Language Inference

Snijders, A., Kiela, D., & Margatina, K. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.160 · Cited by 10

Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecture

Yao, B., Jindal, I., Popa, L. C., Katsis, Y., Ghosh, S., He, L., Lu, Y., Srivastava, S., Li, Y., Hendler, J., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.778 · Cited by 10

In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers

Pradhan, V. K., Schaekermann, M., & Lease, M. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.828187 · Cited by 7

An Information Retrieval Approach to Building Datasets for Hate Speech DetectionPreprint

Rahman, M. M., Balakrishnan, D., Murthy, D., Kutlu, M., & Lease, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09775 · Cited by 16

Abductive Commonsense Reasoning Exploiting Mutually Exclusive Explanations

Zhao, W., Chiu, J., Cardie, C., & Rush, A. M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.831 · Cited by 18

Understanding Large Language Models : Towards Rigorous and Targeted Interpretability Using Probing Classifiers and Self-Rationalisation

Kunz, J. (2024) · Linköping studies in science and technology. Dissertations · doi:10.3384/9789180754712 · Cited by 7

On the (In)Effectiveness of Images for Text Classification

Ma, C., Shen, A., Yoshikawa, H., Iwakura, T., Beck, D., & Baldwin, T. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.4 · Cited by 7

Investigating Labeler Bias in Face Annotation for Machine Learning

Haliburton, L., Ghebremedhin, S., Welsch, R., Schmidt, A., & Mayer, S. (2024) · Frontiers in artificial intelligence and applications · doi:10.3233/faia240191 · Cited by 10

EdinburghNLP at WNUT-2020 Task 2: Leveraging Transformers with Generalized Augmentation for Identifying Informativeness in COVID-19 Tweets

Maveli, N. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.67 · Cited by 6

You Are What You Tweet: Profiling Users by Past Tweets to Improve Hate Speech Detection

Chaudhry, P., & Lease, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-96960-8_13 · Cited by 6

Towards Interpretable Reasoning over Paragraph Effects in Situation

Ren, M., Geng, X., Qin, T., Huang, H., & Jiang, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.548 · Cited by 6

Natural Language Inference with Mixed EffectsPreprint

Gantt, W., Kane, B., & White, A. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10501 · Cited by 6

How Data Workers Shape Datasets: The Role of Positionality in Data Collection and Annotation for Computer Vision

Scheuerman, M. K., Woodruff, A., & Brubaker, J. R. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757481 · Cited by 7

You Are What You Annotate: Towards Better Models through Annotator Representations

Deng, N., Zhang, X., Liu, S., Wu, W. M. C., Wang, L., & Mihalcea, R. F. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.832 · Cited by 19

Investigating Annotator Bias in Abusive Language Datasets

Wich, M., Widmer, C., Hagerer, G., & Groh, G. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_170 · Cited by 12

Explanation-based Finetuning Makes Models More Robust to Spurious Cues

Ludan, J. M., Meng, Y., Nguyen, T. D., Shah, S. A., Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.242 · Cited by 10

Annotation Sensitivity: Training Data Collection Methods Affect Model Performance

Kern, C. A., Eckman, S., Beck, J., Chew, R., Ma, B., & Kreuter, F. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.992 · Cited by 6

Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning

Weber, L., Bruni, E., & Hupkes, D. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.20 · Cited by 25

Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language

Efrat, A., Shaham, U., Kilman, D., & Levy, O. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.344 · Cited by 5

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts)

(2023) · doi:10.18653/v1/2023.acl-tutorials · Cited by 5

What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review

Thorn Jakobsen, T., & Rogers, A. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.354 · Cited by 1

Unifying Data Perspectivism and Personalization: An Application to Social Norms

Plepi, J., Neuendorf, B., Flek, L., & WELCH, C. A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.500 · Cited by 7

CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks

Ismayilzada, M., Paul, D., Montariol, S., Geva, M., & Bosselut, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.607 · Cited by 14

Assessing Gender Bias in Machine Translation

Jain, R. (2024) · International Conference on Applied Artificial Intelligence and Computing (ICAAIC) · doi:10.1109/icaaic60222.2024.10575537 · Cited by 6

Posthoc Verification and the Fallibility of the Ground Truth

Ding, Y., Botzer, N., & Weninger, T. (2022) · doi:10.18653/v1/2022.dadc-1.3 · Cited by 6

Unsupervised Explanation Generation via Correct Instantiations

Cheng, S., Wu, Z., Chen, J., Li, Z., Liu, Y., & Kong, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26494 · Cited by 5

Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future

(2021) · doi:10.18653/v1/2021.bppf-1 · Cited by 4

Segmentation of Leukoaraiosis on Noncontrast Head CT Using CT‐MRI Paired Data Without Human Annotation

Ryu, W., Song, J. W., Lim, J., Lee, J. H., Sunwoo, L., Kim, D., Kim, D., Bae, H., Lee, M., & Kim, B. J. (2025) · Brain and Behavior · doi:10.1002/brb3.70602 · Cited by 3

Modelling Speaker Attribution in Narrative Texts With Biased and Bias-Adjustable Neural Networks

Dönicke, T., Varachkina, H., Weimer, A. M., Gödeke, L., Barth, F., Gittel, B., Holler, A., & Sporleder, C. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.725321 · Cited by 3

Interventional Training for Out-Of-Distribution Natural Language Understanding

Yu, S., Jiang, J., Zhang, H., Niu, Y., Sun, Q., & Bing, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.799 · Cited by 3

Shortcut Learning in Financial Text Mining: Exposing the Overly Optimistic Performance Estimates of Text Classification Models under Distribution Shift

Caron, M. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020933 · Cited by 3

HumSet: Dataset of Multilingual Information Extraction and Classification for Humanitarian Crises Response

Fekih, S., Tamagnone, N., Minixhofer, B., Shrestha, R., Contla, X., Oglethorpe, E., & Rekabsaz, N. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.321 · Cited by 3

Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models

Malaviya, C., Bhatia, S., & Yatskar, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.438 · Cited by 3

SeSQL: A High-Quality Large-Scale Session-Level Chinese Text-to-SQL Dataset

Huang, S., Wang, L., Li, Z., Liu, Z., Dou, C., Yan, F., Xiao, X., Wu, H. R., & Zhang, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44693-1_42 · Cited by 3

Whose truth is it anyway? An experiment on annotation bias in times of factual opinion polarization

van der Velden, M. A. C. G., Löecherbach, F., van Atteveldt, W., Fokkens, A. S., Reuver, M., & Welbers, K. (2025) · Communication Methods and Measures · doi:10.1080/19312458.2025.2562034 · Cited by 3

Exploring the Effect of Author and Reader Identity in Online Story Writing: the STORIESINTHEWILD Corpus.

August, T., Sap, M., Clark, E., Reinecke, K., & Smith, N. A. (2020) · doi:10.18653/v1/2020.nuse-1.6 · Cited by 3

Building Low-Resource NER Models Using Non-Speaker Annotations

Tsygankova, T., Marini, F., Mayhew, S., & Roth, D. (2021) · doi:10.18653/v1/2021.dash-1.11 · Cited by 3

Toward Annotation Efficiency in Biased Learning Settings for Natural Language Processing

Effland, T. (2022) · Columbia Academic Commons (Columbia University) · doi:10.7916/znbv-rz34

Large Language Models for Data Annotation and Synthesis: A Survey

Tan, Z., Li, D., Wang, S., Beigi, A., Jiang, B., Bhattacharjee, A., Karami, M., Li, J., Cheng, L., & Liu, H. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.54 · Cited by 178

Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks

Röttger, P., Vidgen, B., Hovy, D., & Pierrehumbert, J. B. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.13 · Cited by 140

AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators

He, X., Lin, Z., Gong, Y., Jin, A.-L., Zhang, H., Chen, L., Jiao, J., Yiu, S., Duan, N., & Chen, W. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-industry.15 · Cited by 272

A Systematic Comparison of Large Language Models for Data Annotation in NER Tasks

Haq, M. U. U., Rigoni, D., & Sperduti, A. (2026) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4qnuuw7rjs24