The Effect of Natural Distribution Shift on Question Answering Models
Open accessAbstract
We build four new test sets for the Stanford Question Answering Dataset (SQuAD) and evaluate the ability of question-answering systems to generalize to new data. Our first test set is from the original Wikipedia domain and measures the extent to which existing systems overfit the original test set. Despite several years of heavy test set re-use, we find no evidence of adaptive overfitting. The remaining three test sets are constructed from New York Times articles, Reddit posts, and Amazon product reviews and measure robustness to natural distribution shifts. Across a broad range of models, we observe average performance drops of 3.8, 14.0, and 17.4 F1 points, respectively. In contrast, a strong human baseline matches or exceeds the performance of SQuAD models on the original domain and exhibits little to no drop in new domains. Taken together, our results confirm the surprising resilience of the holdout method and emphasize the need to move towards evaluation metrics that incorporate robustness to natural distribution shifts.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Model performance drops when the question answering model is evaluated on different topic domains, such as New York Times articles, Reddit posts, and Amazon product reviews.
“However, Miller et al. [42] found that model performance drops when the question answering model is evaluated on different topic domains, such as New York Times articles, Reddit posts, and Amazon product reviews.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [42]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Image-Based Recommendations on Styles and Substitutes
McAuley, J., Targett, C., Shi, Q., & van den Hengel, A. (2015) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2766462.2767755 · Cited by 2,348 · see citing works
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01603 · Cited by 2,322 · see citing works
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827 · see citing works
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · see citing works
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · see citing works
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · see citing works
AllenNLP: A Deep Semantic Natural Language Processing Platform
Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M. E., Schmitz, M., & Zettlemoyer, L. (2018) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/w18-2501 · Cited by 1,273 · see citing works
Baumgartner, J., Zannettou, S., Keegan, B., Squire, M., & Blackburn, J. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7347 · Cited by 881 · see citing works
Do ImageNet Classifiers Generalize to ImageNet?
Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019) · arXiv · doi:10.48550/arxiv.1902.10811 · Cited by 790 · see citing works
Self-citationMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C., & Renshaw, E. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 778 · see citing works
NewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 2× · see citing works
“1 and NewsQA (Trischler et al., 2017).”
BackgroundSemantically Equivalent Adversarial Rules for Debugging NLP models
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1079 · Cited by 435 · see citing works
Machine Comprehension Using Match-LSTM and Answer PointerPreprint
Wang, S., & Jiang, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07905 · Cited by 424 · see citing works
Simple and Effective Multi-Paragraph Reading Comprehension
Clark, C., & Gardner, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1078 · Cited by 402 · see citing works
SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
Dunn, M., Sagun, L., Higgins, M., Guney, V. U., Cirik, V., & Cho, K. (2017) · arXiv · doi:10.48550/arxiv.1704.05179 · Cited by 359 · see citing works
Learning the Difference that Makes a Difference with Counterfactually-Augmented DataPreprint
Kaushik, D., Hovy, E. H., & Lipton, Z. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.12434 · Cited by 346 · see citing works
Pretrained Transformers Improve Out-of-Distribution Robustness
Hendrycks, D., Liu, X., Wallace, E., Dziedzic, A., Krishnan, R., & Song, D. X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.244 · Cited by 312 · see citing works
Shen, Y., Huang, P.-S., Gao, J., & Chen, W. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098177 · Cited by 262 · see citing works
Preserving Statistical Validity in Adaptive Data Analysis
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. (2015) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/2746539.2746580 · Cited by 258 · see citing works
Modeling Biological Processes for Reading Comprehension
Berant, J., Srikumar, V., Chen, P.-C., Linden, A. V., Harding, B., Huang, B., Clark, P., & Manning, C. D. (2014) · Conference on Empirical Methods in Natural Language Processing · doi:10.3115/v1/d14-1159 · Cited by 241 · see citing works
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., & Chen, D. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5801 · Cited by 220 · see citing works
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading ComprehensionPreprint
Zhang, S., Liu, X., Liu, J., Gao, J., Duh, K., & Durme, B. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12885 · Cited by 201 · see citing works
Reinforced Mnemonic Reader for Machine Reading Comprehension
Hu, M., Peng, Y., Huang, Z., Qiu, X., Wei, F., & Zhou, M. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/570 · Cited by 199 · see citing works
Learning and Evaluating General Linguistic IntelligencePreprint
Yogatama, D., d’Autume, C. D. M., Connor, J. T., Kočiský, T., Chrzanowski, M., Kong, L., Lazaridou, A., Wang, L., Yu, L., Dyer, C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.11373 · Cited by 171 · see citing works
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension
Talmor, A., & Berant, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1485 · Cited by 157 · see citing works
A Meta-Analysis of Overfitting in Machine Learning
Roelofs, R., Shankar, V., Recht, B., Fridovich-Keil, S., Hardt, M., Miller, J. J., & Schmidt, L. (2019) · neural information processing systems · Cited by 138 · see citing works
Self-citationLearning Recurrent Span Representations for Extractive Question AnsweringPreprint
Lee, K., Salant, S., Kwiatkowski, T., Parikh, A. P., Das, D., & Berant, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01436 · Cited by 118 · see citing works
A Mutual Information Maximization Perspective of Language Representation LearningPreprint
Kong, L., d’Autume, C. D. M., Wang, L., Yu, L., Dai, Z., & Yogatama, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.08350 · Cited by 116 · see citing works
The Ladder: A Reliable Leaderboard for Machine Learning Competitions
Blum, A., & Hardt, M. (2015) · arXiv · doi:10.48550/arxiv.1502.04585 · Cited by 92 · see citing works
DCN+: Mixed Objective and Deep Residual Coattention for Question AnsweringPreprint
Xiong, C., Zhong, V., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00106 · Cited by 90 · see citing works
Cold Case: The Lost MNIST Digits
Yadav, C., & Bottou, L. (2019) · arXiv · doi:10.48550/arxiv.1905.10498 · Cited by 69 · see citing works
Evaluating NLP Models via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · Cited by 65 · see citing works
Learning to Compute Word Embeddings On the FlyPreprint
Bahdanau, D., Bosc, T., Jastrzȩbski, S., Grefenstette, E., Vincent, P., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.00286 · Cited by 62 · see citing works
MEMEN: Multi-layer Embedding with Memory Networks for Machine ComprehensionPreprint
Pan, B., Li, H., Zhao, Z., Cao, B., Cai, D., & He, X. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.09098 · Cited by 58 · see citing works
What do Models Learn from Question Answering Datasets?
Sen, P., & Saffari, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.190 · Cited by 51 · see citing works
pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
Joshi, M., Choi, E., Levy, O., Weld, D. S., & Zettlemoyer, L. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1362 · Cited by 47 · see citing works
Ruminating Reader: Reasoning with Gated Multi-hop Attention
Gong, Y., & Bowman, S. R. (2018) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/w18-2601 · Cited by 44 · see citing works
End-to-End Answer Chunk Extraction and Ranking for Reading ComprehensionPreprint
Yu, Y., Zhang, W., Hasan, K. F., Yu, M., Xiang, B., & Zhou, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09996 · Cited by 38 · see citing works
Contextualized Word Representations for Reading Comprehension
Salant, S., & Berant, J. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2088 · Cited by 34 · see citing works
Domain-agnostic Question-Answering with Adversarial Training
Lee, S., Kim, D., & Park, J. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5826 · Cited by 34 · see citing works
An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering
Longpre, S., Lu, Y., Tu, Z., & DuBois, C. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5829 · Cited by 34 · see citing works
Exploring Question Understanding and Adaptation in Neural-Network-Based Question AnsweringPreprint
Zhang, J., Zhu, X., Chen, Q., Dai, L.-R., Wei, S., & Jiang, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.04617 · Cited by 34 · see citing works
A Multi-Stage Memory Augmented Neural Network for Machine Reading Comprehension
Yu, S., Indurthi, S. R., Back, S., & Lee, H. (2018) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/w18-2603 · Cited by 31 · see citing works
The advantages of multiple classes for reducing overfitting from test set reusePreprint
Feldman, V., Frostig, R., & Hardt, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10360 · Cited by 19 · see citing works
Phase Conductor on Multi-layered Attentions for Machine ComprehensionPreprint
Liu, R., Wei, W., Mao, W., & Chikina, M. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.10504 · Cited by 18 · see citing works
Smarnet: Teaching Machines to Read and Comprehend Like HumanPreprint
Chen, Z., Yang, R., Cao, B., Zhao, Z., Cai, D., & He, X. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.02772 · Cited by 16 · see citing works
Natural Analysts in Adaptive Data AnalysisPreprint
Zrnic, T., & Hardt, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.11143 · Cited by 9 · see citing works
Question Answering Using Hierarchical Attention on Top of BERT Features
Osama, R. A. E.-K., El-Makky, N., & Torki, M. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5825 · Cited by 4 · see citing works
Flagged first, then by citations
1 of the 55 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- Hsin-Yuan Huang, Chenguang Zhu, Yelong Shen, and Weizhu Chen. Fusionnet: Fusing via fullyaware attention with application to machine comprehension. arXiv preprint arXiv:1711.07341 , 2017. · arXiv
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 3597-3608, 2019.
- Nikita Kitaev and Dan Klein. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Melbourne, Australia, July 2018. Association for Computational Linguistics.
- Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse. In Advances in Neural Information Processing Systems , pages 999310002, 2019.
- Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you dont know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 784-789, 2018.
- Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. Reasonet: Learning to stop reading in machine comprehension. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1047-1055, 2017.
Cited by
Sorted by citations
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Robust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00780 · Cited by 428 · see citing works
Self-citationWILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518 · see citing works
Utility is in the Eye of the User: A Critique of NLP Leaderboards
Ethayarajh, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.393 · Cited by 147 · see citing works
Flexible Clustered Federated Learning for Client-Level Data Distribution Shift
Duan, M., Liu, D., Ji, X., Wu, Y., Liang, L., Chen, X., Tan, Y., & Ren, A. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3134263 · Cited by 129 · see citing works
Robustness Gym: Unifying the NLP Evaluation Landscape
Goel, K., Rajani, N. F., Vig, J., Taschdjian, Z., Bansal, M., & Ré, C. (2021) · doi:10.18653/v1/2021.naacl-demos.6 · Cited by 109 · see citing works
Measure and Improve Robustness in NLP Models: A Survey
Wang, X., Wang, H., & Yang, D. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.339 · Cited by 90 · see citing works
Predicting with Confidence on Unseen Distributions
Guillory, D., Shankar, V., Ebrahimi, S., Darrell, T., & Schmidt, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00117 · Cited by 85 · see citing works
Self-citationTextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing
Wang, X., Liu, Q., Gui, T., Zhang, Q., Zou, Y., Zhou, X., Ye, J., Zhang, Y., Zheng, R., Pang, Z., et al. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.41 · Cited by 84 · see citing works
Miller, J., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., & Schmidt, L. (2021) · arXiv · doi:10.48550/arxiv.2107.04649 · Cited by 74 · see citing works
Self-citationAngeli, K. D., Gao, S. B., Danciu, I., Durbin, E. B., Wu, X., Stroup, A. M., Doherty, J. A., Schwartz, S., Wiggins, C. L., Damesyn, M. A., et al. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103957 · Cited by 70 · see citing works
Measuring Robustness in Deep Learning Based Compressive SensingPreprint
Darestani, M. Z., Chaudhari, A. S., & Heckel, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06103 · Cited by 66 · see citing works
Akrout, M., Feriani, A., Bellili, F., Mezghani, A., & Hossain, E. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3326399 · Cited by 58 · see citing works
Generative Language Models for Paragraph-Level Question Generation
Ushio, A., Alva-Manchego, F. E., & Camacho-Collados, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.42 · Cited by 41 · see citing works
Choi, M., Pei, J., Kumar, S., Shu, C., & Jurgens, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.699 · Cited by 38 · see citing works
A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures
Baughan, A., Wang, X., Liu, A., Mercurio, A., Chen, J., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581152 · Cited by 32 · see citing works
Varshney, N., Mishra, S. R., & Baral, C. R. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.158 · Cited by 35 · see citing works
Red Teaming Language Model Detectors with Language Models
Shi, Z., Wang, Y., Yin, F., Chen, X., Chang, K.-W., & Hsieh, C. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00639 · Cited by 39 · see citing works
Contrastive Domain Adaptation for Question Answering using Limited Text Corpora
Yue, Z., Kratzwald, B., & Feuerriegel, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.754 · Cited by 27 · see citing works
An Empirical Comparison of LM-based Question and Answer Generation Methods
Ushio, A., Alva-Manchego, F. E., & Camacho-Collados, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.899 · Cited by 30 · see citing works
Knowledge Enhanced Pretrained Language Models: A Compreshensive SurveyPreprint
Wei, X., Wang, S., Zhang, D., Bhatia, P., & Arnold, A. O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08455 · Cited by 38 · see citing works
Reliability Testing for Natural Language Processing Systems
Tan, S., Joty, S., Baxter, K., Taeihagh, A., Bennett, G. A., & Kan, M. (2021) · doi:10.18653/v1/2021.acl-long.321 · Cited by 27 · see citing works
Dealing with textual noise for robust and effective BERT re-ranking
Chen, X., He, B., Hui, K., Sun, L., & Sun, Y. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103135 · Cited by 24 · see citing works
The Evolution of Out-of-Distribution Robustness Throughout Fine-TuningPreprint
Andreassen, A., Bahri, Y., Neyshabur, B., & Roelofs, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15831 · Cited by 29 · see citing works
E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commercePreprint
Zhang, D., Yuan, Z., Liu, Y., Zhuang, F., Chen, H., & Xiong, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.02835 · Cited by 35 · see citing works
Do Offline Metrics Predict Online Performance in Recommender Systems?Preprint
Krauth, K., Dean, S., Zhao, A., Guo, W., Curmei, M., Recht, B., & Jordan, M. I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07931 · Cited by 20 · see citing works
Self-citationTowards Improving Selective Prediction Ability of NLP Systems
Varshney, N., Mishra, S. R., & Baral, C. R. (2022) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2022.repl4nlp-1.23 · Cited by 16 · see citing works
Learning to Perturb Word Embeddings for Out-of-distribution QA
Lee, S., Kang, M., Lee, J., & Hwang, S. J. (2021) · doi:10.18653/v1/2021.acl-long.434 · Cited by 18 · see citing works
TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language ProcessingPreprint
Gui, T., Wang, X., Zhang, Q., Liu, Q., Zou, Y., Zhou, X., Zheng, R., Zhang, C., Wu, Q., Ye, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.11441 · Cited by 20 · see citing works
A Practical Toolkit for Multilingual Question and Answer Generation
Ushio, A., Alva-Manchego, F. E., & Camacho-Collados, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.8 · Cited by 17 · see citing works
Liu, N. F., Lee, T., Jia, R., & Liang, P. (2021) · arXiv (Cornell University) · Cited by 17 · see citing works
MLDemon: Deployment Monitoring for Machine Learning SystemsPreprint
Ginart, A. A., Zhang, M. J., & Zou, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.13621 · Cited by 10 · see citing works
When Invariant Representation Learning Meets Label Shift: Insufficiency and Theoretical Insights
Luo, Y.-W., & Ren, C.-X. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3417214 · Cited by 8 · see citing works
Kamalloo, E., Rezagholizadeh, M., & Ghodsi, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.84 · Cited by 7 · see citing works
Exploring The Landscape of Distributional Robustness for Question Answering Models
Awadalla, A., Wortsman, M., Ilharco, G., Min, S., Magnusson, I., Hajishirzi, H., & Schmidt, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.441 · Cited by 6 · see citing works
Self-citationChen, C., Shen, Z., Klein, D., Stanovsky, G., Downey, D., & Lo, K. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.844 · Cited by 5 · see citing works
Weber, L., Bruni, E., & Hupkes, D. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.20 · Cited by 9 · see citing works
Multimodal Machine Translation
Liu, J. (2021) · IEEE Access · doi:10.1109/access.2021.3115135 · Cited by 5 · see citing works
Desai, A., Gunel, B., Ozturkler, B., Beg, H., Vasanawala, S. S., Hargreaves, B. A., Ré, C., Pauly, J. M., & Chaudhari, A. S. (2021) · CaltechAUTHORS (California Institute of Technology) · Cited by 5 · see citing works
Are Sample-Efficient NLP Models More Robust?
Liu, N. F., Kumar, A., Liang, P., & Jia, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.144 · Cited by 7 · see citing works
Do Question Answering Modeling Improvements Hold Across Benchmarks?
Liu, N. F., Lee, T., Jia, R., & Liang, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.736 · Cited by 9 · see citing works
Schlegel, V., Nenadić, G., & Batista-Navarro, R. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000171 · Cited by 22 · see citing works
Understanding and Testing Generalization of Deep Networks on Out-of-Distribution DataPreprint
Hu, R., Sang, J., Wang, J., Hu, R., & Jiang, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.09190 · Cited by 4 · see citing works
Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math ProblemsPreprint
Ye, T., Xu, Z., Li, Y., & Allen-Zhu, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5250631 · Cited by 9 · see citing works
Benchmarking Low-Shot Robustness to Natural Distribution Shifts
Singh, A., Sarangmath, K., Chattopadhyay, P., & Hoffman, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01487 · Cited by 3 · see citing works
Benchmarking Long-tail Generalization with Likelihood Splits
Godbole, A., & Jia, R. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.71 · Cited by 6 · see citing works
Why do classifier accuracies show linear trends under distribution shift?
Mania, H., & Sra, S. (2020) · arXiv · doi:10.48550/arxiv.2012.15483 · Cited by 3 · see citing works
Hey, T. (2023) · OECD eBooks · doi:10.1787/7b7b1bce-en · Cited by 2 · see citing works
Generalization Differences between End-to-End and Neuro-Symbolic Vision-Language Reasoning Systems
Wang, Z., Thomason, J., & Jia, R. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.345 · Cited by 2 · see citing works
Towards Robust Off-Policy Evaluation via Human Inputs
Singh, H., Joshi, S., Doshi-Velez, F., & Lakkaraju, H. (2022) · AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534198 · Cited by 2 · see citing works
Analyzing the Domain Robustness of Pretrained Language Models, Layer by Layer
Kashyap, A. R., Mehnaz, L., Malik, B., Waheed, A., Hazarika, D., Kan, M., & Shah, R. R. (2021) · National University of Singapore · Cited by 2 · see citing works
Sentence-Based Model Agnostic NLP Interpretability.Preprint
Rychener, Y., Renard, X., Seddah, D., Frossard, P., & Detyniecki, M. (2020) · arXiv (Cornell University) · Cited by 2 · see citing works
QUACKIE: A NLP Classification Task With Ground Truth ExplanationsPreprint
Rychener, Y., Renard, X., Seddah, D., Frossard, P., & Detyniecki, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.13190 · Cited by 3 · see citing works
Covariate Shift in High-Dimensional Random Feature RegressionPreprint
Tripuraneni, N., Adlam, B., & Pennington, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08234 · Cited by 4 · see citing works
New State-of-the-Art for Question Answering on Portuguese SQuAD v1.1
Silva, E. H. M. D., Laterza, J., & Faleiros, T. P. (2022) · Anais do Symposium on Knowledge Discovery, Mining and Learning (KDMiLe) · doi:10.5753/kdmile.2022.227787 · Cited by 1 · see citing works
QAQA-SS: An Improved Fuzzing Approach with Seed Scheduling Based on the UCB Algorithm for QA Systems
Chen, J., Wu, Y., Chen, H., Chen, J., & Akpaku, E. (2024) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c63300.2024.00089 · Cited by 1 · see citing works
Lessons from shortcomings in machine learning for medical imaging
Varoquaux, G. P., & Cheplygina, V. (2023) · OECD eBooks · doi:10.1787/b885eecd-en · Cited by 2 · see citing works
Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLP
Kim, H., Cho, H., Lee, S.-W., Kim, J., Park, C., Lee, S., Yoo, K. M., & Kim, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.392 · Cited by 1 · see citing works
On Degrees of Freedom in Defining and Testing Natural Language Understanding
Sugawara, S., & Tsugita, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.861 · Cited by 1 · see citing works
QA Domain Adaptation using Hidden Space Augmentation and Self-Supervised Contrastive Adaptation
Yue, Z., Zeng, H., Kratzwald, B., Feuerriegel, S., & Wang, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.147 · Cited by 2 · see citing works
Advanced Cross-Modal Gating for Enhanced Multimodal Sentiment AnalysisPreprint
Johnson, E., Patel, R., & Smith, M. (2024) · Preprints.org · doi:10.20944/preprints202408.0265.v1 · Cited by 2 · see citing works
Contrastive Domain Adaptation for Question Answering using Limited Text CorporaPreprint
Yue, Z., Kratzwald, B., & Feuerriegel, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13854 · Cited by 8 · see citing works
Automatic Model Evaluation using Feature Importance Patterns on Unlabeled Data
Silva, I. S., & Veloso, A. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892649
Tourassi, G. D., Shankar, M. A., & Wang, F. (2023) · OECD eBooks · doi:10.1787/0e1b4c2f-en
Eroom’s Law and the decline in the productivity of biopharmaceutical R&D
Scannell, J. W. (2023) · OECD eBooks · doi:10.1787/6f256d17-en
The end of Moore’s Law? Innovation in computer systems continues at a high pace
Kressel, H. (2023) · OECD eBooks · doi:10.1787/63e48242-en · Cited by 1 · see citing works
Quantifying the “cognitive extent” of science and how it has changed over time and across countries
Milojević, S. (2023) · OECD eBooks · doi:10.1787/1f717652-en
Interpretability: Should – and can – we understand the reasoning of machine-learning systems?
Cartwright, H. (2023) · OECD eBooks · doi:10.1787/2d0478b9-en
OECD (2023) · OECD eBooks · doi:10.1787/f1262928-en
Data Sampling and (In)stability in Machine Translation Evaluation
Lo, C.-K., & Knowles, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.826
Withdrawn: Question Answering through Enhanced Question Classification for Domain AdaptationPreprint
Foster, C., Nasir, W., & Warren, E. (2024) · Preprints.org · doi:10.20944/preprints202404.0933.v1
Enhanced Multimodal Integration Using TriFusion Networks for Comprehensive Emotion AnalysisPreprint
Wilson, E., Patel, R., Taylor, A., & Jones, L. (2024) · Preprints.org · doi:10.20944/preprints202408.0576.v1
Borghi, G., Iacomini, E., Oster, M., & Segala, C. (2025) · Oberwolfach Reports · doi:10.4171/owr/2024/56
Measuring Robustness to Natural Distribution Shifts in Image Classification
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., & Schmidt, L. (2020) · arXiv · doi:10.48550/arxiv.2007.00644 · Cited by 206 · see citing works
Self-citation
- Authors
- Miller, J., Krauth, K., Recht, B., & Schmidt, L.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2004.14444
- OpenAlex
- W3022116759
- Cited by
- 74
- References
- 55
- Citations by year
- 2025: 2 · 2024: 6 · 2023: 22 · 2022: 14 · 2021: 22 · 2020: 8
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- question answering systems, SQuAD, test set construction, overfitting, domain generalization, evaluation metrics, New York Times article, Reddit posts, Amazon reviews
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
