Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
Open accessThe DOI matches its Crossref record; 51 of the 54 references it cites were found in a registry or an index.
Abstract
Christopher Clark, Mark Yatskar, Luke Zettlemoyer. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“Main Models: We experiment with both the uncased BERT base model (Devlin et al., 2019), and the same recurrent model used for the synthetic data (see Appendix B).”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 3×
“2019; Jia and Liang, 2017), which were designed to break models that adopt superficial strategies on well known textual entailment (Bowman et al., 2015), reading comprehension (Rajpurkar et al., 2016), and VQA (Antol et al., 2015) datasets.”
BackgroundMethodBottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948 · cited 1×
“We use a popular implementation of the BottomUpToDown (Anderson et al., 2018) VQA model.”
MethodTraining Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 2×
“Our simplest ensemble is a product of experts (Hinton, 2002): p̂i = softmax (log(pi) + log(bi)) Equivalently, p̂i ∝ pi ◦ bi, where ◦ is elementwise multiplication.”
BackgroundMethodVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 5×
“Data: We experiment with a synthetic dataset built by modifying MNLI (Bowman et al., 2015).”
BackgroundMethodInfluentialIncorporating non-local information into information extraction systems by Gibbs sampling
Finkel, J. R., Grenager, T., & Manning, C. (2005) · Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05 · doi:10.3115/1219840.1219885 · Cited by 3,008 · cited 1×
“The bias-only model uses NER tags, identified by running the Stanford NER Tag-ger (Finkel et al., 2005) on the passage, as input.”
MethodYago: a core of semantic knowledge
Suchanek, F. M., Kasneci, G., & Weikum, G. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242667 · Cited by 2,512
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 2×
“…from us-ing words that occur in the context passage, VQA 2.0 (Goyal et al., 2018) selects examples to limit the effectiveness of question-only models, and others have filtered examples solvable by simple baselines (Yang et al., 2018; Zhang et al., 2018b; Clark et al., 2018; Zellers et al., 2018).”
BackgroundInfluentialTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 4×
“Data : We construct a changing-prior QA dataset from TriviaQA (Joshi et al., 2017) by categorizing questions into three classes, Person, Location, and Other, based on what kind of entity they are asking about.”
MethodInfluentialSelf-citationTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650
Self-citationBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 6×
“It has the following stages: Embed : Embed the words using a character CNN, following what was done by Seo et al. (2017), and the fasttext crawl word embed-dings (Mikolov et al., 2018), then run a shared BiLSTM over the results.”
BackgroundMethodInfluentialAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 5×
“2019; Jia and Liang, 2017), which were designed to break models that adopt superficial strategies on well known textual entailment (Bowman et al., 2015), reading comprehension (Rajpurkar et al., 2016), and VQA (Antol et al., 2015) datasets.”
BackgroundMethodInfluentialMitigating Unwanted Biases with Adversarial Learning
Zhang, B. H., Lemoine, B., & Mitchell, M. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278779 · Cited by 1,387 · cited 2×
“…from us-ing words that occur in the context passage, VQA 2.0 (Goyal et al., 2018) selects examples to limit the effectiveness of question-only models, and others have filtered examples solvable by simple baselines (Yang et al., 2018; Zhang et al., 2018b; Clark et al., 2018; Zellers et al., 2018).”
BackgroundInfluentialEnhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113 · cited 2×
“The model we use for NLI is based on ESIM (Chen et al., 2017).”
MethodAudio Adversarial Examples: Targeted Attacks on Speech-to-Text
Carlini, N., & Wagner, D. (2018) · 2018 IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2018.00009 · Cited by 1,035 · cited 1×
“Evaluating models on out-of-domain examples built by applying minor perturbations to existing examples has also been the subject of recent study (Szegedy et al., 2014; Belinkov and Bisk, 2018; Carlini and Wagner, 2018; Glockner et al., 2018).”
BackgroundCoQA: A Conversational Question Answering Challenge
Reddy, S., Chen, D., & Manning, C. D. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00266 · Cited by 975 · cited 2×
“For example, both CoQA (Reddy et al., 2019) and QuAC (Choi et al., 2018) take steps to prevent annotators from us-ing words that occur in the context passage, VQA 2.0 (Goyal et al., 2018) selects examples to limit the effectiveness of question-only models, and others have filtered examples solvable…”
BackgroundInfluentialRight for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944 · cited 5×
“We show scores for individual heuristics used in HANS in Appendix C.”
MethodInfluentialAnnotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907 · cited 1×
“For example, entailment models trained on MNLI (Bowman et al., 2015) will guess an answer based solely on the presence of particular keywords (Gururangan et al., 2018) or whether sentences pairs contain the same words (Mc-Coy et al., 2019), while QA models trained on SQuAD (Rajpurkar et al., 2016)…”
MethodMen Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · cited 2×
“A related task is preventing models from using particular problematic dataset features, which is often studied from the perspective of fairness (Zhao et al., 2017; Burns et al., 2018).”
BackgroundSelf-citationQuAC: Question Answering in Context
Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1241 · Cited by 714 · cited 3×
“For example, both CoQA (Reddy et al., 2019) and QuAC (Choi et al., 2018) take steps to prevent annotators from us-ing words that occur in the context passage, VQA 2.0 (Goyal et al., 2018) selects examples to limit the effectiveness of question-only models, and others have filtered examples solvable…”
BackgroundInfluentialSelf-citationSWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Zellers, R., Bisk, Y., Schwartz, R., & Choi, Y. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1009 · Cited by 690 · cited 1×
“…from us-ing words that occur in the context passage, VQA 2.0 (Goyal et al., 2018) selects examples to limit the effectiveness of question-only models, and others have filtered examples solvable by simple baselines (Yang et al., 2018; Zhang et al., 2018b; Clark et al., 2018; Zellers et al., 2018).”
BackgroundInfluentialDon't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Agrawal, A., Batra, D., Parikh, D., & Kembhavi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00522 · Cited by 566 · cited 11×
“Recent concern about dataset bias has led researchers to re-examine many popular datasets, resulting in the discovery of a wide variety of biases (Agrawal et al., 2018; Anand et al., 2018; Min et al., 2019; Schwartz et al., 2017).”
BackgroundMethodInfluentialMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487 · cited 1×
“For example, entailment models trained on MNLI (Bowman et al., 2015) will guess an answer based solely on the presence of particular keywords (Gururangan et al., 2018) or whether sentences pairs contain the same words (Mc-Coy et al., 2019), while QA models trained on SQuAD (Rajpurkar et al., 2016)…”
MethodThink you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · cited 2×
“…from us-ing words that occur in the context passage, VQA 2.0 (Goyal et al., 2018) selects examples to limit the effectiveness of question-only models, and others have filtered examples solvable by simple baselines (Yang et al., 2018; Zhang et al., 2018b; Clark et al., 2018; Zellers et al., 2018).”
BackgroundInfluentialConstructing Datasets for Multi-hop Reading Comprehension Across Documents
Welbl, J., Stenetorp, P., & Riedel, S. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00021 · Cited by 456 · cited 1×
“Some questions in recent multi-hop QA datasets (Yang et al., 2018; Welbl et al., 2018) can be solved by single-hop models (Chen and Dur-rett, 2019; Min et al., 2019).”
BackgroundMachine Comprehension Using Match-LSTM and Answer Pointer
Wang, S., & Jiang, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07905 · Cited by 408 · cited 1×
“We consider two extractive QA datasets, which we treat as a joint classification task where the model must select the start and end answer token (Wang and Jiang, 2017).”
MethodWomen Also Snowboard: Overcoming Bias in Captioning Models
Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_47 · Cited by 404 · cited 1×
“A related task is preventing models from using particular problematic dataset features, which is often studied from the perspective of fairness (Zhao et al., 2017; Burns et al., 2018).”
BackgroundSimple and Effective Multi-Paragraph Reading Comprehension
Clark, C., & Gardner, M. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1078 · Cited by 396 · cited 1×
“We use the approach of Clark and Gardner (2018) to break up the documents into passages of at most 400 tokens, and rank the passages for each question using their linear paragraph ranker.”
MethodSelf-citationLearning Not to Learn: Training Deep Neural Networks With Biased Data
Kim, B., Kim, H., Kim, K., Kim, S., & Kim, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00922 · Cited by 345 · cited 1×
“A popular approach is to use an adversary to remove information about a target feature, often gender or ethnicity, from a model’s internal representations (Edwards and Storkey, 2016; Wang et al., 2018; Kim et al., 2019).”
BackgroundBreaking NLI Systems with Sentences that Require Simple Lexical Inferences
Glockner, M., Shwartz, V., & Goldberg, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2103 · Cited by 319 · cited 1×
“Evaluating models on out-of-domain examples built by applying minor perturbations to existing examples has also been the subject of recent study (Szegedy et al., 2014; Belinkov and Bisk, 2018; Carlini and Wagner, 2018; Glockner et al., 2018).”
BackgroundYin and Yang: Balancing and Answering Binary Visual Questions
Zhang, P., Goyal, Y., Summers-Stay, D., Batra, D., & Parikh, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.542 · Cited by 312 · cited 2×
“…objectives to force models to understand all the input (Lewis and Fan, 2019), carefully designed model architecture (Agrawal et al., 2018; Zhang et al., 2016), or adversarial removal of class-indicative features from model’s internal representations (Ramakrish-nan et al., 2018; Zhang et…”
Background(2018) · International Conference on Learning Representations · Cited by 285
Obstructing Classification via Projection
Haghighatkhah, P., Meulemans, W., Speckmann, B., Urhausen, J., & Verbeek, K. (2021) · Leibniz international proceedings in informatics · doi:10.4230/lipics.mfcs.2021.56 · Cited by 229
Revisiting Visual Question Answering Baselines
Jabri, A., Joulin, A., & van der Maaten, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_44 · Cited by 218
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
Zhang, S., Liu, X., Liu, J., Gao, J., Duh, K., & Durme, B. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12885 · Cited by 200 · cited 2×
“…from us-ing words that occur in the context passage, VQA 2.0 (Goyal et al., 2018) selects examples to limit the effectiveness of question-only models, and others have filtered examples solvable by simple baselines (Yang et al., 2018; Zhang et al., 2018b; Clark et al., 2018; Zellers et al., 2018).”
BackgroundInfluentialUnlearn Dataset Bias in Natural Language Inference by Fitting the Residual
He, H., Zha, S., & Wang, H. (2019) · Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019) · doi:10.18653/v1/d19-6115 · Cited by 157 · cited 1×
“Concurrently, He et al. (2019) also suggested using a product-of-experts ensemble to train unbiased models, but we consider a wider variety of ensembling approaches and test on additional domains.”
BackgroundProbing the Need for Visual Context in Multimodal Machine Translation
Caglayan, O., Madhyastha, P., Specia, L., & Barrault, L. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1422 · Cited by 139 · cited 1×
“For example, many joint natu-1 github.com/chrisc36/debias ral language processing and vision datasets can be partially solved by models that ignore the vision aspect of the task (Jabri et al., 2016; Zhang et al., 2016; Anand et al., 2018; Caglayan et al., 2019).”
BackgroundCombining Retrieval, Statistics, and Inference to Answer Elementary Science Questions
Clark, P., Etzioni, O., Khot, T., Sabharwal, A., Tafjord, O., Turney, P., & Khashabi, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10325 · Cited by 138 · cited 1×
“Additional examples include story completion (Schwartz et al., 2017) and multiple choice questions (Clark et al., 2016, 2018).”
BackgroundCompositional Questions Do Not Necessitate Multi-hop Reasoning
Min, S., Wallace, E., Singh, S., Gardner, M., Hajishirzi, H., & Zettlemoyer, L. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1416 · Cited by 138 · cited 2×
“Recent concern about dataset bias has led researchers to re-examine many popular datasets, resulting in the discovery of a wide variety of biases (Agrawal et al., 2018; Anand et al., 2018; Min et al., 2019; Schwartz et al., 2017).”
BackgroundSelf-citationRobust Machine Comprehension Models via Adversarial Training
Wang, Y., & Bansal, M. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2091 · Cited by 110
The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task
Schwartz, R., Sap, M., Konstas, I., Zilles, L., Choi, Y., & Smith, N. A. (2017) · Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) · doi:10.18653/v1/k17-1004 · Cited by 108 · cited 2×
“Recent concern about dataset bias has led researchers to re-examine many popular datasets, resulting in the discovery of a wide variety of biases (Agrawal et al., 2018; Anand et al., 2018; Min et al., 2019; Schwartz et al., 2017).”
BackgroundGenerative Question Answering: Learning to Answer the Whole Question.
Lewis, M., & Fan, A. (2018) · International Conference on Learning Representations · Cited by 86
Understanding Dataset Design Choices for Multi-hop Reasoning
Chen, J., & Durrett, G. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1405 · Cited by 82 · cited 1×
“, 2018) can be solved by single-hop models (Chen and Durrett, 2019; Min et al., 2019).”
BackgroundAdversarial Regularization for Visual Question Answering: Strengths, Shortcomings, and Side Effects
Grand, G., & Belinkov, Y. (2019) · Proceedings of the Second Workshop on Shortcomings in Vision and Language · doi:10.18653/v1/w19-1801 · Cited by 76 · cited 3×
“Instead we follow prior work (Grand and Belinkov, 2019; Ramakr-ishnan et al., 2018) and perform model selection on the test set.”
BackgroundMethodLogarithmic opinion pools for conditional random fields
Smith, A., Cohn, T., & Osborne, M. (2005) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1219840.1219843 · Cited by 64 · cited 1×
“We explore several different ensembling methods, building on product-of-expert style approaches (Hinton, 2002; Smith et al., 2005).”
MethodBlindfold Baselines for Embodied QAPreprint
Anand, A., Belilovsky, E., Kastner, K., Larochelle, H., & Courville, A. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.05013 · Cited by 57 · cited 2×
“Recent concern about dataset bias has led researchers to re-examine many popular datasets, resulting in the discovery of a wide variety of biases (Agrawal et al., 2018; Anand et al., 2018; Min et al., 2019; Schwartz et al., 2017).”
BackgroundOn Adversarial Removal of Hypothesis-only Bias in Natural Language Inference
Belinkov, Y., Poliak, A., Shieber, S., Van Durme, B., & Rush, A. (2019) · Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019) · doi:10.18653/v1/s19-1028 · Cited by 56 · cited 1×
“…(Lewis and Fan, 2019), carefully designed model architecture (Agrawal et al., 2018; Zhang et al., 2016), or adversarial removal of class-indicative features from model’s internal representations (Ramakrish-nan et al., 2018; Zhang et al., 2018a; Belinkov et al., 2019; Grand and Belinkov, 2019).”
BackgroundMisleading Failures of Partial-input Baselines
Feng, S., Wallace, E., & Boyd-Graber, J. L. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1554 · Cited by 55
Adversarial Removal of Gender from Deep Image Representations.
Wang, T., Zhao, J., Yatskar, M., Chang, K.-W., & Ordóñez, V. (2018) · Cited by 13
Self-citation
Research works printed but not linked: 4
- Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases https://aclanthology.org/D19-1418/ (Clark et al., EMNLP-IJCNLP 2019)
- Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (Clark et al., EMNLP-IJCNLP 2019)
- Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4069–4082, Hong Kong, China. Association for Computational Linguistics.
- ACL materials are Copyright © 1963–2026 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License.
Cited by
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P., Ding, N., & Soricut, R. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00356 · Cited by 612
“Another supporting evidence is the issue of strong language priors (well-known in the VQA community), which led to interest in adversarial test sets and other methods to overcome strong language biases [1, 30, 14, 7].”
BackgroundCounterfactual VQA: A Cause-Effect Look at Language Bias
Niu, Y., Tang, K., Zhang, H., Lu, Z., Hua, X.-S., & Wen, J.-R. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01251 · Cited by 418
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Chen, L., Yan, X., Xiao, J., Zhang, H., Pu, S., & Zhuang, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01081 · Cited by 341
“2) fusion-based [10, 14, 27]: they late fuse the predicted answer distributions of the two models, and derive the training gradients based on the fused answer distributions.”
BackgroundMethodInfluentialPredictive biases in natural language processing models: A conceptual framework and overview
Shah, D. S., Schwartz, H. A., & Hovy, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.468 · Cited by 241
“tures and use it in ensemble learning (Clark et al., 2019).”
BackgroundHybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
Chen, W., Zha, H., Chen, Z., Xiong, W., Wang, H., & Wang, W. Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.91 · Cited by 201
“Annotation De-biasing As has been suggested in previous papers (Kaushik and Lipton, 2018; Chen and Durrett, 2019; Clark et al., 2019), the existing benchmarks on multi-hop reasoning question answering have annotation biases, which makes designing multi-hop models unnecessary.”
Background
Metadata
- Authors
- Clark, C., Yatskar, M., & Zettlemoyer, L.
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 4067-4080
- OpenAlex
- W2970019270
- Semantic Scholar
- ba783d92d0eaf6a7bff6ced7660150ce38016bbc
- Cited by
- 381
- References
- 54
- Citations by year
- 2026: 15 · 2025: 40 · 2024: 73 · 2023: 81 · 2022: 66 · 2021: 73 · 2020: 31 · 2019: 2
- FWCI
- 16.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- data bias, ensemble methods, bias mitigation, training data quality, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.