RACE: Large-scale ReAding Comprehension Dataset From Examinations
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task.Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning.In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%).We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [112]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, to our knowledge, most of the existing datasets are not of direct relevance to teaching and learning, except for RACE (Lai, Xie, Liu, Yang, & Hovy, 2017) and LearningQ (Chen, Yang, Hauff, & Houben, 2018).”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Lai, Xie, Liu, Yang, & Hovy, 2017
The cited source says what the claim attributes to it.
In the citing paper
“On RACE-h and RACE-m (Lai et al., 2017), Chinchilla greatly outperforms Gopher , improving accuracy by more than 10% in both cases-see Table 7.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Lai et al., 2017
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We consider only the case of human-readable prompts to large models, which contrasts with methods such as prompt-tuning [16,20], which learns a continuous prompt embedding, and auto-prompting [33], which attempts to search for prompts from scratch, both of which require a training step.”
From Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We choose word embedding size d = 100 and use the 100-dimensional Glove word embedding (Pennington et al., 2014) as embedding initialization.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Instead of evaluating the accuracy, researchers need to use F1 score, BLEU (Papineni et al., 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the questions come without candidate spans.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 8×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
MethodResultInfluentialTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 3×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 1×
“, 2016) and recently proposed TriviaQA (Joshi et al., 2017).”
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578 · cited 2×
“, 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the”
BackgroundSelf-citationNewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 11×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodResultInfluentialMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610 · cited 4×
“Although efforts have been made with a similar motivation, including the MCTest dataset created by (Richardson et al., 2013) (containing 500 passages and 2000 questions) and several others (Peñas et al.”
BackgroundMethodInfluentialA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491 · cited 11×
“However, both require limited reasoning ability (Chen et al., 2016).”
BackgroundMethodInfluential
Cited by
Items 301–400 of 1,038
Adversarial Training for Commonsense Inference
Pereira, L. K., Liu, X., Cheng, F., Asahara, M., & Kobayashi, I. (2020) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2020.repl4nlp-1.8 · Cited by 33
Maurya, K. K., & Desarkar, M. S. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3411997 · Cited by 30
“n N o Considering the fact that the current performance ceiling of humans on the RACE dataset is 95% [11] (for identification of correct answer given article, question and four options), confusing humans is a challenging task. p u t f Grammatical Cor-rectness metric received high score over…”
BackgroundFrom F to A on the New York Regents Science Exams — An Overview of the Aristo Project
Clark, P. E., Etzioni, O., Khashabi, D., Khot, T., Mishra, B. D., Richardson, K., Sabharwal, A., Schoenick, C., Tafjord, O., Tandon, N., et al. (2020) · AI Magazine · doi:10.1609/aimag.v41i4.5304 · Cited by 172
Li, Y., & Zhao, H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.176 · Cited by 31
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 714
“While many large-scale annotated general domain QA datasets have been introduced (Ra-jpurkar et al., 2016; Lai et al., 2017; Koˇcisk`y Question: Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting?”
BackgroundMitra, A., Banerjee, P., Pal, K. K., Mishra, S. R., & Baral, C. R. (2019) · arXiv (Cornell University) · Cited by 29
ETC: Encoding Long and Structured Inputs in TransformersPreprint
Ainslie, J., Ontañón, S., Alberti, C., Cvicek, V., Fisher, Z., Pham, P., Ravula, A., Sanghai, S. K., Wang, Q., & Li, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.08483 · Cited by 27
“…process (RoBERTa (Liu et al., 2019)), scaling the internal representation of the model (ALBERT (Lan et al., 2019)), or both (T5 (Raffel et al., 2019)), outperforming the original BERT model in tasks such as GLUE (Wang et al., 2018), SQuAD (Rajpurkar et al., 2016) or RACE (Lai et al., 2017).”
MethodFinding Generalizable Evidence by Learning to Convince Q&A Models
Perez, E., Karamcheti, S., Fergus, R., Weston, J., Kiela, D., & Cho, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1244 · Cited by 27
“Multiple-choice QA: RACE and DREAM For our experiments, we use RACE (Lai et al., 2017) and DREAM (Sun et al.”
BackgroundMethodInfluentialQAConv: Question Answering on Informative Conversations
Wu, C.-S., Madotto, A., Liu, W., Fung, P., & Xiong, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.370 · Cited by 42
Xu, W., Zhang, H., Cai, D., & Lam, W. P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.90 · Cited by 50
Chen, S., Jin, S., & Xie, X. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468569 · Cited by 36
“We build a model based on the base-size BERT to show the early performance of such huge PTMs. ROBERTa: ROBERTa [27] is an improved version of BERT and shows more robust and even better performance than BERT on many tasks such as SQuAD [33] and RACE [25].”
BackgroundMethodCited text span identification for scientific summarisation using pre-trained encoders
Zerva, C., Nghiem, M.-Q., Nguyen, N. T. H., & Ananiadou, S. (2020) · Scientometrics · doi:10.1007/s11192-020-03455-z · Cited by 30
“It has been shown to outperform BERT on various tasks, including several paraphrasing tasks of the GLUE benchmarks as well as question answering tasks (SQUAD) (Rajpurkar et al. 2016), RACE (Lai et al. 2017).”
MethodDual Multi-head Co-attention for Multi-choice Reading Comprehension.Preprint
Zhu, P., Zhao, H., & Li, X. (2020) · arXiv (Cornell University) · Cited by 29
Combining pre-trained language models and structured knowledgePreprint
Colon-Hernandez, P., Havasi, C., Alonso, J. B., Huggins, M., & Breazeal, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.12294 · Cited by 65
“The authors use the RACE (Lai et al. 2017) dataset, and based”
MethodInfluentialFine-Tuned Understanding: Enhancing Social Bot Detection With Transformer-Based Classification
Sallah, A., Alaoui, E. A. A., Agoujil, S., Wani, M. A., Hammad, M. A., Maleh, Y., & El‐Latif, A. A. A. (2024) · IEEE Access · doi:10.1109/access.2024.3440657 · Cited by 43
“Consequently, RoBERTa has demonstrated superior performance over BERT in natural language processing (NLP) benchmarks, such as the General Language Understanding Evaluation (GLUE) [78], the Stanford Question Answering Dataset (SQuAD) [79], and the RACE dataset [80].”
MethodElimiNet: A Model for Eliminating Options for Reading Comprehension with Multiple Choice Questions
Parikh, S., Sai, A. B., Nema, P., & Khapra, M. M. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/594 · Cited by 33
“Finally, [ Lai et al. , 2017 ] adapt the the models proposed in [ Chen et al. , 2016; Dhingra et al. , 2017 ] for cloze-style RC and use them to address the problem of RC-MCQ.”
BackgroundHardalov, M., Mihaylov, T., Zlatkova, D., Dinkov, Y., Koychev, I. K., & Nakov, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.438 · Cited by 111
Li, R., Jiang, Z., Wang, L., Lu, X., Zhao, M., & Chen, D. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106936 · Cited by 28
Benchmarking Robustness of Machine Reading Comprehension Models
Si, C., Yang, Z., Cui, Y., Ma, W., Liu, T., & Wang, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.56 · Cited by 39
jurBERT: A Romanian BERT Model for Legal Judgement Prediction
Masala, M., Iacob, R. C. A., Uban, A. S., Cidota, M. A., Velicu, H., Rebedea, T. E., & Popescu, M. (2021) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2021.nllp-1.8 · Cited by 42
“…et al., 2020; Zhang et al., 2019) have achieved state-of-the-art results on several generic NLP tasks ranging from nat-ural language understanding (Wang et al., 2018, 2019; Lai et al., 2017), question answering (Ra-jpurkar et al., 2018; Reddy et al., 2019) to Text-to-SQL (Yu et al., 2018, 2019b,a).”
BackgroundMLEC-QA: A Chinese Multi-Choice Biomedical Question Answering Dataset
Li, J., Zhong, S., & Chen, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.698 · Cited by 72
“To better understand our dataset, we manually inspected 10 sets of examination papers (2 sets for each sub-field), and summarize the most frequent reasoning types of the questions from MLEC-QA and previous works (Lai et al., 2017; Zhong et al., 2020).”
MethodDiverse Distractor Generation for Constructing High-Quality Multiple Choice Questions
Xie, J., Peng, N., Cai, Y., Wang, T., & Huang, Q. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3138706 · Cited by 27
Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question AnsweringPreprint
Feng, Y., Chen, X., Lin, B. Y., Wang, P., Yan, J., & Ren, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00646 · Cited by 25
“They generally fall into two paradigms: 1) Fine-tuning LMs on large-scale general-domain datasets ( e.g. RACE (Lai et al., 2017)) or on knowledge-rich text.”
BackgroundMulti-class Text Classification using BERT-based Active LearningPreprint
Prabhu, S., Mohamed, M., & Misra, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14289 · Cited by 55
“BERT-based [Devlin et al. 2019; Sanh et al. 2020; Yinhan Liu et al. 2019] Deep Learning models have been proven to achieve state-of-the-art results on GLUE [Wang et al. 2018], RACE [Lai et al. 2017] and SQuAD [Rajpurkar et al. 2018, 2016].”
Don't Make Your LLM an Evaluation Benchmark Cheater
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., & Han, J. (2023) · arXiv · doi:10.48550/arxiv.2311.01964 · Cited by 25
“We use chain-of-thought prompting and reuse the prompts provided by Wei et al. (2022) for evaluation and report the accuracy of LLMs. • Reading Comprehension Tasks : we select three English datasets RACE-Middle and RACE-High (Lai et al., 2017), CoQA (Reddy et al., 2019) and two Chinese datasets CMRC2018 (Cui et al., 2…”
MethodA T5-based interpretable reading comprehension model with more accurate evidence training
Guan, B., Zhu, X., & Yuan, S. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103584 · Cited by 31
Nangia, N., Sugawara, S., Trivedi, H. K., Warstadt, A., Vania, C., & Bowman, S. R. (2021) · doi:10.18653/v1/2021.acl-long.98 · Cited by 38
“We fine-tune RoBERTaLARGE on RACE (Lai et al., 2017), a large-scale multiplechoice QA dataset that is commonly used for training (Sun et al.”
MethodInfluentialTellMeWhy: A Dataset for Answering Why-Questions in Narratives
Lal, Y. K., Chambers, N., Mooney, R. J., & Balasubramanian, N. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.53 · Cited by 65
“For 201 questions, they design multiple-choice questions derived from (Lai et al., 2017) to test understanding of different categories of events.”
BackgroundPeng, W., Hu, Y., Yu, J., Xing, L., & Xie, Y. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107364 · Cited by 30
Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI
Tian, J., Li, Y., Chen, W., Xiao, L., He, H., & Jin, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.303 · Cited by 141
“In most simple reasoning scenarios, such as RACE (Lai et al., 2017) and SQuAD (Ra-jpurkar et al., 2016), the performance of XLNet is usually better than RoBERTa’s.”
MethodLogiformer: A Two-Branch Graph Transformer Network for Interpretable Logical Reasoning
Xu, F., Liu, J., Lin, Q., Pan, Y., & Zhang, L. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532016 · Cited by 38
“A large number of models have achieved competitive performances in some famous datasets, such as SQuAD[23, 24], RACE[14].”
BackgroundWhat’s Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering
Khot, T., Sabharwal, A., & Clark, P. E. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1281 · Cited by 28
“Since this and several recent multi-hop datasets use the multiple-choice setting (Welbl et al., 2018; Khashabi et al., 2018; Lai et al., 2017), we assume access to potential answers to a question.”
BackgroundKaushik, D., Kiela, D., Lipton, Z. C., & Yih, W.-T. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.517 · Cited by 40
“, 2017), RACE (Lai et al., 2017), and TextbookQA (Kembhavi et al.”
BackgroundProceedings of the 2nd Workshop on Natural Language Processing for Conversational AI
(2020) · doi:10.18653/v1/2020.nlp4convai-1 · Cited by 23
“# of passages # of examples MRQA (span-based) 386,384 516,819 DREAM (multi-choice) 6,444 10,197 RACE (multi-choice) 27,933 97,687 MultiWOZ 8,420 298,978* Table 4: Statistics of datasets used.”
InfluentialUnderstanding Commonsense Inference Aptitude of Deep Contextual Representations
Da, J., & Kasai, J. (2019) · Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing · doi:10.18653/v1/d19-6001 · Cited by 22
“(2019) explores fine-tuning on indomain data, similarly to our fine-tuning on the RACE dataset (Lai et al., 2017).”
BackgroundMethodInfluentialAn Investigation of Applying Large Language Models to Spoken Language Learning
Gao, Y., Nuchged, B., Li, Y., & Peng, L. (2023) · Applied Sciences · doi:10.3390/app14010224 · Cited by 41
Multi-source Meta Transfer for Low Resource Multiple-Choice Question Answering
Yan, M., Zhang, H., Jin, D., & Zhou, J. T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.654 · Cited by 34
“We also compare the base-line that simply combines RACE and MCTEST datasets to be one large training source, denoted by MML(M ∪ R), dramatically drops the performance and only achieves 29 .”
BackgroundInfluentialQuestion Answering Infused Pre-training of General-Purpose Contextualized Representations
Jia, R., Lewis, M., & Zettlemoyer, L. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.59 · Cited by 37
Baradaran, R., & Amirkhani, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.08.095 · Cited by 25
Weakly Supervised Learning for Textbook Question Answering
Ma, J., Chai, Q., Huang, J., Liu, J., You, Y., & Zheng, Q. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3180563 · Cited by 28
“The textual ISAAQ is pre-trained on RACE [20], ARC-Easy, ARC-Challenge [21] and OpenBookQA [22] datasets and fine-tuned on CK12-QA.”
MethodAligning Source Visual and Target Language Domains for Unpaired Video Captioning
Liu, F., Wu, X., You, C., Ge, S., Zou, Y., & Sun, X. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3132229 · Cited by 44
“model [33] or an information retrieval model [10]; 2) Medical Image Classification [34] which includes a image segmentation model [35], [36] followed by a classification model; 3) Unpaired Machine Translation [12], [37], [38] which stacks a source-to-pivot translation model followed by a pivot-to-target translation mo…”
MethodA Survey on Neural Machine Reading ComprehensionPreprint
Qiu, B., Xu, C., Xu, J., & Sun, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.03824 · Cited by 31
“Similarly, the evaluation metric is accuracy. et al. , 2017 ] collected more than 20,000 articles and 100,000 questions from Chinese middle and high school students’ English exams, covering a wide range of fields.”
BackgroundComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase ClustersPreprint
Abujabal, A., Roy, R. S., Yahya, M., & Weikum, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.09528 · Cited by 34
“Reading comprehension (RC) is a recently introduced task, where the goal is to answer a question from a given textual paragraph (Kociský et al., 2017; Lai et al., 2017; Rajpurkar et al., 2016; Trischler et al., 2017; Yang et al., 2015).”
BackgroundOptimal Subarchitecture Extraction For BERTPreprint
de Wynter, A., & Perry, D. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10499 · Cited by 32
“The Reading Comprehension from Examinations (RACE) dataset is a multiple-choice questionanswering dataset that emphasizes reading comprehension across multiple NLU tasks (e.g. paraphrasing, summarization, single and multi-sentence reasoning, etcetera), and is collected from the English examinations for high school and…”
BackgroundMethodInfluentialCan language representation models think in bets?
Tang, Z., & Kejriwal, M. (2023) · Royal Society Open Science · doi:10.1098/rsos.221585 · Cited by 23
“Compared with the RoBERTa-large model, DeBERTa improves on SQuAD v. 2.0 by 2.3%, on multi-genre natural language inference (MNLI) [49] by 0.9%, and on large-scale reading comprehension dataset from examinations (RACE) [50] by 3.6%.”
BackgroundInfluentialTSQA: Tabular Scenario Based Question Answering
Li, X., Sun, Y., & Cheng, G. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17570 · Cited by 41
MGRC: An End-to-End Multigranularity Reading Comprehension Model for Question Answering
Liu, Q., Geng, X., Huang, H., Qin, T., Lü, J., & Jiang, D. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3107029 · Cited by 25
“Depending on the answer type, MRC tasks are classified into four categories [11], namely cloze style [12], multiple choice [13], span prediction [2], and free-form answer [14].”
MethodShall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study
Wang, B., Ping, W., Xu, P., McAfee, L., Liu, Z., Shoeybi, M., Dong, Y., Kuchaiev, O., Li, B., Xiao, C., et al. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.482 · Cited by 21
Federated Freeze BERT for text classification
Galal, O., Abdel-Gawad, A. H., & Farouk, M. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00885-x · Cited by 32
Knowledge Fusion and Semantic Knowledge Ranking for Open Domain Question AnsweringPreprint
Banerjee, P., & Baral, C. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03101 · Cited by 24
“…in recent years for natural language question answering such as (Rajpurkar et al., 2016; Joshi et al., 2017; Khashabi et al., 2018a; Richardson et al., 2013; Lai et al., 2017; Reddy et al., 2018; Choi et al., 2018; Tafjord et al., 2018; Mitra et al., 2019b; Sap et al., 2019; Bisk et al., 2019).”
BackgroundMethodContextualized Representations Using Textual Encyclopedic KnowledgePreprint
Joshi, M., Lee, K., Luan, Y., & Toutanova, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12006 · Cited by 34
“The out-of-domain set we evaluate on has examples from BioASQ (Tsatsaronis et al., 2015), DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), RACE (Lai et al., 2017), RelationExtraction (Levy et al., 2017), and TextbookQA (Kembhavi et al., 2017).”
MethodInfluentialBanerjee, P. (2019) · Workshop on Graph Based Methods for Natural Language Processing · doi:10.18653/v1/d19-5310 · Cited by 20
“…2019; Peters et al., 2018) have shown considerable linguistic understanding and perform well in tasks requiring multi-hop reasoning such as question answering (Rajpurkar et al., 2016; Mihaylov et al., 2018; Khashabi et al., 2018; Lai et al., 2017), and document ranking tasks (Callan et al., 2009).”
BackgroundTowards Knowledge Enhanced Language Model for Machine Reading Comprehension
Gong, P., Liu, J., Yang, Y., & He, H. (2020) · IEEE Access · doi:10.1109/access.2020.3044308 · Cited by 24
“With the rapid development of deep learning and increasing availability of datasets (e.g., SQuAD, RACE) [1], [2], machine reading comprehension (MRC) has achieved remarkable advancements in the last few years.”
BackgroundSelf-citationParSQuAD: Machine Translated SQuAD dataset for Persian Question Answering
Abadani, N., Mozafari, J., Fatemi, A., Nematbakhsh, M. A., & Kazemi, A. (2021) · International Conference on Web Research (ICWR) · doi:10.1109/icwr51868.2021.9443126 · Cited by 40
“In recent years, some large-scale QA datasets such as CNN/Daily Mail [2], SQuAD [3], MS MARCO [4], RACE [5] have been generated.”
BackgroundHeadlineStanceChecker: Exploiting summarization to detect headline disinformation
Sepúlveda-Torres, R., Vicente, M., Saquete, E., Lloret, E., & Palomar, M. (2021) · Journal of Web Semantics · doi:10.1016/j.websem.2021.100660 · Cited by 25
Hardalov, M., Koychev, I. K., & Nakov, P. (2019) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-056-4_053 · Cited by 30
“…extractive, e.g., span-based (Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Rajpurkar et al., 2018; Reddy et al., 2019), and non-extractive, e.g., multiple-choice questions (Richardson et al., 2013; Lai et al., 2017; Clark et al., 2018; Mihaylov et al., 2018; Sun et al., 2019a).”
BackgroundMethodResultInfluentialFrame-based Neural Network for Machine Reading Comprehension
Guo, S., Guan, Y., Tan, H., Li, R., & Li, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106889 · Cited by 20
“For example, Lai et al. [27] adopted and modified Gated Attention Reader (GA) [34] and Stanford Attention Reader (Stanford AR) [35] for both cloze-style MRC and multi-choice MRC problems.”
InfluentialQDG: A unified model for automatic question-distractor pairs generation
Shuai, P., Li, L., Liu, S., & Shen, J. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03894-6 · Cited by 29
“Since all distractors are removed from the EQG-RACE, the related distractors must also be extracted from the original RACE dataset [17].”
BackgroundCross-Lingual Transfer Learning for Question AnsweringPreprint
Lee, C., & Lee, H.-Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.06042 · Cited by 32
“…audio (Lee et al., 2018; Li et al., 2018) or video descriptions (Rohrbach et al., 2015) have been studied, but most focus on understanding text documents (Lai et al., 2017; Rajpurkar et al., 2016; Clark et al., 2018; Joshi et al., 2017; He et al., 2017; Nguyen et al., 2016; Trischler et al., 2016).”
BackgroundQuestion Answering Survey: Directions, Challenges, Datasets, Evaluation MatricesPreprint
Pandya, H. A., & Bhatt, B. (2021) · Preprints.org · doi:10.20944/preprints202112.0136.v1 · Cited by 20
Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.06904 · Cited by 20
“…Tafjord et al. 2019a quartz-no_knowledge Tafjord et al. 2019b quartz-with_knowledge Tafjord et al. 2019b race-high Lai et al. 2017 race-middle Lai et al. 2017 superglue-copa Gordon et al. 2012 swag Zellers et al. 2018 wino_grande Sakaguchi et al. 2020 qa/long-form qa eli5-askh Fan et al.…”
BackgroundCorporate Bankruptcy Prediction with Domain-Adapted BERT
Kim, A. G., & Yoon, S. (2021) · Workshop on Economics and Natural Language Processing (ECONLP) · doi:10.18653/v1/2021.econlp-1.4 · Cited by 30
Improving Pretrained Language Model Fine-Tuning With Noise Stability Regularization
Hua, H., Li, X., Dou, D., Xu, C., & Luo, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3330926 · Cited by 38
“Comprehension dataset from examinations (RACE) [81] is a machine reading comprehension dataset that is collected from English reading comprehension exams for middle-and high-school students in China.”
Qin, R., Yang, K., Abbasi, A., Dobolyi, D. G., Seyedi, S. S., Griner, E. L., Kwon, H., Cotes, R. O., Jiang, Z., Clifford, G. D., et al. (2025) · ACM Transactions on Management Information Systems · doi:10.1145/3673906 · Cited by 33
“LLaMA’s ability to outperform GPT-3 on the RACE Reading Comprehension benchmark dataset [45] shows an aptitude to handle single and multi-sentence reasoning as well as ambiguous questions, which should apply to detecting latent mental health disorders.”
Yu, J., Zha, Z.-J., & Yin, J. P. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1217 · Cited by 20
“As shown in Table 1, experiments were conducted on 3 popular data sets in 9 domains, including MCTest, RACE and MultiRC.”
MethodInfluentialSTARC: Structured Annotations for Reading Comprehension
Berzak, Y., Malmaud, J., & Lévy, R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.507 · Cited by 46
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference
Murty, S., Hashimoto, T., & Manning, C. D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.88 · Cited by 33
“In particular, we consider RACE (Lai et al., 2017) and use gold annotations provided in Demszky et al.”
MethodMinD at SemEval-2021 Task 6: Propaganda Detection using Transfer Learning and Multimodal Fusion
Tian, J., Gui, M., Li, C., Yan, M., & Xiao, W. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.150 · Cited by 21
“• RoBERTa (Liu et al., 2019) proposes an improved recipe for training BERT models and boosts the performance on GLUE(Wang et al., 2019), RACE(Lai et al., 2017) and SQuAD(Rajpurkar et al., 2016).”
MethodMa, Y., Zeng, Z., Zhu, D., Li, X., Yang, Y., Xiaoyuan, Y., Zhou, K., & Shen, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.09297 · Cited by 31
Self-citationAutomatic distractor generation in multiple-choice questions: a systematic literature review
Awalurahman, H. W., & Budi, I. D. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2441 · Cited by 27
Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trends
Lan, Y., Li, X., Du, H., Lu, X., Gao, M. Z., Qian, W., & Zhou, A. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3621181 · Cited by 62
“As a result, some MCQ datasets initially designed for QA tasks, including SciQ [130], RACE [131], and FairytaleQA [132], are also leveraged for researches on QG [133–136].”
Benchmarking Machine Reading Comprehension: A Psychological Perspective
Sugawara, S., Stenetorp, P., & Aizawa, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.137 · Cited by 31
“1 (Rajpurkar et al., 2016)), and collected from examinations (RACE (Lai et al., 2017)).”
BackgroundSampat, S. K., Kumar, A., Yang, Y., & Baral, C. R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.289 · Cited by 29
Self-citationCTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations
Li, H., Ding, W., Kang, Y., Liu, T., Wu, Z., & Liu, Z. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.323 · Cited by 37
“Previous works have explored alternative approaches to improve word embedding (Mikolov et al., 2013; Le and Mikolov, 2014; Pennington et al., 2014), which is a low-level linguistic representation.”
BackgroundSingle-dataset Experts for Multi-dataset Question Answering
Friedman, D., Dodge, B., & Chen, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.495 · Cited by 22
“, 2018), RACE (Lai et al., 2017), RelationExtraction (Levy et al.”
Federated Split BERT for Heterogeneous Text Classification
Lit, Z., Sit, S., Wang, J., & Xiao, J. Z. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892845 · Cited by 35
“Pre-trained models such as ELMo [1], GPT [2] and BERT [3] have achieved great success in many NLP tasks, such as sentiment classification [4], natural language inference [5], and question answering [6].”
MethodDIRECT: Toward Dialogue-Based Reading Comprehension Tutoring
Huang, J., Lee, Y., & Kwon, O. (2022) · IEEE Access · doi:10.1109/access.2022.3233224 · Cited by 27
“RACE-M contains 7,139 passages with 28,293 questions [12] and there are 5,708 dialogues developed with 23,982 Question -type turns in DIRECT (Table 2).”
BackgroundMethodInfluentialA Novel Multi-Stage Prompting Approach for Language Agnostic MCQ Generation Using GPT
Maity, S., Deroy, A., & Sarkar, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56063-7_18 · Cited by 52
Chen, G., Yang, J., & Gašević, D. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23204-7_6 · Cited by 23
“For example, Longest , LexRank , and Novel are the only strategies achieving Bleu1 scores greater than 5 and Meteor L scores greater than 6 on RACE.”
BackgroundMethodInfluentialComparing Test Sets with Item Response Theory
Vania, C., Htut, P. M., Huang, W., Mungra, D., Pang, R. Y., Phang, J., Liu, H., Cho, K., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.92 · Cited by 18
“passage, there are concerns that it rarely requires reasoning ability which often involves answers not mentioned in the passage, and thus not reflecting comprehension ability of humans (Lai et al., 2017; Sugawara et al., 2018).”
BackgroundSelf-citationNOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset
Zhang, Q., Wang, L., Yu, S., Wang, S., Wang, Y., Jiang, J., & Lim, E. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.350 · Cited by 28
Williams, E., Rodrigues, P., & Novak, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.02431 · Cited by 30
“At the time of release, the RoBERTa architecture achieved state-of-the-art results on publicly available benchmark datasets such as GLUE [28], RACE [13], and SQuAD [19].”
MethodDeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.06066 · Cited by 18
“With the DeepSeekMoE architecture, we further scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens.”
QASC: A Dataset for Question Answering via Sentence Composition
Khot, T., Clark, P., Guerquin, M., Jansen, P., & Sabharwal, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6319 · Cited by 20
PROST: Physical Reasoning about Objects through Space and Time
Aroca-Ouellette, S., Paik, C., Roncone, A., & Kann, K. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.404 · Cited by 52
Exploring Language Models: A Comprehensive Survey and Analysis
Singh, A. (2023) · International Conference on Research Methodologies in Knowledge Management, Artificial Intelligence and Telecommunication Engineering (RMKMATE) · doi:10.1109/rmkmate59243.2023.10369423 · Cited by 26
“This results in improved performance on essential natural language understanding benchmarks such as GLUE [12], SQuAD (Stanford Question Answering Dataset) [13], and RACE [14].”
BackgroundRemember the Facts? Investigating Answer-Aware Neural Question Generation for Text Comprehension
Steuer, T., Filighera, A., & Rensing, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-52237-7_41 · Cited by 22
“Recently, datasets have been collected, containing questions on different cognitive levels, providing more training data for NQGs in education [5,18].”
BackgroundMethodNLQuAD: A Non-Factoid Long Question Answering Data Set
Soleimani, A., Monz, C., & Worring, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.106 · Cited by 30
“There is also a range of multiple-choice QA tasks such as RACE (Lai et al., 2017), ARC (Clark et al., 2018), SWAQ (Zellers et al., 2018), and COS-MOS QA (Huang et al., 2019) that are clustered together with the short-context QA data sets.”
Review Conversational Reading ComprehensionPreprint
Xu, H., Liu, B., Shu, L., & Yu, P. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00821 · Cited by 24
XTQA: Span-Level Explanations for Textbook Question Answering
Ma, J., Chai, Q., Liu, J., Yin, Q., Wang, P., & Zheng, Q. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3294991 · Cited by 21
“They pretrained the textual ISAAQ on RACE [13], ARC-Easy, ARC-Challenge [14], and OpenBookQA [15] datasets and fine-tuned it on CK12-QA.”
MethodAudio-Oriented Multimodal Machine Comprehension via Dynamic Inter- and Intra-modality Attention
Huang, Z., Liu, F., Wu, X., Ge, S., Wang, H., Fan, W., & Zou, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17548 · Cited by 26
“There are four types of Machine Comprehension (MC) (Chen 2018), including cloze-style (Hermann et al. 2015), multi-choices (Lai et al. 2017), span extraction (Rajpurkar et al. 2016; Rajpurkar, Jia, and Liang 2018), and generative format (Kocisk´y et al. 2018).”
BackgroundInfluentialWang, J., Ji, Y., Sun, J., Yang, Y., & Sakai, T. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.196 · Cited by 16
Self-citationWang, Y., Hu, Z., Liang, J., & Li, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.785 · Cited by 32
“In some simple, fact-based QA tasks, such as SQuAD (Rajpurkar et al., 2016, 2018) and RACE (Lai et al., 2017), machine has surpassed humans in terms of answer accuracy.”
MethodRQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question
Mohammadshahi, A., Scialom, T., Yazdani, M., Yanki, P., Fan, A., Henderson, J., & Saeidi, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.428 · Cited by 53
Natural language processing: An overview of models, transformers and applied practices
Canchila, S., Meneses-Eraso, C., Casanoves-Boix, J., Cortés-Pellicer, P., & Sirvent, F. C. (2024) · Computer Science and Information Systems · doi:10.2298/csis230217031c · Cited by 31
Model Breadcrumbs: Scaling Multi-task Model Merging with Sparse Masks
Davari, M., & Belilovsky, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73226-3_16 · Cited by 169
Li, X., Zhang, Z., Zhu, W., Li, Z., Ni, Y., Gao, P., Yan, J., & Xie, G. (2019) · doi:10.18653/v1/d19-6011 · Cited by 26
“As a result, Table 3 illustrates that pre-training on RACE plays a significant role in the system.”
MethodInfluentialWang, Y., Liu, F., Verspoor, K. M., & Baldwin, T. (2020) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2020.bionlp-1.11 · Cited by 21
“, 2018) with >390k instances, and RACE (Lai et al., 2017) with >97k instances.”
BackgroundMoCA: Incorporating domain pretraining and cross attention for textbook question answering
Xu, F., Lin, Q., Liu, J., Zhang, L., Zhao, T., Chai, Q., Pan, Y., Huang, Y., & Wang, Q. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109588 · Cited by 18
Metadata
- Authors
- Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 785-794
- OpenAlex
- W2606964149
- Cited by
- 1,044
- References
- 19
- Citations by year
- 2026: 37 · 2025: 75 · 2024: 86 · 2023: 145 · 2022: 136 · 2021: 216 · 2020: 169 · 2019: 138 · 2018: 38 · 2017: 3
- FWCI
- 42.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750, FA8750-12-2-0342)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, machine reading comprehension, benchmark dataset, English examinations, Chinese students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.