RACE: Large-scale ReAding Comprehension Dataset From Examinations
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task.Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning.In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%).We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [112]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, to our knowledge, most of the existing datasets are not of direct relevance to teaching and learning, except for RACE (Lai, Xie, Liu, Yang, & Hovy, 2017) and LearningQ (Chen, Yang, Hauff, & Houben, 2018).”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Lai, Xie, Liu, Yang, & Hovy, 2017
The cited source says what the claim attributes to it.
In the citing paper
“On RACE-h and RACE-m (Lai et al., 2017), Chinchilla greatly outperforms Gopher , improving accuracy by more than 10% in both cases-see Table 7.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Lai et al., 2017
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We consider only the case of human-readable prompts to large models, which contrasts with methods such as prompt-tuning [16,20], which learns a continuous prompt embedding, and auto-prompting [33], which attempts to search for prompts from scratch, both of which require a training step.”
From Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We choose word embedding size d = 100 and use the 100-dimensional Glove word embedding (Pennington et al., 2014) as embedding initialization.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Instead of evaluating the accuracy, researchers need to use F1 score, BLEU (Papineni et al., 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the questions come without candidate spans.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 8×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
MethodResultInfluentialTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 3×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 1×
“, 2016) and recently proposed TriviaQA (Joshi et al., 2017).”
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578 · cited 2×
“, 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the”
BackgroundSelf-citationNewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 11×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodResultInfluentialMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610 · cited 4×
“Although efforts have been made with a similar motivation, including the MCTest dataset created by (Richardson et al., 2013) (containing 500 passages and 2000 questions) and several others (Peñas et al.”
BackgroundMethodInfluentialA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491 · cited 11×
“However, both require limited reasoning ability (Chen et al., 2016).”
BackgroundMethodInfluential
Cited by
Items 701–800 of 1,038
Selecting Paragraphs to Answer Questions for Multi-passage Machine Reading Comprehension
Lin, D., Tang, J., Pang, K., Li, S., & Wang, T. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-31624-2_10 · Cited by 3
IIT-KGP at COIN 2019: Using pre-trained Language Models for modeling Machine Comprehension
Sharma, P., & Roychowdhury, S. (2019) · doi:10.18653/v1/d19-6009 · Cited by 3
“We first fine-tuned BERT on the RACE dataset using a maximum sequence length of 350 for 2 epochs.”
BackgroundInfluentialCOVID-19’s Impact on the Telecommunications Companies
Almuqren, L., & Cristea, A. I. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-72654-6_31 · Cited by 4
Wang, Y., Lei, C., & Miao, D. (2021) · IEEE Access · doi:10.1109/access.2021.3079165 · Cited by 3
“In recent years, various machine reading comprehension datasets have been released one after another, such as SQuAD, MS MARCO [16], RACE [17], CNN & Daily Mail [18], and DuReader.”
MethodZero-Shot Clinical Questionnaire Filling From Human-Machine Interactions
Toudeshki, F. G., Jolivet, P., Durand-Salmon, A., & Liednikova, A. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.5 · Cited by 4
“Mishra et al. (2020) described reasoning categories present in RACE and showed that if given passage is a dialogue, NLI model performs better than QA model.”
BackgroundLanguage-based audio retrieval with Converging Tied Layers and Contrastive Loss
Koh, A., & Siong, C. E. (2022) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipaasc55919.2022.9979840 · Cited by 5
Long-Tailed Question Answering in an Open World
Dai, Y. Q., Lang, H., Zheng, Y., Huang, F., & Li, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.351 · Cited by 12
Shan, J., Nishihara, Y., Yamanishi, R., & Maeda, A. (2019) · Conference on Technologies and Applications of Artificial Intelligence (TAAI) · doi:10.1109/taai48200.2019.8959903 · Cited by 3
“In this paper, we use a dataset of past English reading comprehension tests called RACE [3], which has pairs of an article and questions.”
MethodBerre, G. L., & Langlais, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-47358-7_37 · Cited by 3
“Current state-of-the-art models on OpenBookQA reach 78% accuracy using Roberta [11] and an additional pretraining on RACE dataset [4] while BERT Large with no additionnal pretraining achieves 60% accuracy.”
BackgroundMethodJoint Semantic Analysis with Document-Level Cross-Task Coherence Rewards
Aralikatte, R., Abdou, M., Lent, H., Hershcovich, D., & Søgaard, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17484 · Cited by 3
Learning from Limited Labels for Long Legal Dialogue
Hong, J., Chong, D., & Manning, C. D. (2021) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2021.nllp-1.20 · Cited by 12
“We tried using a multiple choice reading comprehension (MRC) head (Richardson et al., 2013; Lai et al., 2017; Chen, 2018), which proved to be an an elegant way of grounding the model, with similarities to contrastive learning, and able to generate dynamic classification options, e.g. unlike year…”
MethodLooking Beyond Sentence-Level Natural Language Inference for Downstream TasksPreprint
Mishra, A., Patel, D., Vijayakumar, A., Li, X., Kapanipathi, P., & Talamadupula, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09099 · Cited by 3
Optimizing Deeper Transformers on Small DatasetsPreprint
Xu, P., Kumar, D., Yang, W., Zi, W., Tang, K., Huang, C., Cheung, J. C. K., Prince, S. J. D., & Cao, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15355 · Cited by 3
“, 2020b) without using the additional RACE dataset (Lai et al., 2017).”
MethodNeural Transfer Learning with Transformers for Social Science Text Analysis.Preprint
Wankmüller, S. (2021) · arXiv (Cornell University) · Cited by 3
Encoding Explanatory Knowledge for Zero-shot Science Question AnsweringPreprint
Zhou, Z., Valentino, M., Landers, D., & Freitas, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.05737 · Cited by 4
Zero-Shot Open-Book Question AnsweringPreprint
Gholami, S., & Noori, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.11520 · Cited by 3
“…of datasets in the literature for natural language QA (Rajpurkar et al., 2016; Joshi et al., 2017; Khashabi et al., 2018; Richardson et al., 2013; Lai et al., 2017; Reddy et al., 2019; Choi et al., 2018; Tafjord et al., 2019; Mitra et al., 2019), as well several solutions to tackle these…”
BackgroundAn AI-based Solution for Enhancing Delivery of Digital Learning for Future TeachersPreprint
Kang, Y., Forkan, A. R. M., Jayaraman, P. P., Wieland, N., Kollias, E., Du, H., Thomson, S., & Li, Y.-F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.01229 · Cited by 7
“This model was trained on a large-scale question-answer dataset, called RACE [35], that contains more than 27k text articles with more than 97l questions from English examinations of middle-school and high-school Chinese students within the 12-18 age range.”
MethodProceedings of the First Workshop on Commonsense Inference in Natural Language Processing
(2019) · doi:10.18653/v1/d19-60 · Cited by 5
“We fine-tune on additional data, but we select only data related to attributes that BERT is deficient in. 3.3.1 Data Selection In our experiments, we use RACE (Lai et al., 2017) as our supplementary dataset.”
InfluentialWhen AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Akhtar, M., Reuel, A., Soni, P., Ahuja, S., Ammanamanchi, P. S., Rawal, R., Zouhar, V., Yadav, S., Whitehouse, C., Ki, D., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.16763 · Cited by 3
More with Less: ZeroQA and Relevant Subset Selection for AI2 Reasoning Challenge
Pătras, C. C.-B., Pîrtoacă, G.-S., Rebedea, T. E., & Ruşeţi, Ş. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.09.334 · Cited by 2
Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models
Pirozelli, P., José, M. M., Filho, P. D. T. P. D. C., Brandão, A. A. F., & Cozman, F. G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71167-1_2 · Cited by 8
The Future of Assessment: Automating Multiple Choice Question Generation
Rani, V. S., Kalavala, R., Chittireddy, A. R., Babu, D. A. R., Suram, P., & Velidandi, S. (2024) · International Conference on IoT Based Control Networks and Intelligent Systems (ICICNIS) · doi:10.1109/icicnis64247.2024.10823166 · Cited by 2
“For distractor generation, we employed the RACE [1] dataset, which provides diverse passages and associated multiple-choice questions, enabling the creation of plausible distractors.”
None of the above: comparing scenarios for answerability detection in question answering systems
Reyes-Montesinos, J., Rodrigo, Á., & Peñas, A. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06765-y · Cited by 6
“Datasets such as CNN/Daily Mail [ 9], SQuAD [3], RACE [4] or QuAIL [7] are constructed with the premise that a correct answer for every question exist within the given context.”
YMCQ: Reasoning-Enhanced MCQ Generation
Dutulescu, A., Ruşeţi, Ş., Iorga, D., Dascălu, M., & McNamara, D. S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98465-5_39 · Cited by 4
Ji, L., Yao, L., & Xu, W. (2025) · Scientific Reports · doi:10.1038/s41598-025-24031-6 · Cited by 2
LLMs in Staging: An Orchestrated LLM Workflow for Structured Augmentation with Fact Scoring
Trimigno, G., Lombardo, G., Tomaiuolo, M., Cagnoni, S., & Poggi, A. (2025) · Future Internet · doi:10.3390/fi17120535 · Cited by 4
Finkelstein, I., & Soffer-Vital, S. (2025) · Educational Research and Evaluation · doi:10.1080/13803611.2025.2596373 · Cited by 6
“These tools, which are based on deep neural networks, are able to generate assessment tasks such as multiple-choice questions and open-answer questions and are founded upon large-scale datasets such as RACE (Lai et al., 2017).”
FiNE: Fine-Grained Neuron-Level Model Editing for Reliable and Safe LLMs
Yang, X., Pan, H., Wang, X., Cao, Y., Li, J., & Wang, M. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3697976 · Cited by 4
TibetanQA: a dataset of Tibetan for Machine reading comprehension
Sun, Y., Dan, Z., Liu, S., & Xiaobing, Z. H. A. O. (2022) · China Scientific Data · doi:10.11922/11-6035.csd.2021.0099.zh · Cited by 2
Help Transformer Improve Performance in Automatic Mathematics Word Problem-Solving
Liu, D., Wang, G., & Yang, J. (2022) · IEEE Access · doi:10.1109/access.2022.3220777 · Cited by 2
“In recent years, lots of machine reading comprehension (MRC) models [1], [2], [3] have been proposed, and achieved remarkable results in various public benchmarks such as SQuAD [4] and RACE [5].”
BackgroundSuryono, W., & Harianto, B. B. (2023) · Technium Social Sciences Journal · doi:10.47577/tssj.v41i1.8512 · Cited by 2
Zhou, Q., Yang, Z., Li, P., & Liu, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.738 · Cited by 11
KoRC: Knowledge Oriented Reading Comprehension Benchmark for Deep Text Understanding
Yao, Z., Liu, Y., Lv, X., Cao, S., Yu, J., Li, J., & Hou, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.743 · Cited by 13
The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models
Tran, S. Q., Nguyen-Thuan, P., Le, U., & Kretchmar, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.113 · Cited by 8
Multi-Grained Evidence Inference for Multi-Choice Reading Comprehension
Zhao, Y., Zhao, H., & Duan, S. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3313885 · Cited by 3
“…which searches for the most proper snippet from the passage as answer [4], [5]; 2) generative task, which needs model to summarize the passage and generate answer [6]; 3) multi-choice task, the focus of this work, which provides several options and aims to select the most suitable one [7], [8].”
BackgroundMethodMultisource Soft Labeling and Hard Negative Sampling for Retrieval Distractor Ranking
Wang, J., Rong, W., Bai, J., Sun, Z., Ouyang, Y., & Xiong, Z. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3325549 · Cited by 3
“1) RACE (See [52]): These two datasets are preprocessed as follows.”
BackgroundCRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model
Zhang, K., Ding, N., Qi, B., Zhu, X., Long, X., & Zhou, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.597 · Cited by 21
Location metadata extraction from Geosocial data of Road Accident using Deep Learning models
Mukherjee, T., Sinhahajari, S., Mukherjee, D., Mallick, H., Middya, A. I., & Roy, S. (2024) · Evolving Systems · doi:10.1007/s12530-024-09637-x · Cited by 2
“Their best model provides cutting-edge outcomes on SQuAD (Sido 2018), GLUE (Wang et al. 2018), and RACE (Lai et al. 2017).”
Exploring State-of-the-Art LLMs from BERT to XLNet: A study over Question Answering
Parra-Navarro, L. M., Batista, E. C. S., & Pacheco, M. A. C. (2024) · IEEE Latin American Conference on Computational Intelligence (LA-CCI) · doi:10.1109/la-cci62337.2024.10814828 · Cited by 5
“In our study, we conducted a comprehensive review of various datasets relevant to open-domain textual question answering (QA), including RACE[29], TriviaQA[30], and XQuAD[31].”
Incorporating Molecular Knowledge in Large Language Models via Multimodal Modeling
Yang, Z., Lv, K., Shu, J. H., Li, Z., & Xiao, P. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3506158 · Cited by 3
Dong, X., Li, Z., Su, H., Xue, J., & Dang, X. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11237-3 · Cited by 4
“We also calculated the average lengths of the articles, questions, correct answers, and distractors in the RACE and RACE++ datasets.”
InfluentialA resilient generative model in few-shot question answering
Zou, A., Chen, Y., Huang, R., & Qin, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113629 · Cited by 2
Mamba-MSCCA-Net: Efficient change detection for remote sensing images
Song, Z., Wu, Y., & Huang, S. (2025) · Displays · doi:10.1016/j.displa.2025.103097 · Cited by 2
Sheng, Y., & Lan, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-36718-3_29 · Cited by 2
A New Fusion Method on Machine Reading Comprehension
Tang, W., Jiang, H., & Xu, K. (2020) · International Conference on Big Data Engineering (BDE) · doi:10.1145/3404512.3404529 · Cited by 2
“MCTest [2] and RACE [3] are more famous in this kind of data set.”
BackgroundOn the Transferability of Minimal Prediction Preserving Inputs in Question Answering
Longpre, S., Lu, Y., & DuBois, C. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.101 · Cited by 2
Bai, Z., Liu, J., Wang, M., Yuan, C., & Wang, X. (2022) · Applied Sciences · doi:10.3390/app12063072 · Cited by 3
TAGNet: a tiny answer-guided network for conversational question generation
Wang, Z., Zhu, H., Liu, M. Z., & Qin, B. (2022) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01737-x · Cited by 4
“Our model is trained on the CoQA dataset, which includes seven different domains: children’s stories [55], literature, middle school English exams [56], news articles [57], Wikipedia articles, Reddit articles [58], and science articles [59].”
MethodSMART: A Stratified Machine Reading Test
Yao, J., Feng, M., Feng, H., Wang, Z., Zhang, Y., & Xue, N. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32233-5_6 · Cited by 2
Option Attentive Capsule Network for Multi-choice Reading Comprehension
Miao, H., Liu, R., & Gao, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-36718-3_26 · Cited by 2
Context Modeling with Evidence Filter for Multiple Choice Question Answering
Yu, S., Zhang, H., Jing, W., & Jiang, J. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747889 · Cited by 2
Introduction to machine reading comprehension
Zhu, C. (2021) · Elsevier eBooks · doi:10.1016/b978-0-323-90118-5.00001-1 · Cited by 6
Zhang, T., Wang, C., Qiu, M., Yang, B., Cai, Z., He, X., & Huang, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.197 · Cited by 4
A Chinese Machine Reading Comprehension Dataset Automatic Generated Based on Knowledge Graph
Zhao, H., Yuan, S., Leng, J., Pan, X., Zhao, X., Ma, Q., & Liang, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-84186-7_18 · Cited by 2
“RACE (Lai et al., 2017) collects around 870K multiple-choice questions that appeared in Chinese middle school students’ English reading comprehension examination.”
BackgroundOutput Layer Go First: Better Fine-tuning by Bridging the Gap with Pre-training
Xiao, T., Dong, Y., & Dong, B. (2021) · doi:10.1109/icnlp52887.2021.00030 · Cited by 2
NQuAD: 70,000+ Questions for Machine Comprehension of the Numerals in Text
Chen, C.-C., Huang, H.-H., & Chen, H.-H. (2021) · International Conference on Information and Knowledge Management · doi:10.1145/3459637.3482155 · Cited by 15
“Based on the above definition, Table 1 shows the statistics of three Chinese MRC datasets (CMRC-2017 [9], DRCD [24], and CMRC-2018 [8]) and four Englishmultiple-choiceMRC datasets (MCTest [22], RACE [15], MCScript [19], and ARC [7]).”
BackgroundUnsupervised Multiple Choices Question Answering: Start Learning from Basic Knowledge
Liu, C.-L., & Lee, H.-Y. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.12 · Cited by 5
“…numerous question answering datasets (Weston et al., 2015; Rajpurkar et al., 2016, 2018; Yang et al., 2018; Trischler et al., 2017; Choi et al., 2018; Joshi et al., 2017; Kwiatkowski et al., 2019; Reddy et al., 2019; Richardson, 2013; Lai et al., 2017a; Khashabi et al., 2018) have been proposed.”
BackgroundMethodInfluentialWhat If Sentence-hood is Hard to Define: A Case Study in Chinese Reading Comprehension
Wang, J., Zhao, H., Zhao, Y., & Shen, L. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.202 · Cited by 4
“, 2016), multi-choice (Lai et al., 2017; Sun et al., 2019) and span extraction (Rajpurkar et al.”
BackgroundVarghese, N., & Punithavalli, M. (2022) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-16-9113-3_1 · Cited by 2
SSIN: Sentence Semantic Interaction Network for Multi-Choice Reading Comprehension
Xu, X., Tohti, T., & Hamdulla, A. (2022) · IEEE Access · doi:10.1109/access.2022.3217479 · Cited by 2
Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer
Lv, X.-T., Ding, N., Qin, Y., Liu, Z.-Y., & Sun, M. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2023.acl-short.24 · Cited by 12
Domain Incremental Lifelong Learning in an Open World
Dai, Y. Q., Lang, H., Zheng, Y., Yu, B., Huang, F., & Li, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.361 · Cited by 10
Operation-Augmented Numerical Reasoning for Question Answering
Zhou, Y., Bao, J., Wu, Y., He, X., & Zhao, T. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3316448 · Cited by 2
Natural Response Generation for Chinese Reading Comprehension
Chen, N., Li, H., Bao, Y., Wang, B., & Li, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.739 · Cited by 3
CK12: A Rounded K12 Knowledge Graph Based Benchmark for Chinese Holistic Cognition Evaluation
You, W., Wang, P., Li, C., Ji, Z., & Bai, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29914 · Cited by 8
“Some benchmarks like GLUE (Wang et al. 2018), Super-GLUE (Wang et al. 2019), MMLU (Hendrycks et al. 2021a), RACE (Lai et al. 2017), GSM8K (Cobbe et al. 2021), LAM-BADA (Paperno et al. 2016), TriviaQA (Joshi et al. 2017), Truthfulqa (Lin, Hilton, and Evans 2021), HELM (Liang et al. 2022) and…”
MethodHigh-quality distractor generation framework for English reading comprehension
Guo, Z., Wang, H., & Guo, G. (2023) · International Academic Exchange Conference on Science and Technology Innovation (IAECST) · doi:10.1109/iaecst60924.2023.10502585 · Cited by 2
“The accuracy of the generated distractors was verified on the RACE dataset [6] and the feasibility of the GDF-HD framework was verified using both automated and manual evaluation.”
MethodInfluentialLabel Distribution Augmented Maximum Likelihood Estimation for Reading Comprehension
Su, L., Guo, J., Fan, Y., Lan, Y., & Cheng, X. (2020) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3336191.3371835 · Cited by 2
“Reading comprehension (RC), aiming to understand natural texts to answer questions, is a challenging task in natural language processing [10, 13, 16, 27, 28] and has became an important component in QA system [1].”
BackgroundExamination-Style Reading Comprehension with Neural augmented Retrieval
Zhang, Y., Zhao, H., & Zhang, Z. (2019) · International Conference on Asian Language Processing · doi:10.1109/ialp48816.2019.9037657 · Cited by 2
“In Chinese dataset the relevant evidence is not tagged, the effectiveness of word importance can only be reflected by the final accuracy of choosing the right answer, so we use an English multiple choice question-answering dataset [8].”
MethodA Sentence-Based Circular Reasoning Model in Multi-Hop Reading Comprehension
Huo, L., & Zhao, X. H. (2020) · IEEE Access · doi:10.1109/access.2020.3025957 · Cited by 3
“2) Multi-choice tasks, such as QAngarooWikiHop andMedHop [15] and RACE [16].”
MethodA Neural Framework for English-Hindi Cross-Lingual Natural Language Inference
Saikh, T., De, A., Bandyopadhyay, D., Gain, B., & Ekbal, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-63830-6_55 · Cited by 2
A guide to the dataset explosion in QA, NLI, and commonsense reasoning
Rogers, A., & Rumshisky, A. (2020) · doi:10.18653/v1/2020.coling-tutorials.5 · Cited by 4
“Multiple-choice questions are harder to generate, as they need good confounds, and often come from curated test collections (Lai et al., 2017).”
BackgroundSelf-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering Data
Yu, D., Sun, K., Yu, D., & Cardie, C. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.6 · Cited by 7
Extraction of Question-related Sentences for Reading Comprehension Tests via Attention Mechanism
Shan, J., Nishihara, Y., Maeda, A., & Yamanishi, R. (2020) · Conference on Technologies and Applications of Artificial Intelligence (TAAI) · doi:10.1109/taai51410.2020.00013 · Cited by 2
“Not like the datasets for QA systems, questions in the RACE dataset are posed for examining the language learners’ reading comprehension skills.”
MethodInfluentialControlling Text Edition by Changing Answers of Specific Questions
Sha, L., Hohenecker, P., & Lukasiewicz, T. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.110 · Cited by 2
Repurposing Entailment for Multi-Hop Question Answering TasksPreprint
Trivedi, H. K., Kwon, H. Y., Khot, T., Sabharwal, A., & Balasubramanian, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09380 · Cited by 8
“Recall that the Reading Strategies results are reported with an additional fine-tuning on another larger QA dataset, RACE (Lai et al., 2017) aside from the target QA datasets we use here.”
BackgroundMethodInfluentialCareful Selection of Knowledge to solve Open Book Question AnsweringPreprint
Banerjee, P., Pal, K. K., Mitra, A., & Baral, C. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.10738 · Cited by 17
“…been proposed for natural language question answering (Rajpurkar et al., 2016; Joshi et al., 2017; Khashabi et al., 2018; Richardson et al., 2013; Lai et al., 2017; Reddy et al., 2018; Choi et al., 2018; Tafjord et al., 2018; Mitra et al., 2019) and many attempts have been made to solve these…”
Answering questions by learning to rank -- Learning to rank by answering questionsPreprint
Pîrtoacă, G.-S., Rebedea, T. E., & Ruşeţi, Ş. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00596 · Cited by 7
“Training the discriminator is performed on the RACE (Reading Comprehension Dataset) which is also collected from English Examinations (Lai et al., 2017).”
MethodNumNet: Machine Reading Comprehension with Numerical ReasoningPreprint
Ran, Q., Lin, Y., Li, P., Zhou, J., & Liu, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06701 · Cited by 18
“In recent years, researchers have published a large number of annotated MRC datasets such as CNN/Daily Mail (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016), RACE (Lai et al., 2017), Trivi-aQA (Joshi et al., 2017) and so on.”
ManyModalQA: Modality Disambiguation and QA over Diverse InputsPreprint
Hannan, D., Jain, A., & Bansal, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08034 · Cited by 10
GenNet : Reading Comprehension with Multiple Choice Questions using Generation and Selection modelPreprint
Ingale, V., & Singh, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04360 · Cited by 2
“We evaluate our model on RACE dataset(Lai et al., 2017) Race is a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions.”
MethodR3: A Reading Comprehension Benchmark Requiring Reasoning ProcessesPreprint
Wang, R., Tao, K., Song, D., Zhang, Z., Ma, X., Su, X. H., & Dai, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.01251 · Cited by 4
“Besides, recent works like HotpotQA (Yang et al., 2018), RACE (Lai et al., 2017), WikiHop (Welbl et al., 2018), etc., require the ability of multi-step reasoning.”
MethodPrerequisites for Explainable Machine Reading Comprehension: A Position Paper.Preprint
Sugawara, S., Stenetorp, P., & Aizawa, A. (2020) · arXiv (Cornell University) · Cited by 3
Adversarial Training for Commonsense InferencePreprint
Pereira, L. K., Liu, X., Cheng, F., Asahara, M., & Kobayashi, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.08156 · Cited by 2
“PSH-SJTU (Li and Xie, 2019) is based on multi-stage fine-tuning XLNET (Yang et al., 2019) on RACE (Lai et al., 2017), SWAG (Zellers et al., 2018) and MC-Script2.”
MethodKnowledgeable Dialogue Reading Comprehension on Key TurnsPreprint
Li, J., Zhang, Z., & Zhao, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13988 · Cited by 4
“Multi-choice Machine Reading Comprehension (MRC) has long been a heated topic, and various datasets and models have been proposed in recent years (Lai et al., 2017; Os-termann et al., 2018; Khashabi et al., 2018; Mostafazadeh et al., 2016; Richardson et al., 2013; Sun et al., 2019).”
BackgroundQuestion Directed Graph Attention Network for Numerical Reasoning over TextPreprint
Chen, K., Xu, W., Cheng, X., Xiaochuan, Z., Zhang, Y., Song, L., Wang, T., Qi, Y., & Chu, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.07448 · Cited by 2
“…from recent improvements of pre-trained deep language models like BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), a considerable progress of MRC have been made on the annotated datasets such as SQuAD (Rajpurkar et al., 2016), RACE (Lai et al., 2017), TriviaQA (Joshi et al., 2017) and so on.”
BackgroundMethodInfluentialComposing Answer from Multi-spans for Reading ComprehensionPreprint
Zhang, Z., Zhang, Y., Zhao, H., Zhou, X., & Zhou, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06141 · Cited by 2
“The multi-choice style MRC like RACE (Lai et al. 2017) requires models to choose a correct answer from a set of answer candidate options.”
MethodUnsupervised Explanation Generation for Machine Reading ComprehensionPreprint
Cui, Y., Liu, T., Wang, S., & Hu, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06737 · Cited by 3
“Compared to the best scores in RACE, the accuracy of the student output is 67.8/66.8, which is lower than the teacher system, and thus we stick to use the teacher system to make the final prediction.3 As the main body system, i.e., ALBERT model, is shared between the teacher and student, the learning of the student wi…”
InfluentialHare, A., Chen, Y., Liu, Y., Liu, Z., & Brinton, C. G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01941 · Cited by 7
“Finally, the RACE Dataset [17], obtained from the project website,3 is a collection of English text used to teach Chinese students in middle and high schools.”
MethodUsing Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension ModelsPreprint
Lin, J., Zou, J., & Ding, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11136 · Cited by 2
“sic RACE dataset (Lai et al., 2017), which requires MRC models to answer multiple-choice reading comprehension questions based on a passage.”
MethodInfluentialControllable Open-ended Question Generation with A New Question Type OntologyPreprint
Cao, S., & Wang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00152 · Cited by 2
Zou, J., & Ding, N. (2021) · Research Square · doi:10.21203/rs.3.rs-813994/v1 · Cited by 2
Zero-Shot Dialogue State Tracking via Cross-Task TransferPreprint
Lin, Z., Liu, B., Madotto, A., Moon, S., Crook, P., Zhou, Z., Wang, Z., Yu, Z., Cho, E., Subba, R., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04655 · Cited by 2
“, 2019), and that of multiple-choice datasets are from RACE (Lai et al., 2017) and DREAM (Sun et al.”
BackgroundInfluentialListReader: Extracting List-form Answers for Opinion QuestionsPreprint
Cui, P., Hu, D., & Hu, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.11692 · Cited by 3
Qu, F., Jia, X., & Wu, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05179 · Cited by 6
“To alleviate the above limitations, we propose to perform QAG on RACE (Lai et al., 2017) .”
MethodMiMo: Unlocking the Reasoning Potential of Language Model - From Pretraining to Posttraining
Xiaomi, L.-C., :, Xia, B., Shen, B., Cici, Zhu, D., Zhang, D., Wang, G., Zhang, H., Liu, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.07608 · Cited by 1
“Reading comprehension : DROP (Dua et al., 2019), RACE (Lai et al., 2017).”
Self-citationMA-MRC: A Multi-answer Machine Reading Comprehension Dataset
Yue, Z., Liu, J., Zhang, C., Wang, C., Jiang, H., Zhang, Y., Tian, X., Cen, Z., Xiao, Y., & Ruan, T. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3592015 · Cited by 3
Weak Positive Sampling and Soft Smooth Labeling for Distractor Generation Data Augmentation
Wang, J., Bai, J., Rong, W., Ouyang, Y., & Xiong, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-4752-2_62 · Cited by 1
Nguyen, N. L.-T., Phan, K. T.-K., Huynh, T. V., & Nguyen, K. V. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19671-1 · Cited by 1
“Therefore, many works investigating NLI tasks at the passage level have received much attention [10, 12, 13].”
Wang, R., Jiang, Y., Tao, Y., Li, M., Wang, X., & Ge, S. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9440-9_8 · Cited by 2
Metadata
- Authors
- Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 785-794
- OpenAlex
- W2606964149
- Cited by
- 1,044
- References
- 19
- Citations by year
- 2026: 37 · 2025: 75 · 2024: 86 · 2023: 145 · 2022: 136 · 2021: 216 · 2020: 169 · 2019: 138 · 2018: 38 · 2017: 3
- FWCI
- 42.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750, FA8750-12-2-0342)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, machine reading comprehension, benchmark dataset, English examinations, Chinese students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.