RACE: Large-scale ReAding Comprehension Dataset From Examinations
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task.Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning.In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%).We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [112]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, to our knowledge, most of the existing datasets are not of direct relevance to teaching and learning, except for RACE (Lai, Xie, Liu, Yang, & Hovy, 2017) and LearningQ (Chen, Yang, Hauff, & Houben, 2018).”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Lai, Xie, Liu, Yang, & Hovy, 2017
The cited source says what the claim attributes to it.
In the citing paper
“On RACE-h and RACE-m (Lai et al., 2017), Chinchilla greatly outperforms Gopher , improving accuracy by more than 10% in both cases-see Table 7.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Lai et al., 2017
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We consider only the case of human-readable prompts to large models, which contrasts with methods such as prompt-tuning [16,20], which learns a continuous prompt embedding, and auto-prompting [33], which attempts to search for prompts from scratch, both of which require a training step.”
From Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We choose word embedding size d = 100 and use the 100-dimensional Glove word embedding (Pennington et al., 2014) as embedding initialization.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Instead of evaluating the accuracy, researchers need to use F1 score, BLEU (Papineni et al., 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the questions come without candidate spans.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 8×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
MethodResultInfluentialTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 3×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 1×
“, 2016) and recently proposed TriviaQA (Joshi et al., 2017).”
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578 · cited 2×
“, 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the”
BackgroundSelf-citationNewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 11×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodResultInfluentialMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610 · cited 4×
“Although efforts have been made with a similar motivation, including the MCTest dataset created by (Richardson et al., 2013) (containing 500 passages and 2000 questions) and several others (Peñas et al.”
BackgroundMethodInfluentialA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491 · cited 11×
“However, both require limited reasoning ability (Chen et al., 2016).”
BackgroundMethodInfluential
Cited by
Items 101–200 of 1,038
Conversational question answering: a survey
Zaib, M., Zhang, W. E., Sheng, Q. Z., Mahmood, A., & Zhang, Y. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01744-y · Cited by 192
“The examples of such datasets include RACE [37], MCTest [78], and MCSript [59], – Descriptive answer datasets allow answers to be in any free-form text.”
BackgroundAutomatic question generation and answer assessment: a survey
Das, B., Majumder, M., Phadikar, S., & Sekh, A. A. (2021) · Research and Practice in Technology Enhanced Learning · doi:10.1186/s41039-021-00151-1 · Cited by 123
Biomedical Question Answering: A Survey of Approaches and Challenges
Jin, Q., Yuan, Z., Xiong, G., Yu, Q., Ying, H., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2023) · ACM Computing Surveys · doi:10.1145/3490238 · Cited by 118
“Many general domain QA datasets that are extracted from examinations have been introduced [89, 95, 145, 176].”
BackgroundLarge-scale Cloze Test Dataset Created by Teachers
Xie, Q., Lai, G., Dai, Z., & Hovy, E. H. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1257 · Cited by 167
“…al., 2016) have reached human performance on the SQuAD dataset (Rajpurkar et al., 2016), their performance is still not comparable to human performance on datasets that focus more on reasoning where the evidence cannot be simply found by a matching behavior (Lai et al., 2017; Xu et al., 2017).”
BackgroundInfluentialSelf-citationEnd-to-end generation of multiple-choice questions using text-to-text transfer transformer models
Rodriguez-Torrealba, R., Garcia-Lopez, E., & Garcia-Cabot, A. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118258 · Cited by 118
Datasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 113
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Goyal, S., Choudhury, A. R., Raje, S. M., Chakaravarthy, V. T., Sabharwal, Y., & Verma, A. (2020) · arXiv · doi:10.48550/arxiv.2001.08950 · Cited by 117
Transforming Question Answering Datasets Into Natural Language Inference Datasets
Demszky, D., Guu, K., & Liang, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.02922 · Cited by 113
“The questions vary greatly in terms of their type and difficulty: questions in QAMR hinge on selecting the right arguments within a single sentence, while questions in RACE, written for middle-and high-schoolers, require holistic reasoning about the text (e.g.”
BackgroundInfluentialSemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge
Ostermann, S., Roth, M., Modi, A., Thater, S., & Pinkal, M. (2018) · Proceedings of The 12th International Workshop on Semantic Evaluation · doi:10.18653/v1/s18-1119 · Cited by 114
“Another recently published multiple choice dataset is RACE (Lai et al., 2017), which contains 100,000 questions on reading examination data.”
BackgroundMethodInfluentialFeTaQA: Free-form Table Question Answering
Nan, L., Hsieh, C., Mao, Z., Lin, X. V., Verma, N., Zhang, R., Kryściński, W., Schoelkopf, H., Kong, R., Tang, X., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00446 · Cited by 111
“For QA over text, a sequence modeling approach is usually adopted to encode the query and the context, and answers are either categorical (Lai et al., 2017), extractive (Rajpurkar et al., 2016; Yang et al., 2018) or abstractive/generative (Ko-cisk´y et al., 2017; Nguyen et al., 2016; Fan et al.,…”
MethodDuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension
Saha, A., Aralikatte, R., Khapra, M. M., & Sankaranarayanan, K. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1156 · Cited by 110
“There are also some datasets for RC with multiple choice questions (Richardson et al., 2013; Berant et al., 2014; Lai et al., 2017) where the task is to select one among k given candidate answers.”
BackgroundDCMN+: Dual Co-Matching Network for Multi-Choice Reading Comprehension
Zhang, S., Zhao, H. Y., Wu, Y., Zhang, Z., Zhou, X., & Zhou, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6502 · Cited by 155
MuTual: A Dataset for Multi-Turn Dialogue Reasoning
Cui, L., Wu, Y., Liu, S., Zhang, Y., & Zhou, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.130 · Cited by 110
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.04434 · Cited by 109
Neural Approaches to Conversational AI
Gao, J., Galley, M., & Li, L. (2019) · now publishers, Inc. eBooks · doi:10.1561/9781680835533 · Cited by 107
NumNet: Machine Reading Comprehension with Numerical Reasoning
Ran, Q., Lin, Y., Li, P., Zhou, J., & Liu, Z. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1251 · Cited by 108
“In recent years, researchers have published a large number of annotated MRC datasets such as CNN/Daily Mail (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016), RACE (Lai et al., 2017), Trivi-aQA (Joshi et al., 2017) and so on.”
A Co-Matching Model for Multi-choice Reading Comprehension
Wang, S., Yu, M., Jiang, J., & Chang, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2118 · Cited by 106
“Recently, Lai et al. (2017) released a new multi-choice machine comprehension dataset called RACE that was extracted from middle and high school English examinations in China.”
BackgroundMethodGREEK-BERT: The Greeks visiting Sesame Street
Koutsikakis, J., Chalkidis, I., Malakasiotis, P., & Androutsopoulos, I. (2020) · Hellenic Conference on Artificial Intelligence · doi:10.1145/3411408.3411440 · Cited by 172
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
Ye, Q., Lin, B. Y., & Ren, X. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.572 · Cited by 103
“…al. 2019b quoref qa/machine reading comprehension Dasigi et al. 2019 race-high qa/multiple-choice qa Lai et al. 2017 race-middle qa/multiple-choice qa Lai et al. 2017 reddit_tifu-title cg/summarization Kim et al. 2019 reddit_tifu-tldr cg/summarization Kim et al. 2019 ropes qa/machine reading…”
BackgroundWhat Makes Reading Comprehension Questions Easier?
Sugawara, S., Inui, K., Sekine, S., & Aizawa, A. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1453 · Cited by 105
“Annotation labels: Our annotation labels (Table 4) were inspired by previous work such as Chen et al. (2016), Trischler et al. (2017), and Lai et al. (2017).”
BackgroundMethodInfluentialCompanion Proceedings of the Web Conference 2022
Guo, K., Defretiere, C., Diefenbach, D., Gravier, C., & Gourru, A. (2022) · doi:10.1145/3487553 · Cited by 107
A Survey on Machine Reading Comprehension—Tasks, Evaluation Metrics and Benchmark Datasets
Zeng, C., Li, S., Li, Q., Hu, J., & Hu, J. (2020) · Applied Sciences · doi:10.3390/app10217640 · Cited by 148
Memory-Efficient Pipeline-Parallel DNN TrainingPreprint
Narayanan, D., Phanishayee, A., Shi, K., Chen, X., & Zaharia, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09503 · Cited by 176
“To further validate the quality of the pre-trained model, we finetuned the pre-trained vanilla and 2BW BERT models on downstream MNLI and RACE tasks (Wang et al., 2019; Lai et al., 2017).”
MethodDeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99
“These datasets include HellaSwag, PIQA, WinoGrande, RACE-Middle, RACE-High, MMLU, ARC-Easy, ARC-Challenge, OpenBookQA, CHID, C-Eval, CMMLU, C3 and CCPM.”
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S., & Huang, J. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.29 · Cited by 91
Learning to Encode Position for Transformer with Continuous Dynamical ModelPreprint
Liu, X., Yu, H., Dhillon, I. S., & Hsieh, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09229 · Cited by 137
“However, the weights w h in dynamic function h ( τ, p τ ; w h ) are not trained in large corpus, given that GLUE/SQuAD/RACE datasets are too small to train dynamics from scratch, we decided to pretrain h alone in WikiText103 [21] data using masked language modeling loss.”
InfluentialMCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
Ostermann, S., Modi, A., Roth, M., Thater, S., & Pinkal, M. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2x36g84ob95p · Cited by 92
A Survey on Machine Reading Comprehension Systems
Baradaran, R., Ghiasi, R., & Amirkhani, H. (2022) · Natural Language Engineering · doi:10.1017/s1351324921000395 · Cited by 143
Reasoning Over Paragraph Effects in Situations
Lin, K., Tafjord, O., Clark, P., & Gardner, M. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5808 · Cited by 92
“We also experiment with first fine-tuning on RACE (Lai et al., 2017) before fine-tuning on ROPES .”
MethodFrom 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo ProjectPreprint
Clark, P. E., Etzioni, O., Khashabi, D., Khot, T., Mishra, B. D., Richardson, K., Sabharwal, A., Schoenick, C., Tafjord, O., Tandon, N., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01958 · Cited by 92
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense ReasoningPreprint
Huang, L., Bras, R. J. L., Bhagavatula, C., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00277 · Cited by 195
“Specifically, we first fine-tune BERT on RACE or SWAG or both, and directly test on C OS - MOS to show the impact of knowledge transfer.”
BackgroundResultInfluentialExplaining NLP Models via Minimal Contrastive Editing (MiCE)
Ross, A., Marasović, A., & Peters, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.336 · Cited by 86
Li, J., Liu, M., Kan, M., Zheng, Z., Wang, Z., Lei, W., Liu, T., & Qin, B. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.238 · Cited by 138
Generative AI in mobile networks: a survey
Karapantelakis, A., Alizadeh, P., Alabbasi, A., Dey, K., & Nikou, A. (2023) · Annals of Telecommunications · doi:10.1007/s12243-023-00980-9 · Cited by 116
“Some of their important benchmarks are crowdsourced questions derived from Wikipedia articles (SQuAD) [49], multiple-choice reading (RACE) [50], and diverse set of text-based tasks covering paraphrasing, sentiment, and linguistic acceptability (GLUE) [51].”
LearningQ: A Large-Scale Dataset for Educational Question Generation
Chen, G., Yang, J., Hauff, C., & Houben, G.-J. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.14987 · Cited by 83
“…all of these datasets are not suitable for educational question generation due to either the limited number of topics (Rajpurkar et al. 2016; Lai et al. 2017) or the loose dependency between documents and questions, i.e., a document might not contain the content necessary to generate a…”
BackgroundMethodRepresentation Learning for Natural Language Processing
Liu, Z., Lin, Y., & Sun, M. (2020) · Springer eBooks · doi:10.1007/978-981-15-5573-2 · Cited by 142
DRCD: a Chinese Machine Reading Comprehension Dataset
Shao, C.-C., Liu, T., Lai, Y., Tseng, Y., & Tsai, S. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00920 · Cited by 80
“But Lai et al. (2017) also indicate that multiple-choice MRC datasets are often far from sufficient for the training of advanced data-driven MRC models because of the expensive data-generation process by human experts.”
BackgroundAlignScore: Evaluating factual consistency with a unified alignment function
Zha, Y., Yang, Y., Li, R., & Hu, Z. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.634 · Cited by 81
Self-citationWang, L., Sun, M., Zhao, W., Shen, K., & Liu, J. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1120 · Cited by 96
“The results are in Table 4 Pretraining on RACE dataset turns out to be the most important factor.”
BackgroundInfluentialRecent Advances in Neural Question Generation
Pan, L., Lei, W., Chua, T., & Kan, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08949 · Cited by 75
“53 RACE (Lai et al., 2017) multiple choice Education 27,933 72,547 2.”
BackgroundCJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension
Duan, X., Wang, B., Wang, Z., Ma, W., Cui, Y., Wu, D., Wang, S., Liu, T., Huo, T., Hu, Z., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32381-3_36 · Cited by 109
“0 [16] focuses on Wikipedia, and RACE [12] concentrates on Chinese middle school students’ English reading comprehension examination questions.”
BackgroundInfluentialExT5: Towards Extreme Multi-Task Scaling for Transfer LearningPreprint
Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H., Mehta, S. V., Zhuang, H., Tran, V. Q., Bahri, D., Ni, J., et al. (2021) · arXiv.org · doi:10.48550/arxiv.2111.10952 · Cited by 241
Understanding Emergent Abilities of Language Models from the Loss Perspective
Du, Z., Zeng, A., Dong, Y., & Tang, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1683 · Cited by 76
Liu, Y., Li, D., Wang, K., Xiong, Z., Shi, F., Wang, J., Li, B., & Hang, B. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103809 · Cited by 130
Hierarchical Attention Flow for Multiple-Choice Reading Comprehension
Zhu, H., Wei, F., Qin, B., & Liu, T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12040 · Cited by 77
“2017) which are trained with 100D pre-trained Glove word embeddings, ‡ indicates the results that we get by running the published code (Lai et al. 2017) of GA Reader and Stanford AR with 300D pre-trained Glove word embeddings.”
BackgroundMethodInfluentialWhat to Pre-Train on? Efficient Intermediate Task Selection
Poth, C., Pfeiffer, J., Rücklé, A., & Gurevych, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.827 · Cited by 142
“83.29 80.06 RACE (Lai et al., 2017) 88K reading comprehension English exams acc.”
BackgroundInfluentialScienceQA: a novel resource for question answering on scholarly articles
Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., & Bhattacharyya, P. (2022) · International Journal on Digital Libraries · doi:10.1007/s00799-022-00329-y · Cited by 261
“RACE : RACE [29] consists of 28,000 passages and 100,000 questions generated by human experts that cover a variety of topics of English examinations for middle and high school level Chinese students to test their ability of understanding and reasoning.”
BackgroundMethodGeneralizing Question Answering System with Pre-trained Language Model Fine-tuning
Su, D., Xu, Y., Winata, G. I., Xu, P., Kim, H., Liu, Z., & Fung, P. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5827 · Cited by 69
“The fact that the single-task model achieves better performance on RACE than that using the multi-task learning method is related to the higher similarity between RACE and NaturalQuestions, which we can figure out from Figure 2.”
BackgroundInfluentialTowards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering
Ma, K., Francis, J., Lu, Q., Nyberg, E., & Oltramari, A. (2019) · doi:10.18653/v1/d19-6003 · Cited by 89
MMM: Multi-Stage Multi-Task Learning for Multi-Choice Reading Comprehension
Jin, D., Gao, S., Kao, J.-Y., Chung, T., & Hakkani-Tür, D. Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6310 · Cited by 78
Perturbation augmentation for fairer NLP
Qian, R., Ross, C., Fernandes, J., Smith, E. M., Kiela, D., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.646 · Cited by 69
Improving Question Answering by Commonsense-Based Pre-training
Zhong, W., Tang, D., Duan, N., Zhou, M., Wang, J., & Yin, J. P. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32233-5_2 · Cited by 86
A Vietnamese Dataset for Evaluating Machine Reading Comprehension
Nguyen, K. V., Nguyen, D.-V., Nguyen, T. H. A., & Nguyen, N. L.-T. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.233 · Cited by 78
Prompting Large Language Models With the Socratic Method
Chang, E. Y. (2023) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc57344.2023.10099179 · Cited by 145
“evaluating the quality and credibility of written materials, from research papers to blog posts [18, 26].”
BackgroundXu, Y., Wang, D., Yu, M., Ritchie, D., Yao, B., Wu, T., Zhang, Z., Li, T. J.-J., Bradford, N., Sun, B., et al. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.34 · Cited by 133
Uni-MoE: Scaling Unified Multimodal LLMs With Mixture of Experts
Li, Y., Jiang, S., Hu, B., Wang, L., Zhong, W., Luo, W., Ma, L., & Zhang, M. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3532688 · Cited by 257
TWEETQA: A Social Media Focused Question Answering Dataset
Xiong, W., Wu, J., Wang, H., Kulkarni, V., Yu, M., Chang, S., Guo, X., & Wang, W. Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1496 · Cited by 65
“In recent et al., 2015; Hill et al., 2015) or quiz-style problems (Richardson et al., 2013; Lai et al., 2017).”
BackgroundA Transformer-Based Approach to Multilingual Fake News Detection in Low-Resource Languages
De, A., Bandyopadhyay, D., Gain, B., & Ekbal, A. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3472619 · Cited by 104
“, [8, 21]) showed progress on many popular NLP tasks such as Reading Comprehension on widely used datasets like SQuAD [22] and RACE [14].”
BackgroundGenerating Distractors for Reading Comprehension Questions from Real Examinations
Gao, Y., Bing, L., Li, P., King, I., & Lyu, M. R. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016423 · Cited by 63
Representation Engineering: A Top-Down Approach to AI Transparency
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01405 · Cited by 65
QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions
Tafjord, O., Gardner, M., Lin, K., & Clark, P. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1608 · Cited by 64
“BERT-PFT (no knowledge): BERT first fine-tuned (“pre-fine-tuned”) on the RACE dataset (Lai et al., 2017; Sun et al., 2019), and then fine-tuned on Q UA RT Z (questions only, no K , both train and test).”
MethodAssessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
Sugawara, S., Stenetorp, P., Inui, K., & Aizawa, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6422 · Cited by 63
DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering
Cao, Q., Trivedi, H. K., Balasubramanian, A., & Balasubramanian, N. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.411 · Cited by 89
A Historical Survey of Advances in Transformer Architectures
Sajun, A. R., Zualkernan, I. A., & Sankalpa, D. (2024) · Applied Sciences · doi:10.3390/app14104316 · Cited by 91
“Multiple corrupted counterparts were generated for each image in the famous benchmark ImageNet dataset using several carefully designed operations.”
MethodIncorporating Structured Commonsense Knowledge in Story Completion
Chen, J., Chen, J., & Zhou, Y. T. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016244 · Cited by 77
Difficulty Controllable Generation of Reading Comprehension Questions
Gao, Y., Bing, L., Chen, W., Lyu, M., & King, I. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/690 · Cited by 60
“Note that the answers of SQuAD questions are text spans in the input sentence, and they are significantly different from RACE questions [Lai et al., 2017] such as “What do you learn from the story?”.”
BackgroundTWilBert: Pre-trained deep bidirectional transformers for Spanish Twitter
González, J. Á., Hurtado, L.-F., & Plà, F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.09.078 · Cited by 69
“H = 2048 was trained and fine-tuned for the ReAding Comprehension from Examinations dataset [18], obtaining significantly lower results than another model trained with L = 24 and H = 1024 (BERT large [6]).”
MethodMulti-hop Question Answering via Reasoning Chains
Chen, J., Lin, S.-T., & Durrett, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02610 · Cited by 61
“Earlier question answering datasets contain some questions of this form (Richardson et al., 2013; Lai et al., 2017), but typically exhibit a limited range of multi-hop phenomena.”
BackgroundChID: A Large-scale Chinese IDiom Dataset for Cloze Test
Zheng, C., Huang, M., & Sun, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1075 · Cited by 57
“RACE (Lai et al., 2017) and CLOTH (Xie et al., 2018) are constructed from questions in examinations designed for secondary and high school students.”
MethodTo Test Machine Comprehension, Start by Defining Comprehension
Dunietz, J., Burnham, G., Bharadwaj, A., Rambow, O., Chu-Carroll, J., & Ferrucci, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.701 · Cited by 58
SG-Net: Syntax Guided Transformer for Language Representation
Zhang, Z., Wu, Y., Zhou, J., Duan, S., Zhao, H. Y., & Wang, R. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3046683 · Cited by 71
“Our multi-choice MRC is evaluated on Large-scale ReAding Comprehension Dataset From Examinations (RACE) dataset [59], which consists of two subsets: RACE-M and RACE-H corresponding to middle school and high school difficulty levels.”
MethodShortGPT: Layers in Large Language Models are More Redundant Than You Expect
Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., Han, X., & Chen, W. (2024) · arXiv · doi:10.48550/arxiv.2403.03853 · Cited by 60
“Furthermore, we note that the approach of reducing the number of layers (ShortGPT/LaCo) outperforms the method of reducing the embedding dimensions (LLM-6 As a pruning method, we further validated the effects of different pruning ratios on model performance.”
MethodCLUE: A Chinese Language Understanding Evaluation BenchmarkPreprint
Xu, L., Zhang, X., Li, L., Hu, H., Cao, C., Liu, W., Li, J., Li, Y., Sun, K., Xu, Y., et al. (2020) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2004.05986 · Cited by 487
FriendsQA: Open-Domain Question Answering on TV Show Transcripts
Yang, Z., & Choi, J. D. (2019) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w19-5923 · Cited by 83
“RACE is a large dataset compiled from English assessments for 1218 years old students (Lai et al., 2017).”
BackgroundDocument Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension
Zheng, B., Wen, H., Liang, Y., Duan, N., Che, W., Jiang, D., Zhou, M., & Liu, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.599 · Cited by 59
Careful Selection of Knowledge to Solve Open Book Question Answering
Banerjee, P., Pal, K. K., Mitra, A., & Baral, C. R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1615 · Cited by 56
“…been proposed for natural language question answering (Rajpurkar et al., 2016; Joshi et al., 2017; Khashabi et al., 2018; Richardson et al., 2013; Lai et al., 2017; Reddy et al., 2018; Choi et al., 2018; Tafjord et al., 2018; Mitra et al., 2019) and many attempts have been made to solve these…”
Deepfake Text Detection: Limitations and Opportunities
Pu, J., Sarwar, Z., Abdullah, S. M., Rehman, A., Kim, Y., Bhattacharya, P., Javed, M., & Viswanath, B. (2023) · 2023 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46215.2023.10179387 · Cited by 53
“RoBERTa is known to outperform BERT on NLP tasks such as GLUE [39], SQuAD [40], and RACE [41].”
MethodGreco, C. M., & Tagarelli, A. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09374-7 · Cited by 104
Enabling All In-Edge Deep Learning: A Literature Review
Joshi, P., Hasanuzzaman, M., Thapa, C., Afli, H., & Scully, T. (2023) · IEEE Access · doi:10.1109/access.2023.3234761 · Cited by 77
“The same model was then able to achieve the SOTA accuracy on the ReAding Comprehension Dataset From Examinations (RACE [42]) dataset with improved throughput by 10% as compared to existing approaches.”
MethodAdaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task
An, Y., Liu, K., Liu, J., Lyu, Y., & Li, S. (2018) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/w18-2611 · Cited by 80
“The paradigm of such datasets ranges from cloze test (Hermann et al., 2015; Hill et al., 2015), multiple choice (Lai et al., 2017), span extraction (Rajpurkar et al., 2016) and answer generation (Nguyen et al., 2016; He et al., 2017).”
BackgroundOptimizing Deeper Transformers on Small Datasets
Xu, P., Kumar, D., Yang, W., Zi, W., Tang, K., Huang, C., Cheung, J. C. K., Prince, S. J. D., & Cao, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.163 · Cited by 53
“, 2020b) without using the additional RACE dataset (Lai et al., 2017).”
MethodTeach Me to Explain: A Review of Datasets for Explainable NLP.Preprint
Wiegreffe, S., & Marasović, A. (2021) · arXiv (Cornell University) · Cited by 64
From Machine Reading Comprehension to Dialogue State Tracking: Bridging the Gap
Gao, S., Agarwal, S., Jin, D., Chung, T., & Hakkani-Tür, D. Z. (2020) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2020.nlp4convai-1.10 · Cited by 63
The role of large language models in agriculture: harvesting the future with LLM intelligence
Shaikh, T. A., Rasool, T., Veningston, K., & Yaseen, S. M. (2024) · Progress in Artificial Intelligence · doi:10.1007/s13748-024-00359-4 · Cited by 86
Relation-aware Video Reading Comprehension for Temporal Language Grounding
Gao, J., Sun, X., Xu, M., Zhou, X., & Ghanem, B. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.324 · Cited by 82
“The latter (Lai et al., 2017; Sun et al., 2019) aims to find the only correct option in the given candidate choices based on the given passage.”
Question answering by reasoning across documents with graph convolutional networks
De Cao, N., Aziz, W., & Titov, I. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1240 · Cited by 54
Evidence Sentence Extraction for Machine Reading Comprehension
Wang, H., Yu, D., Sun, K., Chen, J., Yu, D., McAllester, D., & Roth, D. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1065 · Cited by 60
“The average number of evidence sentences in whole RACE dataset should be higher since questions in RACE-High are more difficult (Lai et al., 2017), and we ignore 10.”
BackgroundMethodInfluentialUncertainty Quantification and Confidence Calibration in Large Language Models: A Survey
Liu, X., Chen, T., Da, L., Chen, C., Lin, Z., & Wei, H. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736569 · Cited by 53
“…[143], GSM8K [13], CalibratedMath [75] Factuality TruthfulQA [76], FEVER [123], HaluE-val [70], FActScore [90] General Knowledge MMLU [41], GPQA [110], HellaSwag [150] Consistency & Am-biguity ParaRel [30], AmbigQA [92], Ambi-gInst [43] Table 3: Categorization of datasets and benchmarks. multi-hop…”
Evaluating NLP Models via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · Cited by 75
Battle of the Wordsmiths: Comparing ChatGPT, GPT-4, Claude, and BardPreprint
Borji, A., & Mohammadian, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4476855 · Cited by 76
Discovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.09251 · Cited by 53
Arabic machine reading comprehension on the Holy Qur’an using CL-AraBERT
Malhas, R., & Elsayed, T. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103068 · Cited by 52
SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
Zhao, T., Lu, X., & Lee, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.47 · Cited by 124
A Survey on Transfer Learning in Natural Language ProcessingPreprint
Alyafeai, Z., Al-shaibani, M. S., & Ahmad, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04239 · Cited by 135
Question Directed Graph Attention Network for Numerical Reasoning over Text
Chen, K., Xu, W., Cheng, X., Xiaochuan, Z., Zhang, Y., Song, L., Wang, T., Qi, Y., & Chu, W. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.549 · Cited by 57
Memory-efficient Transformers via Top-k Attention
Gupta, A., Dar, G., Goodman, S., Ciprut, D., & Berant, J. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.5 · Cited by 123
QuALITY: Question Answering with Long Input Texts, Yes!
Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., et al. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.391 · Cited by 51
Video Moment Retrieval via Comprehensive Relation-Aware Network
Xin, S., Gao, J., Zhu, Y., Wang, X., & Zhou, X. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3250518 · Cited by 59
Text-based Question Difficulty Prediction: A Systematic Review of Automatic Approaches
AlKhuzaey, S., Grasso, F., Payne, T. R., & Tamma, V. A. M. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00362-1 · Cited by 122
Option Comparison Network for Multiple-choice Reading ComprehensionPreprint
Ran, Q., Li, P., Hu, W., & Zhou, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.03033 · Cited by 64
“Therefore, RACE is extremely challenging for MCRC models.”
BackgroundMethodInfluential
Metadata
- Authors
- Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 785-794
- OpenAlex
- W2606964149
- Cited by
- 1,044
- References
- 19
- Citations by year
- 2026: 37 · 2025: 75 · 2024: 86 · 2023: 145 · 2022: 136 · 2021: 216 · 2020: 169 · 2019: 138 · 2018: 38 · 2017: 3
- FWCI
- 42.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750, FA8750-12-2-0342)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, machine reading comprehension, benchmark dataset, English examinations, Chinese students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.