RACE: Large-scale ReAding Comprehension Dataset From Examinations
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task.Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning.In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%).We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [112]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, to our knowledge, most of the existing datasets are not of direct relevance to teaching and learning, except for RACE (Lai, Xie, Liu, Yang, & Hovy, 2017) and LearningQ (Chen, Yang, Hauff, & Houben, 2018).”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Lai, Xie, Liu, Yang, & Hovy, 2017
The cited source says what the claim attributes to it.
In the citing paper
“On RACE-h and RACE-m (Lai et al., 2017), Chinchilla greatly outperforms Gopher , improving accuracy by more than 10% in both cases-see Table 7.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Lai et al., 2017
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We consider only the case of human-readable prompts to large models, which contrasts with methods such as prompt-tuning [16,20], which learns a continuous prompt embedding, and auto-prompting [33], which attempts to search for prompts from scratch, both of which require a training step.”
From Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We choose word embedding size d = 100 and use the 100-dimensional Glove word embedding (Pennington et al., 2014) as embedding initialization.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Instead of evaluating the accuracy, researchers need to use F1 score, BLEU (Papineni et al., 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the questions come without candidate spans.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 8×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
MethodResultInfluentialTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 3×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 1×
“, 2016) and recently proposed TriviaQA (Joshi et al., 2017).”
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578 · cited 2×
“, 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the”
BackgroundSelf-citationNewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 11×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodResultInfluentialMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610 · cited 4×
“Although efforts have been made with a similar motivation, including the MCTest dataset created by (Richardson et al., 2013) (containing 500 passages and 2000 questions) and several others (Peñas et al.”
BackgroundMethodInfluentialA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491 · cited 11×
“However, both require limited reasoning ability (Chen et al., 2016).”
BackgroundMethodInfluential
Cited by
Items 501–600 of 1,038
Fact-driven Logical Reasoning.Preprint
Ouyang, S., Zhang, Z., & Zhao, H. (2021) · arXiv (Cornell University) · Cited by 11
BCA: Bilinear Convolutional Neural Networks and Attention Networks for legal question answering
Zhang, H., Zhang, T., Cao, F., Wang, Z., Zhang, Y., Sun, Y., & Vicente, M. A. (2022) · AI Open · doi:10.1016/j.aiopen.2022.11.002 · Cited by 10
REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training
Jiao, F., Guo, Y., Niu, Y., Ji, F., Li, F.-L., & Nie, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.13 · Cited by 10
Evaluation Paradigms in Question Answering
Rodriguez, P., & Boyd-Graber, J. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.758 · Cited by 24
Fine-Grained Sentiment Analysis with a Fine-Tuned BERT and an Improved Pre-Training BERT
Li, J. (2023) · IEEE International Conference on Image Processing and Computer Applications (ICIPCA) · doi:10.1109/icipca59209.2023.10257673 · Cited by 9
“They also discovered that despite BERT's substantial undertraining, it could perform as well as or even better than nearly every model that was published after it and obtained impressive performance on many datasets, including GLUE [6], SQuAD [7, 8], and RACE [9].”
MethodGeneration of Multiple Choice Questions from Indian Educational Text
Saddish, A., Somaiah, P., Gambhir, V., & Nirmala, S. J. (2023) · Asian Conference on Innovation in Technology (ASIANCON) · doi:10.1109/asiancon58793.2023.10270551 · Cited by 8
“The distractor generation module in the [12] uses a T5 transformer trained on the RACE dataset [13] for context dependent distractors and a sense2vec model for context independent distractors.”
MethodControlling Risk of Web Question Answering
Su, L., Guo, J., Fan, Y., Lan, Y., & Cheng, X. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331261 · Cited by 12
“According to the answer form, the MRC task can be further divided into four categories, namely clozestyle MRC [19], multi-choice MRC [24, 36], span-prediction based MRC [34, 35] and free-form MRC [32].”
MethodCommonsense Inference in Natural Language Processing (COIN) - Shared Task Report
Ostermann, S., Zhang, S., Roth, M., & Clark, P. E. (2019) · doi:10.18653/v1/d19-6007 · Cited by 12
“…used a wide range of external corpora and resources to augment their models, ranging from other machine comprehension data sets such as RACE (Lai et al., 2017) or MCScript (Ostermann et al., 2018a), up to commonsense knowledge databases such as ConceptNet (Speer et al., 2017), WebChild…”
BackgroundMethodInfluentialREM-Net: Recursive Erasure Memory Network for Commonsense Evidence Refinement
Huang, Y., Fang, M., Zhan, X., Cao, Q., & Liang, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16791 · Cited by 8
Common Sense-Based Reasoning Using External Knowledge for Question Answering
Yang, Y., & Kang, S. (2020) · IEEE Access · doi:10.1109/access.2020.3045762 · Cited by 9
“Typical datasets include the MCTest [13], RACE [14], CommonsenseQA [1], and Open-bookQA [15].”
MethodSelf-citationAnswer-Aware Question Generation from Tabular and Textual Data using T5
Pandraju, S., & Mahalingam, S. G. (2021) · International Journal of Emerging Technologies in Learning (iJET) · doi:10.3991/ijet.v16i18.25121 · Cited by 12
“[31] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang and Eduard Hovy, “RACE: Large-scale ReAding Comprehension Dataset from Examinations", In Proceedings of the 2017 Conference on EMNLP, pp. 785–794, Sep. 2017. https://doi.org/10.18653/v1/d17-1082 [32] Mark Przybocki, Kay Peterson, and Sébastien Bronsart, “NIST Metric…”
MethodInfluentialInterpretable modular knowledge reasoning for machine reading comprehension
Ren, M., Huang, H., & Gao, Y. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-06975-2 · Cited by 9
“Figure 3 shows the learning curve of our model, RoBERTa large baseline, and data-augmented model (we selected RACE [38] as the external dataset which contained around 100,000 questions.) on the test set.”
MethodShin, D., Lee, J. H., & Lee, Y. (2022) · System · doi:10.1016/j.system.2022.102863 · Cited by 9
Improving the robustness of machine reading comprehension via contrastive learning
Feng, J., Sun, J., Shao, D., & Cui, J. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03947-w · Cited by 8
Liu, M., Xiang, J., Xu, X., & Hu, H. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3551637 · Cited by 9
“In general, MRC is categorized into four tasks [17], cloze test [40], multiple choice [12], span extraction [29, 30], and free answering [23].”
MethodEliciting and Understanding Cross-task Skills with Task-level Mixture-of-Experts
Ye, Q.-Y., Zha, J., & Ren, X. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.findings-emnlp.189 · Cited by 18
CLER: Cross-task Learning with Expert Representation to Generalize Reading and Understanding
Takahashi, T., Taniguchi, M., Taniguchi, T., & Ohkuma, T. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5824 · Cited by 9
“184 date the model as in-domain settings, and unveiled six out of the twelve test datasets 1 (Dua et al., 2019; Lai et al., 2017; Kembhavi et al., 2017; Levy et al., 2017; Saha et al., 2018) to validate the trained model as out-of-domain settings.”
BackgroundA BERT-based Dual Embedding Model for Chinese Idiom Prediction
Tan, M., & Jiang, J. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.113 · Cited by 11
A Vietnamese Dataset for Evaluating Machine Reading ComprehensionPreprint
Nguyen, K. V., Nguyen, D.-V., Nguyen, A. G.-T., & Nguyen, N. L.-T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.14725 · Cited by 8
“…et al., 2018; Trischler et al., 2016), cloze-style MRC datasets (Hermann et al., 2015; Hill et al., 2015; Cui et al., 2016), reading comprehension with multiple-choice (Richardson et al., 2013; Lai et al., 2017) and conversation-based reading comprehension (Reddy et al., 2019; Sun et al., 2019).”
BackgroundXiao, D., Li, Y., Zhang, H., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12148 · Cited by 8
“The ReAding Comprehension from Examinations (RACE; Lai et al., 2017) (Wei et al., 2019) 82.2 81.2 90.9 87.9 - / - - / - - / - - / - - - MacBERT LARGE (Cui et al., 2020) (Diao et al., 2019) 80.5 79.2 90.2 88.0 - / - - / - - / - - / - - - NEZHA BASE (Wei et al., 2019) 81.4 79.3 90.0 87.4 - / - - / -…”
MethodInfluentialImproving Machine Reading Comprehension with Single-choice Decision and Transfer LearningPreprint
Jiang, Y., Wu, S., Gong, J., Cheng, Y., Meng, P., Lin, W.-L., Chen, Z., & Li, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.03292 · Cited by 13
An MRC Framework for Semantic Role LabelingPreprint
Wang, N., Li, J., Meng, Y., Sun, X., Qiu, H., Wang, Z., Wang, G., & He, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06660 · Cited by 12
“According to the type of the answer, machine reading comprehension can be divided into the following four categories: extractive (Rajpurkar et al., 2016), multiple-choice (Lai et al., 2017), close style (Onishi et al., 2016), and free-form (Nguyen et al., 2016).”
BackgroundHandover QG: Question Generation by Decoder Fusion and Reinforcement Learning
Chung, H.-L., Chan, Y.-H., & Fan, Y.-C. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3426292 · Cited by 8
“…are two sorts of datasets for QG training: (1) factoid-type datasets, originally created for training machine reading comprehension system (e.g., SQuAD [8] or NewsQA [9]), and (2) exam-type datasets, collected from educational reading comprehension tests (e.g., RACE [10], [11] or Learning-Q [12]).”
BackgroundMa, Y., Qing, L., Kang, Y., Liu, J., Zhang, Y., Cheng, Q., Lu, W., & Liu, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127226 · Cited by 13
Aligning large language models with human preferences using historical text edits
Majkutewicz, J., & Szymański, J. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113566 · Cited by 9
Cooking Well, With Love, Is an Art: Transformers on Youtube Hinglish Data
Yadav, S., Kaushik, A., & Sharma, S. (2021) · International Conference on Computational Performance Evaluation (ComPE) · doi:10.1109/compe53109.2021.9752377 · Cited by 12
Keywords-aware dynamic graph neural network for multi-hop reading comprehension
Jia, M., Liao, L., Wang, W., Li, F., Chen, Z., Li, J., & Huang, H. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.05.110 · Cited by 8
Analyzing Multi-Task Learning for Abstractive Text Summarization
Kirstein, F., Wahle, J. P., Ruas, T., & Gipp, B. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.5 · Cited by 13
Tunable Soft Prompts are Messengers in Federated Learning
Dong, C., Xie, Y., Ding, B., Shen, Y., & Li, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.976 · Cited by 23
Distractor Generation Through Text-to-Text Transformer Models
de-Fitero-Dominguez, D., García‐López, E., García‐Cabot, A., del-Hoyo-Gabaldon, J.-A., & Moreno-Cediel, A. (2024) · IEEE Access · doi:10.1109/access.2024.3361673 · Cited by 14
“DG-RACE [13] is a variant of the RACE [19] dataset.”
MethodInfluentialLeite, B., Osório, T. F., & Cardoso, H. L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72315-5_16 · Cited by 11
“Focusing on corpora explicitly created for an educational purpose, RACE [10], CLOTH [21], and FairytaleQA [22] are prime examples of this accomplishment.”
BackgroundTransformer-based Pouranic topic classification in Indian mythology
Paul, A., Seal, S., & Das, D. (2024) · Sadhana · doi:10.1007/s12046-024-02598-6 · Cited by 7
“RoBERTa achieves state-of-the-art performance on GLUE, ReAding Comprehension Dataset From Examinations (RACE) [32], and Stanford Question Answering Dataset (SQuAD) [33].”
Explanatory argument extraction of correct answers in resident medical exams
Goenaga, I., Atutxa, A., Gojenola, K., Oronoz, M., & Agerri, R. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102985 · Cited by 17
Patel, S., Patel, J., Shah, D., Goel, P., & Patel, B. (2024) · International Conference on Data Intelligence and Cognitive Informatics (ICDICI) · doi:10.1109/icdici62993.2024.10810811 · Cited by 8
Can Machines Learn to Comprehend Scientific Literature?
Park, D., Choi, Y., Kim, D., Yu, M., Kim, S., & Kang, J. (2019) · IEEE Access · doi:10.1109/access.2019.2891666 · Cited by 7
“In addition to the models, several large-scale datasets [1], [2], [5]–[7] have been proposed for training such intelligent reading machines.”
BackgroundMethodInfluentialDTC: Transfer learning for commonsense machine comprehension
Han, W., Peng, M., Xie, Q., Hu, G., Gao, W., Wang, H., Zhang, Y., & Liu, Z. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.07.110 · Cited by 7
Self-citationImproved Machine Reading Comprehension Using Data Validation for Weakly Labeled Data
Yang, Y., Kang, S., & Seo, J. (2020) · IEEE Access · doi:10.1109/access.2019.2963569 · Cited by 7
“There are several released datasets from which to choose: MS MARCO [33], NarrativeQA [34], MCtest [35], and Race [36].”
MethodSelf-citationAn Empirical Study of Content Understanding in Conversational Question Answering
Chiang, T.-R., Ye, H.-T., & Chen, Y.-N. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6257 · Cited by 11
Cong, Y., Wu, Y., Liang, X., Pei, J., & Qin, Z. (2021) · Applied Intelligence · doi:10.1007/s10489-020-02168-3 · Cited by 7
“In addition to these datasets, there are some other typical multiple-choice datasets, such as RACE [11], MCTest [27], and CoQA [25].”
Meidinger, M., & Aßenmacher, M. (2021) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0010255108660873 · Cited by 8
“Collections of benchmark data sets for different natural language understanding (NLU) tasks (Rajpurkar et al., 2016; Lai et al., 2017; Wang et al., 2018) have gained massive popularity among researchers in this field.”
BackgroundAdversarial Training for Machine Reading Comprehension with Virtual Embeddings
Yang, Z., Cui, Y., Si, C., Che, W., Liu, T., Wang, S., & Hu, G. (2021) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2021.starsem-1.30 · Cited by 7
“For PQAT, (cid:15) δ is set to 0, (cid:15) P and (cid:15) Q is set to 4e-2 for RACE and 2e-2 for other tasks.”
MethodInfluentialGCRC: A New Challenging MRC Dataset from Gaokao Chinese for Explainable Evaluation
Tan, H., Wang, X., Ji, Y., Li, R., Li, X., Hu, Z., Zhao, Y., & Han, X. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.113 · Cited by 16
“There exist some datasets, collected from standard exams/tests, including English datasets RACE (Lai et al., 2017), DREAM (Sun et al.”
BackgroundMethodExtract, Integrate, Compete: Towards Verification Style Reading Comprehension
Zhang, C., Lai, Y., Feng, Y., & Zhao, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.255 · Cited by 9
Evidence Augment for Multiple-Choice Machine Reading Comprehension by Weak Supervision
Luo, D., Zhang, P., Ma, L., Zhu, X., Zhou, M., Liang, Q., Wang, B., & Wang, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86383-8_29 · Cited by 7
Loginova, E., Benedetto, L., Benoit, D. F., & Cremonesi, P. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_097 · Cited by 22
“We experiment on the large question-answering dataset RACE (Lai et al., 2017) to assess the QDE capabilities of the proposed approach.”
MethodCausal and Masked Language Modeling of Javanese Language using Transformer-based Architectures
Wongso, W., Setiawan, D. S., & Suhartono, D. (2021) · International Conference on Advanced Computer Science and Information Systems · doi:10.1109/icacsis53237.2021.9631331 · Cited by 10
“As a result, they outperformed BERT’s results in SQuAD [25], MNLI [24], and RACE [29] benchmarks.”
ResultElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models
Li, J., Tang, T., Gong, Z., Yang, L., Yu, Z., Chen, Z., Wang, J., Zhao, X. W., & Wen, J.-R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.258 · Cited by 8
UKP-SQUARE: An Online Platform for Question Answering Research
Baumgärtner, T., Wang, K., Sachdeva, R. S., Geigle, G., Eichler, M., Poth, C., Sterz, H., Puerto, H., Ribeiro, L. F. R., Pfeiffer, J., et al. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.2 · Cited by 10
qaAskeR$$^+$$: a novel testing method for question answering software via asking recursive questions
Xie, X., Jin, S., & Chen, S. (2023) · Automated Software Engineering · doi:10.1007/s10515-023-00380-2 · Cited by 10
“…to fill in the blanks (Onishi et al. 2016; Suster and Daelemans 2018b), judge the correct options (Clark et al. 2019; Khashabi et al. 2018; Lai et al. 2017), extract the relevant spans (Rajpurkar et al. 2016, 2018; Yang et al. 2018), and return fluent text answers (Kwiatkowski et al.…”
BackgroundDetecting Privacy Policies Violations Using Natural Language Inference (NLI)
Alshamsan, A. R., & Chaudhry, S. A. (2022) · IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) · doi:10.1109/csde56538.2022.10089347 · Cited by 13
“The ReAding Comprehension from Examinations (RACE) is a dataset from middle and high school students in China for English examinations.”
MethodYiğit, G., & Amasyalı, M. F. (2023) · Knowledge and Information Systems · doi:10.1007/s10115-023-01918-2 · Cited by 7
“• RACE [11]: a reading comprehension dataset including questions from English examinations of middle and high school Chinese students.”
BackgroundMethodFiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning
Ye, Q., Beltagy, I., Peters, M. E., Ren, X., & Hajishirzi, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.454 · Cited by 22
ConfliBERT-Arabic: A Pre-trained Arabic Language Model for Politics, Conflicts and Violence
Alsarra, S., Abdeljaber, L., Yang, W., Zawad, N., Khan, L. R., Brandt, P. T., Osorio, J., & D'Orazio, V. J. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_011 · Cited by 8
“The development of pre-trained language models has been accelerated by the introduction of comprehensive benchmarks in the general NLP domain (Wang et al., 2018, 2019; Rajpurkar et al., 2018; Lai et al., 2017), as well as in biomedical applications (Peng et al., 2019; Gu et al., 2021).”
MethodUniversal Self-Adaptive Prompting
Wan, X., Sun, R., Nakhost, H., Dai, H., Eisenschlos, J. M., Arık, S. Ö., & Pfister, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.461 · Cited by 42
LiveQA: A Question Answering Dataset Over Sports Live
Liu, Q., Jiang, S., Wang, Y., & Li, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-63031-7_23 · Cited by 21
JURI SAYS: An Automatic Judgement Prediction System for the European Court of Human Rights
Medvedeva, M., Xu, X., Wieling, M. B., & Vols, M. (2020) · Frontiers in artificial intelligence and applications · doi:10.3233/faia200883 · Cited by 13
PALRACE: Reading Comprehension Dataset with Human Data and Labeled RationalesPreprint
Zou, J., Zhang, Y., Jin, P., Luo, C., Pan, X., & Ding, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12373 · Cited by 10
Šavelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09325 · Cited by 7
“Robinson et al. [32] apply InstructGPT [30] and Codex to OpenBookQA [27], StoryCloze [28], and RACE-m [23] data sets which focus on multi-hop reasoning, recall, and reading comprehension, reporting 77.”
MethodMcCarroll, N., McShane, P., O’Connell, E., Curran, K., Singh, M., McNamee, E., Clist, A., & Brammer, A. (2025) · SN Computer Science · doi:10.1007/s42979-025-04300-x · Cited by 7
“Models leveraging these pre-trained embeddings have achieved competitive results on all nine of the GLUE natural language understanding benchmarks [ 36], the SQUAD [ 37 ] and RACE [ 38] generic benchmark datasets”
Machine-Generated Questions Attract Instructors When Acquainted with Learning Objectives
Shimmei, M., Bier, N., & Matsuda, N. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36272-9_1 · Cited by 11
Exploring The Landscape of Distributional Robustness for Question Answering Models
Awadalla, A., Wortsman, M., Ilharco, G., Min, S., Magnusson, I., Hajishirzi, H., & Schmidt, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.441 · Cited by 40
Pirozelli, P., José, M. M., Silveira, I. C., Cação, F. N., Peres, S. M., Brandão, A. A. F., Costa, A. H. R., & Cozman, F. G. (2024) · Data Intelligence · doi:10.1162/dint_a_00245 · Cited by 15
“As a consequence of its special architecture, RoBERTa achieves state-of-the-art results on SQuAD 1.1 and 2.0, GLUE [18] and RACE [19].”
InfluentialArQuAD: An Expert-Annotated Arabic Machine Reading Comprehension Dataset
Obeidat, R., Al-Harbi, M., Al‐Ayyoub, M., & Alawneh, L. M. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10248-6 · Cited by 13
“Other notable cloze and multiple-choice datasets include CliCR [40], RACE [41], and MCScript [42].”
Automatic Multiple Choice Question Generation: A Systematic Literature Review
Henry, M. M., Heryanto, N. A., Isnan, M., Nugroho, K. S., & Pardamean, B. (2024) · International Conference on Information Technology and Digital Applications (ICITDA) · doi:10.1109/icitda64560.2024.10810102 · Cited by 10
Artificial Intelligence in Clinics: Enhancing Cardiology Practice
Sakamoto, A., Nakamura, Y., Sato, E., & Kagiyama, N. (2025) · JMA Journal · doi:10.31662/jmaj.2024-0190 · Cited by 9
Metacognitive symbolic distillation framework for multi-choice machine reading comprehension
Yao, J., Xu, X., & He, G. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113130 · Cited by 7
Automating Turkish Educational Quiz Generation Using Large Language Models
Zeinalipour, K., Keptiğ, Y. G., Maggini, M., & Gori, M. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-82150-9_19 · Cited by 18
Navigating the Landscape of Hint Generation Research: From the Past to the Future
Jangra, A., Mozafari, J., Jatowt, A., & Muresan, S. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00751 · Cited by 20
Da, J. (2019) · doi:10.18653/v1/d19-6010 · Cited by 5
Masumura, R., Ihori, M., Tanaka, T., Saito, I., Nishida, K., & Oba, T. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003857 · Cited by 4
“Reading comprehension: Global-context iterative rescoring for GLCLMs is a task similar to reading comprehension in which, given the input of passage sentences and a question sentence, an answer sentence is output [32–34].”
BackgroundAssessing COVID-19 Impacts on College Students via Automated Processing of Free-form Text
Sharma, R. S., Pagadala, S. D., Bharti, P., Chellappan, S., Schmidt, T., & Goyal, R. (2021) · International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC) · doi:10.5220/0010249404590466 · Cited by 6
Contextual and Semantic Fusion Network for Multiple-Choice Reading Comprehension
Duan, Q., Huang, J., & Wu, H. (2021) · IEEE Access · doi:10.1109/access.2021.3068993 · Cited by 6
“[4] first-step match the passage against the question and then select an answer using the first matching result.”
BackgroundMethodInfluentialA Multiple-Choice Machine Reading Comprehension Model with Multi-Granularity Semantic Reasoning
Dai, Y., Fu, Y., & Yang, L. (2021) · Applied Sciences · doi:10.3390/app11177945 · Cited by 6
“The RACE dataset is a representative benchmark dataset for multiplechoice reading comprehension, which is constructed using an English test bank for junior and senior high schools.”
MethodInfluentialWhen Retriever-Reader Meets Scenario-Based Multiple-Choice Questions
Huang, Z., Wu, A., Shen, Y., Cheng, G., & Qu, Y. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.84 · Cited by 8
Meta-learning for compressed language model: A multiple choice question answering study
Yan, M., & Pan, Y. F. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.01.148 · Cited by 7
Li, R., Wang, L., Jiang, Z., Hu, Z., Zhao, M., & Lu, X. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07072-0 · Cited by 7
“In contrast to traditional Machine Reading Comprehension (MRC) task where the question can be answered given a single passage [28, 46, 47] or a limited text set [22, 26, 38], open-domain question answering (OpenQA) is more challenging and realistic as models need to retrieve relevant…”
Unsupervised Domain Adaptation for Question Generation with DomainData Selection and Self-training
Zhu, P., & Hauff, C. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.183 · Cited by 12
A Prompt-Based Representation Individual Enhancement Method for Chinese Idiom Reading Comprehension
Sha, Y., Wu, M., Zeng, Z., Ge, X., Huang, Z., & Wang, H. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-30675-4_50 · Cited by 6
Which Shortcut Solution Do Question Answering Models Prefer to Learn?
Shinoda, K., Sugawara, S., & Aizawa, A. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26590 · Cited by 12
Won’t Get Fooled Again: Answering Questions with False Premises
Hu, S., Luo, Y., Wang, H., Cheng, X., Liu, Z., & Sun, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.309 · Cited by 75
Evaluating the Knowledge Dependency of Questions
Moon, H., Yang, Y., Yu, H., Lee, S., Jeong, M., Park, J., Shin, J., Kim, M., & Choi, S. L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.718 · Cited by 18
“, 2020) on the RACE dataset (Lai et al., 2017), a large-scale reading comprehension MCQ dataset.”
BackgroundMethodSelf-citationMPrompt: Exploring Multi-level Prompt Tuning for Machine Reading Comprehension
Chen, G., Qian, Y., Wang, B., & Li, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.343 · Cited by 11
ConfliBERT-Spanish: A Pre-trained Spanish Language Model for Political Conflict and Violence
Yang, W., Alsarra, S., Abdeljaber, L., Zawad, N., Delaram, Z., Osorio, J., Khan, L. R., Brandt, P. T., & D'Orazio, V. J. (2023) · IEEE Congress on Information Science and Technology (CiSt) · doi:10.1109/cist56084.2023.10409883 · Cited by 8
“Most works on pre-trained language models use comprehensive benchmarks in the general NLP [27]–[30].”
BackgroundTopic Enhanced Multi-head Co-Attention: Generating Distractors for Reading Comprehension
Shuai, P., Wei, Z., Liu, S., Xu, X., & Li, L. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533341 · Cited by 6
“1) RACE: RACE 1 [1] is a large dataset collected from the English exams for middle and high school Chinese students.”
BackgroundMethodLeveraging Structured Information from a Passage to Generate Questions
Xu, J., Sun, Y., Gan, J., Zhou, M., & Wu, D. (2022) · Tsinghua Science & Technology · doi:10.26599/tst.2022.9010034 · Cited by 7
Huang, Z., Shen, Y., Li, X., Wei, Y., Cheng, G., Zhou, L., Dai, X., & Qu, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.07855 · Cited by 6
“the answer can be extracted or abstracted from the document (Rajpurkar et al., 2016; Nguyen et al., 2016; Lai et al., 2017).”
BackgroundBIOMRC: A Dataset for Biomedical Machine Reading ComprehensionPreprint
Stavropoulos, P., Pappas, D., Androutsopoulos, I., & McDonald, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06376 · Cited by 6
“There are also several large open-domain MRC datasets annotated by humans (Kwiatkowski et al., 2019; Rajpurkar et al., 2016, 2018; Trischler et al., 2017; Nguyen et al., 2016; Lai et al., 2017).”
BackgroundCounterfactual Variable Control for Robust and Interpretable Question AnsweringPreprint
Yu, S., Niu, Y., Wang, S., Jiang, J., & Sun, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.05581 · Cited by 8
“Impressively, using Megatron-LM achieved less than 10% error on RACE (Lai et al. 2017).”
MethodHow Can We Know When Language Models KnowPreprint
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2020) · arXiv (Cornell University) · Cited by 11
MoEfication: Conditional Computation of Transformer Models for Efficient InferencePreprint
Zhang, Z., Lin, Y., Liu, Z., Li, P., Sun, M., & Zhou, J. (2021) · arXiv (Cornell University) · Cited by 8
Case-Based Abductive Natural Language InferencePreprint
Valentino, M., Thayaparan, M., & Freitas, A. (2020) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2009.14539 · Cited by 9
“We compare CBANLI against published explainable approaches that are fine-tuned only on ARC, without additional pre-training on related datasets (e.g. OpenBookQA (Mihaylov et al., 2018), RACE (Lai et al., 2017)).”
MethodExploring The Landscape of Distributional Robustness for Question Answering Models
Awadalla, A., Wortsman, M., Ilharco, G., Min, S., Magnusson, I., Hajishirzi, H., & Schmidt, L. (2022) · arXiv · doi:10.48550/arxiv.2210.12517 · Cited by 6
MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
Cai, W., Qin, L., & Huang, J. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716006 · Cited by 17
“The downstream tasks includes: HellaSwag [80], PIQA [4], WinoGrande [57], BoolQ [10], ARC-Easy [11], OBQA [41], RACE [29], MathQA [2].”
Optimizing Answer Generator in Vietnamese Legal Question Answering Systems Using Language Models
Thanh, H. L., Luu, N., Nguyen, T. B., Dao, T., & Sang, D. V. (2025) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3732938 · Cited by 5
“RACE [15] is a large-scale multiple-choice from English examinations in China.”
Examining the Feasibility of AI-Generated Questions in Educational Settings
Zeghouani, O., Ali, Z., van Dijkhuizen, W. S., Hong, J., & Clos, J. (2024) · International Symposium on Trustworthy Autonomous Systems (TAS) · doi:10.1145/3686038.3686652 · Cited by 10
“Since our system is designed for educational purposes, QA datasets such as SciQ, RACE and LearningQ [4, 18, 37] will be particularly useful for producing and evaluating questions that test higher-order cognitive skills.”
Ner4Opt: named entity recognition for optimization modelling from natural language
Kadıoğlu, S., Dakle, P. P., Uppuluri, K., Politi, R., Raghavan, P., Rallabandi, S. K., & Srinivasamurthy, R. (2024) · Constraints · doi:10.1007/s10601-024-09376-5 · Cited by 16
“It achieves state-of-the-art results on well-known NLP benchmarks such as GLUE [43], RACE [44] and SQuAD [45, 46].”
Machine Reading Comprehension for the Tamil Language With Translated SQuAD
Sinthusha, A. V. A., Charles, E. Y. A., & Weerasinghe, R. (2025) · IEEE Access · doi:10.1109/access.2025.3530949 · Cited by 7
Generating language assessment content free from representational harms
Choi, I., & Zu, J. (2025) · Language Testing · doi:10.1177/02655322251349560 · Cited by 5
“Details about the passages in the RACE data are available in Lai et al. (2017).”
Question and Answer Classification with Deep Contextualized Transformer
Luo, H., Liu, N., & Feng, C. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-73103-8_32 · Cited by 7
“We will also use RACE [17] and GLUE [38] datasets to do more test work and make more stable algorithms to solve the question and answer classification issues.”
MethodTransformers for Classifying Fourth Amendment Elements and Factors Tests
Gretok, E. W., Langerman, D., & Oliver, W. M. (2020) · Frontiers in artificial intelligence and applications · doi:10.3233/faia200850 · Cited by 5
Metadata
- Authors
- Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 785-794
- OpenAlex
- W2606964149
- Cited by
- 1,044
- References
- 19
- Citations by year
- 2026: 37 · 2025: 75 · 2024: 86 · 2023: 145 · 2022: 136 · 2021: 216 · 2020: 169 · 2019: 138 · 2018: 38 · 2017: 3
- FWCI
- 42.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750, FA8750-12-2-0342)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, machine reading comprehension, benchmark dataset, English examinations, Chinese students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.