RACE: Large-scale ReAding Comprehension Dataset From Examinations
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task.Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning.In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%).We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [112]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, to our knowledge, most of the existing datasets are not of direct relevance to teaching and learning, except for RACE (Lai, Xie, Liu, Yang, & Hovy, 2017) and LearningQ (Chen, Yang, Hauff, & Houben, 2018).”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Lai, Xie, Liu, Yang, & Hovy, 2017
The cited source says what the claim attributes to it.
In the citing paper
“On RACE-h and RACE-m (Lai et al., 2017), Chinchilla greatly outperforms Gopher , improving accuracy by more than 10% in both cases-see Table 7.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Lai et al., 2017
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We consider only the case of human-readable prompts to large models, which contrasts with methods such as prompt-tuning [16,20], which learns a continuous prompt embedding, and auto-prompting [33], which attempts to search for prompts from scratch, both of which require a training step.”
From Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We choose word embedding size d = 100 and use the 100-dimensional Glove word embedding (Pennington et al., 2014) as embedding initialization.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Instead of evaluating the accuracy, researchers need to use F1 score, BLEU (Papineni et al., 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the questions come without candidate spans.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 8×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
MethodResultInfluentialTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 3×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 1×
“, 2016) and recently proposed TriviaQA (Joshi et al., 2017).”
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578 · cited 2×
“, 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the”
BackgroundSelf-citationNewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 11×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodResultInfluentialMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610 · cited 4×
“Although efforts have been made with a similar motivation, including the MCTest dataset created by (Richardson et al., 2013) (containing 500 passages and 2000 questions) and several others (Peñas et al.”
BackgroundMethodInfluentialA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491 · cited 11×
“However, both require limited reasoning ability (Chen et al., 2016).”
BackgroundMethodInfluential
Cited by
Items 401–500 of 1,038
Sentiment Analysis with Neural Models for Hungarian
Laki, L. J., & Yang, Z. G. (2023) · Acta Polytechnica Hungarica · doi:10.12700/aph.20.5.2023.5.8 · Cited by 24
“RoBERTa is a Robustly optimized BERT pre-training approach, which achieves state-of-the-art results on tasks like GLUE [23], RACE [30] and SQuAD [31], while using less resources due to its optimized pre-training paradigm.”
MethodSpecialist or Generalist? Instruction Tuning for Specific NLP Tasks
Shi, C., Su, Y., Yang, C. H., Yang, Y., & Cai, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.947 · Cited by 45
Self-citationLarge Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU
Koto, F., Aisyah, N., Li, H., & Baldwin, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.760 · Cited by 103
ReCO: A Large Scale Chinese Reading Comprehension Dataset on Opinion
Wang, B., Yao, T., Zhang, Q., Xu, J., & Wang, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6450 · Cited by 26
“Other multiple choices MRC datasets including MCScript (Ostermann et al. 2018) that requires the system to understand the script of daily events, and RACE (Lai et al. 2017) where the questions are collected from the English exams of Chinese students.”
BackgroundMethodQuestion Answering Survey: Directions, Challenges, Datasets, Evaluation Matrices
Pandya, H. A., & Bhatt, B. (2021) · Journal of Xidian University · doi:10.37896/jxu15.4/014 · Cited by 46
“erature from Project Gutenberg, 5 middle and high school English exams from RACE [61], news articles from CNN [38], articles from Wikipedia, Reddit articles from the Writing Prompts dataset [30], and science articles from AI2 Science Questions.”
InfluentialHuang, Y., Liu, J., & Lv, C. (2023) · Applied Sciences · doi:10.3390/app13052924 · Cited by 15
Contrastive classification: A label-independent generalization model for text classification
Liang, Y., Tohti, T., & Hamdulla, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.123130 · Cited by 16
Manakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.4 · Cited by 14
Shaikh, T. A., Rasool, T., & Mir, W. A. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2025.104005 · Cited by 22
Answering questions by learning to rank - Learning to rank by answering questions
Pîrtoacă, G.-S., Rebedea, T. E., & Ruşeţi, Ş. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1256 · Cited by 14
“Training the discriminator is performed on the RACE (Reading Comprehension Dataset) which is also collected from English Examinations (Lai et al., 2017).”
MethodSegatron: Segment-Aware Transformer for Language Modeling and Understanding
Bai, H., Shi, P., Lin, J., Xie, Y., Tan, L., Xiong, K., Gao, W., & Li, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17485 · Cited by 36
“We further test our models with RACE (Lai et al. 2017), which is a large-scale reading comprehension dataset with more than 28,000 passages.”
MethodVLSP 2021-ViMRC Challenge: Vietnamese Machine Reading Comprehension
Nguyen, K. V., Tran, S. Q., Nguyen, L. T., Huynh, T. V., Luu, S. T., & Nguyen, N. L.-T. (2022) · VNU Journal of Science Computer Science and Communication Engineering · doi:10.25073/2588-1086/vnucsce.340 · Cited by 15
“…et al., 2017)), abstractive MRC dataset (NarrativeQA (Koˇciský et al., 2018), RECAM (Zheng et al., 2021)), multiple-choices datasets (RACE (Lai et al., 2017) and MCTest (Richardson et al., 2013)), and conversational reading comprehension dataset (CoQA (Reddy et al., 2019) and ViCoQA (Luu…”
BackgroundInfluentialRetrieval Augmentation for Commonsense Reasoning: A Unified Approach
Yu, W., Zhu, C., Zhang, Z., Wang, S., Zhang, Z., Fang, Y., & Jiang, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.294 · Cited by 46
Dynamic Fusion Networks for Machine Reading ComprehensionPreprint
Xu, Y., Liu, J., Gao, J., Shen, Y., & Liu, X. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04964 · Cited by 39
“To this end, we focus on questions from human English examinations , i.e., the RACE dataset (Lai et al., 2017), because they are specifically designed to test students’ comprehension ability in various aspects, which provides more comprehensive and realistic evaluation on MRC than other popular…”
BackgroundMethodInfluentialMultilingual Question Answering from Formatted Text applied to Conversational AgentsPreprint
Siblini, W., Pasqual, C., Lavielle, A., Challal, M., & Cauchois, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.04659 · Cited by 23
“…day by public and private research laboratories around the world and solve complex language processing tasks: Natural Language Inference [43][37][24], Learning Sentence Similarity and Para-phrasing [11][1], Text Classification [39][42], Reading Comprehension and Question Answering [37][31][21][17].”
BackgroundMulti-branch Attentive TransformerPreprint
Yang, F., Xie, S., Xia, Y., Wu, L., Qin, T., Li, X., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10270 · Cited by 23
Investigating Educational and Noneducational Answer Selection for Educational Question Generation
Steuer, T., Filighera, A., & Tregel, T. (2022) · IEEE Access · doi:10.1109/access.2022.3180838 · Cited by 13
“However, all three datasets [10], [21], [22] only contain a document and corresponding questions and do not annotate the answer span inside the document explicitly.”
BackgroundPersonalizing Task-oriented Dialog Systems via Zero-shot Generalizable Reward Function
Siddique, A. B., Maqbool, M. H., Taywade, K., & Foroosh, H. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557417 · Cited by 25
“The MPNet model has produced cutting-edge results on several natural language processing tasks including GLUE [55], SQuAD [40, 41], RACE [21], and sentiment prediction [29] benchmarks.”
MethodVincentio, K., & Suhartono, D. (2022) · Informatica · doi:10.31449/inf.v46i7.4236 · Cited by 22
A Quantitative Study of NLP Approaches to Question Difficulty Estimation
Benedetto, L. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-36336-8_67 · Cited by 13
What’s the Meaning of Superhuman Performance in Today’s NLU?
Tedeschi, S., Bos, J., Declerck, T., Hajič, J., Hershcovich, D., Hovy, E. H., Koller, A., Krek, S., Schockaert, S., Sennrich, R., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.697 · Cited by 57
Self-citationAutomated multiple-choice question generation in Spanish using neural language models
de-Fitero-Dominguez, D., García‐Cabot, A., & García‐López, E. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10076-7 · Cited by 15
“Additionally, the T5 model is fine-tuned specifically for the distraction generation task, leveraging the DG RACE dataset [18], which is a refined version of the original RACE [32].”
Reading Comprehension with Graph-based Temporal-Casual Reasoning
Sun, Y., Gong, C., & Qu, Y. (2018) · International Conference on Computational Linguistics · Cited by 13
CodeQA: A Question Answering Dataset for Source Code Comprehension
Liu, C., & Wan, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.223 · Cited by 49
Logic Traps in Evaluating Attribution Scores
Ju, Y., Zhang, Y., Zhao, Y., Jiang, Z., Liu, K., & Zhao, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.407 · Cited by 32
“We experiment on RACE (Lai et al., 2017), a large-scale question-answering dataset.”
Visuo-Linguistic Question Answering (VLQA) Challenge
Sampat, S. K., Yang, Y., & Baral, C. R. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.413 · Cited by 27
Self-citationDa, J., & Kasai, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01157 · Cited by 13
“(2019) explores fine-tuning on indomain data, similarly to our fine-tuning on the RACE dataset (Lai et al., 2017).”
BackgroundMethodInfluentialBenchmarking Robustness of Machine Reading Comprehension ModelsPreprint
Si, C., Yang, Z., Cui, Y., Ma, W., Liu, T., & Wang, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14004 · Cited by 13
“To this end, many MRC benchmarks have been constructed in different domains, styles, and languages (Rajpurkar et al., 2016; Lai et al., 2017; Dua et al., 2019b; Cui et al., 2019, inter alia ).”
Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective
Yang, P., Wang, J., Gan, R., Zhu, X., Zhang, L., Wu, Z., Gao, X., Zhang, J., & Sakai, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.474 · Cited by 24
Diagram Perception Networks for Textbook Question Answering via Joint Optimization
Ma, J., Liu, J., Chai, Q., Wang, P., & Tao, J. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01954-z · Cited by 17
“ThetextualISAAQistrained on ARC-Easy (Clark et al., 2018), ARC-Challenge, Open-BookQA (Mihaylov et al., 2018), and RACE (Lai et al., 2017) before being fine-tuned on NDTF and NDMC subtasks.”
Fact-Driven Logical Reasoning for Machine Reading Comprehension
Ouyang, S., Zhang, Z., & Zhao, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29850 · Cited by 37
“There has been remarkable progress in MRC, with human-parity benchmark results re-ported in examination-style MRC datasets like SQuAD (Ra-jpurkar, Jia, and Liang 2018) and RACE (Lai et al. 2017).”
BackgroundMethodTranslucent Answer Predictions in Multi-Hop Reading Comprehension
Bhargav, G. P. S., Glass, M., Garg, D., Shevade, S., Dana, S., Khandelwal, D., Subramaniam, L. V., & Gliozzo, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6272 · Cited by 16
“…based (Unger et al. 2014) vs. text based (reading comprehension) (Joshi et al. 2017), (ii) extractive (Ra-jpurkar et al. 2016) vs. abstractive (Lai et al. 2017), (iii) short (Rajpurkar et al. 2016) vs. long answers (Kwiatkowski et al. 2019), (iv) single-hop (Rajpurkar et al. 2016) vs.…”
Reference Knowledgeable Network for Machine Reading Comprehension
Zhao, Y., Zhang, Z., & Zhao, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3164219 · Cited by 13
Bi-directional CognitiveThinking Network for Machine Reading Comprehension
Peng, W., Hu, Y., Xing, L., Xie, Y., Yu, J., Sun, Y., & Wei, X. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.235 · Cited by 14
Subsentence Extraction from Text Using Coverage-Based Deep Learning Language Models
Lim, J., Sa, I., Ahn, H. S., Gasteiger, N., Lee, S. J., & MacDonald, B. (2021) · Sensors · doi:10.3390/s21082712 · Cited by 13
KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding
Lee, H., Yoon, J., Hwang, B., Joe, S., Min, S.-J., & Gwon, Y. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412023 · Cited by 14
“fewer parameters than BERT, ALBERT has been able to achieve new state-of-the-art results on the GLUE, RACE [10], and SQuAD benchmarks.”
BackgroundSKR-QA: Semantic ranking and knowledge revise for multi-choice question answering
Ren, M., Huang, H., & Gao, Y. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.06.076 · Cited by 12
Steuer, T., Bongard, L., Uhlig, J., & Zimmer, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86436-1_22 · Cited by 21
Data Augmentation for Biomedical Factoid Question Answering
Pappas, D., Malakasiotis, P., & Androutsopoulos, I. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.6 · Cited by 24
ExpMRC: explainability evaluation for machine reading comprehension
Cui, Y., Liu, T., Che, W., Chen, Z., & Wang, S. (2022) · Heliyon · doi:10.1016/j.heliyon.2022.e09290 · Cited by 14
“• RACE+ is a new subset that is similar to RACE [4].”
BackgroundMethodResultInfluentialYou Don’t Need Labeled Data for Open-Book Question Answering
Gholami, S., & Noori, M. (2021) · Applied Sciences · doi:10.3390/app12010111 · Cited by 22
What do we expect from Multiple-choice QA Systems?
Shah, K., Gupta, N., & Roth, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.317 · Cited by 19
On the comparability of Pre-trained Language ModelsPreprint
Aßenmacher, M., & Heumann, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.00781 · Cited by 23
Graph-free Multi-hop Reading Comprehension: A Select-to-Guide StrategyPreprint
Wu, B., Zhang, Z., & Zhao, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.11823 · Cited by 26
“N EURAL methods have achieved impressive results on single-hop machine reading comprehension (MRC) datasets [1], [2], [3], [4], [5], where questions are usually straightforward and answers are easy to find.”
MethodMulti-tasking Dialogue Comprehension with Discourse ParsingPreprint
He, Y., Zhang, Z., & Zhao, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03269 · Cited by 13
“…early stage, MRC was regarded as the form of triple-style (passage, question, answer) question answering (QA) task, such as the cloze-style (Hermann et al., 2015; Hill et al., 2016), multiple-choice (Lai et al., 2017; Sun et al., 2019), and span-QA (Rajpurkar et al., 2016; Rajpurkar et al., 2018).”
BackgroundLabutov, I., Yang, B., Prakash, A., & Azaria, A. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1077 · Cited by 16
“RACE (Lai et al., 2017) is an English exam dataset for middle and high school Chinese students.”
BackgroundMulti-Granular Sequence Encoding via Dilated Compositional Units for Reading Comprehension
Tay, Y., Luu, A. T., & Hui, S. C. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1238 · Cited by 12
“Overall, there is a 6% improvement on the RACE-H dataset and Model RACE-M RACE-H RACE Time Sliding Window (Lai et al., 2017) 37.3 30.4 32.2 N/A Stanford AR (Chen et al., 2016) 44.2 43.0 43.3 N/A GA (Dhingra et al., 2016) 43.7 44.2 44.1 N/A ElimiNet (Parikh et al., 2018) N/A N/A 44.5 N/A Dynamic…”
BackgroundMethodInfluentialAnswer Uncertainty and Unanswerability in Multiple-Choice Machine Reading Comprehension
Raina, V., & Gales, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.82 · Cited by 23
“It is found that pre-training models on RACE (Lai et al., 2017) boosted performance of the best single model to an accuracy of 70.8% on DEV-def and 69.7% on EVL-def.”
BackgroundMethodResultInfluentialExploring Social Biases of Large Language Models in a College Artificial Intelligence Course
Kolisko, S., & Anderson, C. J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.26879 · Cited by 25
“Students were given access to two datasets for each downstream task: the Yelp reviews (Zhang, Zhao, and LeCun 2015) and TweetEval (Rosenthal, Farra, and Nakov 2017) datasets for sentiment analysis; the RACE (Lai et al. 2017) and QuAIL (Rogers et al.”
InfluentialHai-yang, Y., Wang, E., Lang, Q., & Wang, J. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3364155 · Cited by 20
Šavelka, J., Agarwal, A., Bogart, C., & Sakr, M. F. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-53656-4_8 · Cited by 21
Conversational Machine Reading Comprehension for Vietnamese Healthcare Texts
Luu, S. T., Bui, M. N., Nguyen, L. D., Tran, K. V., Nguyen, K. V., & Nguyen, N. L.-T. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-88113-9_44 · Cited by 13
“Many of MRC corpora are constructed on specific domains and open domains in English such as SQuAD [12] (extractive MRC) on Wikipedia articles, RACE [4] (multiple choices MRC) on High school students English Exams domain, and NarrativeQA [7] (abstractive MRC) on books and stories domain.”
BackgroundTeaching Machines to Read, Answer and Explain
Cui, Y., Liu, T., Che, W., Chen, Z., & Wang, S. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3156789 · Cited by 14
FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain
Labrak, Y., Bazoge, A., Dufour, R., Daille, B., Gourraud, P., Morin, E., & Rouvier, M. (2022) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2022.louhi-1.5 · Cited by 48
Sepúlveda-Torres, R., Vicente, M., Saquete, E., Lloret, E., & Palomar, M. (2023) · Data & Knowledge Engineering · doi:10.1016/j.datak.2023.102176 · Cited by 16
CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Models
Niu, C., Li, C., Ng, V., & Luo, B. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00055 · Cited by 11
“[53]), or even few-shot GPT-3 (common sense reasoning [54]).”
BackgroundLearning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?
Qin, C., Joty, S., Li, Q., & Zhao, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.659 · Cited by 22
Ekram, S. M. S., Rahman, A. A., Altaf, M. S., Islam, M. S., Rahman, M. M., Rahman, M. M., Hossain, M. A., & Kamal, A. R. M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.186 · Cited by 35
“Another of this kind is RACE dataset (Lai et al., 2017), collected from middle school and high school English examinations in China.”
BackgroundImproving QA Generalization by Concurrent Modeling of Multiple Biases
Wu, M., Moosavi, N. S., Rücklé, A., & Gurevych, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.74 · Cited by 18
The Application of Text Entailment Techniques in COLIEE 2020
Rabelo, J. C. B., Kim, M., & Goebel, R. G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-79942-7_16 · Cited by 15
Contrastive heterogeneous graphs learning for multi-hop machine reading comprehension
Liao, J., Zhao, X. H., Li, X., Tang, J., & Ge, B. (2021) · World Wide Web · doi:10.1007/s11280-021-00980-6 · Cited by 12
“In recent years, to verify the effect of MRC models, many datasets have been developed, typically, SQuAD 1.0 [26], RACE [22], and SQuAD 2.0 [25].”
MethodInfluentialThe CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal PaddingPreprint
Fegade, P., Chen, T., Gibbons, P. B., & Mowry, T. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10221 · Cited by 17
“For batch size 128, an implementation of masked SDPA in CORA which exploits this masking performs 1.56× faster than an implementation which does not for the RACE dataset and 1.29× for the MNLI dataset.”
InfluentialHassan, E., Talaat, A. S., & Elsabagh, M. A. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01233-3 · Cited by 17
“The RoB-ERTa model performs at the cutting edge on SQuAD [41], RACE [42], and GLUE [2].”
Salehin, I., Sajib, M. T. A., Badhon, N. H., Rifat, M. S. H., Amin, N., & Moon, N. N. (2025) · Engineering Reports · doi:10.1002/eng2.70365 · Cited by 13
How Accurate are GPT-3’s Hypotheses About Social Science Phenomena?
Rosenbusch, H., Stevenson, C. E., & van der Maas, H. L. J. (2023) · Digital Society · doi:10.1007/s44206-023-00054-2 · Cited by 23
“Furthermore, this task diverges quite far from those that LLM’s are typically scored on like text translation, summarization, or simple info extraction (e.g., Akhbardeh et al., 2021; Barrault et al., 2019; Lai et al., 2017).”
Improving Machine Reading Comprehension with Contextualized Commonsense Knowledge
Sun, K., Yu, D., Chen, J., Yu, D., & Cardie, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.598 · Cited by 12
Gu, Y., Gui, X., & Li, D. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.107 · Cited by 10
Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
Xu, C., Zhou, W., Ge, T., Xu, K., McAuley, J., & Wei, F. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.172 · Cited by 16
Unified Question Generation with Continual Lifelong Learning
Yuan, W., Yin, H., He, T., Chen, T., Wang, Q., & Cui, L. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511930 · Cited by 15
“For example, datasets that belong to answer-extraction QG (e.g. SQuAD) might be more similar to answer-abstraction QG datasets (e.g. NarrativeQA) than to multi-choice QG datasets (e.g. RACE).”
MethodInfluentialA Comparison of Transformer-Based Language Models on NLP Benchmarks
Greco, C. M., Tagarelli, A., & Zumpano, E. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08473-7_45 · Cited by 11
SGPT: Semantic graphs based pre-training for aspect-based sentiment analysis
Qian, Y., Chen, C., Wang, Z., Xiao, R., & Tang, H. (2023) · World Wide Web · doi:10.1007/s11280-022-01123-1 · Cited by 15
“Pre-trained language models learn contextualized word representations on large-scale text corpus through a self-supervised learning method, which are fine-tuned on downstream tasks and can obtain the state-of-the-art (SOTA) performance, and are pervasive and have made a tremendous impact in many NLP fields such as rea…”
BackgroundNer4Opt: Named Entity Recognition for Optimization Modelling from Natural Language
Dakle, P. P., Kadıoğlu, S., Uppuluri, K., Politi, R., Raghavan, P., Rallabandi, S. K., & Srinivasamurthy, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-33271-5_20 · Cited by 30
Development of system for generating questions, answers, distractors using transformers
Barlybayev, A., & Matkarimov, B. T. (2024) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v14i2.pp1851-1863 · Cited by 17
“RACE is a comprehensive reading comprehension dataset gathered from English exams meant for middle and high schoolers [72].”
BackgroundA Unified Machine Reading Comprehension Framework for Cohort Selection
Xiong, Y., Peng, W., Chen, Q., Huang, Z., & Tang, B. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3095478 · Cited by 13
Hierarchical Multi-Task Learning for Diagram Question Answering with Multi-Modal Transformer
Yuan, Z., Xiao, P., Wu, X. Y., & Xu, C. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475255 · Cited by 11
“In the pre-training phase, RACE [19], ARC Challenge [7], and VQA abstract scenes[1] datasets are utilized to train the transformer block layers.”
MethodJi, Y., Chen, L., Dou, C., Ma, B., & Li, X. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.96 · Cited by 14
“With the help of various large-scale reading comprehension datasets (Hermann et al., 2015; Hill et al., 2015; Trischler et al., 2016; Rajpurkar et al., 2016; Lai et al., 2017; Rajpurkar et al., 2018), neural networks have achieved a great success on MRC in recent years.”
BackgroundFew Shot Dialogue State Tracking using Meta-learning
Dingliwal, S., Gao, B., Agarwal, S., Lin, C., Chung, T., & Hakkani-Tür, D. Z. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.148 · Cited by 22
Improving Machine Reading Comprehension via Adversarial TrainingPreprint
Yang, Z., Cui, Y., Che, W., Liu, T., Wang, S., & Hu, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03614 · Cited by 15
“These datasets focus on different types of RC tasks, such as cloze-style RC (Hermann et al., 2015; Hill et al., 2016), span-based RC with or without unanswerable questions (Rajpurkar et al., 2016, 2018) and multi-choice RC (Lai et al., 2017).”
BackgroundInfluentialFreeLB: Enhanced Adversarial Training for Language UnderstandingPreprint
Zhu, C., Cheng, Y., Gan, Z., Sun, S., Goldstein, T., & Liu, J. (2019) · arXiv (Cornell University) · Cited by 74
Multi-task Learning with Multi-head Attention for Multi-choice Reading ComprehensionPreprint
Wan, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04992 · Cited by 13
“RACE [Lai et al., 2017] is a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions.”
A Survey On Neural Word EmbeddingsPreprint
Sezerer, E., & Tekır, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.01804 · Cited by 36
More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question AnsweringPreprint
Bai, Y., & Wang, D. (2021) · arXiv.org · doi:10.48550/arxiv.2109.12264 · Cited by 14
“RACE[42] 2017 EN English Exams English Exams English Exams English Exams Multi.”
BackgroundMethodInfluentialAdding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
Miller, E. (2024) · arXiv · doi:10.48550/arxiv.2411.00640 · Cited by 9
“For instance, DROP[6], QuAC[5], RACE[13], and SQuAD[17] are reading-comprehension evals having multiple related questions about independently selected passages of text, and multilingual evals such as MGSM[18] consist of the same question translated into many languages.”
BackgroundWYWEB: A NLP Evaluation Benchmark For Classical Chinese
Zhou, B., Chen, Q., Zhong, X., Zhong, X., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.204 · Cited by 33
Mucciaccia, S. S., Paixão, T., Mutz, F. W., Souza, A. F. D., Badue, C., & Oliveira-Santos, T. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5801 · Cited by 7
XCMRC: Evaluating Cross-Lingual Machine Reading Comprehension
Liu, P., Deng, Y., Zhu, C., & Hu, H. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32233-5_43 · Cited by 10
Enhancing Multiple-choice Machine Reading Comprehension by Punishing Illogical Interpretations
Ju, Y., Zhang, Y., Tian, Z., Liu, K., Cao, X., Zhao, W., Li, J., & Zhao, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.295 · Cited by 10
“We demonstrate the effectiveness of our method through experiments on three representative datasets: RACE (Lai et al., 2017), MULTIRC (Khashabi et al., 2018) and DREAM (Sun et al., 2019).”
BackgroundMethodInfluentialOPERA: Operation-Pivoted Discrete Reasoning over Text
Zhou, Y., Bao, J., Duan, C., Sun, H., Liang, J., Wang, Y., Zhao, J., Wu, Y., He, X., & Zhao, T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.119 · Cited by 9
Improving Pre-trained Language Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_3 · Cited by 10
Zero-Shot Learners for Natural Language Understanding via a Unified Multiple-Choice Perspective
Wang, J., Yang, P., Gan, R., Zhang, Y., Zhang, J., & Sakai, T. (2023) · IEEE Access · doi:10.1109/access.2023.3343123 · Cited by 14
Directional attention weaving for text-grounded conversational question answering
Li, R., Jiang, Z., Wang, L., Lu, X., & Zhao, M. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.01.056 · Cited by 9
Read and Reason with MuSeRC and RuCoS: Datasets for Machine Reading Comprehension for Russian
Fenogenova, A., Mikhailov, V., & Shevelev, D. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.570 · Cited by 13
“Recently, the paradigm has shifted towards a more complex setting, where the model is tested to infer the answer based on interpretive analysis of text (Lai et al., 2017), reasoning over multiple documents (Yang et al.”
BackgroundComparative Analysis of Transformer based Language Models
Pathak, A. (2021) · doi:10.5121/csit.2021.110111 · Cited by 11
“On the contrary, it performs notably dull on natural language inference (NLI) tasks such as ANLI [26], WIC [27] and on many reading comprehension tasks such as DROP [28], MultiRC [29], RACE [13] and QuAC [30].”
BackgroundYan, R., Liao, W., Cui, J., Zhang, H., Hu, Y., & Zhao, D. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449991 · Cited by 15
“choice [19], span prediction [31], as well as other free forms such as narratives [26].”
BackgroundAlignment-Based Graph Network for Judicial Examination Task
Wu, J., & Luo, X. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-82153-1_32 · Cited by 10
Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments
Ivgi, M., Carmon, Y., & Berant, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.544 · Cited by 32
Multilingual Conceptual Coverage in Text-to-Image Models
Saxon, M., & Wang, W. Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.266 · Cited by 24
Weakly-Supervised Questions for Zero-Shot Relation Extraction
Najafi, S., & Fyshe, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.224 · Cited by 12
Biomedical Question Answering: A Comprehensive Review.Preprint
Jin, Q., Yuan, Z. P., Xiong, G., Yu, Q., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2021) · arXiv (Cornell University) · Cited by 11
Metadata
- Authors
- Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 785-794
- OpenAlex
- W2606964149
- Cited by
- 1,044
- References
- 19
- Citations by year
- 2026: 37 · 2025: 75 · 2024: 86 · 2023: 145 · 2022: 136 · 2021: 216 · 2020: 169 · 2019: 138 · 2018: 38 · 2017: 3
- FWCI
- 42.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750, FA8750-12-2-0342)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, machine reading comprehension, benchmark dataset, English examinations, Chinese students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.