RACE: Large-scale ReAding Comprehension Dataset From Examinations
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task.Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning.In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%).We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [112]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, to our knowledge, most of the existing datasets are not of direct relevance to teaching and learning, except for RACE (Lai, Xie, Liu, Yang, & Hovy, 2017) and LearningQ (Chen, Yang, Hauff, & Houben, 2018).”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Lai, Xie, Liu, Yang, & Hovy, 2017
The cited source says what the claim attributes to it.
In the citing paper
“On RACE-h and RACE-m (Lai et al., 2017), Chinchilla greatly outperforms Gopher , improving accuracy by more than 10% in both cases-see Table 7.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Lai et al., 2017
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We consider only the case of human-readable prompts to large models, which contrasts with methods such as prompt-tuning [16,20], which learns a continuous prompt embedding, and auto-prompting [33], which attempts to search for prompts from scratch, both of which require a training step.”
From Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We choose word embedding size d = 100 and use the 100-dimensional Glove word embedding (Pennington et al., 2014) as embedding initialization.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Instead of evaluating the accuracy, researchers need to use F1 score, BLEU (Papineni et al., 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the questions come without candidate spans.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 8×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
MethodResultInfluentialTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 3×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 1×
“, 2016) and recently proposed TriviaQA (Joshi et al., 2017).”
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578 · cited 2×
“, 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the”
BackgroundSelf-citationNewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 11×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodResultInfluentialMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610 · cited 4×
“Although efforts have been made with a similar motivation, including the MCTest dataset created by (Richardson et al., 2013) (containing 500 passages and 2000 questions) and several others (Peñas et al.”
BackgroundMethodInfluentialA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491 · cited 11×
“However, both require limited reasoning ability (Chen et al., 2016).”
BackgroundMethodInfluential
Cited by
Items 601–700 of 1,038
NUT-RC: Noisy User-generated Text-oriented Reading Comprehension
Huang, R., Zou, B., Hong, Y., Zhang, W., Aw, A. T., & Zhou, G. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.242 · Cited by 8
“, 2016) or multiple-choice (Richardson et al., 2013; Lai et al., 2017).”
BackgroundEviDR: Evidence-Emphasized Discrete Reasoning for Reasoning Machine Reading Comprehension
Zhou, Y., Bao, J., Sun, H., Liang, J., Wu, Y., He, X., Zhou, B., & Zhao, T. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88480-2_35 · Cited by 5
Question Difficulty Estimation Based on Attention Model for Question Answering
Song, H.-J., Yoon, S.-H., & Park, S.-B. (2021) · Applied Sciences · doi:10.3390/app112412023 · Cited by 5
Word2Course: Creating Interactive Courses from as Little as a Keyword
Foucher, S., Pascual, D., Richter, O., & Wattenhofer, R. P. (2022) · International Conference on Computer Supported Education · doi:10.5220/0011064700003182 · Cited by 5
“Notable research in that field was conducted by Zhaopeng Qiu et al. (Qiu et al., 2020) who introduced the EDGE framework that is the current state-of-theart on the RACE dataset (Lai et al., 2017) for reading comprehension.”
BackgroundInfluentialDiscourse-Aware Graph Networks for Textual Logical Reasoning
Huang, Y., Liu, L., Xu, K., Fang, M., Lin, L., & Liang, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3280178 · Cited by 9
“Ontheotherhand,recentadvancesintransformer-based pre-trainedlanguagemodels(PLMs)[23],[24],[25],[26], [27]havewitnessedgreatsuccessinextensivenaturallanguage tasks[28],[29],[30],butfaillogicalreasoning[2],[3],[4].”
Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding
Bhardwaj, R., Saha, A., Hoi, S. C. H., & Poria, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.455 · Cited by 13
QASports: A Question Answering Dataset about Sports
Jardim, P. C., Moraes, L. M. P., & Ciferri, C. D. D. A. (2023) · doi:10.5753/dsw.2023.233602 · Cited by 5
AIComprehend: An Adaptive Reading Comprehension Learning Platform Using Machine Learning
Mondia, D. J., Naranjo, K., Galamay, S. T., & Tiglao, N. (2023) · International Symposium on Networks, Computers and Communications · doi:10.1109/isncc58260.2023.10323847 · Cited by 2
“For this study, the team used the ReAding Comprehension dataset from Examinations (RACE) dataset [11] which contains 28,000 text passages and around 100,000 questions made by human experts (English instructors) as reference for the reading comprehension items.”
MethodAI-Driven Assessment in Iranian High School English Classes
Marandi, S. S., & Hosseini, S. (2024) · International and National Conference on E-Learning and E-Teaching (ICeLeT) · doi:10.1109/icelet62507.2024.10493060 · Cited by 9
“The sentences utilized in the generated questions were selected from the Race dataset [6].”
MethodPrompt Learning for Few-Shot Question Answering via Self-Context Data Augmentation
Qiu, J.-Q., Zhang, C.-Y., & Chen, C. L. P. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3483201 · Cited by 5
“QA is one of the most challenging tasks in NLP. Based on the formats of the questions and answers, QA tasks can be categorized into four types [22]: cloze style [23], multiple choice [24], span prediction [1], and free-form answer [25].”
Self Data Augmentation for Open Domain Question Answering
Zhang, Q., Zheng, M., Chen, S., Liu, H., & Fang, M. (2025) · ACM Transactions on Information Systems · doi:10.1145/3707449 · Cited by 9
“Lai et al. [31] collected questions from English examination in China.”
Self-citationSCDE: Sentence Cloze Dataset with High Quality Distractors From Examinations
Kong, X., Gangal, V., & Hovy, E. H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.502 · Cited by 11
Self-citationLuu, S. T., Nguyen, K. V., Nguyen, A. G.-T., & Nguyen, N. L.-T. (2021) · International Conference on Communications and Electronics (ICCE) · doi:10.1109/icce48956.2021.9352127 · Cited by 5
“This model received the best performance on the RACE dataset, which is 50.4% [6].”
InfluentialAct-Aware Slot-Value Predicting in Multi-Domain Dialogue State Tracking
Su, R., Wu, T.-W., & Juang, B.-H. F. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-138 · Cited by 7
“For categorical slots, we implement multiple-choice RC [19] to choose from a predefined set of limited values, e.”
MethodJob and Employee Embeddings: A Joint Deep Learning Approach
Liu, H., & Ge, Y. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3180593 · Cited by 8
“For example, the popular RoBERTa-base model with 110 million parameters is trained on five corpora with 160GB of uncompressed text [23], and it has achieved state-of-the-art results on tasks such as GLUE [31], RACE [32] and SQuAD [33].”
MethodSelf-citationLeveraging greater relations for improving multi-choice reading comprehension
Yan, H., Liu, L., Feng, X., & Huang, Q. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07561-2 · Cited by 5
“However, [19, 21, 36] pointed out that most questions require multi-sentence reasoning in the multi-choice reading comprehension task.”
BackgroundSultan, M. A., Sil, A., & Florian, R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.247 · Cited by 8
“…any confounding ef-1 Target Test B io ASQ (Tsatsaronis et al., 2015) 1,504 DROP (Dua et al., 2019) 1,503 D uo RC (Saha et al., 2018) 1,501 RACE (Lai et al., 2017) 674 R el E x (Levy et al., 2017) 2,948 T extbook QA (Kembhavi et al., 2017) With every training method, we train six models, each…”
Cross-Corpus Readability Compatibility Assessment for English Texts
Li, Z., Ding, H., & Zhang, S. (2023) · IEEE Access · doi:10.1109/access.2023.3315834 · Cited by 8
“567 (Lee,2022)[6], (Martinc,2021)[40], (Vajjala,2021)[10], (Imperial,2021)[42] RACE The Reading Comprehension Dataset From Examination (RACE) is a collection of reading comprehension materials from English exams for middle and high school students in China, divided into two levels based on grades: junior high and seni…”
BackgroundDual Head-wise Coattention Network for Machine Comprehension with Multiple-Choice Questions
Liu, Z., Huang, K., Huang, D., Liu, Z., & Zhao, J. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412013 · Cited by 5
“We use five multiple-choice MC datasets as the target datasets: RACE [10], DREAM [28], and SemEval-2018 Task 11 [19], OpenBookQA [17], and TOEFL [31] which are summarized in Table 2.”
MethodInfluentialIncremental BERT with commonsense representations for multi-choice reading comprehension
Li, R., Wang, L., Jiang, Z., Liu, D., Zhao, M., & Lu, X. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11197-0 · Cited by 7
“We report results on three well-known multi-choice datasets, CosmosQA [13], DREAM [29], and RACE [17], which are summarized in Table 1.”
BackgroundMethodJansen, P. A., Smith, K. J., Moreno, D., & Ortiz, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.596 · Cited by 14
“Model Parameters: We used a vanilla BERT-base-uncased (110M), as well as RoBERTA-Large (355M) pre-trained on the RACE reading comprehension benchmark (Lai et al., 2017).”
MethodNi, J., Zhu, C., Chen, W., & McAuley, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.09492 · Cited by 5
Large Scale Question Answering using Tourism DataPreprint
Contractor, D., Shah, K., Partap, A., Mausam, & Singla, P. (2019) · arXiv.org · doi:10.48550/arxiv.1909.03527 · Cited by 11
“, 2017) Web snippets Span 25-46 N N RACE (Lai et al., 2017) Passages on topics Choices 350 Y N OpenBookQA (Mihaylov et al.”
BackgroundFinding Generalizable Evidence by Learning to Convince Q&A ModelsPreprint
Perez, E., Karamcheti, S., Fergus, R., Weston, J., Kiela, D., & Cho, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05863 · Cited by 15
“Multiple-choice QA: RACE and DREAM For our experiments, we use RACE (Lai et al., 2017) and DREAM (Sun et al.”
BackgroundMethodInfluentialAssessing the Benchmarking Capacity of Machine Reading Comprehension DatasetsPreprint
Sugawara, S., Stenetorp, P., Inui, K., & Aizawa, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09241 · Cited by 5
“Although s ′ 3 and s ′ 7 with RACE were exceptions, their learning did not converge within the spec-ified number of epochs.”
MethodInfluentialDeep Contextual Embeddings for Address Classification in E-commercePreprint
Mangalgi, S., Kumar, L., & Tallamraju, R. B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03020 · Cited by 15
“…obtains state-of-the-art performance on several NLP tasks like question answering, classification etc. Liu et al. [18] propose RoBERTa which is a robust and optimized version of pre-training a BERT based model and achieve new state-of-the-art results on GLUE[35], RACE[15] and SQuAD[27] datasets.”
BackgroundExplainable Natural Language Reasoning via Conceptual Unification.Preprint
Valentino, M., Thayaparan, M., & Freitas, A. (2020) · arXiv (Cornell University) · Cited by 5
Understanding Emergent Abilities of Language Models from the Loss Perspective
Du, Z., Zeng, A., Dong, Y., & Tang, J. (2024) · arXiv · doi:10.48550/arxiv.2403.15796 · Cited by 5
“We use TriviaQA [31] for English.”
Assessing Distractors in Multiple-Choice Tests
Raina, V., Liusie, A., & Gales, M. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.2 · Cited by 9
Fernando, A., & Ranathunga, S. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02520-4 · Cited by 6
“The ablation study results for considering each linguistic entity along with combinations of them, on the LEM strategy are available in Tables 14, 15, 16 in the Appendix 1.”
Jain, Y., & Singh, C. (2019) · doi:10.18653/v1/d19-6008 · Cited by 4
“We have pretrained the model with Race Dataset (Lai et al., 2017) for 1 epoch.”
MethodYu, X., Liu, Q., He, S., Liu, K., Liu, S., Zhao, J., & Zhou, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-84186-7_14 · Cited by 5
“, 2019), or inferring answers based on pieces of evidence from a given document, which is also referred to as non-extractive MRC, including multiple-choice MRC (Richardson et al., 2013; Lai et al., 2017), open domain question answering (Dhingra et al.”
BackgroundInfluentialSyntax-Aware Multi-Spans Generation for Reading Comprehension
Zhang, Z., Zhang, Y., & Zhao, H. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3138679 · Cited by 10
Ask to Understand: Question Generation for Multi-hop Question Answering
Li, J., Ren, M., Gao, Y., & Yang, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-6207-5_2 · Cited by 7
Self-citationTIE: Topological Information Enhanced Structural Reading Comprehension on Web Pages
Zhao, Z., Chen, L., Cao, R., Xu, H., Chen, X., & Yu, K. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.132 · Cited by 8
Multi-Turn and Multi-Granularity Reader for Document-Level Event Extraction
Yang, H., Chen, Y., Liu, K., Zhao, J., Zhao, Z., & Sun, W. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3542925 · Cited by 6
“In recent years, the MRC task has been widely investigated since the release of large-scale corpora [14, 16, 27, 38].”
BackgroundHop, Union, Generate: Explainable Multi-hop Reasoning without Rationale Supervision
Zhao, W., Chiu, J., Cardie, C., & Rush, A. M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1001 · Cited by 13
MRHF: Multi-stage Retrieval and Hierarchical Fusion for Textbook Question Answering
Zhu, P., Wang, Z., Okumura, M., & Yang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-53308-2_8 · Cited by 6
Toward Unified Controllable Text Generation via Regular Expression Instruction
Zheng, X. W., Lin, H., Han, X., & Sun, L. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.1 · Cited by 13
Tomikawa, Y., & Uto, M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-64315-6_27 · Cited by 5
Capraro, V., Paolo, R. D., & Pizziol, V. (2025) · Scientific Reports · doi:10.1038/s41598-025-01715-7 · Cited by 14
“Previous research has introduced benchmarks for evaluating LLMs’ abilities in various domains, including reading comprehension 40,41 , mathematical abilities 42 , reasoning about physical common sense 43 , numerical abilities over financial data 44 , and common-sense reasoning 45,46 .”
ECNU at SemEval-2018 Task 11: Using Deep Learning Method to Address Machine Comprehension Task
Sheng, Y., Lan, M., & Wu, Y. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1175 · Cited by 4
“Inspired by (Lai et al., 2017), we use the state-of-art Gated-Attention Reader which performs well on several datasets.”
MethodPawate, A. G., Madhavan, V., & Chandak, D. (2020) · doi:10.18653/v1/2020.textgraphs-1.12 · Cited by 4
“By improving key hyper-parameter choices, (Liu et al., 2019) we were able to show that BERT can match and even improve on a number of the models that were proposed after it, achieving SOTA performance on GLUE, SQuAD and RACE (Lai et al., 2017).”
MethodEnhanced Few-Shot Learning with Multiple-Pattern-Exploiting Training
Zeng, J., Jiang, Y., Wu, S., & Li, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88483-3_31 · Cited by 4
Interactive Gated Decoder for Machine Reading Comprehension
Cui, Y., Che, W., Yang, Z., Liu, T., Qin, B., Wang, S., & Hu, G. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3501399 · Cited by 4
“Machine reading comprehension is a task to read and comprehend given articles and answer the questions based on them, which consists of various sub-genres, such as cloze-style MRC [11, 12], span-extraction MRC [24], open-domain MRC [3], multiple-choice MRC [17, 25], and so on.”
BackgroundSurvey of Machine Reading Comprehension Models and its Evaluation Metrics
Therasa, M., & Mathivanan, G. (2022) · International Conference on Computing Methodologies and Communication (ICCMC) · doi:10.1109/iccmc53470.2022.9754070 · Cited by 6
Deep human answer understanding for natural reverse QA
Yao, R., Hou, L., Yang, L., Gui, J., & Wu, O. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109625 · Cited by 6
Putra, R. P., & Setiawan, E. B. (2022) · International Conference on Advanced Creative Networks and Intelligent Systems (ICACNIS) · doi:10.1109/icacnis57039.2022.10054870 · Cited by 5
CoLISA: Inner Interaction via Contrastive Learning for Multi-choice Reading Comprehension
Dong, M., Zou, B., Li, Y., & Yu, H. Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28244-7_17 · Cited by 7
Meng, X., Song, Y., Bai, Q., & Wang, T. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110796 · Cited by 6
Few-shot Adaptation Works with UnpredicTable Data
Chan, J. S., Pieler, M., Jao, J., Scheurer, J., & Perez, E. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.102 · Cited by 8
Disentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformers
Zhong, W., Ma, T., Wang, J., Yin, J., Zhao, T., Lin, C.-Y., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.480 · Cited by 5
How Long Is Enough? Exploring the Optimal Intervals of Long-Range Clinical Note Language Modeling
Cahyawijaya, S., Wilie, B., Lovenia, H., Zhong, H., Zhong, M., Ip, Y.-Y. N., & Fung, P. (2022) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2022.louhi-1.19 · Cited by 6
High-efficiency Device-Cloud Collaborative Transformer Model
Jiang, P., Ke, X., Li, C., & Zhou, Y. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00214 · Cited by 9
“We consider tasks include the GLUE benchmark for language understanding [44], 7 widely used text (sentiment / topic) classification tasks (IMDB, AD, DBpe-dia, Yelp-2, Yelp-5, Amazon-2, Amazon-5), SQuAD question answering task, and the RACE reading comprehension dataset [45].”
BackgroundMethodXu, F., Liu, J., Lin, Q., Zhao, T., Zhang, J., & Zhang, L. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3317254 · Cited by 8
“RACE dataset [7] is specially designed to improve the reading skills for both middle school and high school students, which contains about 9.7K examples.”
Investigating better context representations for generative question answering
Francis, S., & Moens, M. (2023) · Information Retrieval · doi:10.1007/s10791-023-09420-7 · Cited by 6
“Multichoice style MRC such as ReAding Comprehension dataset from Examination (RACE) (Lai et al., 2017) requires models to choose a correct answer from a set of answer candidate options.”
In-Context Demonstration Selection with Cross Entropy Difference
Iter, D., Pryzant, R., Xu, R., Wang, S., Liu, Y., Yi‐chong, X., & Zhu, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.81 · Cited by 29
A Multiple Granularity Co-Reasoning Model for Multi-choice Reading Comprehension
Miao, H., Liu, R., & Gao, S. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852176 · Cited by 4
“[1] refined two deep neural network models: Stanford Attentive Reader (SAR) [5] and Gated-Attention Readers (GA) [6] as baselines for this dataset.”
BackgroundMethodInfluentialConstrained Fact Verification for FEVER
Pratapa, A., Jayanthi, S. M., & Nerella, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.629 · Cited by 8
“Firstly, the reasoning models in Clark et al. (2020) were first trained on the RACE multi-choice question answering dataset (Lai et al., 2017) and then fine-tuned on the RuleTaker dataset.”
MethodCompressing Transformer-Based Semantic Parsing Models using Compositional Code Embeddings
Prakash, P., Shashidhar, S. K., Zhao, W., Rongali, S., Khan, H., & Kayser, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.423 · Cited by 7
Multi-Perspective Context Aggregation for Semi-supervised Cloze-style Reading ComprehensionPreprint
Wang, L., Li, S., Zhao, W., Shen, K., Sun, M., Jia, R., & Liu, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.06289 · Cited by 9
“Moreover, given the same amount of text, CLOTH consistently beats RACE and RACE consistently beats Wikipedia.”
MethodInfluentialAmazonQA: A Review-Based Question Answering TaskPreprint
Gupta, M., Kulkarni, N., Chanda, R., Rayasam, A., & Lipton, Z. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04364 · Cited by 38
“, 2016] Crowd-Sourced Wiki Span of words 100K 536 RACE [Lai et al., 2017] Crowd-Sourced English Exams Multiple choice 97l 28K ARC [Clark et al.”
BackgroundReasoning-Driven Question-Answering for Natural Language UnderstandingPreprint
Khashabi, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04926 · Cited by 13
“, 2015), suffer from even lower percentage of such questions (12% and 2% respectively (Lai et al., 2017)).”
BackgroundMethodInfluentialKnowledge Guided Named Entity Recognition for BioMedical TextPreprint
Banerjee, P., Pal, K. K., Devarakonda, M. V., & Baral, C. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03869 · Cited by 8
“External knowledge can also be about commonsense or science, such as in Question Answering tasks (RACE (Lai et al., 2017), OpenBookQA (Mihaylov et al., 2018), SocialIQA (Sap et al., 2019)) etc.”
BackgroundTesting pre-trained Transformer models for Lithuanian news clusteringPreprint
Stankevičius, L., & Lukoševičius, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03461 · Cited by 7
Deepening Hidden Representations from Pre-trained Language ModelsPreprint
Yang, J., & Zhao, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.01940 · Cited by 25
“They include question answering (Rajpurkar et al., 2018; Lai et al., 2017), natural language inference (Williams et al., 2018; Bowman et al., 2015), named entity recognition (Tjong Kim Sang and De Meulder, 2003), sentiment analysis (Socher et al., 2013) and semantic textual similarity (Cer et al.,…”
BackgroundReCO: A Large Scale Chinese Reading Comprehension Dataset on OpinionPreprint
BingningWang, Yao, T., Zhang, Q., Xu, J., & Wang, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12146 · Cited by 4
“Other multiple choices MRC datasets including MCScript (Ostermann et al. 2018) that requires the system to understand the script of daily events, and RACE (Lai et al. 2017) where the questions are collected from the English exams of Chinese students.”
BackgroundMethodReading Comprehension as Natural Language Inference: A Semantic AnalysisPreprint
Mishra, A., Patel, D., Vijayakumar, A., Li, X., Kapanipathi, P., & Talamadupula, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01713 · Cited by 4
“Drawing inspiration from the prior work in the area (Clark et al., 2018; Demszky et al., 2018; Trivedi et al., 2019), we use rule-based conversion to create an NLI version of the largest available RC dataset - RACE (Lai et al., 2017).”
MethodInfluentialCan Transformers Reason About Effects of Actions?Preprint
Banerjee, P., Baral, C. R., Luo, M., Mitra, A., Pal, K. K., Son, T. C., & Varshney, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09938 · Cited by 15
“Training & Testing All our experiments are conducted using Roberta-large, which is finetuned on RACE dataset (Lai et al. 2017) using the hyperparameters mentioned in (Liu et al. 2019) as recommended by Clark, Tafjord, and Richardson.”
MethodFQuAD2.0: French Question Answering and knowing that you know nothingPreprint
Heinrich, Q., Viaud, G., & Belblidia, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.13209 · Cited by 9
“BoolQ (Clark et al., 2019) highlights the surprising difficulty of answering yes/no questions, while RACE (Lai et al., 2017) studies multiple choice Question Answering.”
BackgroundCOVIDRead: A Large-scale Question Answering Dataset on COVID-19Preprint
Saikh, T., Sahoo, S. K., Ekbal, A., & Bhattacharyya, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09321 · Cited by 7
“These two are the standard metrics for evaluating Span-of-Texts based QA Systems, and widely used in various notable QA Systems [31, 11, 32, 30].”
BackgroundMethodImproving Controllability of Educational Question Generation by Keyword ProvisionPreprint
Chan, Y.-H., Chung, H.-L., & Fan, Y.-C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.01012 · Cited by 4
“The case studies are selected from the test set of EGQ-RACE (Jia et al. 2020).”
MethodInfluentialPILs of Knowledge: A Synthetic Benchmark for Evaluating Question Answering Systems in Healthcare
Lunardi, R., Soprano, M., Coppola, P., Mea, V. D., Mizzaro, S., & Roitero, K. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730283 · Cited by 6
“The evaluation process consists of prompting each model with the question and requiring it to select the correct answer from the four possible choices, exactly as done to assess the effectiveness of LLMs in popular benchmarks [11, 13, 21, 32, 35, 60, 63].”
Named Entity Recognition and News Article Classification: A Lightweight Approach
Katranis, I., Troussas, C., Krouska, A., Mylonas, P., & Sgouropoulou, C. (2025) · IEEE Access · doi:10.1109/access.2025.3605709 · Cited by 4
“The resulting ALBERT models set new state-of-the-art results on the GLUE, versions 1.1 and 2.0 of SQuAD ( [35], [36]) and RACE [37] benchmarks.”
InfluentialMulti-choice machine reading comprehension benchmark datasets: A survey
Foolad, S., Kiani, K., & Rastgoo, R. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.114499 · Cited by 4
Multiple-Choice Question Answering Models for Automatic Depression Severity Estimation
Gabín, J., Pérez, A., & Parapar, J. (2021) · XoveTIC Conference (XoveTIC) · doi:10.3390/engproc2021007023 · Cited by 3
“We also used RACE (Large-scale ReAding Comprehension Dataset From Examinations) [7] and SWAG (Large-Scale Adversarial Dataset for Grounded Commonsense Inference) [8], two general-purpose multiple-choice question answering datasets.”
MethodFeeding What You Need by Understanding What You Learned
Wang, X., Liu, B., Xu, F., Long, B., Tang, S., & Wu, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.403 · Cited by 7
Lite Unified Modeling for Discriminative Reading Comprehension
Zhao, Y., Zhao, H., Shen, L., & Zhao, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.594 · Cited by 5
Context Reasoning Attention Network: Generating Plausible Distractors for Multi-choice Questions
Li, M., Wang, L., Liu, H., Wang, W., & Zheng, H.-T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15934-3_49 · Cited by 4
Self-citationAlignment Relation is What You Need for Diagram Parsing
Zhang, X. M., Zhang, L., Hu, X., Liu, J., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3374511 · Cited by 5
“(2) ISAAQ [27] using a pre-trained ResNet-101 to extract the features of all elements on the diagram, and the textual extractor is pre-trained on RACE [45], ARC [46] and OpenBookQA [47].”
MethodSubgraph-Based Attention Network for Multi-Hop Question Answering
Gong, C., Wei, Z., Wang, X., Wang, R., Li, Y., & Zhu, P. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650851 · Cited by 4
“In contrast to single-hop QA [1]–[3], which can be answered only through a single paragraph, multi-hop QA [4]–[6] requires the model to engage in intricate correlation and reasoning across multiple relevant paragraphs, making it more challenging.”
BackgroundSyncIntellects: Orchestrating LLM Inference with Progressive Prediction and QoS-Friendly Control
Lin, X., Zhang, Z., Yue, P., Li, H., Zhang, J., Fan, B., Su, H., & Gong, X. (2024) · International Workshop on Quality of Service · doi:10.1109/iwqos61813.2024.10682949 · Cited by 7
“The evaluation is conducted based on 4 datasets: Natural Questionss [30] dataset, TriviaQA dataset [41], SQuAD dataset [42], Hotpot dataset [43], MMLU [44], RACE [45], and Winogrande [46].”
MethodViMMRC 2.0 — Enhancing Machine Reading Comprehension on Vietnamese Literature Text
Luu, S. T., Hoang, K. T., Pham, T. Q., Nguyen, K. V., & Nguyen, N. L.-T. (2025) · International Journal of Asian Language Processing · doi:10.1142/s2717554525500043 · Cited by 8
Benchmarking AI: Toward Inclusive Evaluation of Language Models
Jovanović, M., & Campbell, M. (2025) · Computer · doi:10.1109/mc.2025.3575880 · Cited by 1
An LSTM model for cloze-style machine comprehension
Wang, S., & Jiang, J. (2018) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 3
Liu, C., & Yu, D. (2019) · doi:10.18653/v1/d19-6012 · Cited by 3
Aksu, T., Liu, Z., Kan, M., & Chen, N. F. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.14 · Cited by 3
“Corpora on reading comprehension are largely limited to written text, e.g. , SQuAD (Rajpurkar et al., 2018b), MARCO (Nguyen et al., 2016), RACE (Lai et al., 2017), TriviaQA (Joshi et al., 2017) and many others (Hermann et al., 2015; Hill et al., 2016; Richardson et al., 2013; Kocisk´y et al., 2017;…”
Answering Fill-in-the-Blank Questions in Portuguese with Transformer Language Models
Oliveira, H. G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86230-5_58 · Cited by 3
Context-guided Triple Matching for Multiple Choice Question Answering
Yao, X., Ma, J., Hu, X., Liu, J., Wang, H., Huang, G., Yang, J., & Li, W. (2023) · IEEE Smart World Congress (SWC) · doi:10.1109/swc57546.2023.10448677 · Cited by 6
“…and Levy], or identifying ( selecting ) the most suitable answers to questions from a set of candidates, known as multiple choice question answering (MCQA) [Duan, Huang, and Wu(2021)], [Li et al.(2021)Li, Jiang, Wang, Lu, Zhao, and Chen], [Zhang et al.(2020a)Zhang, Zhao, Wu, Zhang, Zhou, and Zhou].”
BackgroundMethodCascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models
Malaviya, C., Bhatia, S., & Yatskar, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.438 · Cited by 10
Multi-hop question answering over incomplete knowledge graph with abstract conceptual evidence
Sun, Q., Zhang, C., Hu, Z., Jin, Z., Yu, J., & Liu, L. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04849-1 · Cited by 6
Afzaal, M., Nouri, J., & Aayesha, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42682-7_2 · Cited by 6
“World Knowledge” in Multiple Choice Reading Comprehension
Liusie, A., Raina, V., & Gales, M. (2023) · Fact Extraction and VERification Workshop (FEVER) · doi:10.18653/v1/2023.fever-1.5 · Cited by 10
A Two-Stage Distillation Method: Teach Model to Answer Questions After Comprehending the Document
Sun, R., & Jian, P. (2023) · International Conference on Asian Language Processing · doi:10.1109/ialp61005.2023.10337099 · Cited by 3
“The MRC tasks can be roughly divided into several categories, such as multi-choice [11, 12], span prediction [13, 14, 6], free form [15], knowledge-based MRC [16], multi-passage MRC [17], MRC with unanswerable questions [18, 19] and so on.”
BackgroundGuo, K., Diefenbach, D., Gourru, A., & Gravier, C. (2023) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai59109.2023.00032 · Cited by 12
“RoBERTa achieves better performance than BERT in many benchmarks like GLUE [30], SQuAD [10] and RACE [31], which explains this choice over BERT base or large models.”
MethodDIMN: Dual Integrated Matching Network for multi-choice reading comprehension
Wei, Q., Ma, K., Liu, X., Ji, K., Yang, B., & Abraham, A. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107694 · Cited by 3
Deen, M. M. S. A., Pielka, M., Hees, J., Abdou, B. S., & Sifa, R. (2023) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci52147.2023.10371891 · Cited by 5
“They utilized the RACE dataset [4], which is a large-scale reading comprehension dataset consisting of questions and answers from English exams for Chinese students.”
MethodWang, M., He, X., Liu, L., Fang, Q.-M., Zhang, M., Chen, H., & Liu, Y. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3368288 · Cited by 5
Enhancing Task Performance in Continual Instruction Fine-tuning Through Format Uniformity
Tan, X., Cheng, L., Qiu, X., Shi, S., Cheng, Y., Chu, W., Xu, Y., & Qi, Y. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657920 · Cited by 7
Hierarchical Answer Selection Framework for Multi-passage Machine Reading Comprehension
Li, Z., Xu, J., Lan, Y., Guo, J., Feng, Y., & Cheng, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01012-6_8 · Cited by 3
“At the outset, MRC task dataset was cloze task [2, 3], and then was multiple-choice exam [4] which collect multiple-choice questions from exams.”
Metadata
- Authors
- Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 785-794
- OpenAlex
- W2606964149
- Cited by
- 1,044
- References
- 19
- Citations by year
- 2026: 37 · 2025: 75 · 2024: 86 · 2023: 145 · 2022: 136 · 2021: 216 · 2020: 169 · 2019: 138 · 2018: 38 · 2017: 3
- FWCI
- 42.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750, FA8750-12-2-0342)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, machine reading comprehension, benchmark dataset, English examinations, Chinese students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.