RACE: Large-scale ReAding Comprehension Dataset From Examinations
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task.Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning.In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%).We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [112]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, to our knowledge, most of the existing datasets are not of direct relevance to teaching and learning, except for RACE (Lai, Xie, Liu, Yang, & Hovy, 2017) and LearningQ (Chen, Yang, Hauff, & Houben, 2018).”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Lai, Xie, Liu, Yang, & Hovy, 2017
The cited source says what the claim attributes to it.
In the citing paper
“On RACE-h and RACE-m (Lai et al., 2017), Chinchilla greatly outperforms Gopher , improving accuracy by more than 10% in both cases-see Table 7.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Lai et al., 2017
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We consider only the case of human-readable prompts to large models, which contrasts with methods such as prompt-tuning [16,20], which learns a continuous prompt embedding, and auto-prompting [33], which attempts to search for prompts from scratch, both of which require a training step.”
From Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We choose word embedding size d = 100 and use the 100-dimensional Glove word embedding (Pennington et al., 2014) as embedding initialization.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Instead of evaluating the accuracy, researchers need to use F1 score, BLEU (Papineni et al., 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the questions come without candidate spans.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 8×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
MethodResultInfluentialTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 3×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 1×
“, 2016) and recently proposed TriviaQA (Joshi et al., 2017).”
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578 · cited 2×
“, 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the”
BackgroundSelf-citationNewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 11×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodResultInfluentialMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610 · cited 4×
“Although efforts have been made with a similar motivation, including the MCTest dataset created by (Richardson et al., 2013) (containing 500 passages and 2000 questions) and several others (Peñas et al.”
BackgroundMethodInfluentialA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491 · cited 11×
“However, both require limited reasoning ability (Chen et al., 2016).”
BackgroundMethodInfluential
Cited by
Items 1,001–1,039 of 1,039
Wang, M., Liu, J., Gutiérrez-Basulto, V., Wang, L., & Pan, J. Z. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3588525
Exploring Adversarial Robustness of Masked Auto-Encoders
Luo, Y., Gao, J., Kuang, J., Qi, B., Chen, X., Zhao, Y., Li, F., & Liu, J. (2025) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc65474.2025.11090267
“Training with embedding-level adversarial attacks, which exploit the rich semantic information encoded in embeddings, has been shown to improve the performance more effectively than word-level attack methods [22], [23].”
GISA: Gradual Information Selection Attention for MCQ Difficulty Estimation
Ravikiran, M., Sharma, T., Bhavsar, A. V., & Saluja, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98414-3_14
Optimizing GPT-Based Distractor Generation for the Korean CSAT English ExamPreprint
Jung, C. Y., & Song, S. (2025) · Research Square · doi:10.21203/rs.3.rs-6680435/v1
AI-Based Examination Content Creation: Evaluating Large Language Models for Question Generation
Abdelhamied, S. H., Antr, K. K., Abdellah, C. M., Antr, M. K., Mahrous, Z. M., & Eldin, N. G. (2025) · Intelligent Methods, Systems, and Applications (IMSA) · doi:10.1109/imsa65733.2025.11167467
“2) RACE Dataset: The RACE dataset [14] (Reading Comprehension from Examinations) is a large-scale dataset derived from middle and high school English exams.”
Efficient Knowledge Transfer from Large to Small Language Models via Low-Overhead Query Mechanism
Ahemad, F. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761417 · Cited by 1
Optimized Domain-Specific Text Processing with Keyword Knowledge Distillation (KKD)
Biswas, M., Singhal, A., & Anish, P. R. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227844 · Cited by 1
“For instance, GLUE [1], SQUAD [2], and RACE [3], have achieved near-human levels of proficiency.”
Near Real-Time Sentiment Analysis in Cross Domain Applications
Albergaria, M., Santos, J., Oliveira, P., Faria, R., Marreiros, G., & Faria, L. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-01130-5_34
WizardEvent: Empowering Event Reasoning by Hybrid Event-Aware Data Synthesizing
Tao, Z., Chen, X., Jin, Z., Bai, X., Zhao, H., Hu, W., Tao, C., & Ma, S. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3634839
“We also compare to general test datasets: factual understanding (CSQA [34], HELLASWAG [35], OBQA [36]) and logical reasoning(ARC-Challenge [37], Winogrande [38], RACE-middle, and RACE-high [39]).”
AI-Driven Question Generation System: Benchmarking Large Language Models for Examination Systems
Abdelhamied, S. H., Antr, K. K., Abdellah, C. M., Antr, M. K., Mahrous, Z. M., & Eldin, N. G. (2025) · Student Research Conference on Applied Computing (SRC) · doi:10.1109/src65875.2025.11263698
“2) RACE Dataset: The RACE dataset [14] (Reading Comprehension from Examinations) is a large-scale dataset derived from middle and high school English exams.”
CRIT: Socratic Inquiry for Critical Thinking in LLMs
Chang, E. Y. (2025) · ACM eBooks · doi:10.1145/3749421.3749429
“Critical reading is a crucial component of critical thinking, which involves evaluating the quality and credibility of written materials, from research papers to blog posts [Paul 1990, Lai et al. 2017].”
Pustak AI: Curriculum-Aligned and Interactive Textbooks Using Large Language Models
Sharma, S., Naik, R., Gawas, T., Patil, H., & Korgaonkar, K. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-14583-3_3 · Cited by 1
Cao, R., Zhou, X., Zhang, M., & Guodong, Z. H. O. U. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-3830-0_23
Effective Large Language Model-Based Diverse Distractor Generation for Multiple-Choice Questions
Nguyen, D., Vo, C., & Nguyen, P. H. (2026) · Lecture notes in networks and systems · doi:10.1007/978-981-96-8793-0_34
Smart Reasoning Assistant: An AI-Driven System for Solving Aptitude and Logical Reasoning Problems
Srivatsala, G. N. G., Yelavarti, K. C., & Raju, L. P. (2025) · International Conference on Communication and Electronics Systems (ICCES) · doi:10.1109/icces67310.2025.11337087
“Reading comprehension tasks were done with Flan-T5-base [9] and the Mistral-8×7B [12] which was fine-tuned on the RACE dataset (≈100k records) [16].”
A Vietnamese Dataset for Text Segmentation and Multiple Choices Reading Comprehension
Nguyen, T. H., Do, D. M., Quan, T. X., & Hà, N. V. (2025) · International Conference on Big-data Service and Intelligent Computation (BDSIC) · doi:10.1145/3778265.3778272
“…datasets include SQuAD [29], which provides over 100,000 question-answer pairs based on Wikipedia articles, RACE [13], a large-scale dataset collected from English exams for Chinese students containing multiple-choice questions that demand deep understanding, and Cosmos QA [10], which…”
Zhou, H., Li, L., Xu, X., & Yin, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5719-6_17
Liu, H. T., Zhang, Y., Meng, Q., & Zhang, X. (2025) · International Conference on Big Data Analytics and Artificial Intelligence Application · doi:10.1145/3788108.3788109
“However, datasets like Narrative QA [3] focus on reading comprehension, while RACE [6]and CLOTH[7], sourced from school examinations, cater to the English language, thus failing to address the multifaceted requirements of Chinese fill-in-the-blank question generation across multiple domains.”
Self-citationA Spanish Language Proficiency Dataset for AI Evaluation
Peñas, A., Rodrigo, Á., Giménez, J. F., Pastor, I. S., Moreno‐Álvarez, S., Pérez, A., & Reyes-Montesinos, J. (2026) · Information · doi:10.3390/info17020159
Mikriukov, A., Plaksin, Y., Ravveduto, A., Snigireva, M., Succi, G., Tormasov, A. G., & Трофимова, Е. (2025) · International Conference on Data Science and Intelligent Systems (DSIS) · doi:10.1109/dsis67228.2025.11390564
“For reading comprehension, many datasets have been developed and studied, for instance: SQuAD [23], MCTest [24] and RACE [25], even though the RC-LSAT [16] is more challenging due to its longer sequences and difficulty to grasp.”
Etutor: An End-to-End Multimodal Framework for AI-Driven Educational Video Generation
Meng, H., Liu, K., Li, C., Li, Y., Shang, D., & Jia, Z. (2025) · International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · doi:10.1109/prai67447.2025.11412789
Yan, C., Deng, L., Hao, M., Li, C., & Shan, W. (2026) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac66049.2026.11420589
“To further ensure the generalizability of our experiments, in addition to the MMLU and Wikitext-2 datasets used in the error injection phase, we incorporated CoLA [31], SST-2 [32], CMMLU [33], RACE [34], and WikiText-103 [25] as additional benchmarks for evaluating the final inference performance.”
Teng, Y., & Wang, C. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00643-4
“For instance, Devlin et al. introduced BERT, which when fine-tuned significantly out-performed previous RNN/attention models on RACE (Lai et al. 2017).”
InfluentialAnalyzing AI Evaluation Benchmarks Through Information Retrieval and Network Science
Simeoni, G., Soprano, M., Lunardi, R., Roitero, K., & Mizzaro, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_25
TransformKV: Optimizing Multi-Turn Conversational Services in LLMs via KV Cache Transformation
Zhou, J., Fang, Z., Qin, Y., Chen, W., Zhang, T., Zhang, C., & Zheng, Z. (2026) · IEEE Transactions on Computers · doi:10.1109/tc.2026.3678524
“To assess output quality, we utilize the SQuAD 2.0 [26], RACE [27], and NewsQA [28] datasets.”
InfluentialHuman-Machine Dialogue in the Medical Field. Using Dialog to Collect Important Patient Information
Liednikova, A. (2022) · theses.fr (ABES) · doi:10.70675/b572f8cbz5876z4683z828bzfb09085ddd78 · Cited by 1
Vers la mitigation des biais en traitement neuronal des langues
Berre, G. L. (2023) · theses.fr (ABES) · doi:10.70675/f37c3802z821az4a6dzb877zf631994cacd5 · Cited by 1
Yu, M., Li, H., Singh, P., Li, X., Wang, D., & Hu, L. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792273 · Cited by 13
“In contrast, PIXEL employs closed-form optimization to calculate the minimum required magnitude subject to a cosine similarity threshold.”
Jiang, J., Qu, Y., & Li, H. (2026) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1109/gaiis69281.2026.11519134
“The experiment uses GSM8K, ARC, and RACE as primary training tasks, fine-tunes Seed LM, Vanilla FT, and CDFT on them, respectively, and evaluates performance on five datasets (GSM8K, ARC[20], RACE[21], HumanEval, and TruthfulQA[22]), including the training tasks themselves.”
Chen, Z., Wang, W., Zhu, J., Wang, X., Liu, Y., Liu, Z. T., Luo, G., Sun, P., Su, W., Tian, H., et al. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3702168 · Cited by 1
Structural and Behavioral Analysis of Fine-Tuning SLMs for Domain Adaptation
Özmen, H. B., & Senel, F. A. (2026) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194026500518
Automated Multiple-Choice Question Generation: A Survey from a KDDM Perspective
Mu, N., & Mine, T. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1947-6_45
Leveraging Human-AI Collaboration for a Passage-Based Question Authoring Tool
Demirtaş, M. A., Nam, S., & Griffin, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29760-0_50
F-AVSS: A Fuzzy Layer-Importance Metric for Efficient Large Language Model Compression
Song, Z., Kang, X., & Kang, Z. (2026) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2026.3708997
“Restrictions apply. information richness because a layer whose activations barely move across samples is unlikely to introduce novel, task-relevant signals [13], [14], [15].”
Sayah, B., Nehar, A., Cherroun, H., Bellaouar, S., & Alam, F. (2026) · BenchCouncil Transactions on Benchmarks Standards and Evaluations · doi:10.66834/byjwsd23
WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
Tian, C., Wang, J., Zhao, Q., Chen, K., Liu, J., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2025) · arXiv · doi:10.48550/arxiv.2507.17634
“…et al., 2025), PIQA (Bisk et al., 2020), hellaswag (Zellers et al., 2019) and KOR-Bench (Ma et al., 2025)) (b) Language Understanding (e.g., race (Lai et al., 2017), SQuAD 2.0 (Rajpurkar et al., 2018), TriviaQA (Joshi et al., 2017), NQ (Kwiatkowski et al., 2019) and winogrande (Sakaguchi et al.,…”
InfluentialGrattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
(2021) · International Conference on Learning Representations · Cited by 435
In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT
Shen, X., Chen, Z., Backes, M., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.08979 · Cited by 76
“• RACE [46] is a multiple-choice reading comprehension dataset.”
Metadata
- Authors
- Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 785-794
- OpenAlex
- W2606964149
- Cited by
- 1,044
- References
- 19
- Citations by year
- 2026: 37 · 2025: 75 · 2024: 86 · 2023: 145 · 2022: 136 · 2021: 216 · 2020: 169 · 2019: 138 · 2018: 38 · 2017: 3
- FWCI
- 42.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750, FA8750-12-2-0342)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, machine reading comprehension, benchmark dataset, English examinations, Chinese students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.