RACE: Large-scale ReAding Comprehension Dataset From Examinations
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task.Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning.In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%).We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [112]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, to our knowledge, most of the existing datasets are not of direct relevance to teaching and learning, except for RACE (Lai, Xie, Liu, Yang, & Hovy, 2017) and LearningQ (Chen, Yang, Hauff, & Houben, 2018).”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Lai, Xie, Liu, Yang, & Hovy, 2017
The cited source says what the claim attributes to it.
In the citing paper
“On RACE-h and RACE-m (Lai et al., 2017), Chinchilla greatly outperforms Gopher , improving accuracy by more than 10% in both cases-see Table 7.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Lai et al., 2017
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We consider only the case of human-readable prompts to large models, which contrasts with methods such as prompt-tuning [16,20], which learns a continuous prompt embedding, and auto-prompting [33], which attempts to search for prompts from scratch, both of which require a training step.”
From Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We choose word embedding size d = 100 and use the 100-dimensional Glove word embedding (Pennington et al., 2014) as embedding initialization.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Instead of evaluating the accuracy, researchers need to use F1 score, BLEU (Papineni et al., 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the questions come without candidate spans.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 8×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
MethodResultInfluentialTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 3×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 1×
“, 2016) and recently proposed TriviaQA (Joshi et al., 2017).”
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578 · cited 2×
“, 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the”
BackgroundSelf-citationNewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 11×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodResultInfluentialMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610 · cited 4×
“Although efforts have been made with a similar motivation, including the MCTest dataset created by (Richardson et al., 2013) (containing 500 passages and 2000 questions) and several others (Peñas et al.”
BackgroundMethodInfluentialA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491 · cited 11×
“However, both require limited reasoning ability (Chen et al., 2016).”
BackgroundMethodInfluential
Cited by
Items 801–900 of 1,036
WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models
Liu, J., Ning, K., Su, Y., Han, W., Xu, J., & Zhang, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889153 · Cited by 25
“The general datasets contain BoolQ [14], RACE [15], PIQA [16], RTE [21], and HellaSwag [17], focusing on reading comprehension, commonsense reasoning, and sentence completion.”
Exercise Semantic Embedding for Knowledge Tracking in Open Domain
Cheng, Z. F., & Li, J. (2025) · Information · doi:10.3390/info16040302 · Cited by 1
Zhu, K., Jiang, L., Zhang, D., Zhang, X., & Lu, K. (2024) · doi:10.1145/3722237.3722250 · Cited by 1
“The RACE (Reading Comprehension from Examinations) dataset, a benchmark dataset specifically tailored for reading comprehension tasks [ 14 ] , was leveraged in the model training phase.”
Mozafari, J., Abdallah, A., Piryani, B., & Jatowt, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730299 · Cited by 3
“MCQA requires models to select the correct answer from a set of options, often including distractors designed to challenge the model [10, 28].”
Elimination-based reasoning with LLM for multiple-choice educational question answering
Zhao, Q., & Zhang, M. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00122-2 · Cited by 8
“We thoroughly evaluate the proposed method on several challenging educational QA datasets: RACE (English reading comprehension exams) (Lai et al. 2017), Open-BookQA (elementary science questions requiring common facts) (Alkhaldi 2023), and ARC (AI2 Reasoning Challenge science exam questions) (Clark…”
VMHQA: A Vietnamese Multi-choice Dataset for Mental Health Domain Question Answering
Nguyen, T. A. H., Nguyen, Q.-D., Nguyen, H. M., Nguyen, A. H., & Nguyen, L. T. T. (2025) · ICST Transactions on Scalable Information Systems · doi:10.4108/eetsis.7678 · Cited by 3
“Benchmark datasets like SQuAD [5] and RACE [6] are necessary for upgrading AI models’ ability to extract information from texts.”
Wang, M., Li, Z., Zhao, X., & Guo, Q. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104422 · Cited by 2
Discourse-Aware Language Representation
Zhang, Z., Ouyang, S., & Zhao, H. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3621229 · Cited by 2
Generating Reading Comprehension Exercises with Large Language Models for Educational Applications
Huang, X., Jiang, F., & Xiao, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5699-1_2 · Cited by 3
ESQAD: A curriculum-aligned dataset for question answer generation in Spanish
Badenes-Olmedo, C., Eyzaguirre-Barreda, P., Chu-Artzt, N., & Gayoso-Cabada, J. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.115255 · Cited by 2
Anwar, M. R., Khalid, S., Alshahrani, S., Bilal, H. S. M., & Aldawsari, M. F. (2026) · IEEE Access · doi:10.1109/access.2026.3652006 · Cited by 6
“In contrast, datasets like CLOTH [113] and RACE [30] provide a more direct training setup, with native cloze and multiple-choice formats, respectively.”
Exploring question answering: metric analysis and evaluation framework for enhanced interpretability
Navarro, L. C., Mucciaccia, S. S., Mutz, F. W., Paixão, T. M., Badue, C. S., Souza, A. F. D., & Oliveira-Santos, T. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12053-8 · Cited by 1
“Our formulation draws inspiration from MCQA diagnostic benchmarks such as MMLU [21], ARC [22], TruthfulQA [23], and RACE [24], to design mechanisms to evaluate QA tasks.”
Statistically Profiling Biases in Natural Language Reasoning Datasets and Models
Huang, S., & Zhu, K. Q. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.299 · Cited by 3
Guoqi Zhang, C. Y. (2021) · Academic Journal of Computing & Information Science · doi:10.25236/ajcis.2021.040212 · Cited by 1
Markchom, T., & Liang, H. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.106 · Cited by 1
“In most QA tasks (Chen et al., 2016; Lai et al., 2017), the answers are commonly concrete words appearing in the contexts.”
BackgroundOperation Diagnosis on Procedure Graph: The Task and Dataset
Luo, R., Zhu, Q., Chen, Q., Wang, S., Wei, Z., Sun, W., & Tang, S. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482157 · Cited by 4
“We adopt accuracy, precision, recall and F1 scores as metrics for automatic evaluation, which are widely used in node classification and question answering [1, 18, 37].”
MethodGANDALF: a General Character Name Description Dataset for Long Fiction
Carlsson, F., Sahlgren, M., Olsson, F., & Gyllensten, A. C. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.13 · Cited by 2
“Examples of previous MRC datasets that also formulate their questions as multiple-choice include: RACE (Lai et al., 2017), OpenBookQA (Mihaylov et al., 2018), Mul-tiRC (Khashabi et al., 2018) and RACE-C (Liang et al., 2019).”
BackgroundA Question Answering System Based on Multi-Granularity Semantic Interaction Neural Network
Zhao, R., Wu, S., Mao, Y., & Wang, X. (2021) · International Conference on Network and Information Systems for Computers (ICNISC) · doi:10.1109/icnisc54316.2021.00177 · Cited by 1
Text Difficulty Classification by Combining Machine Learning and Language Features
Ding, H., Zhong, Q., Zhang, S., & Liu, Y. (2022) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-030-89698-0_108 · Cited by 1
Learning to generate complex question with intent prediction from long passage
Pan, Y., Hu, B., Wang, S., Wang, X., Chen, Q., Xu, Z., & Zhang, M. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03651-9 · Cited by 1
“We first construct a question generation dataset on top of RACE [11], containing nearly 100,000 human-generated questions and 28,000 passages from English exams for middle and high school Chinese students.”
MethodSelf-aware cycle curriculum learning for multiple-choice reading comprehension
Chen, H., & Li, Y. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1179 · Cited by 1
“It contains more than 100,000 questions posed by experts, and it focuses more on reasoning skills (Lai et al., 2017).”
BackgroundBhagwatkar, N., Vaidya, K., Singh, A. P., Borikar, S., & Padwad, H. (2023) · INTERNATIONAL JOURNAL OF NEXT-GENERATION COMPUTING · doi:10.47164/ijngc.v14i1.1044 · Cited by 1
Categorizing Citation Relations in Scientific Papers Based on the Contributions of Cited Papers
Chen, P.-C., Huang, H., & Chen, H. (2022) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat55865.2022.00063 · Cited by 1
“[13] construct a dataset from Chinese high school English multiple-choice reading quiz in RACE.”
BackgroundHuman attention during goal-directed reading comprehension relies on task optimization
Zou, J., Zhang, Y., Li, J., Tian, X., & Ding, N. (2023) · eLife · doi:10.7554/elife.87197 · Cited by 9
“In total, 800 question/passage pairs were adapted from the RACE dataset ( Lai et al., 2017 ), a collection of English reading comprehension questions designed for Chinese high school students who learn English as a second language.”
BackgroundMethodCross-task Knowledge Transfer for Extremely Weakly Supervised Text Classification
Park, S., Kim, K., & Lee, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.328 · Cited by 2
“For PLAT QA , we use RoBERTa 4 model trained on RACE (Lai et al., 2017).”
MethodUsing contradictions improves question answering systems
Fortier-Dubois, E., & Rosati, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.72 · Cited by 8
“Specifically, we used a RoBERTa large model (Liu et al., 2019) finetuned on the RACE dataset (Lai et al., 2017), as well as two DeBERTa v3 variants, base and xsmall (He et al., 2021a), finetuned on the SciQ dataset (Welbl et al., 2017).”
MethodPILE: Pairwise Iterative Logits Ensemble for Multi-Teacher Labeled Distillation
Cai, L., Zhang, L., Ma, D., Fan, J., Shi, D., Wu, Y., Cheng, Z., Gu, S., & Yin, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-industry.60 · Cited by 1
Yi, J., Chen, W., Qin, Y., Lin, Y., Ding, N., Xu, H., Liu, Z., Sun, M., & Zhou, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.244 · Cited by 2
PLM-PGHC: A novel de-biasing framework for robust question answering
Yu, S., Wu, M., Zhang, Y., Xie, N., & Huang, L. (2023) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-233029 · Cited by 2
Accurate, Diverse and Multiple Distractor Generation with Mixture of Experts
Qu, F., Wang, C., & Wu, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44693-1_59 · Cited by 3
LEBERT :Lite and Efficiently Optimized BERT PRetraining Approach
Yadav, P., & Sharma, A. R. (2024) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-15219 · Cited by 1
Identifying Valid User Stories Using BERT Pre-trained Natural Language Models
Scoggin, S. B., & Marques-Neto, H. T. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-45648-0_17 · Cited by 2
Probing Pretrained Language Models with Hierarchy Properties
Lovón-Melgarejo, J., Moreno, J. G., Besançon, R., Ferret, O., & Tamine, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56060-6_9 · Cited by 1
Wang, J., Bansal, K., Arapakis, I., Ge, X., & Jose, J. M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56027-9_19 · Cited by 5
Modeling Logical Content and Pattern Information for Contextual Reasoning
Guo, P., Jian, P., & Shi, X. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650051 · Cited by 1
“RACE [24] requires model to choose the right answer from multiple options.”
MethodFive Shades of BERT: Assessing Performance in Question Answering through Comparative Analysis
Raikwar, K., Rajusth, P., Prabhu, S., & Shrivastava, N. (2024) · Asian Conference on Innovation in Technology (ASIANCON) · doi:10.1109/asiancon62057.2024.10837814 · Cited by 1
“…of language to learn from and explores the impact of varying training steps (100k, 300k, and 500k) with a larger batch size (8k) to achieve cutting-edge performance, as demonstrated by results on several benchmarks, including GLUE [13], SQuAD [2], and RACE [15] within the scope of the paper.”
BRoverbs - Measuring how much LLMs understand Portuguese proverbs
Almeida, T. S., Bonás, G. K., & Santos, J. G. A. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5797 · Cited by 5
Multi-Perspective Fusion Network for Commonsense Reading Comprehension
Liu, C., Yan, Z., Si, Q., Zhang, H., Li, B., & Yu, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01716-3_22 · Cited by 1
“Several datasets have been constructed for testing the comprehension ability of a system, such as MCTest [15], SQuAD [14], BAbI [22], TriviaQA [6], RACE [8], and NewsQA [17].”
MethodScholarlyRead: A New Dataset for Scientific Article Reading Comprehension.
Saikh, T., Ekbal, A., & Bhattacharyya, P. (2020) · Language Resources and Evaluation · Cited by 1
A Matching-Integration-Verification Model for Multiple-Choice Reading Comprehension
Xing, L., Hu, Y., Xie, Y., Wang, C., & Hu, Y. S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-55393-7_21 · Cited by 1
Employing Sentence Context in Czech Answer Selection
Medveď, M., Sabol, R., & Horák, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58323-1_12 · Cited by 1
Learning to Answer Word-Meaning-Explanation Questions for Chinese Gaokao Reading Comprehension
Tan, H., Qiang, P., & Li, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60450-9_5 · Cited by 1
Mitra, A., Narayana, S., & Baral, C. R. (2020) · Workshop on Structured Prediction for NLP (SPNLP) · doi:10.18653/v1/2020.spnlp-1.12 · Cited by 1
“…the accuracy of our system (DeepEKR) with the two re-ported solvers, namely BERT (standard BERT multiple choice question solver trained on the QuaRTz dataset) and BERT-PFT-Race ( BERT multiple choice question solver trained on the Race dataset (Lai et al., 2017) and then on the QuaRTz dataset) .”
MethodRAIN: A Relation-based Arithmetic model with Implicit Numbers
Bai, Z., Li, K., Chen, J., Yuan, C., & Wang, X. (2020) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc51575.2020.9345205 · Cited by 1
“The above models perform well on a variety of questions [1], [9]–[11].”
BackgroundChallenge Closed-Book Science Exam: A Meta-Learning Based Question Answering System
Zheng, X., Wang, P., Wang, Q., & Shi, Z. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69886-7_12 · Cited by 1
CSFQGD: Chinese Sentence Fill-in-the-blank Question Generation Dataset for Examination
Zhang, T., Cui, Z., Leng, J., & Liu, Y. (2021) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd49262.2021.9437824 · Cited by 2
Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction
Shen, M., Banerjee, P., & Baral, C. R. (2021) · doi:10.18653/v1/2021.acl-short.117 · Cited by 5
“Recent study (Khot et al., 2020) has shown that finetuning a RACE-finetuned (Lai et al., 2017) RoBERTa model as a start point is much more stable than directly finetuning a RoBERTa model from scratch, we follow the same strategy to start fine-tuning a RACE-finetuned RoBERTa-large model on all synthetic…”
PINGAN Omini-Sinitic at SemEval-2021 Task 4:Reading Comprehension of Abstract Meaning
Wang, Y., Wang, Y., Zhu, H., Zeng, B., Hao, Z., Wang, S., & Xiao, J. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.109 · Cited by 2
BiQuAD: Towards QA based on deeper text understanding
Grimm, F., & Cimiano, P. (2021) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2021.starsem-1.10 · Cited by 1
“The RACE dataset (Lai et al., 2017) relies on a multiple choice setting and leverages data used for the assessment of reading comprehension by humans.”
MethodEnglish Cloze Test Based on BERT
Ding, M., Chen, M., Chen, W., & Cai, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-82147-0_4 · Cited by 1
Knowledge Representation, Reasoning and Learning for Non-Extractive Reading Comprehension
Mitra, A. (2019) · Cited by 1
Teaching Machines to Read and Comprehend Tibetan Text
Sun, Y., Liu, S., Chen, C., Dan, Z., & Zhao, X. (2021) · Journal of Computer and Communications · doi:10.4236/jcc.2021.99011 · Cited by 2
A Better Multiway Attention Framework for Fine-Tuning
Hao, K., Li, J., Hou, C., & Li, P. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-92307-5_44 · Cited by 1
Incorporating Phrases in Latent Query Reformulation for Multi-Hop Question Answering
Tang, J., Hu, S., Chen, Z., Xu, H., & Tan, Z. (2022) · Mathematics · doi:10.3390/math10040646 · Cited by 3
A Review of Machine Reading Comprehension
殷, 明. (2020) · Computer Science and Application · doi:10.12677/csa.2020.1012261 · Cited by 1
Lee, J., Moon, H., Park, C., Seo, J., Eo, S., & Lim, H. (2022) · Applied Sciences · doi:10.3390/app12136686 · Cited by 1
Question Difficulty Estimation with Directional Modality Association in Video Question Answering
Kim, B.-M., & Park, S.-B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08530-7_24 · Cited by 1
Calibration of the Multiple Choice Machine Reading Comprehension
Xing, L., Hu, Y., Xie, Y., Peng, W., Sun, Y., & Zhang, C. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892434 · Cited by 1
“The whole RACE dataset is split into a RACE-middle version and a RACE-high version, which correspond to exams in middle school and high school, respectively.”
MethodInfluentialOn Degrees of Freedom in Defining and Testing Natural Language Understanding
Sugawara, S., & Tsugita, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.861 · Cited by 3
SharPT: Shared Latent Space Prompt Tuning
Pang, B., Yavuz, S., Xiong, C., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.92 · Cited by 3
QuaRTz: An Open-Domain Dataset of Qualitative Relationship QuestionsPreprint
Tafjord, O., Gardner, M., Lin, K., & Clark, P. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03553 · Cited by 23
“BERT-PFT (no knowledge): BERT first fine-tuned (“pre-fine-tuned”) on the RACE dataset (Lai et al., 2017; Sun et al., 2019), and then fine-tuned on Q UA RT Z (questions only, no K , both train and test).”
MethodWhat's Missing: A Knowledge Gap Guided Approach for Multi-hop Question AnsweringPreprint
Khot, T., Sabharwal, A., & Clark, P. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09253 · Cited by 5
“Since this and several recent multi-hop datasets use the multiple-choice setting (Welbl et al., 2018; Khashabi et al., 2018; Lai et al., 2017), we assume access to potential answers to a question.”
BackgroundJing, Y., Xiong, D., & Yan, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05040 · Cited by 3
“To address these limitations, a number of datasets have be built recently, such as MS MARCO (Nguyen et al., 2016), DuReader (He et al., 2018), TriviaQA (Joshi et al., 2017), RACE (Lai et al., 2017), NarrativeQA (Kocisky et al., 2018), SQuAD2.”
MethodDesign and Challenges of Cloze-Style Reading Comprehension Tasks on Multiparty DialoguePreprint
Li, C., Liu, T., & Choi, J. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00773 · Cited by 1
AntNet: Deep Answer Understanding Network for Natural Reverse QA.Preprint
Yang, L., Yin, Q., Hou, L., Gui, J., Wu, O., & Kwok, J. T.-Y. (2019) · arXiv (Cornell University) · Cited by 1
Gestalt: a Stacking Ensemble for SQuAD2.0Preprint
El-Geish, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.07067 · Cited by 2
“Multiple-choice QA systems pick an answer out of multiple choices like in the MCTest [6] and RACE [7] tasks.”
BackgroundZhang, T., Wang, C., Qiu, M., Yang, B., He, X., & Huang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.10327 · Cited by 1
“…et al., 2016; Cui et al., 2019) often leads to poor annotation results. ii) Due to the domain sensitivity, people are more concerned about the reliability of the information sources where the answers are extracted, and the ex-plainability of the answers themselves (Lee et al., 2014; Dalmer, 2017).”
Knowledge Efficient Deep Learning for Natural Language ProcessingPreprint
Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.12878 · Cited by 3
“8 Amazon Turker Performance [1] - - 85.”
BackgroundMethodInfluentialLan, T., Mao, X.-L., Zhao, Z., Wei, W., & Huang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11357 · Cited by 2
Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World KnowledgePreprint
Xu, C., Zhou, W., Ge, T., Xu, K., McAuley, J., & Wei, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.02704 · Cited by 1
“We formulate the task of decoding the cant in a similar format to multi-choice reading comprehension tasks (Lai et al., 2017; Zellers et al., 2018; Clark et al., 2018).”
MethodAn Empirical Study of Offensive Language in Online Interactions
Sarkar, D. (2021) · RIT Scholar Works (Rochester Institute of Technology) · Cited by 1
Zero-shot Task Transfer for Invoice Extraction via Class-aware QA EnsemblePreprint
Damodaran, P., Singh, P., & Achankuju, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.06069 · Cited by 2
Attention-based Approaches for Text Analytics in Social Media and Automatic Summarization
Barba, J. Á. G. (2021) · RiuNet (Universitat Politècnica de València) · doi:10.4995/thesis/10251/172245 · Cited by 4
Audio captioning and retrieval with improved cross-modal objectives
Koh, A. J. J. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/172437 · Cited by 1
Bridging Today and the Future of Humanity: AI Safety in 2024 and BeyondPreprint
Han, S. (2025) · Qeios · doi:10.32388/zhm9uq · Cited by 5
Hechtner, F., Schmidt, L., Seebeck, A., & Weiß, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5146523 · Cited by 4
Deep learning for information extraction from business documents
Sage, C. (2021) · theses.fr (ABES) · doi:10.70675/fd1f6c1fz859bz4230zbf7bz9bb7cdb83ad0 · Cited by 3
Evaluating and Improving the Reasoning Abilities of Language Models
Helwe, C. (2024) · theses.fr (ABES) · doi:10.70675/7fc70e59zdd27z4b0fza38dz03a2c67cda50 · Cited by 1
Document Understanding with Deep Learning Techniques
Nguyen, K.-A. L. (2024) · theses.fr (ABES) · doi:10.70675/b83ce709z5fc9z4a33za0bezd7aaad779ef1 · Cited by 4
Evaluation and Integration of Structural Knowledge for Pre-trained Language Models
Lovón-Melgarejo, J. (2024) · theses.fr (ABES) · doi:10.70675/9b2d6bf6za13az4dfcza45bzca75ad5b25a6 · Cited by 2
Beyond Text Compression: Evaluating Tokenizers Across Scales
Lotz, J. F., Lopes, A. V., Peitz, S., Setiawan, H., & Emili, L. (2025) · arXiv · doi:10.48550/arxiv.2506.03101 · Cited by 1
“• Knowledge understanding (KU): MMLU (Hendrycks et al., 2021), and RACE (Lai et al., 2017).”
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
von Oswald, J., Scherrer, N., Kobayashi, S., Versari, L., Yang, S., Mittal, S., Schlegel, M., Maile, K., Schimpf, Y., Sieberling, O., et al. (2025) · arXiv
“This includes Lambada [74], HellaSwag [114] and RACE-{M,H} [59].”
MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
Wang, J., Tian, C., Chen, K., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2026) · arXiv · doi:10.48550/arxiv.2601.17858
“…(Hong et al., 2025), PIQA (Bisk et al., 2020), hellaswag (Zellers et al., 2019) and KOR-Bench (Ma et al., 2025)); (b) language understanding (race (Lai et al., 2017), SQuAD 2.0 (Rajpurkar et al., 2018), TriviaQA (Joshi et al., 2017), NQ (Kwiatkowski et al., 2019) and wino-grande (Sakaguchi et…”
InfluentialDARC-IT: a DAtaset for Reading Comprehension in Italian
Brunato⋄, D., Valeriani, M., & Dell’Orletta⋄, F. (2018) · Accademia University Press eBooks · doi:10.4000/books.aaccademia.3099
“The same holds for RACE (Lai et al., 2017), which is also the only one specifically designed for educational purposes.”
BackgroundExam Keeper: Detecting Questions with Easy-to-Find Answers
Hsu, T.-L., Dai, S.-C., & Ku, L. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3314130
“To train these models and retrieve similar questions, we use the CLOTH [8] dataset for cloze questions, the RACE [3] dataset for reading comprehension, and the SWAG [9] dataset for fit-the-best-sentence questions.”
MethodODSQA: Open-domain Spoken Question Answering DatasetPreprint
Lee, C., Wang, S.-M., Chang, H.-C., & Lee, H.-Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.02280
“Most QA work focuses on understanding text documents[14, 15, 1, 16].”
BackgroundQuestion answering with textual sequence matching
Wang, S. (2019) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University)
Neural architectures for natural language understanding
Tay, Y. (2019) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/83291
BIG MOOD: Relating Transformers to Explicit Commonsense KnowledgePreprint
Da, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07713
“Yang et al. (2019) achieves state-of-the-art on RACE (Lai et al., 2017) with a Transformer-XL based model (Dai et al., 2019).”
Reconstructed Option Rereading Network for Opinion Questions Reading Comprehension
Qiu, D., Bao, L., Tian, Z., Zhang, Y., Liu, K., Zhao, J., & Liao, X. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32381-3_8
“There are some large-scale datasets for this task, such as MCTest [10] and RACE [4].”
MethodInjecting Hierarchy with U-Net TransformersPreprint
Donahue, D., Lialin, V., & Rumshisky, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.10488 · Cited by 1
“, 2019), and others, with Transformer-based models dominating leaderboards for many common benchmarks (Wang et al., 2018, 2019; Lai et al., 2017).”
BackgroundMultiple-Choice Question Answering Over Semi-Structured Tables
Ni, W. (2019) · ERA: Education and Research Archive (University of Alberta) · doi:10.7939/r3-c9ky-cd27
Narayana, S. (2020) · Arizona State University Library Digital Repository (Arizona State University)
Cross-lingual Training for Multiple-Choice Question Answering
Echegoyen, G., Rodrigo, Á., & Peñas, A. (2020) · RUA, Repositorio Institucional de la Universidad de Alicante (Universidad de Alicante) · doi:10.26342/2020-65-4
“Given the size of the RACE dataset (Lai et al., 2017), it is possible to fine-tune an English model with it, but you cannot do the same for other languages as the datasets are too small.”
BackgroundMethodInformation Extraction from Public Meeting Articles
Virgo, F. G., Chu, C., Ogawa, T., Tanaka, K., Ashihara, K., Nakashima, Y., Takemura, N., Nagahara, H., & Fujikawa, T. (2022) · SN Computer Science · doi:10.1007/s42979-022-01176-z
Questions Generation for Reading Comprehension Using Coherence Relations
Anamika, Pathak, V., Shrivastava, V., & Panday, A. (2021) · Algorithms for intelligent systems · doi:10.1007/978-981-15-6707-0_6
Knowledge graph augmented advanced learning models for commonsense reasoning
Pothuri, A., Veeramallu, H. S. R., & Malik, P. (2021) · IOP Conference Series Materials Science and Engineering · doi:10.1088/1757-899x/1022/1/012038
“Even algorithms are used to pass information from other read-understanding datasets, such as RACE [39] and OpenBookQA [40].”
MethodA Data-Oriented Approach for Closed-Book Question Answering
Suzuki, M., Matsuda, K., Ouchi, H., Suzuki, J., & Inui, K. (2021) · Journal of Natural Language Processing · doi:10.5715/jnlp.28.3
“1 はじめに 質問応答は,自然言語処理における重要な問題の一つであり,古くから研究が続けられている. 特に近年では, SQuAD (Rajpurkar et al. 2016, 2018),MS MARCO (Bajaj et al. 2016),RACE (Lai et al. 2017)といった読解型質問応答の大規模なデータセットが多数提案され,それと同時 に,深層ニューラルネットワークを用いた質問応答の手法が数多く提案されている.また,こ れらのデータセットは,質問応答の研究だけではなく,近年盛んに研究がなされている BERT (Devlin et al. 2019)…”
Pseudoscience Detection Using a Pre-trained Transformer Model with Intelligent ReLabeling
Rajapakse, T. C., & Nawarathna, R. D. (2021) · Lecture notes in networks and systems · doi:10.1007/978-981-33-4355-9_25
What Linguistic Information Does Reading Comprehension Require?
Guan, Y., Li, R., & Guo, S. (2021) · Communications in computer and information science · doi:10.1007/978-981-16-1964-9_20
Metadata
- Authors
- Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 785-794
- OpenAlex
- W2606964149
- Cited by
- 1,044
- References
- 19
- Citations by year
- 2026: 37 · 2025: 75 · 2024: 86 · 2023: 145 · 2022: 136 · 2021: 216 · 2020: 169 · 2019: 138 · 2018: 38 · 2017: 3
- FWCI
- 42.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750, FA8750-12-2-0342)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, machine reading comprehension, benchmark dataset, English examinations, Chinese students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.