RACE: Large-scale ReAding Comprehension Dataset From Examinations
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task.Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning.In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%).We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [112]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, to our knowledge, most of the existing datasets are not of direct relevance to teaching and learning, except for RACE (Lai, Xie, Liu, Yang, & Hovy, 2017) and LearningQ (Chen, Yang, Hauff, & Houben, 2018).”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Lai, Xie, Liu, Yang, & Hovy, 2017
The cited source says what the claim attributes to it.
In the citing paper
“On RACE-h and RACE-m (Lai et al., 2017), Chinchilla greatly outperforms Gopher , improving accuracy by more than 10% in both cases-see Table 7.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Lai et al., 2017
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We consider only the case of human-readable prompts to large models, which contrasts with methods such as prompt-tuning [16,20], which learns a continuous prompt embedding, and auto-prompting [33], which attempts to search for prompts from scratch, both of which require a training step.”
From Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We choose word embedding size d = 100 and use the 100-dimensional Glove word embedding (Pennington et al., 2014) as embedding initialization.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Instead of evaluating the accuracy, researchers need to use F1 score, BLEU (Papineni et al., 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the questions come without candidate spans.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 8×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
MethodResultInfluentialTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 3×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 1×
“, 2016) and recently proposed TriviaQA (Joshi et al., 2017).”
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578 · cited 2×
“, 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the”
BackgroundSelf-citationNewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 11×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodResultInfluentialMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610 · cited 4×
“Although efforts have been made with a similar motivation, including the MCTest dataset created by (Richardson et al., 2013) (containing 500 passages and 2000 questions) and several others (Peñas et al.”
BackgroundMethodInfluentialA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491 · cited 11×
“However, both require limited reasoning ability (Chen et al., 2016).”
BackgroundMethodInfluential
Cited by
Items 201–300 of 1,038
Representation Learning for Natural Language Processing
Liu, Z., Lin, Y., & Sun, M. (2023) · Springer eBooks · doi:10.1007/978-981-99-1600-9 · Cited by 81
PRover: Proof Generation for Interpretable Reasoning over Rules
Saha, S., Ghosh, S., Srivastava, S., & Bansal, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.9 · Cited by 93
HEAD-QA: A Healthcare Dataset for Complex Reasoning
Vilares, D., & Gómez-Rodríguez, C. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1092 · Cited by 46
“With a similar aim, Lai et al. (2017) released 100k questions and 28k passages intended for middle or high school Chinese students, and Zellers et al. (2018) introduced a dataset for common sense reasoning from a spectrum of daily situations.”
BackgroundDeep learning based question answering system in Bengali
Mayeesha, T. T., Sarwar, A. M., & Rahman, R. M. (2020) · Journal of Information and Telecommunication · doi:10.1080/24751839.2020.1833136 · Cited by 66
“From 2015 – 2016, there have been multiple attempts at curating large scale question answering datasets for English such as CNN/ Daily Mail Corpus (Hermann et al., 2015), SQuAD 1.0 (Rajpurkar et al., 2016), RACE (Lai et al., 2017), SQuAD 2.0 (Rajpurkar et al., 2018), QuAC (Choi et al., 2015), CoQA (Reddy et al., 2018)…”
BackgroundInfluentialLogic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text
Wang, S., Zhong, W., Tang, D., Wei, Z., Fan, Z., Jiang, D., Zhou, M., & Duan, N. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.127 · Cited by 48
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
Zhang, Z., Lin, Y., Liu, Z., Li, P., Sun, M., & Zhou, J. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.71 · Cited by 48
“In the experiments, we evaluate MoEfication on two typical kinds of downstream tasks, including GLUE and QA benchmarks (Wang et al., 2019; Rajpurkar et al., 2016; Lai et al., 2017), using T5 and BERT (Raffel et al.”
BackgroundMethodInfluentialExploiting Explicit Paths for Multi-hop Reading Comprehension
Kundu, S., Khot, T., Sabharwal, A., & Clark, P. E. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1263 · Cited by 55
“, 2018) pre-trained on an additional dataset, RACE (Lai et al., 2017) to achieve a score of 55% on this task.”
BackgroundText Compression-aided Transformer Encoding
Li, Z., Zhang, Z., Zhao, H., Wang, R., Chen, K., Utiyama, M., & Sumita, E. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3058341 · Cited by 59
“Multi-choice MRC Our Multi-choice MRC is evaluated on the Large-scale ReAding Comprehension Dataset From Examinations (RACE) dataset [46], which consists of two subsets, RACE-M and RACE-H, corresponding to middle school and high school difficulty levels.”
MethodEQG-RACE: Examination-Type Question Generation
Jia, X., Zhou, W., Sun, X., & Wu, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17553 · Cited by 48
EQG-RACE: Examination-Type Question Generation
Jia, X., Zhou, W., Sun, X., & Wu, Y. (2020) · arXiv · doi:10.48550/arxiv.2012.06106 · Cited by 48
“Different from these datasets, RACE (Lai et al. 2017) is collected from English exams for Chinese students, which is a high-quality examination dataset on reading comprehension in real context.”
BackgroundWhich *BERT? A Survey Organizing Contextualized Encoders
Xia, P., Wu, S., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.608 · Cited by 77
WebSRC: A Dataset for Web-Based Structural Reading Comprehension
Chen, X., Zhao, Z., Chen, L., Ji, J., Zhang, D., Luo, A., Xiong, Y., & Yu, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.343 · Cited by 146
“Traditional datasets for machine reading comprehension (Talmor et al., 2019; Yang et al., 2018; Ra-jpurkar et al., 2016, 2018; Choi et al., 2018; Reddy et al., 2019; Lai et al., 2017) contain plain text passages and QAs about them.”
BackgroundTR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
Ye, D., Lin, Y., Huang, Y., & Sun, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.463 · Cited by 47
DUMA: Reading Comprehension With Transposition Thinking
Zhu, P., Zhang, Z., Zhao, H., & Li, X. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3138683 · Cited by 79
“…datasets and models have been proposed in recent years (Trischler et al. 2017; Zhang et al. 2018a; Nguyen et al. 2016; Rajpurkar et al. 2016; Hermann et al. 2015; Sun et al. 2019a; Lai et al. 2017; Zhang et al. 2018b; Zhu et al. 2018b; Zhang et al. 2020b; Bhargav et al. 2020; Hu et al. 2019).”
BackgroundLeaf: Multiple-Choice Question Generation
Vachev, K., Hardalov, M., Karadzhov, G., Georgiev, G., Koychev, I., & Nakov, P. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-99739-7_41 · Cited by 45
Zero-Shot Dialogue State Tracking via Cross-Task Transfer
Lin, Z., Liu, B., Madotto, A., Moon, S., Zhou, Z., Crook, P., Wang, Z., Zhou, Y., Cho, E., Subba, R., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.622 · Cited by 86
“, 2019), and that of multiple-choice datasets are from RACE (Lai et al., 2017) and DREAM (Sun et al.”
BackgroundInfluentialODSQA: Open-Domain Spoken Question Answering Dataset
Lee, C., Wang, S.-M., Chang, H.-C., & Lee, H.-Y. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639505 · Cited by 65
DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
He, W. P., Liu, K., Liu, J., Lyu, Y., Zhao, S., Xiao, X., Liu, Y., Wang, Y., Wu, H. R., She, Q., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05073 · Cited by 45
“Multiple-choice MRC (Lai et al., 2017) tests comprehension ability via option selection on examination data.”
BackgroundMachine Reading Comprehension: The Role of Contextualized Language Models and BeyondPreprint
Zhang, Z., Zhao, H., & Wang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06249 · Cited by 80
“…the previous and recent dominant techniques by reviewing the systems for the flagship datasets concerning the main types of MRC, cloze-type CNN/DailyMail (Hermann et al. 2015), multi-choice RACE (Lai et al. 2017), and span extraction SQuAD (Rajpurkar et al. 2016; Rajpurkar, Jia, and Liang 2018).”
BackgroundMethodInfluentialAmazonQA: A Review-Based Question Answering Task
Gupta, M., Kulkarni, N., Chanda, R., Rayasam, A., & Lipton, Z. C. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/694 · Cited by 69
“, 2016] Crowd-Sourced Wiki Span of words 100K 536 RACE [Lai et al., 2017] Crowd-Sourced English Exams Multiple choice 97l 28K ARC [Clark et al.”
Background“You are grounded!”: Latent Name Artifacts in Pre-trained Language Models
Shwartz, V., Rudinger, R., & Tafjord, O. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.556 · Cited by 43
Survey of Neural Text Representation Models
Babić, K., Martinčić-Ipšić, S., & Meštrović, A. (2020) · Information · doi:10.3390/info11110511 · Cited by 54
“• RACE (ReAding Comprehension dataset from Examinations) [37] is a dataset for the evaluation of methods in the reading comprehension task, consisting of English exams for Chinese students.”
MethodExplaining Question Answering Models through Text GenerationPreprint
Latcinnik, V., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05569 · Cited by 69
“Multi-choice QA MC-QA has been a popular format for QA (Lai et al., 2017; Zellers et al., 2018; Clark et al., 2018) mostly because it simplifies evaluation of free text answers.”
Repurposing Entailment for Multi-Hop Question Answering Tasks
Trivedi, H. K., Kwon, H. Y., Khot, T., Sabharwal, A., & Balasubramanian, N. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1302 · Cited by 53
“Recall that the Reading Strategies results are reported with an additional fine-tuning on another larger QA dataset, RACE (Lai et al., 2017) aside from the target QA datasets we use here.”
BackgroundMethodInfluentialQuiz-Style Question Generation for News Stories
Lelkes, A. D., Tran, V. Q., & Yu, C. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3449892 · Cited by 43
“The most commonly used question answering datasets only contain the correct answer; available multiple-choice question datasets that include distractors, such as [25] [47], typically have short distractors and/or focus on reading comprehension or a narrow domain.”
You Need to Read Again: Multi-granularity Perception Network for Moment Retrieval in Videos
Sun, X., Wang, X., Gao, J., Liu, Q., & Zhou, X. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532083 · Cited by 58
“The former needs to locate a certain span that seems most related to the given question [23, 24] while the latter is required to select a correct answer in a given candidate choices set[15, 32].”
BackgroundLi, J., Liu, M., Zheng, Z., Zhang, H., Qin, B., Kan, M., & Liu, T. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533364 · Cited by 42
“There are several types of datasets for machine comprehension, such as multiple-choice datasets [10], [12], answer sentence selection datasets [26], [27] and extractive datasets [11], [28]–[30]. b) Discourse parsing for multiparty dialogues: Discourse parsing for multiparty dialogues is a…”
BackgroundPreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution
Chen, H., Fan, Z., Lü, H., Yuille, A., & Rong, S. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1016 · Cited by 77
“In addition to the crawled data, we include the documents from the RACE dataset (Lai et al., 2017).”
MethodA survey of deep learning techniques for machine reading comprehension
Kazi, S., Khoja, S. A., & Daud, A. Y. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10583-4 · Cited by 52
Comprehensive analysis of embeddings and pre-training in NLP
Tripathy, J. K., Sethuraman, S. C., Cruz, M. V., Namburu, A., Mangalraj, P., Nandha Kumar, R., Sudhakar Ilango, S., & Vijayakumar, V. (2021) · Computer Science Review · doi:10.1016/j.cosrev.2021.100433 · Cited by 58
Efficient Video Grounding With Which-Where Reading Comprehension
Gao, J., Xin, S., Ghanem, B., Zhou, X., & Ge, S. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3174136 · Cited by 44
“The latter [48], [49] 206 aims to find only one choice in the given candidate set based 207 on the given passage.”
BackgroundEvaluating Models' Local Decision Boundaries via Contrast Sets
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02709 · Cited by 40
“Model We use the best model from Lin et al. (2019), which is a span extraction model built on top of a RoBERTa model (Liu et al., 2019) that is first finetuned on RACE (Lai et al., 2017).”
MethodHierarchical Learning for Generation with Long Source Sequences
Rohde, T., Wu, X., & Liu, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07545 · Cited by 41
“We evaluate the pre-trained model on three downstream tasks: SQuAD 2.0 (Rajpurkar et al., 2018), MNLI-m (Williams et al., 2018) and RACE (Lai et al., 2017).”
MethodInfluentialMulti-Matching Network for Multiple Choice Reading Comprehension
Tang, M., Cai, J., & Zhuo, H. H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017088 · Cited by 45
“Compared to these datasets where answer is extracted from passage directly, answers for questions in MS-MARCO (Nguyen et al. 2016), RACE (Lai et al. 2017), DuReader (He et al. 2018) are human-generated, which is more challenging.”
BackgroundMethodInfluentialImproving Subject-Area Question Answering with External Knowledge
Pan, X., Sun, K., Yu, D., Chen, J., Ji, H., Cardie, C., & Yu, D. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5804 · Cited by 72
“, 2019) on a large-scale multiple-choice machine reading comprehension dataset RACE (Lai et al., 2017) collected from English-as-a-foreign-language exams, which provides a ground truth reference document instead of a reference corpus for each question.”
BackgroundMethodInfluentialDocRED: A Large-Scale Document-Level Relation Extraction DatasetPreprint
Yao, Y., Ye, D., Li, P., Han, X., Lin, Y., Liu, Z., Liu, Z., Huang, L., Zhou, J., & Sun, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06127 · Cited by 40
“…Mitamura et al., 2015, 2017), fact extraction and verification (Thorne et al., 2018), reading comprehension (Nguyen et al., 2016; Joshi et al., 2017; Lai et al., 2017), sentiment classification (Pang and Lee, 2004; Pret-tenhofer and Stein, 2010), summarization (Nalla-pati et al., 2016) and machine…”
BackgroundMishra, A., Patel, D., Vijayakumar, A., Li, X. L., Kapanipathi, P., & Talamadupula, K. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.104 · Cited by 39
“The Question-Answering (QA) task can easily be cast into the NLI form, and QA datasets (Rajpurkar et al., 2016; Lai et al., 2017b; Khashabi et al., 2018; Sun et al., 2019; Huang et al., 2019) encompass a variety of semantic phenomena that only occur in longer (con)texts.”
BackgroundThe Unbearable Shallow Understanding of Deep Learning
Plebe, A., & Grasso, G. M. (2019) · Minds and Machines · doi:10.1007/s11023-019-09512-8 · Cited by 90
Automatic Distractor Generation for Multiple Choice Questions in Standard Tests
Qiu, Z., Wu, X., & Fan, W. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.189 · Cited by 53
English Machine Reading Comprehension Datasets: A Survey
Dzendzik, D., Foster, J., & Vogel, C. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.693 · Cited by 69
“, 2016) 15k Wikipedia annotators 3 7 what ) RACE (Lai et al., 2017) 98k ER experts 3] 73.”
InfluentialJiang, Z., Araki, J., Ding, H., & Neubig, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00955 · Cited by 39
“…(fictional stories (Richardson et al., 2013)), PIQA (physical (Bisk et al., 2020)), SIQA (social (Sap et al., 2019)), RACE (English comprehension (Lai et al., 2017)), QASC (science (Khot et al., 2020)), MT-test (mixed (Hendrycks et al., 2020)) Extractive SQuAD 1.1 (wikipedia (Rajpurkar et al.,…”
InfluentialTesting Your Question Answering Software via Asking Recursively
Chen, S., Jin, S., & Xie, X. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678670 · Cited by 63
“These datasets are with diverse forms of task, including to fill in the blanks [54], [55], judge the correct options [6], [29], [56], extract the relevant spans [5], [8], [28], and return fluent text answers [7], [9].”
BackgroundLegal Information Retrieval and Entailment Based on BM25, Transformer and Semantic Thesaurus Methods
Kim, M., Rabelo, J. C. B., Okeke, K., & Goebel, R. G. (2022) · The Review of Socionetwork Strategies · doi:10.1007/s12626-022-00103-1 · Cited by 65
“These models have achieved impressive results in a wide range of publicly available benchmarks of different common natural language tasks, such as RACE (reading comprehension) [13] , COPA (common sense reasoning) [14] and RTE (textual entailment) [15], to name a few.”
BackgroundChallenging Reading Comprehension on Daily Conversation: Passage Completion on Multiparty Dialog
Ma, K., Jurczyk, T., & Choi, J. D. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1185 · Cited by 42
“More datasets are available for another type of a reading comprehension task, that is multiple choice question answering, such as MCTest (Richardson et al., 2013), TriviaQA (Joshi et al., 2017), RACE (Lai et al., 2017), and SQuAD (Rajpurkar et al., 2016).”
BackgroundMethodMCScript2.0: A Machine Comprehension Corpus Focused on Script Events and Participants
Ostermann, S., Roth, M., & Pinkal, M. (2019) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s19-1012 · Cited by 44
“In contrast to Wang et al. (2018), we use the non-ensemble version of TriAN without pretraining on RACE (Lai et al., 2017), for better comparability to the other models.”
Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text
Li, D., Hu, B., Chen, Q., Peng, W., & Wang, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.111 · Cited by 45
“With the advent of large scale datasets such as SQuAD (Rajpurkar et al., 2016, 2018), RACE (Lai et al., 2017), and Natural Questions (Kwiatkowski et al., 2019; Lee et al., 2019) on the open domain, machine reading comprehension (MRC) has become a hot topic in the natural language processing field.”
MethodSelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. J. F. (2023) · arXiv · doi:10.48550/arxiv.2303.08896 · Cited by 38
“The answering system A is Longformer (Beltagy et al., 2020) fine-tuned to the RACE dataset.”
BackgroundClinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37
ManyModalQA: Modality Disambiguation and QA over Diverse Inputs
Hannan, D., Jain, A., & Bansal, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6294 · Cited by 37
Chung, H.-L., Chan, Y.-H., & Fan, Y.-C. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.393 · Cited by 46
A review of public datasets in question answering research
Cambazoglu, B. B., Sanderson, M., Scholer, F., & Croft, B. (2020) · SIGIR Forum · doi:10.1145/3483382.3483389 · Cited by 51
“…et al., 2015], CBT [Hill et al., 2016], WDW [Onishi et al., 2016]), where the goal is to predict the missing word in a given piece of text, or multi-choice tasks (e.g., RACE [Lai et al., 2017], MCTest [Richardson et al., 2013]), where the answers are selected from a small set of candidates.”
BackgroundDeep learning-based question answering: a survey
Abdel-Nabi, H., Awajan, A., & Ali, M. Z. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01783-5 · Cited by 54
“” It is a large-scale multiple-choice reading comprehension dataset [146].”
BackgroundAttack Prompt Generation for Red Teaming and Defending Large Language Models
Deng, B., Wang, W., Feng, F., Deng, Y., Wang, Q., & He, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.143 · Cited by 177
New Vietnamese Corpus for Machine Reading Comprehension of Health News Articles
Nguyen, K. V., Huynh, T. V., Nguyen, D.-V., Nguyen, A. G.-T., & Nguyen, N. L.-T. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3527631 · Cited by 55
“, 2016), reading comprehension with multiple-choice (Richardson et al., 2013; Lai et al., 2017), and conversation-based reading comprehension (Reddy et al.”
BackgroundEvaluating Pretrained Transformer-based Models on the Task of Fine-Grained Named Entity Recognition
Lothritz, C., Allix, K., Veiber, L., Bissyandé, T. F., & Klein, J. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.334 · Cited by 51
“Liu et al. (2019) reported that RoBERTa outperforms BERT on the GLUE, SQuAD, and RACE benchmarks.”
MethodConfliBERT: A Pre-trained Language Model for Political Conflict and Violence
Hu, Y., Hosseini, M., Parolin, E. S., Osorio, J., Khan, L. R., Brandt, P. T., & D'Orazio, V. J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.400 · Cited by 90
“The introduction of comprehensive benchmarks accelerated the development of pre-trained language models in the general NLP domain (Wang et al., 2018, 2019; Rajpurkar et al., 2018; Lai et al., 2017) and biomedical applications (Peng et al., 2019; Gu et al., 2021).”
BackgroundCo-Attention Hierarchical Network: Generating Coherent Long Distractors for Reading Comprehension
Zhou, X., Luo, S., & Wu, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6522 · Cited by 34
ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters
Abujabal, A., Roy, R. S., Yahya, M., & Weikum, G. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1027 · Cited by 56
“Reading comprehension (RC) is a recently introduced task, where the goal is to answer a question from a given textual paragraph (Kociský et al., 2017; Lai et al., 2017; Rajpurkar et al., 2016; Trischler et al., 2017; Yang et al., 2015).”
BackgroundWhat Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge
Richardson, K., & Sabharwal, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00331 · Cited by 48
“In line with our goal of obtaining insights into the strongest QA models, we first pre-trained our RoBERTa-large model on the RACE dataset (Lai et al., 2017), a recipe used by several leading models on science benchmarks.”
MethodXiao, D., Li, Y., Zhang, H., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.136 · Cited by 56
A Self-Training Method for Machine Reading Comprehension with Soft Evidence Extraction
Niu, Y., Jiao, F., Zhou, M., Yao, T., Xu, J., & Huang, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.361 · Cited by 40
Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts
Liu, H., Cui, L., Liu, J., & Zhang, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17580 · Cited by 76
Self-citationWhat Language Model to Train if You Have One Million GPU Hours?
Le Scao, T., Wang, T., Hesslow, D., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., Press, O., et al. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.54 · Cited by 33
Improving Question Answering with External KnowledgePreprint
Pan, X., Sun, K., Yu, D., Chen, J., Ji, H., Cardie, C., & Yu, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00993 · Cited by 33
“, 2019) on a large-scale multiple-choice machine reading comprehension dataset RACE (Lai et al., 2017) collected from English-as-a-foreign-language exams, which provides a ground truth reference document instead of a reference corpus for each question.”
BackgroundMethodInfluentialConvolutional Spatial Attention Model for Reading Comprehension with Multiple-Choice Questions
Chen, Z., Cui, Y., Ma, W., Wang, S., & Hu, G. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016276 · Cited by 35
“Also according to the analysis by Lai et al.(2017), a large portion of the questions in RACE need reasoning over various clues, which makes it more challenging and suitable to evaluate the ability of the reading comprehension systems.”
BackgroundMethodInfluentialCombining Similarity and Transformer Methods for Case Law Entailment
Rabelo, J. C. B., Kim, M., & Goebel, R. G. (2019) · International Conference on Artificial Intelligence and Law · doi:10.1145/3322640.3326741 · Cited by 35
“These models have achieved impressive results in a wide range of publicly available benchmarks of dierent common natural language tasks, such as RACE (reading comprehension) [20] , COPA (common sense reasoning) [29], CoLA (linguistic acceptability) [31] and RTE (textual entailment) [11] to name a few.”
BackgroundNguyen, K. V., Tran, K. V., Luu, S. T., Nguyen, A. G.-T., & Nguyen, N. L.-T. (2020) · IEEE Access · doi:10.1109/access.2020.3035701 · Cited by 41
“This method was also used as a baseline in other studies [2], [12], [14].”
BackgroundMethodResultInfluentialBridging Information-Seeking Human Gaze and Machine Reading Comprehension
Malmaud, J., Lévy, R., & Berzak, Y. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.11 · Cited by 30
Controllable Open-ended Question Generation with A New Question Type Ontology
Cao, S., & Wang, L. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.502 · Cited by 63
KagNet: Knowledge-Aware Graph Networks for Commonsense ReasoningPreprint
Lin, B. Y., Chen, X., Chen, J., & Ren, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02151 · Cited by 32
“2) there are also models using multi-task learning to transfer knowledge from other reading comprehension datasets, such as RACE (Lai et al., 2017) and”
BackgroundWhat does BERT Learn from Multiple-Choice Reading Comprehension Datasets?Preprint
Si, C., Wang, S., Kan, M., & Jiang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12391 · Cited by 58
“(Lai et al., 2017) Unlike other datasets where the passages are crowd-soured, the passages in this dataset are collected from the English exams for middle and high school Chinese students.”
BackgroundExploring Summarization to Enhance Headline Stance Detection
Sepúlveda-Torres, R., Vicente, M., Saquete, E., Lloret, E., & Palomar, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-80599-9_22 · Cited by 43
Review on Neural Question Generation for Education Purposes
Faraby, S. A., Adiwijaya, A., & Romadhony, A. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00374-x · Cited by 58
“Additionally, RACE includes general-style questions such as "What would be the best title for the paragraph?"”
InfluentialDomain-agnostic Question-Answering with Adversarial Training
Lee, S., Kim, D., & Park, J. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5826 · Cited by 50
“, 2018), RACE (RA) (Lai et al., 2017), RelationExtraction (RE) (Levy et al.”
MethodRead, Retrospect, Select: An MRC Framework to Short Text Entity Linking
Gu, Y., Qu, X., Wang, Z., Huai, B., Yuan, N. J., & Gui, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17528 · Cited by 34
Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models
Lin, J., Zou, J., & Ding, N. (2021) · doi:10.18653/v1/2021.acl-short.43 · Cited by 59
“sic RACE dataset (Lai et al., 2017), which requires MRC models to answer multiple-choice reading comprehension questions based on a passage.”
MethodInfluentialDual Co-Matching Network for Multi-choice Reading ComprehensionPreprint
Zhang, S., Zhao, H., Wu, Y., Zhang, Z., Zhou, X., & Zhou, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09381 · Cited by 31
“With the built matching representations M p q , M p o , M q o for three sequence pairs, we concatenate them as the final representation C ∈ R 3 l for each passage-question-option triplet.”
BackgroundMethodAn Overview of Neural Network CompressionPreprint
O’Neill, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03669 · Cited by 180
“5.5% the size of original BERT and has produced state of the art results on top NLP benchmarks such as GLUE [184], SQuAD [149] and RACE [98].”
BackgroundInfluentialTweetBERT: A Pretrained Language Representation Model for Twitter Text AnalysisPreprint
Qudar, M. M. A., & Mago, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11091 · Cited by 65
“English passages and 100,000 questions [34].”
Prompting Techniques for Secure Code Generation: A Systematic Investigation
Tony, C., Ferreyra, N. E. D., Mutas, M., Dhiff, S., & Scandariato, R. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3722108 · Cited by 109
“…A R C [ 8 ] , U n i fi e d Q A [ 46 ] , O p e n B oo k Q A [ 66 ] , C o Q A [ 80 ] , Q u A C [ 15 ] , D R O P [ 21 ] , S Q u A D [ 79 ] , R A C E [ 52 ] , 2020 N e u r I P S 24786 F e w - s h o t S D L S U P E R G L U E [ 99 ] , R T E [ 11 ] , A N L I [ 68 ] , S A T a n a l o g y [ 97 ] R e y n o…”
Ni, J., Zhu, C., Chen, W., & McAuley, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1030 · Cited by 47
An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering
Longpre, S., Lu, Y., Tu, Z., & DuBois, C. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5829 · Cited by 30
“As a requisite first step to a representation that generalizes, transfer learning (in the form of large pre-trained language models such as Peters et al. (2018); Howard and Ruder (2018); Devlin et al.”
BackgroundVlad, G.-A., Zaharia, G.-E., Cercel, D.-C., Chiru, C.-G., & Trăușan-Matu, Ș. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.160 · Cited by 40
Unsupervised Energy-based Adversarial Domain Adaptation for Cross-domain Text Classification
Zou, H., Yang, J., & Wu, X. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.103 · Cited by 42
“Large pre-trained language models, e.g. BERT (Devlin et al., 2019), XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019b), achieve outstanding results on challenging NLP benchmarks, i.e. GLUE (Wang et al., 2018), RACE (Lai et al., 2017), and SQuAD (Rajpurkar et al., 2016).”
BackgroundFrom LSAT: The Progress and Challenges of Complex Reasoning
Wang, S., Liu, Z., Zhong, W., Zhou, M., Wei, Z., Chen, Z., & Duan, N. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3164218 · Cited by 82
Qu, F., Jia, X., & Wu, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.202 · Cited by 30
“To alleviate the above limitations, we propose to perform QAG on RACE (Lai et al., 2017) .”
MethodMulti-range Reasoning for Machine ComprehensionPreprint
Tay, Y., Tuan, L. A., & Hui, S. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09074 · Cited by 40
“Competitor result are reported from [18, 37].”
BackgroundMethodInfluentialTransfer Learning from Transformers to Fake News Challenge Stance Detection (FNC-1) TaskPreprint
Slovikovskaya, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14353 · Cited by 66
How Additional Knowledge can Improve Natural Language Commonsense Question Answering?Preprint
Mitra, A., Banerjee, P., Pal, K. K., Mishra, S. R., & Baral, C. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.08855 · Cited by 63
“Another challenging QA task is when the multiple-choice questions do not have sufficient knowledge to answer correctly given a passage, context or options, as in ARC (Clark et al., 2018), RACE (Lai et al., 2017), and OpenBook QA (Mihaylov et al., 2018).”
BackgroundZhao, Z., Zhou, G., Xie, Z., Wu, L., & Huang, J. X. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3658673 · Cited by 29
“The RACE dataset is a large-scale collection of data compiled by Lai et al. [17] from English exams administered to middle and high school students in China.”
MethodInteractive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · arXiv · doi:10.48550/arxiv.2208.07852 · Cited by 204
“The RACE dataset [14] is a multiple-choice reading comprehension dataset built from English examinations used for benchmarking this task.”
Jing, Y., Xiong, D., & Yan, Z. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1249 · Cited by 41
“To address these limitations, a number of datasets have be built recently, such as MS MARCO (Nguyen et al., 2016), DuReader (He et al., 2018), TriviaQA (Joshi et al., 2017), RACE (Lai et al., 2017), NarrativeQA (Kocisky et al., 2018), SQuAD2.”
MethodBioMRC: A Dataset for Biomedical Machine Reading Comprehension
Pappas, D., Stavropoulos, P., Androutsopoulos, I., & McDonald, R. (2020) · Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing · doi:10.18653/v1/2020.bionlp-1.15 · Cited by 29
EduQG: A Multi-Format Multiple-Choice Dataset for the Educational Domain
Hadifar, A., Bitew, S. K., Deleu, J., Develder, C., & Demeester, T. (2023) · IEEE Access · doi:10.1109/access.2023.3248790 · Cited by 49
“Different variants of T5 devised to generate distractors include: (i) T5 finetuned on RACE [60], (ii) T5 finetuned on EduQG, and (iii) T5 multi-stage finetuned, first on RACE and then EduQG (RACE → EduQG).”
InfluentialR-Trans: RNN Transformer Network for Chinese Machine Reading Comprehension
Liu, S., Zhang, S., Zhang, X., & Wang, H. (2019) · IEEE Access · doi:10.1109/access.2019.2901547 · Cited by 33
“‘‘RACE: Large-scale reading comprehension dataset from examinations.’’”
BackgroundLi, K., Xian, X., Wang, J., & Yu, N. (2018) · Applied Surface Science · doi:10.1016/j.apsusc.2018.11.214 · Cited by 32
“Like the CLOTH dataset [99], RACE [36] is also collected from English exams for middleschool and high-school Chinese students.”
BackgroundMethodInfluentialProQA: Structural Prompt-based Pre-training for Unified Question Answering
Zhong, W., Gao, Y., Ding, N., Qin, Y., Liu, Z., Zhou, M., Wang, J., Yin, J. P., & Duan, N. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.313 · Cited by 41
Learning to Reuse Distractors to Support Multiple-Choice Question Generation in Education
Bitew, S. K., Hadifar, A., Sterckx, L., Deleu, J., Develder, C., & Demeester, T. (2022) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2022.3226523 · Cited by 51
“There have also been a number of works on generating distractors in the context of machine comprehension [40].”
BackgroundPerformance analysis of large language models in the domain of legal argument mining
Zubaer, A. A., Granitzer, M., & Mitrović, J. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1278796 · Cited by 62
“Development of pre-trained language models (PLMs) like BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has been utilized to achieve state-of-the-art results on diverse benchmark datasets,e.g. GLUE (Wang et al., 2018), SQuAD (Rajpurkar et al., 2016), and RACE (Lai et al., 2017).”
Huang, Z., Shen, Y., Li, X., Wei, Y., Cheng, G., Zhou, L., Dai, X., & Qu, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1597 · Cited by 42
“the answer can be extracted or abstracted from the document (Rajpurkar et al., 2016; Nguyen et al., 2016; Lai et al., 2017).”
Background
Metadata
- Authors
- Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 785-794
- OpenAlex
- W2606964149
- Cited by
- 1,044
- References
- 19
- Citations by year
- 2026: 37 · 2025: 75 · 2024: 86 · 2023: 145 · 2022: 136 · 2021: 216 · 2020: 169 · 2019: 138 · 2018: 38 · 2017: 3
- FWCI
- 42.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750, FA8750-12-2-0342)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, machine reading comprehension, benchmark dataset, English examinations, Chinese students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.