RACE: Large-scale ReAding Comprehension Dataset From Examinations
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task.Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning.In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%).We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [112]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, to our knowledge, most of the existing datasets are not of direct relevance to teaching and learning, except for RACE (Lai, Xie, Liu, Yang, & Hovy, 2017) and LearningQ (Chen, Yang, Hauff, & Houben, 2018).”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Lai, Xie, Liu, Yang, & Hovy, 2017
The cited source says what the claim attributes to it.
In the citing paper
“On RACE-h and RACE-m (Lai et al., 2017), Chinchilla greatly outperforms Gopher , improving accuracy by more than 10% in both cases-see Table 7.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Lai et al., 2017
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We consider only the case of human-readable prompts to large models, which contrasts with methods such as prompt-tuning [16,20], which learns a continuous prompt embedding, and auto-prompting [33], which attempts to search for prompts from scratch, both of which require a training step.”
From Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We choose word embedding size d = 100 and use the 100-dimensional Glove word embedding (Pennington et al., 2014) as embedding initialization.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Instead of evaluating the accuracy, researchers need to use F1 score, BLEU (Papineni et al., 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the questions come without candidate spans.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 8×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
MethodResultInfluentialTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 3×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 1×
“, 2016) and recently proposed TriviaQA (Joshi et al., 2017).”
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578 · cited 2×
“, 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the”
BackgroundSelf-citationNewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 11×
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
BackgroundMethodResultInfluentialMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610 · cited 4×
“Although efforts have been made with a similar motivation, including the MCTest dataset created by (Richardson et al., 2013) (containing 500 passages and 2000 questions) and several others (Peñas et al.”
BackgroundMethodInfluentialA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491 · cited 11×
“However, both require limited reasoning ability (Chen et al., 2016).”
BackgroundMethodInfluential
Cited by
Items 901–1,000 of 1,036
Unsupervised Pronoun Resolution via Masked Noun-Phrase PredictionPreprint
Shen, M., Banerjee, P., & Baral, C. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12392
“Recent study (Khot et al., 2020) has shown that finetuning a RACE-finetuned (Lai et al., 2017) RoBERTa model as a start point is much more stable than directly finetuning a RoBERTa model from scratch, we follow the same strategy to start fine-tuning a RACE-finetuned RoBERTa-large model on all synthetic…”
Prediction or Comparison: Toward Interpretable Qualitative Reasoning
Ren, M., Huang, H., & Gao, Y. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.59
Answering Chinese Elementary School Social Study Multiple Choice QuestionsPreprint
Lee, D., Liang, C.-C., & Su, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.02893
“Afterwards, the neural network models are widely adopted due to the availability of large data-sets (such as RACE [13] and SNLI [14]).”
MethodAudio-Oriented Multimodal Machine Comprehension: Task, Dataset and ModelPreprint
Huang, Z., Liu, F., Wu, X., Ge, S., Wang, H., Fan, W., & Zou, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.01571
“There are four types of Machine Comprehension (MC) (Chen 2018), including cloze-style (Hermann et al. 2015), multi-choices (Lai et al. 2017), span extraction (Rajpurkar et al. 2016; Rajpurkar, Jia, and Liang 2018), and generative format (Kociský et al. 2018).”
InfluentialChen, Z., Lei, Y., Liu, P., & Guo, G. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.110
EviDR: Evidence-Emphasized Discrete Reasoning for Reasoning Machine Reading ComprehensionPreprint
Zhou, Y., Bao, J., Sun, H., Liang, J., Wu, Y., He, X., Zhou, B., & Zhao, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.07994
“Most of these studies focus on MRC datasets which have been released around span extraction, e.g., SQUAD [19], conversational state tracking, e.g., CoQA [21] and QuAC [4], passage retrieval, e.g., HotpotQA [24], multi-choice selection, e.g., RACE [14] and CommonsenseQA [23], and answer generation, e.g., MS-MARCO [18]…”
BackgroundGeoCQA: A Large-Scale Geography-Domain Chinese Question Answering Dataset from Examination
Cui, Z. S., Wang, B., & Ju, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88483-3_13
A Framework for Rationale Extraction for Deep QA modelsPreprint
Ramnath, S., Nema, P., Sahni, D., & Khapra, M. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04620
“A wide variety of Deep QA models (Seo et al., 2016; Xiong et al., 2016; Yu et al., 2018) have been introduced to solve RCQA tasks across various datasets (Rajpurkar et al., 2016; Lai et al., 2017; Nguyen et al., 2016; Joshi et al., 2017).”
BackgroundMethodممدوح, د. (2021) · مجلة البحث في التربية وعلم النفس · doi:10.21608/mathj.2021.90384.1132
Diving into Text Representation Learning with Deep Hashing
Zou, A., Hao, W., Jin, D., & Tian, Y. (2021) · International Conference on Electronic Information Engineering and Computer Science (EIECS) · doi:10.1109/eiecs53707.2021.9587985
Extract, Integrate, Compete: Towards Verification Style Reading ComprehensionPreprint
Zhang, C., Lai, Y., Feng, Y., & Zhao, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05149 · Cited by 1
“Lastly, the passages in VGaokao, which contain 1,159 Chinese characters on average, are approximately 3 times longer than that of RACE (Lai et al., 2017) and 10 times longer than that of C3 (Sun et al.”
BackgroundMethodInfluentialCorporate Bankruptcy Prediction with BERT Model
Kim, A., & Yoon, S. (2021)
CodeQA: A Question Answering Dataset for Source Code ComprehensionPreprint
Liu, C., & Wan, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08365 · Cited by 18
“A few examples of QA datasets in each category are CNN & Daily Mail (Hermann et al., 2015), RACE (Lai et al., 2017), SQuAD (Rajpurkar et al., 2016), MS MARCO (Nguyen et al., 2016).”
Muppet: Massive Multi-task Representations with Pre-FinetuningPreprint
Aghajanyan, A., Gupta, A., Shrivastava, A., Chen, X., Zettlemoyer, L., & Gupta, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11038 · Cited by 109
“The pre-finetuning scale is reported in terms of the number of datasets. upon prior state of the art methods for RTE (Ben-tivogli et al., 2009) and HellaSWAG (Zellers et al., 2019), as well as improve upon vanilla pre-trained representations for MNLI (Williams et al., 2018a), SQuAD (Rajpurkar et al., 2016a), BoolQ (Cla…”
BackgroundMethodInfluentialSkeIn: Sketchy-Intensive Reading Comprehension Model for Multi-choice Biomedical Questions
Li, J., Zhong, S., Chen, K., & Li, T. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91415-8_47
Task Optimization Leads to Human-like Top-down and Bottom-up Attention during Reading ComprehensionPreprint
Zou, J., & Ding, N. (2021) · Research Square · doi:10.21203/rs.3.rs-1034025/v1
Automatically Solving Elementary Science Questions: A Survey
Nagdev, S., Radke, M. A., & Ramanath, M. (2022) · IETE Technical Review · doi:10.1080/02564602.2022.2048716
“Formultiple choice type questions,MCTest [64], RACE [65], OpenBookQA/OBQA [66], ARC [41], QASC [67], CommonsenseQA/CQA [68], PIQA [69], SIQA [70], and Winogrande [71] are used.”
BackgroundMethodInfluentialJeong, S., Oh, D., Park, K., & Lim, H. (2022) · Applied Sciences · doi:10.3390/app12094099
“RACE [1] and SQuAD [2] are passage-based reading comprehension datasets.”
BackgroundMethodIncorporating Option and Out-of-domain Knowledge for Multi-choice Machine Reading Comprehension
Xu, Y. F., Shi, S., & Huang, H. (2021) · IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS) · doi:10.1109/ccis53392.2021.9754687
Tsuchiya, M., & Watarai, T. (2022) · Transactions of the Japanese Society for Artificial Intelligence · doi:10.1527/tjsai.37-4_a-lc3
Lin, Y., Liang, D., & Kang, X. (2022) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1145/3532213.3532241
“3532241 MRC such as the form of RACE [2], which selects the correct answer from the candidates under the condition of given articles and questions.”
MethodInfluentialKang, D., Lee, S., Park, E., & Na, S. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.147
“…(Liu et al., 2019), dynamic masking in the masked language modeling (MLM) objective dynamically while revisiting the next sentence prediction owing to its uncertain effectiveness has achieved promising results on GLUE (Wang et al., 2018), RACE (Lai et al., 2017), and SQuAD (Rajpurkar et al., 2016).”
BackgroundMultiple Choice Question Answering Using Attention Based Ranking and Transfer Learning
Kadam, N., & Kumar, M. A. (2022) · IEEE Region 10 Symposium (TENSYMP) · doi:10.1109/tensymp54529.2022.9864511 · Cited by 2
“Improving answer verifier discriminator The best AVD model exists is a Trained on BERt architecture on the RACE dataset [18].”
MethodInfluentialPirozelli, P., José Marcos, M., Igor, S., Cação, F. N., Peres, S. M., Brandão, A. A. F., R., C. A. H., & Cozman, F. G. (2022) · Research Square · doi:10.21203/rs.3.rs-2046889/v1
ViMRC - VLSP 2021: Context-Aware Answer Extraction in Vietnamese Question Answering
Le, T. T. H. (2022) · VNU Journal of Science Computer Science and Communication Engineering · doi:10.25073/2588-1086/vnucsce.316
“Machine Reading Comprehension has attracted significant attention from researchers, with the release of many task-specific datasets [5, 6, 7].”
BackgroundNumerical reasoning based on transformer
Xü, W. (2022) · International Conference on Computer Science and Communication Technology (ICCSCT) · doi:10.1117/12.2661770
U3E: Unsupervised and Erasure-based Evidence Extraction for Machine Reading Comprehension
He, S., Shi, S., Wu, C., & Huang, H. (2022) · IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS) · doi:10.1109/ccis57298.2022.10016420 · Cited by 1
Inter- and Intra-sentence Interaction Network for Multi-choice Reading Comprehension
Xu, X., Tohti, T., & Hamdulla, A. (2022) · International Conference on Virtual Reality, Human-Computer Interaction and Artificial Intelligence (VRHCIAI) · doi:10.1109/vrhciai57205.2022.00018
ЗАГАЛЬНОДОСТУПНІ НАБОРИ ДАНИХ ТА МЕТРИКИ ДЛЯ СПРИЯННЯ ДОСЛІДЖЕННЯМ СИСТЕМ НАДАННЯ ВІДПОВІДЕЙ
Vyshniak, M. Y., & Pyrozhenko, M. Y. (2023) · Таврійський науковий вісник Серія Технічні науки · doi:10.32851/tnv-tech.2023.1.2 · Cited by 2
Human Attention during Goal-directed Reading Comprehension Relies on Task OptimizationPreprint
Zou, J., Zhang, Y., Li, J., Tian, X., & Ding, N. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.04.25.538252
SADE: A Self-Adaptive Expert for Multi-Dataset Question Answering
Peng, Y., Wang, Q., Mao, Z., & Zhang, Y. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097208
Human Attention during Goal-directed Reading Comprehension Relies on Task OptimizationPreprint
Zou, J., Zhang, Y., Li, J., Tian, X., & Ding, N. (2023) · eLife · doi:10.7554/elife.87197.1 · Cited by 1
MSQA: An Unsupervised Domain-Adapted Question Answering Method Based on Multiple Source Domains
Peng, T., Zhang, X., Huang, J., Liu, J., Hu, X., & He, R. (2023) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops56833.2023.10150310
“The model performance is validated on five out-of-domain datasets, which are DROP [4], RACE [13], BioASQ [33], TextbookQA”
MethodInfluentialTerm Similarity-aware Extensive and Intensive Reading For Multiple Choice Question Answering
Li, X., Zhang, J., & Ma, J. (2022) · IEEE Smart World Congress (SWC) · doi:10.1109/smartworld-uic-atc-scalcom-digitaltwin-pricomp-metaverse56740.2022.00179
“However, evidence sentences for multiple choice tasks are usually multiple sentences [3], [24].”
BackgroundMethodInfluentialMeng, F., He, J.-F., Xu, X., Ya-juan, Z., & Sun, L. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-4752-2_37
JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions
Yu, M., Gu, Y., Guo, X., Feng, Y., Zhu, X., Greenspan, M., Campbell, M. S., & Gan, C. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.713
Learning Invariant Representation Improves Robustness for MRC Models
Hai, Y., Liang, W., Meng, H., Liu, T., & Wang, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.241
“We set the learning rate lr to 3 e − 5 for all models on all datasets, excepting that lr of all models is 2 e − 5 for ReClor, lr of RoBERTa is 3e-6 on RACE and NP-BoolQ, and ALBERT has lr 1e-5 for RACE, BoolQ and NP-BoolQ.”
MethodInfluentialSentence and Document Representation Learning
Ding, N., Lin, Y., Liu, Z., & Sun, M. (2023) · Springer eBooks · doi:10.1007/978-981-99-1600-9_4 · Cited by 2
Sanches, M. F., de Sá, J. M. C., de Souza, R. R., Souza, A. M. D., dos Reis, J. C., & Villas, L. A. (2023) · SN Computer Science · doi:10.1007/s42979-023-02148-7 · Cited by 1
“SN Computer Science achieves state-of-the-art results on GLUE [10], RACE [11] and SQuAD [12], was trained with BooksCorpus [7], CCNEWS [13], OPENWEBTEXT [14] and STORIES [15] datasets.”
MethodMachine Reading Comprehension for Multiclass Questions on Thai Corpus
Lapchaicharoenkit, T. (2019) · Chula Digiverse · doi:10.58837/chula.the.2019.168
“[13] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy, "RACE: Large-scale ReAding Comprehension Dataset From Examinations," in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, sep 2017: Association for Computational Linguistics, pp. 785-794.”
BackgroundMethodInfluentialTraining NLI Models Through Universal Adversarial Attack
Lin, J., Liu, W., Zou, J., & Ding, N. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-6207-5_19
“For example, in multiple-choice task, e.g., RACE (Lai et al., 2017), the model can be fooled to choose a certain biased option as the answer.”
MethodMulti-hop question answering using sparse graphs
Hemmati, N., & Ghassem-Sani, G. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107128 · Cited by 3
Question difficulty estimation via enhanced directional modality association transformer
Kim, B.-M., Park, G.-M., & Park, S.-B. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04988-5
“The reason why RoBERTa is employed for this purpose is that it achieves higher performance than BERT in many QA tasks such as SQuAD [32], RACE [24], and GLUE [38].”
MethodAn efficient confusing choices decoupling framework for multi-choice tasks over texts
Wang, Y., Bao, J., Duan, C., Wu, Y., He, X., Zhu, C., & Zhao, T. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08795-4 · Cited by 1
“Multi-Choice Question Answering We select the widely used multi-choice question-answering dataset, RACE [13], to evaluate the proposed mechanisms in a more complex application.”
InfluentialPOSPAN: Position-Constrained Span Masking for Language Model Pre-training
Zhang, Z., Shen, L., Zhao, Y., Chen, M., & He, X. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615197
Zou, J., Zhang, Y., Li, J., Tian, X., & Ding, N. (2023) · eLife · doi:10.7554/elife.87197.2.sa4
Ancient Chinese Machine Reading Comprehension Exception Question Dataset with a Non-trivial Model
Rao, D., Huang, G., & Jiang, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-7022-3_14
FlashMe: Automatic Flashcard Generation
Nair, N., Pikle, S., Save, S., Varghese, R., & Sonawane, K. V. (2023) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt56998.2023.10308164
“The RACE dataset contained four essential characteristics: article (a paragraph of context), answer (a character representing the correct option), query, and options (answer) [14].”
BackgroundHuang, S. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-8184-7_5
Keyword Provision Question Generation for Facilitating Educational Reading Comprehension Preparation
Chan, Y.-H., Chung, H.-L., & Fan, Y.-C. (2022) · International Conference on Natural Language Generation · doi:10.18653/v1/2022.inlg-main.15
“300 context paragraphs and the corresponding answers were randomly selected from the test set of EQG-RACE data (Jia et al., 2020).”
MethodInfluentialFew-shot Unified Question Answering: Tuning Models or Prompts?
Bansal, S., Yavuz, S., Pang, B., Bhat, M., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.550 · Cited by 3
Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models
Chen, W., Xu, X., Han, X., Lin, Y., Xie, R., Liu, Z., Sun, M., & Zhou, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.738
Machine Reading Comprehension using Case-based Reasoning
Thai, D., Agarwal, D., Chaudhary, M., Zhao, W., Das, R., Lee, J. Y., Hajishirzi, H., Zaheer, M., & McCallum, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.564 · Cited by 6
Explicit Alignment and Many-to-many Entailment Based Reasoning for Conversational Machine Reading
Luo, Y., Tian, S., Yuan, C., & Wang, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.866 · Cited by 1
Makahleh, M., Obiedat, R., & Quwaider, M. (2023) · International Conference on Social Networks Analysis, Management and Security (SNAMS) · doi:10.1109/snams60348.2023.10375456
“For instance, benchmark datasets like SQuAD (Stanford Question Answering Dataset) [6] and RACE (Reading Comprehension from Examinations) [7] each provide over 10.000 question-answer pairs, offering extensive and diverse content for training models to achieve robust comprehension.”
BackgroundMethodInfluentialWorldCALL2023: Official Conference Proceedings
(2024) · WorldCALL official conference proceedings · doi:10.22492/issn.2759-1182.2023 · Cited by 10
SpanBERT-based Multilayer Fusion Model for Extractive Reading Comprehension
Zhang, P., He, L., & Xi, D. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.0150149 · Cited by 1
“Representative datasets for this type of task include MCTest [13] and RACE [14], while representative models include DCMN+ [15].”
MethodA Unified Recombination and Adversarial Framework for Machine Reading Comprehension
Liao, J., Wu, Y., Su, Y. R., & Dong, Z. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-0837-6_7
MDA: Masked Language Modeling Meets Data Augmentation for Multiple Choice Question Answering
Yao, X., Huang, Z., Ma, J., Chen, J., & Yang, J. (2023) · IEEE Smart World Congress (SWC) · doi:10.1109/swc57546.2023.10449210
“M/H) [14] and DREAM [15].”
CDUMA: An Adaptive Approach for Mitigating Confounder for MCQA
Li, S., Lu, C., Bai, G., Liu, Y., Liu, X., Zhang, Z., & Liu, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446341
“In order to answer these questions correctly, the model not only needs to infer from the passage[1] but also requires an understanding of common knowledge[2].”
BackgroundAttend All Options at Once: Full Context Input for Multi-choice Reading Comprehension
Wang, R., Verberne, S., & Spruit, M. R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56027-9_24
Contrastive Learning Based on AMR Graph for Logic Reasoning
He, Y., & Wang, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-1277-9_15
Indonesian generative chatbot model for student services using GPT
Priccilia, S., & Girsang, A. S. (2024) · International Journal of Informatics and Communication Technology (IJ-ICT) · doi:10.11591/ijict.v13i1.pp50-56 · Cited by 2
“GPT notably outperformed other models in question-answering tasks on datasets like RACE [11] and Story Cloze [12] with improvements of 5.7% and 8.9%, respectively.”
MethodInfluentialNumerical reasoning reading comprehension on Vietnamese COVID-19 news: task, corpus, and challenges
Nguyen, K. V., Le, T. V., & Pham-Phuc, T. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09744-5
“…span-extraction reading comprehension tasks (SQuAD-style) [10, 18], abstractive reading comprehension tasks [24, 25], reading comprehension with multiple choice questions [6, 12, 26], numerical reasoning reading comprehension tasks [9], and expanding MRC research to low-resource languages [5, 27].”
Automated Multi-Choice Question Answering System using Natural Language Processing
Madoo, R., & Hole, V. (2024) · International Conference for Innovation in Technology (INOCON) · doi:10.1109/inocon60754.2024.10511814
“The Race dataset serves as a valuable resource for large-scale reading comprehension tasks, offering an extensive collection of over 28,000 passages and nearly 100,000 questions [28].”
BackgroundA Large Financial Chat ModelPreprint
Ding, F. (2024) · doi:10.33774/coe-2024-lsc94
Choi, S., Huang, J., & Kwon, O. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63031-6_26
Tasks for LLMs and Their Evaluation
Ostapuk, N., & Audiffren, J. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_6 · Cited by 2
Transformer-Based Biomedical Text Extraction
Khalaf, R. A., & Bernasconi, A. (2024) · Elsevier eBooks · doi:10.1016/b978-0-323-95502-7.00036-1
Chinese cyber-violent Speech Detection and Analysis Based on Pre-trained Model
Zhou, S. (2024) · International Conference on Computing, Networks and Internet of Things (CNIOT) · doi:10.1145/3670105.3670179
Exploring the Perception of Indian Educators Towards Use of Generative AI Tools for Assessment
Pillai, N. S., & Arora, P. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-2418-9.ch002
Phan, K. T., Le, T., & Tran, N. T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70816-9_18
ConClue: Conditional Clue Extraction for Multiple Choice Question Answering
Yang, W., Yang, J., Li, W., & Guo, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70552-6_11
MADE: A Universal Fine-Tuning Framework to Enhance Robustness of Machine Reading Comprehension
Cao, Y., & Wang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72359-9_15
Lan, J., Liang, R., Liu, H., He, C., & Hao, T. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-7007-6_9
Self-citationGenerative Artificial Intelligence in Teaching, Learning, and Assessment: An Indian Perspective
Arora, P., & Pillai, N. S. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-5518-3.ch004 · Cited by 1
A Comparative Study on Pre-Trained Models Based on BERT
Zhang, M. (2024) · doi:10.1109/icnlp60986.2024.10692659
“Reference [16] presents a new English benchmark for the evaluation of models in solving reading comprehension problems.”
BackgroundSönmez, Y., & Özyurt, F. (2024) · The Springer series in applied machine learning · doi:10.1007/978-3-031-69499-8_11
ELO-Mask: Effective and Layerwise Optimization of Mask for Sparse LLMs
Xiang, B., Wu, J., Han, X., Gu, Q., Chao, F., Yang, X. P., Wu, F., & Fu, X. (2024) · IEEE Access · doi:10.1109/access.2024.3498904
Qadg: Generating question–answer-distractors pairs for real examination
Zhou, H. Y., & Li, L. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10658-5
“In Table 1, we provide an example from the RACE [3] dataset, which includes a passage, a question stem, an answer, three distractors, and an example generated by our model after the ground truth.”
InfluentialMuLoR: a multi-graph contrastive network for logical reasoning
Xiao, J., Lin, G., Yu, X., & Li, P. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02286-1
Generative AI text summarization performance analysis prospects
Tsypliak, O., & Артемчук, В. О. (2024) · Modeling Control and Information Technologies · doi:10.31713/mcit.2024.037
HR-SKGs: Hyper-Relational Semantic Knowledge Graphs for Multi-hop Reading Comprehension
Zhang, Y., Cheng, B. H., & Chen, Y.-L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889591 · Cited by 1
“The rapid advancement of this task is largely attributed to the creation of various large-scale MRC datasets in recent years, such as CNN/Daily Mail [4], RACE [5], SQuAD [6], [7], and CoQA [8].”
KPQDG: a key sentence prompt-tuning for automatic question-distractor pairs generation
Yin, Z., & Li, L. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11149-x · Cited by 2
“In contrast, RACE [4] possesses such an advantage as it is derived from genuine Chinese high school and college entrance examination reading comprehension prompts.”
Towards Explainable Medical Machine Reading Comprehension With Rationale Generation
Li, D., He, S., Hu, B., & Chen, Q. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3555105 · Cited by 1
“M ACHINE reading comprehension (MRC) has received increasing attention in the natural language processing field with the release of large-scale datasets such as SQuAD [1], [2], RACE [3], and Natural Questions [4], [5] on the open domain.”
Binary mask tuning on gradient: Towards multi-data question answering
Gong, C., Wei, Z., Zhu, P., & Miao, D. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113505
ViEduQA: A New Vietnamese Dataset for Question Answer Generation in Education
Nguyen, T.-P., Le, H.-L., Hùng, P. Q., Huy, N. Q., Phan, X.-H., & Nguyen, M.-T. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4288-5_34
Imran, A., & Gopalakrishnan, K. (2025) · Apress eBooks · doi:10.1007/979-8-8688-0989-7_4 · Cited by 1
Wei, Y., & Li, L. (2024) · International Conference on Information Science and Education (ICISE-IE) · doi:10.1109/icise-ie64355.2024.11025434
“These datasets are: CODAH [11], Com2Sense [12], CosmosQA [13], HellaSWAG [14], PhysicalIQA [15], RACE-Middle [16], RiddleSense [17], SocialIQA [18], SWAG [19], and WinoGrade [20].”
Reddy, M. (2025) · International Conference on Artificial Intelligence and Machine Learning Applications Theme: Healthcare and Internet of Things (AIMLA) · doi:10.1109/aimla63829.2025.11040611
Preaching to the ChoIR: Lessons IR Should Share with AI
Demartini, G., Hauff, C., Lease, M., Mizzaro, S., Roitero, K., Sanderson, M., Scholer, F., Shah, C. S., Spina, D., Thomas, P., et al. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744612 · Cited by 1
“Common benchmark examples include ARC-C [32], CRASS [46], HellaSwag [178], MMLU [59], OpenBookQA [90], RACE [73], and SciQ [171].”
Wang, M., Liu, J., Gutiérrez-Basulto, V., Wang, L., & Pan, J. Z. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3588525
Exploring Adversarial Robustness of Masked Auto-Encoders
Luo, Y., Gao, J., Kuang, J., Qi, B., Chen, X., Zhao, Y., Li, F., & Liu, J. (2025) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc65474.2025.11090267
“Training with embedding-level adversarial attacks, which exploit the rich semantic information encoded in embeddings, has been shown to improve the performance more effectively than word-level attack methods [22], [23].”
GISA: Gradual Information Selection Attention for MCQ Difficulty Estimation
Ravikiran, M., Sharma, T., Bhavsar, A. V., & Saluja, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98414-3_14
Optimizing GPT-Based Distractor Generation for the Korean CSAT English ExamPreprint
Jung, C. Y., & Song, S. (2025) · Research Square · doi:10.21203/rs.3.rs-6680435/v1
Metadata
- Authors
- Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 785-794
- OpenAlex
- W2606964149
- Cited by
- 1,044
- References
- 19
- Citations by year
- 2026: 37 · 2025: 75 · 2024: 86 · 2023: 145 · 2022: 136 · 2021: 216 · 2020: 169 · 2019: 138 · 2018: 38 · 2017: 3
- FWCI
- 42.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750, FA8750-12-2-0342)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, machine reading comprehension, benchmark dataset, English examinations, Chinese students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.