Logiqa 2.0—An improved dataset for logical reasoning in natural language understanding
Abstract
NLP research on logical reasoning regains momentum with the recent releases of a handful of datasets, notably LogiQA and Reclor. Logical reasoning is exploited in many probing tasks over large Pre-trained Language Models (PLMs) and downstream tasks like question-answering and dialogue systems. In this paper, we release LogiQA 2.0. The dataset is an amendment and re-annotation of LogiQA in 2020, a large-scale logical reasoning reading comprehension dataset adapted from the Chinese Civil Service Examination. We increase the data size, refine the texts with manual translation by professionals, and improve the quality by removing items with distinctive cultural features like Chinese idioms. Furthermore, we conduct a fine-grained annotation on the dataset and turn it into a two-way natural language inference (NLI) task, resulting in 35k premise-hypothesis pairs with gold labels, making it the first large-scale NLI dataset for complex logical reasoning. Compared to Question Answering, Natural Language Inference excels in generalizability and helps downstream tasks better. We establish a baseline for logical reasoning in NLI and incite further research.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Continuous monitoring and bias testing are essential to identify and address biases that may emerge from AI's predictive patterns.
“Continuous monitoring and bias testing are essential to identify and address any biases that may emerge from AI's predictive patterns [220], [219].”
From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [220]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · see citing works
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · see citing works
Some Philosophical Problems from the Standpoint of Artificial Intelligence
McCarthy, J., & Hayes, P. J. (1981) · Elsevier eBooks · doi:10.1016/b978-0-934613-03-3.50033-7 · Cited by 3,296 · see citing works
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · see citing works
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · see citing works
Pre-Training With Whole Word Masking for Chinese BERT
Cui, Y., Che, W., Liu, T., Qin, B., & Yang, Z. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3124365 · Cited by 1,722 · see citing works
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · see citing works
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674 · see citing works
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944 · see citing works
Annotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907 · see citing works
Revisiting Pre-Trained Models for Chinese Natural Language Processing
Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., & Hu, G. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.58 · Cited by 900 · see citing works
Levesque, H. J., Davis, E., & Morgenstern, L. (2011) · Cited by 879 · see citing works
The logic theory machine-A complex information processing system
Newell, A., & Simon, H. (1956) · IEEE Transactions on Information Theory · doi:10.1109/tit.1956.1056797 · Cited by 846 · see citing works
The Third PASCAL Recognizing Textual Entailment Challenge
Giampiccolo, D., Magnini, B., Dagan, I., & Dolan, B. (2007) · ACL-PASCAL@ACL · doi:10.3115/1654536.1654538 · Cited by 801 · see citing works
A survey of human-in-the-loop for machine learning
Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., & He, L. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.05.014 · Cited by 800 · see citing works
McCarthy, J. (1960) · Cited by 799 · see citing works
A Unified MRC Framework for Named Entity Recognition
Li, X., Feng, J., Meng, Y., Han, Q., Wu, F., & Li, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.519 · Cited by 697 · see citing works
Hypothesis Only Baselines in Natural Language Inference
Poliak, A., Naradowsky, J., Haldar, A., Rudinger, R., & Durme, B. V. (2018) · International Workshop on Semantic Evaluation · doi:10.18653/v1/s18-2023 · Cited by 685 · see citing works
A SICK cure for the evaluation of compositional distributional semantic models
Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Raffaella Bernardi, B., & Zamparelli, R. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/39qdhuevzbqa · Cited by 680 · see citing works
The NarrativeQA Reading Comprehension Challenge
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00023 · Cited by 596 · cited 1× · see citing works
“DuoRC [83] and Narrative QA [84] raise challenges by introducing two passages about the same facts.”
BackgroundSocial IQa: Commonsense Reasoning about Social Interactions
Sap, M., Rashkin, H., Chen, D., LeBras, R., & Choi, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1454 · Cited by 588 · see citing works
A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1223 · Cited by 494 · see citing works
Zero-Shot Relation Extraction via Reading Comprehension
Levy, O., Seo, M., Choi, E., & Zettlemoyer, L. (2017) · Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) · doi:10.18653/v1/k17-1034 · Cited by 482 · cited 1× · see citing works
“There is a fundamental connection between machine reading comprehension and natural language inference [15], both tasks rely heavily on reasoning skills, and both are general because many NLP tasks can be cast into MRC [19], [20] or NLI [21].”
BackgroundZero-Shot Relation Extraction via Reading Comprehension
Levy, O., Seo, M., Choi, E., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1706.04115 · Cited by 482
Welbl, J., Stenetorp, P., & Riedel, S. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.1162/tacl_a_00021 · Cited by 459 · see citing works
A Robustly Optimized BERT Pre-training Approach with Post-training
Liu, Z., Lin, W., Shi, Y., & Zhao, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-84186-7_31 · Cited by 431 · see citing works
SciTaiL: A Textual Entailment Dataset from Science Question Answering
Khot, T., Sabharwal, A., & Clark, P. E. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12022 · Cited by 416 · see citing works
A Concise Introduction to Logic
O’Regan, G. (2018) · Springer eBooks · doi:10.1007/978-3-319-75844-2_6 · Cited by 401 · see citing works
Probing Neural Network Comprehension of Natural Language Arguments
Niven, T., & Kao, H.-Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1459 · Cited by 395 · see citing works
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1246 · Cited by 393 · cited 1× · see citing works
“Drop [85] tests discrete numerical reasoning over the context.”
BackgroundBoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 391 · see citing works
Natural Language Input for a Computer Problem Solving System
Bobrow, D. G. (1964) · Defense Technical Information Center (DTIC) · Cited by 326 · see citing works
Geva, M., Goldberg, Y., & Berant, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1107 · Cited by 284 · see citing works
Hutchinson, B., Smart, A., Hanna, A., Denton, R., Greer, C., Kjartansson, O., Barnes, P., & Mitchell, M. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445918 · Cited by 279 · see citing works
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
Huang, L., Bras, R. J. L., Bhagavatula, C., & Choi, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1243 · Cited by 278 · see citing works
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · arXiv · doi:10.48550/arxiv.1903.00161 · Cited by 238 · see citing works
Question Answering Using Enhanced Lexical Semantic Models
Yih, W.-T., Chang, M., Meek, C., & Pastusiak, A. (2013) · Cited by 221 · see citing works
Theories of the syllogism: A meta-analysis.
Khemlani, S., & Johnson-Laird, P. N. (2012) · Psychological Bulletin · doi:10.1037/a0026841 · Cited by 213 · see citing works
LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Jian, L., Cui, L., Liu, H., Huang, D., Wang, Y., & Zhang, Y. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/501 · Cited by 209 · see citing works
Artificial Intelligence, Logic and Formalizing Common Sense
McCarthy, J. (1989) · Springer eBooks · doi:10.1007/978-94-009-2448-2_6 · Cited by 208 · see citing works
Transforming Question Answering Datasets Into Natural Language Inference DatasetsPreprint
Demszky, D., Guu, K., & Liang, P. (2018) · arXiv.org · doi:10.48550/arxiv.1809.02922 · Cited by 192 · see citing works
ReClor: A Reading Comprehension Dataset Requiring Logical ReasoningPreprint
Yu, W., Jiang, Z., Dong, Y., & Feng, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04326 · Cited by 179 · see citing works
QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2022) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 177 · see citing works
Transformers as Soft Reasoners over Language
Clark, P., Tafjord, O., & Richardson, K. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/537 · Cited by 176 · see citing works
Natural logic for textual inference
MacCartney, B., & Manning, C. D. (2007) · doi:10.3115/1654536.1654575 · Cited by 158 · see citing works
CPT: a pre-trained unbalanced transformer for both Chinese language understanding and generation
Shao, Y., Geng, Z., Liu, Y., Dai, J., Yan, H., Yang, F., Li, Z., Bao, H., & Qiu, X. (2024) · Science China Information Sciences · doi:10.1007/s11432-021-3536-5 · Cited by 155 · see citing works
How Can We Accelerate Progress Towards Human-like Linguistic Generalization?
Linzen, T. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.465 · Cited by 148 · see citing works
Cann, R. (1993) · Cambridge University Press eBooks · doi:10.1017/cbo9781139166317 · Cited by 141 · see citing works
Two modes of mental representation and problem solution in syllogistic reasoning
Ford, M. (1995) · Cognition · doi:10.1016/0010-0277(94)00625-u · Cited by 133 · see citing works
Entailment as Few-Shot LearnerPreprint
Wang, S., Fang, H., Khabsa, M., Mao, H., & Ma, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14690 · Cited by 130 · see citing works
The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants
Habernal, I., Wachsmuth, H., Gurevych, I., & Stein, B. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1175 · Cited by 124 · see citing works
Abzianidze, L., Bjerva, J., Evang, K., Haagsma, H., van Noord, R., Ludmann, P., Nguyen, D.-D., & Bos, J. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2039 · Cited by 118 · see citing works
DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension
Saha, A., Aralikatte, R., Khapra, M. M., & Sankaranarayanan, K. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1156 · Cited by 111 · see citing works
CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text
Sinha, K., Sodhani, S., Dong, J., Pineau, J., & Hamilton, W. L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1458 · Cited by 110 · see citing works
MuTual: A Dataset for Multi-Turn Dialogue Reasoning
Cui, L., Wu, Y., Liu, S., Zhang, Y., & Zhou, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.130 · Cited by 110 · see citing works
Self-citationPulman, S. (1996) · Cited by 98 · see citing works
Explaining Answers with Entailment Trees
Dalvi, B. B., Jansen, P. A., Tafjord, O., Xie, Z., Smith, H., Pipatanangkura, L., & Clark, P. E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.585 · Cited by 96 · see citing works
MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
Ostermann, S., Modi, A., Roth, M., Thater, S., & Pinkal, M. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2x36g84ob95p · Cited by 95 · see citing works
Extracting Social Power Relationships from Natural Language
Bramsen, P., Escobar-Molano, M. L., Patel, A., & Alonso, R. D. H. (2011) · Cited by 90 · see citing works
MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script KnowledgePreprint
Ostermann, S., Modi, A., Roth, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.05223 · Cited by 68 · see citing works
Logic for natural language analysis
Pereira, F. C. N. (1982) · ERA · Cited by 66 · see citing works
Martín, A. G., Huertas‐Tato, J., Huertas-García, Á., Villar-Rodríguez, G., & Camacho, D. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109265 · Cited by 62 · see citing works
Can Neural Networks Understand Monotonicity Reasoning?
Yanaka, H., Mineshima, K., Bekki, D., Inui, K., Sekine, S., Abzianidze, L., & Bos, J. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4804 · Cited by 61 · see citing works
Pay Attention to the Ending:Strong Neural Baselines for the ROC Story Cloze Task
Cai, Z., Tu, L., & Gimpel, K. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2097 · Cited by 59 · see citing works
DAGN: Discourse-Aware Graph Network for Logical Reasoning
Huang, Y., Fang, M., Cao, Y. H., Wang, L., & Liang, X. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.467 · Cited by 59 · see citing works
Can NLI Models Verify QA Systems’ Predictions?
Chen, J., Choi, E., & Durrett, G. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.324 · Cited by 51 · see citing works
HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning
Yanaka, H., Mineshima, K., Bekki, D., Inui, K., Sekine, S., Abzianidze, L., & Bos, J. (2019) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s19-1027 · Cited by 49 · see citing works
Explaining The Efficacy of Counterfactually Augmented DataPreprint
Kaushik, D., Setlur, A., Hovy, E. H., & Lipton, Z. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02114 · Cited by 44 · see citing works
Exploring the Efficacy of Automatically Generated Counterfactuals for Sentiment Analysis
Yang, L., Li, J., Cunningham, P., Zhang, Y., Smyth, B., & Dong, R. (2021) · doi:10.18653/v1/2021.acl-long.26 · Cited by 43 · see citing works
Mishra, A., Patel, D., Vijayakumar, A., Li, X. L., Kapanipathi, P., & Talamadupula, K. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.104 · Cited by 39 · see citing works
Learning Logic Rules for Document-Level Relation Extraction
Ru, D., Sun, C., Feng, J., Qiu, L., Zhou, H., Zhang, W., Yu, Y. H., & Li, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.95 · Cited by 39 · see citing works
Combining Natural Logic and Shallow Reasoning for Question Answering
Angeli, G., Nayak, N., & Manning, C. D. (2016) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p16-1042 · Cited by 37 · see citing works
Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts
Liu, H., Cui, L., Liu, J., & Zhang, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17580 · Cited by 35 · see citing works
Utama, P. A., Bambrick, J., Moosavi, N. S., & Gurevych, I. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.199 · Cited by 28 · see citing works
ConjNLI: Natural Language Inference Over Conjunctive Sentences
Saha, S., Nie, Y., & Bansal, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.661 · Cited by 26 · see citing works
TaxiNLI: Taking a Ride up the NLU Hill
Joshi, P., Aditya, S., Sathe, A., & Choudhury, M. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.4 · Cited by 24 · see citing works
Cohen, V., & Gokaslan, A. (2020) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/3416063 · Cited by 22 · see citing works
A Rationale-Centric Framework for Human-in-the-loop Machine Learning
Lu, J., Yang, L., Namee, B. M., & Zhang, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.481 · Cited by 22 · see citing works
Logical reasoning in natural language: It is all about knowledge
Iwańska, L. (1993) · Minds and Machines · doi:10.1007/bf00974107 · Cited by 16 · see citing works
Textual Inference: getting logic from humans.
Kalouli, A.-L., Real, L., & de Paiva, V. (2017) · Cited by 12 · see citing works
Neural Natural Logic Inference for Interpretable Question Answering
Shi, J., Ding, X., Du, L., Liu, T., & Qin, B. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.298 · Cited by 12 · see citing works
Improving Machine Reading Comprehension with Contextualized Commonsense Knowledge
Sun, K., Yu, D., Chen, J., Yu, D., & Cardie, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.598 · Cited by 10 · see citing works
Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI
Sia, S., Belyy, A., Almahairi, A., Khabsa, M., Zettlemoyer, L., & Mathias, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26174 · Cited by 7 · see citing works
Reading Comprehension as Natural Language Inference: A Semantic AnalysisPreprint
Mishra, A., Patel, D., Vijayakumar, A., Li, X., Kapanipathi, P., & Talamadupula, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01713 · Cited by 4 · see citing works
Logical Reasoning for Task Oriented Dialogue Systems
Beygi, S., Fazel-Zarandi, M., Cervone, A., Krishnan, P., & Jonnalagadda, S. (2022) · Workshop on e-Commerce and NLP (ECNLP) · doi:10.18653/v1/2022.ecnlp-1.10 · Cited by 2 · see citing works
Exposing Length Divergence Bias of Textual Matching Models.Preprint
Jiang, L., Lyu, T., Meng, C., Lyu, X., & Yin, D. (2021) · arXiv (Cornell University) · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works
“…(NLU) : NLU is central to enhancing the comprehension and contextualization of human language in AI systems, and involves key capabilities such as semantic analysis, named entity recognition, sentiment analysis, textual entailment, and machine reading comprehension [190], [191], [192], [193].”
BackgroundSurvey of Cultural Awareness in Language Models: Text and Beyond
Pawar, S., Park, J., Jin, J., Arora, A., Myung, J., Yadav, S., Haznitrama, F. G., Song, I., Oh, A. H., & Augenstein, I. (2025) · Computational Linguistics · doi:10.1162/coli.a.14 · Cited by 77 · see citing works
A review: Insight into smart and sustainable ultra-precision machining augmented by intelligent IoT
Xu, Z., Zhu, T., Luo, F. L., Zhang, B., Poon, H., Yip, W. S., & To, S. (2024) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2024.03.008 · Cited by 44 · see citing works
An embedded end-to-end voice assistant
Lazzaroni, L., Bellotti, F., & Berta, R. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108998 · Cited by 22 · see citing works
A survey on large language models with multilingualism: recent advances and new frontiers
Huang, K., Mo, F., Zhang, X., Li, H., Li, Y., Zhang, Y., yi, W., Mao, Y., Liu, J., Xu, Y., et al. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11534-5 · Cited by 73 · see citing works
An extensive bibliometric analysis of artificial intelligence techniques from 2013 to 2023
Bajpai, A., Yadav, S., & Nagwani, N. K. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07021-3 · Cited by 20 · see citing works
Empower LlaMa 2 for Advanced Logical Reasoning in Natural Language UnderstandingPreprint
Watanabe, N., Kinasaka, K., & Nakamura, A. (2024) · Research Square · doi:10.21203/rs.3.rs-4333417/v1 · Cited by 5 · see citing works
Deng, Y., Qiu, X., Tan, X., Qu, C., Pan, J., Cheng, Y., Xu, Y., & Chu, W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887899 · Cited by 4 · see citing works
Chen, X., Chen, Z., & Cheng, S. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00041-2 · Cited by 6 · see citing works
Multi-choice machine reading comprehension benchmark datasets: A survey
Foolad, S., Kiani, K., & Rastgoo, R. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.114499 · Cited by 3 · see citing works
The Philosophy of Language Models
Millière, R., & Buckner, C. J. (2026) · Philosophy Compass · doi:10.1111/phc3.70095 · Cited by 3 · see citing works
Salsabil, A. A., Setiawan, E. B., & Kurniawan, I. (2024) · Kinetik Game Technology Information System Computer Network Computing Electronics and Control · doi:10.22219/kinetik.v9i2.1940 · Cited by 5 · see citing works
Bao, Q., Gendron, G., Peng, A. Y., Zhong, W., Tan, N. Ö., Chen, Y., Witbrock, M., & Liu, J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-6603-4_22 · Cited by 2 · see citing works
Toward Enhancing Reasoning Capabilities of LLMs: An Approach via Synthetic Logic Corpus
Morishita, T., Morio, G., Yamaguchi, A., & Sogawa, Y. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.520 · Cited by 1 · see citing works
Neuro-symbolic NLP: taxonomy, assessment, and directions
Chatzikyriakidis, S., & Lappin, S. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1797587 · Cited by 1 · see citing works
MLRQA: A Dataset with Multimodal Logical Reasoning Challenges
Xiao, J., Lin, G., & Li, P. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0122-6_1
MuLoR: a multi-graph contrastive network for logical reasoning
Xiao, J., Lin, G., Yu, X., & Li, P. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02286-1
Reasoning Capabilities and Invariability of Large Language Models
Raganato, A., Peñaloza, R., Viviani, M., & Pasi, G. (2024) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat62293.2024.00025
Improve Semantic Reasoning Ability by Combining Multi-task Learning and Deformation Model
Wang, D., & Yan, Z. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.05.021
FoVer: First-Order Logic Verification for Natural Language Reasoning
Pei, Y., Du, Y., & Jin, X. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.41
LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?Preprint
Ye, M., He, H., Zhong, Q., Zhang, J. J., Liu, J., & Du, B. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6055453
Hierarchical Attribution Analysis of Deficiencies in the Reasoning Ability of Large Language Models
Jiang, Z., Zhang, H., Wang, F., Yan, X., & Yang, Y. (2026) · International Conference on Computer, Artificial Intelligence and Control Engineering (CAICE) · doi:10.1145/3804601.3804629
Investigating Reasoning in Large Language Models with Counterfactual Knowledge Graphs
Yan, F., Yao, J., Chen, M. K., Sun, Y., Zhang, Q., Zhang, X., Luo, R., & Tao, D. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817451
Neurosymbolic Large Language Models: A Survey of Symbolic Integration, Reasoning and Explainability
Rani, M., Mishra, B. K., & Thakker, D. (2026) · Information Systems Frontiers · doi:10.1007/s10796-026-10794-4 · Cited by 1 · see citing works
Saravanan, K., Hithesh, K., Maheshwari, A., Vishnuram, P., Bajaj, M., Mohapatra, B., Sharma, R., & Sahu, B. K. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-7449-0_19
Logic-MCTS: Enhancing Logical Reasoning in Small Language Models via Monte Carlo Tree Search
Yang, J., Yang, Y., Zou, D., Zhong, J., Shan, R., & Zhao, B. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728926
SLR: Automated Synthesis for Scalable Logical Reasoning
Helff, L., Omar, A., Friedrich, F., Wüst, A., Shindo, H., Mitchell, R., Woydt, T., Schramowski, P., Stammer, W., & Kersting, K. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.16
- Authors
- Liu, H., Liu, J., Cui, L., Teng, Z., Duan, N., Zhou, M., & Zhang, Y.
- Affiliations
- Westlake University (CN)
- Fudan University, Shanghai, China
- Zhejiang University (CN)
- Microsoft Research Asia (China) (CN)
- Langboat Technology, Beijing, China
- Venue
- IEEE/ACM Transactions on Audio Speech and Language Processing
- Published
- 2023
- Volume
- 31
- Pages
- 2947-2962
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W4383340139
- Cited by
- 26
- References
- 97
- Citations by year
- 2026: 9 · 2025: 10 · 2024: 7
- FWCI
- 2.45
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- logical reasoning, natural language inference, reading comprehension, data annotation, Chinese idioms, large pre-trained language models, question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
