A Decomposable Attention Model for Natural Language Inference
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
We propose a simple neural architecture for natural language inference.Our approach uses attention to decompose the problem into subproblems that can be solved separately, thus making it trivially parallelizable.On the Stanford Natural Language Inference (SNLI) dataset, we obtain state-of-the-art results with almost an order of magnitude fewer parameters than previous work and without relying on any word-order information.Adding intra-sentence attention that takes a minimum amount of order into account yields further improvements.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2014) · Cited by 33,419
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 6×
“Given two sentences, where each word is represented by an embedding vector, we first create a soft alignment matrix using neural attention (Bahdanau et al., 2015).”
MethodInfluential(2010) · Cited by 8,261
(2011) · Cited by 6,993
(1989) · neural information processing systems · Cited by 3,736
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 7×
“We evaluate our approach on the Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015).”
BackgroundMethodInfluentialStatistical machine translation
(2010) · Choice Reviews Online · doi:10.5860/choice.47-6293 · Cited by 1,309
Long Short-Term Memory-Networks for Machine Reading
Cheng, J., Dong, L., & Lapata, M. (2016) · arXiv · doi:10.48550/arxiv.1601.06733 · Cited by 1,198
Cited by
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
“For applications involving text pairs, a common pattern is to independently encode text pairs before applying bidirectional cross attention, such as Parikh et al. (2016); Seo et al.”
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
Self-Attention Generative Adversarial Networks
Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A. (2018) · arXiv · doi:10.48550/arxiv.1805.08318 · Cited by 3,353
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
“jor benchmark for machine learning work on sentence understanding and spurring work on core representation learning techniques for NLU, such as attention (Wang and Jiang, 2016; Parikh et al., 2016), memory (Munkhdalai and Yu, 2017), and the use of parse structure (Mou et al.”
BackgroundBERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Taming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01268 · Cited by 2,514
Taming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2020) · arXiv · doi:10.48550/arxiv.2012.09841 · Cited by 2,514
Self-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505
“Attention-based models have therefore used position encodings or biased attention weights based on distance (Parikh et al., 2016).”
MethodSelf-citationNatural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
Self-citationRoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957
RoFormer: Enhanced Transformer with Rotary Position Embedding
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021) · arXiv · doi:10.48550/arxiv.2104.09864 · Cited by 1,952
“On the other side, the previous work Parikh et al. [2016], Shaw et al. [2018], Huang et al. [2018], Dai et al. [2019], Yang et al. [2019], Raffel et al. [2020], Ke et al. [2020], He et al. [2020], Huang et al. [2020] focuses on relative position encoding, which typically encodes the relative…”
ResultA Structured Self-attentive Sentence Embedding
Lin, Z., Feng, M., Santos, C. N. D., Yu, M., Xiang, B., Zhou, B., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.03130 · Cited by 1,743
Deep Learning--based Text Classification: A Comprehensive Review
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., & Gao, J. (2022) · ACM Computing Surveys · doi:10.1145/3439726 · Cited by 1,538
AllenNLP: A Deep Semantic Natural Language Processing Platform
Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M., Schmitz, M., & Zettlemoyer, L. (2018) · Proceedings of Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/w18-2501 · Cited by 1,252
Bidirectional LSTM with attention mechanism and convolutional layer for text classification
Liu, G., & Guo, J. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.01.078 · Cited by 1,127
Enhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113
Deep learning modelling techniques: Current progress, applications, advantages, and challenges
Ahmed, S. F., Alam, M. S. B., Hassan, M., Rozbu, M. R., Ishtiak, T., Rafa, N., Mofijur, M., Shawkat Ali, A. B. M., & Gandomi, A. H. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10466-8 · Cited by 1,079
Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.-X., & Yan, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.00235 · Cited by 1,053
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944
Annotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907
Transformer Interpretability Beyond Attention Visualization
Chefer, H., Gur, S., & Wolf, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00084 · Cited by 903
End-to-End Human Pose and Mesh Reconstruction with Transformers
Lin, K., Wang, L., & Liu, Z. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00199 · Cited by 740
“Recent studies [36, 28, 53] have shown that attention mechanisms improve the performance on various language tasks.”
BackgroundBilateral Multi-Perspective Matching for Natural Language Sentences
Wang, Z., Hamza, W., & Florian, R. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/579 · Cited by 739
Gated Self-Matching Networks for Reading Comprehension and Question Answering
Wang, W., Yang, N., Wei, F., Chang, B., & Zhou, M. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1018 · Cited by 666
An Empirical Study of Spatial Attention Mechanisms in Deep Networks
Zhu, X., Cheng, D., Zhang, Z., Lin, S., & Dai, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00679 · Cited by 659
Universal Adversarial Triggers for Attacking and Analyzing NLP
Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1221 · Cited by 642
Attention in Natural Language Processing
Galassi, A., Lippi, M., & Torroni, P. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3019893 · Cited by 635
Learned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572
Multi-Scale Self-Guided Attention for Medical Image Segmentation
Sinha, A., & Dolz, J. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2986926 · Cited by 570
NAIS: Neural Attentive Item Similarity Model for Recommendation
He, X., He, Z., Song, J., Liu, Z., Jiang, Y.-G., & Chua, T.-S. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2831682 · Cited by 569
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation
Chen, M. X., Firat, O., Bapna, A., Johnson, M., Macherey, W., Foster, G., Jones, L., Schuster, M., Shazeer, N., Parmar, N., et al. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1008 · Cited by 564
Self-citationDeep Learning for Entity Matching: A Design Space Exploration
Mudgal, S., Li, H., Rekatsinas, T., Doan, A., Park, Y., Krishnan, G., Deep, R., Arcaute, E., & Raghavendra, V. (2018) · Proceedings of the 2018 International Conference on Management of Data · doi:10.1145/3183713.3196926 · Cited by 550
Explainable machine learning in materials science
Zhong, X., Gallagher, B., Liu, S., Kailkhura, B., Hiszpanski, A., & Han, T. Y.-J. (2022) · npj Computational Materials · doi:10.1038/s41524-022-00884-7 · Cited by 501
Jain, S., & Wallace, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10186 · Cited by 467
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462
“We adapted two neural entailment models, DecompAttn (Parikh et al., 2016) and DGEM (Khot et al., 2018), to the task of answering multiple-choice questions.”
MethodInfluentialA Survey on Data Augmentation for Text Classification
Bayer, M., Kaufhold, M.-A., & Reuter, C. (2023) · ACM Computing Surveys · doi:10.1145/3544558 · Cited by 425
Attention mechanism in neural networks: where it comes and where it goes
Soydaner, D. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07366-3 · Cited by 414
SciTaiL: A Textual Entailment Dataset from Science Question Answering
Khot, T., Sabharwal, A., & Clark, P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12022 · Cited by 411
Adversarial Attacks on Deep-learning Models in Natural Language Processing: A Survey
Zhang, W. E., Sheng, Q. Z., Alhazmi, A., & Li, C. (2020) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3374217 · Cited by 407
Are Sixteen Heads Really Better than One?
Michel, P., Levy, O., & Neubig, G. (2019) · arXiv · doi:10.48550/arxiv.1905.10650 · Cited by 404
Neural Machine Translation: A Review
Stahlberg, F. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12007 · Cited by 397
e-SNLI: Natural language inference with natural language explanations
Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., & Blunsom, P. (2018) · arXiv · doi:10.48550/arxiv.1812.01193 · Cited by 367
“Glockner et al. show that models achieving high accuracies on SNLI, such as [21, 22, 6], show dramatically reduced performance on this simpler dataset, while the model of Chen et al. [5] is more robust due to incorporating exter-nal knowledge.”
BackgroundResultAxial Attention in Multidimensional Transformers
Ho, J., Kalchbrenner, N., Weissenborn, D., & Salimans, T. (2019) · arXiv · doi:10.48550/arxiv.1912.12180 · Cited by 370
Lee, J., Lee, I., & Kang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08082 · Cited by 349
Revisiting Video Saliency Prediction in the Deep Learning Era
Wang, W., Shen, J., Xie, J., Cheng, M.-M., Ling, H., & Borji, A. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2924417 · Cited by 330
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
Shen, D., Wang, G., Wang, W., Min, M. R., Su, Q., Zhang, Y., Li, C., Henao, R., & Carin, L. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1041 · Cited by 328
Li, M., Hsu, W., Xie, X., Cong, J., & Gao, W. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.2968472 · Cited by 321
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences
Glockner, M., Shwartz, V., & Goldberg, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2103 · Cited by 319
“Finally, DECOMPOSABLE ATTENTION (Parikh et al., 2016) performs soft alignment of words from the premise to words in the hypothesis using attention mechanism, and decomposes the task into comparison of aligned words.”
BackgroundMethodVisual attention methods in deep learning: An in-depth survey
Hassanin, M., Anwar, S., Radwan, I., Khan, F. S., & Mian, A. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102417 · Cited by 317
Improved Neural Relation Detection for Knowledge Base Question Answering
Yu, M., Yin, W., Hasan, K. S., dos Santos, C., Xiang, B., & Zhou, B. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1053 · Cited by 314
Deep Semantic Role Labeling With Self-Attention
Tan, Z., Wang, M., Xie, J., Chen, Y., & Shi, X. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11928 · Cited by 306
Evolution of Semantic Similarity—A Survey
Chandrasekaran, D., & Mago, V. (2022) · ACM Computing Surveys · doi:10.1145/3440755 · Cited by 303
Attention, please! A survey of neural attention models in deep learning
de Santana Correia, A., & Colombini, E. L. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10148-x · Cited by 298
Co-Occurrent Features in Semantic Segmentation
Zhang, H., Zhang, H., Wang, C., & Xie, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00064 · Cited by 294
MEF-GAN: Multi-Exposure Image Fusion via Generative Adversarial Networks
Xu, H., Ma, J., & Zhang, X.-P. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2999855 · Cited by 282
Revisiting Video Saliency: A Large-Scale Benchmark and a New Model
Wang, W., Shen, J., Guo, F., Cheng, M.-M., & Borji, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00514 · Cited by 268
Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I., & Gurevych, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1213 · Cited by 265
Recent advances in deep learning based dialogue systems: a systematic survey
Ni, J., Young, T., Pandelea, V., Xue, F., & Cambria, E. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10248-8 · Cited by 264
A Spatio-temporal Transformer for 3D Human Motion Prediction
Aksan, E., Kaufmann, M., Cao, P., & Hilliges, O. (2021) · 2021 International Conference on 3D Vision (3DV) · doi:10.1109/3dv53792.2021.00066 · Cited by 262
Neural Natural Language Inference Models Enhanced with External Knowledge
Chen, Q., Zhu, X., Ling, Z.-H., Inkpen, D., & Wei, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1224 · Cited by 253
Synthesizer: Rethinking Self-Attention in Transformer Models
Tay, Y., Bahri, D., Metzler, D., Juan, D.-C., Zhao, Z., & Zheng, C. (2020) · arXiv · doi:10.48550/arxiv.2005.00743 · Cited by 251
 3: Accelerating attention mechanisms in neural networks with approximation
Ham, T. J., Jung, S. J., Kim, S., Oh, Y. H., Park, Y., Song, Y., Park, J.-H., Lee, S., Park, K., Lee, J. W., et al. (2020) · 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca47549.2020.00035 · Cited by 242
TabFact: A Large-scale Dataset for Table-based Fact Verification
Chen, W., Wang, H., Chen, J., Zhang, Y., Wang, H., Li, S., Zhou, X., & Wang, W. Y. (2019) · arXiv · doi:10.48550/arxiv.1909.02164 · Cited by 239
“With the surge of deep learning, there have been many powerful algorithms like the Decomposed Model (Parikh et al., 2016), Enhanced-LSTM (Chen et al., 2017) and BERT (Devlin et al., 2019).”
MethodSummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238
Show Your Work: Improved Reporting of Experimental Results
Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1224 · Cited by 233
More than Bags of Words: Sentiment Analysis with Word Embeddings
Rudkowsky, E., Haselmayer, M., Wastian, M., Jenny, M., Emrich, Š., & Sedlmair, M. (2018) · Communication Methods and Measures · doi:10.1080/19312458.2018.1455817 · Cited by 219
“Several approaches (Le & Mikolov, 2014; Parikh, Täckström, Das, & Uszkoreit, 2016) build on the concept of distributed word embeddings but use more sophisticated modeling techniques than a simple averaging of “ standard ” word embeddings.”
MethodA Comparison of Transformer and LSTM Encoder Decoder Models for ASR
Zeyer, A., Bahar, P., Irie, K., Schluter, R., & Ney, H. (2019) · 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru46091.2019.9004025 · Cited by 218
“More recently, the Transformer model was introduced [22], which is based on self-attention [23–25].”
MethodThe Fact Extraction and VERification (FEVER) Shared Task
Thorne, J., Vlachos, A., Cocarascu, O., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the First Workshop on Fact Extraction and VERification (FEVER) · doi:10.18653/v1/w18-5501 · Cited by 212
“, 2016), Decomposable Attention (Parikh et al., 2016) or similar to them.”
BackgroundMethodInfluentialCalibration of Pre-trained Transformers
Desai, S., & Durrett, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.21 · Cited by 209
Explainable Automated Fact-Checking for Public Health Claims
Kotonya, N., & Toni, F. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.623 · Cited by 195
Learning to Write with Cooperative Discriminators
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., & Choi, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1152 · Cited by 192
Reasoning with Sarcasm by Reading In-Between
Tay, Y., Luu, A. T., Hui, S. C., & Su, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1093 · Cited by 192
“Intraattention can be interpreted as a self-targetted coattention and is seeing a lot promising results in many recent works (Vaswani et al., 2017; Parikh et al., 2016; Tay et al., 2017a; Shen et al., 2017).”
ResultFine-grained Fact Verification with Kernel Graph Attention Network
Liu, Z., Xiong, C., Sun, M., & Liu, Z. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.655 · Cited by 183
“Many fact verification systems leverage Natural Language Inference (NLI) techniques (Chen et al., 2017b; Ghaeini et al., 2018; Parikh et al., 2016; Radford et al., 2018; Peters et al., 2018; Li et al., 2019) to verify the claim.”
BackgroundTransformers as Soft Reasoners over Language
Clark, P., Tafjord, O., & Richardson, K. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/537 · Cited by 176
Human-like problem-solving abilities in large language models using ChatGPT
Orrù, G., Piarulli, A., Conversano, C., & Gemignani, A. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1199350 · Cited by 170
Theoretical Limitations of Self-Attention in Neural Sequence Models
Hahn, M. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00306 · Cited by 165
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Press, O., Smith, N. A., & Lewis, M. (2021) · arXiv · doi:10.48550/arxiv.2108.12409 · Cited by 161
Multi-modal Attention Network Learning for Semantic Source Code Retrieval
Wan, Y., Shu, J., Sui, Y., Xu, G., Zhao, Z., Wu, J., & Yu, P. (2019) · 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00012 · Cited by 161
“In this condition, we call it intra-attention (also well known as self-attention) [48], [49], which exhibits a better balance between ability to model long-range dependencies and computational and statistical efficiency.”
BackgroundInterpretation of Natural Language Rules in Conversational Machine Reading
Saeidi, M., Bartolo, M., Lewis, P., Singh, S., Rocktäschel, T., Sheldon, M., Bouchard, G., & Riedel, S. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1233 · Cited by 141
“For neural models, we use Decomposed Attention Model (DAM) (Parikh et al., 2016) trained on each the SNLI and ShARC corpora using ELMO embeddings (Peters et al.”
MethodHeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers
Dong, P., Sun, M., Lu, A., Xie, Y., Liu, K., Kong, Z., Meng, X., Li, Z., Lin, X., Fang, Z., et al. (2023) · 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071047 · Cited by 128
“Transformers [2], [39] have recently made an attractive resurgence in the form of Vision Transformers (ViTs) [11], showing strong versatility in NLP [3], [26], [48], computer vision (e.”
BackgroundConvolutional Self-Attention Networks
Yang, B., Wang, L., Wong, D. F., Chao, L. S., & Tu, Z. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1407 · Cited by 128
Rethinking search: making domain experts out of dilettantes
Metzler, D., Tay, Y., Bahri, D., & Najork, M. (2021) · ACM SIGIR Forum · doi:10.1145/3476415.3476428 · Cited by 112
“, 2015], and entailment detection [Parikh et al., 2016; Tay et al., 2018b].”
BackgroundMethodKochmar, E., Vu, D. D., Belfer, R., Gupta, V., Serban, I. V., & Pineau, J. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00267-x · Cited by 112
UCL Machine Reading Group: Four Factor Framework For Fact Finding (HexaF)
Yoneda, T., Mitchell, J., Welbl, J., Stenetorp, P., & Riedel, S. (2018) · Proceedings of the First Workshop on Fact Extraction and VERification (FEVER) · doi:10.18653/v1/w18-5515 · Cited by 101
“The Jack the Reader (Weissenborn et al., 2018) framework was used for both DAM and ESIM.”
MethodScaling Autoregressive Video Models
Weissenborn, D., Täckström, O., & Uszkoreit, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02634 · Cited by 93
Self-citationInterpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference
Ghaeini, R., Fern, X., & Tadepalli, P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1537 · Cited by 91
“Several pieces of prior work in NLI have attempted to visualize the attention layer to provide some understanding of their models (Ghaeini et al., 2018b; Parikh et al., 2016).”
BackgroundHyperLex: A Large-Scale Evaluation of Graded Lexical Entailment
Vulić, I., Gerz, D., Kiela, D., Hill, F., & Korhonen, A. (2016) · arXiv · doi:10.48550/arxiv.1608.02117 · Cited by 90
Geometry-Free View Synthesis: Transformers and no 3D Priors
Rombach, R., Esser, P., & Ommer, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01409 · Cited by 75
Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference
Belinkov, Y., Poliak, A., Shieber, S., Van Durme, B., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1084 · Cited by 73
Importance of Self-Attention for Sentiment Analysis
Letarte, G., Paradis, F., Giguère, P., & Laviolette, F. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5429 · Cited by 70
A Review on Fact Extraction and Verification
Bekoulis, G., Papagiannopoulou, C., & Deligiannis, N. (2023) · ACM Computing Surveys · doi:10.1145/3485127 · Cited by 48
Putting the Horse before the Cart: A Generator-Evaluator Framework for Question Generation from Text
Kumar, V., Ramakrishnan, G., & Li, Y.-F. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1076 · Cited by 46
“We therefore, design a novel reward function, a decomposable attention (Parikh et al., 2016) based similarity scorer (DAS).”
BackgroundMethodUltra-fine Entity Typing with Indirect Supervision from Natural Language Inference
Li, B., Yin, W., & Chen, M. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00479 · Cited by 35
Learning to Predict Denotational Probabilities For Modeling Entailment
Lai, A., & Hockenmaier, J. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1068 · Cited by 33
Learning a Product Relevance Model from Click-Through Data in E-Commerce
Yao, S., Tan, J., Chen, X., Yang, K., Xiao, R., Deng, H., & Wan, X. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3450129 · Cited by 33
“DecompAtt [19] leverages attention mechanism for alignment.”
TIS Transformer: remapping the human proteome using deep learning
Clauwaert, J., McVey, Z., Gupta, R., & Menschaert, G. (2023) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqad021 · Cited by 26
Language-Agnostic Representation Learning for Product Search on E-Commerce Platforms
Ahuja, A., Rao, N., Katariya, S., Subbian, K., & Reddy, C. K. (2020) · Proceedings of the 13th International Conference on Web Search and Data Mining · doi:10.1145/3336191.3371852 · Cited by 26
TIS Transformer: Remapping the Human Proteome Using Deep Learning
Clauwaert, J., Gupta, R., McVey, Z., & Menschaert, G. (2021) · doi:10.1101/2021.11.18.468957 · Cited by 26
Traceable LLM-based validation of statements in knowledge graphs
Adam, D., & Kliegr, T. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104128 · Cited by 23
Metadata
- Authors
- Parikh, A., Täckström, O., Das, D., & Uszkoreit, J.
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 2249-2255
- OpenAlex
- W2413794162
- Cited by
- 1,296
- References
- 29
- FWCI
- 83.68
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.