CoChat
Sign Up
Cited by 261Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Leveraging Knowledge Bases in LSTMs for Improving Machine Reading

Open access
Yang, Bishan, Mitchell, Tom·2017·Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/p17-1132
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper focuses on how to take advantage of external knowledge bases (KBs) to improve recurrent neural networks for machine reading.Traditional methods that exploit knowledge from KBs encode knowledge as discrete indicator features.Not only do these features generalize poorly, but they require task-specific feature engineering to achieve good performance.We propose KBLSTM, a novel neural model that leverages continuous representations of KBs to enhance the learning of recurrent neural networks for machine reading.To effectively integrate background knowledge with information from the currently processed text, our model employs an attention mechanism with a sentinel to adaptively decide whether to attend to background knowledge and which information from KBs is useful.Experimental results show that our model achieves accuracies that surpass the previous state-of-the-art results for both entity extraction and event extraction on the widely used ACE2005 dataset.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/p17-1132

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Wilcoxon signed-rank test is applicable for most NLP setups.

    “The test is actually applicable for most NLP setups and it has been used widely (e.g.(Søgaard et al., 2014;Søgaard, 2013;Yang and Mitchell, 2017)) due to its improved power compared to the sign test variants.”

    From The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing (2018) · cites this paper as Yang and Mitchell, 2017

    Section 4.1.1 (Implementation Details; page unavailable) reports using a Wilcoxon rank-sum test to compare NLP models, so the paper provides an example of a Wilcoxon test in an NLP setup. It does not report using the signed-rank test or establish that signed-rank testing applies to most NLP setups; it also does not compare its power with sign-test variants. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, and Yoshua Bengio. 2016. A neural knowledge lan- guage model. arXiv preprint arXiv:1608.00318 . · arXiv
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Represen- tations (ICLR).
  3. Antoine Bordes, Nicolas Usunier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. In Advances in Neural Information Processing Systems (NIPS). pages 2787-2795.
  4. Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2011. Learning structured embed- dings of knowledge bases. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelli- gence.
  5. Kai-Wei Chang, Wen-tau Yih, Bishan Yang, and Christopher Meek. 2014. Typed tensor decompo- sition of knowledge bases for relation extraction. In Empirical Methods in Natural Language Processing (EMNLP).
  6. Danqi Chen, Jason Bolton, and Christopher D Man- ning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Pro- ceedings of the 54th Annual Meeting of the Associa- tion for Computational Linguistics (ACL).
  7. Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In Proceedings of the Annual Meeting of the Associ- ation for Computational Linguistics (ACL). pages 167-176.
  8. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP).
  9. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research 12(Jul):2121-2159.
  10. Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. Transactions of the Association for Computational Linguistics 2:477-490.
  11. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition- based dependency parsing with stack long short- term memory. In Proceedings of the Annual Meet- ing of the Association for Computational Linguistics (ACL).
  12. Jeffrey L Elman. 1990. Finding structure in time. Cog- nitive science 14(2):179-211.
  13. Charles J Fillmore. 1976. Frame semantics and the na- ture of language. Annals of the New York Academy of Sciences 280(1):20-32.
  14. Alex Graves, Santiago Fernández, and Jürgen Schmid- huber. 2005. Bidirectional lstm networks for im- proved phoneme classification and recognition. In International Conference on Artificial Neural Net- works. Springer, pages 799-804.
  15. Kelvin Guu, John Miller, and Percy Liang. 2015. Traversing knowledge graphs in vector space. In Proceedings of the 2015 Conference on Empir- ical Methods in Natural Language Processing (EMNLP).
  16. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Su- leyman, and Phil Blunsom. 2015. Teaching ma- chines to read and comprehend. In Advances in Neu- ral Information Processing Systems (NIPS). pages 1693-1701.
  17. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735-1780.
  18. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: the 90% solution. In Proceedings of the human lan- guage technology conference of the NAACL, Com- panion Volume: Short Papers. pages 57-60.
  19. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirec- tional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991 . · arXiv
  20. Guillaume Lample, Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics (NAACL).
  21. Ni Lao, Tom Mitchell, and William W Cohen. 2011. Random walk inference and learning in a large scale knowledge base. In Proceedings of the Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP). pages 529-539.
  22. Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In Proceedings of the Annual Meeting of the Association for Computa- tional Linguistics (ACL). pages 402-412.
  23. Qi Li, Heng Ji, Yu Hong, and Sujian Li. 2014. Con- structing information networks using one single model. In Empirical Methods in Natural Language Processing (EMNLP). pages 1846-1851.
  24. Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global fea- tures. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). pages 73-82.
  25. Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing when to look: Adaptive at- tention via a visual sentinel for image captioning. In Proceedings of the 30th IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR).
  26. Xuezhe Ma and Eduard Hovy. 2016. End-to-end se- quence labeling via bi-directional lstm-cnns-crf. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (ACL).
  27. Alexander Miller, Adam Fisch, Jesse Dodge, Amir- Hossein Karimi, Antoine Bordes, and Jason We- ston. 2016. Key-value memory networks for directly reading documents. In Empirical Methods in Natu- ral Language Processing (EMNLP).
  28. T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, J. Betteridge, A. Carlson, B. Dalvi, M. Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Platanios, A. Rit- ter, M. Samadi, B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen, A. Saparov, M. Greaves, and J. Welling. 2015. Never-ending learning. In Pro- ceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI).
  29. Makoto Miwa and Mohit Bansal. 2016. End-to-end re- lation extraction using lstms on sequences and tree structures. Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (ACL) .
  30. Ndapandula Nakashole and Tom M Mitchell. 2015. A knowledge-intensive model for prepositional phrase attachment. In Proceedings of the Annual Meet- ing of the Association for Computational Linguistics (ACL). pages 365-375.
  31. Thien Huu Nguyen, Kyunghyun Cho, and Ralph Gr- ishman. 2016. Joint event extraction via recurrent neural networks. In North American Chapter of the Association for Computational Linguistics (NAACL- HLT). pages 300-309.
  32. Altaf Rahman and Vincent Ng. 2011. Coreference res- olution with world knowledge. In Proceedings of the 49th Annual Meeting of the Association for Com- putational Linguistics (ACL). pages 814-824.
  33. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Com- putational Natural Language Learning (CoNLL). pages 147-155.
  34. Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Pro- ceedings of HLT-NAACL.
  35. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning with neural ten- sor networks for knowledge base completion. In Advances in Neural Information Processing Systems (NIPS). pages 926-934.
  36. Kristina Toutanova, Danqi Chen, Patrick Pantel, Pallavi Choudhury, and Michael Gamon. 2015. Represent- ing text for joint embedding of text and knowledge bases. In Association for Computational Linguistics (ACL).
  37. Jason Weston, Antoine Bordes, Oksana Yakhnenko, and Nicolas Usunier. 2013. Connecting language and knowledge bases with embedding models for re- lation extraction. In Proceedings of the 2013 Con- ference on Empirical Methods in Natural Language Processing (EMNLP).
  38. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Towards universal paraphrastic sen- tence embeddings. Proceedings of the International Conference on Learning Representations (ICLR) .
  39. Derry Tanti Wijaya. 2016. VerbKB: A Knowledge Base of Verbs for Natural Language Understanding. Ph.D. thesis, Carnegie Mellon University. WordNet. 2010. About wordnet. http://wordnet.princeton.edu.
  40. Bishan Yang and Tom Mitchell. 2016. Joint extrac- tion of events and entities within a document con- text. In North American Chapter of the Association for Computational Linguistics (NAACL-HLT). pages 289-299.
  41. Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. International Conference on Learning Rep- resentations (ICLR) .
  42. Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural net- works. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).

Cited by

Sorted by citations

Authors
Yang, B., & Mitchell, T.
Affiliations
Carnegie Mellon University (US)
ORCID
Mitchell, Tom
Venue
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2017
Pages
1436-1446
ISSN
0736-587X (ISSN-L)
DOI
10.18653/v1/p17-1132
OpenAlex
W2741075451
Cited by
261
References
3
Citations by year
2026: 3 · 2025: 4 · 2024: 9 · 2023: 21 · 2022: 19 · 2021: 57 · 2020: 62 · 2019: 56 · 2018: 28 · 2017: 2
FWCI
18.46
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1247489, 1065251)
  • Defense Advanced Research Projects Agency (FA8750-13-2-0005, FA8750)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
knowledge base, machine reading, recurrent neural networks, attention mechanism, named entity recognition, event extraction, ACE2005, background knowledge integration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review