CoChat

Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling

Open access
Logan, Robert, Liu, Nelson F., Peters, Matthew E., Gardner, Matt, Singh, Sameer · 2019 · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

The DOI matches its Crossref record; 28 of the 31 references it cites were found in a registry or an index.

Abstract

Modeling human language requires the ability to not only generate fluent text but also encode factual knowledge.However, traditional language models are only capable of remembering facts seen at training time, and often have difficulty recalling them.To address this, we introduce the knowledge graph language model (KGLM), a neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context.These mechanisms enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens.We also introduce the Linked WikiText-2 dataset, 1 a corpus of annotated text aligned to the Wikidata knowledge graph whose contents (roughly) match the popular WikiText-2 benchmark (Merity et al., 2017).In experiments, we demonstrate that the KGLM achieves significantly better performance than a strong baseline language model.We additionally compare different language models' ability to complete sentences requiring factual knowledge, and show that the KGLM outperforms even very large language models in generating facts.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/p19-1598

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Long Short-Term Memory

    Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×

    “RNN language models (Mikolov et al., 2010) parameterize this distribution using a recurrent structure: (1) We use LSTMs (Hochreiter and Schmidhuber, 1997) as the recurrent module in this paper.”
    Method
  • Adam: A Method for Stochastic Optimization

    Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×

    “However, we perform optimization using Adam (Kingma and Ba, 2015) with learning rate 1e-3 instead of NT-ASGD, having found that it is more stable.”
    Method
  • Untitled work

    (2014) · Cited by 33,419

  • Untitled work

    (2013) · Cited by 5,726

  • Recurrent neural network based language model

    Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 2×

    “RNN language models (Mikolov et al., 2010) parameterize this distribution using a recurrent structure:”
    Method
  • Wikidata: a free collaborative knowledgebase

    Vrandečić, D., & Krötzsch, M. (2014) · Communications of the ACM · doi:10.1145/2629489 · Cited by 3,342 · cited 14×

    “Accordingly, we use the neural-el (Gupta et al., 2017) entity linker to identify additional links to Wikidata, and identify coreferences using Stanford CoreNLP 2 to cover pronouns, nominals, and other tokens missed by the linker.”
    MethodInfluential
  • Untitled work

    (2013) · Cited by 2,056

  • Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models

    Serban, I., Sordoni, A., Bengio, Y., Courville, A., & Pineau, J. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.9883 · Cited by 1,615 · cited 2×

    “…skilled at generating grammatical sentences, and previous work has shown that language models also possess some degree of common-sense reasoning and basic knowledge (Vinyals and Le, 2015; Serban et al., 2016; Trinh and Le, 2019), their ability to generate fac-tually correct text is quite limited.”
    Background
  • Incorporating Copying Mechanism in Sequence-to-Sequence Learning

    Gu, J., Lu, Z., Li, H., & Li, V. O. K. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1154 · Cited by 1,446 · cited 1×

    “To obtain probabilities for words in the alias vocabulary, we use a copy mechanism Gu et al. (2016).”
    Method
  • A Neural Conversational Model

    Vinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400 · cited 1×

    “Although language models are quite skilled at generating grammatical sentences, and previous work has shown that language models also possess some degree of common-sense reasoning and basic knowledge (Vinyals and Le, 2015; Serban et al., 2016; Trinh and Le, 2019), their ability to generate factually correct text is qu…”
    Background
  • Regularizing and Optimizing LSTM Language Models

    Merity, S., Keskar, N. S., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.02182 · Cited by 697 · cited 6×

    “For instance, when conditioned on the text at the top of Figure 1, an AWD-LSTM language model (Merity et al., 2018) trained on Wikitext-2 assigns higher probability to the word “PlayStation” than “Game Boy”, even though this sentence appears verbatim in the training data.”
    BackgroundMethodInfluential
  • Building applied natural language generation systems

    REITER, E., & DALE, R. (1997) · Natural Language Engineering · doi:10.1017/s1351324997001502 · Cited by 631 · cited 1×

    “For a survey of early non-neural text generation meth-ods we refer the reader to Reiter and Dale (1997).”
    Method
  • Challenges in Data-to-Document Generation

    Wiseman, S., Shieber, S., & Rush, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1239 · Cited by 524 · cited 4×

    “Our goals are much more aligned to the data-to-text task (Ahn et al., 2016; Lebret et al., 2016; Wiseman et al., 2017; Yang et al., 2017; Gardent et al., 2017; Ferreira et al., 2018), where a small table-sized KB is provided to generate a short piece of text; we are interested in language models…”
    BackgroundMethodInfluential
  • Recurrent Neural Network Grammars

    Dyer, C., Kuncoro, A., Ballesteros, M., & Smith, N. A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1024 · Cited by 517 · cited 1×

    “Given samples from a proposal distribution q ( E | x ) the marginal distribution is: This approach is used to evaluate models in Ji et al. (2017) and Dyer et al. (2016).”
    Method
  • Neural Text Generation from Structured Data with Application to the Biography Domain

    Lebret, R., Grangier, D., & Auli, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1128 · Cited by 444 · cited 4×

    “, 2017), and Wikipedia info-boxes (Lebret et al., 2016).”
    BackgroundMethodInfluential
  • Pointer Sentinel Mixture Models

    Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.07843 · Cited by 427 · cited 3×

    “Secondly, according to correspondence with Merity et al. (2017), WikiText-2 was collected by querying the Wikipedia Text API.”
    BackgroundMethodInfluential
  • The WebNLG Challenge: Generating Text from RDF Data

    Gardent, C., Shimorina, A., Narayan, S., & Perez-Beltrachini, L. (2017) · Proceedings of the 10th International Conference on Natural Language Generation · doi:10.18653/v1/w17-3518 · Cited by 342

  • Entity Linking via Joint Encoding of Types, Descriptions, and Context

    Gupta, N., Singh, S., & Roth, D. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1284 · Cited by 227 · cited 2×

    “Accordingly, we use the neural-el (Gupta et al., 2017) entity linker to identify additional links to Wikidata, and identify coreferences using Stanford CoreNLP2 to cover pronouns, nominals, and other tokens missed by the linker.”
    MethodSelf-citation
  • FRAGE: Frequency-Agnostic Word RepresentationPreprint

    Gong, C., He, D., Tan, X., Qin, T., Wang, L.-W., & Liu, T.-Y. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1809.06858 · Cited by 155 · cited 3×

    “General language modeling Also related are the recent papers proposing modifications to the AWD-LSTM that improve performance on Wikitext-2 (Gong et al., 2018; Yang et al., 2018; Krause et al., 2018).”
    Background
  • A Neural Knowledge Language Model

    Ahn, S., Choi, H., Pärnamaa, T., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.00318 · Cited by 101 · cited 8×

    “(ii) The neural knowledge language model (NKLM) (Ahn et al., 2016) which established the idea of leveraging knowledge graphs in neural language models.”
    BackgroundMethodResultInfluential
  • Dynamic Entity Representations in Neural Language Models

    Ji, Y., Tan, C., Martschat, S., Choi, Y., & Smith, N. A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1195 · Cited by 94 · cited 8×

    “Following Ji et al. (2017), we compute q ( E | x ) using a discriminative version of our model that predicts annotations for the current token instead of for the next token.”
    BackgroundMethodInfluential
  • Dynamic Evaluation of Neural Sequence Models

    Krause, B., Kahembwe, E., Murray, I., & Renals, S. (2017) · arXiv · doi:10.48550/arxiv.1709.07432 · Cited by 90 · cited 2×

    “General language modeling Also related are the recent papers proposing modifications to the AWD-LSTM that improve performance on Wikitext-2 (Gong et al., 2018; Yang et al., 2018; Krause et al., 2018).”
    Background
  • Enriching the WebNLG corpus

    Ferreira, T. C., Moussallem, D., Krahmer, E., & Wubben, S. (2018) · International Conference on Natural Language Generation · doi:10.18653/v1/w18-6521 · Cited by 73 · cited 1×

    “…much more aligned to the data-to-text task (Ahn et al., 2016; Lebret et al., 2016; Wiseman et al., 2017; Yang et al., 2017; Gardent et al., 2017; Ferreira et al., 2018), where a small table-sized KB is provided to generate a short piece of text; we are interested in language models that…”
    Background
  • Reference-Aware Language Models

    Yang, Z., Blunsom, P., Dyer, C., & Ling, W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1197 · Cited by 68 · cited 2×

    “Our goals are much more aligned to the data-to-text task (Ahn et al., 2016; Lebret et al., 2016; Wiseman et al., 2017; Yang et al., 2017; Gardent et al., 2017; Ferreira et al., 2018), where a small table-sized KB is provided to generate a short piece of text; we are interested in language models…”
    BackgroundMethod
  • Numeracy for language models: Evaluating and improving their ability to predict numbers

    Spithourakis, G., & Riedel, S. (2018) · UCL Discovery (University College London) · Cited by 63

  • Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

    Yang, Z., Dai, Z., Salakhutdinov, R., & Cohen, W. W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.03953 · Cited by 62

  • “…recent and performant language model, KGLM obtains not only a lower overall perplexity, but also a substantially lower unknown-penalized perplexity (Ueberla, 1994; Ahn et al., 2016), a metric that allows fair comparisons between models that accurately model rare tokens and ones that predict them…”
    BackgroundMethod
  • Do Language Models Have Common Sense

    Trinh, T. H., & Le, Q. V. (2018) · Cited by 16

Research works printed but not linked: 2

  1. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In Proc. of ICLR.
  2. Oriol Vinyals and Quoc V. Le. 2015. A neural con- versational model. Proc. of ICML Deep Learning Workshop.

Cited by

See all 197

Metadata

Authors
Logan, R., Liu, N. F., Peters, M. E., Gardner, M., & Singh, S.
Affiliations
  • University of California, Irvine (US)
  • University of Washington (US)
  • Allen Institute for Artificial Intelligence (US)
Venue
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Published
2019
Pages
5962-5971
ISSN
0736-587X (ISSN-L)
OpenAlex
W2951048068
Cited by
168
References
31
Citations by year
2026: 3 · 2025: 12 · 2024: 24 · 2023: 15 · 2022: 27 · 2021: 33 · 2020: 47 · 2019: 7
FWCI
10.91
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (1817183)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
knowledge graph retrieval, Wikidata, sentence completion

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.