Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 28 of the 31 references it cites were found in a registry or an index.

Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling

Open access
Logan, Robert, Liu, Nelson F., Peters, Matthew E., Gardner, Matt, Singh, Sameer · 2019 · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Abstract

Modeling human language requires the ability to not only generate fluent text but also encode factual knowledge.However, traditional language models are only capable of remembering facts seen at training time, and often have difficulty recalling them.To address this, we introduce the knowledge graph language model (KGLM), a neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context.These mechanisms enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens.We also introduce the Linked WikiText-2 dataset, 1 a corpus of annotated text aligned to the Wikidata knowledge graph whose contents (roughly) match the popular WikiText-2 benchmark (Merity et al., 2017).In experiments, we demonstrate that the KGLM achieves significantly better performance than a strong baseline language model.We additionally compare different language models' ability to complete sentences requiring factual knowledge, and show that the KGLM outperforms even very large language models in generating facts.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/p19-1598

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Long Short-Term Memory

    Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×

    “RNN language models (Mikolov et al., 2010) parameterize this distribution using a recurrent structure: (1) We use LSTMs (Hochreiter and Schmidhuber, 1997) as the recurrent module in this paper.”
    Method
  • Adam: A Method for Stochastic Optimization

    Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×

    “However, we perform optimization using Adam (Kingma and Ba, 2015) with learning rate 1e-3 instead of NT-ASGD, having found that it is more stable.”
    Method
  • Untitled work

    (2014) · Cited by 33,419

  • Untitled work

    (2013) · Cited by 5,726

  • Recurrent neural network based language model

    Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 2×

    “RNN language models (Mikolov et al., 2010) parameterize this distribution using a recurrent structure:”
    Method
  • Wikidata: a free collaborative knowledgebase

    Vrandečić, D., & Krötzsch, M. (2014) · Communications of the ACM · doi:10.1145/2629489 · Cited by 3,342 · cited 14×

    “Accordingly, we use the neural-el (Gupta et al., 2017) entity linker to identify additional links to Wikidata, and identify coreferences using Stanford CoreNLP 2 to cover pronouns, nominals, and other tokens missed by the linker.”
    MethodInfluential
  • Untitled work

    (2013) · Cited by 2,056

  • Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models

    Serban, I., Sordoni, A., Bengio, Y., Courville, A., & Pineau, J. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.9883 · Cited by 1,615 · cited 2×

    “…skilled at generating grammatical sentences, and previous work has shown that language models also possess some degree of common-sense reasoning and basic knowledge (Vinyals and Le, 2015; Serban et al., 2016; Trinh and Le, 2019), their ability to generate fac-tually correct text is quite limited.”
    Background
  • Incorporating Copying Mechanism in Sequence-to-Sequence Learning

    Gu, J., Lu, Z., Li, H., & Li, V. O. K. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1154 · Cited by 1,446 · cited 1×

    “To obtain probabilities for words in the alias vocabulary, we use a copy mechanism Gu et al. (2016).”
    Method
  • A Neural Conversational Model

    Vinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400 · cited 1×

    “Although language models are quite skilled at generating grammatical sentences, and previous work has shown that language models also possess some degree of common-sense reasoning and basic knowledge (Vinyals and Le, 2015; Serban et al., 2016; Trinh and Le, 2019), their ability to generate factually correct text is qu…”
    Background

Show all 31 references

Research works printed but not linked: 2

  1. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In Proc. of ICLR.
  2. Oriol Vinyals and Quoc V. Le. 2015. A neural con- versational model. Proc. of ICML Deep Learning Workshop.

Cited by

See all 196

Metadata

Authors
Logan, R., Liu, N. F., Peters, M. E., Gardner, M., & Singh, S.
Affiliations
  • University of California, Irvine (US)
  • University of Washington (US)
  • Allen Institute for Artificial Intelligence (US)
Venue
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Published
2019
Pages
5962-5971
ISSN
0736-587X (ISSN-L)
OpenAlex
W2951048068
Cited by
168
References
31
Citations by year
2026: 3 · 2025: 12 · 2024: 24 · 2023: 15 · 2022: 27 · 2021: 33 · 2020: 47 · 2019: 7
FWCI
10.91
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (1817183)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
knowledge graph retrieval, Wikidata, sentence completion

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling. https://app.cochat.ai/papers/10.18653/v1/p19-1598
The paper
Logan, R., Liu, N. F., Peters, M. E., Gardner, M., & Singh, S. (2019). Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. https://doi.org/10.18653/v1/p19-1598

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.