Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries
Open accessAbstract
Pretrained language models have been suggested as a possible alternative or complement to structured knowledge bases.However, this emerging LM-as-KB paradigm has so far only been considered in a very limited setting, which only allows handling 21k entities whose name is found in common LM vocabularies.Furthermore, a major benefit of this paradigm, i.e., querying the KB using natural language paraphrases, is underexplored.Here we formulate two basic requirements for treating LMs as KBs: (i) the ability to store a large number facts involving a large number of entities and (ii) the ability to query stored facts.We explore three entity representations that allow LMs to handle millions of entities and present a detailed case study on paraphrased querying of facts stored in LMs, thereby providing a proof-of-concept that language models can indeed serve as knowledge bases.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Parametric memory can implicitly integrate learned information during the inference process, without requiring a mechanism explicitly designed for integrating diverse information, as is necessary in non-parametric memory driven systems.
“Parametric memory can implicitly integrate learned information during the inference process, without requiring a mechanism explicitly designed for integrating diverse information, as is necessary in non-parametric memory driven systems [287, 288].”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [287]
Language models can efficiently query and implicitly store factual knowledge akin to knowledge bases.
“Language models have been shown to efficiently query and 'implicitly' store factual knowledge akin to knowledge bases [249, 287, 288].”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [287]
Language models serve a function similar to traditional knowledge bases in retrieving information.
“Recent research suggests that language models serve a function similar to that of traditional knowledge bases in retrieving information [248, 249, 287, 289].”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [287]
Large language models have the ability to retrieve factual knowledge as explicit knowledge bases, presenting an opportunity to construct more comprehensive knowledge graphs within recommender systems.
“The ability of Large Language Models to retrieve factual knowledge as explicit knowledge bases [53-56, 56-61] has been stirred discussed, which presents an opportunity to construct more comprehensive knowledge graphs within recommender systems.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [59]
The citation lists haven’t been fetched yet
References 69 · Cited by 78
- Authors
- Heinzerling, B., & Inui, K.
- Affiliations
- Tohoku University (JP)
- Corresponding author
- Heinzerling, Benjamin
- ORCID
- Inui, Kentaro
- Venue
- Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
- Published
- 2021
- Pages
- 1772-1791
- OpenAlex
- W3146844750
- Cited by
- 78
- References
- 69
- Citations by year
- 2026: 4 · 2025: 11 · 2024: 17 · 2023: 26 · 2022: 17 · 2021: 3
- FWCI
- 7.28
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the European Chapter of the Association for Computational Linguistics (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- entity representation, storage capacity, natural language querying, entity linking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
