Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 8

The DOI matches its Crossref record; 96 of the 96 references it cites were found in a registry or an index.

Completeness, Recall, and Negation in Open-world Knowledge Bases: A Survey

Razniewski, Simon, Arnaout, Hiba, Ghosh, Shrestha, Suchanek, Fabian·2024·ACM Computing Surveys
DOI 10.1145/3639563
Open this paper in CoChat — free account

Abstract

General-purpose knowledge bases (KBs) are a cornerstone of knowledge-centric AI. Many of them are constructed pragmatically from web sources and are thus far from complete. This poses challenges for the consumption as well as the curation of their content. While several surveys target the problem of completing incomplete KBs, the first problem is arguably to know whether and where the KB is incomplete in the first place, and to which degree. In this survey, we discuss how knowledge about completeness, recall, and negation in KBs can be expressed, extracted, and inferred. We cover (i) the logical foundations of knowledge representation and querying under partial closed-world semantics; (ii) the estimation of this information via statistical patterns; (iii) the extraction of information about recall from KBs and text; (iv) the identification of interesting negative statements; and (v) relaxed notions of relative recall. This survey is targeted at two types of audiences: (1) practitioners who are interested in tracking KB quality, focusing extraction efforts, and building quality-aware downstream applications; and (2) data management, knowledge base, and semantic web researchers who wish to understand the state-of-the-art of knowledge bases beyond the open-world assumption. Consequently, our survey presents both fundamental methodologies and the results that they have produced, and gives practice-oriented recommendations on how to choose between different approaches for a problem at hand.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3639563

Cites

96 of 96 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 26

  1. Christoph Alt Marc Hübner and Leonhard Hennig. 2019. Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. In Annual Meeting of the Association for Computational Linguistics (ACL). 1388–1398.
  2. Hiba Arnaout. 2023. Enriching Open-world Knowledge Graphs with Expressive Negative Statements. Ph. D. Dissertation.
  3. Hiba Arnaout and Simon Razniewski. 2023. Can large language models generate salient negative statements? In Knowledge Base Construction from Pre-trained Language Models Workshop at ISWC 2023 (KBC-LM).
  4. Frank Benford. 1938. The law of anomalous numbers. Proc. Am. Philos. Societ. (1938).
  5. Kurt Bollacker Robert Cook and Patrick Tufts. 2007. Freebase: A shared database of structured general human knowledge. In Proceedings of the 22nd National Conference on Artificial Intelligence - Volume 2 (AAAI’07). 10.5555/1619797.1619981 · doi:10.5555/1619797.1619981
  6. Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng, Lei Li, and Yanghua Xiao. 2023. Say what you mean! Large language models speak too positively about negative commonsense knowledge. In ACL.
  7. Herbert P. Grice. 1975. Logic and conversation. In Speech Acts. Brill.
  8. Chadi Helwe Chloé Clavel and Fabian Suchanek. 2021. Reasoning with transformer-based models: Deep learning but shallow reasoning. In International Conference on Automated Knowledge Base Construction (AKBC).
  9. Johannes Hoffart Dragan Milchevski and Gerhard Weikum. 2014. STICS: Searching with strings things and cats. In Conference on Research and Development in Information Retrieval.
  10. Bernhard Hollunder and Franz Baader. 1991. Qualifying number restrictions in concept languages. In Proceedings of the 2nd International Conference on Principles of Knowledge Representation and Reasoning (KR’91). Morgan Kaufmann Publishers Inc. 335–346.
  11. Martin Josifoski, Nicola De Cao, Maxime Peyrard, Fabio Petroni, and Robert West. 2022. GenIE: Generative information extraction. In NAACL.
  12. Nicholas Klein, Filip Ilievski, Hayden Freedman, and Pedro Szekely. 2022. Identifying surprising facts in Wikidata. In Wikidata Workshop.
  13. Alon Y. Levy. 1996. Obtaining complete answers from incomplete databases. In Proceedings of the 22th International Conference on Very Large Data Bases (VLDB’96). 402–412.
  14. Mausam Michael Schmitz Stephen Soderland Robert Bart and Oren Etzioni. 2012. Open language learning for information extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL’12). Association for Computational Linguistics 523–534.
  15. Ndapandula Nakashole Gerhard Weikum and Fabian Suchanek. 2012. PATTY: A taxonomy of relational patterns with semantic types. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL’12). Association for Computational Linguistics 1135–1145.
  16. Jeff Z. Pan Simon Razniewski Jan-Christoph Kalo Sneha Singhania Jiaoyan Chen Stefan Dietze Hajira Jabeen Janna Omeliyanenko Wen Zhang Matteo Lissandrini Russa Biswas Gerard de Melo Angela Bonifati Edlira Vakaj Mauro Dragoni and Damien Graux. 2023. Large Language Models and Knowledge Graphs: Opportunities and Challenges. arXiv:2308.06374 · arXiv
  17. Pierre-Henri Paris Syrine El Aoud and Fabian M. Suchanek. 2021. The vagueness of vagueness in noun phrases. In International Conference on Automated Knowledge Base Construction (AKBC’21). https://imt.hal.science/hal-03344675
  18. Soumajit Pramanik Jesujoba Alabi Rishiraj Saha Roy and Gerhard Weikum. 2021. UNIQORN: Unified question answering over RDF knowledge graphs and natural language text. arXiv:2108.08614. · arXiv
  19. Radityo Eko Prasojo Fariz Darari Simon Razniewski and Werner Nutt. 2016. Managing and consuming completeness information for Wikidata using COOL-WD. CEUR Workshop Proceedings on 7th International Workshop on Consuming Linked Data 1666 (2016).
  20. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
  21. Tara Safavi and Danai Koutra. 2020. Generating negative commonsense knowledge. In 4th Knowledge Representation and Reasoning Meets Machine Learning Workshop at NeurIPS.
  22. Sneha Singhania, Jan-Christoph Kalo, Simon Razniewski, and Jeff Z. Pan. 2023. Knowledge base construction from pre-trained language models. In Challenge at ISWC 2023.
  23. Sneha Singhania, Tuan-Phong Nguyen, and Simon Razniewski. 2022. LM-KBC: Knowledge base construction from pre-trained language models. In Challenge at ISWC 2022. Retrieved from https://ceur-ws.org/Vol-3274/paper1.pdf
  24. Jannik Strötgen and Michael Gertz. 2010. HeidelTime: High quality rule-based extraction and normalization of temporal expressions. In Proceedings of the 5th International Workshop on Semantic Evaluation (SemEval’10). Association for Computational Linguistics 321–324.
  25. Fabian M. Suchanek and Anh Tuan Luu. 2023. Knowledge bases and language models: Complementing forces. In RuleML+RR Invited Paper.
  26. Ian H. Witten and David N. Milne. 2008. An effective low-cost measure of semantic relatedness obtained from Wikipedia links. In Proceeding of AAAI Workshop on Wikipedia and Artificial Intelligence: an Evolving Synergy. AAAI Press 25–30.

Metadata

Authors
Razniewski, S., Arnaout, H., Ghosh, S., & Suchanek, F.
Venue
ACM Computing Surveys
Publisher
Association for Computing Machinery (ACM)
Published
2024-02-23
Volume
56
Issue
6
Pages
1-42
ISSN
  • 0360-0300
  • 1557-7341
DOI
10.1145/3639563
Cited by
8
References
96
Language
en
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.