The DOI matches its Crossref record; 96 of the 96 references it cites were found in a registry or an index.
Completeness, Recall, and Negation in Open-world Knowledge Bases: A Survey
Abstract
General-purpose knowledge bases (KBs) are a cornerstone of knowledge-centric AI. Many of them are constructed pragmatically from web sources and are thus far from complete. This poses challenges for the consumption as well as the curation of their content. While several surveys target the problem of completing incomplete KBs, the first problem is arguably to know whether and where the KB is incomplete in the first place, and to which degree. In this survey, we discuss how knowledge about completeness, recall, and negation in KBs can be expressed, extracted, and inferred. We cover (i) the logical foundations of knowledge representation and querying under partial closed-world semantics; (ii) the estimation of this information via statistical patterns; (iii) the extraction of information about recall from KBs and text; (iv) the identification of interesting negative statements; and (v) relaxed notions of relative recall. This survey is targeted at two types of audiences: (1) practitioners who are interested in tracking KB quality, focusing extraction efforts, and building quality-aware downstream applications; and (2) data management, knowledge base, and semantic web researchers who wish to understand the state-of-the-art of knowledge bases beyond the open-world assumption. Consequently, our survey presents both fundamental methodologies and the results that they have produced, and gives practice-oriented recommendations on how to choose between different approaches for a problem at hand.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Advances In Experimental Social Psychology
(2022) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(22)x0002-6 · Cited by 22,259
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
Probability Inequalities for sums of Bounded Random Variables
Hoeffding, W. (1994) · Springer series in statistics · doi:10.1007/978-1-4612-0865-5_26 · Cited by 6,683
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Wikidata: a free collaborative knowledgebase
Vrandečić, D., & Krötzsch, M. (2014) · Communications of the ACM · doi:10.1145/2629489 · Cited by 3,342
A Survey on Knowledge Graphs: Representation, Acquisition, and Applications
Ji, S., Pan, S., Cambria, E., Marttinen, P., & Yu, P. S. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3070843 · Cited by 2,947
Yago: a core of semantic knowledge
Suchanek, F. M., Kasneci, G., & Weikum, G. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242667 · Cited by 2,512
Self-citationDBpedia: A Nucleus for a Web of Open Data
Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., & Ives, Z. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-76298-0_52 · Cited by 2,454
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
Speer, R., Chin, J., & Havasi, C. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11164 · Cited by 2,312
Research works printed but not linked: 26
- Christoph Alt Marc Hübner and Leonhard Hennig. 2019. Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. In Annual Meeting of the Association for Computational Linguistics (ACL). 1388–1398.
- Hiba Arnaout. 2023. Enriching Open-world Knowledge Graphs with Expressive Negative Statements. Ph. D. Dissertation.
- Hiba Arnaout and Simon Razniewski. 2023. Can large language models generate salient negative statements? In Knowledge Base Construction from Pre-trained Language Models Workshop at ISWC 2023 (KBC-LM).
- Frank Benford. 1938. The law of anomalous numbers. Proc. Am. Philos. Societ. (1938).
- Kurt Bollacker Robert Cook and Patrick Tufts. 2007. Freebase: A shared database of structured general human knowledge. In Proceedings of the 22nd National Conference on Artificial Intelligence - Volume 2 (AAAI’07). 10.5555/1619797.1619981 · doi:10.5555/1619797.1619981
- Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng, Lei Li, and Yanghua Xiao. 2023. Say what you mean! Large language models speak too positively about negative commonsense knowledge. In ACL.
- Herbert P. Grice. 1975. Logic and conversation. In Speech Acts. Brill.
- Chadi Helwe Chloé Clavel and Fabian Suchanek. 2021. Reasoning with transformer-based models: Deep learning but shallow reasoning. In International Conference on Automated Knowledge Base Construction (AKBC).
- Johannes Hoffart Dragan Milchevski and Gerhard Weikum. 2014. STICS: Searching with strings things and cats. In Conference on Research and Development in Information Retrieval.
- Bernhard Hollunder and Franz Baader. 1991. Qualifying number restrictions in concept languages. In Proceedings of the 2nd International Conference on Principles of Knowledge Representation and Reasoning (KR’91). Morgan Kaufmann Publishers Inc. 335–346.
- Martin Josifoski, Nicola De Cao, Maxime Peyrard, Fabio Petroni, and Robert West. 2022. GenIE: Generative information extraction. In NAACL.
- Nicholas Klein, Filip Ilievski, Hayden Freedman, and Pedro Szekely. 2022. Identifying surprising facts in Wikidata. In Wikidata Workshop.
- Alon Y. Levy. 1996. Obtaining complete answers from incomplete databases. In Proceedings of the 22th International Conference on Very Large Data Bases (VLDB’96). 402–412.
- Mausam Michael Schmitz Stephen Soderland Robert Bart and Oren Etzioni. 2012. Open language learning for information extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL’12). Association for Computational Linguistics 523–534.
- Ndapandula Nakashole Gerhard Weikum and Fabian Suchanek. 2012. PATTY: A taxonomy of relational patterns with semantic types. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL’12). Association for Computational Linguistics 1135–1145.
- Jeff Z. Pan Simon Razniewski Jan-Christoph Kalo Sneha Singhania Jiaoyan Chen Stefan Dietze Hajira Jabeen Janna Omeliyanenko Wen Zhang Matteo Lissandrini Russa Biswas Gerard de Melo Angela Bonifati Edlira Vakaj Mauro Dragoni and Damien Graux. 2023. Large Language Models and Knowledge Graphs: Opportunities and Challenges. arXiv:2308.06374 · arXiv
- Pierre-Henri Paris Syrine El Aoud and Fabian M. Suchanek. 2021. The vagueness of vagueness in noun phrases. In International Conference on Automated Knowledge Base Construction (AKBC’21). https://imt.hal.science/hal-03344675
- Soumajit Pramanik Jesujoba Alabi Rishiraj Saha Roy and Gerhard Weikum. 2021. UNIQORN: Unified question answering over RDF knowledge graphs and natural language text. arXiv:2108.08614. · arXiv
- Radityo Eko Prasojo Fariz Darari Simon Razniewski and Werner Nutt. 2016. Managing and consuming completeness information for Wikidata using COOL-WD. CEUR Workshop Proceedings on 7th International Workshop on Consuming Linked Data 1666 (2016).
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- Tara Safavi and Danai Koutra. 2020. Generating negative commonsense knowledge. In 4th Knowledge Representation and Reasoning Meets Machine Learning Workshop at NeurIPS.
- Sneha Singhania, Jan-Christoph Kalo, Simon Razniewski, and Jeff Z. Pan. 2023. Knowledge base construction from pre-trained language models. In Challenge at ISWC 2023.
- Sneha Singhania, Tuan-Phong Nguyen, and Simon Razniewski. 2022. LM-KBC: Knowledge base construction from pre-trained language models. In Challenge at ISWC 2022. Retrieved from https://ceur-ws.org/Vol-3274/paper1.pdf
- Jannik Strötgen and Michael Gertz. 2010. HeidelTime: High quality rule-based extraction and normalization of temporal expressions. In Proceedings of the 5th International Workshop on Semantic Evaluation (SemEval’10). Association for Computational Linguistics 321–324.
- Fabian M. Suchanek and Anh Tuan Luu. 2023. Knowledge bases and language models: Complementing forces. In RuleML+RR Invited Paper.
- Ian H. Witten and David N. Milne. 2008. An effective low-cost measure of semantic relatedness obtained from Wikipedia links. In Proceeding of AAAI Workshop on Wikipedia and Artificial Intelligence: an Evolving Synergy. AAAI Press 25–30.
Metadata
- Authors
- Razniewski, S., Arnaout, H., Ghosh, S., & Suchanek, F.
- Venue
- ACM Computing Surveys
- Published
- 2024-02-23
- Volume
- 56
- Issue
- 6
- Pages
- 1-42
- ISSN
- 0360-0300
- 1557-7341
- Cited by
- 8
- References
- 96
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Completeness, Recall, and Negation in Open-world Knowledge Bases: A Survey. https://app.cochat.ai/papers/10.1145/3639563
- The paper
- Razniewski, S., Arnaout, H., Ghosh, S., & Suchanek, F. (2024). Completeness, Recall, and Negation in Open-world Knowledge Bases: A Survey. ACM Computing Surveys. https://doi.org/10.1145/3639563
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
