The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Snowball: extracting relations from large plain-text collections
Abstract
Text documents often contain valuable structured data that is hidden Yin regular English sentences. This data is best exploited infavailable as arelational table that we could use for answering precise queries or running data mining tasks.We explore a technique for extracting such tables from document collections that requires only a handful of training examples from users. These examples are used to generate extraction patterns, that in turn result in new tuples being extracted from the document collection.We build on this idea and present our Snowball system. Snowball introduces novel strategies for generating patterns and extracting tuples from plain-text documents.At each iteration of the extraction process, Snowball evaluates the quality of these patterns and tuples without human intervention,and keeps only the most reliable ones for the next iteration. In this paper we also develop a scalable evaluation methodology and metrics for our task, and present a thorough experimental evaluation of Snowball and comparable techniques over a collection of more than 300,000 newspaper documents.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Indexing by latent semantic analysis
Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199009)41:6<391::aid-asi1>3.0.co;2-9 · Cited by 12,751
Introduction to Modern Information Retrieval
Salton, G., & McGill, M. J. (1983) · Virtual Defense Library (Ministerio de Defensa) · Cited by 11,392
Combining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Automatic text processing: the transformation, analysis, and retrieval of information by computer
(1989) · Choice Reviews Online · doi:10.5860/choice.27-0351 · Cited by 3,188
Information Retrieval: Data Structures and Algorithms
Frakes, W. B., & Baeza-Yates, R. A. (1992) · Cited by 2,757
Unsupervised word sense disambiguation rivaling supervised methods
Yarowsky, D. (1995) · Proceedings of the 33rd annual meeting on Association for Computational Linguistics - · doi:10.3115/981658.981684 · Cited by 2,385
Extracting Patterns and Relations from the World Wide Web
Brin, S. (1999) · Lecture Notes in Computer Science · doi:10.1007/10704656_11 · Cited by 999
Unsupervised Models for Named Entity Classification
Collins, M., & Singer, Y. (1999) · Cited by 819
Automatically generating extraction patterns from untagged text
Riloff, E. (1996) · Cited by 621
Learning to construct knowledge bases from the World Wide Web
Craven, M., DiPasquo, D., Freitag, D., McCallum, A., Mitchell, T., Nigam, K., & Slattery, S. (2000) · Artificial Intelligence · doi:10.1016/s0004-3702(00)00004-7 · Cited by 464
Cited by
Distant supervision for relation extraction without labeled data
Mintz, M., Bills, S., Snow, R., & Jurafsky, D. (2009) · Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - ACL-IJCNLP '09 · doi:10.3115/1690219.1690287 · Cited by 2,752
Yago: a core of semantic knowledge
Suchanek, F. M., Kasneci, G., & Weikum, G. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242667 · Cited by 2,512
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.20 · Cited by 1,305
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
Unsupervised named-entity extraction from the Web: An experimental study
Etzioni, O., Cafarella, M., Downey, D., Popescu, A.-M., Shaked, T., Soderland, S., Weld, D. S., & Yates, A. (2005) · Artificial Intelligence · doi:10.1016/j.artint.2005.03.001 · Cited by 1,124
Open information extraction from the web
Etzioni, O., Banko, M., Soderland, S., & Weld, D. S. (2008) · Communications of the ACM · doi:10.1145/1409360.1409378 · Cited by 1,012
How Can We Know What Language Models Know?
Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00324 · Cited by 1,002
YAGO: A Large Ontology from Wikipedia and WordNet
Suchanek, F. M., Kasneci, G., & Weikum, G. (2008) · Journal of Web Semantics · doi:10.1016/j.websem.2008.06.001 · Cited by 904
Dependency tree kernels for relation extraction
Culotta, A., & Sorensen, J. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1219009 · Cited by 814
Exploring various knowledge in relation extraction
GuoDong, Z., Jian, S., Jie, Z., & Min, Z. (2005) · Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05 · doi:10.3115/1219840.1219893 · Cited by 764
Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions
Shen, W., Wang, J., & Han, J. (2015) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2014.2327028 · Cited by 761
Web-scale information extraction in knowitall: (preliminary results)
Etzioni, O., Cafarella, M., Downey, D., Kok, S., Popescu, A.-M., Shaked, T., Soderland, S., Weld, D. S., & Yates, A. (2004) · Proceedings of the 13th international conference on World Wide Web · doi:10.1145/988672.988687 · Cited by 747
Sarawagi, S. (2008) · Foundations and Trends in Databases · doi:10.1561/1900000003 · Cited by 707
Aggarwal, C. C. (2015) · Data Mining · doi:10.1007/978-3-319-14142-8_13 · Cited by 606
Collective knowledge systems: Where the Social Web meets the Semantic Web
Gruber, T. (2008) · Journal of Web Semantics · doi:10.1016/j.websem.2007.11.011 · Cited by 606
Random Walk Inference and Learning in A Large Scale Knowledge Base
Lao, N., Mitchell, T. M., & Cohen, W. W. (2018) · Figshare · doi:10.1184/r1/6476267 · Cited by 482
Coupled semi-supervised learning for information extraction
Carlson, A., Betteridge, J., Wang, R. C., Hruschka, E. R., & Mitchell, T. M. (2010) · Proceedings of the third ACM international conference on Web search and data mining · doi:10.1145/1718487.1718501 · Cited by 416
A Comprehensive Survey on Automatic Knowledge Graph Construction
Zhong, L., Wu, J., Li, Q., Peng, H., & Wu, X. (2024) · ACM Computing Surveys · doi:10.1145/3618295 · Cited by 382
Data-driven materials research enabled by natural language processing and information extraction
Olivetti, E. A., Cole, J. M., Kim, E., Kononova, O., Ceder, G., Han, T. Y.-J., & Hiszpanski, A. M. (2020) · Applied Physics Reviews · doi:10.1063/5.0021106 · Cited by 302
“76 For the modified Snowball algorithm, the quaternary relationships included the property specifier, chemical entity mention, property value, and then unit.”
MethodClassifying semantic relations in bioscience texts
Rosario, B., & Hearst, M. A. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1219010 · Cited by 257
StatSnowball: a statistical approach to extracting entity relationships
Zhu, J., Nie, Z., Liu, X., Zhang, B., & Wen, J.-R. (2009) · Proceedings of the 18th international conference on World wide web · doi:10.1145/1526709.1526724 · Cited by 252
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification
Schick, T., Schmid, H., & Schütze, H. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.488 · Cited by 184
A stopping criterion for active learning
Vlachos, A. (2008) · Computer Speech & Language · doi:10.1016/j.csl.2007.12.001 · Cited by 153
RevMiner: an extractive interface for navigating reviews on a smartphone
Huang, J., Etzioni, O., Zettlemoyer, L., Clark, K., & Lee, C. (2012) · Proceedings of the 25th annual ACM symposium on User interface software and technology · doi:10.1145/2380116.2380120 · Cited by 37
Extraction of temporal facts and events from Wikipedia
Kuzey, E., & Weikum, G. (2012) · Proceedings of the 2nd Temporal Web Analytics Workshop · doi:10.1145/2169095.2169101 · Cited by 32
Metadata
- Authors
- Agichtein, E., & Gravano, L.
- Venue
- Proceedings of the fifth ACM conference on Digital libraries
- Publisher
- ACM
- Published
- 2000-06
- Pages
- 85-94
- OpenAlex
- W2103931177
- Cited by
- 1,147
- References
- 16
- FWCI
- 11.86
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.