The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Distant supervision for relation extraction without labeled data
Open accessAbstract
Modern models of relation extraction for tasks like ACE are based on supervised learning of relations from small hand-labeled corpora. We investigate an alternative paradigm that does not require labeled corpora, avoiding the domain dependence of ACE-style algorithms, and allowing the use of corpora of any size. Our experiments use Freebase, a large semantic database of several thousand relations, to provide distant supervision. For each pair of entities that appears in some Freebase relation, we find all sentences containing those entities in a large unlabeled corpus and extract textual features to train a relation classifier. Our algorithm combines the advantages of supervised IE (combining 400,000 noisy pattern features in a probabilistic classifier) and unsupervised IE (extracting large numbers of relations from large corpora of any domain). Our model is able to extract 10,000 instances of 102 relations at a precision of 67.6%. We also analyze feature performance, showing that syntactic parse features are particularly helpful for relations that are ambiguous or lexically distant in their expression.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Freebase: a collaboratively created graph database for structuring human knowledge
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., & Taylor, J. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376746 · Cited by 4,852
Automatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · Proceedings of the 14th conference on Computational linguistics - · doi:10.3115/992133.992154 · Cited by 3,253
Incorporating non-local information into information extraction systems by Gibbs sampling
Finkel, J. R., Grenager, T., & Manning, C. (2005) · Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05 · doi:10.3115/1219840.1219885 · Cited by 3,008
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
Self-citationOpen information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · International Joint Conference on Artificial Intelligence · Cited by 1,398
Snowball: extracting relations from large plain-text collections
Agichtein, E., & Gravano, L. (2000) · Proceedings of the fifth ACM conference on Digital libraries · doi:10.1145/336597.336644 · Cited by 1,147
Unsupervised named-entity extraction from the Web: An experimental study
Etzioni, O., Cafarella, M., Downey, D., Popescu, A.-M., Shaked, T., Soderland, S., Weld, D. S., & Yates, A. (2005) · Artificial Intelligence · doi:10.1016/j.artint.2005.03.001 · Cited by 1,124
The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation
Doddington, G. R., Mitchell, A., Przybocki, M. A., Ramshaw, L., Strassel, S. M., & Weischedel, R. (2004) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4o6ysc5sicba · Cited by 1,029
Open information extraction from the web
Etzioni, O., Banko, M., Soderland, S., & Weld, D. S. (2008) · Communications of the ACM · doi:10.1145/1409360.1409378 · Cited by 1,012
Extracting Patterns and Relations from the World Wide Web
Brin, S. (1999) · Lecture Notes in Computer Science · doi:10.1007/10704656_11 · Cited by 999
Cited by
Items 101–113 of 113
Harvesting facts from textual web sources by constrained label propagation
Wang, Y., Yang, B., Qu, L., Spaniol, M., & Weikum, G. (2011) · Proceedings of the 20th ACM international conference on Information and knowledge management · doi:10.1145/2063576.2063698 · Cited by 37
Automatic Construction of Subject Knowledge Graph based on Educational Big Data
Su, Y., & Zhang, Y. (2020) · Proceedings of the 2020 3rd International Conference on Big Data and Education · doi:10.1145/3396452.3396458 · Cited by 34
Barbosa, N. M., Hayes, J., Kaushik, S., & Wang, Y. (2022) · ACM Transactions on Accessible Computing · doi:10.1145/3519032 · Cited by 28
“We used a combination of data collected via distant supervision [45] (i.e., weakly labeling via heuristics) and manual labeling on Amazon Mechanical Turk (AMT).”
BackgroundMethodInfluentialAutomated clinical knowledge graph generation framework for evidence based medicine.
Alam, F., Giglou, H. B., & Malik, K. M. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120964 · Cited by 27
Leveraging Semantic Parsing for Relation Linking over Knowledge Bases
Mihindukulasooriya, N., Rossiello, G., Kapanipathi, P., Abdelaziz, I., Ravishankar, S., Yu, M., Gliozzo, A., Roukos, S., & Gray, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-62419-4_23 · Cited by 24
Investigating Software Usage in the Social Sciences: A Knowledge Graph Approach
Schindler, D., Zapilko, B., & Krüger, F. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-49461-2_16 · Cited by 24
Machine Learning for Refining Knowledge Graphs: A Survey
Subagdja, B., Shanthoshigaa, D., Wang, Z., & Tan, A.-H. (2024) · ACM Computing Surveys · doi:10.1145/3640313 · Cited by 24
Combination of conditional random field with a rule based method in the extraction of PICO elements.
Chabou, S., & Iglewski, M. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0699-2 · Cited by 19
Hoang, L., Jiang, L., & Kilicoglu, H. (2021) · doi:10.1101/2021.09.14.21263586 · Cited by 5
“One well-known technique is distant supervision 13 , based on using domain knowledge in external knowledge bases.”
BackgroundMethodBuilding a large gene expression-cancer knowledge base with limited human annotations
Marchesin, S., Menotti, L., Giachelle, F., Silvello, G., & Alonso, O. (2023) · Database · doi:10.1093/database/baad061 · Cited by 4
Post-Estimation Smoothing: A Simple Baseline for Learning with Side Information
Rolf, E., Jordan, M. I., & Recht, B. (2020) · arXiv · doi:10.48550/arxiv.2003.05955 · Cited by 3
Attributable by Construction: Claim-Anchored Provenance for Multi-Document Summarization
Guan, S. (2026) · arXiv · doi:10.48550/arxiv.2606.23989
“Silver training labels come from distant supervision (Mintz et al., 2009), aligning gold reference sentences to clusters by M sel entailment; neither M ver nor M eval touches them, so the selector trains independently of the other modules (Appendix D).”
CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Zhao, J., Shin, C., Huang, T.-H., GNVV, S. S. S. N., & Sala, F. (2026) · arXiv · doi:10.48550/arxiv.2603.00039
Metadata
- Authors
- Mintz, M., Bills, S., Snow, R., & Jurafsky, D.
- Venue
- Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - ACL-IJCNLP '09
- Published
- 2009
- Volume
- 2
- Pages
- 1003
- OpenAlex
- W2107598941
- Cited by
- 2,752
- References
- 25
- FWCI
- 30.08
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.