Snorkel: rapid training data creation with weak supervision
Open accessThe DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Abstract
Labeling training data is increasingly the largest bottleneck in deploying machine learning systems. We present Snorkel, a first-of-its-kind system that enables users to train state-of-the-art models without hand labeling any training data. Instead, users write labeling functions that express arbitrary heuristics, which can have unknown accuracies and correlations. Snorkel denoises their outputs without access to ground truth by incorporating the first end-to-end implementation of our recently proposed machine learning paradigm, data programming. We present a flexible interface layer for writing labeling functions based on our experience over the past year collaborating with companies, agencies, and research labs. In a user study, subject matter experts build models 2.8X faster and increase predictive performance an average 45.5% versus seven hours of hand labeling. We study the modeling tradeoffs in this new setting and propose an optimizer for automating tradeoff decisions that gives up to 1.8X speedup per pipeline execution. In two collaborations, with the U.S. Department of Veterans Affairs and the U.S. Food and Drug Administration, and on four open-source text and image data sets representative of other deployments, Snorkel provides 132% average improvements to predictive performance over prior heuristic approaches and comes within an average 3.60% of the predictive performance of large hand-curated training sets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Annotation schemes developed to iteratively improve imperfect data -such as weak supervision (Ratner et al. 2017), targeted data cleaning (Chen, Yu, and Bowman 2022), and active learning (Yuan, Lin, and Boyd-Graber 2020;Li et al. 2022) -avoid LLM pitfalls by allowing human validation to refine the original model.”
From Can Large Language Models Transform Computational Social Science? (2024) · cites this paper as Ratner et al. 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · cited 1×
“Transfer learning considers the strategy of repurposing models trained on different datasets or tasks where labeled training data is more abundant [34].”
BackgroundTensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 5,839
Combining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196
Caspi, R., Billington, R., Ferrer, L., Foerster, H., Fulcher, C. A., Keseler, I. M., Kothari, A., Krummenacker, M., Latendresse, M., Mueller, L. A., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkv1164 · Cited by 3,395
Automatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · Proceedings of the 14th conference on Computational linguistics - · doi:10.3115/992133.992154 · Cited by 3,253
DBpedia – A large-scale, multilingual knowledge base extracted from Wikipedia
Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P. N., Hellmann, S., Morsey, M., van Kleef, P., Auer, S., et al. (2015) · Semantic Web · doi:10.3233/sw-140134 · Cited by 3,168
Distant supervision for relation extraction without labeled data
Mintz, M., Bills, S., Snow, R., & Jurafsky, D. (2009) · Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - ACL-IJCNLP '09 · doi:10.3115/1690219.1690287 · Cited by 2,752
International Conference on Machine Learning (ICML)-2005
津田, 宏. (2006) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 2,558
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
Dawid, A. P., & Skene, A. M. (1979) · Applied Statistics · doi:10.2307/2346806 · Cited by 1,692 · cited 1×
“Researchers have therefore considered how to estimate the accuracy of label sources without a gold standard with which to compare—a classic problem [13]—and combine these estimates into labels that can be used to train an arbitrary end model.”
BackgroundFinding a "Kneedle" in a Haystack: Detecting Knee Points in System Behavior
Satopaa, V., Albrecht, J., Irwin, D., & Raghavan, B. (2011) · 2011 31st International Conference on Distributed Computing Systems Workshops · doi:10.1109/icdcsw.2011.20 · Cited by 1,274
Modeling Relations and Their Mentions without Labeled Text
Riedel, S., Yao, L., & McCallum, A. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15939-8_10 · Cited by 1,270
Settles, B. (2012) · Cited by 1,183
The Comparative Toxicogenomics Database: update 2019
Davis, A. P., Grondin, C. J., Johnson, R. J., Sciaky, D., McMorran, R., Wiegers, J., Wiegers, T. C., & Mattingly, C. J. (2019) · Nucleic Acids Research · doi:10.1093/nar/gky868 · Cited by 976
Knowledge-Based Weak Supervision for Information Extraction of Overlapping Relations
Hoffmann, R., Zhang, C., Ling, X., Zettlemoyer, L., & Weld, D. S. (2011) · Cited by 874
Neural Information Processing Systems (NIPS)
Shang, X., Zhu, Z., Leimkuhler, B., & Storkey, A. (2015) · Cited by 831
Human computation: a survey and taxonomy of a growing field
Quinn, A. J., & Bederson, B. B. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979148 · Cited by 801
The Comparative Toxicogenomics Database: update 2017
Davis, A. P., Grondin, C. J., Johnson, R. J., Sciaky, D., King, B. L., McMorran, R., Wiegers, J., Wiegers, T. C., & Mattingly, C. J. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw838 · Cited by 544
Dong, X. L., & Srivastava, D. (2015) · Synthesis Lectures on Data Management · doi:10.1007/978-3-031-01853-4 · Cited by 539
Probability of error of some adaptive pattern-recognition machines
Scudder, H. (1965) · IEEE Transactions on Information Theory · doi:10.1109/tit.1965.1053799 · Cited by 518
The Comparative Toxicogenomics Database: update 2013
Davis, A. P., Murphy, C. G., Johnson, R., Lay, J. M., Lennon-Hopkins, K., Saraceni-Richards, C., Sciaky, D., King, B. L., Rosenstein, M. C., Wiegers, T. C., et al. (2013) · Nucleic Acids Research · doi:10.1093/nar/gks994 · Cited by 497
Caspi, R., Altman, T., Dale, J. M., Dreher, K., Fulcher, C. A., Gilham, F., Kaipa, P., Karthikeyan, A. S., Kothari, A., Krummenacker, M., et al. (2010) · Nucleic Acids Research · doi:10.1093/nar/gkp875 · Cited by 483
HoloClean: holistic data repairs with probabilistic inference
Rekatsinas, T., Chu, X., Ilyas, I. F., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3137628.3137631 · Cited by 480
Self-citationA Survey of Crowdsourcing Systems
Yuen, M.-C., King, I., & Leung, K.-S. (2011) · 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing · doi:10.1109/passat/socialcom.2011.203 · Cited by 392
Data Programming: Creating Large Training Sets, Quickly
Ratner, A., De, C., Wu, S., Selsam, D., & Ré, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07723 · Cited by 350
Self-citationLi, Y., Gao, J., Meng, C., Li, Q., Su, L., Zhao, B., Fan, W., & Han, J. (2016) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2897350.2897352 · Cited by 336
The 2017 Conference on Empirical Methods on Natural Language Processing (EMNLP)
Narayan, S., Gardent, C., Cohen, S. B., & Shimorina, A. (2015) · Cited by 276
Label-Free Supervision of Neural Networks with Physics and Domain Knowledge
Stewart, R., & Ermon, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10934 · Cited by 274
Spectral Methods meet EM: A Provably Optimal Algorithm for CrowdsourcingPreprint
Zhang, Y., Chen, X., Zhou, D., & Jordan, M. I. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.3824 · Cited by 266
A Bayesian approach to discovering truth from conflicting sources for data integration
Zhao, B., Rubinstein, B. I. P., Gemmell, J., & Han, J. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2168651.2168656 · Cited by 264
Learning to Extract Relations from the Web using Minimal Supervision
Bunescu, R., & Mooney, R. J. (2007) · Cited by 261
Reducing Wrong Labels in Distant Supervision for Relation Extraction
Takamatsu, S., Sato, I., & Nakagawa, H. (2012) · Cited by 221
Wei, C.-H., Peng, Y., Leaman, R., Davis, A. P., Mattingly, C. J., Li, J., Wiegers, T. C., & Lu, Z. (2016) · Database · doi:10.1093/database/baw032 · Cited by 198
Learning the Structure of Generative Models without Labeled Data
Bach, S. H., He, B. D., Ratner, A., & Ré, C. (2017) · PubMed · doi:10.48550/arxiv.1703.00854 · Cited by 190
Self-citationSemi-Supervised Learning (Adaptive Computation and Machine Learning)
Chapelle, O., Schölkopf, B., & Zien, A. (2006) · The MIT Press eBooks · Cited by 185
Snuba: automating weak supervision to label training data
Varma, P., & Ré, C. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3291264.3291268 · Cited by 184
Self-citationGeneralized Expectation Criteria for Semi-Supervised Learning with Weakly Labeled Data
Mann, G., & McCallum, A. (2010) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.5555/1756006.1756038 · Cited by 173
Aggregating crowdsourced binary ratings
Dalvi, N., Dasgupta, A., Kumar, R., & Rastogi, V. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488414 · Cited by 158
Data Programming: Creating Large Training Sets, Quickly.
Ratner, A., De, C., Wu, S., Selsam, D., & Ré, C. (2016) · PubMed · Cited by 150
Self-citationTraining Complex Models with Multi-Task Weak Supervision
Ratner, A., Hancock, B., Dunnmon, J., Sala, F., Pandey, S., & Ré, C. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014763 · Cited by 145
Self-citationLearning with a probabilistic teacher
Agrawala, A. K. (1970) · IEEE Transactions on Information Theory · doi:10.1109/tit.1970.1054472 · Cited by 139
Pochampally, R., Das Sarma, A., Dong, X. L., Meliou, A., & Srivastava, D. (2014) · Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2588555.2593674 · Cited by 128
Improved Pattern Learning for Bootstrapped Entity Extraction
Gupta, S., & Manning, C. D. (2014) · Conference on Computational Natural Language Learning · doi:10.3115/v1/w14-1611 · Cited by 121
Ranking and combining multiple predictors without labeled data
Parisi, F., Strino, F., Nadler, B., & Kluger, Y. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1219097111 · Cited by 120
Training Classifiers with Natural Language Explanations
Hancock, B., Varma, P., Wang, S., Bringmann, M., Liang, P., & Ré, C. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1175 · Cited by 109
Self-citationLearning from measurements in exponential families
Liang, P., Jordan, M. I., & Klein, D. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553457 · Cited by 108
Davis, A. P., Wiegers, T. C., Roberts, P. M., King, B. L., Lay, J. M., Lennon-Hopkins, K., Sciaky, D., Johnson, R., Keating, H., Greene, N., et al. (2013) · Database · doi:10.1093/database/bat080 · Cited by 107
Weakly supervised classification of aortic valve malformations using unlabeled cardiac MRI sequences
Fries, J. A., Varma, P., Chen, V. S., Xiao, K., Tejeda, H., Saha, P., Dunnmon, J., Chubb, H., Maskatia, S., Fiterau, M., et al. (2019) · Nature Communications · doi:10.1038/s41467-019-11012-3 · Cited by 103
Self-citationModeling annotators: a generative approach to learning from annotator rationales
Zaidan, O. F., & Eisner, J. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613721 · Cited by 101
Fonduer: Knowledge Base Construction from Richly Formatted Data
Wu, S., Hsiao, L., Cheng, X., Hancock, B. J., Rekatsinas, T., Levis, P. A., & Ré, C. (2018) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3183713.3183729 · Cited by 99
Self-citationDeepDive: declarative knowledge base construction
Zhang, C., Ré, C., Cafarella, M., De, C., Ratner, A., Shin, J., Wang, F., & Wu, S. (2017) · Communications of the ACM · doi:10.1145/3060586 · Cited by 82
Self-citationWhat do a Million News Articles Look like
Corney, D. P. A., Albakour, D., Martinez-Alvarez, M., & Moussa, S. H. (2016) · Cited by 61
SLiMFast: Guaranteed Results for Data Fusion and Source Reliability
Rekatsinas, T., Joglekar, M., Garcia-Molina, H., Parameswaran, A., & Ré, C. (2017) · Proceedings of the 2017 ACM International Conference on Management of Data · doi:10.1145/3035918.3035951 · Cited by 60
Self-citationPattern Learning for Relation Extraction with a Hierarchical Topic Model
Alfonseca, E., Delort, J.-Y., & Garrido, G. (2013) · Cited by 58
Scheffer, T. (2009) · IGI Global eBooks · doi:10.4018/978-1-60566-010-3.ch272 · Cited by 56
Inferring Generative Model Structure with Static Analysis
Varma, P., He, B. D., Bajaj, P., Banerjee, I., Khandwala, N., Rubin, D. L., & Ré, C. (2017) · PubMed · doi:10.48550/arxiv.1709.02477 · Cited by 56
Self-citationHoloClean: Holistic Data Repairs with Probabilistic InferencePreprint
Rekatsinas, T., Chu, X., Ilyas, I. F., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.00820 · Cited by 54
Comprehensive and reliable crowd assessment algorithms
Joglekar, M., García-Molina, H., & Parameswaran, A. (2015) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2015.7113284 · Cited by 53
Error Rate Analysis of Labeling by Crowdsourcing
Li, H.-W. (2013) · International Conference on Machine Learning · Cited by 51
Snorkel MeTaL: Weak Supervision for Multi-Task Learning
Ratner, A., Hancock, B., Dunnmon, J., Goldman, R., & Ré, C. (2018) · Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning · doi:10.1145/3209889.3209898 · Cited by 43
Self-citationSnorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial ScalePreprint
Bach, S. H., Rodriguez, D., Liu, Y., Luo, C., Shao, H., Xia, C., Sen, S., Ratner, A., Hancock, B. J., Alborzi, H., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00417 · Cited by 37
Self-citationKu, J. P., Hicks, J. L., Hastie, T. J., Leskovec, J., Ré, C., & Delp, S. L. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv071 · Cited by 32
Self-citation
Cited by
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationOpportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310
Scientific discovery in the age of artificial intelligence.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06221-2 · Cited by 1,823
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective
Roh, Y., Heo, G., & Whang, S. E. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2946162 · Cited by 968
Metadata
- Authors
- Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C.
- Venue
- Proceedings of the VLDB Endowment
- Published
- 2017-11
- Volume
- 11
- Issue
- 3
- Pages
- 269-282
- ISSN
- 2150-8097 (ISSN-L)
- OpenAlex
- W2769041395
- Cited by
- 777
- References
- 70
- FWCI
- 44.12
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.