Snorkel MeTaL: Weak Supervision for Multi-Task Learning
Open accessAbstract
Many real-world machine learning problems are challenging to tackle for two reasons: (i) they involve multiple sub-tasks at different levels of granularity; and (ii) they require large volumes of labeled training data. We propose Snorkel MeTaL, an end-to-end system for multi-task learning that leverages weak supervision provided at multiple levels of granularity by domain expert users. In MeTaL, a user specifies a problem consisting of multiple, hierarchically-related sub-tasks---for example, classifying a document at multiple levels of granularity---and then provides labeling functions for each sub-task as weak supervision. MeTaL learns a re-weighted model of these labeling functions, and uses the combined signal to train a hierarchical multi-task network which is automatically compiled from the structure of the sub-tasks. Using MeTaL on a radiology report triage task and a fine-grained news classification task, we achieve average gains of 11.2 accuracy points over a baseline supervised approach and 9.5 accuracy points over the predictions of the user-provided labeling functions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The multi-task setup can confer benefits across very similar tasks.
“though it has been shown that the multi-task setup can confer benefits across very similar tasks Liu et al. (2019b); Ratner et al. (2018).”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Ratner et al. (2018)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 15
- [1]Bach SH, He B, Ratner AJ, and Ré C. Learning the structure of generative models without labeled data. In ICML, 2017. PMC6417840 30882087
- [2]Caruana R. Multitask learning: A knowledge-based source of inductive bias. In ICML, pages 41– 48, 1993.
- [3]Mitchell T. Never-ending learning. In AAAI, 2015.
- [5]Kazuma H, Caiming X, Yoshimasa T, and Socher R. A joint many-task model: Growing a neural network for multiple NLP tasks. CoRR, abs/1611.01587, 2016. · arXiv
- [6]Lewis DD, Yang Y, Rose TG, and Li F. Rcv1: A new benchmark collection for text categorization research. JMLR, 5(Apr): 361– 397, 2004.
- [7]Mintz M, Bills S, Snow R, and Jurafsky D. Distant supervision for relation extraction without labeled data. In Proc ACL, pages 1003– 1011, 2009.
- [8]Mnih Volodymyr and Hinton Geoffrey E. Learning to label aerial images from noisy data. In ICML, pages 567– 574, 2012.
- [10]Paszke A Automatic differentiation in pytorch. In NIPS-W, 2017.
- [11]Platanios E, Poon H, Mitchell TM, and Horvitz EJ. Estimating accuracy from unlabeled data: A probabilistic logic approach. In NIPS, pages 4364– 4373, 2017.
- [12]Ratner AJ, De Sa CM, Wu S, Selsam D, and Ré C. Data programming: Creating large training sets, quickly. In Adv Neural Inf Process Syst, pages 3567– 3575, 2016. 29872252 PMC5985238
- [13]Ratner AJ, Bach SH, Ehrenberg H, Fries J, Wu S, and Ré C. Snorkel: Rapid training data creation with weak supervision. In VLDB, 2018. PMC7075849 32214778
- [14]Rekatsinas T, Joglekar M, Garcia-Molina H, Parameswaran A, and Ré C. Slimfast: Guaranteed results for data fusion and source reliability. In SIGMOD, 2017.
- [15]Ruder S. An overview of multi-task learning in deep neural networks. CoRR, abs/1706.05098, 2017 URL http://arxiv.org/abs/1706.05098. · arXiv
- [16]Søgaard A and Goldberg Y. Deep multi-task learning with low level tasks supervised at lower layers. In Proc ACL, volume 2, pages 231– 235, 2016.
- [17]Xiao T, Xia T, Yang Y, Huang C, and Wang X. Learning from massive noisy labeled data for image classification. In CVPR, pages 2691– 2699, 2015.
Cited by
Sorted by citations
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A., & Lungren, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.117 · Cited by 262 · see citing works
Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A. Y., & Lungren, M. P. (2020) · arXiv · doi:10.48550/arxiv.2004.09167 · Cited by 103 · see citing works
- Authors
- Ratner, A., Hancock, B., Dunnmon, J., Goldman, R., & Ré, C.
- Affiliations
- Stanford University, Stanford, CA
- Stanford University (US)
- Venue
- Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning
- Publisher
- ACM
- Published
- 2018-06-15
- Volume
- 2018
- Pages
- 1-4
- PMC
- PMC6436830
- OpenAlex
- W2804236178
- Cited by
- 43
- References
- 0
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 1 · 2023: 6 · 2022: 4 · 2021: 7 · 2020: 9 · 2019: 12 · 2017: 1
- FWCI
- 2.86
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Workshop on Data Management for End-to-End Machine Learning (DEEM) (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- Defense Advanced Research Projects Agency (FA87501720095, N6600115C4043)
- National Institutes of Health (U54EB02040)
- Office of Naval Research (N000141712266)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multi-task learning, weakly supervised learning, hierarchical multi-task learning, labeling functions, end-to-end learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
