CoChat
Sign Up
Cited by 54

Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale

Bach, Stephen H., Rodriguez, Daniel, Liu, Yintao, Luo, Chong, Shao, Haidong, Xia, Cassandra, Sen, Souvik, Ratner, Alex +5 more·2019·Proceedings of the 2019 International Conference on Management of Data
DOI 10.1145/3299869.3314036PMC PMC6879379
Open this paper in CoChat — free account

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3299869.3314036

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Both WS and UWS are probabilistic graphical models that do not account for confounding factors (WS operates on discrete labels, while UWS handles continuous scores).

    “Both WS and UWS are probabilistic graphical models that do not account for confounding factors (WS operates on discrete labels, while UWS handles continuous scores).”

    From CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation (2026) · cites this paper as Bach et al., 2019

    The Background section describes Snorkel DryBell’s weak-supervision generative model using discrete label votes, and the Sampling-Free Generative Model section specifies a conditionally independent model. Our copy does not discuss UWS or establish that both WS and UWS fail to account for confounding factors; the conditional-independence assumption alone does not support that broader claim. The indexed passages provide no page numbers. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 34

  1. [1]Abadi Martín, Barham Paul, Chen Jianmin, Chen Zhifeng, Davis Andy, Dean Jeffrey, Devin Matthieu, Ghemawat Sanjay, Irving Geoffrey, Isard Michael, Kudlur Manjunath, Levenberg Josh, Monga Rajat, Moore Sherry, Murray Derek G., Steiner Benoit, Tucker Paul, Vasudevan Vijay, Warden Pete, Wicke Martin, Yu Yuan, and Zheng Xiaoqiang. 2016 TensorFlow: A System for Large-scale Machine Learning. In USENIX Conference on Operating Systems Design and Implementation (OSDI).
  2. [2]Agrawala AK. 1970 Learning with a Probabilistic Teacher. IEEE Transactions on Infomation Theory 16 (1970), 373– 379.
  3. [3]Bach Stephen H., He Bryan, Ratner Alexander, and Ré Christopher. 2017 Learning the Structure of Generative Models without Labeled Data. In International Conference on Machine Learning (ICML). PMC6417840 30882087
  4. [4]Baylor Denis, Breck Eric, Cheng Heng-Tze, Fiedel Noah, Foo Chuan Yu, Haque Zakaria, Haykal Salem, Ispir Mustafa, Jain Vihan, Koc Levent, 2017 TFX: A TensorFlow-based production-scale machine learning platform. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD).
  5. [5]Blum A and Mitchell T. 1998 Combining Labeled and Unlabeled Data with Co-Training. In Workshop on Computational Learning Theory (COLT).
  6. [6]Bootkrajang Jakramate and Kabán Ata. 2012 Label-noise robust logistic regression and its applications. In Joint European conference on machine learning and knowledge discovery in databases Springer, 143– 158.
  7. [7]Chapelle O, Schölkopf B, and Zien A (Eds.). 2006 Semi-Supervised Learning. MIT Press.
  8. [8]Cubuk Ekin D, Zoph Barret, Mane Dandelion, Vasudevan Vijay, and Le Quoc V. 2018 AutoAugment: Learning Augmentation Policies from Data. arXiv preprint arXiv:1805.09501 (2018). · arXiv
  9. [9]Dalvi Nilesh, Dasgupta Anirban, Kumar Ravi, and Rastogi Vibhor. 2013 Aggregating crowdsourced binary ratings. In Proceedings of the 22nd international conference on World Wide Web ACM, 285– 294.
  10. [13]Grave Edouard, Cisse Moustapha M, and Joulin Armand. 2017 Unbounded cache model for online language modeling with open vocabulary. In Advances in Neural Information Processing Systems (NeurIPS.
  11. [14]Gupta Sonal, Shah Rushin, Mohit Mrinal, Kumar Anuj, and Lewis Mike. 2018 Semantic parsing for task oriented dialog using hierarchical representations. arXiv preprint arXiv:1810.07942 (2018). · arXiv
  12. [15]Hinton Geoffrey, Vinyals Oriol, and Dean Jeff. 2015 Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015). · arXiv
  13. [16]Hochreiter Sepp and Schmidhuber Jürgen. 1997 Long short-term memory. Neural Computation 9, 8 (1997), 1735– 1780. 9377276
  14. [18]Lee Dong-Hyun. 2013 Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop on Challenges in Representation Learning.
  15. [19]Li Yaliang, Gao Jing, Meng Chuishi, Li Qi, Su Lu, Zhao Bo, Fan Wei, and Han Jiawei. 2015 A Survey on Truth Discovery. SIGKDD Explor. Newsl 17, 2(2015).
  16. [20]Mahajan Dhruv, Girshick Ross, Ramanathan Vignesh, He Kaiming, Paluri Manohar, Li Yixuan, Bharambe Ashwin, and van der Maaten Laurens. 2018 Exploring the Limits of Weakly Supervised Pretraining. In European Conference on Computer Vision (ECCV).
  17. [21]Manning Christopher, Surdeanu Mihai, Bauer John, Finkel Jenny, Bethard Steven, and McClosky David. 2014 The Stanford CoreNLP natural language processing toolkit. In Annual meeting of the Association for Computational Linguistics: System Demonstrations.
  18. [22]McMahan H Brendan, Holt Gary, Sculley David, Young Michael, Ebner Dietmar, Grady Julian, Nie Lan, Phillips Todd, Davydov Eugene, Golovin Daniel, 2013 Ad click prediction: A view from the trenches. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD).
  19. [23]Mintz Mike, Bills Steven, Snow Rion, and Jurafsky Dan. 2009 Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2 Association for Computational Linguistics, 1003– 1011.
  20. [24]Mnih Volodymyr and Hinton Geoffrey E. 2012 Learning to label aerial images from noisy data. In Proceedings of the 29th International conference on machine learning (ICML-12) 567– 574.
  21. [26]Pan SJ and Yang Q. 2010 A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering 22, 10 (2010), 1345– 1359.
  22. [27]Pan Sinno Jialin, Yang Qiang, 2010 A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22, 10 (2010), 1345– 1359.
  23. [28]Rahm Erhard and Do Hong Hai. 2000 Data cleaning: Problems and current approaches. IEEE Data Eng. Bull. 23, 4 (2000), 3– 13.
  24. [29]Ratner Alexander, Bach Stephen H, Ehrenberg Henry, Fries Jason, Wu Sen, and Ré Christopher. 2017 Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment 11, 3 (2017), 269– 282. 29770249 PMC5951191
  25. [30]Ratner Alexander J, De Sa Christopher M, Wu Sen, Selsam Daniel, and Ré Christopher. 2016 Data programming: Creating large training sets, quickly. In Advances in neural information processing systems. 3567– 3575. 29872252 PMC5985238
  26. [31]Ratner Alexander J, Hancock Braden, Dunnmon Jared, Sala Frederic, Pandey Shreyash, and Ré Christopher. 2019 Training Complex Models with Multi-Task Weak Supervision. In AAAI. PMC6765366 31565535
  27. [32]Rekatsinas Theodoros, Chu Xu, Ilyas Ihab F., and Ré Christopher. 2017 HoloClean: Holistic Data Repairs with Probabilistic Inference. PVLDB 10, 11 (2017), 1190– 1201.
  28. [33]Rekatsinas Theodoros, Joglekar Manas, Garcia-Molina Hector, Parameswaran Aditya, and Ré Christopher. 2017 SLiMFast: Guaranteed Results for Data Fusion and Source Reliability. In ACM SIGMOD International Conference on Management of Data (SIGMOD).
  29. [34]Scudder HJ. 1965 Probability of Error of some Adaptive Pattern-Recognition Machines. IEEE Transactions on Infomation Theory 11 (1965), 363– 371.
  30. [37]Sun Chen, Shrivastava Abhinav, Singh Saurabh, and Gupta Abhinav. 2017 Revisiting Unreasonable Effectiveness of Data in Deep Learning Era. CoRR abs/1707.02968 (2017). arXiv:1707.02968 http://arxiv.org/abs/1707.02968 · arXiv
  31. [38]Xian Yongqin, Lampert Christoph H, Schiele Bernt, and Akata Zeynep. 2018 Zero-shot learning - A comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence (2018). 30028691
  32. [39]Zhai Andrew, Kislyuk Dmitry, Jing Yushi, Feng Michael, Tzeng Eric, Donahue Jeff, Du Yue Li, and Darrell Trevor. 2017 Visual discovery at Pinterest. In International Conference on the World Wide Web (WWW).
  33. [40]Zhang Ce, Shin Jaeho, Ré Christopher, Cafarella Michael, and Niu Feng. 2016 Extracting databases from dark data with deepdive. In Proceedings of the 2016 International Conference on Management of Data ACM, 847– 859. PMC5350112 28316365
  34. [41]Zhao Bo, Rubinstein Benjamin IP, Gemmell Jim, and Han Jiawei. 2012 A Bayesian approach to discovering truth from conflicting sources for data integration. PVLDB 5, 6 (2012), 550– 561.

Cited by

Sorted by citations

Authors
Bach, S. H., Rodriguez, D., Liu, Y., Luo, C., Shao, H., Xia, C., Sen, S., Ratner, A., Hancock, B., Alborzi, H., Kuchhal, R., Ré, C., & Malkin, R.
Affiliations
  • Brown University, Providence, RI, USA
  • Google, Mountain View, CA, USA
  • Stanford University, Stanford, CA, USA
Venue
Proceedings of the 2019 International Conference on Management of Data
Publisher
ACM
Published
2019-06-25
Pages
362-375
DOI
10.1145/3299869.3314036
PMC
PMC6879379
Cited by
54
References
0
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review