The DOI matches its Crossref record; 157 of the 157 references it cites were found in a registry or an index.
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective
Open accessAbstract
Data collection is a major bottleneck in machine learning and an active research topic in multiple communities. There are largely two reasons data collection has recently become a critical issue. First, as machine learning is becoming more widely-used, we are seeing new applications that do not necessarily have enough labeled data. Second, unlike traditional machine learning, deep learning techniques automatically generate features, which saves feature engineering costs, but in return may require larger amounts of labeled data. Interestingly, recent research in data collection comes not only from the machine learning, natural language, and computer vision communities, but also from the data management community due to the importance of handling large amounts of data. In this survey, we perform a comprehensive study of data collection from a data management point of view. Data collection largely consists of data acquisition, data labeling, and improvement of existing data or models. We provide a research landscape of these operations, provide guidelines on which technique to use when, and identify interesting research challenges. The integration of machine learning and data management for data collection is part of a larger trend of Big data and Artificial Intelligence (AI) integration and opens many opportunities for new research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 25,387
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
Weiss, K., Khoshgoftaar, T. M., & Wang, D. (2016) · Journal of Big Data · doi:10.1186/s40537-016-0043-6 · Cited by 6,260
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
Combining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Distributed Representations of Sentences and Documents
Le, Q. V., & Mikolov, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.4053 · Cited by 4,934
Freebase: a collaboratively created graph database for structuring human knowledge
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., & Taylor, J. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376746 · Cited by 4,852
Semi-Supervised Learning Literature Survey
Zhu, X. (2005) · Minds at UW (University of Wisconsin) · Cited by 3,844
Semi-supervised learning using Gaussian fields and harmonic functions
Zhu, X., Zoubin Ghahramani, A., & Lafferty, J. (2003) · Cited by 3,509
A Survey on Deep Transfer Learning
Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., & Liu, C. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01424-7_27 · Cited by 3,165
One-shot learning of object categories
Fei-Fei, L., Fergus, R., & Perona, P. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.79 · Cited by 3,029
Distant supervision for relation extraction without labeled data
Mintz, M., Bills, S., Snow, R., & Jurafsky, D. (2009) · Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - ACL-IJCNLP '09 · doi:10.3115/1690219.1690287 · Cited by 2,752
Yago: a core of semantic knowledge
Suchanek, F. M., Kasneci, G., & Weikum, G. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242667 · Cited by 2,512
Unsupervised word sense disambiguation rivaling supervised methods
Yarowsky, D. (1995) · Proceedings of the 33rd annual meeting on Association for Computational Linguistics - · doi:10.3115/981658.981684 · Cited by 2,385
Alcalá‐Fdez, J., Fernández, A., Luengo, J., Derrac, J., García, S., Sánchez, L., & Herrera, F. (2011) · Consultation of the Doctoral Thesis Database (TESEO) (Ministerio de Educación, Cultura y Deporte) · Cited by 2,328
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 3×
“The difference between the models can identified using model interpretation tools like LIME [120] and Anchors [121] and be used by the user to further debug and improve the labeling functions.”
BackgroundMethodToward an architecture for never-ending language learning
Carlson, A., Betteridge, J., Kisiel, B., Settles, B., Hruschka, E., & Mitchell, T. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7519 · Cited by 2,000
A brief introduction to weakly supervised learning
Zhou, Z.-H. (2018) · National Science Review · doi:10.1093/nsr/nwx106 · Cited by 1,899
Classification in the Presence of Label Noise: A Survey
Frenay, B., & Verleysen, M. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2292894 · Cited by 1,777
A sequential algorithm for training text classifiers
Lewis, D., & Gale, W. A. (1994) · arXiv (Cornell University) · doi:10.5555/188490.188495 · Cited by 1,731
Seung, H. S., Opper, M., & Sompolinsky, H. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130417 · Cited by 1,651
Synthetic Data for Text Localisation in Natural Images
Gupta, A., Vedaldi, A., & Zisserman, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.254 · Cited by 1,563
(2015) · Springer eBooks · doi:10.1007/978-1-4899-7637-6 · Cited by 1,550
Knowledge vault: a web-scale approach to probabilistic knowledge fusion
Dong, X., Gabrilovich, E., Heitz, G., Horn, W., Lao, N., Murphy, K., Strohmann, T., Sun, S., & Zhang, W. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623623 · Cited by 1,533
NIPS 2016 Tutorial: Generative Adversarial Networks
Goodfellow, I. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.00160 · Cited by 1,229
A Sequential Algorithm for Training Text Classifiers
Lewis, D. D., & Gale, W. A. (1994) · SIGIR ’94 · doi:10.1007/978-1-4471-2099-5_1 · Cited by 1,208
Tri-training: exploiting unlabeled data using three classifiers
Zhou, Z.-H., & Li, M. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.186 · Cited by 1,171
Get another label? improving data quality and data mining using multiple, noisy labelers
Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401965 · Cited by 1,165
AutoAugment: Learning Augmentation Policies from Data
Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., & Le, Q. V. (2018) · arXiv · doi:10.48550/arxiv.1805.09501 · Cited by 1,106
An analysis of active learning strategies for sequence labeling tasks
Settles, B., & Craven, M. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613855 · Cited by 972
RoadRunner: Towards Automatic Data Extraction from Large Web Sites
Valter Crescenzi, A., Mecca, G., & Paolo Merialdo, A. (2001) · Cited by 928
Learning from massive noisy labeled data for image classification
Xiao, T., Xia, T., Yang, Y., Huang, C., & Wang, X. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298885 · Cited by 907
Toward Optimal Active Learning through Sampling Estimation of Error Reduction
Roy, N., & McCallum, A. (2001) · Cited by 833
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3157794.3157797 · Cited by 777
Web-scale information extraction in knowitall: (preliminary results)
Etzioni, O., Cafarella, M., Downey, D., Kok, S., Popescu, A.-M., Shaked, T., Soderland, S., Weld, D. S., & Yates, A. (2004) · Proceedings of the 13th international conference on World Wide Web · doi:10.1145/988672.988687 · Cited by 747
Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition
Jaderberg, M., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.2227 · Cited by 726
Open Language Learning for Information Extraction
Schmitz, M., Soderland, S. G., Bart, R., & Etzioni, O. (2012) · Cited by 694
Employing EM and Pool-Based Active Learning for Text Classification
McCallum, A., & Nigam, K. (1998) · Cited by 689
WebTables: exploring the power of tables on the web
Cafarella, M. J., Halevy, A., Wang, D. Z., Wu, E., & Zhang, Y. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1453856.1453916 · Cited by 629
CrowdDB: answering queries with crowdsourcing
Franklin, M. J., Kossmann, D., Kraska, T., Ramesh, S., & Xin, R. (2011) · Proceedings of the 2011 ACM SIGMOD International Conference on Management of data · doi:10.1145/1989323.1989331 · Cited by 623
Patki, N., Wedge, R., & Veeramachaneni, K. (2016) · 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2016.49 · Cited by 616
Adversarial Example Generation with Syntactically Controlled Paraphrase Networks
Iyyer, M., Wieting, J., Gimpel, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1170 · Cited by 615
Transfer Learning in Natural Language Processing
Ruder, S., Peters, M. E., Swayamdipta, S., & Wolf, T. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-5004 · Cited by 587
Wrangler: interactive visual specification of data transformation scripts
Kandel, S., Paepcke, A., Hellerstein, J., & Heer, J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979444 · Cited by 581
Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study
Triguero, I., García, S., & Herrera, F. (2015) · Knowledge and Information Systems · doi:10.1007/s10115-013-0706-y · Cited by 573
Data synthesis based on generative adversarial networks
Park, N., Mohammadi, M., Gorde, K., Jajodia, S., Park, H., & Kim, Y. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3231751.3231757 · Cited by 531
Principles of Data Integration
(2012) · Elsevier eBooks · doi:10.1016/c2011-0-06130-6 · Cited by 518
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2020) · The VLDB Journal · doi:10.1007/s00778-019-00552-1 · Cited by 510
CrowdER: crowdsourcing entity resolution
Wang, J., Kraska, T., Franklin, M. J., & Feng, J. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2350229.2350263 · Cited by 507
Settles, B. (2012) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01560-1 · Cited by 502
Potter's Wheel: An Interactive Data Cleaning System
Raman, V., & Hellerstein, J. L. (2001) · Cited by 492
Multiple-Instance Active Learning
Settles, B., Craven, M. W., & Ray, S. (2007) · neural information processing systems · Cited by 489
HoloClean: holistic data repairs with probabilistic inference
Rekatsinas, T., Chu, X., Ilyas, I. F., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3137628.3137631 · Cited by 480
Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions
Zhu, X., Lafferty, J., & Ghahramani, Z. (2003) · Cited by 471
Iterative Learning for Reliable Crowdsourcing Systems
Karger, D. R., Oh, S., & Shah, D. (2011) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 468
Semantically Equivalent Adversarial Rules for Debugging NLP models
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1079 · Cited by 432
Web data extraction, applications and techniques: A survey
Ferrara, E., De Meo, P., Fiumara, G., & Baumgartner, R. (2014) · Knowledge-Based Systems · doi:10.1016/j.knosys.2014.07.007 · Cited by 396
Quality Control in Crowdsourcing Systems: Issues and Directions
Allahbakhsh, M., Benatallah, B., Ignjatovic, A., Motahari-Nezhad, H. R., Bertino, E., & Dustdar, S. (2013) · IEEE Internet Computing · doi:10.1109/mic.2013.20 · Cited by 385
Semi-supervised regression with co-training
Zhou, Z., & Li, M. H. (2005) · International Joint Conference on Artificial Intelligence · Cited by 365
A survey on heterogeneous transfer learning
Day, O., & Khoshgoftaar, T. M. (2017) · Journal of Big Data · doi:10.1186/s40537-017-0089-0 · Cited by 360
Webly Supervised Learning of Convolutional Networks
Chen, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.168 · Cited by 351
Data Programming: Creating Large Training Sets, Quickly
Ratner, A., De, C., Wu, S., Selsam, D., & Ré, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07723 · Cited by 350
TFX: A TensorFlow-Based Production-Scale Machine Learning Platform
Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., et al. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098021 · Cited by 350
Self-citationQuery Learning Strategies Using Boosting and Bagging
Abe, N., & Mamitsuka, H. (1998) · International Conference on Machine Learning · Cited by 348
Generating Multi-label Discrete Patient Records using Generative Adversarial Networks
Choi, E., Biswal, S., Malin, B., Duke, J. D., Stewart, W. F., & Sun, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.06490 · Cited by 341
Learning Deep Object Detectors from 3D Models
Peng, X., Sun, B., Ali, K., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.151 · Cited by 338
Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., & Allahbakhsh, M. (2019) · ACM Computing Surveys · doi:10.1145/3148148 · Cited by 313
TurKit: human computation algorithms on mechanical turk
Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866040 · Cited by 307
Crowdsourced Data Management: A Survey
Li, G., Wang, J., Zheng, Y., & Franklin, M. J. (2016) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2016.2535242 · Cited by 283
ActiveClean: interactive data cleaning for statistical modeling
Krishnan, S., Wang, J., Wu, E., Franklin, M. J., & Goldberg, K. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/2994509.2994514 · Cited by 264
Data integration for the relational web
Cafarella, M. J., Halevy, A., & Khoussainova, N. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687627.1687750 · Cited by 243
InfoGather: entity augmentation and attribute discovery by holistic matching with web tables
Yakout, M., Ganjam, K., Chakrabarti, K., & Chaudhuri, S. (2012) · Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2213836.2213848 · Cited by 226
Profiler: integrated statistical analysis and visualization for data quality assessment
Kandel, S., Parikh, R., Paepcke, A., Hellerstein, J. M., & Heer, J. (2012) · Proceedings of the International Working Conference on Advanced Visual Interfaces · doi:10.1145/2254556.2254659 · Cited by 220
Corleone: hands-off crowdsourcing for entity matching
Gokhale, C., Das, S., Doan, A., Naughton, J. F., Rampalli, N., Shavlik, J., & Zhu, X. (2014) · Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2588555.2588576 · Cited by 219
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets
Chang, J. C., Amershi, S., & Kamar, E. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3026044 · Cited by 217
Data Lifecycle Challenges in Production Machine Learning: A Survey
Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2018) · ACM SIGMOD Record · doi:10.1145/3299887.3299891 · Cited by 214
Self-citationData Management Challenges in Production Machine Learning
Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2017) · Proceedings of the 2017 ACM International Conference on Management of Data · doi:10.1145/3035918.3054782 · Cited by 208
Self-citationCrowdScreen: algorithms for filtering data with humans
Parameswaran, A. G., Garcia-Molina, H., Park, H., Polyzotis, N., Ramesh, A., & Widom, J. (2012) · Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2213836.2213878 · Cited by 207
Paraphrasing Revisited with Neural Machine Translation
Mallinson, J., Sennrich, R., & Lapata, M. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1083 · Cited by 206
Synthesizing Tabular Data using Generative Adversarial Networks
Xu, L., & Veeramachaneni, K. K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.11264 · Cited by 206
Zhou, Y., & Goldman, S. (2005) · 16th IEEE International Conference on Tools with Artificial Intelligence · doi:10.1109/ictai.2004.48 · Cited by 203
Crowdsourced Databases: Query Processing with People
Marcus, A., Wu, E., Karger, D. R., Madden, S. R., & Miller, R. C. (2011) · Cited by 191
Data Curation at Scale: The Data Tamer System
Stonebraker, M. R., Beskales, G., Pagan, A., Bruckner, D., Cherniack, M., Xu, S., Analytics, V., Ilyas, I. F., & Zdonik, S. (2013) · Cited by 191
Learning the Structure of Generative Models without Labeled Data
Bach, S. H., He, B. D., Ratner, A., & Ré, C. (2017) · PubMed · doi:10.48550/arxiv.1703.00854 · Cited by 190
Learning to Compose Domain-Specific Transformations for Data Augmentation
Ratner, A., Ehrenberg, H. R., Hussain, Z., Dunnmon, J., & Ré, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.01643 · Cited by 180
Goods: Organizing Google's Datasets
Halevy, A., Korn, F., Noy, N. F., Olston, C., Polyzotis, N., Roy, S., & Whang, S. E. (2016) · Proceedings of the 2016 International Conference on Management of Data · doi:10.1145/2882903.2903730 · Cited by 171
Self-citationLearning-Based Video Motion Magnification
Oh, T.-H., Jaroensri, R., Kim, C., Elgharib, M., Durand, F., Freeman, W. T., & Matusik, W. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_39 · Cited by 167
Scaling up crowd-sourcing to very large datasets: a case for active learning
Mozafari, B., Sarkar, P., Franklin, M., Jordan, M., & Madden, S. (2014) · Proceedings of the VLDB Endowment · doi:10.14778/2735471.2735474 · Cited by 165
Google fusion tables: web-centered data management and collaboration
Gonzalez, H., Halevy, A. Y., Jensen, C. S., Langen, A., Madhavan, J., Shapley, R., Shen, W., & Goldberg-Kidon, J. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807286 · Cited by 162
Aurum: A Data Discovery System
Castro Fernandez, R., Abedjan, Z., Koko, F., Yuan, G., Madden, S., & Stonebraker, M. (2018) · 2018 IEEE 34th International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2018.00094 · Cited by 162
Active Learning for Regression Based on Query by Committee
Burbidge, R., Rowland, J. J., & King, R. D. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-77226-2_22 · Cited by 160
Efficient co-regularised least squares regression
Brefeld, U., Gärtner, T., Scheffer, T., et al. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143862 · Cited by 154
Spreadsheet table transformations from examples
Harris, W. R., & Gulwani, S. (2011) · ACM SIGPLAN Notices · doi:10.1145/1993316.1993536 · Cited by 153
Challenges in Data Crowdsourcing
García-Molina, H., Joglekar, M., Marcus, A., Parameswaran, A., & Verroios, V. (2016) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2016.2518669 · Cited by 145
DataHub: Collaborative Data Science & Dataset Version Management at ScalePreprint
Bhardwaj, A., Bhattacherjee, S., Chavan, A., Deshpande, A., Elmore, A. J., Madden, S. R., & Parameswaran, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.0798 · Cited by 138
The jabberwocky programming environment for structured social computing
Ahmad, S., Battle, A. J., Malkani, Z., & Kamvar, S. (2011) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2047196.2047203 · Cited by 132
Harvesting relational tables from lists on the web
Elmeleegy, H., Madhavan, J., & Halevy, A. Y. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687627.1687749 · Cited by 128
Adaptive Sensitive Reweighting to Mitigate Bias in Fairness-aware Classification
Krasanakis, E., Spyromitros-Xioufis, E., Papadopoulos, S., & Kompatsiaris, Y. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186133 · Cited by 127
The GAN Landscape: Losses, Architectures, Regularization, and NormalizationPreprint
Kurach, K., Lučić, M., Zhai, X., Michalski, M., & Gelly, S. (2018) · arXiv (Cornell University) · Cited by 125
Data Wrangling: The Challenging Yourney from the Wild to the Lake.
Terrizzano, I. G., Schwarz, P., Roth, M. T., & Colino, J. E. (2015) · Conference on Innovative Data Systems Research · Cited by 122
AutoMan: a platform for integrating human-based and digital computation
Barowy, D. W., Curtsinger, C., Berger, E. D., & McGregor, A. (2012) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2384616.2384663 · Cited by 118
Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale
Bach, S. H., Rodriguez, D., Liu, Y., Luo, C., Shao, H., Xia, C., Sen, S., Ratner, A., Hancock, B., Alborzi, H., et al. (2019) · Proceedings of the 2019 International Conference on Management of Data · doi:10.1145/3299869.3314036 · Cited by 118
Crowdsourced enumeration queries
Trushkowsky, B., Kraska, T., Franklin, M. J. M., & Sarkar, P. (2013) · Proceedings - International Conference on Data Engineering · doi:10.1109/icde.2013.6544865 · Cited by 117
Structured labeling for facilitating concept evolution in machine learning
Kulesza, T., Amershi, S., Caruana, R., Fisher, D., & Charles, D. X. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557238 · Cited by 116
Data Integration: The Current Status and the Way Forward.
Stonebraker, M. R., & Ilyas, I. F. (2018) · IEEE Data(base) Engineering Bulletin · Cited by 110
To Join or Not to Join? Thinking Twice about Joins before Feature Selection
Kumar, A., Naughton, J. F., Patel, J. M., & Zhu, X. (2016) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2882903.2882952 · Cited by 103
Seeping Semantics: Linking Datasets Using Word Embeddings for Data Discovery
Fernandez, R. C., Mansour, E. A. H., Qahtan, A. A., Elmagarmid, A. K., Ilyas, I. F., Madden, S. R., Ouzzani, M., Stonebraker, M. R., & Tang, N. (2018) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2018.00093 · Cited by 102
Spreadsheet table transformations from examples
Harris, W. R., & Gulwani, S. (2011) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/1993498.1993536 · Cited by 101
Semi-supervised active learning for sequence labeling
Tomanek, K., & Hahn, U. (2009) · doi:10.3115/1690219.1690291 · Cited by 94
ReNoun: Fact Extraction for Nominal Attributes
Yahya, M., Whang, S. E., Gupta, R., & Halevy, A. Y. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1038 · Cited by 93
Self-citationMarcus, A., Karger, D. R., Madden, S. R., Miller, R. C., & Oh, S. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2535568.2448944 · Cited by 92
BoostClean: Automated Error Detection and Repair for Machine LearningPreprint
Krishnan, S., Franklin, M. J. M., Goldberg, K. Y., & Wu, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.01299 · Cited by 87
Asking the Right Questions in Crowd Data Sourcing
Boim, R., Greenshpan, O., Milo, T., Novgorodov, S., Polyzotis, N., & Tan, W.-C. (2012) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2012.122 · Cited by 84
Crowdsourced Data Management: Industry and Academic Perspectives
Marcus, A., & Parameswaran, A. (2015) · Foundations and Trends in Databases · doi:10.1561/1900000044 · Cited by 83
Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision
Kim, J. O., Sterman, S., Cohen, A. A. B., & Bernstein, M. S. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998196 · Cited by 83
Towards linear algebra over normalized data
Chen, L., Kumar, A., Naughton, J. F., & Patel, J. M. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3137628.3137633 · Cited by 81
Ratner, A., Bach, S. H., Ehrenberg, H. R., & Ré, C. (2017) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3035918.3056442 · Cited by 77
Gupta, R., Halevy, A. Y., Wang, X., Whang, S. E., & Wu, F. Q. (2014) · Proceedings of the VLDB Endowment · doi:10.14778/2732286.2732288 · Cited by 70
Self-citationExploiting Unlabeled Data in Content-Based Image Retrieval
Zhou, Z., Chen, K., & Jiang, Y. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30115-8_48 · Cited by 67
Data Cleaning for Accurate, Fair, and Robust Models: A Big Data - AI Integration Approach
Tae, K. H., Roh, Y., Oh, Y. H., Kim, H., & Whang, S. E. (2019) · Proceedings of the 3rd International Workshop on Data Management for End-to-End Machine Learning · doi:10.1145/3329486.3329493 · Cited by 62 · cited 2×
“More recently, MLCLEAN [166] has been proposed to integrate three data operations: traditional data cleaning, model unfairness mitigation where the goal is to remove data bias that causes model fairness, and data sanitization where the goal is to remove data poisoning.”
BackgroundSelf-citationDeepDive: A Data Management System for Automatic Knowledge Base Construction
Zhang, C., Konda, P., & Mallory, E. K. (2015) · Cited by 58
Park, H., & Widom, J. (2014) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2588555.2610503 · Cited by 57
Collaborative data analytics with DataHub
Bhardwaj, A., Deshpande, A., Elmore, A. J., Karger, D. R., Madden, S. R., Parameswaran, A., Subramanyam, H., Wu, E., & Zhang, R. V. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2824032.2824100 · Cited by 56
Mechanical Novel: Crowdsourcing Complex Work Through Reflection and Revision
Kim, J. O., Sterman, S., Cohen, A. A. B., & Bernstein, M. S. (2017) · Understanding innovation · doi:10.1007/978-3-319-60967-6_5 · Cited by 56
Managing Google’s data lake: an overview of the Goods system
Olston, C., Korn, F. R., Noy, N. F., Polyzotis, N., Whang, S. E., & Roy, S. (2016) · IEEE Data(base) Engineering Bulletin · Cited by 55
Self-citationNavigating the Data Lake with DATAMARAN: Automatically Extracting Structure from Log Datasets
Gao, Y., Huang, S., & Parameswaran, A. (2018) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3183713.3183746 · Cited by 55
Resolvable vs. Irresolvable Disagreement
Schaekermann, M., Goh, J., Larson, K. S., & Law, E. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274423 · Cited by 54
Snorkel MeTaL: Weak Supervision for Multi-Task Learning
Ratner, A., Hancock, B., Dunnmon, J., Goldman, R., & Ré, C. (2018) · Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning · doi:10.1145/3209889.3209898 · Cited by 43
Cleaning crowdsourced labels using oracles for statistical classification
Dolatshah, M., Teoh, M., Wang, J., & Pei, J. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3297753.3297758 · Cited by 38
Reverb: Recommending code-related web pages
Sawadsky, N., Murphy, G. C., & Jiresal, R. (2013) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2013.6606627 · Cited by 36
Communicating Context to the Crowd for Complex Writing Tasks
Salehi, N., Teevan, J. B., Iqbal, S. T., & Kamar, E. S. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998332 · Cited by 34
Enabling and Optimizing Non-linear Feature Interactions in Factorized Linear Algebra
Li, S., Chen, L., & Kumar, A. (2019) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3299869.3319878 · Cited by 34
Automatic Image Dataset Construction from Click-through Logs Using Deep Neural Network
Bai, Y., Yang, K., Yu, W., Xu, C. X., Ma, W., & Zhao, T. (2015) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2733373.2806243 · Cited by 30
A Demo of the Data Civilizer System
Fernandez, R. C., Deng, D., Mansour, E. A. H., Qahtan, A. A., Tao, W., Abedjan, Z., Elmagarmid, A. K., Ilyas, I. F., Madden, S. R., Ouzzani, M., et al. (2017) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3035918.3058740 · Cited by 30
Xia, Y., Cao, X., Wen, F., & Sun, J. Z. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10593-2_26 · Cited by 24
Are key-foreign key joins safe to avoid when learning high-capacity classifiers?
Shah, V., Kumar, A., & Zhu, X. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3157794.3157804 · Cited by 22
The Role of Massively Multi-Task and Weak Supervision in Software 2.0.
Ratner, A., Hancock, B. J., & Ré, C. (2019) · Conference on Innovative Data Systems Research · Cited by 22
Osprey: Weak Supervision of Imbalanced Extraction Problems without Code
Bringer, E., Israeli, A., Shoham, Y., Ratner, A., & Ré, C. (2019) · doi:10.1145/3329486.3329492 · Cited by 21
Crowdsourcing large scale wrapper inference
Crescenzi, V., Merialdo, P., & Qiu, D. (2014) · Distributed and Parallel Databases · doi:10.1007/s10619-014-7163-9 · Cited by 19
Data programming with DDLite: putting humans in a different part of the loop
Ehrenberg, H. R., Shin, J., Ratner, A., Fries, J., & Ré, C. (2016) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/2939502.2939515 · Cited by 19
Scaling Graph-based Semi Supervised Learning to Large Number of Labels Using Count-Min SketchPreprint
Talukdar, P. P., & Cohen, W. W. (2013) · arXiv (Cornell University) · doi:10.1184/r1/6476321 · Cited by 18
Foundations of Crowd Data Sourcing
Amsterdamer, Y., & Milo, T. (2015) · ACM SIGMOD Record · doi:10.1145/2737817.2737819 · Cited by 14
Deep representation of industrial components using simulated images
Kim, S., Choe, G., Ahn, B., et al. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989232 · Cited by 7
Research works printed but not linked: 25
- 10.1145/2213836.2213912 · doi:10.1145/2213836.2213912
- 10.14778/1938545.1938547 · doi:10.14778/1938545.1938547
- 10.1007/978-1-4614-8265-9_1154 · doi:10.1007/978-1-4614-8265-9_1154
- 10.1145/2723372.2723725 · doi:10.1145/2723372.2723725
- 10.1007/s00778-011-0223-0 · doi:10.1007/s00778-011-0223-0
- chakrabarti, 2016, Data services leveraging bing's data assets, IEEE Data Eng Bull, 39, 15
- deng, 2017, The data civilizer system, Proc Biennial Conf Innovative Data Syst Res
- ravi, 2016, Large scale distributed semi-supervised learning using streaming approximation, Proc Int Conf Artif Intell Statistics, 519
- 10.14778/3229863.3240492 · doi:10.14778/3229863.3240492
- 10.1007/978-3-319-12637-1_27 · doi:10.1007/978-3-319-12637-1_27
- 10.1007/s10619-014-7163-9 · DOI
- park, 2014, Crowdfill: Collecting structured data from the crowd, Proc ACM SIGMOD Int Conf Manage Data, 577
- 10.14778/2367502.2367555 · doi:10.14778/2367502.2367555
- 10.1145/2882903.2882952 · DOI
- ehrenberg, 2016, Data programming with ddlite: Putting humans in a different part of the loop, Proc Workshop Human-In-the-Loop Data Analytics, 10.1145/2939502.2939515 · DOI
- yu, 2010, Keyword search in relational databases: A survey, IEEE Data Eng Bulletin, 33, 67
- 10.14778/1687553.1687622 · doi:10.14778/1687553.1687622
- mahdisoltani, 2015, YAGO3: A knowledge base from multilingual wikipedias, Proc Biennial Conf Innovative Data Syst Res
- mitchell, 2015, Never-ending learning, Proc Association Advancement Artif Intell, 2302
- goodfellow, 2016, Deep Learning
- sindhwani, 2005, A co-regularized approach to semi-supervised learning with multiple views, Proc ICML Workshop Learn Multiple Views
- 10.14778/2824032.2824035 · doi:10.14778/2824032.2824035
- halevy, 2013, Data publishing and sharing using fusion tables, Proc Biennial Conf Innovative Data Syst Res
- 10.1145/1807128.1807158 · doi:10.1145/1807128.1807158
- dekel, 2009, Vox populi: Collecting high-quality labels from a crowd, Proc 22nd Annu Conf Learn Theory
Cited by
AutoML: A survey of the state-of-the-art
He, X., Zhao, K., & Chu, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106622 · Cited by 1,691
Baduge, S. K., Thilakarathna, S., Perera, J. S., Arashpour, M., Sharafi, P., Teodosio, B., Shringi, A., & Mendis, P. (2022) · Automation in Construction · doi:10.1016/j.autcon.2022.104440 · Cited by 1,007
“The majority of the time for an end-to-end ML/DL model deployment is spent on the dataset which involves collecting, cleaning, analyzing, visualizing and feature engineering [2].”
BackgroundA Unifying Review of Deep and Shallow Anomaly Detection
Ruff, L., Kauffmann, J. R., Vandermeulen, R. A., Montavon, G., Samek, W., Kloft, M., Dietterich, T. G., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3052449 · Cited by 914
A review of machine learning applications in wildfire science and management
Jain, P., Coogan, S. C. P., Subramanian, S. G., Crowley, M., Taylor, S., & Flannigan, M. D. (2020) · Environmental Reviews · doi:10.1139/er-2020-0019 · Cited by 765
Artificial Intelligence in Education: AIEd for Personalised Learning Pathways
Tapalova, O., & Zhiyenbayeva, N. (2022) · Electronic Journal of e-Learning · doi:10.34190/ejel.20.5.2597 · Cited by 633
Metadata
- Authors
- Roh, Y., Heo, G., & Whang, S. E.
- Venue
- IEEE Transactions on Knowledge and Data Engineering
- Published
- 2021-04-01
- Volume
- 33
- Issue
- 4
- Pages
- 1328-1347
- ISSN
- 1041-4347 (ISSN-L)
- 1558-2191
- 2326-3865
- OpenAlex
- W2979417040
- Cited by
- 968
- References
- 157
- FWCI
- 122.09
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- National Research Foundation of Korea (NRF-2018R1A5A1059921)
- SK Telecom
- Google AI Focused Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.