The DOI matches its Crossref record; 181 of the 181 references it cites were found in a registry or an index.
Data-centric Artificial Intelligence: A Survey
Open accessAbstract
Artificial Intelligence (AI) is making a profound impact in almost every domain. A vital enabler of its great success is the availability of abundant and high-quality data for building machine learning models. Recently, the role of data in AI has been significantly magnified, giving rise to the emerging concept of data-centric AI . The attention of researchers and practitioners has gradually shifted from advancing model design to enhancing the quality and quantity of the data. In this survey, we discuss the necessity of data-centric AI, followed by a holistic view of three general data-centric goals (training data development, inference data development, and data maintenance) and the representative methods. We also organize the existing literature from automation and collaboration perspectives, discuss the challenges, and tabulate the benchmarks for various tasks. We believe this is the first comprehensive survey that provides a global view of a spectrum of tasks across various stages of the data lifecycle. We hope it can help the readers efficiently grasp a broad picture of this field, and equip them with the techniques and further research ideas to systematically engineer data for building AI systems. A companion list of data-centric AI resources will be regularly updated on https://github.com/daochenzha/data-centric-AI .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Generative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
Abdi, H., & Williams, L. J. (2010) · WIREs Computational Statistics · doi:10.1002/wics.101 · Cited by 11,104
Graph neural networks: A review of methods and applications
Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., & Sun, M. (2020) · AI Open · doi:10.1016/j.aiopen.2021.01.001 · Cited by 6,880
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
Adversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317
Freebase: a collaboratively created graph database for structuring human knowledge
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., & Taylor, J. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376746 · Cited by 4,852
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 4×
“On the other hand, tuning the inference data can help obtain the desirable outputs, such as tuning prompts for language models [146].”
InfluentialSelf-citationLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 4×
“On the other hand, tuning the inference data can help obtain the desirable outputs, such as tuning prompts for language models [146].”
InfluentialSelf-citationPractical Black-Box Attacks against Machine Learning
Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2017) · Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security · doi:10.1145/3052973.3053009 · Cited by 3,495
Deep Learning Based Recommender System: A Survey and New Perspectives
Zhang, S., Yao, L., Sun, A., & Tay, Y. (2020) · ACM Computing Surveys · doi:10.1145/3285029 · Cited by 3,485
Deep learning for computer vision: A brief review
Voulodimos, A., Doulamis, N., Doulamis, A., & Protopapadakis, E. (2018) · Computational Intelligence and Neuroscience · doi:10.1155/2018/7068349 · Cited by 3,403
Deep learning for healthcare: review, opportunities and challenges
Miotto, R., Wang, F., Wang, S., Jiang, X., & Dudley, J. T. (2018) · Briefings in Bioinformatics · doi:10.1093/bib/bbx044 · Cited by 3,076
Adapting visual category models to new domains
Saenko, K., Kulis, B., Fritz, M., & Darrell, T. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_16 · Cited by 2,847
Deep Learning for Anomaly Detection: A Review
Pang, G., Shen, C., Cao, L., & Hengel, A. V. D. (2022) · ACM Computing Surveys · doi:10.1145/3439950 · Cited by 2,837
Apache Spark: A unified engine for big data processing
Zaharia, M., Xin, R. S., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M. J., et al. (2016) · Communications of the ACM · doi:10.1145/2934664 · Cited by 2,757
Distant supervision for relation extraction without labeled data
Mintz, M., Bills, S., Snow, R., & Jurafsky, D. (2009) · Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - ACL-IJCNLP '09 · doi:10.3115/1690219.1690287 · Cited by 2,752
Data integration: a theoretical perspective
Lenzerini, M. (2002) · Proceedings of the twenty-first ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems · doi:10.1145/543613.543644 · Cited by 2,494
Feature Selection: A Data Perspective
Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R. P., Tang, J., & Liu, H. (2018) · ACM Computing Surveys · doi:10.1145/3136625 · Cited by 2,442
An Overview of Overfitting and its Solutions
Ying, X. (2019) · Journal of Physics: Conference Series · doi:10.1088/1742-6596/1168/2/022022 · Cited by 2,352
Time series classification from scratch with deep neural networks: A strong baseline
Wang, Z., Yan, W., & Oates, T. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966039 · Cited by 2,299
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Robust physical-world attacks on deep learning visual classification
Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00175 · Cited by 2,173
Evasion attacks against machine learning at test time
Biggio, B., Corona, I., Maiorca, D., Nelson, B. A., Srndic, N., Laskov, P., Giacinto, G., & Roli, F. (2017) · CINECA IRIS Institutial Research Information System (University of Genoa) · Cited by 1,870
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,740
Pipino, L. L., Lee, Y. W., & Wang, R. Y. (2002) · Communications of the ACM · doi:10.1145/505248.506010 · Cited by 1,724
Keulen, D. S. (2014) · Cited by 1,677
Integrating structured biological data by Kernel Maximum Mean Discrepancy
Borgwardt, K. M., Gretton, A., Rasch, M. J., Kriegel, H.-P., Schölkopf, B., & Smola, A. J. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl242 · Cited by 1,674
Chen, P.-Y., Zhang, H., Sharma, Y., Yi, J., & Hsieh, C.-J. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140448 · Cited by 1,656
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
Hendrycks, D., & Dietterich, T. (2019) · arXiv · doi:10.48550/arxiv.1903.12261 · Cited by 1,335
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.20 · Cited by 1,305
Methodologies for data quality assessment and improvement
Batini, C., Cappiello, C., Francalanci, C., & Maurino, A. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541883 · Cited by 1,288
Facial Expression Analysis under Partial Occlusion: A Survey
Zhang, L., Verma, B., Tjondronegoro, D., & Chandran, V. (2019) · ACM Computing Surveys · doi:10.1145/3158369 · Cited by 1,258
Covariate Shift Adaptation by Importance Weighted Cross Validation
Sugiyama, M., Krauledat, M., & Müller, K. (2007) · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · doi:10.5555/1314498.1390324 · Cited by 1,226
A Survey of Deep Active Learning
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Gupta, B. B., Chen, X., & Wang, X. (2022) · ACM Computing Surveys · doi:10.1145/3472291 · Cited by 1,169
Counterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061
Domain adaptation for medical image analysis: A survey
Guan, H., & Liu, M. (2022) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2021.3117407 · Cited by 1,057
A Survey of Clustering Algorithms for Big Data: Taxonomy and Empirical Analysis
Fahad, A., Alshatri, N., Tari, Z., Alamri, A., Khalil, I., Zomaya, A. Y., Foufou, S., & Bouras, A. (2014) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2014.2330519 · Cited by 1,042
How Can We Know What Language Models Know?
Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00324 · Cited by 1,002
Self-citationChowdhary, K. R. (2020) · Fundamentals of Artificial Intelligence · doi:10.1007/978-81-322-3972-7_19 · Cited by 1,002
A review of medical image data augmentation techniques for deep learning applications
Chlap, P., Min, H., Vandenberg, N., Dowling, J., Holloway, L., & Haworth, A. (2021) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13261 · Cited by 956
Evasion Attacks against Machine Learning at Test Time
Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G., & Roli, F. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40994-3_25 · Cited by 890
Synthetic data augmentation using GAN for improved liver lesion classification
Frid-Adar, M., Klang, E., Amitai, M., Goldberger, J., & Greenspan, H. (2018) · 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) · doi:10.1109/isbi.2018.8363576 · Cited by 860
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3157794.3157797 · Cited by 777
Uniclust databases of clustered and deeply annotated protein sequences and alignments
Mirdita, M., von den Driesch, L., Galiez, C., Martin, M. J., Söding, J., & Steinegger, M. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw1081 · Cited by 759
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.185 · Cited by 747
Effect of Data Scaling Methods on Machine Learning Algorithms and Model Performance
Ahsan, M., Mahmud, M., Saha, P., Gupta, K., & Siddique, Z. (2021) · Technologies · doi:10.3390/technologies9030052 · Cited by 744
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Time Series Data Augmentation for Deep Learning: A Survey
Wen, Q., Sun, L., Yang, F., Song, X., Gao, J., Wang, X., & Xu, H. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/631 · Cited by 727
Self-citationAn empirical survey of data augmentation for time series classification with neural networks
Iwana, B. K., & Uchida, S. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254841 · Cited by 725
A Survey of Data Augmentation Approaches for NLP
Feng, S., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., & Hovy, E. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.84 · Cited by 670
What Makes a Visualization Memorable?
Borkin, M. A., Vo, A. A., Bylinskii, Z., Isola, P., Sunkavalli, S., Oliva, A., & Pfister, H. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.234 · Cited by 666
Hamilton, J. D. (2020) · Princeton University Press eBooks · doi:10.2307/j.ctv14jx6sm · Cited by 665
Universal Adversarial Triggers for Attacking and Analyzing NLP
Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1221 · Cited by 642
Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective
Hazelwood, K., Bird, S., Brooks, D., Chintala, S., Diril, U., Dzhulgakov, D., Fawzy, M., Jia, B., Jia, Y., Kalro, A., et al. (2018) · 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2018.00059 · Cited by 590
Automatic Database Management System Tuning Through Large-scale Machine Learning
Van Aken, D., Pavlo, A., Gordon, G. J., & Zhang, B. (2017) · Proceedings of the 2017 ACM International Conference on Management of Data · doi:10.1145/3035918.3064029 · Cited by 573
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Deep learning for financial applications : A survey
Ozbayoglu, A. M., Gudelek, M. U., & Sezer, O. B. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106384 · Cited by 545
TSFEL: Time Series Feature Extraction Library
Barandas, M., Folgado, D., Fernandes, L., Santos, S., Abreu, M., Bota, P., Liu, H., Schultz, T., & Gamboa, H. (2020) · SoftwareX · doi:10.1016/j.softx.2020.100456 · Cited by 525
Feature selection and analysis on correlated gas sensor data with recursive feature elimination
Yan, K., & Zhang, D. (2015) · Sensors and Actuators B: Chemical · doi:10.1016/j.snb.2015.02.025 · Cited by 516
Cascaded Diffusion Models for High Fidelity Image Generation
Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., & Salimans, T. (2021) · arXiv · doi:10.48550/arxiv.2106.15282 · Cited by 514
Voyager: Exploratory Analysis via Faceted Browsing of Visualization Recommendations
Wongsuphasawat, K., Moritz, D., Anand, A., Mackinlay, J., Howe, B., & Heer, J. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2467191 · Cited by 507
CrowdER: crowdsourcing entity resolution
Wang, J., Kraska, T., Franklin, M. J., & Feng, J. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2350229.2350263 · Cited by 507
Scribbler: Controlling Deep Image Synthesis with Sketch and Color
Sangkloy, P., Lu, J., Fang, C., Yu, F., & Hays, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.723 · Cited by 497
Xanthopoulos, P., Pardalos, P. M., & Trafalis, T. B. (2013) · SpringerBriefs in Optimization · doi:10.1007/978-1-4419-9878-1_4 · Cited by 474
Intrusion detection model using fusion of chi-square feature selection and multi class SVM
Sumaiya Thaseen, I., & Aswani Kumar, C. (2017) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2015.12.004 · Cited by 465
Benchmark and Survey of Automated Machine Learning Frameworks
Zöller, M.-A., & Huber, M. F. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11854 · Cited by 448
Missing data imputation: focusing on single imputation.
Zhang, Z. (2016) · PubMed · doi:10.3978/j.issn.2305-5839.2015.12.38 · Cited by 410
Bank, D., Koenigstein, N., & Giryes, R. (2023) · Machine Learning for Data Science Handbook · doi:10.1007/978-3-031-24628-9_16 · Cited by 406
A Survey of Crowdsourcing Systems
Yuen, M.-C., King, I., & Leung, K.-S. (2011) · 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing · doi:10.1109/passat/socialcom.2011.203 · Cited by 392
Conditional Functional Dependencies for Data Cleaning
Bohannon, P., Fan, W., Geerts, F., Jia, X., & Kementsietsidis, A. (2007) · 2007 IEEE 23rd International Conference on Data Engineering · doi:10.1109/icde.2007.367920 · Cited by 375
Artificial Intelligence Safety and Security
(2018) · Chapman & Hall/CRC eBooks · doi:10.1201/9781351251389 · Cited by 364
The Science of Visual Data Communication: What Works
Franconeri, S. L., Padilla, L. M., Shah, P., Zacks, J. M., & Hullman, J. (2021) · Psychological Science in the Public Interest · doi:10.1177/15291006211051956 · Cited by 360
Webb, S. (2018) · Nature · doi:10.1038/d41586-018-02174-z · Cited by 353
Data Programming: Creating Large Training Sets, Quickly
Ratner, A., De, C., Wu, S., Selsam, D., & Ré, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07723 · Cited by 350
TFX: A TensorFlow-Based Production-Scale Machine Learning Platform
Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., et al. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098021 · Cited by 350
Chen, J., Yang, Z., & Yang, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.194 · Cited by 347
FACE: Feasible and Actionable Counterfactual Explanations
Poyiadzi, R., Sokol, K., Santos-Rodriguez, R., De Bie, T., & Flach, P. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375850 · Cited by 317
Toward a Quantitative Survey of Dimension Reduction Techniques
Espadoto, M., Martins, R. M., Kerren, A., Hirata, N. S. T., & Telea, A. C. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2944182 · Cited by 309
Overview and Importance of Data Quality for Machine Learning Tasks
Jain, A., Patel, H., Nagalapatti, L., Gupta, N., Mehta, S., Guttula, S., Mujumdar, S., Afzal, S., Sharma Mittal, R., & Munigala, V. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3406477 · Cited by 293
Algorithmic recourse: From counterfactual explanations to interventions
Karimi, A.-H., Schölkopf, B., & Valera, I. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445899 · Cited by 274
Multi-Objective Counterfactual Explanations
Dandl, S., Molnar, C., Binder, M., & Bischl, B. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58112-1_31 · Cited by 271
Tuning database configuration parameters with iTuned
Duan, S., Thummala, V., & Babu, S. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687627.1687767 · Cited by 267
A Survey on Trajectory Data Management, Analytics, and Learning
Wang, S., Bao, Z., Culpepper, J. S., & Cong, G. (2022) · ACM Computing Surveys · doi:10.1145/3440207 · Cited by 267
Discovering denial constraints
Chu, X., Ilyas, I. F., & Papotti, P. (2013) · Proceedings of the VLDB Endowment · doi:10.14778/2536258.2536262 · Cited by 262
An end-to-end learning-based cost estimator
Sun, J., & Li, G. (2019) · Proceedings of the VLDB Endowment · doi:10.14778/3368289.3368296 · Cited by 256
Benchmarking Adversarial Robustness on Image Classification
Dong, Y., Fu, Q.-A., Yang, X., Pang, T., Su, H., Xiao, Z., & Zhu, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00040 · Cited by 247
Embedded Unsupervised Feature Selection
Wang, S., Tang, J., & Liu, H. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9211 · Cited by 244
Automating large-scale data quality verification
Schelter, S., Lange, D., Schmidt, P., Celikel, M., Biessmann, F., & Grafberger, A. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3229863.3229867 · Cited by 240
A Benchmark for Data Imputation Methods
Jäger, S., Allhorn, A., & Bießmann, F. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.693674 · Cited by 231
Sharma, S., Henderson, J., & Ghosh, J. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375812 · Cited by 226
A survey of big data management: Taxonomy and state-of-the-art
Siddiqa, A., Hashem, I. A. T., Yaqoob, I., Marjani, M., Shamshirband, S., Gani, A., & Nasaruddin, F. (2016) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2016.04.008 · Cited by 226
Feature Extraction: A Survey of the Types, Techniques, Applications
Salau, A. O., & Jain, S. (2019) · 2019 International Conference on Signal Processing and Communication (ICSC) · doi:10.1109/icsc45622.2019.8938371 · Cited by 226
DeepEye: Towards Automatic Data Visualization
Luo, Y., Qin, X., Tang, N., & Li, G. (2018) · 2018 IEEE 34th International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2018.00019 · Cited by 218
A Marketplace for Data: An Algorithmic Solution
Agarwal, A., Dahleh, M., & Sarkar, T. (2019) · Proceedings of the 2019 ACM Conference on Economics and Computation · doi:10.1145/3328526.3329589 · Cited by 211
Data Augmentation for Deep Graph Learning: A Survey
Ding, K., Xu, Z., Tong, H., & Liu, H. (2022) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3575637.3575646 · Cited by 211
Comparison-Based Inverse Classification for Interpretability in Machine Learning
Laugel, T., Lesot, M., Marsala, C., Renard, X., & Detyniecki, M. (2018) · Communications in computer and information science · doi:10.1007/978-3-319-91473-2_9 · Cited by 204
Zhou, Y., & Goldman, S. (2005) · 16th IEEE International Conference on Tools with Artificial Intelligence · doi:10.1109/ictai.2004.48 · Cited by 203
Feature Engineering for Predictive Modeling Using Reinforcement Learning
Khurana, U., Samulowitz, H., & Turaga, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11678 · Cited by 202
DACE: Distribution-Aware Counterfactual Explanation by Mixed-Integer Linear Optimization
Kanamori, K., Takagi, T., Kobayashi, K., & Arimura, H. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/395 · Cited by 200
Rethinking Data Augmentation for Image Super-resolution: A Comprehensive Analysis and a New Strategy
Yoo, J., Ahn, N., & Sohn, K.-A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00840 · Cited by 199
Table union search on open data
Nargesian, F., Zhu, E., Pu, K. Q., & Miller, R. J. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3192965.3192973 · Cited by 198
Xanthopoulos, P., Pardalos, P. M., & Trafalis, T. B. (2012) · SpringerBriefs in optimization · doi:10.1007/978-1-4419-9878-1 · Cited by 198
Graph-based semi-supervised learning: A review
Chong, Y., Ding, Y., Yan, Q., & Pan, S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.12.130 · Cited by 194
Multiaccuracy: Black-Box Post-Processing for Fairness in Classification
Kim, M. P., Ghorbani, A., & Zou, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314287 · Cited by 193
Using Random Undersampling to Alleviate Class Imbalance on Tweet Sentiment Data
Prusa, J. D., Khoshgoftaar, T. M., Dittman, D. J., & Napolitano, A. (2015) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri.2015.39 · Cited by 193
Benchmark of filter methods for feature selection in high-dimensional gene expression survival data
Bommert, A., Welchowski, T., Schmid, M., & Rahnenführer, J. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbab354 · Cited by 182
TTS-GAN: A Transformer-Based Time-Series Generative Adversarial Network
Li, X., Metsis, V., Wang, H., & Ngu, A. H. H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-09342-5_13 · Cited by 176
Interpretability and fairness evaluation of deep learning models on MIMIC-IV dataset
Meng, C., Trinh, L., Xu, N., Enouen, J., & Liu, Y. (2022) · Scientific Reports · doi:10.1038/s41598-022-11012-2 · Cited by 174
A crowdsourcing open platform for literature curation in UniProt
Wang, Y., Wang, Q., Huang, H., Huang, W., Chen, Y., McGarvey, P. B., Wu, C. H., & Arighi, C. N. (2021) · PLoS Biology · doi:10.1371/journal.pbio.3001464 · Cited by 173
Bisong, E. (2019) · Building Machine Learning and Deep Learning Models on Google Cloud Platform · doi:10.1007/978-1-4842-4470-8_18 · Cited by 166
Comparison of Different Image Data Augmentation Approaches
Nanni, L., Paci, M., Brahnam, S., & Lumini, A. (2021) · Journal of Imaging · doi:10.3390/jimaging7120254 · Cited by 157
Regularized Learning for Domain Adaptation under Label ShiftsPreprint
Azizzadenesheli, K., Liu, A., Yang, F., & Anandkumar, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.09734 · Cited by 154
Self-citationCARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation AlgorithmsPreprint
Pawelczyk, M., Bielawski, S., Heuvel, J. V. D., Richter, T., & Kasneci, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.00783 · Cited by 154
GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation
Yoo, K. M., Park, D., Kang, J., Lee, S.-W., & Park, W. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.192 · Cited by 151
A benchmark and comparison of active learning for logistic regression
Yang, Y., & Loog, M. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.06.004 · Cited by 149
Systematic review of data-centric approaches in artificial intelligence and machine learning
Singh, P. (2023) · Data Science and Management · doi:10.1016/j.dsm.2023.06.001 · Cited by 136
Dataset Discovery in Data Lakes
Bogatu, A., Fernandes, A. A. A., Paton, N. W., & Κωνσταντίνου, Ν. (2020) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde48307.2020.00067 · Cited by 133
Hsu, W.-N., Zhang, Y., & Glass, J. R. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8268911 · Cited by 131
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2020) · arXiv · doi:10.48550/arxiv.2012.15723 · Cited by 124
DataPerf: Benchmarks for Data-Centric AI DevelopmentPreprint
Mazumder, M., Banbury, C., Yao, X., Karlaš, B., Rojas, W. G., Diamos, S., Diamos, G., He, L., Parrish, A., Kirk, H. R., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.10062 · Cited by 123
A Feature Extraction & Selection Benchmark for Structural Health Monitoring
Buckley, T., Ghosh, B., & Pakrashi, V. (2022) · Structural Health Monitoring · doi:10.1177/14759217221111141 · Cited by 120
Obukhov, A. D., & Krasnyanskiy, M. N. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-63322-6_8 · Cited by 117
Zhang, X., Chang, Z., Li, Y., Wu, H. L., Tan, J. R., Li, F., & Cui, B. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3538598.3538604 · Cited by 113
Data Integration: The Current Status and the Way Forward.
Stonebraker, M. R., & Ilyas, I. F. (2018) · IEEE Data(base) Engineering Bulletin · Cited by 110
Knowledge Graph Quality Management: a Comprehensive Survey
Xue, B., & Zou, L. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3150080 · Cited by 108
To Join or Not to Join? Thinking Twice about Joins before Feature Selection
Kumar, A., Naughton, J. F., Patel, J. M., & Zhu, X. (2016) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2882903.2882952 · Cited by 103
In-Processing Modeling Techniques for Machine Learning Fairness: A Survey
Wan, M., Zha, D., Liu, N., & Zou, N. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3551390 · Cited by 99
Self-citationData Mining and Machine Learning to Promote Smart Cities: A Systematic Review from 2000 to 2018
de Souza, J. T., de Francisco, A. C., Piekarski, C. M., & do Prado, G. F. (2019) · Sustainability · doi:10.3390/su11041077 · Cited by 96
Do Image Classifiers Generalize Across Time?
Shankar, V., Dave, A., Roelofs, R., Ramanan, D., Recht, B., & Schmidt, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00952 · Cited by 92
Meta-AAD: Active Anomaly Detection with Deep Reinforcement Learning
Zha, D., Lai, K.-H., Wan, M., & Hu, X. (2020) · Proceedings · doi:10.1109/icdm50108.2020.00086 · Cited by 91
Self-citationComparison of Instance Selection Algorithms II. Results and Comments
Grochowski, M., & Jankowski, N. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24844-6_87 · Cited by 89
FOCUS: Flexible Optimizable Counterfactual Explanations for Tree Ensembles
Lučić, A., Oosterhuis, H., Haned, H., & de Rijke, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i5.20468 · Cited by 88
Finding representative patterns with ordered projections
Riquelme, J. C., Aguilar–Ruiz, J. S., & Toro, M. (2002) · Pattern Recognition · doi:10.1016/s0031-3203(02)00119-x · Cited by 84
BERTese: Learning to Speak to BERT
Haviv, A., Berant, J., & Globerson, A. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.316 · Cited by 84
Slice Finder: Automated Data Slicing for Model Validation
Chung, Y., Kraska, T., Polyzotis, N., Tae, K. H., & Whang, S. E. (2019) · 2019 IEEE 35th International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2019.00139 · Cited by 83
Bridging the Semantic Gap with SQL Query Logs in Natural Language Interfaces to Databases
Baik, C., Jagadish, H. V., & Li, Y. (2019) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2019.00041 · Cited by 83
He, Y., Tang, J., Ouyang, H. J., Kang, C., Yin, D., & Chang, Y. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983835 · Cited by 76
Heffetz, Y., Vainshtein, R., Katz, G., & Rokach, L. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403261 · Cited by 57
SliceLine: Fast, Linear-Algebra-based Slice Finding for ML Model Debugging
Sagadeeva, S., & Böehm, M. (2021) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3448016.3457323 · Cited by 55
Snowy: Recommending Utterances for Conversational Visual Analysis
Srinivasan, A., & Setlur, V. (2021) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3472749.3474792 · Cited by 54
A Comparison of Radial and Linear Charts for Visualizing Daily Patterns
Waldner, M., Diehl, A., Gračanin, D., et al. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934784 · Cited by 53
Learning to Recommend Visualizations from Data
Qian, X., Rossi, R. A., Du, F., Kim, S., Koh, E., Malik, S., Lee, T. Y., & Chan, J. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467224 · Cited by 53
Automated Anomaly Detection via Curiosity-Guided Search and Self-Imitation Learning
Li, Y., Chen, Z., Zha, D., Zhou, K., Jin, H., Chen, H., & Hu, X. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3105636 · Cited by 52
Self-citationSaporta, G. (2002) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(01)00072-x · Cited by 48
Jia, R., Wu, F., Sun, X., Xu, J., Dao, D., Kailkhura, B., Zhang, C., Li, B., & Song, D. X. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00814 · Cited by 46
Towards Similarity-Aware Time-Series Classification
Zha, D., Lai, K.-H., Zhou, K., & Hu, X. (2022) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611977172.23 · Cited by 46
Self-citationCohen, T., Roberts, K. E., Gururaj, A. E., Chen, X., Pournejati, S., Alter, G. C., Hersh, W., Demner‐Fushman, D., Lucila Ohno-machado, A., & Xu, H. (2017) · Database · doi:10.1093/database/bax061 · Cited by 42
PyODDS: An End-to-end Outlier Detection System with Automated Machine Learning
Li, Y., Zha, D., Venugopal, P. K., Zou, N., & Hu, X. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366424.3383530 · Cited by 42
Self-citationCounterfactual Explanations for Oblique Decision Trees:Exact, Efficient Algorithms
Carreira-Perpiñán, M. Á., & Hada, S. S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16851 · Cited by 41
An Evaluation-Focused Framework for Visualization Recommendation Algorithms
Zeng, Z., Moh, P., Du, F., Hoffswell, J., Lee, T. Y., Malik, S., Koh, E., & Battle, L. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114814 · Cited by 38
Data Quality: The Role of Empiricism
Sadiq, S. W., Dasu, T., Dong, X. L., Freire, J., Ilyas, I. F., Link, S., Miller, M. J., Naumann, F., Zhou, X., & Srivastava, D. (2018) · ACM SIGMOD Record · doi:10.1145/3186549.3186559 · Cited by 37
Annotator Rationales for Labeling Tasks in Crowdsourcing
Kutlu, M., McDonnell, T., Elsayed, T., & Lease, M. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12012 · Cited by 36
Active Ensemble Learning for Knowledge Graph Error Detection
Dong, J., Zhang, Q., Huang, X. X., Tan, Q., Zha, D., & Zihao, Z. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3570368 · Cited by 33
Self-citationData excellence for AI: why should you care?
Aroyo, L., Lease, M., Paritosh, P., & Schaekermann, M. (2022) · Interactions · doi:10.1145/3517337 · Cited by 31
Sharma, S., Henderson, J., & Ghosh, J. (2019) · arXiv · doi:10.48550/arxiv.1905.07857 · Cited by 28
Comparison of Instance Selection and Construction Methods with Various Classifiers
Blachnik, M., & Kordos, M. (2020) · Applied Sciences · doi:10.3390/app10113933 · Cited by 27
Zhou, X., Jin, L., Sun, J., Zhao, X., Yu, X., Feng, J., Li, S., Wang, T., Li, K., & Liu, L. (2021) · Proceedings of the VLDB Endowment · doi:10.14778/3476311.3476334 · Cited by 25
BED: A Real-Time Object Detection System for Edge Devices
Wang, G., Bhat, Z. P., Jiang, Z., Chen, Y.-W., Zha, D., Reyes, A. C., Niktash, A., Ulkar, M. G., Okman, E., Cai, X., et al. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557168 · Cited by 24
Self-citationMIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset
Meng, C., Trinh, L., Xu, N., & Liu, Y. (2021) · doi:10.21203/rs.3.rs-402058/v1 · Cited by 21
AutoOD: Neural Architecture Search for Outlier Detection
Li, Y., Chen, Z., Zha, D., Zhou, K., Jin, H., Chen, H., & Hu, X. (2021) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde51399.2021.00210 · Cited by 19
Self-citationTowards Automated Imbalanced Learning with Deep Hierarchical Reinforcement Learning
Zha, D., Lai, K.-H., Tan, Q., Ding, S., Zou, N., & Hu, X. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557474 · Cited by 17
Self-citationEstimating the Number and Sizes of Fuzzy-Duplicate Clusters
Heise, A., Kasneci, G., & Naumann, F. (2014) · doi:10.1145/2661829.2661885 · Cited by 16
Quantitative program slicing: Separating statements by relevance
Santelices, R., Zhang, Y., Jiang, S., Cai, H., & Zhang, Y. (2013) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2013.6606695 · Cited by 16
Data-centric AI: Techniques and Future Perspectives
Zha, D., Lai, K.-H., Yang, F., Zou, N., Gao, H., & Hu, X. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599553 · Cited by 16
Self-citationFairness-Aware Unsupervised Feature Selection
Xing, X., Liu, H., Chen, C., et al. (2021) · Proceedings of the ... ACM International Conference on Information & Knowledge Management · doi:10.1145/3459637.3482106 · Cited by 10
Vijayan, A., Fatima, S., Sowmya, A., & Vafaee, F. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac315 · Cited by 9
Building Data Curation Processes with Crowd Intelligence
Chen, T., Han, L., Demartini, G., Indulska, M., & Sadiq, S. W. (2020) · Lecture notes in business information processing · doi:10.1007/978-3-030-58135-0_3 · Cited by 7
Active Feature Selection for the Mutual Information Criterion
Schnapp, S., & Sivan Sabato, A. (2021) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v35i11.17144 · Cited by 7
Becker, M., Burkart, N., Birnstill, P., & Beyerer, J. (2021) · Advances in Artificial Intelligence and Machine Learning · doi:10.54364/aaiml.2021.1107 · Cited by 7
Research works printed but not linked: 101
- Mohamed Abdelaal Christian Hammacher and Harald Schoening. 2023. REIN: A comprehensive benchmark framework for data cleaning methods in ML pipelines. Retrieved from https://arXiv:2302.04702
- Peshawa Jamal Muhammad Ali, Rezhna H. Faraj, Erbil Koya, Peshawa J. Muhammad Ali, and Rezhna H. Faraj. 2014. Data normalization and standardization: A technical report. Mach. Learn. Tech. Rep. 1, 1 (2014), 1–6.
- 10.1109/4236.612229 · doi:10.1109/4236.612229
- Patricia C. Arocena, Boris Glavic, Giansalvatore Mecca, Renée J. Miller, Paolo Papotti, and Donatello Santoro. 2016. Benchmarking data curation systems. IEEE Data Eng. Bull. 39, 2 (2016), 47–62.
- B. Azhagusundari, Antony Selvadoss Thanamani et al. 2013. Feature selection based on information gain. Int. J. Innov.e Technol. Explor. Eng. 2, 2 (2013), 18–21.
- Matias Barenstein. 2019. Propublica’s compas data revisited. Retrieved from https://arXiv:1906.04711
- 10.1109/HICSS.1995.375612 · doi:10.1109/hicss.1995.375612
- Eyal Betzalel Coby Penso Aviv Navon and Ethan Fetaya. 2022. A study on the evaluation of generative models. Retrieved from https://arXiv:2206.10935
- Anant Bhardwaj, Souvik Bhattacherjee, Amit Chavan, Amol Deshpande, Aaron J. Elmore, Samuel Madden, and Aditya G. Parameswaran. 2015. Datahub: Collaborative data science & dataset version management at scale. In Proceedings of the CIDR.
- Pierre Blanchart. 2021. An exact counterfactual-example-based approach to tree-ensemble models interpretability. Retrieved from https://arXiv:2105.14820
- Benedikt Boecking, Willie Neiswanger, Eric Xing, and Artur Dubrawski. 2021. Interactive weak supervision: Learning useful heuristics for data labeling. In Proceedings of the ICLR.
- Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the FAccT.
- 10.1007/978-1-4614-7485-2_17 · doi:10.1007/978-1-4614-7485-2_17
- Surajit Chaudhuri and Vivek R. Narasayya. 1997. An efficient, cost-driven index selection tool for Microsoft SQL server. In Proceedings of the VLDB.
- 10.1109/ICDE.2011.5767949 · doi:10.1109/icde.2011.5767949
- 10.5555/1622407.1622416 · doi:10.5555/1622407.1622416
- Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Proceedings of the NeurIPS.
- 10.5555/1622737.1622744 · doi:10.5555/1622737.1622744
- Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. 2019. Autoaugment: Learning augmentation policies from data. In Proceedings of the CVPR.
- Ofer Dekel and Ohad Shamir. 2009. Vox Populi: Collecting high-quality labels from a crowd. In Proceedings of the COLT.
- Meghana Deodhar Xiao Ma Yixin Cai Alex Koes Alex Beutel and Jilin Chen. 2022. A Human-ML collaboration framework for improving video content reviews. Retrieved from https://arXiv:2210.09500
- Luc Desnoyers. 2011. Toward a taxonomy of visuals in science communication. Techn. Commun. 58, 2 (2011), 119–134.
- Amit Dhurandhar Tejaswini Pedapati Avinash Balakrishnan Pin-Yu Chen Karthikeyan Shanmugam and Ruchir Puri. 2019. Model agnostic contrastive explanations for structured data. Retrieved from https://arXiv:1906.00117
- Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. 2021. Retiring adult: New datasets for fair machine learning. In Proceedings of the NeurIPS.
- Sirui Ding Ruixiang Tang Daochen Zha Na Zou Kai Zhang Xiaoqian Jiang and Xia Hu. 2023. Fairly predicting graft failure in liver transplant for organ assigning. Retrieved from https://arXiv:2302.09400
- Iddo Drori Yamuna Krishnamurthy Remi Rampin Raoni de Paula Lourenco Jorge Piazentin Ono Kyunghyun Cho Claudio Silva and Juliana Freire. 2021. AlphaD3M: Machine learning pipeline synthesis. Retrieved from https://arXiv:2111.02508
- Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R. Arabnia. 2021. A brief review of domain adaptation. In Proceedings from ICDATA and IKE. 877–894.
- Raul Castro Fernandez, Ziawasch Abedjan, Famien Koko, Gina Yuan, Samuel Madden, and Michael Stonebraker. 2018. Aurum: A data discovery system. In Proceedings of the ICDE.
- Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and robust automated machine learning. In Proceedings of the NeurIPS.
- 10.1145/3448016.3457334 · doi:10.1145/3448016.3457334
- Edwin Gamboa, Alejandro Libreros, Matthias Hirth, and Dan Dubiner. 2022. Human-AI collaboration for improving the identification of cars for autonomous driving. In Proceedings of the CIKM Workshop.
- Amirata Ghorbani, Michael Kim, and James Zou. 2020. A distributional framework for data valuation. In Proceedings of the ICML.
- Amirata Ghorbani and James Zou. 2019. Data shapley: Equitable valuation of data for machine learning. In Proceedings of the ICML.
- Pieter Gijsbers Marcos L. P. Bueno Stefan Coors Erin LeDell Sébastien Poirier Janek Thomas Bernd Bischl and Joaquin Vanschoren. 2022. Amlb: An automl benchmark. Retrieved from https://arXiv:2207.12560
- Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. 2009. Covariate shift by kernel mean matching. Dataset Shift Mach. Learn. 3, 4 (2009), 5.
- Georges G. Grinstein Patrick Hoffman Ronald M. Pickett and Sharon J. Laskowski. 2002. Benchmark Development for the Evaluation of Visualization for Data Mining. Morgan Kaufmann Publishers Inc. San Francisco CA USA 129–176.
- Keren Gu Brandon Yang Jiquan Ngiam Quoc Le and Jonathon Shlens. 2019. Using videos to evaluate image model robustness. Retrieved from https://arXiv:1904.10076
- Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. 2022. G-Mixup: Graph data augmentation for graph classification. In Proceedings of the ICML.
- Haibo He, Yang Bai, Edwardo A. Garcia, and Shutao Li. 2008. ADASYN: Adaptive synthetic sampling approach for imbalanced learning. In Proceedings of the WCCI.
- Herodotos Herodotou, Harold Lim, Gang Luo, Nedyalko Borisov, Liang Dong, Fatma Bilgen Cetin, and Shivnath Babu. 2011. Starfish: A self-tuning system for big data analytics. In Proceedings of the CIDR.
- Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Proceedings of the NeurIPS.
- Sarah Hooper, Michael Wornow, Ying Hang Seah, Peter Kellman, Hui Xue, Frederic Sala, Curtis Langlotz, and Christopher Re. 2021. Cut out the annotator, keep the cutout: Better segmentation with weak supervision. In Proceedings of the ICLR.
- Johannes Jakubik Michael Vössing Niklas Kühl Jannis Walk and Gerhard Satzger. 2022. Data-centric artificial intelligence. Retrieved from https://arXiv:2212.11854
- Mohammad Hossein Jarrahi Ali Memariani and Shion Guha. 2022. The principles of data-centric AI (DCAI). Retrieved from https://arXiv:2211.14611
- Minqi Jiang Chaochuan Hou Ao Zheng Xiyang Hu Songqiao Han Hailiang Huang Xiangnan He Philip S. Yu and Yue Zhao. 2023. Weakly supervised anomaly detection: A survey. Retrieved from https://arXiv:2302.04549
- Zhimeng Jiang, Kaixiong Zhou, Zirui Liu, Li Li, Rui Chen, Soo-Hyun Choi, and Xia Hu. 2022. An information fusion approach to learning with instance-dependent label noise. In Proceedings of the ICLR.
- Shankar Kanthara, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. Chart-to-text: A large-scale benchmark for chart summarization. In Proceedings of the ACL.
- Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL.
- Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. 2021. Variational diffusion models. In Proceedings of the NeurIPS.
- Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao et al. 2021. Wilds: A benchmark of in-the-wild distribution shifts. In Proceedings of the ICML.
- Sanjay Krishnan and Eugene Wu. 2019. Alphaclean: Automatic generation of data cleaning pipelines. Retrieved from https://arXiv:1904.11827
- 10.1609/aaai.v35i18.18012 · doi:10.1609/aaai.v35i18.18012
- Kwei-Herng Lai, Daochen Zha, Junjie Xu, Yue Zhao, Guanchu Wang, and Xia Hu. 2021. Revisiting time series outlier detection: Definitions and benchmarks. In Proceedings of the NeurIPS.
- Kamakshi Lakshminarayan, Steven A. Harp, Robert P. Goldman, Tariq Samad et al. 1996. Imputation of missing data using machine learning techniques. In Proceedings of the KDD.
- Peng Li Xi Rao Jennifer Blase Yue Zhang Xu Chu and Ce Zhang. 2019. Cleanml: A benchmark for joint data cleaning and machine learning [experiments and analysis]. Retrieved from https://arXiv:1904.09483
- Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. 2018. Detecting and correcting for label shift with black box predictors. In Proceedings of the ICML.
- Zhining Liu, Pengfei Wei, Jing Jiang, Wei Cao, Jiang Bian, and Yi Chang. 2020. MESA: Boost ensemble imbalanced learning with meta-sampler. In Proceedings of the NeurIPS.
- 10.14778/3421424.3421425 · doi:10.14778/3421424.3421425
- Diego Martinex Daochen Zha Qiaoyu Tan and Xia Hu. 2023. Towards personalized preprocessing pipeline search. Retrieved from https://arXiv:2302.14329
- 10.1145/3318464.3380597 · doi:10.1145/3318464.3380597
- Mitar Milutinovic, Brandon Schoenfeld, Diego Martinez-Garcia, Saswati Ray, Sujen Shah, and David Yan. 2020. On evaluation of automl systems. In Proceedings of the ICML Workshop.
- Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. Retrieved from https://arXiv:1312.5602
- Andrew Ng. 2021. Data-centric AI resource hub. Snorkel AI. Retrieved February 8, 2023 from https://snorkel.ai/
- Andrew Ng. 2023. Landing AI. Landing AI. Retrieved February 8, 2023 from https://landing.ai/
- Erkin Otles, Jeeheh Oh, Benjamin Li, Michelle Bochinski, Hyeon Joo, Justin Ortwine, Erica Shenoy, Laraine Washer, Vincent B. Young, Krishna Rao et al. 2021. Mind the performance gap: Examining dataset shift during prospective validation. In Proceedings of the MLHC.
- Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray et al. 2022. Training language models to follow instructions with human feedback. In Proceedings of the NeurIPS.
- 10.1007/978-3-319-92639-1_60 · doi:10.1007/978-3-319-92639-1_60
- 10.1007/978-3-319-18818-8_2 · doi:10.1007/978-3-319-18818-8_2
- 10.14778/2733004.2733009 · doi:10.14778/2733004.2733009
- Neoklis Polyzotis and Matei Zaharia. 2021. What can data-centric AI learn from data and ML engineering? Retrieved from https://arXiv:2112.06439
- Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever et al. 2018. Improving language understanding by generative pre-training. OpenAI (2018).
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever et al. 2019. Language models are unsupervised multitask learners. OpenAI (2019).
- Alexander Ratner. 2023. Scale AI. Snorkel AI. Retrieved February 8, 2023 from https://snorkel.ai/
- 10.1109/ICDEW49219.2020.00035 · doi:10.1109/icdew49219.2020.00035
- Timo Schick and Hinrich Schütze. 2020. Few-shot text generation with pattern-exploiting training. Retrieved from https://arXiv:2012.11926
- Nabeel Seedat Fergus Imrie and Mihaela van der Schaar. 2022. DC-Check: A data-centric AI checklist to guide the development of reliable machine learning systems. Retrieved from https://arXiv:2211.05764
- Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. 2018. Poison frogs! Targeted clean-label poisoning attacks on neural networks. In Proceedings of the NeurIPS.
- Zheyan Shen Jiashuo Liu Yue He Xingxuan Zhang Renzhe Xu Han Yu and Peng Cui. 2021. Towards out-of-distribution generalization: A survey. Retrieved from https://arXiv:2108.13624
- Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. 2020. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. In Proceedings of the NeurIPS.
- Michael Stonebraker, Daniel Bruckner, Ihab F. Ilyas, George Beskales, Mitch Cherniack, Stanley B. Zdonik, Alexander Pagan, and Shan Xu. 2013. Data curation at scale: The data tamer system. In Proceedings of the CIDR.
- Oliver Sutton. 2012. Introduction to k nearest neighbour classification and condensed nearest neighbour data reduction. University lectures, University of Leicester 1 (2012).
- Wei Tang and Matthew Lease. 2011. Semi-supervised consensus labeling for crowdsourcing. In Proceedings of the SIGIR Workshop.
- Yuchao Tao Ryan McKenna Michael Hay Ashwin Machanavajjhala and Gerome Miklau. 2021. Benchmarking differentially private synthetic data generation algorithms. Retrieved from https://arXiv:2112.09238
- Saravanan Thirumuruganathan, Nan Tang, Mourad Ouzzani, and AnHai Doan. 2020. Data curation with deep learning. In Proceedings of the EDBT.
- 10.1109/ICDE.2000.839397 · doi:10.1109/icde.2000.839397
- Alexandr Wang. 2023. Scale AI. Scale AI. Retrieved February 8, 2023 from https://scale.com/
- Yidong Wang, Hao Chen, Yue Fan, Sun Wang, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo et al. 2022. Usb: A unified semi-supervised learning benchmark for classification. In Proceedings of the NeurIPS.
- Patrick Henry Winston. 1984. Artificial Intelligence. Addison-Wesley Longman Publishing.
- Jin Yong Yoo John X. Morris Eli Lifland and Yanjun Qi. 2020. Searching for a search method: Benchmarking search algorithms for generating nlp adversarial examples. Retrieved from https://arXiv:2009.06368
- Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. In Proceedings of the NeurIPS.
- 10.1190/1.1444352 · doi:10.1190/1.1444352
- Daochen Zha Zaid Pervaiz Bhat Kwei-Herng Lai Fan Yang and Xia Hu. 2023. Data-centric AI: Perspectives and challenges. Retrieved from https://arXiv:2301.04819
- 10.1007/s10115-018-1280-0 · doi:10.1007/s10115-018-1280-0
- 10.24963/ijcai.2022/862 · doi:10.24963/ijcai.2022/862
- Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Ji Liu. 2021. Douzero: Mastering doudizhu with self-play deep reinforcement learning. In Proceedings of the ICML.
- Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In Proceedings of the ICLR.
- Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2019. Self-attention generative adversarial networks. In Proceedings of the IICML.
- Jieyu Zhang Cheng-Yu Hsieh Yue Yu Chao Zhang and Alexander Ratner. 2022. A survey on programmatic weak supervision. Retrieved from https://arXiv:2202.05433
- 10.1109/TFUZZ.2019.2959995 · doi:10.1109/tfuzz.2019.2959995
- Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of the NeurIPS.
- Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. 2020. Rethinking pre-training and self-training. In Proceedings of the NeurIPS.
Cited by
The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review
Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01196-4 · Cited by 169
FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
Rotstein, N., Bensaïd, D., Brody, S., Ganz, R., & Kimmel, R. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00559 · Cited by 66
Impacts of generative artificial intelligence on the future of labor market: a systematic review
Salari, N., Beiromvand, M., Hosseinian-Far, A., Habibi, J., Babajani, F., & Mohammadi, M. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100652 · Cited by 49
“This progress was supported by a rapid increase in the availability of large datasets, which were deemed necessary for training these models (Zha et al., 2025).”
Semantically Orthogonal Framework for Citation Classification: Disentangling Intent and Content
Duan, C., & Tan, Z. (2026) · Lecture Notes in Computer Science · doi:10.1007/978-3-032-05409-8_12 · Cited by 3
Metadata
- Authors
- Zha, D., Bhat, Z. P., Lai, K.-H., Yang, F., Jiang, Z., Zhong, S., & Hu, X.
- Venue
- ACM Computing Surveys
- Published
- 2025-01-24
- Volume
- 57
- Issue
- 5
- Pages
- 1-42
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4406085118
- Cited by
- 334
- References
- 181
- FWCI
- 110.69
- Type
- journal article
- Language
- en
- Open access
- green
- Funders
- NSF (IIS-2224843, ITE-2429680)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.