The DOI matches its Crossref record; 71 of the 73 references it cites were found in a registry or an index.
Data Lifecycle Challenges in Production Machine Learning: A Survey
Abstract
Machine learning has become an essential tool for gleaning knowledge from data and tackling a diverse set of computationally hard tasks. However, the accuracy of a machine learned model is deeply tied to the data that it is trained on. Designing and building robust processes and tools that make it easier to analyze, validate, and transform data that is fed into large-scale machine learning systems poses data management challenges. Drawn from our experience in developing data-centric infrastructure for a production machine learning platform at Google, we summarize some of the interesting research challenges that we encountered, and survey some of the relevant literature from the data management and machine learning communities. Specifically, we explore challenges in three main areas of focus - data understanding, data validation and cleaning, and data preparation. In each of these areas, we try to explore how different constraints are imposed on the solutions depending on where in the lifecycle of a model the problems are encountered and who encounters them.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Data lifecycle activities form a critical and compelling subgroup of activities, which profoundly affect ML projects' success (Munappy et al., 2020;Polyzotis et al., 2018;Wan et al., 2019).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Polyzotis et al., 2018
Citations
Cites
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“Some of the typical transformations applied include normalization, bucketization, winsorizing, one-hot encoding, feature crosses, and using a pre-trained model or embeddings (mappings from values such as words to real numbers) to extract features [51].”
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · cited 1×
“Some promising techniques include autoencoders and restricted Boltzmann Machines [12].”
MethodStatistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · cited 1×
“Identifying new kinds of “spam” [34] is an open challenge as well.”
BackgroundPearson, K. (1900) · The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science · doi:10.1080/14786440009463897 · Cited by 4,069
Detection of Abrupt Changes: Theory and Application
Riedel, K. S. (1994) · Technometrics · doi:10.1080/00401706.1994.10485821 · Cited by 3,518 · cited 1×
“Time series analysis [10, 27, 18] can help track these statistics.”
BackgroundOn the probable error of a coefficient of correlation deduced from a small sample
Fisher, R. A. (1921) · Adelaide Research & Scholarship (AR&S) (University of Adelaide) · Cited by 2,089
Statistical Methods for Research Workers
Fisher, R. A. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_6 · Cited by 1,425
Ding, H., Trajcevski, G., Scheuermann, P., Wang, X., & Keogh, E. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1454159.1454226 · Cited by 1,354
Detection of Abrupt Changes: Theory and Application
Riedel, K. S., Basseville, M., Nikiforov, I. V., et al. (1994) · Technometrics · doi:10.2307/1269388 · Cited by 1,330
Settles, B. (2012) · Cited by 1,183
Get another label? improving data quality and data mining using multiple, noisy labelers
Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401965 · Cited by 1,165 · cited 1×
“Sheng et al. [65] shows how improving the quality of the already available labels can better improve model accuracy than collecting more examples.”
BackgroundOn Choosing and Bounding Probability Metrics
Gibbs, A. L., & Su, F. E. (2002) · International Statistical Review · doi:10.1111/j.1751-5823.2002.tb00178.x · Cited by 1,163 · cited 1×
“Other metrics that can be used include the L 1 and L ∞ metrics or Earth Mover’s distance [32, 70].”
MethodInferring causal impact using Bayesian structural time-series models
Brodersen, K. H., Gallusser, F., Koehler, J., Remy, N., & Scott, S. L. (2015) · The Annals of Applied Statistics · doi:10.1214/14-aoas788 · Cited by 961 · cited 1×
“Time series analysis [10, 27, 18] can help track these statistics.”
Background(2015) · neural information processing systems · Cited by 912
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3157794.3157797 · Cited by 777
Dremel: interactive analysis of web-scale datasets
Melnik, S., Gubarev, A., Long, J. J., Romer, G., Shivakumar, S., Tolton, M., & Vassilakis, T. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920886 · Cited by 711
BlinkDB: queries with bounded errors and bounded response times on very large data
Agarwal, S., Mozafari, B., Panda, A., Milner, H., Madden, S., & Stoica, I. (2013) · Proceedings of the 8th ACM European Conference on Computer Systems · doi:10.1145/2465351.2465355 · Cited by 694
WebTables: exploring the power of tables on the web
Cafarella, M. J., Halevy, A., Wang, D. Z., Wu, E., & Zhang, Y. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1453856.1453916 · Cited by 629
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2020) · The VLDB Journal · doi:10.1007/s00778-019-00552-1 · Cited by 510
Pearson, K. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_2 · Cited by 454 · cited 1×
“For example the chisquared test can be used to check homogeneity [55] by rejecting the null hypothesis for the distributions being the same.”
MethodA cost-based model and effective heuristic for repairing constraints by value modification
Bohannon, P., Fan, W., Flaster, M., & Rastogi, R. (2005) · Proceedings of the 2005 ACM SIGMOD international conference on Management of data · doi:10.1145/1066157.1066175 · Cited by 424 · cited 1×
“In the literature, cost-based models [16, 42] and conflict hypergraphs [23] are proposed.”
BackgroundData Programming: Creating Large Training Sets, Quickly
Ratner, A., De, C., Wu, S., Selsam, D., & Ré, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07723 · Cited by 350 · cited 1×
“Weak supervision is used in Snorkel [57, 56] where hand labeling is avoided altogether, and workers can programmatically generate lower-quality labels, which are then denoised with generative models.”
MethodTFX: A TensorFlow-Based Production-Scale Machine Learning Platform
Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., et al. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098021 · Cited by 350
Self-citationHolistic data cleaning: Putting violations into context
Chu, X., Ilyas, I. F., & Papotti, P. (2013) · 2013 IEEE 29th International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2013.6544847 · Cited by 326 · cited 1×
“In the literature, cost-based models [16, 42] and conflict hypergraphs [23] are proposed.”
BackgroundMLbase: A Distributed Machine-learning System
Kraska, T., Talwalkar, A., & Duchi, J. C. (2013) · Cited by 306
A multidisciplinary survey on discrimination analysis
Romei, A., & Ruggieri, S. (2014) · The Knowledge Engineering Review · doi:10.1017/s0269888913000039 · Cited by 288 · cited 1×
“Recently, determining if a trained model is “fair” [59] has become a critical issue.”
BackgroundActiveClean: interactive data cleaning for statistical modeling
Krishnan, S., Wang, J., Wu, E., Franklin, M. J., & Goldberg, K. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/2994509.2994514 · Cited by 264
S ee DB: efficient data-driven visualization recommendations to support visual analytics
Vartak, M., Rahman, S., Madden, S., Parameswaran, A., & Polyzotis, N. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2831360.2831371 · Cited by 222 · cited 2×
“Other metrics that can be used include the L1 and L∞ metrics or Earth Mover’s distance [32, 70].”
MethodSelf-citationOn approximating optimum repairs for functional dependency violations
Kolahi, S., & Lakshmanan, L. V. S. (2009) · Proceedings of the 12th International Conference on Database Theory · doi:10.1145/1514894.1514901 · Cited by 206 · cited 1×
“In the literature, cost-based models [16, 42] and conflict hypergraphs [23] are proposed.”
BackgroundData scientists in software teams: State of the art and challenges
Kim, M., Zimmermann, T., DeLine, R., & Begel, A. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2754374 · Cited by 199
Data Curation at Scale: The Data Tamer System
Stonebraker, M. R., Beskales, G., Pagan, A., Bruckner, D., Cherniack, M., Xu, S., Analytics, V., Ilyas, I. F., & Zdonik, S. (2013) · Cited by 191
SystemML: declarative machine learning on spark
Boehm, M., Dusenberry, M. W., Eriksson, D., Evfimievski, A. V., Manshadi, F. M., Pansare, N., Reinwald, B., Reiss, F. R., Sen, P., Surve, A. C., et al. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/3007263.3007279 · Cited by 184
Goods: Organizing Google's Datasets
Halevy, A., Korn, F., Noy, N. F., Olston, C., Polyzotis, N., Roy, S., & Whang, S. E. (2016) · Proceedings of the 2016 International Conference on Management of Data · doi:10.1145/2882903.2903730 · Cited by 171
Self-citationBigDansing: A System for Big Data Cleansing
Khayyat, Z., Ilyas, I. F., Jindal, A., Madden, S. R., Ouzzani, M., Papotti, P., Quiané-Ruiz, J.-A., Tang, N., & Yin, S. (2015) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2723372.2747646 · Cited by 163
Effortless data exploration with zenvisage: an expressive and interactive visual analytics system
Siddiqui, T., Kim, A., Lee, J., Karahalios, K. G., & Parameswaran, A. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/3025111.3025126 · Cited by 149
Towards Unified Data and Lifecycle Management for Deep Learning
Miao, H., Li, A., Davis, L. S., & Deshpande, A. (2017) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2017.112 · Cited by 144
DataHub: Collaborative Data Science & Dataset Version Management at ScalePreprint
Bhardwaj, A., Bhattacherjee, S., Chavan, A., Deshpande, A., Elmore, A. J., Madden, S. R., & Parameswaran, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.0798 · Cited by 138
Probabilistic demand forecasting at scale
Böse, J.-H., Flunkert, V., Gasthaus, J. A., Januschowski, T., Lange, D., Salinas, D., Schelter, S., Seeger, M., & Wang, Y. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3137765.3137775 · Cited by 138 · cited 1×
“In this article, driven from our experience in building data management infrastructure for a large-scale ML platform [11], we identify several core challenges in the management of ML data that are relevant to [11] and other ML platforms [17].”
BackgroundMacroBase: Prioritizing Attention in Fast Data
Bailis, P. D., Gan, E., Madden, S. R., Narayanan, D., Rong, K., & Suri, S. (2017) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3035918.3035928 · Cited by 135
KeystoneML: Optimizing Pipelines for Large-Scale Advanced Analytics
Sparks, E., Venkataraman, S., Kaftan, T., Franklin, M. J. M., & Recht, B. (2017) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2017.109 · Cited by 133
A Roadmap for a Rigorous Science of Interpretability.Preprint
Doshi‐Velez, F., & Kim, B. (2017) · arXiv (Cornell University) · Cited by 119
Model Selection Management Systems
Kumar, A., McCann, R., Naughton, J. F., & Patel, J. M. (2016) · ACM SIGMOD Record · doi:10.1145/2935694.2935698 · Cited by 118
Volkovs, M., Chiang, F., Szlichta, J., & Miller, R. J. (2014) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2014.6816655 · Cited by 105 · cited 1×
“The data validation can be done by comparing data with its previous versions, and alerts can be raised based on accumulated evidence to date [71].”
Brainwash: A data system for feature engineering
Anderson, M. R., Antenucci, D., Bittorf, V., Burgess, M., Cafarella, M., Kumar, A., Niu, F., Park, Y., Ré, C., & Zhang, C. (2013) · Cited by 104
Intelligent Rollups in Multidimensional OLAP Data
Sathe, G., & Sarawagi, S. (2001) · Cited by 103
On the relative trust between inconsistent data and inaccurate constraints
Beskales, G., Ilyas, I. F., Golab, L., & Galiullin, A. A. (2013) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2013.6544854 · Cited by 103
To Join or Not to Join? Thinking Twice about Joins before Feature Selection
Kumar, A., Naughton, J. F., Patel, J. M., & Zhu, X. (2016) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2882903.2882952 · Cited by 103
A unified model for data and constraint repair
Chiang, F., & Miller, R. J. (2011) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2011.5767833 · Cited by 99
Crankshaw, D., Bailis, P. D., Gonzalez, J. E., Li, H., Zhang, Z., Franklin, M. J. M., Ghodsi, A., & Jordan, M. I. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.3809 · Cited by 84
Controlling False Discoveries During Interactive Data Exploration
Zhao, Z., Stefani, L. D., Zgraggen, E., Binnig, C., Upfal, E., & Kraska, T. (2017) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3035918.3064019 · Cited by 83 · cited 1×
“The QUDE system [14, 75] takes a statistical approach and provides an interactive data exploration framework that uses multiple hypothesis testing to control the false positives.”
MethodData X-Ray: A Diagnostic Tool for Data Errors
Wang, X., Dong, X. L., & Meliou, A. (2015) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2723372.2750549 · Cited by 81
Visual exploration of machine learning results using data cube analysis
Kahng, M., Fang, D., & Chau, D. H. (2016) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/2939502.2939503 · Cited by 80 · cited 1×
“For example, MLCube [39] is a tool for visually exploring ML results that enable users to define slices using feature conditions to compute aggregate statistics and evaluation metrics over the slices.”
MethodAutomatically Tracking Metadata and Provenance of Machine Learning Experiments
Schelter, S., Böse, J.-H., Kirschnick, J., Klein, T., & Seufert, S. (2017) · Cited by 78
Chen, B.-C., Chen, L., Lin, Y., & Ramakrishnan, R. (2005) · Very Large Data Bases · Cited by 70
Feature Engineering for Knowledge Base ConstructionPreprint
Ré, C., Sadeghian, A., Shan, Z., Shin, J., Wang, F., Wu, S., & Zhang, C. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.6439 · Cited by 66 · cited 1×
“Feature engineering is a well-studied problem [7, 44, 43, 74, 58, 46, 8, 67] with a suite of techniques that are largely designed based on experience of ML experts.”
BackgroundMODELDB: A System for Machine Learning Model Management.
Vartak, M. (2017) · Conference on Innovative Data Systems Research · Cited by 66
Materialization Optimizations for Feature Selection Workloads
Zhang, C., Kumar, A., & Ré, C. (2016) · ACM Transactions on Database Systems · doi:10.1145/2877204 · Cited by 66 · cited 1×
“Feature engineering is a well-studied problem [7, 44, 43, 74, 58, 46, 8, 67] with a suite of techniques that are largely designed based on experience of ML experts.”
BackgroundInput selection for fast feature engineering
Anderson, M. R., & Cafarella, M. (2016) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2016.7498272 · Cited by 64 · cited 1×
“Feature engineering is a well-studied problem [7, 44, 43, 74, 58, 46, 8, 67] with a suite of techniques that are largely designed based on experience of ML experts.”
BackgroundA Common Runtime for High Performance Data Analysis.
Palkar, S., Thomas, J. J., Shanbhag, A., Schwarzkopf, M., Amarasinghe, S. P., & Zaharia, M. (2017) · Conference on Innovative Data Systems Research · Cited by 35
Inspector gadget: a framework for custom monitoring and debugging of distributed dataflows
Olston, C., & Reed, B. C. (2011) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/1989323.1989459 · Cited by 33
Interactive data exploration with smart drill-down
Joglekar, M., García-Molina, H., & Parameswaran, A. (2016) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2016.7498300 · Cited by 32 · cited 1×
“Similarly, smart drill-downs [38] discover and summarize interesting slices of the entire data.”
BackgroundAre key-foreign key joins safe to avoid when learning high-capacity classifiers?
Shah, V., Kumar, A., & Zhu, X. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3157794.3157804 · Cited by 30
A Demo of the Data Civilizer System
Fernandez, R. C., Deng, D., Mansour, E. A. H., Qahtan, A. A., Tao, W., Abedjan, Z., Elmagarmid, A. K., Ilyas, I. F., Madden, S. R., Ouzzani, M., et al. (2017) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3035918.3058740 · Cited by 30
Toward Sustainable Insights, or Why Polygamy is Bad for You.
Binnig, C., Stefani, L. D., Kraska, T., Upfal, E., Zgraggen, E., & Zhao, Z. (2017) · Conference on Innovative Data Systems Research · Cited by 28
Feature selection in enterprise analytics: a demonstration using an R-based data analytics system
Konda, P., Kumar, A., Ré, C., & Sashikanth, V. (2013) · Proceedings of the VLDB Endowment · doi:10.14778/2536274.2536302 · Cited by 24
Crowdsourcing large scale wrapper inference
Crescenzi, V., Merialdo, P., & Qiu, D. (2014) · Distributed and Parallel Databases · doi:10.1007/s10619-014-7163-9 · Cited by 19 · cited 1×
“Crowd-sourcing frameworks like Alfred [25] help moderately skilled crowd workers in extracting knowledge from a corpus of unstructured documents.”
BackgroundData scientists in software teams: state of the art and challenges
Kim, M., Zimmermann, T., DeLine, R. A., & Begel, A. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3182515 · Cited by 19
Tutorial summary: Active learning
Dasgupta, S., & Langford, J. C. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553559 · Cited by 15 · cited 1×
“In active learning [26, 63, 52], the learning procedure decides how best to enrich the data iteratively.”
MethodTowards large-scale data discovery: position paper
Fernandez, R. C., Abedjan, Z., Madden, S. R., & Stonebraker, M. R. (2016) · International Workshop on Exploratory Search in Databases and the Web (ExploreDB) · doi:10.1145/2948674.2948675 · Cited by 12
ProvDB: A System for Lifecycle Management of Collaborative Analysis WorkflowsPreprint
Miao, H., Chavan, A., & Deshpande, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.04963 · Cited by 4 · cited 1×
“ProvDB [49] provides a unified provenance and metadata management system to support lifecycles of complex collaborative data science workflows.”
Background
Research works printed but not linked: 2
- J. M. Hellerstein , V. Sreekanti , J. E. Gonzales , Sudhansku, Arora, A. Bhattacharyya , S. Das , A. Dey , M. Donsky , G. Fierro , S. Nag , K. Ramachandran , C. She , E. Sun , C. Steinbach , and V. Subramanian . Establishing common ground with data context . In CIDR , 2017 . J. M. Hellerstein, V. Sreekanti, J. E. Gonzales, Sudhansku, Arora, A. Bhattacharyya, S. Das, A. Dey, M. Donsky, G. Fierro, S. Nag, K. Ramachandran, C. She, E. Sun, C. Steinbach, and V. Subramanian. Establishing common ground with data context. In CIDR, 2017.
- A. Jenkinson . Beyond segmentation. Journal of Targeting, Measurement and Analysis for Marketing, (1):60--72 , 1994 . A. Jenkinson. Beyond segmentation. Journal of Targeting, Measurement and Analysis for Marketing, (1):60--72, 1994.
Cited by
A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
Li, Q., Wen, Z., Wu, Z., Hu, S., Wang, N., Li, Y., Liu, X., & He, B. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3124599 · Cited by 1,407
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective
Roh, Y., Heo, G., & Whang, S. E. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2946162 · Cited by 968
“In fact, data management plays a role in almost all aspects of machine learning [4], [5].”
BackgroundSelf-citation“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781
“Multiple recent studies examine the challenges of production machine learning [6, 100, 101].”
BackgroundChallenges in deploying machine learning: A survey of case studies
Paleyes, A., Urma, R.-G., & Lawrence, N. D. (2023) · ACM Computing Surveys · doi:10.1145/3533378 · Cited by 638
“[15].”
Data collection and quality challenges in deep learning: a data-centric AI perspective
Whang, S. E., Roh, Y., Song, H., & Lee, J.-G. (2023) · The VLDB Journal · doi:10.1007/s00778-022-00775-9 · Cited by 573
Self-citation
Metadata
- Authors
- Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M.
- Affiliations
- Google (United States) (US)
- Korea Advanced Institute of Science and Technology (KR)
- Venue
- ACM SIGMOD Record
- Published
- 2018-12-11
- Volume
- 47
- Issue
- 2
- Pages
- 17-28
- ISSN
- 0163-5808 (ISSN-L)
- OpenAlex
- W2905588001
- Semantic Scholar
- 18c2d0b44a0c292ce76696bfe1dfd6839fe09ff9
- Cited by
- 214
- References
- 73
- Citations by year
- 2026: 29 · 2025: 29 · 2024: 34 · 2023: 30 · 2022: 24 · 2021: 31 · 2020: 26 · 2019: 11
- FWCI
- 13.76
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGMOD Record (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Data Quality and Management
- Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Keywords
- data lifecycle, machine learning in production, data management, data validation, data cleaning, data preparation, data understanding, machine learning platform
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.