The DOI matches its Crossref record; 89 of the 89 references it cites were found in a registry or an index.
Profiling relational data: a survey
Open accessAbstract
Profiling data to determine metadata about a given dataset is an important and frequent activity of any IT professional and researcher and is necessary for various use-cases. It encompasses a vast array of methods to examine datasets and produce metadata. Among the simpler results are statistics, such as the number of null values and distinct values in a column, its data type, or the most frequent patterns of its data values. Metadata that are more difficult to compute involve multiple columns, namely correlations, unique column combinations, functional dependencies, and inclusion dependencies. Further techniques detect conditional properties of the dataset at hand. This survey provides a classification of data profiling tasks and comprehensively reviews the state of the art for each class. In addition, we review data profiling tools and systems from research and industry. We conclude with an outlook on the future of data profiling beyond traditional profiling tasks and beyond relational databases.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Jain, A. K., Murty, M. N., & Flynn, P. J. (1999) · ACM Computing Surveys · doi:10.1145/331499.331504 · Cited by 13,199
Fast Algorithms for Mining Association Rules in Large Databases
Agrawal, R., & Srikant, R. (1994) · Very Large Data Bases · Cited by 9,419
Opinion Mining and Sentiment Analysis
Pang, B., & Lee, L. (2008) · now publishers, Inc. eBooks · doi:10.1561/9781601981516 · Cited by 6,720
Mining frequent patterns without candidate generation
Han, J., Pei, J., & Yin, Y. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335372 · Cited by 6,402
Opinion mining and sentiment analysis
Pang, B., & Lee, L. (2008) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000011 · Cited by 6,372
Survey of Clustering Algorithms
Xu, R., & WunschII, D. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2005.845141 · Cited by 6,197
Data mining: an overview from a database perspective
Chen, M.-S., Han, J., & Yu, P. S. (1996) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/69.553155 · Cited by 2,232
Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals
Gray, J., Chaudhuri, S., Bosworth, A., Layman, A., Reichart, D., Venkatrao, M., Pellow, F., & Pirahesh, H. (1997) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009726021843 · Cited by 2,134
Euzenat, J., & Shvaiko, P. (2007) · Cited by 2,124
Sentiment Analysis and Subjectivity
Bing, L. (2010) · Cited by 2,078
Scalable algorithms for association mining
Zaki, M. J. (2000) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/69.846291 · Cited by 1,751
Pipino, L. L., Lee, Y. W., & Wang, R. Y. (2002) · Communications of the ACM · doi:10.1145/505248.506010 · Cited by 1,724
Brill, E. (1995) · Cited by 1,520
(2000) · Cited by 1,467
Data preparation for data mining
Pyle, D. (1999) · Cited by 1,368
Generic Schema Matching with Cupid
Madhavan, J., Bernstein, P. A., & Rahm, E. (2001) · Cited by 1,296
Beyond market baskets: generalizing association rules to correlations
Brin, S., Motwani, R., & Silverstein, C. (1997) · Proceedings of the 1997 ACM SIGMOD international conference on Management of data - SIGMOD '97 · doi:10.1145/253260.253327 · Cited by 1,281
Challenges and opportunities with big data
Labrinidis, A., & Jagadish, H. V. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2367502.2367572 · Cited by 1,070
Algorithms for association rule mining — a general survey and comparison
Hipp, J., Güntzer, U., & Nakhaeizadeh, G. (2000) · ACM SIGKDD Explorations Newsletter · doi:10.1145/360402.360421 · Cited by 962
Christen, P. (2012) · Springer eBooks · doi:10.1007/978-3-642-31164-2 · Cited by 732
Sarawagi, S. (2008) · Foundations and Trends in Databases · doi:10.1561/1900000003 · Cited by 707
Tane: An Efficient Algorithm for Discovering Functional and Approximate Dependencies
Huhtala, Y. (1999) · The Computer Journal · doi:10.1093/comjnl/42.2.100 · Cited by 598
Juola, P. (2007) · now publishers, Inc. eBooks · doi:10.1561/9781601981196 · Cited by 558
Improved histograms for selectivity estimation of range predicates
Poosala, V., Haas, P. J., Ioannidis, Y. E., & Shekita, E. J. (1996) · Proceedings of the 1996 ACM SIGMOD international conference on Management of data - SIGMOD '96 · doi:10.1145/233269.233342 · Cited by 556
The History of Histograms (abridged)
Ioannidis, Y. (2003) · Elsevier eBooks · doi:10.1016/b978-012722442-8/50011-2 · Cited by 502
Potter's Wheel: An Interactive Data Cleaning System
Raman, V., & Hellerstein, J. L. (2001) · Cited by 492
Incremental Clustering for Mining in a Data Warehousing Environment
Ester, M., Kriegel, H., Sander, J., Wimmer, M. H., & Xu, X. (1998) · Cited by 474
Synopses for Massive Data: Samples, Histograms, Wavelets, Sketches
Cormode, G., Garofalakis, M., Haas, P. J., & Jermaine, C. (2011) · Foundations and Trends in Databases · doi:10.1561/1900000004 · Cited by 468
Efficient mining of both positive and negative association rules
Wu, X., Zhang, C., & Zhang, S. (2004) · ACM Transactions on Information Systems · doi:10.1145/1010614.1010616 · Cited by 466
Conditional functional dependencies for capturing data inconsistencies
Fan, W., Geerts, F., Jia, X., & Kementsietsidis, A. (2008) · ACM Transactions on Database Systems · doi:10.1145/1366102.1366103 · Cited by 460
Using a hash-based method with transaction trimming for mining association rules
Park, J. S., Chen, M.-S., & Yu, P. S. (1997) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/69.634757 · Cited by 440
The MADlib analytics library: or MAD skills, the SQL
Hellerstein, J. M., Ré, C., Schoppmann, F., Wang, D. Z., Fratkin, E., Gorajek, A., Ng, K. S., Welton, C., Feng, X., Li, K., et al. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2367502.2367510 · Cited by 373
CORDS: automatic discovery of correlations and soft functional dependencies
Ilyas, I. F., Markl, V., Haas, P., Brown, P., & Aboulnaga, A. (2004) · Proceedings of the 2004 ACM SIGMOD international conference on Management of data · doi:10.1145/1007568.1007641 · Cited by 315
NADEEF: a commodity data cleaning system
Dallachiesa, M., Ebaid, A., Eldawy, A., Elmagarmid, A., Ilyas, I. F., Ouzzani, M., & Tang, N. (2013) · Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2463676.2465327 · Cited by 313
Improving data quality: consistency and accuracy
Cong, G., Fan, W., Geerts, F., et al. (2007) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 313
Beyond market baskets: generalizing association rules to correlations
Brin, S., Motwani, R., & Silverstein, C. (1997) · ACM SIGMOD Record · doi:10.1145/253262.253327 · Cited by 308
Holmes, D. I. (1994) · Computers and the Humanities · doi:10.1007/bf01830689 · Cited by 272
Discovering data quality rules
Chiang, F., & Miller, R. J. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1453856.1453980 · Cited by 265
Discovering denial constraints
Chu, X., Ilyas, I. F., & Papotti, P. (2013) · Proceedings of the VLDB Endowment · doi:10.14778/2536258.2536262 · Cited by 262
Discovering Conditional Functional Dependencies
Fan, W., Geerts, F., Li, J., & Xiong, M. (2011) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2010.154 · Cited by 253
Approximate inference of functional dependencies from relations
Kivinen, J., & Mannila, H. (1995) · Theoretical Computer Science · doi:10.1016/0304-3975(95)00028-u · Cited by 241
On schema matching with opaque column names and data values
Kang, J., & Naughton, J. F. (2003) · Proceedings of the 2003 ACM SIGMOD international conference on Management of data · doi:10.1145/872757.872783 · Cited by 238
Reasoning about record matching rules
Fan, W., Jia, X., Li, J., & Ma, S. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687627.1687674 · Cited by 226
Statistical profile estimation in database systems
Mannino, M. V., Chu, P., & Sager, T. (1988) · ACM Computing Surveys · doi:10.1145/62061.62063 · Cited by 225
Database dependency discovery: a machine learning approach
Flach, P. A., & Savnik, I. (1999) · Bristol Research (University of Bristol) · doi:10.5555/1216155.1216159 · Cited by 220
Profiler: integrated statistical analysis and visualization for data quality assessment
Kandel, S., Parikh, R., Paepcke, A., Hellerstein, J. M., & Heer, J. (2012) · Proceedings of the International Working Conference on Advanced Visual Interfaces · doi:10.1145/2254556.2254659 · Cited by 220
Functional dependency discovery: an experimental evaluation of seven algorithms
Papenbrock, T., Ehrlich, J., Marten, J., Neubert, T., Rudolph, J.-P., Schönberg, M., Zwiener, J., & Naumann, F. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2794367.2794377 · Cited by 220
Self-citationDiscovering all most specific sentences
Gunopulos, D., Khardon, R., Mannila, H., Saluja, S., Toivonen, H., & Sharma, R. S. (2003) · ACM Transactions on Database Systems · doi:10.1145/777943.777945 · Cited by 216
On generating near-optimal tableaux for conditional functional dependencies
Golab, L., Karloff, H., Korn, F., Srivastava, D., & Yu, B. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1453856.1453900 · Cited by 200
Self-citationMining database structure; or, how to build a data quality browser
Dasu, T., Johnson, T. J., Muthukrishnan, S., & Shkapenyuk, V. (2002) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/564691.564719 · Cited by 191
Data Curation at Scale: The Data Tamer System
Stonebraker, M. R., Beskales, G., Pagan, A., Bruckner, D., Cherniack, M., Xu, S., Analytics, V., Ilyas, I. F., & Zdonik, S. (2013) · Cited by 191
Wyss, C., Giannella, C., & Robertson, E. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44801-2_11 · Cited by 170
Regular expression learning for information extraction
Li, Y., Krishnamurthy, R., Raghavan, S., Vaithyanathan, S., & Jagadish, H. V. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613719 · Cited by 165
LODStats – An Extensible Framework for High-Performance Dataset Analytics
Auer, S., Demter, J., Martin, M., & Lehmann, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33876-2_31 · Cited by 163
Naumann, F. (2014) · ACM SIGMOD Record · doi:10.1145/2590989.2590995 · Cited by 161
Self-citationLucchesi, C. L., & Osborn, S. L. (1978) · Journal of Computer and System Sciences · doi:10.1016/0022-0000(78)90009-0 · Cited by 160
Identifying extended entity-relationship object structures in relational schemas
Markowitz, V. M., & Makowsky, J. A. (1990) · IEEE Transactions on Software Engineering · doi:10.1109/32.57618 · Cited by 152
Inferring XML schema definitions from XML data
Bex, G. J., Neven, F., & Vansummeren, S. (2007) · Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) · Cited by 126
Mignet, L., Barbosa, D., & Veltri, P. (2003) · International World Wide Web Conference (WWW) · doi:10.1145/775152.775223 · Cited by 122
Summarization – compressing data into an informative representation
Chandola, V., & Kumar, V. (2006) · Knowledge and Information Systems · doi:10.1007/s10115-006-0039-1 · Cited by 115
Towards correcting input data errors probabilistically using integrity constraints
Khoussainova, N., Bałazińska, M., & Suciu, D. M. (2006) · ACM International Workshop on Data Engineering for Wireless and Mobile Access (MobiDE) · doi:10.1145/1140104.1140114 · Cited by 111
Literature Fingerprinting: A New Method for Visual Literary Analysis
Keim, D. A., & Oelke, D. (2007) · IEEE Symposium on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2007.4389004 · Cited by 110
Order dependency in the relational model
Ginsburg, S., & Hull, R. B. (1983) · Theoretical Computer Science · doi:10.1016/0304-3975(83)90084-1 · Cited by 108
Golab, L., Karloff, H. J., Korn, F. R., Saha, A., & Srivastava, D. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687627.1687693 · Cited by 108
Self-citationExtending dependencies with conditions
Bravo, L., Fan, W., & Ma, S. (2007) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 107
Metric Functional Dependencies
Koudas, N., Saha, A., Srivastava, D., et al. (2009) · Proceedings - International Conference on Data Engineering · doi:10.1109/icde.2009.219 · Cited by 95
Mining functional dependencies from data
Yao, H., & Hamilton, H. J. (2007) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-007-0083-9 · Cited by 92
On multi-column foreign key discovery
Zhang, M., Hadjieleftheriou, M., Ooi, B. C., Procopiuc, C. M., & Srivastava, D. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920944 · Cited by 91
Statistical distortion: consequences of data cleaning
Dasu, T., & Loh, J. M. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2350229.2350279 · Cited by 87
Das, A., Ng, W. K., & Woon, Y.-K. (2001) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/502585.502665 · Cited by 85
Using queries to improve database reverse engineering
Petit, J.-M., Kouloumdjian, J., Boulicaut, J., & Toumani, F. (1994) · Lecture notes in computer science · doi:10.1007/3-540-58786-1_91 · Cited by 79
Profiling and mining RDF data with ProLOD++
Abedjan, Z., Gruetze, T., Anja Jentzsch, D., & Naumann, F. (2014) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2014.6816740 · Cited by 69
Self-citationDiscovery of complex glitch patterns: A novel approach to Quantitative Data Cleaning
Berti‐Équille, L., Dasu, T., & Srivastava, D. (2011) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2011.5767864 · Cited by 63
Support the data enthusiast: challenges for next-generation data-analysis systems
Morton, K., Bałazińska, M., Grossman, D., & Mackinlay, J. D. (2014) · Proceedings of the VLDB Endowment · doi:10.14778/2732279.2732282 · Cited by 62
Information-theoretic tools for mining database structure from large data sets
Andritsos, P., Miller, R. J., & Tsaparas, P. (2004) · Proceedings - ACM-SIGMOD International Conference on Management of Data · doi:10.1145/1007568.1007650 · Cited by 55
Automatic discovery of attributes in relational databases
Zhang, M., Hadjieleftheriou, M., Ooi, B. C., Procopiuc, C. M., & Srivastava, D. (2011) · Proceedings - ACM-SIGMOD International Conference on Management of Data · doi:10.1145/1989323.1989336 · Cited by 51
Estimating the confidence of conditional functional dependencies
Cormode, G., Golab, L., Korn, F. R., McGregor, A., Srivastava, D., & Zhang, X. (2009) · Proceedings - ACM-SIGMOD International Conference on Management of Data · doi:10.1145/1559845.1559895 · Cited by 48
Self-citationDetecting inconsistencies in distributed data
Fan, W., Geerts, F., Ma, S., & Müller, H. (2010) · Proceedings - International Conference on Data Engineering · doi:10.1109/icde.2010.5447855 · Cited by 43
Golab, L., Karloff, H. J., Korn, F. R., & Srivastava, D. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1921060 · Cited by 40
Self-citationFan, W., Geerts, F., & Jia, X. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1454159.1454200 · Cited by 39
Incremental Detection of Inconsistencies in Distributed Data
Fan, W., Li, J., Tang, N., & Yu, W. (2012) · Proceedings - International Conference on Data Engineering · doi:10.1109/icde.2012.82 · Cited by 38
Byrnes, M. E. (2008) · CRC Press eBooks · doi:10.1201/9781420059151-15 · Cited by 36
GDR: a system for guided data repair
Yakout, M., Elmagarmid, A. K., Neville, J., & Ouzzani, M. (2010) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/1807167.1807325 · Cited by 34
RuleMiner: Data quality rules discovery
Chu, X., Ilyas, I. F., Papotti, P., & Ye, Y. (2014) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2014.6816746 · Cited by 30
Logical Foundations of Possibilistic Keys
Köehler, H., Leck, U., Link, S., et al. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11558-0_13 · Cited by 29
Synonym Analysis for Predicate Expansion
Abedjan, Z., & Naumann, F. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-38288-8_10 · Cited by 29
Self-citationOn repairing structural problems in semi-structured data
Korn, F. R., Saha, B., Srivastava, D., & Ying, S. (2013) · Proceedings of the VLDB Endowment · doi:10.14778/2536360.2536361 · Cited by 28
Linear correlation discovery in databases: a data mining approach
Chiang, R. H. L., Chua, C. E. H., & Lim, E. (2004) · Data & Knowledge Engineering · doi:10.1016/j.datak.2004.09.002 · Cited by 21
Efficient and Effective Analysis of Data Quality using Pattern Tableaux.
Golab, L., Korn, F. R., & Srivastava, D. (2011) · Cited by 18
Self-citation
Cited by
Information Quality Assessment for Data Fusion Systems
Becerra, M. A., Tobón, C., Castro-Ospina, A. E., & Peluffo-Ordóñez, D. H. (2021) · Data · doi:10.3390/data6060060 · Cited by 36
A Survey of Data Quality Measurement and Monitoring Tools
Ehrlinger, L., & Wöß, W. (2022) · Frontiers in Big Data · doi:10.3389/fdata.2022.850611 · Cited by 25
Metadata
- Authors
- Abedjan, Z., Golab, L., & Naumann, F.
- Venue
- The VLDB Journal
- Published
- 2015-06-02
- Volume
- 24
- Issue
- 4
- Pages
- 557-581
- ISSN
- 0949-877X (ISSN-L)
- 1066-8888
- OpenAlex
- W760598031
- Cited by
- 283
- References
- 89
- FWCI
- 33.00
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.