Robust and efficient fuzzy match for online data cleaning
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
To ensure high data quality, data warehouses must validate and cleanse incoming data tuples from external sources. In many situations, clean tuples must match acceptable tuples in reference tables. For example, product name and description fields in a sales record from a distributor must match the pre-recorded name and description fields in a product reference relation.A significant challenge in such a scenario is to implement an efficient and accurate fuzzy match operation that can effectively clean an incoming tuple if it fails to match exactly with any tuple in the reference relation. In this paper, we propose a new similarity function which overcomes limitations of commonly used similarity functions, and develop an efficient fuzzy match algorithm. We demonstrate the effectiveness of our techniques by evaluating them on real datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Baeza-Yates, R. A., & Ribeiro-Neto, B. A. (1999) · Cited by 12,609
Identification of common molecular subsequences
Smith, T. F., & Waterman, M. S. (1981) · Journal of Molecular Biology · doi:10.1016/0022-2836(81)90087-5 · Cited by 9,940
Motwani, R., & Raghavan, P. (1995) · Cambridge University Press eBooks · doi:10.1017/cbo9780511814075 · Cited by 5,123
Self-citationApproximate nearest neighbors: towards removing the curse of dimensionality
Indyk, P., & Motwani, R. (1998) · Proceedings of the thirtieth annual ACM symposium on Theory of computing - STOC '98 · doi:10.1145/276698.276876 · Cited by 4,188
Self-citationOn the resemblance and containment of documents
Broder, A. Z. (2002) · Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171) · doi:10.1109/sequen.1997.666900 · Cited by 1,698
M-tree: An Efficient Access Method for Similarity Search in Metric Spaces
Ciaccia, P., Patella, M., & Zezula, P. (1997) · Cited by 1,527
The State of Record Linkage and Current Research Problems
Winkler, W. E. (1999) · Cited by 1,042
The merge/purge problem for large databases
Hernández, M. A., & Stolfo, S. J. (1995) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/223784.223807 · Cited by 809
Interactive deduplication using active learning
Sarawagi, S., & Bhamidipaty, A. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775087 · Cited by 692
Size-Estimation Framework with Applications to Transitive Closure and Reachability
Cohen, E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1534 · Cited by 416
Cited by
Duplicate Record Detection: A Survey
Elmagarmid, A. K., Ipeirotis, P. G., & Verykios, V. S. (2007) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2007.250581 · Cited by 1,267
Sarawagi, S. (2008) · Foundations and Trends in Databases · doi:10.1561/1900000003 · Cited by 707
Collective entity resolution in relational data
Bhattacharya, I., & Getoor, L. (2007) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/1217299.1217304 · Cited by 593
Reference reconciliation in complex information spaces
Dong, X., Halevy, A., & Madhavan, J. (2005) · Proceedings of the 2005 ACM SIGMOD international conference on Management of data · doi:10.1145/1066157.1066168 · Cited by 518
Enterprise Data Analysis and Visualization: An Interview Study
Kandel, S., Paepcke, A., Hellerstein, J. M., & Heer, J. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.219 · Cited by 439
Efficient exact set-similarity joins
Arasu, A., Ganti, V., & Kaushik, R. (2006) · doi:10.5555/1182635.1164206 · Cited by 406
Self-citationEntity resolution: theory, practice & open challenges
Getoor, L., & Machanavajjhala, A. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2367502.2367564 · Cited by 348
Efficient similarity joins for near-duplicate detection
Xiao, C., Wang, W., Lin, X., Yu, J. X., & Wang, G. (2011) · ACM Transactions on Database Systems · doi:10.1145/2000824.2000825 · Cited by 331
Record linkage: similarity measures and algorithms
Koudas, N., Sarawagi, S., & Srivastava, D. (2006) · Proceedings of the 2006 ACM SIGMOD international conference on Management of data · doi:10.1145/1142473.1142599 · Cited by 289
A Latent Dirichlet Model for Unsupervised Entity Resolution
Bhattacharya, I., & Getoor, L. (2006) · Proceedings of the 2006 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972764.5 · Cited by 266
Foundations of Data Quality Management
Fan, W., & Geerts, F. (2012) · Synthesis Lectures on Data Management · doi:10.1007/978-3-031-01892-3 · Cited by 251
Iterative record linkage for cleaning and integration
Bhattacharya, I., & Getoor, L. (2004) · Proceedings of the 9th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery · doi:10.1145/1008694.1008697 · Cited by 206
“More recently, there have been adaptive approaches which make use of multiple attributes and use labeled data [24, 2, 1, 3].”
BackgroundResultOn active learning of record matching packages
Arasu, A., Götz, M., & Kaushik, R. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807252 · Cited by 180
Guiding Novice Web Workers in Making Image Descriptions Using Templates
Morash, V. S., Siu, Y.-T., Miele, J. A., Hasty, L., & Landau, S. (2015) · ACM Transactions on Accessible Computing · doi:10.1145/2764916 · Cited by 69
“In comparing two descriptions, the grams from the first description were compared to those of a second using the Jaccard similarity distance [Chaudhuri et al. 2003].”
Method
Metadata
- Authors
- Chaudhuri, S., Ganjam, K., Ganti, V., & Motwani, R.
- Venue
- Proceedings of the 2003 ACM SIGMOD international conference on Management of data
- Publisher
- ACM
- Published
- 2003-06-09
- Pages
- 313-324
- ISSN
- 0730-8078 (ISSN-L)
- OpenAlex
- W2105423800
- Cited by
- 457
- References
- 16
- FWCI
- 39.73
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.