The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Interactive deduplication using active learning
Abstract
Deduplication is a key operation in integrating data from multiple sources. The main challenge in this task is designing a function that can resolve when a pair of records refer to the same entity in spite of various data inconsistencies. Most existing systems use hand-coded functions. One way to overcome the tedium of hand-coding is to train a classifier to distinguish between duplicates and non-duplicates. The success of this method critically hinges on being able to provide a covering and challenging set of training pairs that bring out the subtlety of deduplication function. This is non-trivial because it requires manually searching for various data inconsistencies between any two records spread apart in large lists.We present our design of a learning-based deduplication system that uses a novel method of interactively discovering challenging training pairs using active learning. Our experiments on real-life datasets show that active learning significantly reduces the number of instances needed to achieve high accuracy. We investigate various design issues that arise in building a system to provide interactive response, fast convergence, and interpretable output.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1992) · Cited by 23,431
A Tutorial on Support Vector Machines for Pattern Recognition
Burges, C. J. C. (1998) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009715923555 · Cited by 16,429
Data mining and knowledge discovery: making sense out of data
Feyyad, U. M. (1996) · IEEE Expert · doi:10.1109/64.539013 · Cited by 4,612
A TUTORIAL ON SUPPORT VECTOR MACHINES FOR PATTERN RECOGNITION, DATA MINING AND KNOWLEDGE DISCOVERY
Burger, C. J. C. (1998) · Cited by 3,313
A guided tour to approximate string matching
Navarro, G. (2001) · ACM Computing Surveys · doi:10.1145/375360.375365 · Cited by 2,561
Seung, H. S., Opper, M., & Sompolinsky, H. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130417 · Cited by 1,651
Improving generalization with active learning
Cohn, D., Atlas, L., & Ladner, R. (1994) · Machine Learning · doi:10.1007/bf00993277 · Cited by 1,591
Selective Sampling Using the Query by Committee Algorithm
Freund, Y., Seung, H. S., Shamir, E., & Tishby, N. (1997) · Machine Learning · doi:10.1023/a:1007330508534 · Cited by 1,097
Efficient clustering of high-dimensional data sets with application to reference matching
McCallum, A., Nigam, K., & Ungar, L. H. (2000) · Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/347090.347123 · Cited by 1,096
The State of Record Linkage and Current Research Problems
Winkler, W. E. (1999) · Cited by 1,015
Cited by
Chen, M., Mao, S., & Liu, Y. (2014) · Mobile Networks and Applications · doi:10.1007/s11036-013-0489-0 · Cited by 2,913
Duplicate Record Detection: A Survey
Elmagarmid, A. K., Ipeirotis, P. G., & Verykios, V. S. (2007) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2007.250581 · Cited by 1,267
Toward Scalable Systems for Big Data Analytics: A Technology Tutorial
Hu, H., Wen, Y., Chua, T.-S., & Li, X. (2014) · IEEE Access · doi:10.1109/access.2014.2332453 · Cited by 1,099
Adaptive duplicate detection using learnable string similarity measures
Bilenko, M., & Mooney, R. J. (2003) · Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/956750.956759 · Cited by 926
Sarawagi, S. (2008) · Foundations and Trends in Databases · doi:10.1561/1900000003 · Cited by 707
Self-citationA Survey of Indexing Techniques for Scalable Record Linkage and Deduplication
Christen, P. (2012) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2011.127 · Cited by 679
Collective entity resolution in relational data
Bhattacharya, I., & Getoor, L. (2007) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/1217299.1217304 · Cited by 593
(2006) · doi:10.5555/1182635.1164130 · Cited by 553
Dong, X. L., & Srivastava, D. (2015) · Synthesis Lectures on Data Management · doi:10.1007/978-3-031-01853-4 · Cited by 539
Reference reconciliation in complex information spaces
Dong, X., Halevy, A., & Madhavan, J. (2005) · Proceedings of the 2005 ACM SIGMOD international conference on Management of data · doi:10.1145/1066157.1066168 · Cited by 518
CrowdER: crowdsourcing entity resolution
Wang, J., Kraska, T., Franklin, M. J., & Feng, J. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2350229.2350263 · Cited by 507
Robust and efficient fuzzy match for online data cleaning
Chaudhuri, S., Ganjam, K., Ganti, V., & Motwani, R. (2003) · Proceedings of the 2003 ACM SIGMOD international conference on Management of data · doi:10.1145/872757.872796 · Cited by 457
Efficient similarity joins for near duplicate detection
Xiao, C., Wang, W., Lin, X., & Yu, J. X. (2008) · Proceedings of the 17th international conference on World Wide Web · doi:10.1145/1367497.1367516 · Cited by 455
Semantic-integration research in the database community: A brief survey
Doan, A., & Halevy, A. Y. (2005) · doi:10.1609/aimag.v26i1.1801 · Cited by 421
Efficient exact set-similarity joins
Arasu, A., Ganti, V., & Kaushik, R. (2006) · doi:10.5555/1182635.1164206 · Cited by 406
Entity Resolution with Markov Logic
Singla, P., & Domingos, P. (2006) · Sixth International Conference on Data Mining (ICDM'06) · doi:10.1109/icdm.2006.65 · Cited by 359
Entity resolution: theory, practice & open challenges
Getoor, L., & Machanavajjhala, A. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2367502.2367564 · Cited by 348
Principles of dataspace systems
Halevy, A., Franklin, M., & Maier, D. (2006) · Proceedings of the twenty-fifth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems · doi:10.1145/1142351.1142352 · Cited by 346
Deep entity matching with pre-trained language models
Li, Y., Li, J., Suhara, Y., Doan, A., & Tan, W.-C. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3421424.3421431 · Cited by 339
Kandel, S., Heer, J., Plaisant, C., Kennedy, J., van Ham, F., Riche, N. H., Weaver, C., Lee, B., Brodbeck, D., & Buono, P. (2011) · Information Visualization · doi:10.1177/1473871611415994 · Cited by 332
Efficient similarity joins for near-duplicate detection
Xiao, C., Wang, W., Lin, X., Yu, J. X., & Wang, G. (2011) · ACM Transactions on Database Systems · doi:10.1145/2000824.2000825 · Cited by 331
Efficient k-Anonymization Using Clustering Techniques
Byun, J.-W., Kamra, A., Bertino, E., & Li, N. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-71703-4_18 · Cited by 329
Crowdsourced Data Management: A Survey
Li, G., Wang, J., Zheng, Y., & Franklin, M. J. (2016) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2016.2535242 · Cited by 283
An interactive clustering-based approach to integrating source query interfaces on the deep Web
Wu, W., Yu, C., Doan, A., & Meng, W. (2004) · Proceedings of the 2004 ACM SIGMOD international conference on Management of data · doi:10.1145/1007568.1007582 · Cited by 273
Similarity encoding for learning with dirty categorical variables
Cerda, P., Varoquaux, G., & Kégl, B. (2018) · Machine Learning · doi:10.1007/s10994-018-5724-2 · Cited by 267
A Latent Dirichlet Model for Unsupervised Entity Resolution
Bhattacharya, I., & Getoor, L. (2006) · Proceedings of the 2006 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972764.5 · Cited by 266
Magellan: toward building entity matching management systems
Konda, P., Das, S., Suganthan G. C., P., Doan, A., Ardalan, A., Ballard, J. R., Li, H., Panahi, F., Zhang, H., Naughton, J., et al. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/2994509.2994535 · Cited by 262
Purpose based access control for privacy protection in relational database systems
Byun, J.-W., & Li, N. (2008) · The VLDB Journal · doi:10.1007/s00778-006-0023-0 · Cited by 259
Rule-based deduplication of article records from bibliographic databases
Jiang, Y., Lin, C., Meng, W., Yu, C., Cohen, A. M., & Smalheiser, N. R. (2014) · Database · doi:10.1093/database/bat086 · Cited by 41
The Effect of Suspicious Profiles on People Recommenders
Pizzato, L. A., Akehurst, J., Silvestrini, C., Yacef, K., Koprinska, I., & Kay, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-31454-4_19 · Cited by 8
Metadata
- Authors
- Sarawagi, S., & Bhamidipaty, A.
- Venue
- Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining
- Publisher
- ACM
- Published
- 2002-07-23
- Pages
- 269-278
- OpenAlex
- W2067566391
- Cited by
- 692
- References
- 30
- FWCI
- 27.46
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.