The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Interactive deduplication using active learning
Abstract
Deduplication is a key operation in integrating data from multiple sources. The main challenge in this task is designing a function that can resolve when a pair of records refer to the same entity in spite of various data inconsistencies. Most existing systems use hand-coded functions. One way to overcome the tedium of hand-coding is to train a classifier to distinguish between duplicates and non-duplicates. The success of this method critically hinges on being able to provide a covering and challenging set of training pairs that bring out the subtlety of deduplication function. This is non-trivial because it requires manually searching for various data inconsistencies between any two records spread apart in large lists.We present our design of a learning-based deduplication system that uses a novel method of interactively discovering challenging training pairs using active learning. Our experiments on real-life datasets show that active learning significantly reduces the number of instances needed to achieve high accuracy. We investigate various design issues that arise in building a system to provide interactive response, fast convergence, and interpretable output.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1992) · Cited by 23,431
A Tutorial on Support Vector Machines for Pattern Recognition
Burges, C. J. C. (1998) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009715923555 · Cited by 16,429
Data mining and knowledge discovery: making sense out of data
Feyyad, U. M. (1996) · IEEE Expert · doi:10.1109/64.539013 · Cited by 4,612
A TUTORIAL ON SUPPORT VECTOR MACHINES FOR PATTERN RECOGNITION, DATA MINING AND KNOWLEDGE DISCOVERY
Burger, C. J. C. (1998) · Cited by 3,313
A guided tour to approximate string matching
Navarro, G. (2001) · ACM Computing Surveys · doi:10.1145/375360.375365 · Cited by 2,561
Seung, H. S., Opper, M., & Sompolinsky, H. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130417 · Cited by 1,651
Improving generalization with active learning
Cohn, D., Atlas, L., & Ladner, R. (1994) · Machine Learning · doi:10.1007/bf00993277 · Cited by 1,591
Selective Sampling Using the Query by Committee Algorithm
Freund, Y., Seung, H. S., Shamir, E., & Tishby, N. (1997) · Machine Learning · doi:10.1023/a:1007330508534 · Cited by 1,097
Efficient clustering of high-dimensional data sets with application to reference matching
McCallum, A., Nigam, K., & Ungar, L. H. (2000) · Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/347090.347123 · Cited by 1,096
The State of Record Linkage and Current Research Problems
Winkler, W. E. (1999) · Cited by 1,015
Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem
Hernández, M. A., & Stolfo, S. J. (1998) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009761603038 · Cited by 812
Less is More: Active Learning with Support Vector Machines
Schohn, G., & Cohn, D. A. (2000) · Cited by 773
Employing EM and Pool-Based Active Learning for Text Classification
McCallum, A., & Nigam, K. (1998) · Cited by 689
Digital libraries and autonomous citation indexing
Lawrence, S., Lee Giles, C., & Bollacker, K. (1999) · Computer · doi:10.1109/2.769447 · Cited by 607
Potter's Wheel: An Interactive Data Cleaning System
Raman, V., & Hellerstein, J. L. (2001) · Cited by 492
The field matching problem: Algorithms and applications
Monge, A. E., & Elkan, C. (1996) · Cited by 471
Learning and making decisions when costs and probabilities are both unknown
Zadrozny, B., & Elkan, C. (2001) · Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/502512.502540 · Cited by 445
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/15324430152733142 · Cited by 385
Declarative Data Cleaning: Language, Model, and Algorithms
Galhardas, H., Florescu, D., Shasha, D. E., Simon, E., & Saita, C.-A. (2001) · Cited by 320
The Effect of Adding Relevance Information in a Relevance Feedback Environment
Buckley, C., Salton, G., & Allan, J. (1994) · SIGIR ’94 · doi:10.1007/978-1-4471-2099-5_30 · Cited by 302
Data Mining Using MLC a Machine Learning Library in C
Kohavi, R., Sommerfield, D., & Dougherty, J. (1997) · Cited by 298
Winkler, W. E. (1995) · Wiley series in probability and statistics · doi:10.1002/9781118150504.ch20 · Cited by 222
Automatic segmentation of text into structured records
Borkar, V., Deshmukh, K., & Sarawagi, S. (2001) · Proceedings of the 2001 ACM SIGMOD international conference on Management of data · doi:10.1145/375663.375682 · Cited by 217
Self-citationA probability analysis on the value of unlabeled data for classification problems
Zhang, T., & Oles, F. J. (2000) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 202
Committee-Based Sample Selection for Probabilistic Classifiers
Argamon, S., & Dagan, I. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.612 · Cited by 164
Active Learning with Committees for Text Categorization
Liere, R., & Tadepalli, P. V. (1997) · Cited by 146
Data mining using /spl Mscr/ℒ/spl Cscr/++ a machine learning library in C++
Kohavi, R., Sommerfield, D. A., & Dougherty, J. (2005) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/tai.1996.560457 · Cited by 146
Active learning using adaptive resampling
Iyengar, V. S., Apte, C. V., & Zhang, T. (2000) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/347090.347110 · Cited by 90
Improving Short-Text Classification using Unlabeled Data for Classification Problems
Zelikovitz, S., & Hirsh, H. (2000) · International Conference on Machine Learning · Cited by 47
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/153244302760185243
Cited by
Chen, M., Mao, S., & Liu, Y. (2014) · Mobile Networks and Applications · doi:10.1007/s11036-013-0489-0 · Cited by 2,913
Duplicate Record Detection: A Survey
Elmagarmid, A. K., Ipeirotis, P. G., & Verykios, V. S. (2007) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2007.250581 · Cited by 1,267
Toward Scalable Systems for Big Data Analytics: A Technology Tutorial
Hu, H., Wen, Y., Chua, T.-S., & Li, X. (2014) · IEEE Access · doi:10.1109/access.2014.2332453 · Cited by 1,099
Adaptive duplicate detection using learnable string similarity measures
Bilenko, M., & Mooney, R. J. (2003) · Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/956750.956759 · Cited by 926
Sarawagi, S. (2008) · Foundations and Trends in Databases · doi:10.1561/1900000003 · Cited by 707
Self-citation
Metadata
- Authors
- Sarawagi, S., & Bhamidipaty, A.
- Venue
- Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining
- Publisher
- ACM
- Published
- 2002-07-23
- Pages
- 269-278
- OpenAlex
- W2067566391
- Cited by
- 692
- References
- 30
- FWCI
- 27.46
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.