The DOI matches its Crossref record.
A Survey of Indexing Techniques for Scalable Record Linkage and Deduplication
Abstract
Record linkage is the process of matching records from several databases that refer to the same entities. When applied on a single database, this process is known as deduplication. Increasingly, matched data are becoming important in many application areas, because they can contain information that is not available otherwise, or that is too costly to acquire. Removing duplicate records in a single database is a crucial step in the data cleaning process, because duplicates can severely influence the outcomes of any subsequent data processing or data mining. With the increasing size of today's databases, the complexity of the matching process becomes one of the major challenges for record linkage and deduplication. In recent years, various indexing techniques have been developed for record linkage and deduplication. They are aimed at reducing the number of record pairs to be compared in the matching process by removing obvious nonmatching pairs, while at the same time maintaining high matching quality. This paper presents a survey of 12 variations of 6 indexing techniques. Their complexity is analyzed, and their performance and scalability is evaluated within an experimental framework using both synthetic and real data sets. No such detailed survey has so far been published.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions
Shen, W., Wang, J., & Han, J. (2015) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2014.2327028 · Cited by 761
CrowdER: crowdsourcing entity resolution
Wang, J., Kraska, T., Franklin, M. J., & Feng, J. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2350229.2350263 · Cited by 507
Deep entity matching with pre-trained language models
Li, Y., Li, J., Suhara, Y., Doan, A., & Tan, W.-C. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3421424.3421431 · Cited by 339
Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow
Wongsuphasawat, K., Smilkov, D., Wexler, J., Wilson, J., Mane, D., Fritz, D., Krishnan, D., Viegas, F. B., & Wattenberg, M. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744878 · Cited by 331
A taxonomy of privacy-preserving record linkage techniques
Vatsalan, D., Christen, P., & Verykios, V. S. (2013) · Information Systems · doi:10.1016/j.is.2012.11.005 · Cited by 295
Self-citationSimilarity encoding for learning with dirty categorical variables
Cerda, P., Varoquaux, G., & Kégl, B. (2018) · Machine Learning · doi:10.1007/s10994-018-5724-2 · Cited by 267
Magellan: toward building entity matching management systems
Konda, P., Das, S., Suganthan G. C., P., Doan, A., Ardalan, A., Ballard, J. R., Li, H., Panahi, F., Zhang, H., Naughton, J., et al. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/2994509.2994535 · Cited by 262
A note on using the F-measure for evaluating record linkage algorithms
Hand, D., & Christen, P. (2018) · Statistics and Computing · doi:10.1007/s11222-017-9746-6 · Cited by 247
Self-citationOptimization of the mainzelliste software for fast privacy-preserving record linkage
Rohde, F., Franke, M., Sehili, Z., Lablans, M., & Rahm, E. (2021) · Journal of Translational Medicine · doi:10.1186/s12967-020-02678-1 · Cited by 14
A Survey on Truth Discovery: Concepts, Methods, Applications, and Opportunities
Wang, S., Zhang, H., Sheng, Q. Z., Li, X., Sun, Z., Cai, T., Zhang, W. E., Yang, J., & Gao, Q. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2024.3423677 · Cited by 11
Metadata
- Authors
- Christen, P.
- Venue
- IEEE Transactions on Knowledge and Data Engineering
- Published
- 2012-09
- Volume
- 24
- Issue
- 9
- Pages
- 1537-1555
- ISSN
- 1041-4347 (ISSN-L)
- OpenAlex
- W2031250218
- Cited by
- 679
- References
- 0
- FWCI
- 51.59
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.