Comparing record linkage software programs and algorithms using real-world data.
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Linkage of medical databases, including insurer claims and electronic health records (EHRs), is increasingly common. However, few studies have investigated the behavior and output of linkage software. To determine how linkage quality is affected by different algorithms, blocking variables, methods for string matching and weight determination, and decision rules, we compared the performance of 4 nonproprietary linkage software packages linking patient identifiers from noninteroperable inpatient and outpatient EHRs. We linked datasets using first and last name, gender, and date of birth (DOB). We evaluated DOB and year of birth (YOB) as blocking variables and used exact and inexact matching methods. We compared the weights assigned to record pairs and evaluated how matching weights corresponded to a gold standard, medical record number. Deduplicated datasets contained 69,523 inpatient and 176,154 outpatient records, respectively. Linkage runs blocking on DOB produced weights ranging in number from 8 for exact matching to 64,273 for inexact matching. Linkage runs blocking on YOB produced 8 to 916,806 weights. Exact matching matched record pairs with identical test characteristics (sensitivity 90.48%, specificity 99.78%) for the highest ranked group, but algorithms differentially prioritized certain variables. Inexact matching behaved more variably, leading to dramatic differences in sensitivity (range 0.04-93.36%) and positive predictive value (PPV) (range 86.67-97.35%), even for the most highly ranked record pairs. Blocking on DOB led to higher PPV of highly ranked record pairs. An ensemble approach based on averaging scaled matching weights led to modestly improved accuracy. In summary, we found few differences in the rankings of record pairs with the highest matching weights across 4 linkage packages. Performance was more consistent for exact string matching than for inexact string matching. Most methods and software packages performed similarly when comparing matching accuracy with the gold standard. In some settings, an ensemble matching approach may outperform individual linkage algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Probabilistic record linkage algorithms can help facilitate the combination of patient records across health systems and data sets, reducing gaps in missing patient data [64,65].”
From Bias in medical AI: Implications for clinical decision-making (2024) · cites this paper as [65]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444 · cited 1×
“, using the expectation-maximization [EM] algorithm [10]).”
Fellegi, I. P., & Sunter, A. B. (1969) · Journal of the American Statistical Association · doi:10.1080/01621459.1969.10501049 · Cited by 2,475 · cited 2×
“Probabilistic weight determination uses statistical modeling based on estimated probabilities that records match, given equality or similarity of linking variables [9].”
Methodvan der Laan, M. J., Polley, E. C., & Hubbard, A. E. (2007) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1309 · Cited by 1,719
Agniel, D., Kohane, I. S., & Weber, G. M. (2018) · BMJ · doi:10.1136/bmj.k1479 · Cited by 391 · cited 1×
“This specific group underscores the importance of understanding real-world healthcare practices—such as the naming of newborn infants in inpatient settings as “Male” or “Female” with mothers’ surnames—when interpreting EHR data [27].”
BackgroundData linkage: a powerful research tool with potential problems
Bohensky, M. A., Jolley, D., Sundararajan, V., Evans, S., Pilcher, D. V., Scott, I., & Brand, C. A. (2010) · BMC Health Services Research · doi:10.1186/1472-6963-10-346 · Cited by 234
Sayers, A., Ben-Shlomo, Y., Blom, A. W., & Steele, F. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyv322 · Cited by 186
Campbell, K. M., Deck, D., & Krupski, A. (2008) · Health Informatics Journal · doi:10.1177/1460458208088855 · Cited by 156
A guide to evaluating linkage quality for the analysis of linked data
Harron, K. L., Doidge, J., Knight, H. E., Gilbert, R. E., Goldstein, H., Cromwell, D. A., & van der Meulen, J. (2017) · International Journal of Epidemiology · doi:10.1093/ije/dyx177 · Cited by 156 · cited 1×
“Furthermore, evaluating linkage error using a gold standard—even an alloyed gold standard—provides important information on linkage quality [7].”
MethodData quality: A statistical perspective
Karr, A. F., Sanil, A. P., & Banks, D. L. (2005) · Statistical Methodology · doi:10.1016/j.stamet.2005.08.005 · Cited by 150 · cited 1×
“Nonetheless, in terms of the linking variables tested, both datasets seemed to be rather good quality, in part perhaps because of human health and financial incentives [24].”
BackgroundSelf-citationGUILD: GUidance for Information about Linking Data sets†
Gilbert, R. E., Lafferty, R., Hagger‐Johnson, G., Harron, K. L., Zhang, L., Smith, P. W. F., Dibben, C. J. L., & Goldstein, H. (2017) · Journal of Public Health · doi:10.1093/pubmed/fdx037 · Cited by 146
Impact of unlinked deaths and coding changes on mortality trends in the Swiss National Cohort
Schmidlin, K., Clough-Gorr, K. M., Spoerri, A., Egger, M., Zwahlen, M., & Cohort, F. T. S. N. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-1 · Cited by 128
(1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00530-7 · Cited by 123
When to conduct probabilistic linkage vs. deterministic linkage? A simulation study
Zhu, Y., Matsuyama, Y., Ohashi, Y., & Setoguchi, S. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.05.012 · Cited by 103 · cited 2×
“Prior work has shown that probabilistic approaches, which are often more time consuming, generally perform better in settings with low-quality data or high levels of missingness [8].”
BackgroundSelf-citationLariscy, J. T. (2011) · Journal of Aging and Health · doi:10.1177/0898264311421369 · Cited by 88
Patorno, E., Gopalakrishnan, C., Franklin, J. M., Brodovicz, K. G., Masso‐Gonzalez, E., Bartels, D. B., Liu, J., & Schneeweiss, S. (2018) · Diabetes, Obesity and Metabolism · doi:10.1111/dom.13184 · Cited by 84
Ford, J. B., Roberts, C. L., & Taylor, L. K. (2006) · Paediatric and Perinatal Epidemiology · doi:10.1111/j.1365-3016.2006.00715.x · Cited by 76
Aldridge, R. W., Shaji, K., Hayward, A. C., & Abubakar, I. (2015) · PLoS ONE · doi:10.1371/journal.pone.0136179 · Cited by 65
The effect of data cleaning on record linkage quality
Randall, S. M., Ferrante, A. M., Boyd, J., & Semmens, J. B. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-64 · Cited by 64 · cited 2×
“Nonetheless, such processes may come at a cost, as heavy cleaning may decrease overall linkage quality [25].”
BackgroundA New Method for Assessing How Sensitivity and Specificity of Linkage Studies Affects Estimation
Moore, C. L., Amin, J., Gidding, H. F., & Law, M. G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0103690 · Cited by 41
A transparent and transportable methodology for evaluating Data Linkage software
Ferrante, A., & Boyd, J. (2012) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2011.10.006 · Cited by 39 · cited 3×
“Another study created synthetic datasets to compare the quality and performance of 10 different linkage packages but did not examine the impact of different matching thresholds [17].”
BackgroundMethodInfluentialHagger-Johnson, G., Harron, K., Fleming, T., Gilbert, R., Goldstein, H., Landy, R., & Parslow, R. C. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-008118 · Cited by 37
The impact of record‐linkage bias in the Cox model
Baldi, I., Ponti, A., Zanetti, R., Ciccone, G., Merletti, F., & Gregori, D. (2010) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2009.01119.x · Cited by 33
The EpiLink Record Linkage Software
Tittarelli, A., Tagliabue, G., Maghini, A., Fabiano, S., Crosignani, P., Tessandori, R., & Contiero, P. (2005) · Methods of Information in Medicine · doi:10.1055/s-0038-1633924 · Cited by 32 · cited 5×
“R did not run at all for inexact string matching or for exact string matching with EpiLink weight determination, presumably because R is single-threaded and holds all objects in (real or virtual) memory.”
MethodInfluentialContiero, P., Tittarelli, A., Tagliabue, G., Maghini, A., Fabiano, S., Crosignani, P., & Tessandori, R. (2005) · PubMed · Cited by 29
Utilising identifier error variation in linkage of large administrative data sources
Harron, K. L., Hagger‐Johnson, G., Gilbert, R. E., & Goldstein, H. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0306-8 · Cited by 26 · cited 1×
“Identifying and accounting for identifier errors when linking data, especially on ZIP Code, helps to reduce bias caused by linkage errors [28].”
BackgroundPerforming studies using the UK Clinical Practice Research Datalink: to link or not to link?
McDonald, L., Schultze, A., Carroll, R., & Ramagopalan, S. V. (2018) · European Journal of Epidemiology · doi:10.1007/s10654-018-0389-5 · Cited by 26
Cornish, R., Macleod, J., Carpenter, J. R., & Tilling, K. M. (2017) · Emerging Themes in Epidemiology · doi:10.1186/s12982-017-0068-0 · Cited by 23
Research works printed but not linked: 2
- J McIllece, 2014, A simplified approach to administrative record linkage in the quarterly census of employment and wages., JSM Proceedings: Survey Research Methods, 4392
- D Krewski, 2005, The effect of record linkage errors on risk estimates in cohort mortality studies, Statistics Canada, 31, 13
Cited by
Bias in medical AI: Implications for clinical decision-making
Cross, J. L., Choma, M. A., & Onofrey, J. A. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000651 · Cited by 607
Dagan, N., Elnekave, E., Barda, N., Bregman-Amitai, O., Bar, A., Orlovsky, M., Bachmat, E., & Balicer, R. D. (2020) · Nature Medicine · doi:10.1038/s41591-019-0720-z · Cited by 113
Asher, J., Resnick, D. M., Brite, J., Brackbill, R. M., & Cone, J. E. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17186937 · Cited by 37
“[32] reviewed multiple software options for record linkage: R (Version 3.”
MethodA Context-Independent Ontological Linked Data Alignment Approach to Instance Matching
Barbosa, A., Bittencourt, I. I., Siqueira, S. W. M., Dermeval, D., & da Cruz, N. J. T. (2022) · International Journal on Semantic Web and Information Systems · doi:10.4018/ijswis.295977 · Cited by 24
“In Databases, the problem is known by record linkage (Gu et al., 2003; Karr et al., 2019), which aims to identify and link resources that are judged to represent the same real-world entity.”
BackgroundTahamont, S., Jelveh, Z., Chalfin, A. J., Yan, S., & Hansen, B. (2020) · Journal of Quantitative Criminology · doi:10.1007/s10940-020-09461-x · Cited by 25
Metadata
- Authors
- Karr, A. F., Taylor, M. T., West, S. L., Setoguchi, S., Kou, T. D., Gerhard, T., & Horton, D. B.
- Affiliations
- RTI International (US)
- Rutgers, The State University of New Jersey (US)
- Bristol-Myers Squibb (United States) (US)
- Corresponding author
- Karr, Alan F.
- ORCID
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2019-09-24
- Volume
- 14
- Issue
- 9
- Pages
- e0221459
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC6759179
- OpenAlex
- W2976642796
- Cited by
- 27
- References
- 27
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 6 · 2023: 2 · 2022: 5 · 2021: 6 · 2020: 4 · 2019: 1
- FWCI
- 2.86
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- The Medicine Forum (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Bristol-Myers Squibb
- Topics
- Physical Sciences › Computer Science › Information Systems › Data Quality and Management
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- data linkage, electronic health records, string matching, linkage quality
- MeSH terms
- Algorithms, Humans, Medical Record Linkage, Software, Databases, Factual, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.